Image labeling method for fruit and vegetable grading recognition deep learning model training sample

Abstract

The invention discloses an image labeling method for a fruit and vegetable grading recognition deep learning model training sample. The method comprises the following steps: obtaining a fruit and vegetable image; the obtained fruit and vegetable images are preprocessed; cutting the preprocessed image into small blocks to obtain a plurality of block images of the small blocks; performing integral labeling on the block images, wherein the block images without fruit and vegetable defects and black spots are labeled as a first type; the block images with fruit and vegetable defects are marked as a second class; and marking the block images without fruit and vegetable defects but with black spots as a third class. According to the method, labeling is carried out by taking each block image as a unit, in the labeling process, fruit and vegetable defects, black spots or fruit stems do not need to be searched and labeled on the images one by one, and only qualitative judgment and labeling are needed to be carried out on whether the block images have the fruit and vegetable defects or black spots or not, so that the labeling steps are remarkably simplified, the labeling workload is reduced, and the labeling efficiency is improved. The labeling work intensity is reduced, and meanwhile, the labeling efficiency is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic fruit and vegetable grading, and particularly relates to a fruit and vegetable grading and recognition deep learning model training sample image labeling method. BACKGROUND

[0002] In recent years, many scholars at home and abroad have researched agricultural product quality detection methods based on machine vision. However, the traditional machine learning method has problems such as the need for human detailed description knowledge assistance, human interference factors, and the like in grading fruit and vegetable surface quality, and has disadvantages such as long calculation time and low accuracy in processing a large amount of data.

[0003] With the development of science and technology, deep learning has gradually emerged and gradually applied to image recognition. The complex network structure and data training model not only reduce the workload, but also improve the detection speed and accuracy. As one of the work contents of deep learning, labeling directly determines the accuracy of the model after training. In fruit and vegetable grading, the labeling work of the existing model training is basically completed by artificial, which is very time-consuming and laborious, especially for some fruits and vegetables such as gold orange and pear, in addition to fruit defects such as scab and bruise, the fruit stem and a large number of black spots are also distributed. Whether it is a fruit stem or a black spot, it is easy to be misjudged as a fruit defect during labeling. In order to achieve high recognition accuracy, a large number of black spot samples are also needed during model training. Therefore, it is necessary to distinguish and label each point during the labeling process, which greatly increases the labeling workload and difficulty, and makes the labeling work very intensive and laborious, and the labeling efficiency is low, and the labeling accuracy is also low, which affects the model training result. SUMMARY

[0004] The purpose of the present application is to provide a fruit and vegetable grading and recognition deep learning model training sample image labeling method.

[0005] The technical solution for achieving the purpose of the present application is: a fruit and vegetable grading and recognition deep learning model training sample image labeling method, comprising the following steps: S1. obtaining a fruit and vegetable image; S2. pre-processing the fruit and vegetable image obtained in step S1, and the pre-processing operation includes image segmentation processing: extracting the fruit and vegetable part of interest in the image, and removing the background part useless for image recognition; S3. cutting the image after the pre-processing in step S2 into small pieces to obtain a plurality of small piece cutting images; S4. performing overall labeling on the cutting images in step S3, wherein: the cutting image without fruit and vegetable defects and black spots is labeled as the first type; the cutting image with fruit and vegetable defects is labeled as the second type; The third category is marked for the image of the cut piece of the fruitless vegetable with defects but with black spots; The image with one or more color blocks showing gray is an image with black spots, and vice versa, the image without any color block showing gray is an image without black spots; the image with any one of scabbing, wrinkling, pricking and bruising is an image with fruit and vegetable defects, and vice versa, the image without any fruit and vegetable defects including scabbing, wrinkling, pricking and bruising is an image without fruit and vegetable defects.

[0006] Further, the step of cutting the image preprocessed in step S2 into small pieces in step S3 is to perform a class sector segmentation on the image preprocessed in step S2 with the center point of the image as the center of the circle.

[0007] Further, the step of cutting the image preprocessed in step S2 into small pieces in step S3 is to cut the image preprocessed in step S2 into pieces in the form of a 256*256 matrix.

[0008] Further, the specific step of cutting the image preprocessed in step S2 into small pieces in step S3 is to use python language and call OpenCV machine vision library to program and realize the automatic cutting function of the image.

[0009] Further, the image segmentation processing method in step S2 is any one of the threshold-based segmentation method, the region-based segmentation method, the edge-based segmentation method and the superpixel segmentation method.

[0010] Further, the image segmentation processing method in step S2 is the threshold-based segmentation method, and the threshold-based segmentation method adopts the H component threshold segmentation method based on the HSV model.

[0011] Further, the preprocessing operation in step S2 further includes a binarization operation and a noise reduction processing, and the image segmentation processing, the binarization operation and the noise reduction processing are sequentially performed.

[0012] Further, the noise reduction processing is performed by judging the area size. In the noise reduction processing, the expansion and contraction noise reduction method can be used, or the noise reduction can be performed by judging the area size.

[0013] Further, step S1 is to obtain fruit and vegetable images from multiple angles through multiple cameras, and after obtaining the fruit and vegetable images, the number of images is increased through a data enhancement method, and the data enhancement method includes a cutting, flipping and random cropping method.

[0014] Further, the fruit and vegetable defects in step S4 are any one of scabbing, wrinkling, pricking or bruising.

[0015] The image labeling method for training samples of the fruit and vegetable grading and recognition deep learning model of the present application labels all contents in the cut block image region as the labeling target and performs overall labeling on the cut block image. Unlike the traditional method, the labeling method no longer takes the fruit and vegetable defects, black spots or fruit stems on the fruit and vegetable image as the labeling target, but labels each cut block image. In the labeling process, it is not necessary to find and label the fruit and vegetable defects, black spots or fruit stems one by one on the image, but only to qualitatively determine and label whether the cut block image has fruit and vegetable defects or black spots. This labeling method avoids the cumbersome process of "quantitative" labeling of fruit and vegetable defects, black spots or fruit stems in the traditional method, significantly simplifies the labeling steps, reduces the labeling workload and labor intensity, and significantly improves the labeling efficiency.

[0016] The number of fruit stems on the fruit and vegetable is one, the amount is small, and the image features of the fruit stem are generally uniform, which has less interference on the fruit and vegetable defect recognition. In contrast, the number of black spots on the fruit and vegetable is indefinite, usually more, and the amount is large, and the image features of the black spots are diverse in shape, which has greater interference on the fruit and vegetable defect recognition. In combination with these characteristics of the fruit and vegetable, the image labeling method for training samples of the fruit and vegetable grading and recognition deep learning model proposed by the present application, in the case that the fruit and vegetable image simultaneously has three characteristics of fruit and vegetable defects, black spots or fruit stems, selects "fruit and vegetable defects" and "black spots" as two core judgment indexes when labeling the cut block image, and divides the labeling results into three categories. On the one hand, "fruit stem" is not selected as a judgment index because the presence or absence of the fruit stem does not affect the quality of the fruit and vegetable, so its absence will not affect the labeling classification result. This design not only meets the actual needs of fruit and vegetable quality evaluation, but also avoids the complexity caused by labeling "fruit stem". Compared with the classification method using three judgment indexes of "fruit and vegetable defects", "black spots" and "fruit stem", the present application only uses two judgment indexes of "fruit and vegetable defects" and "black spots", which significantly reduces the complexity of labeling, simplifies the process of manual labeling and improves the labeling efficiency. On the other hand, in the labeling classification, the cut block image without fruit and vegetable defects but with black spots is separately divided into a category, which is specially used for the identification and training of black spots by the deep learning model. This design enables the model to learn and train black spots individually, thereby effectively avoiding the misjudgment of black spots as fruit and vegetable defects by the model, significantly improving the recognition accuracy of black spots by the model and further improving the overall precision and reliability of fruit and vegetable grading and recognition. Through this optimization, the present application ensures the labeling efficiency while taking into account the accuracy and practicality of model training.

[0017] The present application breaks through the inherent mode of traditional point-by-point labeling, realizes "one-time labeling, full-factor coverage", and greatly improves the efficiency and economy of data labeling while ensuring the accuracy of subsequent model detection. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of the image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the present application; Figure 2 is the original image of the fruit and vegetable image in the image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the present application; Figure 3 is Figure 2 the image obtained after image segmentation processing; Figure 4 is Figure 3 the image obtained after binarization operation; Figure 5 is Figure 4 the image obtained after noise reduction processing; Figure 6 is Figure 2 the image obtained after pretreatment; Figure 7 is the image obtained after the pretreated image is cut into pieces in matrix form; Figure 8 is a comparison chart of the recognition accuracy of ResNet50, VGG16 and AlexNet three models under different cutting piece quantities in matrix form; Figure 9 is a schematic diagram of class fan-shaped segmentation of the pretreated image; Figure 10 is a single piece cutting image obtained after class fan-shaped segmentation of the pretreated image. DETAILED DESCRIPTION

[0019] The preferred embodiment of the image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the present application will be described in detail below in combination with the drawings: As shown in Figure 1 , an image labeling method of a fruit and vegetable grading recognition deep learning model training sample, taking gold orange as an example, includes the following steps: S1. Obtain a fruit and vegetable image; S2. Pretreat the fruit and vegetable image obtained in step S1, and the pretreatment operation includes image segmentation processing: extract the fruit and vegetable part of interest in the image, and remove the background part useless for image recognition; S3. Cut the image pretreated in step S2 into small pieces to obtain a plurality of small piece cutting images; S4. Label the cutting image in step S3 as a whole, wherein: Images of cut fruits and vegetables without defects or black spots are labeled as Category 1 and marked as "good"; Images of cut fruits and vegetables with defects are labeled as Category 2 and marked as "bad"; Images of cut pieces of fruits and vegetables without defects but with black spots are labeled as Category 3 and denoted as "black spots"; An image with one or more grayscale patches is considered an image with black spots; conversely, an image without any grayscale patches is considered an image without black spots. An image with any of the following fruit and vegetable defects—scabs, wrinkles, punctures, and bruises—is considered an image with fruit and vegetable defects; conversely, an image without any of these defects is considered an image without fruit and vegetable defects.

[0020] This invention provides an image annotation method for training samples in a deep learning model for fruit and vegetable grading recognition. The method annotates fruit and vegetable images so that the annotated images can be used as training samples to train a deep learning model for fruit and vegetable grading recognition. In the fruit and vegetable image annotation process, images of the fruits and vegetables are first captured using a camera. The acquired fruit and vegetable images are shown below. Figure 2 As shown.

[0021] This invention discloses an image annotation method for training samples of a deep learning model for fruit and vegetable grading recognition. To obtain complete features of fruits and vegetables and improve the accuracy of grading recognition, images of fruits and vegetables can be acquired using multiple cameras from multiple angles, such as using three or more cameras to capture images from the top and sides. After acquiring the images, data augmentation processing can be performed, including but not limited to image cropping, flipping, and random cropping, to increase the diversity of training samples and improve the accuracy of model training. The data augmentation methods include cropping, flipping, and random cropping.

[0022] This invention provides an image annotation method for training samples of a deep learning model for fruit and vegetable grading recognition, such as... Figure 2 In the obtained fruit and vegetable images shown, the parts to be labeled are the fruit and vegetable portions, i.e. Figure 2 The yellow kumquat portion is highlighted, while the background outside the fruit and vegetable area is irrelevant and should not be included in the labeling process. After acquiring the fruit and vegetable images, preprocessing is performed, including image segmentation. The segmented image is shown below. Figure 3 As shown, only the fruit and vegetable portion is retained, while the background portion is replaced with black.

[0023] The image labeling method for training samples of the fruit and vegetable grading recognition deep learning model of the present application extracts the fruit and vegetable part of interest in the image through segmentation, removes the background part useless for image recognition, focuses the segmented image on the fruit and vegetable part, avoids background interference, ensures the high quality of the training samples, provides reliable data basis for model training, and enables the model to focus more on the extraction of fruit and vegetable features, thereby improving the accuracy of the learning model training grading recognition.

[0024] The image segmentation processing method in step S2 of the image labeling method for training samples of the fruit and vegetable grading recognition deep learning model of the present application can be but is not limited to threshold-based segmentation, region-based segmentation, edge-based segmentation or superpixel segmentation.

[0025] The image segmentation processing method in step S2 of the image labeling method for training samples of the fruit and vegetable grading recognition deep learning model of the present application is preferably threshold-based segmentation. Compared with region-based segmentation, edge-based segmentation and superpixel segmentation, threshold-based segmentation only needs to calculate the gray value of the pixel, without complex region or edge analysis, the algorithm is simple, the processing speed is relatively fast, the calculation efficiency is high, the image recognition time can be greatly shortened and the calculation resources can be saved; and the method requires less calculation resources, and can run on devices with limited resources (such as embedded systems); in addition, it mainly depends on threshold selection, involves fewer parameters, and is easy to adjust and optimize.

[0026] The threshold-based segmentation method can be specifically operated by graying the image and then performing threshold segmentation processing through a gray histogram. This method divides the pixels in the image into foreground (fruit and vegetable part) and background by setting one or more gray thresholds. The specific implementation process is as follows: according to the color features (such as the yellow color of gold orange) of the fruit and vegetable image, one or more gray thresholds are selected, the threshold can be manually set, automatically determined based on histogram analysis or adaptive algorithm (such as Otsu algorithm); compare the gray value of each pixel in the image with the threshold, if the pixel gray value is greater than the threshold, classify it as foreground (fruit and vegetable part); otherwise, classify it as background; set the pixel value of the background part to a fixed value (such as 0 or 255).

[0027] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the present application can also be based on the HSV model to realize threshold segmentation of a certain component in the specific operation of the threshold-based segmentation method. Specifically, the original RGB image is converted into an HSV image, the HSV image can better separate color information and reduce the influence of light changes on the segmentation result; the HSV image includes hue (Hue), saturation (Saturation), and brightness (Value) three components, and the threshold segmentation of the hue (Hue) component, i.e., the H component, is taken as an example, the H component (hue) is extracted from the HSV image, the value range of the H component is usually 0° to 360°, which represents the type of color (such as red, green, blue, etc.); according to the color characteristics of fruits and vegetables, the threshold range of the H component is set, for example, for the yellow Jincheng, the threshold range of the H component can be set to 20° to 40°; each pixel in the image is traversed, and it is judged whether the H component is within the set threshold range, if the H component is within the threshold range, the pixel is classified as foreground (fruit and vegetable part), if the H component is not within the threshold range, the pixel is classified as background; the pixel value of the background part is set to a fixed value (such as 0 or 255).

[0028] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the present application converts the image into a gray image, and then processes the threshold segmentation through a gray histogram, in the process of converting the image into a gray image, the color information is lost, only the brightness information is relied on, the processing is sensitive to light changes, the brightness changes will directly affect the gray value, resulting in unstable segmentation effect, and it is difficult to process scenes with complex colors or overlapping brightness distributions. When threshold segmentation is performed through the HSV model, the color information is retained, the segmentation can be performed through the hue (Hue), saturation (Saturation), and brightness (Value) three components, and the HSV model separates the brightness (Value) from the color information (Hue and Saturation), which can reduce the influence of light changes by adjusting the hue and saturation; in addition, different components can be selected for segmentation, which is more flexible and can accurately segment the target through color characteristics.

[0029] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the present application utilizes the HSV model to perform image segmentation processing, among the hue (Hue), saturation (Saturation), and brightness (Value) three components, the H component can effectively distinguish fruits and vegetables of different colors, and is particularly suitable for scenes with obvious color characteristics (such as yellow Jincheng and red apples). Therefore, the image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the present application, in combination with the bright color characteristics in the Jincheng image, preferably adopts the H component threshold segmentation method based on the HSV model based on the threshold-based segmentation method.

[0030] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, preferably, the preprocessing operation in step S2 further includes binarization operation and noise reduction processing, and the three operations are sequentially performed in the order of image segmentation processing binarization operation noise reduction processing to ensure the accuracy and robustness of the segmentation result. Specifically, the binarization operation is performed on the segmented image, and the image is converted into a black and white binary image. As shown in Figure 4 , the purpose of the binarization operation is to further clarify the boundaries of the foreground and background, so as to facilitate the subsequent noise reduction processing. Then, the binarized image is subjected to noise reduction processing to remove small noise points in the image. As shown in Figure 5 , the purpose of the noise reduction processing is to improve the image quality and reduce the interference of noise points on subsequent labeling and model training. After sequentially processing in the order of image segmentation processing binarization operation noise reduction processing, the finally obtained image after preprocessing is as shown in Figure 6 .

[0031] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, by binarization operation, the boundaries of the fruit and vegetable part and the background part are clarified, and the influence of the fuzzy area on the segmentation result is avoided. By noise reduction processing, small noise points are removed to ensure the cleanliness and integrity of the segmentation result and enhance the image quality. The high-quality image segmentation result after binarization operation and noise reduction processing provides a reliable data basis for subsequent labeling and model training.

[0032] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, in the noise reduction processing, the inflation and contraction noise reduction method can be used, or the noise can be reduced by judging the area size. Among them, the inflation and contraction noise reduction method is to inflate and contract the image through morphological operation (such as opening operation and closing operation) to remove small noise points or fill small holes. The noise reduction method based on area size is to analyze the connected regions, count the area of each connected region in the image, and select the target region and noise region according to the set area threshold. Since the inflation and contraction noise reduction method will change the area of the fruit, the noise can be removed without changing the shape and area of the target region by judging the area size, which is especially suitable for scenes with high accuracy requirements for fruit area, and after combining connected region analysis and area threshold screening, the noise reduction target can be quickly achieved. Therefore, preferably, the noise reduction method based on area size is used in the noise reduction processing.

[0033] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, after the fruit and vegetable image is preprocessed, the image preprocessed in step S2 is cut into small pieces to obtain a plurality of small piece cut images. Then, each cut image is labeled as a whole, wherein: the cut image without fruit and vegetable defects and without black spots is labeled as "good"; the cut image with fruit and vegetable defects is labeled as "bad"; and the cut image without fruit and vegetable defects but with black spots is labeled as "black spot".

[0034] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, the first implementation of the process of cutting the image preprocessed in step S2 into small pieces in step S3 is to cut the image preprocessed in step S2 in a matrix form, as shown in the following formula (1). Figure 7 When the image is cut, the size of the block will directly affect the recognition effect. The finer the block is, the more detailed information of the cut image can be described, and the larger the receptive field of the fruit and vegetable defects or black spots in each cut image is. However, the dimension of feature extraction will increase exponentially, thereby increasing the time complexity of the algorithm. Meanwhile, too fine block will also make the cut image too sparse and lose statistical features, causing overfitting phenomenon and reducing the recognition rate.

[0035] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, the cut image is cut in a matrix form of 64*64, 128*128 and 256*256 respectively, and the labeled image after cutting is used to train ResNet50, VGG16 and AlexNet three kinds of models. The recognition accuracy of the three kinds of models after training is shown in the following formula (2). Figure 8 As can be seen from the formula (2), after the image is cut in a matrix form of 256*256, the recognition accuracy of the trained model is higher. Figure 8 Therefore, in the image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, preferably, the first implementation of the process of cutting the image preprocessed in step S2 into small pieces in step S3 is to cut the image preprocessed in step S2 in a matrix form of 256*256. Under the cut image of 256*256, the labeling amount and the labeling accuracy are considered at the same time. While ensuring that the labeling amount is not too large, the receptive field of the fruit and vegetable defects or black spots in the cut image is also larger, and the labeling accuracy is also higher.

[0036] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, the second implementation of the process of cutting the image preprocessed in step S2 into small pieces in step S3 is to take the center point of the image as the center, and perform class sector segmentation on the image preprocessed in step S2, as shown in the following formula (3). Figure 9The segmented each block image includes two cutting edges, and the intersection of the two cutting edges of each block image is the center point of the S2 preprocessed image. The center point of the S2 preprocessed image falls on the fruit and vegetable part, and the segmented single block image is as shown in FIG. 3. Figure 10

[0037] The image annotation method of the fruit and vegetable grading and recognition deep learning model training sample of the present application can be that the small block cutting process in step S3 after the image preprocessed in step S2 can be cutting in a matrix form; or can be fan-shaped segmentation.

[0038] When cutting in a matrix form, each segmented block image does not have a specific feature. Some block images can be all fruit and vegetable parts or all background parts. Some block images can have fruit and vegetable parts and background parts. The content of the block image is chaotic, and the staff cannot form a more unified and clear cognition of each block image during annotation, especially the black points on the fruit and vegetable parts and the background parts, which are all black and easy to be confused and mislabeled. In addition, if the black point on the fruit and vegetable part appears on the cutting line, the black point on the cutting line will be usually identified as a background part, resulting in the black point on the cutting line being unable to be labeled and missing, and the accuracy of the annotation is low.

[0039] When fan-shaped segmentation is used, each segmented block image has fruit and vegetable parts and background parts, and the fruit and vegetable parts are extended outward from the center angle of the two cutting edges of the block image and located in the center angle region of the block image. The background part is located on the outside of the block image. Each segmented block image has the same and unified feature, which can allow the staff to form a clearer cognition and clearly define the content of the annotation, greatly facilitating the annotation and improving the efficiency of the annotation. In addition, under the feature that the fruit and vegetable part is located in the center angle region of the block image, and the background part is located on the outside of the block image, the region between the fruit and vegetable part and the background part is clearly divided. Whether it is a fruit and vegetable defect with different colors or a black point with the same black color as the background part, they can be clearly distinguished in the block image and accurately labeled. Even if a black image appears on the cutting edge of the block image, the clear region division between the fruit and vegetable part and the background part can make the black image on the cutting line have a clear region attribution. For example, if the black image is located on the fruit and vegetable part, it is a black point on the fruit and vegetable surface. If the black image is located on the background part, it is a background. This makes the black point on the fruit and vegetable part on the cutting line not be mistaken for a background, which greatly improves the accuracy of the annotation.

[0040] ​The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, the operation of cutting the image pre-processed in step S2 into small pieces in step S3 can be programmed by calling the OpenCV machine vision library by using the python language, and the automatic cutting of the image is carried out. Through the setting and adjustment of the cutting parameters such as the number of rows, the number of columns or the cutting angle under the class sector cutting, the size and quantity of the cutting can be flexibly controlled. It can not only be uniformly cut, but also be non-uniformly cut.

[0041] The image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, by dividing the fruit and vegetable image into a plurality of small pieces, forming a small piece cutting image, the boundary of each cutting image is self-defined region. When labeling, all contents in the cutting image region are taken as the labeling target, and the cutting image is labeled as a whole. Unlike traditional methods, this labeling method no longer takes the specific features such as fruit and vegetable defects, black spots or fruit stems on the fruit and vegetable image as the labeling target, but labels each cutting image. In the labeling process, it is not necessary to find and label fruit and vegetable defects, black spots or fruit stems one by one on the image, but only to qualitatively determine whether the cutting image has fruit and vegetable defects or black spots and label it. This labeling method avoids the cumbersome process of "quantitative" labeling of fruit and vegetable defects, black spots or fruit stems in the traditional method, significantly simplifies the labeling steps, reduces the labeling workload, reduces the labeling work intensity, and greatly improves the labeling efficiency.

[0042] The number of fruit stems on fruits and vegetables is one, the quantity is small, and the image features of the fruit stem are generally uniform, which has less interference on the defect recognition of fruits and vegetables. In contrast, the number of black spots on fruits and vegetables is indefinite, and the quantity is usually large. At the same time, the image features of the black spots are diverse, and the shapes are different, which has greater interference on the defect recognition of fruits and vegetables. In combination with these features of fruits and vegetables, the image labeling method of the fruit and vegetable grading recognition deep learning model training sample provided in the application selects "fruit and vegetable defects" and "black spots" as two core judgment indexes when labeling the cut image under the condition that the fruit and vegetable image simultaneously exists three features of fruit and vegetable defects, black spots or fruit stems, and divides the labeling result into three categories. On the one hand, the "fruit stem" is not selected as a judgment index because the existence or nonexistence of the fruit stem does not affect the quality of the fruits and vegetables, and therefore the absence of the fruit stem will not affect the labeling classification result. This design not only meets the actual needs of fruit and vegetable quality evaluation, but also avoids the complexity caused by labeling the "fruit stem". Compared with the classification method using three judgment indexes of "fruit and vegetable defects", "black spots" and "fruit stem", the application only uses two judgment indexes of "fruit and vegetable defects" and "black spots", which significantly reduces the complexity of labeling, simplifies the process of manual labeling, and improves the labeling efficiency. On the other hand, in the labeling classification, the cut image without fruit and vegetable defects but with black spots is separately divided into a category, which is specially used for the identification and training of the black spots by the deep learning model. This design enables the model to learn and train the black spots individually, thereby effectively avoiding the misjudgment of the black spots as fruit and vegetable defects by the model, significantly improving the identification accuracy of the black spots by the model, and further improving the overall precision and reliability of the fruit and vegetable grading recognition. Through this optimization, the application guarantees the labeling efficiency while taking into account the accuracy and practicality of the model training.

[0043] Taking a kumquat picture as an example, when the traditional labeling software is used to label the kumquat picture point by point, the number of black spots on different kumquat pictures is different, and the time required is different, which is about 1-2.5 minutes. It takes about 60 minutes to label 50 kumquat images, and the average labeling time of each kumquat image is 1.2 minutes. When 50 kumquat images are labeled by using the image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, 1056 cut images are obtained by using a 256*256 block method and removing the background part. The labeling time of these 1056 cut images is 27 minutes, and the average labeling time of each kumquat image is 0.54 minutes. Compared with the traditional labeling time of 1.2 minutes per block, the labeling time of the labeling method of the application is greatly shortened, and the labeling efficiency is greatly improved.

[0044] The present application only needs image block level annotation, breaks through the inherent mode of traditional point-by-point pixel extreme annotation (drawing each black point or defect with a brush) of each training image, realizes "one-time annotation, full-factor coverage", guarantees the subsequent model detection accuracy, and greatly improves the data annotation efficiency and economy.

[0045] The image annotation method for training samples of the fruit and vegetable grading recognition deep learning model of the present application pre-processes the fruit and vegetable images before cutting the images, extracts the fruit and vegetable parts of interest in the images, and removes the background parts useless for image recognition. After removing the background, the image data volume is reduced, the computational complexity of subsequent cutting and annotation is reduced, thereby improving the overall processing efficiency; and by extracting the fruit and vegetable parts, the interference of background noise on model training is avoided, so that the model can focus more on the extraction and learning of fruit and vegetable features, improving the recognition accuracy; and after removing the background, the annotation target is more clear, unnecessary annotation areas are reduced, the workload and time cost of manual annotation are reduced; and the pre-processed image data is cleaner and more concentrated, which helps the model to learn the key features of fruits and vegetables (defects, black spots, etc.) more efficiently, thereby improving the generalization ability and recognition accuracy of the model; and after removing the useless background, the image file size is reduced, saving storage space and reducing the consumption of computing resources in the training process; and by removing the background interference, the model can better adapt to fruit and vegetable images in different environments, enhancing the adaptability to complex scenes.

[0046] The present application is not limited to the specific details of the present application, and can be further deduced or replaced without departing from the concept of the present application, and should be considered as belonging to the protection scope of the present application.

Claims

1. An image labeling method for training samples of a fruit and vegetable grading recognition deep learning model, characterized in that: The method comprises the following steps: S1. Obtain a fruit and vegetable image; S2. Preprocess the fruit and vegetable image obtained in step S1, and the preprocessing operation comprises image segmentation processing: extract the fruit and vegetable part of interest in the image, and remove the background part useless for image recognition; S3. Cut the image after preprocessing in step S2 into small blocks to obtain a plurality of small block images; S4. Label the block images in step S3 as a whole, wherein: The block image without fruit and vegetable defects and without black spots is labeled as the first type; The block image with fruit and vegetable defects is labeled as the second type; The block image without fruit and vegetable defects but with black spots is labeled as the third type; The image with more than one gray color block is an image with black spots, otherwise, the image without any gray color block is an image without black spots; the image with any one of scab, wrinkled skin, stab injury and crush injury is an image with fruit and vegetable defects, otherwise, the image without any fruit and vegetable defects including scab, wrinkled skin, stab injury and crush injury is an image without fruit and vegetable defects.

2. The image labeling method for training samples of a fruit and vegetable grading and recognition deep learning model according to claim 1, characterized in that: In step S3, the image after preprocessing in step S2 is segmented into small blocks in a fan shape with the center point of the image as the center.

3. The image labeling method for training samples of a fruit and vegetable grading and recognition deep learning model according to claim 1, characterized in that: In step S3, the image after preprocessing in step S2 is cut into small blocks in the form of a 256*256 matrix.

4. The image labeling method for training samples of a fruit and vegetable grading and recognition deep learning model according to claim 1, characterized in that: In step S3, the specific steps of cutting the image after preprocessing in step S2 into small blocks are to use python language and call OpenCV machine vision library programming to realize the automatic cutting function of the image.

5. The image labeling method for training samples of a fruit and vegetable grading and recognition deep learning model according to claim 1, characterized in that: The image segmentation processing method in step S2 is any one of threshold-based segmentation method, region-based segmentation method, edge-based segmentation method and superpixel segmentation method.

6. The image labeling method for training samples of a fruit and vegetable grading and recognition deep learning model according to claim 5, characterized in that: The image segmentation processing method in step S2 is threshold-based segmentation method, and the threshold-based segmentation method uses H component threshold segmentation method based on HSV model.

7. The image labeling method for training samples of a fruit and vegetable grading and recognition deep learning model according to claim 1, characterized in that: The preprocessing operation in step S2 further comprises binarization operation and noise reduction processing, and the image segmentation processing, binarization operation and noise reduction processing are performed in sequence.

8. The image labeling method for training samples of a fruit and vegetable grading and recognition deep learning model according to claim 7, characterized in that: The noise reduction processing is performed by judging the area size.

9. The image labeling method for training samples of a fruit and vegetable grading and recognition deep learning model according to claim 1, characterized in that: Step S1 is to obtain fruit and vegetable images from multiple angles through multiple cameras, and after obtaining the fruit and vegetable images, the number of images is increased through data enhancement method, and the data enhancement method includes cutting, flipping and random cropping method.

Images