Citrus surface defect segmentation method driven by superpixel features

Abstract

The present invention discloses a superpixel feature-driven citrus surface defect segmentation method. The method comprises: inputting a citrus sample image into a citrus surface defect segmentation model constructed with a Unet model as a benchmark model to generate a predicted label, constructing a loss function according to the real label and the predicted label corresponding to the citrus sample image, and then training the citrus surface defect segmentation model; inputting the citrus sample image to be tested into the trained citrus surface defect segmentation model to obtain a segmentation result image of the citrus sample to be tested. The present invention aggregates the key features of targets of different scales in the image by generating multi-scale superpixels to form regional features to improve the model's learning of the key features of the target, and utilizes the superpixel grid's ability to capture the target boundary to constrain the boundary error of the predicted label, thereby improving the model's segmentation ability to achieve accurate segmentation of citrus surface defects, and providing a reliable basis for citrus quality assessment and sorting.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to a superpixel feature-driven citrus surface defect segmentation method. Background Art

[0002] Surface defects are important indicators for quality assessment, and many countries have strict requirements for this in their grading standards. Therefore, accurate detection and segmentation of citrus surface defects are of great significance for citrus quality assessment and sorting.

[0003] Defect detection methods based on machine vision can, to a certain extent, reduce the problems of low efficiency and strong subjectivity caused by manual sorting. Deep learning has also further promoted the development of fruit and vegetable quality assessment. Convolutional neural networks have good performance in processing two-dimensional data, and their end-to-end feature learning capabilities provide strong technical support for citrus surface defect detection (Liu W, Wang Z, Liu X, et al. A survey of deep neural network architectures and their applications [J]. Neurocomputing, 2017, 234: 11-26.). Among them, semantic segmentation based on deep learning provides a more fine-grained understanding of image content by assigning each pixel in the image to a specific semantic category, and locates and segments the contours and boundaries of the target object by automatically learning image features (Chan S, Huang C, Bai C, et al. Res2-unext: a novel deep learning framework for few-shot cell imagesegmentation [J]. Multimedia Tools and Applications, 2022, 81 (10): 13275-13288.).

[0004] Since citrus surface defects may occur at different stages of growth, harvesting, transportation, and storage, the color, shape, size and other characteristics of the defects are completely different (Feng J, Wang Z, Wang S, et al. MSDD-YOLOX: An enhanced YOLOX for real-time surface defect detection of oranges by type [J]. European Journal of Agronomy, 2023, 149: 126918.). In addition, citrus is a spherical fruit. When using images for detection, there will be problems such as uneven surface brightness and distortion, which makes the characteristics of surface defects variable. However, the current detection methods still have certain limitations in dealing with changes in defect characteristics and complex and diverse defects. The extraction of key defect features determines the accuracy of detection (Chen Y, An X, Gao S, et al. A deep learning-based vision system combining detection and tracking for fast on-line citrus sorting [J]. Frontiers in Plant Science, 2021, 12.).

[0005] The Simple Linear Iterative Clustering (SLIC) superpixel algorithm is a simple and easy-to-implement algorithm that converts a color image into a 5-dimensional feature vector in CIELAB color space and XY coordinates, then constructs a distance metric for the 5-dimensional feature vector to perform local clustering of image pixels. The SLIC algorithm captures image redundancy by aggregating discrete pixels, provides convenient primitives for calculating image features, and greatly reduces the complexity of subsequent image processing tasks, which is conducive to the extraction of key features (Achanta R, Shaji A, Smith K, et al. Slic superpixels compared to state-of-the-art superpixel methods [J]. Ieee Transactions On Pattern Analysis and Machine Intelligence, 2012, 34 (11): 2274-2282.). However, the SLIC algorithm can only achieve pixel clustering at a single scale and cannot adapt to the multi-scale variations of citrus surface defects.

[0006] Citrus surface defects have numerous causes, multiple defect types, and high intra-class variability, making accurate clustering difficult. Furthermore, the complex texture and low contrast of citrus surfaces lead to blurred boundaries and difficult to extract small defect features, further increasing the difficulty of surface defect identification and segmentation. Summary of the Invention

[0007] To address the problems of the background technology, this paper proposes a superpixel feature-driven citrus surface defect segmentation method. By constructing multi-scale superpixel regional feature branches through the superpixel method to capture defects at different scales, the discrete pixel features are aggregated into regional key features, and a superpixel loss function is constructed to clarify the fuzzy boundary, thereby improving the model segmentation performance.

[0008] The technical solution adopted in the present invention is:

[0009] The superpixel feature-driven citrus surface defect segmentation method includes the following steps:

[0010] 1) Collecting citrus sample images, performing image data enhancement processing on the citrus sample images, and annotating the surface defects, stalks, and navels of the citrus in the citrus sample images after image data enhancement processing to obtain a citrus surface defect dataset;

[0011] 2) For the citrus surface defect dataset, a citrus surface defect segmentation model is constructed using the Unet model as the benchmark model. The Unet model is a Unet semantic segmentation model;

[0012] 3) Input the citrus sample images in the citrus surface defect dataset into the citrus surface defect segmentation model, generate the model's predicted labels for the citrus sample images, and construct a loss function L according to the true labels corresponding to the citrus sample images in the citrus surface defect dataset and the predicted labels of the citrus sample images. SP Then, the citrus surface defect segmentation model is trained;

[0013] 4) The image of the citrus sample to be tested is input into the trained citrus surface defect segmentation model to obtain the surface defect, navel and stalk segmentation result images of the citrus sample to be tested.

[0014] The step 2) is specifically as follows:

[0015] 2.1) For citrus sample images in the citrus surface defect dataset, the simple linear iterative clustering method (SLIC) is used to generate superpixel grids of three grid scales from large to small in each citrus sample image;

[0016] 2.2) A citrus surface defect segmentation model is constructed based on the generated superpixel grid maps of the three grid scales using the Unet model as the benchmark model.

[0017] In step 3), the loss function L SP The construction is as follows:

[0018] L SP =L S +L Dice +L Focal

[0019]

[0020]

[0021]

[0022] Among them, L dice is the Dice loss function, L Focal is the Focal Loss loss function, L s is the superpixel boundary error loss function, P i and G i denote the predicted label and true label of the citrus surface defect segmentation model, i is the target category index, c is the value used to iterate all pixels, N is the total number of categories, ɑ and γ denote the class weight and weighting degree used to control the easy-to-classify pixels, respectively, and g i，c Represents the one-hot encoding of the true label, p i，c is the matrix of predicted values for each class;

[0023] Area Error is the total error value of the superpixel grid, which is calculated specifically as follows: based on the minimum-sized superpixel grid map obtained in step 2.1), after setting the maximum threshold M and minimum threshold N of the number of predicted label pixels in the superpixel grid, the error value of each superpixel grid in the superpixel grid map is determined:

[0024] When the number of pixels of the predicted label in the superpixel grid is greater than the minimum threshold N and less than the maximum threshold M, the number of pixels in the superpixel grid excluding the predicted label is calculated as the error value of the superpixel grid; when the number of pixels of the predicted label in the superpixel grid is greater than or equal to the maximum threshold M or less than or equal to the minimum threshold N, the error value of the superpixel grid is zero; finally, the sum of the error values of all superpixel grids is taken as the total error value of the superpixel grid;

[0025] Area GT It is the sum of the number of true label pixels in the superpixel grid.

[0026] In step 2.2), the topological structure of the citrus surface defect segmentation model is specifically as follows:

[0027] Includes convolutional layers, downsampling layers, and upsampling layers;

[0028] The citrus sample image after image data enhancement and the three superpixel grid maps with grid scales from large to small are respectively passed through different convolutional layers to obtain the citrus feature map X1 and the three superpixel grid feature maps with grid scales from large to small. The citrus feature map X1 is sequentially passed through four downsampling layers to obtain the citrus feature map X2, citrus feature map X3, citrus feature map X4 and citrus feature map X5 respectively. Then, the three superpixel grid feature maps with grid scales from large to small are respectively concatenated with the citrus feature map X2, citrus feature map X3 and citrus feature map X4 to obtain the skip connection feature map S1, skip connection feature map S2 and skip connection feature map S3. Subsequently, the citrus feature map X5 is sequentially passed through four upsampling layers to obtain the citrus feature map M1, citrus feature map M2, citrus feature map M3 and citrus feature map M4 respectively.

[0029] The citrus feature map M1, citrus feature map M2, citrus feature map M3 and citrus feature map M4 are respectively concatenated with the citrus feature map X1, skip connection feature map S1, skip connection feature map S2 and skip connection feature map S3 to obtain the citrus feature map Y1, citrus feature map Y2, citrus feature map Y3 and citrus feature map Y4. Finally, the citrus feature map Y4 is convolved to obtain the segmentation result.

[0030] In the step 1), the image data enhancement processing includes random angle rotation, mirror flipping, upside down flipping, Gaussian blurring, mean blurring, median blurring, adding Gaussian noise and shearing operation processing.

[0031] The beneficial effects of the present invention are: by generating multi-scale superpixels to aggregate the key features of targets of different scales in the image, regional features are formed to improve the model's learning of the key features of the target, and a new loss function is designed by utilizing the superpixel grid's ability to capture the target boundary to constrain the boundary error of the predicted label, thereby improving the model's segmentation ability and achieving accurate segmentation of citrus surface defects, providing a reliable basis for citrus quality assessment and sorting. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the method of the present invention.

[0033] Figure 2 This is a picture of the collected citrus samples.

[0034] Figure 3 is the surface map of citrus in the citrus surface defects dataset.

[0035] Figure 4 This is the effect of a simple linear iterative clustering method generating a superpixel grid on the dataset image.

[0036] Figure 5It is a multi-scale superpixel feature map.

[0037] Figure 6 It is a schematic diagram of the superpixel boundary error loss function. DETAILED DESCRIPTION

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] like Figure 1 As shown, the citrus surface defect segmentation method includes the following steps:

[0040] 1) Collect citrus sample images with rock sugar orange as the experimental object, the citrus sample images are as follows Figure 2 As shown in the figure, the image resolution is 224×224. The citrus sample image is subjected to image data enhancement processing by random angle rotation, mirror flipping, upside down flipping, Gaussian blurring, mean blurring, median blurring, adding Gaussian noise, and shearing operations. The surface defects, stalks, and navels of the citrus sample image after image data enhancement are annotated. The annotated targets include three types of defects: spots, ulcers, and deformities, as well as five types of segmentation targets for stalks and navels. The results are as follows: Figure 3 Citrus surface defect dataset shown;

[0041] 2) For the citrus surface defect dataset, a citrus surface defect segmentation model was constructed using the Unet model as the benchmark model;

[0042] 2.1) For each citrus sample image in the citrus surface defect dataset, the simple linear iterative clustering method (SLIC) is used to convert each citrus sample image in the citrus surface defect dataset into the CIELAB color space. K initial cluster centers are set in each citrus sample image in the citrus surface defect dataset, and each cluster center corresponds to a regular grid;

[0043] 2.2) Calculate the gradient values of all pixels in the neighborhood of the cluster center and set the pixel with the lowest gradient as the new cluster center to avoid the superpixel center being on the edge;

[0044] 2.3) Calculate by the following formula until the cluster center of each pixel no longer changes, and the generated superpixel grid is as follows: Figure 4 As shown;

[0045]

[0046] Where D is the distance metric between the color and space of all pixels in the citrus sample image and the cluster center. The cluster center is the center of the superpixel grid. c The color distance between each pixel and the cluster center, d sis the spatial distance between each pixel and the cluster center, S is the maximum spatial distance, and m is the relative importance value for determining color similarity and spatial proximity;

[0047] 2.4) The superpixels of large, medium and small scales are determined by the segmentation target sizes annotated in the citrus surface defect dataset. The superpixel grids of three grid scales are generated by setting the value of k in the simple linear iterative clustering method. Figure 4 As shown in the figure, the larger the value of k, the smaller the scale of the generated superpixel grid, so that the superpixel grids of these three scales can adhere to the segmentation targets of different scales. According to the statistics of the labels of the citrus surface defect dataset, the three values of k are 100, 300, and 500 respectively;

[0048] 2.5) A citrus surface defect segmentation model is constructed based on the generated superpixel grid maps of the three grid scales using the Unet model as the benchmark model.

[0049] Specifically, the topological structure of the citrus surface defect segmentation model is as follows:

[0050] Includes convolutional layers, downsampling layers, and upsampling layers;

[0051] The citrus sample image after image data enhancement and the three superpixel grid maps with grid scales from large to small are respectively passed through different convolutional layers to obtain the citrus feature map X1 and the three superpixel grid feature maps with grid scales from large to small. The citrus feature map X1 is sequentially passed through four downsampling layers to obtain the citrus feature map X2, citrus feature map X3, citrus feature map X4 and citrus feature map X5 respectively.

[0052] Among them, the convolution kernel size of the two convolution layers is 3×3, the step size of the convolution layer passed by the citrus sample image is 1, the step size of the convolution layer passed by the superpixel grid map is 2, and the scales of the citrus feature map X1, citrus feature map X2, citrus feature map X3, citrus feature map X4 and citrus feature map X5 are H×W×C, and W is the image width, H is the image height, and C is the number of channels;

[0053] Then, the three feature map scales from large to small superpixel grid feature maps are concatenated with citrus feature map X2, citrus feature map X3, and citrus feature map X4 to obtain skip connection feature map S1, skip connection feature map S2, and skip connection feature map S3, where the scales of skip connection feature map S1, skip connection feature map S2, and skip connection feature map S3 are respectively and Then, the citrus feature map X5 is sequentially subjected to four upsampling layers to obtain the citrus feature map M1, citrus feature map M2, citrus feature map M3 and citrus feature map M4 respectively;

[0054] The citrus feature maps M1, M2, M3 and M4 are concatenated with the citrus feature map X1, the skip connection feature map S1, the skip connection feature map S2 and the skip connection feature map S3 to obtain the citrus feature maps Y1, Y2, Y3 and Y4. The scales of the citrus feature maps Y1, Y2, Y3 and Y4 are H×W×C, and finally the citrus feature map Y4 is convolved to obtain the segmentation result, with a convolution kernel size of 1×1 and a step size of 1.

[0055] 3) Input the citrus sample images in the citrus surface defect dataset into the citrus surface defect segmentation model, generate the model's predicted labels for the citrus sample images, and construct a loss function L according to the true labels corresponding to the citrus sample images in the citrus surface defect dataset and the predicted labels of the citrus sample images. SP Then, the citrus surface defect segmentation model is trained;

[0056] Specifically, the loss function L SP The settings are as follows:

[0057] L SP =L S +L Dice +L Focal

[0058]

[0059]

[0060]

[0061] Among them, L dice is the Dice loss function, L Focal is the Focal Loss loss function, L s is the superpixel boundary error loss function, P i and G i denote the predicted label and true label of the citrus surface defect segmentation model, i is the target category index, c is the value used to iterate all pixels, N is the total number of categories, ɑ and γ denote the class weight and weighting degree used to control the easy-to-classify pixels, respectively, and g i，c Represents the one-hot encoding of the true label, p i，c is the matrix of predicted values for each class;

[0062] Area Erroris the total error value of the superpixel grid, which is calculated specifically according to the following method: based on the superpixel grid map of minimum size obtained in step 2.1), after setting the maximum threshold M and minimum threshold N of the number of predicted label pixels in the superpixel grid, the error value of each superpixel grid in the superpixel grid map is determined.

[0063] When the number of pixels of the predicted label in the superpixel grid is greater than the minimum threshold N and less than the maximum threshold M, the number of pixels in the superpixel grid excluding the predicted label is calculated as the error value of the superpixel grid; when the number of pixels of the predicted label in the superpixel grid is greater than or equal to the maximum threshold M or less than or equal to the minimum threshold N, the error value of the superpixel grid is zero; finally, the sum of the error values of all superpixel grids is taken as the total error value of the superpixel grid;

[0064] Area GT It is the sum of the number of true label pixels in the superpixel grid.

[0065] 4) The image of the citrus sample to be tested is input into the trained citrus surface defect segmentation model to obtain the surface defect, navel and stalk segmentation result images of the citrus sample to be tested.

[0066] The performance of the citrus surface defect segmentation model proposed in the present invention is compared with other semantic segmentation models. As shown in Table 1, the citrus surface defect segmentation model proposed in the present invention achieves better results, among which the average pixel accuracy mPA and average intersection over union (mIoU) are both the best, and Dice is the set similarity measurement index.

[0067] Table 1 Performance comparison of the model of the present invention and other models

[0068]

[0069] The present invention aggregates the key features of targets of different scales in the image by generating multi-scale superpixels to form regional features to improve the model's learning of the key features of the target. It also uses the superpixel grid's ability to capture the target boundary to design a new loss function to constrain the boundary error of the predicted label, thereby improving the model's segmentation ability and achieving accurate segmentation of citrus surface defects, providing a reliable basis for citrus quality assessment and sorting.

Claims

1. A superpixel feature-driven citrus surface defect segmentation method, characterized by: The method comprises the following steps: 1) Collecting citrus sample images, performing image data enhancement processing on the citrus sample images, and annotating the surface defects, stalks, and navels of the citrus in the citrus sample images after image data enhancement processing to obtain a citrus surface defect dataset; 2) Build a citrus surface defect segmentation model based on the citrus surface defect dataset; 3) Input the citrus sample images in the citrus surface defect dataset into the citrus surface defect segmentation model to generate predicted labels for the citrus sample images. Construct a loss function L according to the true labels corresponding to the citrus sample images in the citrus surface defect dataset and the predicted labels of the citrus sample images. SP Then, the citrus surface defect segmentation model is trained; 4) inputting the image of the citrus sample to be tested into the trained citrus surface defect segmentation model to obtain a segmentation result image of the citrus sample to be tested; The step 2) is specifically as follows: 2.1) For the citrus sample images in the citrus surface defect dataset, a simple linear iterative clustering method is used to generate superpixel grids of three grid scales from large to small in each citrus sample image; 2.2) Based on the generated superpixel grid maps of the three grid scales, a citrus surface defect segmentation model was constructed using the Unet model as the benchmark model; In step 3), the loss function L SP The construction is as follows: L SP =L S +L Dice +L Focal Among them, L dice is the Dice loss function, L Focal is the Focal Loss loss function, L s is the superpixel boundary error loss function, P i and G i denote the predicted label and true label of the citrus surface defect segmentation model, i is the target category index, c is the value used to iterate all pixels, N is the total number of categories, ɑ and γ denote the class weight and weighting degree used to control the easy-to-classify pixels, respectively, and g i，c Represents the one-hot encoding of the true label, p i，c is the matrix of predicted values for each class; Area Error is the total error value of the superpixel grid, which is calculated specifically as follows: based on the minimum-sized superpixel grid map obtained in step 2.1), after setting the maximum threshold M and minimum threshold N of the number of predicted label pixels in the superpixel grid, the error value of each superpixel grid is determined: When the number of pixels of the predicted label in the superpixel grid is greater than the minimum threshold N and less than the maximum threshold M, the number of pixels in the superpixel grid excluding the predicted label is calculated as the error value of the superpixel grid; when the number of pixels of the predicted label in the superpixel grid is greater than or equal to the maximum threshold M or less than or equal to the minimum threshold N, the error value of the superpixel grid is zero; finally, the sum of the error values of all superpixel grids is taken as the total error value of the superpixel grid; Area GT It is the sum of the number of true label pixels in the superpixel grid.

2. The method for segmenting citrus surface defects according to claim 1, characterized in that: In step 2.2), the topological structure of the citrus surface defect segmentation model is specifically as follows: Includes convolutional layers, downsampling layers, and upsampling layers; The citrus sample image after image data enhancement and the three superpixel grid maps with grid scales from large to small are respectively passed through different convolutional layers to obtain the citrus feature map X1 and the three superpixel grid feature maps with grid scales from large to small. The citrus feature map X1 is sequentially passed through four downsampling layers to obtain the citrus feature map X2, citrus feature map X3, citrus feature map X4 and citrus feature map X5 respectively. Then, the three superpixel grid feature maps with scales from large to small are respectively concatenated with the citrus feature map X2, citrus feature map X3 and citrus feature map X4 to obtain the skip connection feature map S1, skip connection feature map S2 and skip connection feature map S3. Subsequently, the citrus feature map X5 is sequentially passed through four upsampling layers to obtain the citrus feature map M1, citrus feature map M2, citrus feature map M3 and citrus feature map M4 respectively. The citrus feature map M1, citrus feature map M2, citrus feature map M3 and citrus feature map M4 are respectively concatenated with the citrus feature map X1, skip connection feature map S1, skip connection feature map S2 and skip connection feature map S3 to obtain the citrus feature map Y1, citrus feature map Y2, citrus feature map Y3 and citrus feature map Y4. Finally, the citrus feature map Y4 is convolved to obtain the segmentation result.

3. The citrus surface defect segmentation method according to claim 1, characterized in that: In the step 1), the image data enhancement processing includes random angle rotation, mirror flipping, upside down flipping, Gaussian blurring, mean blurring, median blurring, adding Gaussian noise and shearing operation processing.

Images