Eagle peach sorting system and device based on visual detection and deep neural network

Abstract

The present invention discloses a sorting system and device for honey peaches based on visual detection and deep neural network. The method includes: acquiring an image of a fleshy fruit to be sorted; performing image segmentation processing on the image of the fleshy fruit to obtain a first feature; processing the first feature through an appearance defect detection model to obtain an appearance detection result; and performing elimination processing on the fleshy fruit based on the appearance detection result to obtain a first fruit set; determining the quality category of the fruit according to the type and shape area of each fruit in the first fruit set, and sorting the fruit according to the quality category. The embodiments of the present application are beneficial to improving the detection efficiency of the appearance defect detection model and the sorting accuracy. This method can be widely applied to the technical field of sorting systems.

Description

Technical Field

[0001] The present invention relates to the technical field of sorting systems, and in particular to a nectarine sorting system and device based on visual detection and deep neural network. Background Art

[0002] Currently, the technology for fruit sorting mainly relies on manual implementation, which is inefficient and easily affected by subjective factors, and prone to errors. Although some automated sorting devices have emerged, for fruits with complex appearance features such as fleshy fruits, the related technologies still have problems in terms of accuracy and efficiency. Summary of the Invention

[0003] An object of the present invention is to solve at least to some extent one of the technical problems existing in the prior art.

[0004] To this end, an object of the present invention is to provide an efficient and accurate nectarine sorting system and device based on visual detection and deep neural network.

[0005] To achieve the above technical object, the technical solutions adopted in the embodiments of the present invention include:

[0006] On the one hand, an embodiment of the present invention provides a nectarine sorting method based on visual detection and deep neural network, including the following steps:

[0007] The nectarine sorting method based on visual detection and deep neural network in the embodiment of the present invention includes: obtaining an image of a fleshy fruit to be sorted; performing image segmentation processing on the image of the fleshy fruit to obtain a first feature; processing the first feature through an appearance defect detection model to obtain an appearance detection result; and performing elimination processing on the fleshy fruit according to the appearance detection result to obtain a first fruit set; determining the quality category of the fruit according to the type and shape area of each fruit in the first fruit set, and sorting the fruit according to the quality category. In the embodiment of the present application, by performing image segmentation processing on the image of the fleshy fruit, the detection efficiency of the appearance defect detection model is improved; through the processing of the appearance defect detection model, the fruits with appearance defects are eliminated; then, according to the type and shape area of the fruit, the quality category of the fruit is determined, and the fruit is sorted according to the quality category, thereby improving the accuracy of sorting.

[0008] In addition, according to the nectarine sorting method based on visual detection and deep neural network in the above embodiment of the present invention, the following additional technical features may also be included:

[0009] Further, the nectarine sorting method based on visual detection and deep neural network in the embodiment of the present invention further includes:

[0010] Scan each fruit in the first fruit set through an infrared camera, perform meat quality detection on the fruits, and update the first fruit set according to the meat quality detection results;

[0011] The scanning of each fruit in the first fruit set through an infrared camera and the meat quality detection of the fruits include:

[0012] Provide an adjustable light source through a lighting device, scan the first fruit through an infrared camera, and determine the thermal radiation information of the first fruit; the first fruit set includes the first fruit;

[0013] Determine the internal characteristics of the first fruit according to the thermal radiation information, and perform meat quality detection on the first fruit.

[0014] Further, in an embodiment of the present invention, the image segmentation processing of the meat quality fruit image to obtain the first feature includes:

[0015] Perform Gaussian filtering processing on the meat quality fruit image to obtain a first image;

[0016] Calculate the gradient of the first image to obtain first information;

[0017] Perform non-maximum suppression processing on the first information to obtain second information;

[0018] Perform double-threshold screening processing on the second information to obtain the first feature.

[0019] Further, in an embodiment of the present invention, the processing of the first feature through an appearance defect detection model to obtain an appearance detection result includes the following steps:

[0020] Preprocess the first feature to obtain third information;

[0021] Extract the features of the third information through a basic network to obtain a second feature;

[0022] Process the second feature through a feature pyramid to obtain a third feature;

[0023] Perform prediction on the third feature through a detection head to obtain an appearance detection result.

[0024] Further, in an embodiment of the present invention, the appearance defect detection model is trained through the following steps:

[0025] Obtain a meat quality fruit image sample to be sorted; the meat quality fruit image sample includes the true sorting result of the meat quality fruit sample;

[0026] Perform image segmentation processing on the meat quality fruit image sample to obtain a first feature sample;

[0027] Process the first feature sample through an appearance defect detection model to obtain a sample detection result; and calculate a loss value according to the sample detection result and the true sorting result using a loss function, and update the parameters of the appearance defect detection model according to the loss value to obtain the trained appearance defect detection model.

[0028] Further, in an embodiment of the present invention, the fleshy fruit image sample includes positive samples and negative samples, and the loss value is determined through the following steps:

[0029] Determine a first confidence loss, a first localization loss, and a first category loss according to the positive samples;

[0030] Determine a second confidence loss according to the negative samples;

[0031] Determine a loss value according to the first confidence loss, the second confidence loss, the first localization loss, and the first category loss.

[0032] Further, in an embodiment of the present invention, the determining the quality category of the fruit according to the type and shape area of each fruit in the first fruit set includes:

[0033] Determine a first empirical value according to the quotient of the weight and shape area of each fruit in the first fruit set; look up the corresponding quality in the classification standard according to the type and the first empirical value to determine the quality category of the fruit;

[0034] Or, determine a second empirical value according to the quotient of the weight and shape area of each fruit in the first fruit set; sort the second empirical values in descending or ascending order according to the type, and determine the quality category of the fruit according to the sorting result.

[0035] On the other hand, an embodiment of the present invention proposes a sorting system for peach with sharp beak based on visual detection and deep neural network, including:

[0036] An image acquisition module for acquiring an image of a fleshy fruit to be sorted;

[0037] An image processing module for performing image segmentation processing on the fleshy fruit image to obtain a first feature;

[0038] A neural network classification module for processing the first feature through an appearance defect detection model to obtain an appearance detection result; and performing elimination processing on the fleshy fruit according to the appearance detection result to obtain a first fruit set;

[0039] The sorting control module is used to determine the quality category of each fruit in the first fruit set according to the type and shape area of each fruit, and sort the fruits according to the quality category.

[0040] On the other hand, an embodiment of the present invention provides a nectarine sorting device based on visual detection and deep neural network, including:

[0041] At least one processor;

[0042] At least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor is caused to implement the above-mentioned nectarine sorting method based on visual detection and deep neural network.

[0044] On the other hand, an embodiment of the present invention provides a storage medium, in which a program executable by a processor is stored, and the program executable by the processor is used to implement the above-mentioned nectarine sorting method based on visual detection and deep neural network when executed by the processor.

[0045] The method provided in this application includes: acquiring an image of a fleshy fruit to be sorted; performing image segmentation processing on the image of the fleshy fruit to obtain a first feature; processing the first feature through an appearance defect detection model to obtain an appearance detection result; and performing elimination processing on the fleshy fruit through the appearance detection result to obtain a first fruit set; determining the quality category of the fruit according to the type and shape area of each fruit in the first fruit set, and sorting the fruit according to the quality category. By performing image segmentation processing on the image of the fleshy fruit in the embodiment of this application, the detection efficiency of the appearance defect detection model is improved; through the processing of the appearance defect detection model, the fruits with appearance defects are eliminated; then, according to the type and shape area of the fruit, the quality category of the fruit is determined, and the fruit is sorted through the quality category, improving the accuracy of sorting. Description of the Drawings

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following introduces the drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the drawings introduced below are only for conveniently and clearly expressing some embodiments of the technical solutions in the present invention, and those skilled in the art can also obtain other drawings based on these drawings without creative efforts.

[0047] Figure 1 It is a schematic flowchart of an embodiment of the nectarine sorting method based on visual detection and deep neural network provided by the present invention;

[0048] Figure 2 Schematic structural diagram of an embodiment of the sorting device provided by the present invention;

[0049] Figure 3 Schematic flow diagram of an embodiment of the meat quality detection process provided by the present invention;

[0050] Figure 4 Schematic diagram of an embodiment of the Gaussian filtering provided by the present invention;

[0051] Figure 5 Schematic diagram of an embodiment of the gradient processing provided by the present invention;

[0052] Figure 6 Schematic diagram of an embodiment of the non - maximum suppression provided by the present invention;

[0053] Figure 7 Schematic diagram of another embodiment of the non - maximum suppression provided by the present invention;

[0054] Figure 8 Schematic diagram of an embodiment of the positive and negative gradients provided by the present invention;

[0055] Figure 9 Schematic diagram of an embodiment of the edge classification provided by the present invention;

[0056] Figure 10 Schematic structural diagram of an embodiment of the nectarine sorting system based on visual detection and deep neural network provided by the present invention;

[0057] Figure 11 Schematic structural diagram of an embodiment of the nectarine sorting device based on visual detection and deep neural network provided by the present invention. Detailed implementation manners

[0058] The embodiments of the present invention are described in detail below. The examples of the embodiments are shown in the drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention, and should not be construed as a limitation of the present invention. For the step numbers in the following embodiments, they are only set for the convenience of explanation and illustration, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0059] Currently, the technology for fruit sorting mainly relies on manual implementation, which is inefficient and easily affected by subjective factors. Although some automated sorting devices have emerged, for fleshy fruits with complex appearance features, the existing technology still has problems in terms of accuracy and efficiency.

[0060] The following describes in detail the nectarine sorting method and system based on visual detection and deep neural network according to the embodiments of the present invention with reference to the accompanying drawings. First, the nectarine sorting method based on visual detection and deep neural network according to the embodiments of the present invention will be described with reference to the accompanying drawings.

[0061] Refer to Figure 1 , in the embodiments of the present invention, a nectarine sorting method based on visual detection and deep neural network is provided. The nectarine sorting method based on visual detection and deep neural network in the embodiments of the present invention can be applied to a terminal, or to a server, or can also be software running on a terminal or a server, etc. The terminal can be a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The nectarine sorting method based on visual detection and deep neural network in the embodiments of the present invention mainly includes the following steps:

[0062] S100: Obtain an image of the fleshy fruit to be sorted;

[0063] S200: Perform image segmentation processing on the fleshy fruit image to obtain a first feature;

[0064] S300: Process the first feature through an appearance defect detection model to obtain an appearance detection result; and perform elimination processing on the fleshy fruit according to the appearance detection result to obtain a first fruit set;

[0065] S400: Determine the quality category of the fruit according to the type and shape area of each fruit in the first fruit set, and sort the fruit according to the quality category.

[0066] In some possible implementation manners, the purpose of the present invention is to provide a fleshy fruit sorting device based on visual detection and deep neural network, aiming to solve the problems existing in the prior art through image processing technology and neural network algorithms. The specific main ideas include the following:

[0067] 1. Appearance defect detection: Use image processing technology, including the detection and recognition of defects such as appearance scratches, spots, and wormholes, to ensure the accuracy of sorting.

[0068] 2. Image segmentation algorithm: Use the Canny operator to implement image segmentation, making the shape features more clearly visible, which helps the sorting device to identify and inspect the shape of the fleshy fruit.

[0069] 3. Infrared camera scanning: Use an infrared camera for scanning to detect the pulp quality of fleshy fruits and ensure the qualification of pulp quality.

[0070] 4. Determine the category of fleshy fruits based on the weight and shape - area ratio range value: By analyzing the numerical values of the weight and shape - area ratio range of fleshy fruits, classify the fruits into different categories for corresponding sorting processing.

[0071] 5. Data sampling and neural network classification: Construct a sample library, use the neural network algorithm to classify fleshy fruit samples, and implement dynamic adjustment of parameters through a feedback mechanism to improve the accuracy and efficiency of sorting.

[0072] The fleshy fruit sorting device of the present invention includes an image acquisition module, an image processing module, a neural network classification module, and a sorting control module. By combining the above - mentioned technical points and reasonably configuring the corresponding hardware and software, automatic, accurate, and efficient sorting of fleshy fruits can be achieved. By introducing image - processing and deep neural network technologies, it is possible to detect and sort fleshy fruits more accurately and quickly, improving the quality and efficiency of sorting. Specifically, the present application can provide a fleshy fruit sorting device based on visual detection and deep neural network; a fleshy fruit sorting device including an image acquisition module, an image processing module, a neural network classification module, and a sorting control module. A method for detecting appearance defects of fleshy fruits based on image processing, including steps of using the Canny operator to implement image segmentation and detecting and identifying defects such as appearance scratches, spots, and wormholes. A method for detecting the pulp quality of fleshy fruits based on an infrared camera, including detecting and determining the pulp quality of fleshy fruits by infrared camera scanning. A method for classifying the pulp quality based on weight and shape - area ratio, including determining the category of pulp quality according to the numerical range of weight and shape - area ratio. A method for classifying fleshy fruit samples based on a neural network, including steps of constructing a sample library, using a neural network to classify fleshy fruit samples, and implementing parameter adjustment through feedback.

[0073] It can be understood that the image of the fleshy fruit to be sorted in the embodiment of the present application can be obtained by photographing the fleshy fruit to be sorted on, for example, Figure 2 a sorting device, and then sorting the fruit through the sorting method proposed in the present application, and placing the fruit in a corresponding container according to the sorting result. It can be understood that the fruits in the embodiment of the present application can be honey - peaches, or other peaches, or fruits such as mangoes and pears. The present application does not limit the specific types of fruits.

[0074] Optionally, in an embodiment of the present invention, the method further includes:

[0075] Each fruit in the first fruit set is scanned by an infrared camera to perform meat quality detection on the fruit, and the first fruit set is updated according to the meat quality detection result;

[0076] Each fruit in the first fruit set is scanned by an infrared camera to perform meat quality detection on the fruit, including:

[0077] An adjustable light source is provided by a lighting device, and the first fruit is scanned by an infrared camera to determine the thermal radiation information of the first fruit; the first fruit set includes the first fruit;

[0078] According to the thermal radiation information, the internal characteristics of the first fruit are determined to perform meat quality detection on the first fruit.

[0079] In some possible implementation manners, referring to Figure 3 As shown in an embodiment, the fruit is photographed by an infrared camera to determine the meat quality of the fruit, and the fruits that do not meet the requirements are eliminated and the first fruit set is updated. Specifically, a special lighting device is used to adjust or transmit the light source to enhance the details and contrast of the captured picture. An industrial camera digital camera is used to obtain the meat quality image of the fruit, and the infrared lens can capture the thermal radiation of the fruit meat, thereby providing information about the internal characteristics of the fruit, such as ripeness, moisture distribution, etc. Through the meat quality detection in the embodiments of the present application, the accuracy of the detection is improved.

[0080] Optionally, in an embodiment of the present invention, image segmentation processing is performed on the meat quality fruit image to obtain a first feature, including:

[0081] Perform Gaussian filtering processing on the meat quality fruit image to obtain a first image;

[0082] Calculate the gradient of the first image to obtain a first piece of information;

[0083] Perform non-maximum suppression processing on the first piece of information to obtain a second piece of information;

[0084] Perform double-threshold screening processing on the second piece of information to obtain a first feature.

[0085] In some possible embodiments, the embodiments of the present application obtain the circumscribed rectangle of a polygon through the Canny operator. It can be understood that the edge of an image refers to the part where the local area brightness of the image changes significantly. The gray profile of this area can be regarded as a step, that is, it changes sharply from one gray value to another gray value with a large difference in a very small buffer area. Most of the information of the image is concentrated in the edge part of the image. The determination and extraction of the image edge are very important for the recognition and understanding of the entire image scene, and it is also an important feature relied on for image segmentation. Edge detection is mainly the measurement, detection, and localization of the gray change of the image. Canny edge detection is a method of detecting edges using a multi-level edge detection algorithm. Generally, the purpose of edge detection is to significantly reduce the data scale of the image while preserving the original image attributes. There are various algorithms for edge detection. The goal of Canny is to find an optimal edge detection algorithm. The meaning of optimal edge detection is:

[0086] (1) Optimal detection: The algorithm can identify as many actual edges in the image as possible, and the probabilities of missing real edges and misdetecting non-edges are both as small as possible;

[0087] (2) Optimal positioning criterion: The position of the detected edge point is the closest to the position of the actual edge point, or the degree to which the detected edge deviates from the true edge of the object due to noise influence is the smallest;

[0088] (3) One-to-one correspondence between detected points and edge points: The edge points detected by the operator should correspond one-to-one to the actual edge points. To meet these requirements, Canny uses the calculus of variations, which is a method of finding functions that optimize specific functions. Optimal detection is represented by four exponential function terms, but it is very similar to the first derivative of the Gaussian function.

[0089] It should be noted that Canny edge detection is divided into the following steps:

[0090] Step S11: Apply Gaussian filtering to smooth the image, aiming to remove noise.

[0091] Step S12: Calculate the image gradient to obtain possible edges;

[0092] Step S13: Apply non-maximum suppression technology to eliminate edge misdetection;

[0093] Step S14: Apply the method of double thresholds to screen edge information;

[0094] Step S15: Use hysteresis technology to track the boundary.

[0095] Among them, in the process of applying Gaussian filtering to remove image noise, since the image edges are very vulnerable to noise interference, in order to avoid detecting incorrect edge information, it is usually necessary to filter the image to remove the noise. The purpose of filtering is to smooth some non-edge areas with weak textures so as to obtain more accurate edges. In the actual processing process, Gaussian filtering is usually used to remove the noise in the image. Refer to Figure 4 As shown in an embodiment, the process of using the Gaussian filter T to filter the pixel points with a pixel value of 226 in the original image O to obtain the value of this point in the filtered result image D. During the filtering process, the weighted average of the pixels around the pixel point is calculated through the filter to obtain the final filtering result. For the Gaussian filter T, the closer a point is to the center, the greater its weight value. For the pixel point with a pixel value of 226 in the image O, the calculation process and result of filtering using the filter T are: Result = 1 / 56×(197×1 + 25×1 + 106×2 + 156×1 + 159×1 + 149×1 + 40×1 + 107×4 + 5×3 + 71×1 + 163×2 + 198×4 + 226×8 + 223×4 + 156×2 + 222×1 + 37×3 + 68×4 + 193×3 + 157×1 + 42×1 + 72×1 + 250×2 + 41×1 + 75×1) = 138.

[0096] It can be understood that generally, the Gaussian filter (Gaussian kernel) is not fixed, and the size of the filter is also variable. The size of the Gaussian kernel plays a very important role in the effect of edge detection. The larger the kernel of the filter, the lower the sensitivity of the edge information to the noise. However, the larger the kernel, the more the positioning error of edge detection will increase. Generally speaking, a 5×5 kernel can meet most situations.

[0097] In the process of calculating the gradient, the embodiments of the present application only need to focus on the direction of the gradient, and the direction of the gradient is perpendicular to the direction of the edge. The edge detection operator returns Gx in the horizontal direction and Gy in the vertical direction.

[0098] The direction of the gradient is always perpendicular to the edge. Usually, the nearby values are taken as horizontal (left, right), vertical (up, down), diagonal (upper right, upper left, lower left, lower right), etc., 8 different directions.

[0099] Therefore, when calculating the gradient, two values, namely the magnitude and the angle (representing the direction of the gradient), will be obtained. Figure 5 Shows the representation method of the gradient. Among them, each gradient contains two different values, namely the magnitude and the angle. For the convenience of observation, a visual representation method is used here. For example, the value "2↑" at the upper left vertex actually represents a binary pair "(2, 90)", indicating that the magnitude of the gradient is 2 and the angle is 90°.

[0100] The non-maximum suppression process, after obtaining the magnitude and direction of the gradient, traverses the pixels in the image and removes all non-edge points. In the specific implementation, the pixels are traversed one by one to determine whether the current pixel is the maximum value among the surrounding pixels with the same gradient direction, and whether to suppress the point is determined according to the judgment result.

[0101] As can be seen from the above description, this step is a process of edge thinning. For each pixel point:

[0102] (1) If the point is a local maximum in the positive / negative gradient direction, then keep the point.

[0103] (2) If not, then suppress the point (set it to zero). For Figure 6 example, points A, B, and C have the same direction (the gradient direction is perpendicular to the edge). Determine whether these three points are local maxima of their respective positions: if so, keep the point; otherwise, suppress the point (set it to zero).

[0104] Through comparison and judgment, it can be seen that point A has the largest local value, so keep point A (referred to as the edge), and the other two points (B and C) are suppressed (set to zero). In Figure 7 the figure, the points on the black background are all local maxima in the upward gradient direction (horizontal edge). Therefore, these points will be kept; the other points are suppressed (processed as 0). This means that these points on the black background will ultimately be processed as edge points, while other points are processed as non-edge points.

[0105] "Positive / negative gradient direction" refers to the gradient directions in opposite directions. For example, in Figure 8 the figure, the pixel points on the black background are all local maxima in the vertical gradient direction (upward and downward) (i.e., horizontal edge). These points will ultimately be processed as edge points. After the above processing, for several edge points in the same direction, basically only one is kept, thus achieving the purpose of edge thinning.

[0106] In the process of applying double thresholds to determine edges, after non-maximum suppression is completed, the strong edges of the image are already in the currently obtained edge image. However, some spurious edges may also be in the edge image. These spurious edges may be generated by the real image or due to noise. The spurious edges generated by noise must be removed.

[0107] Therefore, two thresholds are set, namely the high threshold (maxVal) and the low threshold (minVal). According to the relationship between the gradient magnitude of the current edge pixel and these two thresholds, the attributes of the edge are judged. The specific steps are as follows:

[0108] S21: If the gradient magnitude of the current edge pixel is greater than or equal to the high threshold, then mark the current edge pixel as a strong edge (to be kept).

[0109] S22: If the gradient magnitude of the current edge pixel is between the high threshold and the low threshold, mark the current edge pixel as a virtual edge (to be retained).

[0110] S23: If the gradient magnitude of the current edge pixel is less than or equal to the low threshold, suppress the current edge pixel (to be discarded).

[0111] In the above process, the virtual edges are obtained in the embodiments of the present application and need to be further processed. Generally, it is determined which case the virtual edge belongs to by judging whether the virtual edge is connected to the strong edge. Usually, for a virtual edge: if it is connected to the strong edge, the edge is processed as an edge. If it is not connected to the strong edge, the edge is a weak edge and is suppressed.

[0112] Such as Figure 9 , the left side shows three edge information, and the right side is a schematic diagram of classifying the edge information, and the specific classification is as follows:

[0113] The gradient value of point A is greater than maxVal, so A is a strong edge.

[0114] The gradient values of points B and C are between maxVal and minVal, so B and C are virtual edges.

[0115] The gradient value of point D is less than minVal, so D is suppressed (discarded). It should be noted that the high threshold and the low threshold are not fixed and need to be defined for different images.

[0116] Optionally, in an embodiment of the present invention, the first feature is processed by an appearance defect detection model to obtain an appearance detection result, including the following steps:

[0117] Preprocess the first feature to obtain the third information;

[0118] Extract the features of the third information through the basic network to obtain the second feature;

[0119] Process the second feature through the feature pyramid to obtain the third feature;

[0120] Predict the third feature through the detection head to obtain the appearance detection result.

[0121] In some possible embodiments, for appearance defect detection (such as appearance scratches, spots, wormholes, etc.), shape recognition, and appearance inspection, a Yolo v8 neural network model is used. If a defect is detected in the appearance, it is determined as a non-conforming product category such as appearance scratches, spots, wormholes, etc. The appearance detection method is a very common detection method, which mainly focuses on the morphology, color, smell, and other physical properties of agricultural products. For example, for fruits and vegetables, the appearance detection method can judge their maturity and freshness by observing features such as fruit shape, color, and texture. For animal products, the appearance detection method can determine the freshness of the product and whether it is contaminated by observing features such as texture, color, and smell.

[0122] YOLOv8 is an object detection model based on the YOLO (You Only Look Once) series of algorithms. It is an improved version of YOLOv3, which further improves the detection accuracy and speed by introducing some new technologies and optimizations.

[0123] The calculation model of YOLOv8 mainly includes the following key steps:

[0124] S31. Input preprocessing: The model accepts the input image and performs preprocessing on the image. The preprocessing includes operations such as resizing the image to a fixed input size and normalizing the pixel values.

[0125] S32. Base network: YOLOv8 uses Darknet-53 as the base network. Darknet-53 is a 53-layer convolutional neural network used to extract features of the image.

[0126] S33. Feature pyramid: Based on Darknet-53, YOLOv8 introduces a feature pyramid network. The feature pyramid network obtains multi-scale feature representations by applying convolutional kernels of different scales to feature maps at different levels. This helps to detect targets of different sizes.

[0127] S34. Detection head: YOLOv8 uses three detection heads, which are respectively used to predict target boxes of different scales. Each detection head consists of a series of convolutional layers and fully connected layers, which are used to convert the feature map into the coordinates and class probabilities of the target box.

[0128] S35. Prediction output: By decoding the output of the detection head, information such as the position, class, and confidence of each target box can be obtained. To improve the detection accuracy, YOLOv8 also uses some techniques such as multi-scale prediction, screening threshold, and non-maximum suppression.

[0129] S36. Post - processing: Finally, the model performs post - processing on the prediction results, including operations such as filtering target boxes with low confidence, decoding the positions of the boxes, and applying non - maximum suppression.

[0130] Generally speaking, YOLOv8 achieves efficient and accurate object detection by combining technologies such as Darknet - 53, feature pyramid, and multi - scale prediction. It strikes a good balance between speed and accuracy and is suitable for object detection tasks in real - time scenarios.

[0131] Optionally, in an embodiment of the present invention, the appearance defect detection model is trained through the following steps:

[0132] Obtain image samples of fleshy fruits to be sorted; the image samples of fleshy fruits include the true sorting results of the fleshy fruit samples.

[0133] Perform image segmentation processing on the image samples of fleshy fruits to obtain the first feature samples.

[0134] Process the first feature samples through the appearance defect detection model to obtain sample detection results; calculate the loss value according to the sample detection results and the true sorting results using a loss function, and update the parameters of the appearance defect detection model according to the loss value to obtain the trained appearance defect detection model.

[0135] Optionally, in an embodiment of the present invention, the image samples of fleshy fruits include positive samples and negative samples, and the loss value is determined through the following steps:

[0136] Determine the first confidence loss, the first localization loss, and the first category loss according to the positive samples.

[0137] Determine the second confidence loss according to the negative samples.

[0138] Determine the loss value according to the first confidence loss, the second confidence loss, the first localization loss, and the first category loss.

[0139] In some possible embodiments, the embodiments of the present application select samples through positive and negative sample matching rules. Assign a positive sample to each groundtrue box, and this positive sample is a prediction box with the largest overlapping area with the gt_box among all bboxes, that is, the prediction box with the largest iou with the gt_box. However, if this rule is used to find positive samples, the number of positive samples is very small, which will make it difficult to train the network. If a sample is not a positive sample, it has neither a localization loss nor a classification loss, only a confidence loss. Therefore, the embodiments of the present application divide the loss into three parts: confidence loss, localization loss, and classification loss. Only positive samples in the embodiments of the present application have these three types of losses, while negative samples only have confidence loss. Therefore, during the loss calculation process, the anchor template needs to be divided into two cases of positive samples and negative samples to calculate the loss. To improve the prediction efficiency and accuracy of the model.

[0140] Optionally, in an embodiment of the present invention, according to the type and shape area of each fruit in the first fruit set, determining the quality category of the fruit includes:

[0141] Determine a first empirical value according to the quotient of the weight and shape area of each fruit in the first fruit set; find the corresponding quality in the classification standard according to the type and the first empirical value to determine the quality category of the fruit;

[0142] Or, determine a second empirical value according to the quotient of the weight and shape area of each fruit in the first fruit set; sort the second empirical values in descending or ascending order according to the type, and determine the quality category of the fruit according to the sorting result.

[0143] In some possible embodiments, the embodiments of the present application implement an image segmentation algorithm through the Canny operator to obtain the circumscribed rectangle area cm 2 , and obtain the product weight g through gravity sensing; and determine the category level of the nectarine according to the ratio range value of the weight to the circumscribed rectangle area; in the embodiments of the present application, the calculation process of the first empirical value is: k = weight / area, with the unit of g / cm2. The embodiments of the present application can perform quality classification through a classification standard. The classification standard can be in the form of a table, and the specific definition can be: Class-A: k >= 0.5; Class-B: 0.5 > k >= 0.3; Class-C: k < 0.3. Of course, the calculation process of the second empirical value in the embodiments of the present application is the same as that of the first empirical value. For the same type of fruit, sort the second empirical values and divide the quality category of the fruit according to the sorting result.

[0144] It can be understood that the detection image processing algorithm for the quality of the flesh of fleshy fruits can be implemented in various ways. Specifically:

[0145] 1. Color analysis algorithm: By obtaining color information in the image, such as RGB channel values, the maturity of the flesh can be evaluated. Mature flesh usually has specific color characteristics, such as the skin color and flesh color of the fruit, and the quality of the flesh can be judged based on these characteristics.

[0146] 2 Shape feature extraction algorithm: This algorithm evaluates the integrity and quality of the fruit by extracting the shape features of the flesh, such as roundness, aspect ratio, etc. For example, more mature fruits usually have more regular shapes.

[0147] 3. Texture analysis algorithm: By analyzing the texture characteristics of the pulp, such as lines, cell structure, etc., the texture and taste of the fruit can be evaluated. Mature and high-quality pulp often has delicate and uniform texture characteristics.

[0148] 4 Algorithms based on deep learning: In recent years, deep learning technology has made great progress in the field of image processing. By using deep learning models such as convolutional neural networks (CNN), features can be extracted and classified from pulp images to determine the quality of the pulp. This method usually requires a large amount of training data and model optimization.

[0149] It should be noted that different fruits may require different algorithms to detect the quality of the flesh. In specific practical applications, technical personnel in this field also need to select a suitable algorithm based on the characteristics of the fruit.

[0150] From the perspective of image processing, the following methods can be used to judge the uniformity of the flesh:

[0151] 1. Grayscale equalization: Grayscale equalization technology can enhance the contrast of the image, making the details in the image more clearly visible. By grayscale equalization of the image, the uniformity of the flesh can be observed more accurately, such as whether there are obvious light and dark areas or color differences.

[0152] 2. Texture analysis: Texture feature extraction algorithms can be used to evaluate the uniformity of the flesh. Specifically, they include gray-level co-occurrence matrix (GLCM), local binary pattern (LBP), etc. These algorithms can analyze the texture information in the image, such as the continuity and uniformity of the lines, to determine the uniformity of the flesh.

[0153] 3. Mean filter: The mean filter can smooth the noise in the image and average the grayscale value in the image. By performing mean filter processing on the image, the subtle noise in the pulp image can be reduced, making the uniformity of the pulp easier to observe.

[0154] 4. Fleshy part segmentation: Through image segmentation algorithms, the pulp is separated from other parts (such as peel and seeds), and then the uniformity analysis is carried out for the pulp part. Common image segmentation methods include threshold segmentation, edge detection, region-based segmentation, etc. According to the uniformity characteristics of the pulp, the quality of the pulp can be evaluated.

[0155] It should be noted that judging the uniformity of the pulp texture not only depends on the image processing algorithm, but also needs to be comprehensively evaluated in combination with the actual scenario and experience. Therefore, in practical applications, image processing can be combined with other technical means to obtain more accurate results.

[0156] The embodiment of this application adopts the grid method of image processing to test the uniformity of the pulp texture. The grid method divides the image into a series of grids or small regions, and then analyzes and compares each small region to evaluate the uniformity of the pulp.

[0157] The specific steps are as follows:

[0158] 1. Image segmentation: The pulp image is segmented into grids or small regions. Such as based on threshold segmentation, edge detection, region growing and other methods, the pulp image is segmented into a series of smaller regions.

[0159] 2. Feature extraction: For each small region, relevant features can be extracted to describe the uniformity of the pulp texture. Specific features include the mean value of brightness (gray level), variance, texture features (such as gray level co-occurrence matrix, local binary pattern, etc.).

[0160] 3. Feature comparison: For the features extracted from each small region, they can be compared with the corresponding features of other regions. For example, the variance of the feature values of each small region can be calculated and compared with the variances of other regions to evaluate the uniformity of the pulp.

[0161] 4. Result analysis: According to the results of feature comparison, the information about the uniformity of the pulp texture can be obtained. If the feature values of all small regions are similar and the variance is small, it can be judged that the pulp texture is relatively uniform; on the contrary, if the feature values vary greatly and the variance is large, it may indicate that the pulp texture is uneven.

[0162] It should be noted that in addition to the grid method of image processing, other technical means can also be combined to further evaluate the uniformity of the pulp texture, such as using machine learning algorithms for classification and evaluation. At the same time, more factors need to be considered for accurately evaluating the uniformity of the pulp texture, such as lighting conditions, shooting angles, etc. Therefore, combining multiple methods will be more accurate and reliable.

[0163] In summary, the method provided by the embodiment of the present application includes: obtaining an image of a fleshy fruit to be sorted; performing image segmentation processing on the image of the fleshy fruit to obtain a first feature; processing the first feature through an appearance defect detection model to obtain an appearance detection result; and performing elimination processing on the fleshy fruit based on the appearance detection result to obtain a first fruit set; determining the quality category of the fruit according to the type and shape area of each fruit in the first fruit set, and sorting the fruit according to the quality category. By performing image segmentation processing on the image of the fleshy fruit, the detection efficiency of the appearance defect detection model is improved in the embodiment of the present application; through the processing of the appearance defect detection model, the fruits with appearance defects are eliminated; then, according to the type and shape area of the fruit, the quality category of the fruit is determined, and the fruit is sorted according to the quality category, thereby improving the accuracy of sorting.

[0164] Secondly, refer to the attached Figure 10 Describe a nectarine sorting system based on visual detection and deep neural network proposed according to an embodiment of the present invention.

[0165] Figure 10 FIG. is a schematic structural diagram of a nectarine sorting system based on visual detection and deep neural network according to an embodiment of the present invention. The system specifically includes:

[0166] An image acquisition module 310, configured to obtain an image of a fleshy fruit to be sorted;

[0167] An image processing module 320, configured to perform image segmentation processing on the image of the fleshy fruit to obtain a first feature;

[0168] A neural network classification module 330, configured to process the first feature through an appearance defect detection model to obtain an appearance detection result; and perform elimination processing on the fleshy fruit based on the appearance detection result to obtain a first fruit set;

[0169] A sorting control module 340, configured to determine the quality category of the fruit according to the type and shape area of each fruit in the first fruit set, and sort the fruit according to the quality category.

[0170] It can be seen that the content in the above method embodiments is applicable to the system embodiments of the present invention. The functions specifically implemented by the system embodiments of the present invention are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0171] Refer to Figure 11 , an embodiment of the present invention provides a nectarine sorting device based on visual detection and deep neural network, including:

[0172] At least one processor 410;

[0173] At least one memory 420, configured to store at least one program;

[0174] When the at least one program is executed by the at least one processor 410, the at least one processor 410 is caused to implement the nectarine sorting method based on visual detection and deep neural network.

[0175] Similarly, the content in the above method embodiments is applicable to the present device embodiment. The functions specifically implemented by the present device embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0176] The embodiment of the present invention further provides a computer-readable storage medium, which stores a program executable by a processor. The program executable by the processor is used to execute the above nectarine sorting method based on visual detection and deep neural network when executed by the processor.

[0177] Similarly, the content in the above method embodiments is applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0178] In some alternative embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0179] In addition, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It should also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. Rather, given the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Thus, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It should also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0180] If the described function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several programs for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0181] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable programs for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by a program execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can retrieve and execute programs from the program execution system, apparatus, or device), or in conjunction with these program execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with a program execution system, apparatus, or device.

[0182] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0183] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0184] In the foregoing description of the present specification, descriptions with reference to the terms "one embodiment / example", "another embodiment / example", or "certain embodiments / examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0185] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0186] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of the present invention.

Claims

1. A sorting method for honey peaches based on visual detection and deep neural network, characterized in that, It includes the following steps: Obtain the images of fleshy fruits to be sorted; Perform image segmentation processing on the images of the fleshy fruits to obtain the first features; Process the first features through an appearance defect detection model to obtain an appearance detection result; and perform elimination processing on the fleshy fruits according to the appearance detection result to obtain a first fruit set; Determine the quality category of the fruits according to the type and shape area of each fruit in the first fruit set, and sort the fruits according to the quality category; The appearance defect detection model is trained through the following steps: Obtain image samples of fleshy fruits to be sorted; the image samples of the fleshy fruits include the true sorting results of the fleshy fruit samples; Perform image segmentation processing on the image samples of the fleshy fruits to obtain first feature samples; Process the first feature samples through an appearance defect detection model to obtain sample detection results; and calculate a loss value according to the sample detection results and the true sorting results using a loss function, and update the parameters of the appearance defect detection model according to the loss value to obtain the trained appearance defect detection model; The image samples of the fleshy fruits include positive samples and negative samples, and the loss value is determined through the following steps: Determine a first confidence loss, a first localization loss, and a first category loss according to the positive samples; Determine a second confidence loss according to the negative samples; determine the loss value according to the first confidence loss, the second confidence loss, the first localization loss, and the first category loss; During the loss calculation, the anchor template is divided into two cases of positive samples and negative samples to calculate the loss; The positive samples have a first confidence loss, a first localization loss, and a first category loss, while the negative samples only have a second confidence loss; The method further includes: Scan each fruit in the first fruit set through an infrared camera to perform flesh detection on the fruits, and update the first fruit set according to the flesh detection results; The step of scanning each fruit in the first fruit set through an infrared camera to perform flesh detection on the fruits includes: providing an adjustable light source through a lighting device, scanning the first fruit through an infrared camera to determine the thermal radiation information of the first fruit; the first fruit set includes the first fruit; Determine the internal features of the first fruit according to the thermal radiation information, and perform flesh detection on the first fruit; The step of performing image segmentation processing on the images of the fleshy fruits to obtain the first features includes: Perform Gaussian filtering processing on the images of the fleshy fruits to obtain a first image; Calculate the gradient of the first image to obtain first information; Perform non-maximum suppression processing on the first information to obtain second information; Perform double-threshold screening processing on the second information to obtain the first features; The step of determining the quality category of the fruits according to the type and shape area of each fruit in the first fruit set includes: Determine a first empirical value according to the quotient of the weight and the shape area of each fruit in the first fruit set; search for the corresponding quality in the classification standard according to the type and the first empirical value to determine the quality category of the fruits; Alternatively, determine a second empirical value according to the quotient of the weight of each fruit in the first fruit set and the shape area; sort the second empirical values in descending or ascending order according to the types, and determine the quality category of the fruit according to the sorting result. The processing of the second information through double-threshold screening includes: Set two thresholds, namely a high threshold and a low threshold; judge the attribute of the edge according to the relationship between the gradient magnitude of the current edge pixel and the high threshold and the low threshold. The specific steps are as follows: If the gradient magnitude of the current edge pixel is greater than or equal to the high threshold, mark the current edge pixel as a strong edge and it needs to be retained. If the gradient magnitude of the current edge pixel is between the high threshold and the low threshold, mark the current edge pixel as a virtual edge and it needs to be retained. If the gradient magnitude of the current edge pixel is less than or equal to the low threshold, suppress the current edge pixel and it needs to be discarded.

2. The sorting method for honey peaches based on visual detection and deep neural network according to claim 1, characterized in that, The processing of the first feature through the appearance defect detection model to obtain the appearance detection result includes the following steps: Preprocess the first feature to obtain third information. Extract the features of the third information through the basic network to obtain the second feature. Process the second feature through the feature pyramid to obtain the third feature. Predict the third feature through the detection head to obtain the appearance detection result.

3. A nectarine sorting system for visual detection and deep neural network for implementing the nectarine sorting method of visual detection and deep neural network described in claim 1, characterized in that, Including: An image acquisition module for acquiring an image of a fleshy fruit to be sorted. An image processing module for performing image segmentation processing on the image of the fleshy fruit to obtain the first feature. A neural network classification module for processing the first feature through an appearance defect detection model to obtain an appearance detection result; and performing elimination processing on the fleshy fruit according to the appearance detection result to obtain the first fruit set. A sorting control module for determining the quality category of the fruit according to the type and shape area of each fruit in the first fruit set, and sorting the fruit according to the quality category.

4. A nectarine sorting device based on visual detection and deep neural network, characterized in that, Including: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method for sorting nectarines based on visual detection and deep neural network according to any one of claims 1 to 2.

5. A computer-readable storage medium storing a program executable by a processor, characterized in that, The program executable by the processor is used to implement the method for sorting nectarines based on visual detection and deep neural network according to any one of claims 1 to 2 when executed by the processor.

Images