Agricultural product sorting method and device based on machine vision

Abstract

This application relates to the field of intelligent sorting technology for agricultural products, and provides a machine vision-based method and apparatus for sorting agricultural products, including: acquiring texture images and depth images of target agricultural products; inputting the texture images and depth images into a pre-trained instance segmentation model for cross-modal feature fusion and coarse segmentation processing, and outputting an initial instance mask; calculating the distribution mutation features in the corresponding region of the initial instance mask based on the depth image, and determining whether a preset mutation condition is met; if met, performing fine segmentation processing on the initial instance mask to obtain a precise individual mask; otherwise, directly marking it as a precise individual mask; evaluating the quality level of the precise individual mask and generating sorting control instructions. This application effectively solves the problems of segmentation errors and defect omissions caused by adhesion and stacking during the sorting process of agricultural products, and improves sorting accuracy.

Description

Technical Field

[0001] This application relates to the field of intelligent sorting technology for agricultural products, specifically to a sorting method and apparatus for agricultural products based on machine vision. Background Technology

[0002] Currently, in the field of machine vision sorting of spherical or near-spherical agricultural products, high-speed conveying with dense flow is usually adopted to improve production capacity. However, due to insufficient uniformity of feeding and fluctuations in the operation of the guiding mechanism, fruits are prone to sticking together or even stacking vertically, forming complex imaging scenarios of multiple targets sticking together. Under such conditions, existing technical solutions face multiple dilemmas: traditional two-dimensional image segmentation algorithms rely heavily on visible contour information. When the overlap of fruits is high or the color and texture are similar, multiple individuals are easily mistakenly merged into a single individual, or surface spots are misjudged as segmentation boundaries, resulting in oversegmentation or undersegmentation; pure three-dimensional depth map segmentation can distinguish fruits at different height levels based on depth jumps, but it cannot identify individuals at the same height level that are in close contact, and at the same time, it completely loses key information such as color and surface texture for defect detection; single deep learning instance segmentation models have a natural upper limit to their generalization ability for severe occlusion and complex stacking patterns, and usually do not output quantitative occlusion indicators. More importantly, existing technologies generally lack a refined assessment of the degree of surface occlusion of the segmented individuals. This results in either directly sending occluded individuals into the grading process, leading to missed defects, or aggressive rejection, resulting in waste of good products, which seriously affects sorting accuracy and food safety. Summary of the Invention

[0003] To address the problems of individual segmentation errors and inaccurate quality assessment caused by the sticking and stacking of agricultural products in existing technologies, which in turn affect the sorting accuracy, this application proposes a machine vision-based agricultural product sorting method and device, which achieves accurate segmentation and differentiated quality control of sticking and stacked agricultural products.

[0004] To achieve the above objectives, this application adopts the following technical solution: A machine vision-based method for sorting agricultural products includes: Acquire the texture and depth images of the target agricultural product; The texture image and depth image are input into the pre-trained instance segmentation model for cross-modal feature fusion and coarse segmentation, and the initial instance mask is output. The distribution mutation features within the region corresponding to the initial instance mask are calculated based on the depth image, and it is determined whether the distribution mutation features meet the preset mutation conditions. If the distribution mutation features meet the mutation conditions, then the initial instance mask is finely segmented based on the depth image to obtain multiple precise individual masks; If the distribution mutation characteristics do not meet the mutation conditions, the initial instance mask is marked as the precise individual mask; The quality level of precise individual masks is evaluated, and corresponding sorting control instructions are generated.

[0005] The above scheme achieves adaptive and accurate separation of individual agricultural products under complex stacking conditions by combining multimodal image acquisition and cross-modal fusion coarse segmentation with a strategy of triggering fine segmentation by depth distribution mutation detection, thus avoiding the segmentation failure problem of single modality in adhesive scenarios.

[0006] As one implementation method, determining whether the distribution mutation characteristics meet preset mutation conditions includes: The depth gradient vector of each pixel in the region corresponding to the initial instance mask is calculated based on the depth image to obtain the gradient vector field. Histogram statistics of gradient directions in the gradient vector field are performed based on a sliding window of preset size to obtain a gradient direction distribution histogram. The directional distribution characteristics of the corresponding sliding window are calculated based on the gradient directional distribution histogram; the directional distribution characteristics include directional span and quadrant distribution uniformity. If the directional distribution characteristics meet the preset stacking distribution conditions, the corresponding sliding window is marked as a stacking region; Traverse the gradient vector field to count the number of regions in the stacked region. If the number of regions is greater than a preset threshold, calculate the distance between the geometric center points of every two different stacked regions to obtain the region spacing. If there is a spacing value within the preset spacing range in the regional spacing, then the distribution mutation feature is determined to meet the mutation condition.

[0007] This implementation method, through the analysis of the directional span and quadrant uniformity of the gradient vector field, can geometrically distinguish between surface depressions of a single fruit and V-shaped grooves formed by the stacking of multiple fruits, effectively preventing false triggering caused by natural structures such as causal depressions and improving the robustness of stacking detection.

[0008] As one implementation method, the method for obtaining the spacing interval includes: Calculate the equivalent diameter of the mask based on the total number of pixels in the initial instance mask; The spacing upper limit is constructed based on the mask equivalent diameter and a preset upper limit coefficient; the upper limit coefficient is constructed based on the projection constraints of the stacking geometry model of the target agricultural product; Calculate the discrete diameter of the pixels inside any two stacked regions, and mark the sum of the two discrete diameters as the lower limit of the spacing; The interval formed by the upper and lower limits of the spacing is marked as the spacing interval.

[0009] This implementation uses a spherical contact geometry model to construct a dynamic spacing range, which not only eliminates the pseudo-stacked boundaries caused by noise splitting, but also avoids misjudging the independent defects of separated fruits as stacking, thus ensuring the physical rationality of fine segmentation triggering.

[0010] As one implementation method, the calculation of the discrete diameter includes: Pixels whose gradient magnitude in the stacked region is greater than a preset gradient threshold are selected to obtain a high-magnitude pixel set; the preset gradient threshold is determined based on the average gradient magnitude of the stacked region. Perform connected component analysis on the high-amplitude pixel set and retain the connected component with the largest area to obtain the maximum connected component; The maximum Euclidean distance between all pairs of pixels within the largest connected region is marked as the discrete diameter of the corresponding stacked region.

[0011] This implementation extracts the effective geometric scale of the stacked region through adaptive gradient threshold filtering and connected component analysis, eliminating low-amplitude noise interference and making the calculation of the lower limit of the spacing more accurately reflect the physical width of the stacked boundary.

[0012] As one implementation method, the fine segmentation processing of the initial instance mask based on the depth image includes: The depth gradient magnitude of each pixel in the region corresponding to the initial instance mask is calculated based on the depth image to obtain the gradient magnitude map. Based on the gradient magnitude map, the watershed algorithm is used to segment the initial instance mask to obtain the initial segmentation boundary. The initial segmentation boundary is locally corrected based on the edge features of the texture image to obtain the corrected segmentation boundary. Based on the corrected segmentation boundary, the initial instance mask is segmented into multiple initial individual masks. The individual occlusion degree of multiple initial individual masks is evaluated and filtered to obtain multiple precise individual masks.

[0013] This implementation adopts a cascaded strategy that uses depth to dominate the segmentation direction and texture to assist in accuracy correction. While ensuring the correctness of the segmentation topology, it improves the boundary positioning accuracy and selects reliable individuals through occlusion evaluation, thus forming a complete fine segmentation closed loop.

[0014] As one implementation method, the local correction of the initial segmentation boundary based on the edge features of the texture image includes: Within the local neighborhood of the initial segmentation boundary, detect whether there is a texture edge in the corresponding texture image, and the positional deviation of the texture edge from the initial segmentation boundary is within a preset pixel range; If a texture edge that satisfies the positional deviation condition exists, a depth value profile is extracted on the corresponding depth image along the normal direction of the texture edge, and a depth jump is determined based on the depth value profile; the depth jump refers to the statistical difference of the depth values ​​in the depth value profile being greater than a preset jump threshold. If there is a depth jump, the texture edge replaces the initial segmentation boundary in the corresponding local neighborhood to obtain the corrected segmentation boundary; If there are no texture edges or depth transitions, the initial segmentation boundary within the corresponding local neighborhood is marked as the corrected segmentation boundary.

[0015] This implementation establishes a verification mechanism of "texture proposal and depth adjudication". Correction is only adopted when the texture edge is confirmed by a depth geometric change. This fundamentally eliminates the misleading effect of texture noise such as water stains and color spots on the segmentation boundary, and takes into account both accuracy and noise resistance.

[0016] As one implementation method, the precise individual mask is obtained by: The initial individual mask is fitted with a complete three-dimensional geometric model based on the depth image to obtain the unobstructed projected area. Calculate the pixel ratio of the initial individual mask to the unobstructed projected area to obtain the overall visibility; Based on depth and texture images, surface features of agricultural products within the area corresponding to the initial individual mask are extracted; The visibility of key areas is obtained by comparing and calculating the surface features of agricultural products with the preset key area features; the key area features are the features of the corresponding areas in multiple areas of agricultural products where the probability of defects occurring is higher than a preset probability threshold. A weighted fusion assessment of overall visibility and key area visibility is performed to obtain an occlusion score. If the occlusion score is less than the preset occlusion score threshold, the corresponding initial individual mask is marked as a reflow individual, and a reflow control command is generated. If the occlusion score is greater than or equal to the preset occlusion score threshold, the corresponding initial individual mask is marked as the precise individual mask.

[0017] This implementation method introduces biological morphology priors and performs weighted occlusion assessment on areas with high defect incidence, avoiding the drawbacks of the traditional overall area method that ignores the occlusion of key parts. Combined with the reflow mechanism, it ensures that all fruits entering the quality assessment have sufficient surface visibility, significantly reducing the missed detection rate.

[0018] In one implementation, the instance segmentation model includes a dual-branch feature extraction backbone, a cross-modal feature fusion module, and an instance segmentation output head; The dual-branch feature extraction backbone includes a texture feature extraction branch and a depth feature extraction branch, which are used to extract texture feature maps and depth geometric feature maps from the texture image and the depth image, respectively; The cross-modal feature fusion module is used to fuse texture feature maps and depth geometric feature maps at multiple scales to obtain a multi-scale fused feature map; the feature fusion method is channel dimension concatenation, element-wise addition, or feature weighted fusion based on an attention mechanism. The instance segmentation output head is used to output pixel-by-pixel instance-aware feature maps through convolutional layers based on multi-scale fused feature maps, and to perform pixel-by-pixel classification or mask decoding on the instance-aware feature maps to obtain an initial instance mask.

[0019] This implementation uses a dual-branch architecture and multi-scale cross-modal fusion to enable the network to learn appearance details and geometric structure information simultaneously, enhancing the model's semantic understanding and instance perception of adhered and overlapping targets, and providing a high-quality coarse segmentation foundation for subsequent fine processing.

[0020] As one implementation method, the quality level evaluation of the precise individual mask includes: Based on the texture image, color analysis and surface defect detection are performed on the corresponding area of ​​the precise individual mask to obtain the texture evaluation result; Based on depth images, the size and shape of the corresponding region of the precise individual mask are measured and evaluated to obtain geometric evaluation results; Based on the texture evaluation results and geometric evaluation results, the quality grade of agricultural products corresponding to the precise individual masks is determined according to the preset quality grade judgment rules. Based on the quality grade of agricultural products, corresponding sorting control instructions are generated.

[0021] This implementation method comprehensively utilizes texture and depth information for multi-dimensional quality judgment, enabling simultaneous evaluation of appearance defects and geometric specifications, and ensuring the comprehensiveness and accuracy of sorting decisions.

[0022] Furthermore, this application also provides a machine vision-based agricultural product sorting device, applied to a machine vision-based agricultural product sorting method as described above, comprising: The image acquisition module is used to acquire texture and depth images of the target agricultural product; The coarse segmentation module is used to input texture images and depth images into a pre-trained instance segmentation model for cross-modal feature fusion and coarse segmentation processing, and output an initial instance mask; The stacking detection module is used to calculate the distribution mutation features in the region corresponding to the initial instance mask based on the depth image, and to determine whether the distribution mutation features meet the preset mutation conditions. The fine segmentation module is used to perform fine segmentation on the initial instance mask based on the depth image if the distribution mutation features meet the mutation conditions, thereby obtaining multiple precise individual masks; if the distribution mutation features do not meet the mutation conditions, the initial instance mask is marked as a precise individual mask. The sorting instruction generation module is used to evaluate the quality level of precise individual masks and generate corresponding sorting control instructions.

[0023] The above-mentioned device solution achieves the technical effects corresponding to the aforementioned methods by modularly integrating multimodal sensing, intelligent segmentation, and quality decision-making functions, providing a reliable hardware carrier for high-speed agricultural product sorting lines.

[0024] Beneficial effects: This application provides a machine vision-based method and apparatus for sorting agricultural products. Through multimodal image fusion and cross-modal instance segmentation, coarse segmentation is achieved, leveraging the complementarity of texture and depth information to significantly improve the initial separation capability of adhered and stacked targets. A stacking criterion is constructed through the directional span and quadrant uniformity analysis of the depth gradient vector field. Combined with dynamic spacing interval verification based on spherical projection constraints, it accurately distinguishes between real stacks and surface depressions from a geometric perspective, avoiding false triggering. A cascaded boundary correction mechanism of "texture proposal, depth adjudication" effectively suppresses surface noise interference by utilizing the high-resolution advantage of texture while using depth jumps as physical authenticity verification. By fusing overall visibility with weighted occlusion assessment based on biomorphological priors for key area visibility, coupled with individual-level backflow control, it ensures that high-incidence areas of key surface defects on fruits entering the quality assessment stage are fully exposed, fundamentally solving the problems of missed detection and false rejection caused by occlusion. This forms a complete closed-loop intelligent sorting system from perception, segmentation, assessment to control. Attached Figure Description

[0025] Figure 1 A flowchart illustrating a machine vision-based agricultural product sorting method provided in this application embodiment; Figure 2 A flowchart illustrating a method for determining whether a distribution mutation feature meets a preset mutation condition, provided in an embodiment of this application; Figure 3 This application provides a schematic flowchart of a method for fine segmentation of an initial instance mask based on a depth image, as illustrated in an embodiment of the present application. Figure 4 This is a device architecture diagram of a machine vision-based agricultural product sorting device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0028] Example 1 like Figure 1 As shown in the figure, this embodiment provides a machine vision-based method for sorting agricultural products. This method achieves accurate identification and sorting of individual agricultural products on a high-speed conveyor belt through multimodal data fusion and adaptive processing strategies. Specifically, the method includes the following steps.

[0029] Step S100 involves acquiring texture and depth images of the target agricultural product. Specifically, texture images are typically acquired using a high-resolution color camera to record details such as color, spots, and texture on the surface of the agricultural product; depth images are acquired using a depth sensor (e.g., a line laser profilometer or a structured light camera) to record the three-dimensional geometry and height distribution information of the agricultural product's surface. During acquisition, pixel-level spatial registration of the texture and depth images is required to ensure that the coordinates of the same physical point strictly correspond in both images. This dual-modal data acquisition method utilizes the complementarity of texture and geometric information, overcoming the difficulty of accurately segmenting single two-dimensional images when there is uneven lighting or fruit adhesion, while also avoiding the limitation of losing surface defect features in single three-dimensional data, providing a complete data foundation for subsequent processing.

[0030] Step S200 involves inputting the texture image and depth image into the pre-trained instance segmentation model for cross-modal feature fusion and coarse segmentation, outputting an initial instance mask. Specifically, the pre-trained instance segmentation model can simultaneously receive image data from two modalities and perform feature-level interaction and fusion within the network. The coarse segmentation here refers to the model first separating potential agricultural product target regions from the complex background and attempting preliminary instance-level differentiation of adhered or overlapping targets. The output initial instance mask represents the model's preliminary prediction result for a single agricultural product. It should be noted that due to potential severe stacking or occlusion of agricultural products on the conveyor belt, relying solely on the model's end-to-end prediction often fails to obtain perfect individual boundaries under all conditions. Therefore, this initial instance mask may be an accurate single-fruit mask or a merged mask containing multiple adhered fruits. Its accuracy is insufficient for direct use in refined quality grading and requires further verification and optimization in subsequent steps.

[0031] Step S300 involves calculating the distribution mutation features within the corresponding region of the initial instance mask based on the depth image, and determining whether the distribution mutation features meet preset mutation conditions. This step is the core triggering mechanism of the technical solution of this application. Specifically, the distribution mutation features are quantitative indicators used to characterize the degree of drastic depth changes and their spatial distribution patterns within the mask region. The significance of introducing this feature lies in geometrically distinguishing between "natural depressions of a single fruit" and "boundary gullies formed by stacked fruits." For example, although the fruit stalk depression on the surface of an apple also produces depth changes, its gradient distribution usually exhibits local, unidirectional convergence characteristics; while the V-shaped contact area formed by two stacked fruits often exhibits bidirectional opposition or multidirectional divergent mutation characteristics in its depth gradient. By determining whether the distribution mutation features meet preset conditions, the system can intelligently identify which initial masks truly have stacking risks, thereby avoiding sending all masks into the time-consuming fine segmentation process. This prevents missegmentation of natural depressions of single fruits and significantly improves the detection rate and processing efficiency of real stacked targets.

[0032] In step S400, if the distribution mutation features meet the mutation conditions, the initial instance mask is finely segmented based on the depth image to obtain multiple precise individual masks; if the distribution mutation features do not meet the mutation conditions, the initial instance mask is marked as a precise individual mask. Specifically, this is an adaptive conditional branching processing strategy. When stacking is detected, the system activates the fine segmentation process, using the rich geometric boundary information contained in the depth image to perform secondary fragmentation and correction on the initial mask, separating the multiple fruits that were originally stuck together into independent precise individuals; when no stacking is detected or the mutation features are not obvious, the system considers the current initial instance mask to be accurate enough and directly passes it as a precise individual mask to the next stage. This design not only realizes the dynamic allocation of computing resources and avoids ineffective computing power consumption in simple scenarios, but more importantly, it establishes a safety barrier to ensure that additional segmentation operations are only introduced when stacking interference is certain, minimizing the risk of damage to normal fruits due to overprocessing.

[0033] Step S500 involves evaluating the quality grade of the precise individual masks and generating corresponding sorting control instructions. Specifically, the quality grade evaluation is conducted after obtaining reliable precise individual masks, meaning the evaluation object is a verified, clearly defined individual agricultural product entity. The evaluation process integrates color and defect information from the texture image with size and shape information from the depth image to determine the quality grade of the agricultural product according to preset grading standards. Subsequently, based on the determined quality grade, corresponding sorting control instructions are generated to drive the backend actuators (such as valves, levers, or robotic arms) to divert the agricultural product to the corresponding collection or rejection channel. Through the aforementioned layers of screening and correction, the masks that finally enter the quality evaluation stage possess high accuracy and integrity, thereby fundamentally ensuring the reliability of sorting decisions and effectively solving the problem of inaccurate grading or missed detection caused by segmentation errors in traditional solutions.

[0034] Example 2 Based on Example 1, this example further details the specific implementation method for determining whether the distribution mutation characteristics meet the preset mutation conditions. For example... Figure 2 As shown, as one implementation method, determining whether the distribution mutation feature meets the preset mutation conditions includes: Step S301: Calculating the depth gradient vector of each pixel in the region corresponding to the initial instance mask based on the depth image to obtain a gradient vector field; Step S302: Performing histogram statistics on the gradient directions in the gradient vector field based on a sliding window of a preset size to obtain a gradient direction distribution histogram; Step S303: Calculating the directional distribution characteristics of the corresponding sliding window based on the gradient direction distribution histogram; the directional distribution characteristics include directional span and quadrant distribution uniformity; Step S304: If the directional distribution characteristics meet the preset stacking distribution conditions, then the corresponding sliding window is marked as a stacking region; Step S305: Traversing the gradient vector field to count the number of stacking regions; if the number of regions is greater than a preset threshold, then calculating the distance between the geometric center points of every two different stacking regions to obtain the region spacing; Step S306: If there is a spacing value in the region spacing that is within a preset spacing interval, then it is determined that the distribution mutation feature meets the mutation conditions.

[0035] Specifically, the above steps construct a complete stacking detection logic chain from local gradient analysis to global geometric verification. When calculating the gradient vector field, the Sobel or Prewitt operator can be used to process the depth image to obtain the depth change rate of each pixel in the horizontal and vertical directions, thereby synthesizing a gradient vector with direction and magnitude. This step transforms abstract depth values ​​into vector data representing changes in surface orientation. Subsequently, a sliding window of a preset size (e.g., 15×15 pixels) is used to traverse the gradient vector field, quantifying and statistically analyzing the gradient direction angles (0° to 360°) within the window to generate a gradient direction distribution histogram. The core purpose of introducing the two indicators, directional span and quadrant distribution uniformity, is to geometrically distinguish between the natural depressions on the surface of a single fruit and the boundary gullies formed by the stacking of multiple fruits. For example, while the pedicel depression on an apple surface also generates a depth gradient, its gradient direction typically exhibits a localized, unidirectional convergence characteristic, with a small directional span and highly uneven quadrant distribution. In contrast, the V-shaped contact area formed by two stacked fruits often has a depth gradient pointing towards the center of the groove, resulting in the gradient direction within the window covering more than half a circumference (i.e., a directional span greater than 180°), and showing significant distribution in all four quadrants. Only when the sliding window simultaneously satisfies both the directional span threshold and the quadrant uniformity threshold is it marked as a stacked region. This dual constraint mechanism effectively eliminates false triggers caused by the structure of a single fruit, such as pedicel depressions or insect holes. Furthermore, global verification is performed using the number of regions and the region spacing to ensure that the detected stacked regions spatially conform to the topological relationship of multi-fruit contact, rather than isolated noise points.

[0036] As one implementation method, the method for obtaining the spacing interval includes: calculating the mask equivalent diameter based on the total number of pixels in the initial instance mask; constructing the spacing upper limit based on the mask equivalent diameter and a preset upper limit coefficient; the upper limit coefficient is constructed based on the projection constraints of the stacked geometric model of the target agricultural product; calculating the discrete diameter of the pixels inside any two stacked regions respectively, and marking the sum of the two discrete diameters as the spacing lower limit; and marking the interval formed by the spacing upper limit and the spacing lower limit as the spacing interval.

[0037] Specifically, the dynamic construction of the spacing interval is a key defense against misjudgment in this embodiment. The equivalent diameter of the mask can be calculated using the formula... The calculation yields a value where A represents the total pixel area of ​​the initial instance mask, a parameter that characterizes the macroscopic scale of the currently detected object. The upper limit coefficient is set to a range of 0.6 to 0.85. This value is not arbitrarily chosen but derived based on the physical constraints of the sphere contact geometry model on the two-dimensional projection plane. When two spheres of equal diameter are in close contact, the distance between their contact boundaries in the two-dimensional projection is necessarily less than the projected diameter of the spheres. In actual observation, considering the slight differences in fruit shape and variations in imaging angle, this distance rarely exceeds 0.85 times the projected diameter. Therefore, if the upper limit coefficient is set too large (e.g., greater than 0.85), two completely separated but closely spaced independent fruits may be misjudged as a stack, leading to unnecessary fine segmentation. Conversely, if the upper limit coefficient is set too small (e.g., less than 0.6), stacked fruits with loose contact or gaps may be missed. In a preferred example of this application, the upper limit coefficient is set to 0.75, which minimizes the false alarm rate while ensuring the detection rate. The lower limit of the spacing is determined by the sum of the discrete diameters of the two stacked regions. This represents the minimum physically incompressible gap between two potential stacked entities, eliminating spurious stacking boundaries caused by noise splitting (i.e., cases where the spacing is too small). Through dynamic constraints on the upper and lower limits, the system can adapt to agricultural products of different sizes, ensuring the physical rationality of fine-segmentation triggering.

[0038] As one implementation method, the discrete diameter calculation includes: filtering pixels in the stacked region whose gradient magnitude is greater than a preset gradient threshold to obtain a high-magnitude pixel set; the preset gradient threshold is determined based on the average gradient magnitude of the stacked region; performing connected component analysis on the high-magnitude pixel set and retaining the connected component with the largest area to obtain the maximum connected component; and marking the maximum Euclidean distance between all pairs of pixels in the maximum connected component as the discrete diameter of the corresponding stacked region.

[0039] Specifically, the calculation of the discrete diameter aims to accurately quantify the effective geometric scale of the stacked region. In the gradient vector field, not all pixels represent the true stacking boundary; low-amplitude regions often correspond to gentle changes in surface curvature or sensor noise. Therefore, this embodiment first determines an adaptive threshold based on the average gradient magnitude within the stacked region, retaining only the set of high-amplitude pixels with gradient magnitudes higher than this threshold, thereby filtering out background interference. Subsequently, connected component analysis is performed on the high-amplitude pixel set, and the connected component with the largest area is selected as the carrier of the effective stacking feature. This step further eliminates the influence of isolated noise points. Finally, the maximum value of the Euclidean distance between all pairs of pixels within this largest connected component is calculated and used as the discrete diameter. This definition reflects the most significant extension length of the stacked region on the projection plane, and more accurately reflects the physical span of the stacking boundary than a simple region width or circumcircle diameter. When the sum of the discrete diameters of two stacked regions is used as the lower limit of the spacing, a minimum safe distance based on the target's own scale is actually established. Only when the center-to-center distance between two stacked regions is greater than this minimum safe distance and less than the aforementioned upper limit of the spacing is it confirmed as an effective stacking structure. This strategy, which combines local gradient strength selection with global geometric metric calculation, significantly improves the robustness and accuracy of the stacking decision algorithm under complex conditions.

[0040] Example 3 After determining in Example 1 that the distribution mutation features meet the mutation conditions and trigger fine segmentation processing, this example further details the specific implementation of fine segmentation processing of the initial instance mask based on the depth image. For example... Figure 3 As shown, in one implementation method, the initial instance mask is finely segmented based on the depth image, including: Step S401: Calculate the depth gradient magnitude of each pixel in the corresponding region of the initial instance mask based on the depth image to obtain a gradient magnitude map; Step S402: Based on the gradient magnitude map, perform boundary segmentation on the initial instance mask using the watershed algorithm to obtain the initial segmentation boundary; Step S403: Perform local correction on the initial segmentation boundary based on the edge features of the texture image to obtain the corrected segmentation boundary; Step S404: Perform individual segmentation on the initial instance mask based on the corrected segmentation boundary to obtain multiple initial individual masks; Step S405: Evaluate and filter the individual occlusion degree of the multiple initial individual masks to obtain multiple precise individual masks.

[0041] Specifically, the above steps construct a cascaded segmentation strategy of "depth-driven structure and texture-supplemented accuracy". First, a gradient magnitude map is calculated using the depth image and a watershed algorithm is executed. This step leverages the natural immunity of 3D geometric information to illumination changes and surface texture noise, accurately capturing the V-shaped groove structure formed by fruit stacking from a macroscopic perspective, thus obtaining an initial segmentation boundary with correct topological structure. However, limited by the physical resolution of the depth sensor, the boundary obtained solely based on depth information is often coarse or jagged, making it difficult to accurately fit the true contour of the fruit. Therefore, this application introduces edge features from the texture image to locally correct the initial segmentation boundary, aiming to utilize the high spatial resolution advantage of color images to improve the boundary positioning accuracy. However, to prevent texture noise from misleading segmentation, this correction is not unconditionally adopted but must undergo rigorous subsequent verification. Finally, based on the corrected boundary, the initial instance mask is decomposed into multiple independent initial individual masks, and the accurate individual masks with complete surface information are selected by evaluating the degree of occlusion, thus forming a complete closed loop from coarse segmentation to fine segmentation to quality verification.

[0042] As one implementation method, local correction of the initial segmentation boundary based on the edge features of the texture image includes: detecting whether a texture edge exists in the corresponding texture image within the local neighborhood of the initial segmentation boundary, and ensuring that the positional deviation between the texture edge and the initial segmentation boundary is within a preset pixel range; if a texture edge that satisfies the positional deviation condition exists, then a depth value profile is extracted on the corresponding depth image along the normal direction of the texture edge, and a depth jump is determined based on the depth value profile; a depth jump refers to a statistical difference in depth values ​​of the depth value profile that is greater than a preset jump threshold; if a depth jump exists, then the texture edge replaces the initial segmentation boundary in the corresponding local neighborhood to obtain the corrected segmentation boundary; if no texture edge or depth jump exists, then the initial segmentation boundary in the corresponding local neighborhood is marked as the corrected segmentation boundary.

[0043] Specifically, this correction mechanism embodies the core design philosophy of "texture proposal, depth adjudication," serving as a crucial safeguard for ensuring segmentation accuracy and robustness. During execution, the system first establishes a narrow search band (e.g., a local neighborhood 3 to 5 pixels wide) on both sides of the initial segmentation boundary. Within this region, the Canny or Sobel operator is used to detect texture edges. The purpose of setting positional deviation constraints is to ensure that the texture edges to be corrected are spatially highly consistent with the initial depth-guided boundary, avoiding the erroneous manipulation of irrelevant texture details far from the boundary into segmentation lines. When a candidate texture edge that meets the criteria is detected, the system does not directly adopt it but treats it as a "proposal" and immediately initiates a depth verification procedure: along the normal direction of the texture edge's pixel, a profile line is extracted on the depth image, spanning both sides of the edge, and the statistical difference in depth values ​​between the regions on both sides of the profile line is calculated. This statistical difference can take various forms, such as the difference between the median depth, the difference between the mean depth, or the difference between the weighted average depths, to resist interference from individual abnormal noise points.

[0044] The physical mechanism of introducing depth jump verification lies in the fact that the real fruit stacking boundary is not only a dividing line of color or brightness, but also a surface of height abrupt change in physical space. Conversely, water stains, reflective spots, shadows, or natural color spots commonly found on the surface of agricultural products, although they appear as significant edge features in the texture image, are usually continuous and smooth in the depth dimension, without substantial height jumps. For example, in actual sorting scenarios, if a dry water mark on the surface of an apple runs through the vicinity of the stacking area, traditional pure vision algorithms are very likely to misjudge it as a fruit dividing line, leading to oversegmentation. However, in this embodiment, although the water mark produces a strong edge response in the texture image and is located close to the initial boundary, because the corresponding depth profile shows that the height difference between the two sides is only 0.1mm (far less than the preset 2mm jump threshold), the system determines that there is no depth jump at this point, and therefore rejects the texture edge, maintaining the original depth segmentation boundary unchanged. Conversely, if a real fruit contact seam is detected, its depth profile usually shows a steep drop of several millimeters or even centimeters, far exceeding the preset threshold. Only then will the system confirm the validity of the texture edge and replace the initial boundary with it. This dual verification mechanism fundamentally eliminates the contamination of 3D segmentation results by 2D texture noise, so that the final corrected segmentation boundary has both pixel-level positioning accuracy and maintains strict geometric authenticity.

[0045] Example 4 Based on the multiple initial individual masks obtained through fine segmentation in Example 3, this example further details the method for obtaining precise individual masks, aiming to solve the problem that the individual surface may still be obscured after fine segmentation, leading to missed defect detection. As one implementation method, the acquisition of precise individual masks includes: fitting a complete three-dimensional geometric model of the initial individual mask based on a depth image to obtain an unobstructed projected area; calculating the pixel ratio of the initial individual mask to the unobstructed projected area to obtain overall visibility; extracting agricultural product surface features within the corresponding area of ​​the initial individual mask based on the depth image and texture image; comparing the agricultural product surface features with preset key area features to obtain key area visibility; key area features are the features of the corresponding areas in multiple areas of the agricultural product where the probability of defect occurrence is higher than a preset probability threshold; performing a weighted fusion evaluation of the overall visibility and key area visibility to obtain an occlusion degree score; if the occlusion degree score is less than a preset occlusion score threshold, the corresponding initial individual mask is marked as a reflow individual, and a reflow control command is generated; if the occlusion degree score is greater than or equal to the preset occlusion score threshold, the corresponding initial individual mask is marked as a precise individual mask.

[0046] Specifically, the above steps construct a dual occlusion assessment mechanism that integrates geometric integrity and biological morphological priors. First, a complete 3D geometric model is fitted to the initial individual mask based on the depth image. The physical significance of this step lies in inferring the fruit's expected complete morphology from the currently incomplete observation data. For example, for spherical agricultural products such as apples and citrus fruits, a spherical or ellipsoidal model can be used to perform least-squares fitting of the depth point cloud within the mask, thereby reconstructing the 3D surface of the fruit in an ideal unoccluded state. Projecting this fitted model along the imaging optical axis onto the 2D image plane yields the unoccluded projected area. Subsequently, the ratio of the actual pixel area of ​​the initial individual mask to this unoccluded projected area is calculated as the overall visibility. Overall visibility reflects the macroscopic integrity of the fruit surface. If this value is too low (e.g., below 50%), it indicates that the fruit is severely occluded or that the segmentation result itself has a large deviation. Such samples usually do not have direct value for quality assessment and can be used as a first-level rapid screening indicator.

[0047] However, relying solely on overall visibility often fails to meet the stringent requirements for defect detection in agricultural product sorting. In actual production, it is common for most of the fruit surface to be visible, but precisely the critical areas where defects are most prevalent are obscured. Therefore, this application introduces a key area visibility assessment based on prior knowledge of biomorphology, which is the core feature distinguishing this embodiment from general computer vision technology. Specifically, the distribution of defects on the surface of agricultural products is not uniform and random, but follows specific biological and mechanical contact laws. For example, biological defects such as rot and insect infestation are more common in the pedicel and calyx areas because these natural depressions are prone to water and dust accumulation and have thinner epidermis; while mechanical defects such as bruises and abrasions are highly concentrated in the equatorial ring region because this area is where the fruit most frequently contacts the conveyor belt, guide vanes, and adjacent fruits, and experiences the greatest stress. Therefore, the preset key area features cover specific anatomical areas where the probability of these defects occurring is higher than a preset probability threshold.

[0048] When calculating the visibility of key areas, the system first locates the theoretical projection position of each key area from the current viewpoint based on the fitted 3D geometric model. For example, the fruit stalk cavity area can be defined as a circular area with a preset radius centered on the extreme point of curvature at the top of the model; the equatorial ring area can be defined as an annular strip extending to both sides with a preset width, centered on the maximum diameter of the model. Next, the ratio of the actual visible area of ​​each key area within the initial individual mask to its theoretical complete area is calculated. More importantly, this embodiment assigns differentiated weights to the visibility of each key area. Specifically, the weight of the equatorial ring is usually set higher than that of the fruit stalk cavity and calyx cavity. This weight allocation logic is based on two considerations: first, the equatorial ring is the primary visual focus for consumers to observe the appearance of the fruit, and its appearance quality directly affects the commodity value; second, on high-speed automated sorting lines, the equatorial ring bears the vast majority of mechanical impacts and is a major area of ​​hidden damage. By weighted summation of the visibility of key areas, the system can quantitatively assess risk samples that "appear complete but are actually not visible in key areas," thus effectively making up for the blind spots of the traditional overall area method in terms of refined quality control.

[0049] After obtaining overall visibility and key area visibility, the system performs a weighted fusion evaluation to obtain the final occlusion score. The fusion strategy can be flexibly configured according to the fault tolerance requirements of the actual production line. For example, a weighted average method or a conservative strategy that takes the smaller of the two values ​​can be used to ensure that severe occlusion in any dimension can be accurately captured. Subsequently, the occlusion score is compared with a preset occlusion score threshold. If the score is greater than or equal to the threshold, it indicates that the surface information of the individual is sufficient and the key area is well exposed. It is marked as a precise individual mask and sent to the subsequent quality level evaluation stage. Conversely, if the score is less than the threshold, it is determined that although the individual has been successfully segmented, it does not meet the reliable evaluation conditions due to severe occlusion. At this time, the system marks it as a reflow individual and generates a corresponding reflow control instruction.

[0050] It should be noted that the execution of the recirculation control command constitutes a crucial link in the "detection-evaluation-feedback" closed loop of this application's technical solution. Specifically, when the sorting execution mechanism receives the recirculation control command, it drives a specific pneumatic nozzle or mechanical lever to precisely remove the recirculated individual from the main sorting channel and blow it into or guide it into a dedicated recirculation channel. The end of the recirculation channel connects to the pre-buffer bin or elevator inlet at the feeding end, allowing the individual to re-enter the sorting cycle. During the re-feeding and conveying process, the fruit will naturally tumble and adjust its posture, which greatly increases the probability that the previously obscured surface will be exposed when it passes through the imaging station again, thus being recaptured by the system and detected. This mechanism avoids the waste of good products or the missed detection of defects due to poor single imaging angles, significantly improving the overall sorting accuracy and material utilization rate of the entire line. In contrast, without this recirculation mechanism, relying solely on algorithmic interpolation or guesswork to fill in the obscured information not only fails to guarantee the authenticity of defect detection but may also introduce false features leading to misjudgments. Therefore, this embodiment fundamentally solves the problem of individual-level quality control of agricultural products under complex stacking conditions by incorporating biomorphological priors into occlusion assessment and combining it with physical backflow control.

[0051] Example 5 Building upon Example 1, this example further details the specific architecture of the pre-trained instance segmentation model, providing a clear structural framework for the aforementioned method steps. As one implementation, the instance segmentation model includes a dual-branch feature extraction backbone, a cross-modal feature fusion module, and an instance segmentation output head. This architecture design aims to support the collaborative processing of multimodal information at the hardware data structure level, preventing the model from being understood merely as an abstract functional limitation, and ensuring that the technical solution has a definite physical implementation foundation.

[0052] Specifically, the dual-branch feature extraction backbone includes a texture feature extraction branch and a depth feature extraction branch, used to extract texture feature maps and depth geometric feature maps from texture images and depth images, respectively. The texture feature extraction branch is typically constructed using convolutional neural networks (such as ResNet or lightweight MobileNet), and its core function is to capture information on the color distribution, texture details, and appearance defects of agricultural products from two-dimensional color images. This information is crucial for identifying quality indicators such as surface blemishes and chromaticity. The depth feature extraction branch can be constructed using convolutional neural networks or point cloud processing networks like PointNet, focusing on extracting surface geometry, curvature changes, and height distribution information from three-dimensional depth data. This type of information is insensitive to changes in illumination and can effectively characterize the spatial topological relationships formed by fruit stacking. The two branches are designed to operate in parallel and independently, but the feature maps output at each layer must maintain a spatial resolution correspondence so that subsequent modules can perform accurate pixel-level alignment and fusion. This dual-branch parallel extraction mechanism allows the model to simultaneously obtain two complementary representations of appearance semantics and geometric structure, overcoming the perceptual blind spots of a single modality under complex conditions.

[0053] As one implementation method, the cross-modal feature fusion module is used to fuse texture feature maps and depth geometric feature maps at multiple scales to obtain a multi-scale fused feature map. The feature fusion methods include channel-dimensional concatenation, element-wise addition, or attention-based weighted feature fusion. The multi-scale fusion strategy ensures that the model can utilize high-resolution features to preserve fine edge details while also acquiring global semantic context using low-resolution features. This is particularly crucial for handling agricultural products of different sizes and varying degrees of stacking. Each of the three specific fusion methods has its own implementation and applicable scenarios. Channel-dimensional concatenation directly connects texture and depth feature maps at the same scale along the channel dimension, then connects them to a convolutional layer for dimensionality reduction and feature integration. This method preserves the original modal information to the greatest extent and is suitable for scenarios with extremely high detail requirements and sufficient computational resources. Element-wise addition directly superimposes the values ​​at corresponding positions of the two feature maps and can be followed by an activation function for non-linear transformation. This method has few parameters and is computationally efficient, making it suitable for embedded or real-time sorting devices. Feature-weighted fusion based on attention mechanisms is more intelligent. By introducing channel attention or spatial attention modules, it automatically learns the degree of dependence on the two modalities at different spatial locations and feature channels, dynamically adjusting the fusion weights. For example, in flat areas on the fruit surface, the model may rely more on texture features; while at V-shaped grooves at stacked boundaries, the model automatically enhances the weight of depth features. This adaptive fusion mechanism significantly improves the robustness of the model in complex, adhesive scenarios, making it the preferred solution for handling unstructured agricultural product sorting environments in this application.

[0054] In one implementation, the instance segmentation output head is used to output pixel-by-pixel instance-aware feature maps based on a multi-scale fused feature map through convolutional layers, and then perform pixel-by-pixel classification or mask decoding on the instance-aware feature maps to obtain an initial instance mask. Specifically, the output head takes the highest-resolution fused feature map as input and generates an instance-aware feature map through at least one convolutional layer. Each pixel vector in this feature map encodes the embedding representation of the instance to which that location belongs or the probability distribution of belonging to each candidate instance. Subsequently, through pixel-by-pixel classification or mask decoding operations, pixels with similar embedding encodings or high probability of belonging are aggregated into independent instance sets, i.e., the initial instance mask is output.

[0055] Example 6 After obtaining a verified precise individual mask through the foregoing embodiments, this embodiment further details the specific implementation of quality grade assessment and sorting control instruction generation. As one implementation method, quality grade assessment of the precise individual mask includes: performing color analysis and surface defect detection on the corresponding area of ​​the precise individual mask based on a texture image to obtain a texture assessment result; performing size measurement and morphological assessment on the corresponding area of ​​the precise individual mask based on a depth image to obtain a geometric assessment result; determining the agricultural product quality grade corresponding to the precise individual mask according to preset quality grade determination rules based on the texture and geometric assessment results; and generating corresponding sorting control instructions based on the agricultural product quality grade.

[0056] Specifically, this step requires that the evaluation object be a "precise individual mask." This means that each target entering the quality evaluation stage has undergone multiple verifications, including cross-modal coarse segmentation, stacking detection, fine segmentation correction, and occlusion degree screening, eliminating interference from adhering bodies, severely occluded bodies, and false targets, thus ensuring the purity and reliability of subsequent feature extraction. In terms of texture evaluation, the system uses high-resolution texture images to perform refined analysis of the mask-covered area. Color analysis not only calculates the average RGB or HSV values ​​but also further quantifies the proportion of colored area, color uniformity index, and saturation distribution of specific color gamuts to evaluate the fruit's ripeness and appearance attractiveness. Surface defect detection uses a pre-trained defect recognition model or traditional image processing algorithms to accurately locate and classify defects such as rotten spots, insect holes, rust spots, cracks, and dents, while recording the type, quantity, area, and location information of the defects. Since the mask boundary precisely matches the fruit contour at this point, it effectively avoids false triggering of defect detection by background noise or adjacent fruit textures, significantly improving the confidence of the texture evaluation results.

[0057] In terms of geometric assessment, the system fully utilizes the three-dimensional spatial information contained in depth images, overcoming the limitations of two-dimensional images which are susceptible to changes in lighting and perspective distortion. Size measurement can estimate the maximum transverse diameter, longitudinal diameter, projected area, and even volume of the fruit based on depth point data, providing an objective basis for specification grading. Morphological assessment calculates parameters such as roundness, symmetry, and fruit shape index by fitting a geometric model, used to identify deformed fruits or underdeveloped individuals. For example, in apple sorting, the curved surface morphology of the fruit top and shoulder can be reconstructed using depth data to determine whether flattening or skewness exists. This geometric assessment based on real three-dimensional data, compared to estimation methods relying solely on two-dimensional projected area, can more accurately reflect the actual commercial specifications and morphological quality of agricultural products.

[0058] After obtaining the texture and geometric evaluation results, the system makes a comprehensive decision according to the preset quality grade judgment rules. These rules can take various forms to adapt to different sorting criteria. For example, a multi-dimensional lookup table method can be used, pre-establishing a mapping relationship table between color grade, defect grade, size grade, and final quality grade, and determining the result through retrieval and matching; a weighted scoring method can also be used, assigning weight coefficients to each indicator (e.g., defect weight is higher than color weight), calculating the comprehensive score, and then dividing the grade range; a decision tree or rule engine can also be used to simulate the logical judgment process of human sorting experts, prioritizing the elimination of individuals with fatal defects, and then finely grading qualified products. Regardless of the specific rules used, the core is to achieve simultaneous consideration of appearance quality and internal specifications, avoiding misjudgments caused by single-dimensional evaluation, such as misjudging large fruits with hidden defects as superior grade, or mistakenly rejecting fruits with slightly inferior appearance but meeting specifications and without substantial damage.

[0059] Finally, based on the determined quality grade of the agricultural product, the system generates corresponding sorting control instructions. These instructions typically include information such as the target channel number, actuator type, and action sequence, and are transmitted in real time to the back-end sorting execution unit via an industrial bus. The execution unit then drives the corresponding air valve nozzles, mechanical levers, or flexible grippers to precisely divert the agricultural product to the corresponding collection trough or rejection port. Through this multimodal quality grade assessment mechanism, this application completes a full technical closed loop from perception, segmentation, verification to decision execution, ensuring that every agricultural product flowing out of the high-speed sorting line undergoes comprehensive, objective, and precise quality control.

[0060] Example 7 like Figure 4As shown, this embodiment provides a machine vision-based agricultural product sorting device 600, applied to the machine vision-based agricultural product sorting method described in the preceding embodiments. This device integrates multimodal perception, intelligent segmentation algorithms, and sorting control logic into a unified hardware system through a modular architecture, providing a deployable technical carrier for high-speed agricultural product sorting lines. Specifically, the device includes an image acquisition module 601, a coarse segmentation module 602, a stacking detection module 603, a fine segmentation module 604, and a sorting instruction generation module 605. The modules interact with each other in real time via a high-speed data bus or shared memory mechanism.

[0061] Image acquisition module 601 is used to acquire texture and depth images of the target agricultural product. Specifically, the image acquisition module typically includes a high-resolution color industrial camera and a depth sensor (such as a line laser profilometer or structured light camera), both fixed above the conveyor belt by a precision calibration bracket, maintaining parallel optical axes or a known angle between them. In actual deployment, this module is also equipped with a synchronous triggering unit to simultaneously excite both sensors to acquire data when the agricultural product passes through the inspection station, and to ensure pixel-level spatial registration accuracy between the texture and depth images through hardware-level timestamp alignment. This dual-modal synchronous acquisition mechanism is the data entry point for the entire device, and the acquired registered image pairs directly determine the reliability of subsequent cross-modal fusion processing, effectively overcoming the perception blind spots of a single sensor under complex operating conditions.

[0062] The coarse segmentation module 602 is used to input texture images and depth images into a pre-trained instance segmentation model for cross-modal feature fusion and coarse segmentation processing, outputting an initial instance mask. Specifically, the coarse segmentation module is equipped with a high-performance computing unit (such as a GPU or embedded AI acceleration chip), on which the dual-branch feature extraction backbone and cross-modal feature fusion module as described in Example 5 are loaded and run. This module receives raw image data from the image acquisition module and outputs the initial instance mask through network forward inference. As the primary screening unit of the device, the coarse segmentation module utilizes the generalization ability of the deep learning model to quickly separate candidate targets from the background, defining the region of interest for subsequent fine processing, and significantly reducing the computational load of full-image processing.

[0063] The stacking detection module 603 is used to calculate the distribution mutation features within the corresponding region of the initial instance mask based on the depth image, and to determine whether the distribution mutation features meet the preset mutation conditions. Specifically, the stacking detection module is configured to execute algorithm logic such as gradient vector field construction, orientation histogram statistics, and dynamic spacing interval verification as described in Embodiment 2. This module communicates with the coarse segmentation module and performs depth geometry analysis only on the output initial instance mask region, rather than traversing the entire depth image. This on-demand triggering design allows the device to complete the stacking determination within milliseconds, ensuring both accurate capture of real adhered targets and avoiding false responses to natural depressions in single fruits. The stacking detection module essentially acts as an adaptive routing switch within the device; its determination result directly determines whether the data stream enters the time-consuming fine segmentation channel or passes directly to the quality evaluation channel, thereby achieving dynamic optimization allocation of computing resources.

[0064] The fine segmentation module 604 is used to perform fine segmentation processing on the initial instance mask based on the depth image if the distribution mutation features meet the mutation conditions, thereby obtaining multiple precise individual masks; if the distribution mutation features do not meet the mutation conditions, the initial instance mask is marked as a precise individual mask. Specifically, the fine segmentation module integrates watershed segmentation, texture-depth cascade correction, and occlusion degree evaluation and reflow control logic as described in Embodiments 3 and 4. When the "mutation condition met" signal is received from the stacking detection module, the module activates the depth gradient magnitude calculation and boundary correction subroutine to break the aggregate into independent individuals and perform visibility scoring on each individual; if the score is lower than the threshold, a reflow control command is further generated. Conversely, if the "mutation condition not met" signal is received, the module directly passes through and marks the initial instance mask as a precise individual mask. This conditional branching processing mechanism enables the device to have flexible processing capabilities to cope with different stacking degrees, ensuring that the final output precise individual mask meets the quality evaluation admission standards in terms of geometric integrity and surface visibility.

[0065] The sorting instruction generation module 605 is used to evaluate the quality level of precise individual masks and generate corresponding sorting control instructions. Specifically, the sorting instruction generation module is equipped with a quality evaluation algorithm unit and an IO control interface. This module receives precise individual masks from the fine segmentation module, combines the original texture image and depth image to perform color analysis, defect detection, size measurement, and morphological evaluation as described in Example 6, and determines the quality level according to preset level rules. Subsequently, the module maps the quality level to specific sorting control instructions (such as channel number, valve opening sequence, etc.) and sends them to the back-end actuator in real time via industrial Ethernet or fieldbus. Through the collaborative work of the above modules, the device provided in this application realizes a complete hardware closed loop from multi-dimensional perception and adaptive segmentation to precise decision execution, providing reliable technical support for the intelligent upgrading of post-harvest processing of agricultural products.

[0066] Example 8 To more clearly illustrate the application value and technical effect of the technical solution of this application in a real industrial production environment, this embodiment provides a specific application scenario verification case based on a high-speed agricultural product sorting line.

[0067] In this application scenario, the sorting target is set to Fuji apples, and the sorting line conveyor belt speed is set to 2.0 m / s to simulate high-speed sorting conditions during peak production season. The image acquisition module is equipped with a 4096×3072 resolution linear color camera and a 5kHz sampling frequency 3D line laser profile sensor, both mounted 1.2 meters above the conveyor belt. Hardware triggers enable microsecond-level synchronous acquisition and pixel-level spatial registration. Under these conditions, the apples flow densely through the detection area, with frequent instances of close contact and stacking, placing extremely high demands on segmentation accuracy and defect detection rate.

[0068] Specifically, when two Fuji apples are laterally stacked and pass through the imaging station on a conveyor belt, the system first performs cross-modal coarse segmentation processing as described in Examples 1 and 5. The texture image captures the red coloration and gloss information of the fruit surface, while the depth image records the V-shaped groove geometry formed by the stacking. The pre-trained instance segmentation model extracts features through a dual-branch backbone and fuses them through an attention mechanism, outputting an initial instance mask containing the two adhered fruits. At this point, the mask appears as an elliptical connected region with a major axis of approximately 160 mm on a two-dimensional projection. If used directly for quality assessment, the system will be unable to distinguish the number of individuals, and surface defects in the masked area will be completely lost.

[0069] Subsequently, the stacking detection module intervenes. The system calculates the gradient vector field within the initial instance mask region based on the depth image and performs orientation histogram statistics using a 15×15 pixel sliding window. In the actual data, the gradient direction span of the stacking contact area reaches 210°, and the pixel count in all four quadrants exceeds the preset threshold, satisfying the stacking distribution conditions and being marked as a stacking region. Furthermore, the system detects that the center-to-center distance between the two stacking regions is 95mm. According to the dynamic spacing interval construction method described in Example 3, the equivalent diameter of the initial instance mask is approximately 142mm. Taking the upper limit coefficient k=0.75, the upper limit of the spacing is calculated to be 106.5mm; at the same time, the sum of the discrete diameters calculated based on the high amplitude pixel set is 48mm, constituting the lower limit of the spacing. Since the measured spacing of 95mm falls within the effective interval of [48mm, 106.5mm], the system accurately determines that the initial instance mask has real stacking, thereby triggering the fine segmentation process. This process effectively eliminates false triggers caused by single fruit pedicel depressions (usually with a spacing of less than 30 mm or a directional span of less than 120°), proving the accuracy of the distribution mutation feature detection under actual working conditions.

[0070] Upon entering the fine segmentation stage, the system first executes a watershed algorithm based on the depth gradient magnitude map to obtain an initial segmentation boundary roughly following the direction of the stacked grooves. However, limited by the point cloud density of the laser sensor, this boundary exhibits an aliasing error of approximately 3 to 5 pixels. At this point, the texture-depth cascade correction mechanism comes into play. The system detects a clear texture edge within the neighborhood of the initial boundary, with a positional deviation of only 2 pixels. After extracting the depth profile along the normal direction of this edge, it is found that the median difference in depth values ​​on both sides is 8.5 mm, far exceeding the preset 2 mm jump threshold, confirming it as a real physical contact seam. Therefore, the system replaces the original rough boundary with this texture edge, improving the segmentation accuracy to the sub-millimeter level. In contrast, in the same batch of tests, a dried water stain on the surface of a fruit runs through the vicinity of the stacked area. Although it produced a strong edge response on the texture image and was located close to the initial boundary, it failed the depth jump verification and was automatically filtered by the system because the corresponding depth profile height difference was only 0.15 mm, thus avoiding over-segmentation errors. This fully verifies the robustness of the "texture proposal, depth adjudication" strategy in Example 3 under complex surface noise interference.

[0071] After fine segmentation, the system obtained two independent initial individual masks. For one severely occluded individual, the occlusion assessment module calculated its overall visibility to be 62%, which is higher than the 50% rapid screening threshold. However, based on prior biomorphological assessment of key regions, the visible proportion of the equatorial ring region of this fruit was only 35%. Considering that the equatorial ring is a high-incidence area for bruising and has a high weight, the weighted occlusion score after fusion was 0.58, lower than the preset occlusion score threshold of 0.65. Accordingly, the system marked this individual as a return individual and generated a sorting control command to drive the air valve nozzle to precisely blow it into the return channel and send it back to the feeding end. At the same time, another individual with sufficient surface exposure had an occlusion score of 0.92, was marked as a precise individual mask, and directly entered the quality grade assessment stage. Ultimately, due to its uniform coloring, lack of defects, and qualified fruit diameter, it was diverted to the first-grade channel.

[0072] After passing through the elevator and flexible deflector again, the returned fruit underwent natural tumbling and attitude adjustment. When it passed the imaging station a second time, the previously obscured equatorial ring area was fully exposed. The system executed the complete inspection process again, this time improving the occlusion score to 0.95, and successfully detecting a minor impact damage of 3mm in diameter in that area. Based on a comprehensive judgment of the texture and geometric evaluation results, the fruit was classified as a second-grade product and diverted to the corresponding channel. Without this return mechanism, the fruit would most likely have been misjudged as a qualified product and entered the first-grade market due to the invisibility of key parts, or it might have been conservatively rejected during the initial inspection due to acceptable overall visibility but missing key information, resulting in waste.

[0073] As can be seen from the above complete application examples, the technical solution provided in this application can organically link stack detection, fine segmentation, occlusion assessment and physical backflow control into an adaptive closed-loop system under high-speed dynamic conditions.

[0074] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for sorting agricultural products based on machine vision, characterized in that, include: Acquire the texture and depth images of the target agricultural product; The texture image and depth image are input into the pre-trained instance segmentation model for cross-modal feature fusion and coarse segmentation, and the initial instance mask is output. The distribution mutation features within the region corresponding to the initial instance mask are calculated based on the depth image, and it is determined whether the distribution mutation features meet the preset mutation conditions. If the distribution mutation features meet the mutation conditions, then the initial instance mask is finely segmented based on the depth image to obtain multiple precise individual masks; If the distribution mutation characteristics do not meet the mutation conditions, the initial instance mask is marked as the precise individual mask; The quality level of precise individual masks is evaluated, and corresponding sorting control instructions are generated.

2. The method for sorting agricultural products based on machine vision according to claim 1, characterized in that, The determination of whether the distribution mutation characteristics meet the preset mutation conditions includes: The depth gradient vector of each pixel in the region corresponding to the initial instance mask is calculated based on the depth image to obtain the gradient vector field. Histogram statistics of gradient directions in the gradient vector field are performed based on a sliding window of preset size to obtain a gradient direction distribution histogram. The directional distribution characteristics of the corresponding sliding window are calculated based on the gradient directional distribution histogram; the directional distribution characteristics include directional span and quadrant distribution uniformity. If the directional distribution characteristics meet the preset stacking distribution conditions, the corresponding sliding window is marked as a stacking region; Traverse the gradient vector field to count the number of regions in the stacked region. If the number of regions is greater than a preset threshold, calculate the distance between the geometric center points of every two different stacked regions to obtain the region spacing. If there is a spacing value within the preset spacing range in the regional spacing, then the distribution mutation feature is determined to meet the mutation condition.

3. The method for sorting agricultural products based on machine vision according to claim 2, characterized in that, The methods for obtaining the spacing interval include: Calculate the mask's equivalent diameter based on the total number of pixels in the initial instance mask; The spacing upper limit is constructed based on the mask equivalent diameter and a preset upper limit coefficient; the upper limit coefficient is constructed based on the projection constraints of the stacking geometry model of the target agricultural product; Calculate the discrete diameter of the pixels inside any two stacked regions, and mark the sum of the two discrete diameters as the lower limit of the spacing; The interval formed by the upper and lower limits of the spacing is marked as the spacing interval.

4. The method for sorting agricultural products based on machine vision according to claim 3, characterized in that, The method for calculating the discrete diameter includes: Pixels whose gradient magnitude in the stacked region is greater than a preset gradient threshold are selected to obtain a high-magnitude pixel set; the preset gradient threshold is determined based on the average gradient magnitude of the stacked region. Perform connected component analysis on the high-amplitude pixel set and retain the connected component with the largest area to obtain the maximum connected component; The maximum Euclidean distance between all pairs of pixels within the largest connected region is marked as the discrete diameter of the corresponding stacked region.

5. The method for sorting agricultural products based on machine vision according to claim 1, characterized in that, The fine segmentation process of the initial instance mask based on the depth image includes: The depth gradient magnitude of each pixel in the region corresponding to the initial instance mask is calculated based on the depth image to obtain the gradient magnitude map. Based on the gradient magnitude map, the watershed algorithm is used to segment the initial instance mask to obtain the initial segmentation boundary. The initial segmentation boundary is locally corrected based on the edge features of the texture image to obtain the corrected segmentation boundary. Based on the corrected segmentation boundary, the initial instance mask is segmented into multiple initial individual masks. The individual occlusion degree of multiple initial individual masks is evaluated and filtered to obtain multiple precise individual masks.

6. The method for sorting agricultural products based on machine vision according to claim 5, characterized in that, The local correction of the initial segmentation boundary based on the edge features of the texture image includes: Within the local neighborhood of the initial segmentation boundary, detect whether there is a texture edge in the corresponding texture image, and the positional deviation of the texture edge from the initial segmentation boundary is within a preset pixel range; If a texture edge that satisfies the positional deviation condition exists, a depth value profile is extracted on the corresponding depth image along the normal direction of the texture edge, and a depth jump is determined based on the depth value profile; the depth jump refers to the statistical difference of the depth values ​​in the depth value profile being greater than a preset jump threshold. If there is a depth jump, the texture edge replaces the initial segmentation boundary in the corresponding local neighborhood to obtain the corrected segmentation boundary; If there are no texture edges or depth transitions, the initial segmentation boundary within the corresponding local neighborhood is marked as the corrected segmentation boundary.

7. The method for sorting agricultural products based on machine vision according to claim 5, characterized in that, The method for obtaining the precise individual mask includes: The initial individual mask is fitted with a complete three-dimensional geometric model based on the depth image to obtain the unobstructed projected area. Calculate the pixel ratio of the initial individual mask to the unobstructed projected area to obtain the overall visibility; Based on depth and texture images, surface features of agricultural products within the area corresponding to the initial individual mask are extracted; The visibility of key areas is obtained by comparing and calculating the surface features of agricultural products with the preset key area features; the key area features are the features of the corresponding areas in multiple areas of agricultural products where the probability of defects occurring is higher than a preset probability threshold. A weighted fusion assessment of overall visibility and key area visibility is performed to obtain an occlusion score. If the occlusion score is less than the preset occlusion score threshold, the corresponding initial individual mask is marked as a reflow individual, and a reflow control command is generated. If the occlusion score is greater than or equal to the preset occlusion score threshold, the corresponding initial individual mask is marked as the precise individual mask.

8. The method for sorting agricultural products based on machine vision according to claim 1, characterized in that, The instance segmentation model includes a dual-branch feature extraction backbone, a cross-modal feature fusion module, and an instance segmentation output head; The dual-branch feature extraction backbone includes a texture feature extraction branch and a depth feature extraction branch, which are used to extract texture feature maps and depth geometric feature maps from the texture image and the depth image, respectively; The cross-modal feature fusion module is used to fuse texture feature maps and depth geometric feature maps at multiple scales to obtain a multi-scale fused feature map; the feature fusion method is channel dimension concatenation, element-wise addition, or feature weighted fusion based on an attention mechanism. The instance segmentation output head is used to output pixel-by-pixel instance-aware feature maps through convolutional layers based on multi-scale fused feature maps, and to perform pixel-by-pixel classification or mask decoding on the instance-aware feature maps to obtain an initial instance mask.

9. The method for sorting agricultural products based on machine vision according to claim 1, characterized in that, The quality level evaluation of precise individual masks includes: Based on the texture image, color analysis and surface defect detection are performed on the corresponding area of ​​the precise individual mask to obtain the texture evaluation result; Based on depth images, the size and shape of the corresponding region of the precise individual mask are measured and evaluated to obtain geometric evaluation results; Based on the texture evaluation results and geometric evaluation results, the quality grade of agricultural products corresponding to the precise individual masks is determined according to the preset quality grade judgment rules. Based on the quality grade of agricultural products, corresponding sorting control instructions are generated.

10. A machine vision-based agricultural product sorting device, applied to the machine vision-based agricultural product sorting method as described in any one of claims 1 to 9, characterized in that, include: The image acquisition module is used to acquire texture and depth images of the target agricultural product; The coarse segmentation module is used to input texture images and depth images into a pre-trained instance segmentation model for cross-modal feature fusion and coarse segmentation processing, and output an initial instance mask; The stacking detection module is used to calculate the distribution mutation features in the region corresponding to the initial instance mask based on the depth image, and to determine whether the distribution mutation features meet the preset mutation conditions. The fine segmentation module is used to perform fine segmentation on the initial instance mask based on the depth image if the distribution mutation features meet the mutation conditions, thereby obtaining multiple precise individual masks; if the distribution mutation features do not meet the mutation conditions, the initial instance mask is marked as a precise individual mask. The sorting instruction generation module is used to evaluate the quality level of precise individual masks and generate corresponding sorting control instructions.

Images