Machine vision-based wine grape thinning device and method

Abstract

The present application relates to wine grape thinning technical field, specifically, it is a kind of wine grape thinning device and method based on machine vision, method includes the following steps: step S1: multimodal image acquisition and adaptive enhancement preprocessing;Step S2: accurate segmentation of cluster and peduncle based on improved loss function;Step S3: dynamic decision of peduncle clamping point based on mechanical optimal model;Step S4: thinning intensity decision based on density clustering and random forest regression;Step S5: trajectory planning of mechanical arm and mechanical claw collaborative pruning;Step S6: thinning effect verification and closed-loop correction.The present application can preferably carry out wine grape thinning.

Description

Technical Field

[0001] This invention relates to the field of grape thinning technology, and more specifically, to a grape thinning device and method based on machine vision. Background Technology

[0002] Fruit thinning in wine grapes is a key agronomic measure for precisely controlling vine load and improving fruit quality. In major wine grape producing areas, cultivation methods are mostly based on trellises or grid-like structures, resulting in complex canopy structures and densely packed, irregularly distributed bunches. Currently, grape thinning in wine grapes relies primarily on manual experience: workers subjectively decide which bunches to remove and how many to remove based on visual assessment of bunch size, berry density, and bunch spacing. This method has the following prominent problems: First, manual thinning is inefficient and costly. Vineyards are large-scale, with concentrated thinning periods, and there is a shortage of skilled workers. The limited daily workload per person leads to delays in the thinning window, affecting the effectiveness of quality control.

[0003] Second, inconsistent human judgment standards. Different workers have vastly different experiences, and the same bunch of fruit may be subject to drastically different thinning decisions by different workers, resulting in uneven spatial distribution of thinning intensity, which in turn affects the uniformity of fruit ripening.

[0004] Third, there is a lack of intelligent sensing and decision-making methods. Existing research mainly focuses on thinning equipment for large fruits such as apples and citrus. For crops like wine grapes, which have small berries, grow in clusters, and are subject to severe canopy shading, a mature machine vision recognition and intelligent decision-making system has not yet been developed. In particular, under complex canopy environments, issues such as the precise segmentation of bunches and stalks, the quantification of berry cluster density, and the dynamic calculation of the optimal clamping point for stalks remain technical bottlenecks restricting the development of automated thinning equipment.

[0005] Fourth, existing mechanical fruit thinning devices pose a risk of damage. The few vibration or comb-type fruit thinning mechanisms that have been attempted lack the ability to perceive the individual characteristics of the fruit bunches, which can easily cause fruit to fall off, fruit stalks to tear, or the entire fruit bunch to fall off, making it impossible to achieve precise and low-damage thinning operations.

[0006] In recent years, machine vision and deep learning technologies have been widely applied in agriculture, such as apple detection based on Mask R-CNN and grape bunch segmentation based on DeepLab. However, existing research mostly focuses on the recognition level and lacks deep integration with downstream thinning decisions, robotic arm motion planning, and end effector control. In particular, there are still no systematic solutions for mechanical decisions regarding fruit stem clamping points, quantification of thinning intensity based on fruit spatial distribution, and closed-loop verification feedback after thinning.

[0007] Therefore, there is an urgent need for a machine vision-based grape thinning device and method that can automatically and accurately identify grape bunches in complex canopy environments, make intelligent decisions on thinning targets, optimize the mechanical selection of stalk clamping points, and drive mechanical claws to complete low-damage pruning, thereby achieving intelligent and precise grape thinning operations. Summary of the Invention

[0008] The present invention provides a machine vision-based grape thinning device and method for winemaking, which can overcome some or all of the defects of the prior art.

[0009] The machine vision-based grape thinning method for winemaking according to the present invention includes the following steps: Step S1: Multimodal image acquisition and adaptive enhancement preprocessing; By using an RGB-D camera deployed at the end of a robotic arm, synchronous RGB and depth images of the grape canopy are acquired to obtain two-dimensional color information and three-dimensional spatial coordinates of the bunches, stems, and leaf canopy. Adaptive gamma correction is applied to the RGB images for enhancement. The enhanced RGB images are then used to remove backgrounds from the depth images, retaining pixels with depth values ​​within a preset threshold range to generate region of interest images. Step S2: Accurate segmentation of ear of fruit and peduncle based on improved loss function; A Mask R-CNN instance segmentation network was constructed and trained using transfer learning on a labeled dataset. The total loss function included an edge-preserving loss term was adopted. After training converged, the segmentation model was obtained, and the output included ear mask, stalk mask, and 3D point cloud of ear. Step S3: Dynamic decision-making on the fruit stalk clamping point based on the optimal mechanical model; Extract the stalk skeleton line, calculate the diameter of each point and the distance to the connection point with the ear of fruit; construct the cantilever beam mechanical model of the stalk and calculate the bending deflection at the clamping point; calculate the position of the ear of gravity and the equivalent force arm length; define the comprehensive evaluation value of the clamping point, select the point with the largest evaluation value as the best clamping point, and output its three-dimensional coordinates in the robot base coordinate system. Step S4: Decision on the sparseness intensity based on density clustering and random forest regression; Connectivity analysis is performed on the fruit cluster mask to identify individual fruit grains, and DBSCAN clustering is performed on the set of fruit grain center points. The density of each cluster is calculated. A random forest regression model is constructed, with the projected area of ​​the fruit cluster, the total number of fruit grains, the density variance of each cluster, the relative canopy height of the fruit cluster, and the light intensity as input features, and the thinning intensity coefficient is output. When the thinning intensity coefficient is greater than the thinning threshold, the thinning targets are selected in descending order of density. When the cumulative number of thinned fruit grains reaches the target value, a list of thinning targets is generated. Step S5: Robotic arm trajectory planning and robotic gripper collaborative shearing; Based on the list of targets to be removed, the spatial trajectory of the robotic arm joints is planned using fifth-order polynomial interpolation to achieve collision-free motion; the robotic arm is driven to move to the optimal gripping point, and the bionic flexible gripper is activated to clamp the fruit stalk with the calculated gripping force; the rotary shearing mechanism is activated to complete the shearing; the robotic arm is reset, and the above steps are repeated until the entire list of targets to be removed is completed. Step S6: Verification of sludge removal effect and closed-loop correction; After the thinning operation is completed, the canopy image after thinning is acquired again; the remaining fruit clusters are re-identified using the segmentation model in step S2, the actual number of thinned fruit clusters is calculated, and the thinning deviation is defined; if the absolute value of the deviation exceeds the allowable deviation threshold, a correction instruction is executed: if the deviation is positive, the subsequent thinning target is skipped; if the deviation is negative, step S4 is re-executed for the remaining fruit clusters to generate supplementary thinning targets.

[0010] Preferably, the adaptive gamma correction enhancement function in step S1 is: ; in, For pixel coordinates, The enhanced pixel value, The pixel value is the pixel coordinate. The expression for the adaptive gamma value is: ; in, For The average grayscale value of a local window centered at 15×15 pixels. The grayscale mean of the entire image. The standard deviation of grayscale in a local window. To prevent division by zero for very small positive numbers; The depth threshold for deep background removal is determined by an adaptive algorithm: ; in, and These are the mean and standard deviation of the depth image, respectively, retaining the depth values. Pixels within the range.

[0011] Preferably, in step S2, the total loss function described in step S2... Defined as: ; in, For classification cross-entropy loss, Smooth L1 loss for bounding box regression. For masking binary cross-entropy loss, These are the edge loss weighting coefficients. The edge-preserving loss is expressed as: ; in, The total number of samples, For the first The gradient of the mask is predicted for each sample. To correspond to the gradient of the real mask, It is an L2 norm; The edge loss weighting coefficient Employing a dynamic adjustment strategy: ; in, For training rounds, For the preheating round, These are the initial weights.

[0012] Preferably, in step S3, the skeleton line is refined from the fruit stalk mask using the Zhang-Suen parallel thinning algorithm. Extract from the data and prune branches using the eight-neighbor tracing algorithm, retaining only the main skeleton line connected to the ear mask; The bending deflection function for:: ; in, The weight of the grape bunch is estimated by multiplying the volume of the three-dimensional point cloud of the bunch by a preset empirical value for grape berry density. The equivalent force arm length from the gripping point to the center of gravity of the ear of fruit. The elastic modulus of the fruit stalk, The moment of inertia of the clamping point section, The radius of the fruit stalk at the clamping point; The center of gravity of the ear Centroid calculation based on the 3D point cloud of the ear of fruit: ; in, For point cloud points, For the first The three-dimensional coordinates of each point; The equivalent force arm length The calculation formula is: ; in, For the center of gravity of the ear of fruit, Here are the coordinates of the clamping point. For the Euclidean norm, The angle between the direction of the grain's gravity and the axis of the fruit stalk; The comprehensive evaluation value of the clamping point for: ; in, For the maximum allowable deflection, To achieve the optimal clamping diameter, This represents the minimum distance between the clamping point and the adjacent fruit. For safe distance threshold, The weighting coefficients and Select As the optimal clamping point This is the set of points representing the skeletal framework of the fruit stalk.

[0013] Preferably, the neighborhood radius of the core point in the DBSCAN clustering in step S4 is ,in The average diameter of the fruit. The radius coefficient; The density of the cluster The calculation formula is: ; in, The number of fruits within a cluster. Let be the radius of the equivalent sphere of the cluster; The random forest regression model The input feature vector is: ; in, The projected area of ​​the ear of fruit. The total number of fruits, The density variance of each cluster, The relative height of the fruit bunch to the canopy. The light intensity is the model output; the light intensity coefficient is removed. : ; The density variance The calculation formula is: ; in, The total number of clusters, For the first The density of each cluster, The average density of all clusters; when At that time, according to density Select the thinning targets in descending order, and the cumulative number of thinned fruits meets the following condition: ; in, To remove the index set of the target cluster, To remove the initiation threshold; generate a list of removal targets. .

[0014] Preferably, the trajectory planning function for the fifth-order polynomial interpolation in step S5 is: ; in, For the robotic arm joint angles over time A changing function, The coefficients are undetermined; the boundary conditions are: Solve for the coefficients to Obtain a smooth trajectory, where This is the initial joint angle. For the target joint angle, For exercise time; The clamping force The calculation formula is: ; in, For safety reasons, For the weight of the ear of fruit, The coefficient of friction between the gripper and the fruit stalk.

[0015] Preferably, the formula for calculating the sparse deviation in step S6 is: ; in, The actual number of fruits removed. To eliminate the strength coefficient, This represents the total number of original fruit pieces. To eliminate the deviation rate; if Then execute the correction instruction: if If so, then skip the subsequent evacuation targets; if Then, for the remaining ears of fruit, step S4 is repeated to generate supplementary thinning targets. This is the allowable deviation threshold.

[0016] This invention provides a machine vision-based grape thinning device for winemaking, which employs the aforementioned machine vision-based grape thinning method for winemaking and includes: A tracked mobile chassis is used for autonomous navigation and movement between grape rows. The front end of the tracked mobile chassis is equipped with a lidar and an inertial measurement unit to construct an environmental map between grape rows and achieve autonomous navigation. A six-degree-of-freedom collaborative robotic arm is mounted on a tracked mobile chassis; An end effector, installed at the end of a robotic arm; The embedded industrial computer is electrically connected to the tracked mobile chassis, the six-degree-of-freedom collaborative robotic arm, and the end effector, respectively. The remote monitoring terminal communicates with the embedded industrial control computer via a wireless network to display the operation status in real time and receive manual intervention commands.

[0017] Preferably, the end effector includes: The visual acquisition module includes an RGB-D camera, a ring-shaped LED fill light, and an optical protective cover; The biomimetic flexible gripping module includes two gripping fingers driven by stepper motors, with silicone pads and pressure sensors on the inner side of the gripping fingers; The rotary shearing module includes a brushless DC motor and a ring-shaped rotary blade; The embedded industrial control computer is internally equipped with: The image processing unit is equipped with a GPU acceleration module for performing image acquisition and segmentation; The decision analysis unit is used to perform clamp point calculations and target removal decisions. The motion control unit is used to perform trajectory planning and servo control.

[0018] Preferably, the coefficient of friction of the silicone pad is... satisfy: ; in, This is the base value for the static friction coefficient. For pressure-sensitive increments, This is the actual clamping force. For reference pressure.

[0019] The beneficial effects of this invention are as follows: This invention constructs an RGB-D visual acquisition system to simultaneously acquire visible light and depth images, and employs an adaptive gamma correction enhancement algorithm to effectively overcome interference from large variations in field lighting and complex leaf canopy backgrounds. Based on this, a Mask R-CNN instance segmentation network is built, and an edge-preserving loss term (calculating the gradient difference between the predicted mask and the real mask) is introduced into the total loss function, significantly improving the segmentation accuracy of the ear outline and stalk boundary. Compared to conventional segmentation methods, this invention improves the recognition accuracy of the slender stalk by approximately 15%–20%, providing reliable basic data for subsequent gripping point decisions.

[0020] This invention incorporates a cantilever beam mechanics model into the selection of fruit stem clamping points. By calculating the bending deflection at the clamping point and combining it with the fruit stem diameter and the safe distance from the fruit, a comprehensive evaluation value is constructed. This evaluation function simultaneously considers mechanical stability (minimum deflection), geometric fit (diameter close to the optimal value of 3mm), and obstacle avoidance safety (away from the fruit), and can automatically select the most suitable clamping point. Compared to manual clamping based on experience or clamping at fixed positions, this invention can reduce the risk of fruit stem slippage by approximately 40% and reduce the rate of accidental fruit damage by more than 50%.

[0021] This invention employs the DBSCAN clustering algorithm to divide fruit berries into clusters and calculate the density of each cluster, solving the problem of quantifying the density of berries within the fruit bunch. Simultaneously, a random forest regression model is constructed, taking the projected area of ​​the fruit bunch, the total number of berries, the density variance, the relative height, and the light intensity as inputs, and outputting a thinning intensity coefficient. This model can learn the nonlinear mapping relationship from manual fruit thinning experience, enabling personalized thinning decisions for each fruit bunch. Compared to traditional manual visual inspection or fixed-ratio thinning, this invention can prioritize the thinning of inner, overly dense, or poorly developed berry clusters based on the compactness of the berry spatial distribution, resulting in a more uniform spatial distribution of fruit after thinning, which is beneficial for subsequent photosynthesis and quality improvement.

[0022] This invention, after the robotic arm removes fruit, calculates the deviation between the actual number of fruit removed and the target value by re-acquiring images. If the deviation exceeds 5%, it automatically triggers a supplementary thinning or skipping command, forming a closed-loop feedback control. This mechanism effectively compensates for potential omissions or over-thinning in single-shot identification and execution, significantly improving the reliability of the operation. Experiments show that the thinning accuracy after closed-loop correction can reach within ±3%, far exceeding the ±15% of manual operation.

[0023] The biomimetic flexible gripper of this invention adaptively adjusts its clamping force based on the weight of the fruit bunch and the coefficient of friction. The coefficient of friction of the silicone pad increases tangentially with the clamping force, thus maintaining stability while avoiding damage to the fruit stalks during clamping. The rotary shearing cuts in from the lower side, resulting in high cutting efficiency and minimizing overall movement of the fruit bunch. Testing shows that a single thinning action takes approximately 2-3 seconds, leaving a clean cut on the fruit stalk without tearing. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a machine vision-based grape thinning device in an embodiment. Figure 2 This is a flowchart of a machine vision-based grape thinning method for winemaking, as described in this embodiment. Detailed Implementation

[0025] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention. Example

[0026] This embodiment focuses on Cabernet Sauvignon wine grapes typically grown under trellises in the Helan Mountain East Foothills region of Ningxia, and describes the thinning operation performed during the fruit enlargement period.

[0027] like Figure 1 As shown, this embodiment provides a machine vision-based grape thinning device for winemaking, including: The tracked mobile chassis is wide enough to fit the row spacing of grapes. A 16-line LiDAR and a nine-axis inertial measurement unit are installed at the front of the chassis for row navigation.

[0028] The six-DOF collaborative robotic arm (model: UR10e) has an end-effector load capacity of 5kg and a repeatability of ±0.1mm. Each joint of the six-DOF collaborative robotic arm integrates a torque sensor to detect abnormal collisions during the clearing process and trigger an emergency stop when an abnormal external force exceeding 5N is detected.

[0029] End effector: Integrates an Intel RealSense D435i RGB-D camera (depth range 0.2~3m, RGB resolution 1920×1080), a ring LED fill light (color temperature 5000K), an optical protective cover, a bionic flexible gripper (bionic flexible gripping module), and a rotary shearing mechanism (rotary shearing module). The bionic flexible gripper includes two fingers driven by stepper motors, with silicone pads (bionic bumps on the surface, 1mm in diameter, 2mm spacing) attached to the inner side of the fingers. A miniature pressure sensor (range 0~10N, accuracy ±0.05N) is installed at the base of the fingers. The shearing mechanism uses a brushless DC motor to drive a ring-shaped rotating blade, with an adjustable blade speed (30~60 rpm). The coefficient of friction of the silicone pads on the inner wall of the bionic flexible gripper is... satisfy: ; in, This is the base value for the static friction coefficient. For pressure-sensitive increments, This is the actual clamping force. For reference pressure.

[0030] The silicone pad has a biomimetic bump structure on its surface, and the end of the gripper finger is integrated with a miniature pressure sensor for real-time feedback of the gripping force. The rotary shearing mechanism is driven by a brushless DC motor, and the blade is a ring-shaped rotating blade. The blade's rotation plane is at an angle of 30° to 45° with the gripper's holding plane, ensuring that the blade cuts into the fruit stem from below during shearing.

[0031] The embedded industrial PC (NVIDIA Jetson AGX Xavier, 32GB RAM, 512-core Volta GPU) comes pre-installed with software environments including Ubuntu 18.04, ROSMelodic, and PyTorch 1.8. Internally, the embedded industrial PC deploys: The image processing unit is equipped with a GPU acceleration module for performing image acquisition and segmentation; The decision analysis unit is used to perform clamp point calculations and target removal decisions. The motion control unit is used to perform trajectory planning and servo control.

[0032] The remote monitoring terminal (tablet computer) communicates with the industrial control computer via 5G / WiFi.

[0033] like Figure 2 As shown, this embodiment provides a machine vision-based method for thinning wine grapes, which includes the following steps: Step S1: Multimodal image acquisition and adaptive enhancement preprocessing; The robotic arm, carrying an end effector, moves to a position approximately 0.5 meters in front of the grape canopy, where an RGB-D camera simultaneously acquires RGB images. and depth images The ring-shaped LED fill light automatically turns on to ensure uniform illumination.

[0034] Adaptive gamma correction enhancement is applied to the RGB image. The average grayscale value of the entire image is calculated. For each pixel Take a 15×15 pixel local window centered on it, and calculate the average gray value within the window. and standard deviation .make To prevent division by zero for extremely small positive numbers, an adaptive gamma value is calculated. : ; Then perform a gamma transformation pixel by pixel: ; in, For pixel coordinates, The enhanced pixel value, The pixel value is the pixel coordinate. This operation enhances local contrast, making the boundary between the ear of fruit and the canopy of leaves clearer. Background removal is then performed. Depth image calculation is then performed. mean and standard deviation Set depth threshold , Preserve depth values ​​in The pixels within are mapped to the enhanced RGB image to generate the region of interest image. It effectively removed the distant foliage and ground background.

[0035] Step S2: Accurate segmentation of ear of fruit and peduncle based on improved loss function; A pre-built labeled dataset was constructed: 2000 images of wine grapes under different lighting conditions, varieties, and growth stages were collected, and the LabelMe tool was used to perform pixel-level annotations on the grape bunch outline, pedicel center line, and berry distribution.

[0036] Construct a Mask R-CNN instance segmentation network and perform transfer learning on top of ImageNet pre-trained weights. Total loss function. for: ; in For Mask R-CNN standard loss, These are the edge loss weighting coefficients. Edge preservation loss. To calculate the predicted mask gradient Compared with the real mask gradient The mean L2 distance is expressed as: Dynamic adjustment of edge loss weights: In the early stages of training, edge learning is strengthened, and then gradually weakened in the later stages.

[0037] Training parameters: batch size=4, initial learning rate 0.001, momentum 0.9, training for 50 epochs. After model convergence, real-time data is collected... Perform reasoning and output the ear mask. Fruit stalk mask 3D point cloud of fruit ears .

[0038] Step S3: Dynamic decision-making on the fruit stalk clamping point based on the optimal mechanical model; From the fruit stalk cover The skeleton lines were extracted. The Zhang-Suen parallel thinning algorithm was used to iteratively remove edge pixels, obtaining skeleton lines of single-pixel width. Then, the eight-neighbor tracing algorithm was used to prune branches, retaining only the main skeleton lines connected to the ear mask. Calculate each skeleton point. diameter (Scan the mask width perpendicular to the skeleton direction). Starting from the ear connection point, calculate the distance to each point along the skeleton line. Estimate the ear weight. : Three-dimensional point cloud of ear of fruit The volume is calculated by projecting the point cloud onto a two-dimensional plane and integrating by depth. This volume is then multiplied by the empirical value for grape berry density (1.05 g / cm³) to obtain the mass, which is then multiplied by the acceleration due to gravity. (Center of gravity of the bunch) For the centroid of the point cloud: ; in, For point cloud points, For the first The three-dimensional coordinates of each point; For candidate grip points (Any point on the skeleton line), calculate its distance from the center of gravity. The distance and the angle with the axis of the fruit stalk To obtain an equivalent arm Moment of inertia of cross section fruit stalk elastic modulus Take the measured value of 200 MPa. Bending deflection: ; Simultaneously measure the minimum distance between the clamping point and the adjacent fruit. Set the maximum allowable deflection. Optimal clamping diameter Safe distance threshold Weight Calculate the comprehensive evaluation value. : ; Traverse all points of the skeleton line and select As the optimal clamping point This is the set of points representing the fruit stalk skeleton line. Its coordinates are transformed to the robot arm's base coordinate system using a hand-eye calibration matrix.

[0039] Step S4: Decision on the sparseness intensity based on density clustering and random forest regression; Masking of the ears Perform connected component analysis to identify individual fruit grains and obtain the set of fruit grain center points. Calculate the average diameter of the fruit. (Approximately 10mm). Set DBSCAN clustering parameters: neighborhood radius. radius coefficient The minimum number of points, MinPts, is 3. After clustering, multiple clusters are obtained, and the equivalent sphere radius is calculated for each cluster. (Maximum distance from a point within the cluster to the centroid) and the number of berries Cluster density .

[0040] Calculate overall characteristics: projected area of ​​the ear of fruit (Mask pixel area × pixel equivalent), total number of pixels The variance of the density of each cluster relative height of the ear of grain to the canopy (Distance from the center of the ear of fruit to the ground divided by the total height of the canopy), light intensity (Irradiance estimated from depth images).

[0041] eigenvectors Input a pre-trained random forest regression model (trained from 2000 sets of manually thinned fruit data, 100 trees, maximum depth 10), where, The projected area of ​​the ear of fruit. The total number of fruits, The density variance of each cluster, The relative height of the fruit bunch to the canopy. Light intensity; output sparse intensity coefficient .like (Remove the activation threshold), then proceed by density. Select clusters from high to low density and accumulate their fruit counts. Until Mark the ears of fruit corresponding to these clusters as thinning targets and generate a list. .

[0042] Step S5: Robotic arm trajectory planning and robotic gripper collaborative shearing; For each evacuation target, a collision-free trajectory for the robotic arm from its current position to the gripping point is planned using quintic polynomial interpolation. The trajectory planning function for quintic polynomial interpolation is as follows: ; in, For the robotic arm joint angles over time A changing function, The coefficients are undetermined; the boundary conditions are: Solve for the coefficients to Obtain a smooth trajectory, where This is the initial joint angle. For the target joint angle, This refers to the duration of exercise.

[0043] Set exercise time Initial joint angle Target joint angle read by encoder Calculations are performed using inverse kinematics. The coefficients are then solved. Then, joint commands are sent in real time. After the robotic arm reaches the gripping point, the gripping force is calculated: based on the weight of the ear of fruit estimated in step S3. coefficient of friction with silicone pad (Dynamic value), take the safety factor ,have to The stepper motor drives the gripper to clamp the fruit stem with this force, and the pressure sensor provides real-time feedback. The gripper stops when the set value is reached. The brushless DC motor is then started, and the blade speed is adjusted to 45 rpm. The annular blade cuts into the fruit stem from the lower side, taking approximately 0.5 seconds to cut. The force sensor detects a sudden drop in clamping force (the fruit stem breaks off), the motor stops, and the gripper releases. The robotic arm resets to the next target starting point and repeats the above process.

[0044] Step S6: Verification of sludge removal effect and closed-loop correction; After completing all the thinning targets, the robotic arm carrying the camera returns to the starting area of ​​the operation to re-acquire images of the canopy after thinning. The segmentation model from step S2 is used to identify the remaining fruit clusters, and the actual number of thinned fruit is counted. Calculate the slack removal deviation: ; like (Allowable deviation threshold), when When (too sparse), skip certain preset sparse targets in subsequent operations; when If insufficient thinning occurs, repeat step S4 for the undone ears of fruit to generate supplementary thinning targets and then repeat step S5 again. Usually, one correction is enough to control the deviation within 3%.

[0045] In this embodiment, the tracked mobile chassis autonomously navigates between grape rows at a speed of 0.2 m / s, with LiDAR and IMU constructing a local map in real time. The robotic arm dynamically adjusts its base position as the chassis moves, enabling simultaneous thinning and pruning. The vision system pre-identifies 1-2 grape bunches ahead, transmitting their coordinates to the robotic arm planner to synchronize the arm's movement with the chassis, reducing waiting time. A remote monitoring terminal displays the operation video, current thinning targets, cumulative thinning quantity, and deviation statistics. The operator can intervene at any time to modify the thinning intensity or pause the operation.

[0046] This embodiment addresses the technical bottleneck in wine grape cultivation, which relies primarily on manual experience for fruit thinning and lacks intelligent methods. It systematically overcomes core challenges such as precise cluster analysis in complex canopy environments, intelligent decision-making regarding thinning targets, dynamic calculation of fruit stem clamping points, and low-damage pruning execution. The research results can be directly applied to the development of intelligent fruit thinning equipment for wine grapes, promoting the industry's transformation from labor-intensive to technology-intensive. This has significant practical application value and broad market prospects for promoting the intelligent development of the wine grape industry in the Helan Mountain East Foothills region and even nationwide.

[0047] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A machine vision-based method for thinning wine grapes, characterized in that, Includes the following steps: Step S1: Multimodal image acquisition and adaptive enhancement preprocessing; By using an RGB-D camera deployed at the end of a robotic arm, synchronous RGB and depth images of the grape canopy are acquired to obtain two-dimensional color information and three-dimensional spatial coordinates of the bunches, stems, and leaf canopy. Adaptive gamma correction is applied to the RGB image for enhancement; background removal is performed on the enhanced RGB image using the depth image, retaining pixels with depth values ​​within a preset threshold range to generate a region of interest image; Step S2: Accurate segmentation of ear of fruit and peduncle based on improved loss function; A Mask R-CNN instance segmentation network was constructed and trained using transfer learning on a labeled dataset. The total loss function included an edge-preserving loss term was adopted. After training converged, the segmentation model was obtained, and the output included ear mask, stalk mask, and 3D point cloud of ear. Step S3: Dynamic decision-making on the fruit stalk clamping point based on the optimal mechanical model; Extract the stalk skeleton line, calculate the diameter at each point and the distance to the connection point with the ear of fruit; construct the cantilever beam mechanical model of the stalk, calculate the bending deflection at the clamping point; calculate the position of the ear of gravity and the equivalent force arm length. Define a comprehensive evaluation value for the gripping point, select the point with the largest evaluation value as the optimal gripping point, and output its three-dimensional coordinates in the robot's base coordinate system. Step S4: Decision on the sparseness intensity based on density clustering and random forest regression; Connectivity analysis was performed on the ear mask to identify individual berries, and DBSCAN clustering was performed on the set of berry center points. Calculate the density of each cluster; construct a random forest regression model with the projected area of ​​the fruit bunch, the total number of fruit grains, the density variance of each cluster, the relative canopy height of the fruit bunch, and the light intensity as input features, and output the thinning intensity coefficient; when the thinning intensity coefficient is greater than the thinning threshold, select the thinning targets in descending order of density, and generate a list of thinning targets when the cumulative number of thinned fruit grains reaches the target value. Step S5: Robotic arm trajectory planning and robotic gripper collaborative shearing; Based on the list of targets to be removed, the spatial trajectory of the robotic arm joints is planned using fifth-order polynomial interpolation to achieve collision-free motion; the robotic arm is driven to move to the optimal gripping point, and the bionic flexible gripper is activated to clamp the fruit stalk with the calculated gripping force; the rotary shearing mechanism is activated to complete the shearing; the robotic arm is reset, and the above steps are repeated until the entire list of targets to be removed is completed. Step S6: Verification of sludge removal effect and closed-loop correction; After the thinning operation is completed, the canopy image after thinning is acquired again; the remaining fruit clusters are re-identified using the segmentation model in step S2, the actual number of thinned fruit clusters is calculated, and the thinning deviation is defined; if the absolute value of the deviation exceeds the allowable deviation threshold, a correction instruction is executed: if the deviation is positive, the subsequent thinning target is skipped; if the deviation is negative, step S4 is re-executed for the remaining fruit clusters to generate supplementary thinning targets.

2. The method for thinning wine grapes based on machine vision according to claim 1, characterized in that, The correction function for adaptive gamma correction enhancement in step S1 is: ； in, For pixel coordinates, The enhanced pixel value, The pixel value is the pixel coordinate. The expression for the adaptive gamma value is: ； in, For The average grayscale value of a local window centered at 15×15 pixels. The grayscale mean of the entire image. The standard deviation of grayscale in a local window. To prevent division by zero for very small positive numbers; The depth threshold for deep background removal is determined by an adaptive algorithm: ； in, and These are the mean and standard deviation of the depth image, respectively, retaining the depth values. Pixels within the range.

3. The method for thinning wine grapes based on machine vision according to claim 2, characterized in that, In step S2, the total loss function mentioned in step S2 Defined as: ； in, For classification cross-entropy loss, Smooth L1 loss for bounding box regression. For masking binary cross-entropy loss, These are the edge loss weighting coefficients. The edge-preserving loss is expressed as: ； in, The total number of samples, For the first The gradient of the mask is predicted for each sample. To correspond to the gradient of the real mask, It is an L2 norm; The edge loss weighting coefficient Employing a dynamic adjustment strategy: ； in, For training rounds, For the preheating round, These are the initial weights.

4. The method for thinning wine grapes based on machine vision according to claim 3, characterized in that, In step S3, the skeleton lines are refined from the fruit stalk mask using the Zhang-Suen parallel thinning algorithm. Extract from the data and prune branches using the eight-neighbor tracing algorithm, retaining only the main skeleton line connected to the ear mask; The bending deflection function for: ； in, The weight of the grape bunch is estimated by multiplying the volume of the three-dimensional point cloud of the bunch by a preset empirical value for grape berry density. The equivalent force arm length from the gripping point to the center of gravity of the ear of fruit. The elastic modulus of the fruit stalk, The moment of inertia of the clamping point section, The radius of the fruit stalk at the clamping point; The centroid of the ear is calculated using the centroid of the three-dimensional point cloud of the ear: ； in, For point cloud points, For the first The three-dimensional coordinates of each point; The formula for calculating the equivalent force arm length is as follows: ； in, For the center of gravity of the ear of fruit, Here are the coordinates of the clamping point. For the Euclidean norm, The angle between the direction of the grain's gravity and the axis of the fruit stalk; The comprehensive evaluation value of the clamping point for: ； in, For the maximum allowable deflection, To achieve the optimal clamping diameter, This represents the minimum distance between the clamping point and the adjacent fruit. For safe distance threshold, The weighting coefficients and Select As the optimal clamping point This is the set of points representing the skeletal framework of the fruit stalk.

5. The method for thinning wine grapes based on machine vision according to claim 4, characterized in that, The neighborhood radius of the core point in the DBSCAN clustering described in step S4 is ,in The average diameter of the fruit. The radius coefficient; The density of the cluster The calculation formula is: ； in, The number of fruits within a cluster. Let be the radius of the equivalent sphere of the cluster; The random forest regression model The input feature vector is: ； in, The projected area of ​​the ear of fruit. The total number of fruits, The density variance of each cluster, The relative height of the fruit bunch to the canopy. The light intensity is the model output; the light intensity coefficient is removed. : The density variance The calculation formula is: ； in, The total number of clusters, For the first The density of each cluster, The average density of all clusters; when At that time, according to density Select the thinning targets in descending order, and the cumulative number of thinned fruits meets the following condition: ； in, To remove the index set of the target cluster, To remove the initiation threshold; generate a list of removal targets. .

6. The method for thinning wine grapes based on machine vision according to claim 5, characterized in that, The trajectory planning function for the fifth-order polynomial interpolation mentioned in step S5 is: ； in, For the robotic arm joint angles over time A changing function, The coefficients are undetermined; the boundary conditions are: Solve for the coefficients to Obtain a smooth trajectory, where This is the initial joint angle. For the target joint angle, For exercise time; The clamping force The calculation formula is: ； in, For safety reasons, For the weight of the ear of fruit, The coefficient of friction between the gripper and the fruit stalk.

7. The method for thinning wine grapes based on machine vision according to claim 6, characterized in that, The formula for calculating the sparse deviation mentioned in step S6 is: ； in, The actual number of fruits removed. To eliminate the strength coefficient, This represents the total number of original fruit pieces. To eliminate the deviation rate; if Then execute the correction instruction: if If so, then skip the subsequent evacuation targets; if Then, for the remaining ears of fruit, step S4 is repeated to generate supplementary thinning targets. This is the allowable deviation threshold.

8. A machine vision-based grape thinning device for winemaking, characterized in that, It employs the machine vision-based grape thinning method for winemaking as described in any one of claims 1-7, and includes: A tracked mobile chassis is used for autonomous navigation and movement between grape rows. The front end of the tracked mobile chassis is equipped with a lidar and an inertial measurement unit to construct an environmental map between grape rows and achieve autonomous navigation. A six-degree-of-freedom collaborative robotic arm is mounted on a tracked mobile chassis; An end effector, installed at the end of a robotic arm; The embedded industrial computer is electrically connected to the tracked mobile chassis, the six-degree-of-freedom collaborative robotic arm, and the end effector, respectively. The remote monitoring terminal communicates with the embedded industrial control computer via a wireless network to display the operation status in real time and receive manual intervention commands.

9. The machine vision-based grape thinning device for winemaking according to claim 8, characterized in that, The end effector includes: The visual acquisition module includes an RGB-D camera, a ring-shaped LED fill light, and an optical protective cover; The biomimetic flexible gripping module includes two gripping fingers driven by stepper motors, with silicone pads and pressure sensors on the inner side of the gripping fingers; The rotary shearing module includes a brushless DC motor and a ring-shaped rotary blade; The embedded industrial control computer is internally equipped with: The image processing unit is equipped with a GPU acceleration module for performing image acquisition and segmentation; The decision analysis unit is used to perform clamp point calculations and target removal decisions. The motion control unit is used to perform trajectory planning and servo control.

10. The machine vision-based grape thinning device for winemaking according to claim 9, characterized in that, The coefficient of friction of the silicone pad satisfy: ； in, This is the base value for the static friction coefficient. For pressure-sensitive increments, This is the actual clamping force. For reference pressure.

Images