A flat peach picking robot and picking method based on pickability discrimination

Abstract

The application discloses a kind of based on pickability hierarchical discrimination's flat peach picking robot and picking method.The robot includes wheeled chassis, six-axis linkage mechanical arm, two-stage visual perception system, air suction-screw coordination end effector and closed-loop feedback control system.Two-stage visual perception system uses long-distance laser radar and depth camera to realize rough positioning of fruit tree, and near-distance end effector depth camera is combined with YOLO-BAMS instance segmentation model to realize accurate identification and pickability hierarchical discrimination of flat peach.Pickability discrimination uses hierarchical progressive strategy, first, ripeness discrimination is carried out, second, based on mature fruit, the contour integrity of non-connected region is extracted by morphological closing operation η Difference region is extracted, and the eccentricity e Further discrimination of the type of obstruction; in combination with the visible rate r v Quantify the degree of obstruction, and output picking decision comprehensively.End effector integrates cylinder control vacuum nozzle to realize air blowing leaf-air suction auxiliary cooperative operation, and four-finger flexible finger is driven by cam guide groove to realize screwing envelope gripping.The application realizes "perception-decision-execution" closed-loop control, and solves the technical problems of low flat peach recognition rate, difficult picking of obstructed fruit and high mechanical damage rate in natural orchard environment.

Description

Technical Field

[0001] This invention relates to the fields of agricultural automation equipment and computer vision technology, specifically to a peach-picking robot and picking method that integrates deep learning instance segmentation and mechanical twisting and grasping. Background Technology

[0002] Flat peaches are characterized by their thin skin, juiciness, short stems, and flat shape. In natural orchard environments, they are easily obstructed by branches and leaves, and the overlapping of clusters of fruit poses higher requirements for the target recognition accuracy and non-destructive grasping capabilities of harvesting robots. While existing deep learning-based instance segmentation models (such as Mask R-CNN and YOLO-seg series) have been widely applied to fruit detection tasks, they still have the following significant limitations in peach picking scenarios: First, under severe occlusion conditions, segmentation masks are prone to breakage, depression, or misjudgment of connected regions, making it impossible to output continuous and reliable fruit contour information, thus making it difficult to accurately determine whether the fruit is ready for picking; Second, most existing methods only output category labels, location coordinates, and maturity information, lacking quantitative analysis of occlusion type (such as branch occlusion, leaf occlusion), occlusion degree, and contour integrity, and the recognition results are only at the "presence or absence of fruit" level, which is difficult to meet the decision-making needs of precise picking; Third, lightweight models, in order to balance computational efficiency, often weaken the feature extraction ability of weak boundaries and long and thin occluded regions, resulting in insufficient robustness of recognition under complex lighting and background conditions. The aforementioned defects prevent the harvesting robot from effectively distinguishing between fully exposed, slightly obscured, and severely obscured fruits. The grasping decision lacks a quantitative risk basis, leading to problems such as missed picking, accidental picking, grasping failure, and mechanical damage.

[0003] For the above-mentioned occlusion scenarios, existing studies have attempted to improve the instance segmentation model by introducing boundary enhancement modules, multi-scale feature fusion or attention mechanisms to improve the continuity of the segmentation mask, but the following shortcomings still exist: (1) There is a lack of effective connection between visual perception output and picking action. The fruit mask obtained by segmentation cannot be further converted into "picking probability" or hierarchical discrimination strategy. The robot cannot adjust the grasping posture or end effector action mode autonomously according to the occlusion type (such as branch and leaf occlusion, fruit overlap) and the integrity of the outline; (2) Existing end effectors mostly adopt fixed strategies (such as parallel grippers or vacuum suction cups), lack the ability to cooperate in different occlusion scenarios, and it is difficult to achieve the segmented cooperative operation of "first blowing away leaf occlusion with air, then suctioning to assist positioning, and finally twisting and enveloping to grasp". For fruits that are slightly occluded but whose outlines can still be restored, existing systems often abandon them or forcibly pull them, resulting in low success rate of grasping and high mechanical damage rate; (3) Existing technologies lack a closed-loop feedback mechanism of "perception-decision-execution", and the vision system cannot adaptively adjust the criteria for picking based on the actual grasping results, thus making it difficult to form a continuously optimized picking strategy. Summary of the Invention

[0004] To address the shortcomings of the aforementioned technologies, the purpose of this invention is to provide a peach-picking robot and method based on pickability grading. Through deep integration of algorithms and mechanical structures, this invention solves the technical problems of low peach recognition rate, difficulty in picking obscured fruits, and high mechanical damage rate in natural orchard environments.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A peach-picking robot based on pickability determination, characterized in that it includes: Walking chassis module: chassis (41), walking wheels (45), lidar and depth camera (39); Robotic arm and its connecting components: end effector overall structure (29), end effector and robotic arm connection communication control module (30), depth camera (31), depth camera connection plate (32), six-axis linkage robotic arm (33), robotic arm spindle (34), robotic arm fixing module (35), load-bearing device (36), lifting platform (37); End effector: pressure sensor (1), silicone material (2), vacuum nozzle (3), screw fixing module (4), bearing slider (5), first linkage structure (6), screw cam slider (7), nut M6 (8), bolt M6 (9), base flange structure (10), stepper motor (11), control module for connecting with robotic arm (12), fixing screw (13), motor fixing base (14), communication module (15), second linkage structure fixing module (16), second linkage structure (17), coupling module (18), reduction gear protective cover (19), cam guide groove (20), finger connection fixing module (21), flexible finger part (22), vacuum nozzle fixing connection plate (23), nozzle fixing bolt (24), reduction gear driven wheel (25), reduction gear driving wheel (26), transmission shaft (27), cylinder (28); Control modules: PC-side display (38), PC-side control module (40), PLC control module (42), power control module (43), protective cover (44). Collection feedback module: weight sensor (46), collection box (47).

[0007] The lidar and depth camera (39) are located above the front of the chassis (41) for three-dimensional perception of the orchard environment and coarse positioning of fruit trees; the depth camera (31) is located at the bottom of the end effector structure (29) for close-range accurate identification of peaches and judgment of their pickability; the PC control module (40) has a built-in YOLO-BAMS instance segmentation model and a pickability classification algorithm, receives dual-stage visual data and outputs picking decisions; the cylinder (28) controls the vacuum nozzle (3) to achieve coordinated operation of air blowing to disperse leaves and air suction assistance; the stepper motor (11) meshes with the drive shaft (27), the reduction gear drive wheel (26) and the reduction gear driven wheel (25) to drive the rotary cam slider (7) to move in the cam guide groove (20), thereby driving the flexible finger part (22) to achieve rotary envelope grasping; the weight sensing device (46) is located at the bottom of the collection box (47) for closed-loop feedback to determine whether the picking is successful.

[0008] The two-stage visual perception includes: The first stage: The lidar and depth camera (39) acquire RGB images and depth point cloud data of the orchard environment to achieve coarse positioning of the fruit trees. The PC control module (40) controls the chassis (41) to move to the target area. The second stage: The depth camera (31) acquires close-range RGB-D images, inputs them into the YOLO-BAMS instance segmentation model, and outputs the peach maturity label, bounding box coordinates and segmentation mask. Combined with the depth information, it is converted to the robot coordinate system to achieve precise positioning of the center point of the peach to be picked.

[0009] The YOLO-BAMS instance segmentation model is an improvement on the YOLOv12n-seg architecture, including: The backbone network employs a boundary-local perception initial feature fusion module to extract and fuse multi-scale initial features. This module, located at the network front end, uses a dual-branch parallel structure: the first branch explicitly extracts edge information using the SobelConv operator to achieve accurate fruit segmentation; the second branch uses a zero-filling max-pooling layer to capture the maximum response in local regions, enhancing the perception of texture, color, and morphological details, thereby assisting in the identification of maturity indicators such as fruit blush, background color gradient, and peel gloss, as well as harvestable status information such as foliage occlusion. The outputs of the two branches are concatenated along the channel dimension, then subjected to a convolutional layer with a stride of 2 for secondary fusion and spatial downsampling of composite features, and finally mapped to the target number of channels via a 1×1 convolution.

[0010] The backbone network also includes a C3K2-Faster-EMA module and an ADown module. The C3K2-Faster-EMA module utilizes PConv partial convolutions to achieve low-redundancy spatial mixing, performing convolution calculations only on a subset of channels, thus preserving local texture and edge details while reducing redundant computation. Subsequently, a lightweight MLP consisting of 1×1 convolutions is used to achieve channel interaction. Finally, an efficient multi-scale attention mechanism, EMA, is introduced to adaptively recalibrate the multi-scale spatial channel responses, enhancing the saliency of the fruit's main body and edge regions while suppressing interference from complex backgrounds. This module as a whole employs residual connections to maintain the original information flow and training stability. The ADown module first performs pre-smoothing and noise reduction using 2×2 average pooling (with a stride of 1) to maintain the spatial size. Then, it is divided into two branches in the channel dimension: branch A uses 3×3 convolution to achieve learning-based downsampling and extract deep features; branch B uses max pooling with a stride of 2 to retain significant texture and boundary information, and adjusts the number of channels using 1×1 convolution. After the outputs of the two branches are concatenated, multi-path fusion of learning-based and regular downsampling is achieved, thereby enhancing the model's adaptability to scale.

[0011] The neck network uses the A2C2f module for multi-scale feature fusion and achieves cross-layer feature aggregation through upsampling and concatenation operations.

[0012] The head network employs a multi-task hierarchical segmentation detection head to achieve joint output of instance segmentation, maturity classification, and harvestable state prediction. This multi-task hierarchical segmentation detection head is based on shared convolutional features, retaining the prototype generation branch and the mask coefficient prediction branch, while decoupling the classification branch into two parallel subtasks: one is a three-class classification of maturity, dividing peaches into immature, semi-ripe, and mature categories; the other is for mature peaches to be classified into harvestable and unharvestable states, thus achieving instance-level multi-task output.

[0013] In summary, the boundary and local perception initial feature fusion module integrates low-level edge information and mid-level local context features through parallel processing and multi-level fusion, providing the instance segmentation network with initial feature representations that are highly sensitive to fruit boundaries and surface details; the C3K2-Faster-EMA module improves the separability and boundary representation ability of features in occluded scenes with lower computational cost; the ADown module combines the strong response preservation ability of max pooling with the background detail capture ability of average pooling plus convolution to achieve a balance between resolution reduction and information integration; and the multi-task hierarchical segmentation detection head reduces computational overhead and improves inference efficiency through a prototype-coefficient separation prediction strategy.

[0014] The harvestability grading algorithm employs a four-level progressive strategy: Level 1: Maturity determination. Based on the features output by the YOLO-BAMS instance segmentation model, subsequent determinations are performed only on mature fruits; Level 2: Contour connectivity analysis. Contour integrity is calculated. ,in A c Let be the area of ​​the fruit's circumscribed contour. A h The area of ​​the inscribed maximum convex hull; η If the value is ≥0.85, it is considered basically connected, and the harvesting decision is initiated directly. η When the value is less than 0.85, the occlusion type is determined; Level 3: Occlusion type determination. This applies to the segmentation mask. M After morphological closing operation, we get M ', difference region ; Calculate eccentricity ,in A D For the area of ​​the difference, L D Perimeter of the region of difference; e When the value is less than 0.6, it is determined to be branch obstruction, triggering the robotic arm's view switching mechanism; e A value ≥0.6 indicates leaf occlusion, triggering occlusion degree quantification; Level 4: Occlusion degree quantification. Calculate fruit visibility. r v , ρ The fruit filling coefficient is set to 0.785; r v Fruits with a hull size greater than 0.7 are considered ready to be picked; otherwise, they are not. When multiple fruits overlap, [follow the instructions]. r v Determine the harvesting priority by sorting in descending order.

[0015] The harvestable state S Defined by the comprehensive decision function:

[0016] When the outline completeness hour, It is determined that it is ready for harvesting; when the centrifugal rate is... And fruit visibility hour, If the item is deemed ready to be picked, it is considered ready to be picked; otherwise, the output is... If a ripe peach is obscured by tree branches, the robotic arm needs to be moved to change the viewing angle and reassess whether it can be harvested.

[0017] The air-blowing-air-suction coordinated operation of the end effector includes: Air blowing stage: When the end effector approaches the peach to be picked, the cylinder (28) controls the vacuum nozzle (3) to perform the air blowing action, blowing away the leaves covering the peach surface, improving the recognition accuracy of the depth camera (31) and the accuracy of the center point positioning; Air suction stage: During the process of the flexible finger part (22) twisting and wrapping the peach, the cylinder (28) switches the vacuum nozzle (3) to the air suction mode, adsorbing the surface of the peach fruit, increasing the shedding force and reducing mechanical contact damage.

[0018] The twisting envelope mechanism includes: The cam guide groove (20) consists of four guide rails that are 90° apart. The rotating cam slider (7) moves along the guide rail. The stepper motor (11) drives the transmission shaft (27), which transmits torque to the cam shaft through the meshing of the reduction gear drive wheel (26) and the reduction gear driven wheel (25). The cam slider (7) drives the second linkage structure (17), the first linkage structure (6) and the flexible finger part (22) to realize the four-finger rotating and enveloping action. The flexible finger part (22) is equipped with a pressure sensor (1) and silicone material (2) on the inner side. When rotating and grasping, it envelops the fruit along the suture line of the peach fruit and from the bottom depression of the fruit, conforming to the flat shape of the peach.

[0019] The closed-loop feedback control includes: After the end effector grabs the peach and places it in the collection box (47), the weight sensor (46) detects the weight change and feeds it back to the PC control module (40). The PC control module (40) determines whether the weight increment has reached the preset threshold. After confirming that the picking is successful, it sends a command to control the six-axis linkage robotic arm (33) to return to the initial position. If the weight does not change, the picking is determined to be a failure and the re-identification and grabbing process is triggered.

[0020] A method for harvesting peaches based on the aforementioned robot, characterized by comprising the following steps: S1: Long-distance coarse positioning: LiDAR and depth camera (39) acquire orchard environment data, PC control module (40) identifies fruit tree location and plans path, and controls chassis (41) to move to target area; S2: Close-range precise recognition: The six-axis linkage robotic arm (33) adjusts the pose to bring the depth camera (31) closer to the target, collects RGB-D images and inputs them into the YOLO-BAMS model, and outputs maturity labels, segmentation masks and center point coordinates; S3: Pickability determination: The PC control module (40) runs the pickability classification algorithm, and sequentially executes maturity determination → connectivity analysis → shading type determination → shading degree quantification, and outputs picking decision S; S4: Fetch and execute: If S=1, the robotic arm controls the end effector to move to the gripping point, the cylinder (28) executes the air blowing of the diffuser, the stepper motor (11) drives the flexible finger (22) to twist and wrap, and at the same time the vacuum nozzle (3) switches to air suction to assist in the detachment; S5: Placement feedback: The robotic arm places the peaches into the collection box (47), and the weight sensor (46) detects the weight change and feeds it back to the PC control module (40). After successful confirmation, it returns to the initial position to complete a single harvest.

[0021] The steps S The training and deployment of the YOLO-BAMS model in section 2 includes: constructing a dataset of RGB images of peaches in a natural orchard environment; using the X-AnyLabeling tool for data annotation; employing the minimum bounding rectangle annotation method to reduce background interference; performing data augmentation operations on the dataset, including random rotation, brightness and contrast adjustment, noise addition, and Mosaic enhancement, to expand the dataset size and improve the model's generalization ability; converting the annotated JSON file format to XML format, and then to TXT text format required for YOLO model training; dividing the training set, validation set, and test set by a specific ratio to train and improve the YOLO-BAMS detection model, and performing model validation; and deploying the model on the robot's mobile device by using transform acceleration and intermediate model conversion.

[0022] The steps S The second part of the grasping center point coordinate transformation includes: realizing spatial coordinate system transformation by calling the depth camera of the robot port; the output results include target confidence information, corresponding to the spatial coordinates of the grasping center point in the robot coordinate system, so as to realize the motion positioning of the robot's control end effector; setting the communication protocol and the robot arm MoveIt! control program to realize the recognition and grasping of peaches.

[0023] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: Dual-stage vision-driven robotic arm motion: Long-range lidar and depth camera (39) achieve coarse positioning of fruit trees and control the movement of chassis (41); near-range end effector depth camera (31) combined with YOLO-BAMS model for accurate identification, outputting the coordinates of the grasping center point to drive the movement of robotic arm (33); Pickability determination guides end effector action: the pickability state S output by the algorithm directly determines the end effector's action mode. S When the value is 0, the robotic arm switches the view or skips the view. S When =1, the coordinated grasping action of air blowing, turning, and air suction is initiated; Air blowing to disperse leaves improves algorithm accuracy, and algorithm output optimizes the timing of grasping: the air blowing stage removes leaf obstruction and improves the recognition accuracy of the depth camera (31); the type of obstruction (branch / leaf) in the pickability judgment determines the air blowing intensity and twisting angle of the end effector; Weight feedback closed-loop verification algorithm - execution consistency: The weight sensing device (46) verifies the success or failure of picking and feeds back to the PC control module (40) to optimize the subsequent identification threshold. Attached Figure Description

[0024] To more clearly illustrate the existing technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments are briefly described below.

[0025] Figure 1 This is a schematic diagram of the end effector structure in this invention;

[0026] Figure 2 This is a schematic diagram of the cam guide groove structure in this invention;

[0027] Figure 3 This is a schematic diagram of the reduction gear meshing transmission in this invention;

[0028] Figure 4 This is a schematic diagram of the six-axis linkage robotic arm structure in this invention;

[0029] Figure 5 This is a schematic diagram of the overall structure of the peach-picking robot of the present invention;

[0030] Figure 6 This is a force analysis diagram of the cam slider in this invention;

[0031] Figure 7 This is a diagram of the algorithm-mechanical fusion control system architecture of the present invention;

[0032] Figure 8 This is a flowchart of the two-stage visual perception and automated harvesting process in this invention.

[0033] Figure 9 This is a flowchart illustrating the improved principle of the YOLO-BAMS instance segmentation module in this invention.

[0034] Figure 10 This is a flowchart of the harvestability grading process in this invention;

[0035] Figure 11 This is a diagram of the robot grasping closed-loop control architecture based on pickability discrimination in this invention;

[0036] Figure 12 This is a module architecture diagram of the pluckability-based crawling method in this invention;

[0037] Figure 13This is a diagram of the YOLO-BAMS instance segmentation network structure in this invention;

[0038] Figure 14 Visualization of the YOLO-BAMS instance segmentation effect in this invention;

[0039] Figure 15 This is a comparative experiment of the instance segmentation model in this invention on the test set;

[0040] Figure 16 Indoor recognition and harvesting experiment of the YOLO-BAMS instance segmentation model in this invention;

[0041] Figure 17 These are actual harvesting test data for the peach-harvesting robot in this invention. Detailed Implementation

[0042] The accompanying drawings, in conjunction with the embodiments of this application, will provide a clearer description of the embodiments and more explicitly highlight the technical solutions and advantages of this application. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0043] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] like Figure 1-3As shown, the present invention provides an overall structure (29) of an end effector for peach picking, including: a pressure sensor (1) and a silicone material (2) disposed on the inner surface of the flexible finger part (22) for detecting contact pressure and buffering mechanical impact; a screw fixing module (4) cooperating with a screw cam slider (7) and disposed directly above the four guide rails of the cam guide groove (20); the cam guide groove (20) is composed of four guide rails at 90° to each other; a bearing slider (5) fixedly connected to the finger connection fixing module (21) and axially cooperating with the first connecting rod structure (6); and a base flange structure (10) and a second connecting rod structure fixing module (16) fixed by nuts M6 (8) and bolts M6 (9). The stepper motor (11) is located directly below the base flange structure (10). The motor mounting base (14) is fixed to the lower end of the control module (12) connected to the robotic arm by fixing screws (13). The transmission shaft (27) connects the stepper motor (11) and the reduction gear drive wheel (26). The torque is transmitted to the cam shaft through the meshing of the reduction gear driven wheel (25), which drives the rotating cam slider (7) to move along the guide rail, thereby driving the second linkage structure (17), the first linkage structure (6), and the flexible finger part (22) to achieve the rotating envelope. The vacuum nozzle (3) is fixed by the vacuum nozzle fixing connection plate (23) and the nozzle fixing bolt (24). The cylinder (28) controls its extension and retraction to achieve the switching between air blowing and air suction.

[0045] The connection control module (12) with the robotic arm and the end effector-robotic arm connection communication control module (30) are the same connection interface components, respectively located on the end effector side and the robotic arm side.

[0046] The expression for the force analysis of the cam slider is: ;in, F G represents the force exerted by the cam slider on the push rod, and G represents the local load borne by the push rod. F R1 This indicates the force applied by the push rod to both sides of the guide rail of the cam guide groove. Φ This indicates the friction angle; the push rod is made of TPU material and can be 3D printed. b Indicates the cantilever length. l Indicates the length of the guide rail.

[0047] The power transmission consists of a stepper motor (11), a drive shaft (27), a driven wheel of a reduction gear (25), and a driving wheel of a reduction gear (26). The picking force when the peach fruit rotates is provided by the stepper motor (11), which is connected to the base flange structure (10) through the drive shaft (27), and then transmitted to the second link (17) and the first link (6). The finger part (22) is driven to twist and grasp the peach by the bearing slider (5) and the finger connection fixing module (21).

[0048] The air suction structure consists of a vacuum nozzle (3), a vacuum nozzle fixing plate (23), a nozzle fixing bolt (24), and a cylinder (28). The vacuum nozzle is fixed to the end of the cylinder, and the air blowing and air suction actions of the vacuum nozzle can be realized by the extension and retraction of the cylinder.

[0049] like Figure 3 As shown, the gear meshing between the driven gear (25) and the driving gear (26) of the reduction gear provides smooth and accurate power transmission characteristics. Torque is transmitted between the drive shaft (27) and the driving gear shaft (26), and torque and power are transmitted between the driven gear shaft (25) and the cam guide groove (20).

[0050] like Figure 4 As shown, this invention provides a robotic arm design suitable for peach harvesting, comprising the following steps: The main components of the robotic arm are: end effector overall structure (29), robotic arm communication connection control module (30), depth camera (31), depth camera connection plate (32), six-axis linkage robotic arm (33), robotic arm spindle (34), robotic arm fixing module (35), load-bearing device (36), and lifting platform (37). The depth camera (31), installed at the bottom of the end effector overall structure, combined with a target recognition algorithm, performs target recognition and spatial positioning of the peaches in the orchard environment. The robotic arm controls the end effector to reach the predetermined picking position, and the peaches are picked using the aforementioned dedicated end effector for peach picking.

[0051] like Figure 5 As shown, the overall structure includes: The chassis (41) is equipped with a walking wheel (45) below it, and a lifting platform (37), a load-bearing device (36), and a robotic arm fixing module (35) are arranged on top of it in sequence. The six-axis linkage robotic arm (33) and the end effector are installed on the load-bearing device (36). The PC control module (40) is located in the middle of the chassis (41), the PC display screen (38) is placed above it, and the laser radar and depth camera (39) are placed in front of it on the right. The PLC control module (42) and the power control module (43) are placed below the protective cover (44). The collection box (47) is placed at the rear of the chassis (41), and a weight sensing device (46) is installed at the bottom.

[0052] like Figure 7-10 As shown, the integrated control process of the algorithm and mechanical structure is as follows.

[0053] (1) Long-distance coarse positioning stage: The lidar and depth camera (39) acquire RGB images and depth point clouds of the orchard environment. The PC-side control module (40) runs the SLAM algorithm to build an environmental map, identify the location of the fruit trees, and plan the walking path of the chassis (41). At this stage, only the orientation of the fruit trees is output, and no fine recognition is performed to reduce the computational load.

[0054] (2) Close-range precise identification stage: The six-axis linkage robotic arm (33) adjusts its pose so that the depth camera (31) is 0.3-0.5m away from the target fruit. The acquired RGB-D image is input into the YOLO-BAMS model: The BL-AFFM module of the backbone network extracts multi-scale features; the C3K2-faster-EMA module enhances the representation of occluded fruit features through a separation and enhanced attention mechanism; the A2C2f module of the neck network fuses cross-layer features; and the MHS-SegHead of the head network outputs a segmentation mask, maturity classification, and bounding box. The depth image and segmentation mask are fused, and the coordinate system is transformed to output the three-dimensional coordinates of the center point.

[0055] (3) Harvestability assessment stage: The comprehensive judgment adopts a hierarchical decision-making strategy: first, maturity is determined, and subsequent judgments are only performed on mature fruits; second, fruit region connectivity analysis is conducted, based on the integrity of the outline. η The calculation is performed as follows: ;in, A c The area of ​​the region enclosed by the circumscribed outline of the fruit; A h This represents the area of ​​the largest inscribed convex hull within the region. Based on... η The connectivity of the values ​​is classified: if η If the value is ≥0.85, the fruit outline is considered to remain basically connected, and it is deemed ready for harvesting; if... η If the value is less than 0.85, it indicates that the fruit region may be occluded, and the occlusion type is determined. Secondly, for fruit regions with insufficient connectivity, morphological operations are used to analyze the occlusion features. The fruit segmentation mask is then applied. M After morphological closing operation, the following was obtained M' The difference set of the two constitutes the region of difference. D The calculation formula is as follows: Further extract the geometric morphological features of the differential regions, where the formula for calculating the eccentricity is: ;in, A D For the area of ​​the difference, L D Its outer contour perimeter. Set a shape discrimination threshold: when... eWhen <0.6, the difference area is elongated and classified as branch obstruction; the fruit obstructed by branches is determined to be unharvestable and triggers the perspective switching mechanism, and the robotic arm (33) moves 30° to reshoot; e When the value is ≥0.6, the degree of shading of the fruit by the leaves is further quantified.

[0056] Secondly, a further quantitative analysis of the degree of shading was conducted: fruit surface visibility rate. r v The calculation formula is: ;in, ρ The fruit filling coefficient is 0.785. r v The visible pixel area of ​​the fruit in the instance segmentation result; R To fit the equivalent radius of a complete fruit; r v It is negatively correlated with the degree of occlusion: when r v If the leaf density is >0.7, harvesting is deemed feasible under the current shading conditions and is therefore considered harvestable; conversely, if the density is below 0.7, harvesting is deemed temporarily impossible due to excessive leaf shading. Finally, through... r v The numerical values ​​are sorted in descending order to prioritize the picking of multiple fruits, which further leads to the planning of picking routes for ripe peaches and the output of picking decisions.

[0057] (4) Algorithm-driven crawling and execution phase: Harvesting status S When =1, the PC control module (40) sends a command; the robotic arm (33) moves to the gripping point; the cylinder (28) controls the vacuum nozzle (3) to perform 0.3MPa air blowing, which lasts for 0.5s to blow away the leaves on the surface of the peach to be picked; the center point is confirmed a second time by the depth camera (31); the stepper motor (11) drives the flexible finger (22) at 15r / min to twist 120° to envelop the bottom of the peach; the air nozzle is switched to 0.05MPa to perform air suction to assist the fruit to fall off; the pressure sensor (1) determines that the gripping is successful when the contact force is less than 2N.

[0058] (5) Closed-loop feedback stage: The robotic arm (33) places the peaches into the collection box (47), and the weight sensor (46) detects the weight increment. Success is confirmed when the increment exceeds 100g, and the PC control module (40) records the fruit. r v and η The data is used to optimize subsequent judgment thresholds; if the increment is less than 100g, the judgment fails and a re-grabbing process is triggered.

[0059] The robot grasping method in this invention implements a differentiated grasping strategy based on the aforementioned harvestability grading results, including the following steps:

[0060] Step 1001: Input the harvestability judgment results and make harvesting analysis decisions.

[0061] Harvesting decision based on the harvestability grading algorithm described in claim 4 S And intermediate discrimination parameters, to analyze the harvestable status of the current target peach: like S =0, terminate the current capture process, and the PC control module (40) triggers the corresponding strategy according to the occlusion type: when it is determined to be branch occlusion ( e When <0.6), the robotic arm (33) perspective switching mechanism is triggered, moving 30° to re-shoot and perform secondary discrimination; when it is determined to be leaf occlusion and r v If the value is ≤0.7, skip the current fruit and proceed to identify the next target; if S =1, extracting the intermediate parameter set generated during the pickability determination process. Θ ={ η , e , r v , M , D},in M For instance segmentation mask, D The region of difference serves as the input constraint for subsequent grasping posture planning and end effector motion control.

[0062] Step 1002: Multimodal feature extraction and grasp point optimization based on pluckability constraints.

[0063] In the image acquisition stage, a depth camera (31) is used to acquire close-range RGB-D images of the target peach, and simultaneously acquire depth maps and RGB images. The depth map is used to provide three-dimensional spatial information of the orchard scene and point cloud data of the fruit surface, while the RGB image is used to provide visual features such as fruit color, texture and outline.

[0064] For the acquired RGB images, a convolutional neural network is used to extract color image features to capture the visual characteristics of the target peach, such as color, texture, and outline. For the acquired depth images, a depth feature extractor is used to extract depth image features to capture the spatial geometric characteristics of the target in the scene.

[0065] The deep features and color features are fused and input into the ResUNet feature extraction network to further improve the deep features after multimodal fusion. This process is a multi-scale feature extraction of point cloud data to adapt to the feature expression needs of peaches at different sizes and scales in the image, and to ensure that peaches in different growth states and locations can be effectively identified.

[0066] Specifically, the instance segmentation mask based on the pluckability discrimination output. M In the ResUNet feature extraction process, a mask attention weighting mechanism is introduced: high weights are assigned to features within the masked region, while high weights are assigned to features in the difference region. D By suppressing corresponding features, the main features of the fruit are enhanced, the interference of branches and leaves is weakened, and the accuracy of center point positioning is improved.

[0067] Step 1003: Grasping posture generation and end effector motion planning based on retrieval parameters.

[0068] Based on the multi-scale feature map output from the above steps, combined with the pluckability parameter set Θ The system generates a graspability score for each point in the space, quantifying the probability that the point is a peach-grabbing point; it analyzes the graspability scores to select high-grabability points, i.e., the most suitable spatial points for performing the grasping operation; the inputs are point cloud data and a set of pickability parameters. Θ The output is the gripping posture, including information such as the gripping position, gripping angle, twisting amplitude, and air suction intensity of the end effector.

[0069] The grasping posture generation is based on the pluckability discrimination result and differentiated parameter configuration: When the outline completeness η When the value is ≥0.85, the fruit is considered to be basically unobstructed, and the standard grasping posture is adopted: twisting angle 120°, air suction pressure -0.05MPa, and the air blowing stage is skipped; When determined to be leaf occlusion ( e ≥0.6) and r v When the value is >0.7, based on visibility... r v Dynamically adjust the air blowing intensity and the rotation compensation angle: The air blowing pressure is set to 0.3MPa × (1.2- r v Continue for 0.5 seconds to clear blade obstruction; the rotation angle compensation is 120° + 15° × (1 - r v To increase the envelope range and adapt to the morphological uncertainty caused by partial occlusion; the inhalation pressure is maintained at -0.05 MPa; The grasping point is preferentially located at the bottom depression of the fruit, and the fruit is twisted along the suture line. This strategy utilizes a segmentation mask in the harvestability assessment. M Geometric center and region of difference D The spatial distribution relationship is automatically determined.

[0070] Step 1004: Robot execution and closed-loop grasping completion stage based on pickability judgment.

[0071] Based on the grasping posture output in step 1003, the PC-side control module (40) sends instructions to the end effector to execute differentiated grasping actions: The robotic arm (33) moves to the gripping point, and the end effector's overall structure (29) adjusts its posture so that the flexible finger part (22) aligns with the bottom of the fruit; If the current target is obscured by leaves ( e ≥0.6 and r v ≤0.9), the cylinder (28) controls the vacuum nozzle (3) to perform the air blowing action planned in step 903, blowing away the leaves covering the surface of the peach, and the depth camera (31) confirms the center point coordinates for the second time; The stepper motor (11) drives the flexible finger (22) at 15r / min to perform the twisting angle planned in step 903 to enclose the bottom of the peach. During the twisting process, the pressure sensor (1) monitors the contact force in real time. When the contact force is less than 2N, the current enclosed state is maintained. The air nozzle is switched to the suction pressure planned in step 903 to assist in fruit detachment; the robotic arm (33) places the peach in the collection box (47), the weight sensor (46) detects the weight increment, and when the increment is greater than 100g, the harvest is confirmed to be successful, and the PC control module (40) records the harvestability parameter set of the fruit. Θ This is used to optimize the subsequent discrimination threshold; if the increment is less than 100g, the judgment fails and a re-identification and capture process is triggered.

[0072] Thus, the single-harvest process based on the harvestability assessment is completed, achieving closed-loop control of "assessment-planning-execution-feedback".

[0073] This invention achieves the following technical effects through the deep integration of algorithms and mechanical structures: (1) The BL-AFFM and MHS-SegHead modules of the YOLO-BAMS model improve the recognition accuracy under occlusion conditions while maintaining lightweight design, with an instance segmentation mAP of 92.06%; (2) The occlusion type and degree output by the algorithm are directly mapped to the robotic arm action (view switching, air blowing intensity, and twisting angle), which increases the success rate of picking to over 90%. (3) The air-blowing-twisting-air-suction coordinated end effector is designed with a four-finger enveloping trajectory for the flat shape of the peach to reduce mechanical damage during harvesting; (4) Weight sensing closed-loop feedback verifies the success or failure of harvesting, realizing complete closed-loop control of a single harvesting task.

[0074] like Figure 14 , Figure 15 As shown, the improved YOLO-BAMS model was tested on the peach dataset, and it performed well in identifying and segmenting immature, pickable, and unpickable peaches, with no missed or false detections. Visual labels and coordinate information were also generated for easy viewing.

[0075] like Figure 16 As shown, under the collaborative mechanism of peach recognition-positioning-grabbing in this invention, the positioning of the center point to be grasped by the picking robot is as follows:

[0076] Specifically, firstly, a dataset of flat peaches in a natural orchard environment is constructed to train and improve the YOLO-BAMS model. The model is then validated to confirm its ability to perform high-quality segmentation and harvestability assessment of flat peaches. Next, the model is deployed on the robot's mobile end using transform acceleration and intermediate model conversion. Spatial coordinate system transformation is achieved by calling the depth camera on the robot's port. The output includes target confidence information and the spatial coordinates of the grasping center point in the robot's coordinate system, enabling the robot's end effector to perform motion localization. This process acquires and transforms the coordinate information required for grasping. Communication protocol settings and the MoveIt! control program for the robotic arm are then configured to recognize and grasp flat peaches. A successful grasping and placement of a flat peach by the end effector is considered a successful grasping process, thus achieving a collaborative mechanism of "recognition-localization-grabbing."

[0077] The above description is merely a specific embodiment of this patent, enabling those skilled in the art to understand the method and core idea of ​​the present invention. For those skilled in the art, based on the ideas of this invention, there may be changes in the specific implementation methods and application scope. Therefore, this application is not limited to the specific embodiments shown herein, but is to be accorded the widest scope of the principles and novelty claimed herein.

Claims

1. A peach-picking robot based on pickability determination, characterized in that, include: The walking chassis module is used to carry the robot and enable it to move autonomously within the orchard; The robotic arm module, mounted on the walking chassis module, is used to drive the end effector to the location of the fruit to be picked; The dual-stage visual perception module includes a first visual sensor mounted on the walking chassis and a second visual sensor mounted on the end effector, wherein the first visual sensor is used for coarse positioning of the target area of ​​the fruit tree, and the second visual sensor is used for precise identification of the target peach and positioning of the grasping center. The end effector includes a flexible envelope gripping mechanism, a twisting drive mechanism, and an air blowing-air suction coordinated mechanism. The twisting drive mechanism drives the flexible envelope gripping mechanism to envelop the peach and twist it away along the fruit stem. The air blowing-air suction coordinated mechanism is used to blow away the leaves that are blocking the fruit before picking and to assist the fruit in detaching during the picking process. The control module has a built-in YOLO-BAMS instance segmentation model and a pickability classification algorithm, which is used to generate picking decisions based on the two-stage visual perception results and control the robotic arm and end effector to perform picking actions. The collection and feedback module, including a collection box and a weight detection device, is used to detect the harvesting results and feed back the harvesting success information to the control module to form a closed-loop control for harvesting.

2. The peach-picking robot according to claim 1, characterized in that, The two-stage visual perception module includes: The first stage utilizes LiDAR and depth cameras to acquire RGB images and depth point cloud data of the orchard, achieving coarse localization of the target area of ​​the fruit trees. The second stage uses a depth camera mounted on the end effector to acquire RGB-D images of the target peaches, and inputs them into the YOLO-BAMS instance segmentation model to obtain maturity labels, instance segmentation masks, pickability discrimination, and the coordinates of the capture center point.

3. The peach-harvesting robot according to claim 2, characterized in that, The YOLO-BAMS instance segmentation model includes: A boundary-local perception initial feature fusion module is used to extract multi-scale boundary information; The C3K2-Faster-EMA module is used to enhance the representation of fruit boundary features in occluded scenarios; The ADown module is used to perform fusion downsampling of learning-based and rule-based methods; Multi-scale feature fusion is achieved using the A2C2f module; A multi-task hierarchical segmentation detection head is used to simultaneously output instance segmentation results, maturity classification results, and harvestable status prediction results.

4. The peach-picking robot according to claim 1, characterized in that, The pickability grading algorithm includes: Maturity assessment; contour connectivity analysis; occlusion type assessment; occlusion degree quantification; Based on the above judgment results, output the harvestable status and harvesting priority of the target fruit.

5. The peach-harvesting robot according to claim 4, characterized in that, The contour connectivity analysis uses contour completeness. η Perform the calculation: ;in A c Let be the area of ​​the fruit's circumscribed contour. A h The area of ​​the inscribed maximum convex hull is used; when η≥0.85, the fruit outline is determined to be connected; otherwise, the occlusion type is determined.

6. The peach-harvesting robot according to claim 5, characterized in that, The occlusion type determination is achieved by obtaining the difference region through morphological closing operation using the fruit instance segmentation mask, and then based on the eccentricity of the difference region: Distinguish between branch shading and leaf shading; in A D For the area of ​​the difference, L D The perimeter of the region of difference.

7. The peach-harvesting robot according to claim 6, characterized in that, The degree of occlusion is quantified based on the fruit visibility rate. r v Calculations are performed when the fruit visibility rate r v When the target fruit exceeds a preset threshold, it is determined that the target fruit can be picked, and the picking priority of multiple target fruits is sorted according to the fruit visibility value.

8. The peach-picking robot according to claim 1, characterized in that, The end effector includes: A twisting mechanism driven by a stepper motor; a four-finger flexible enveloping mechanism driven by a cam guide groove; and a vacuum nozzle controlled by a cylinder; wherein the cylinder controls the vacuum nozzle to perform air blowing and air suction actions, and the stepper motor drives the flexible enveloping mechanism to complete the twisting and enveloping harvesting along the peach suture line.

9. The peach-picking robot according to claim 1, characterized in that, The weight detection device detects changes in the weight of the collection box. When the weight increment reaches a preset threshold, it determines that the picking is successful and controls the robotic arm to return to the initial position. When the weight increment does not reach the preset threshold, it triggers the target re-identification and re-grabbing process.

10. A method for harvesting peaches using a peach-harvesting robot according to any one of claims 1 to 9, characterized in that, Includes the following steps: S1. Obtain orchard environmental information and locate the target area; S2. Acquire RGB-D images of the target peaches and input them into the YOLO-BAMS instance segmentation model to obtain maturity, instance segmentation results, pickability determination, and center coordinates. S3. Run the harvestability classification algorithm to comprehensively analyze maturity, contour connectivity, occlusion type and occlusion degree, and generate harvesting decisions. S4. Based on the harvesting decision, the end effector is controlled to perform coordinated harvesting of air blowing, twisting envelope, and air suction. S5. Based on the weight detection results, complete the harvest feedback and execute the next round of harvesting according to the feedback results.

11. The harvesting method according to claim 10, characterized in that, Step S2 includes constructing a dataset of peach images in a natural orchard environment, annotating and augmenting the images, training a YOLO-BAMS instance segmentation model, and deploying it on a robot control platform after training.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the peach picking method according to claim 10 or 11.

Images