Intelligent apple picking robot system with rigid and flexible heterogeneous mechanical arms cooperatively working

Abstract

The invention discloses an intelligent apple picking robot system with rigid and flexible heterogeneous mechanical arms cooperatively working, and solves the problems of missing picking of fruits on the inner side of a fruit tree, low single-arm picking efficiency and mechanical arm positioning drift caused by advancing vibration in the prior art through the cooperative design of a moving platform, a picking mechanism, a control system and a multi-sensor fusion system. The mobile platform adopts a crawler-type chassis, the front end is provided with an ultrasonic sensor to realize obstacle avoidance, and a middle collection box is provided with an infrared sensor to monitor a full-load state; two groups of rigid mechanical arms and two groups of flexible mechanical arms of the picking mechanism adopt a dynamic priority scheduling mechanism to realize labor division picking based on binocular camera and depth camera data, the rigid arms are responsible for picking low-shielding fruits on the periphery of a fruit tree, and the flexible arms go deep into a branch and leaf area to pick high-shielding fruits, so that full coverage of single fruit tree picking is realized; the multi-sensor tight coupling positioning system is mainly based on a laser radar, and integrates an IMU and a differential GNSS-RTK to realize centimeter-level positioning in a GPS denial environment. The actually measured target picking success rate of the rigid-flexible mechanical arm treatment score of the single fruit tree reaches 87.2%.

Description

Technical Field

[0001] This invention belongs to the field of agricultural robot technology, specifically an intelligent apple-picking robot system with a rigid-flexible heterogeneous robotic arm working collaboratively. Background Technology

[0002] With the intelligent transformation of agriculture, harvesting robots are gradually replacing traditional manual harvesting. However, in complex open orchard environments, existing apple harvesting robots mostly adopt a single rigid or flexible robotic arm structure, which has the following problems: (1) Rigid robotic arms have high precision but insufficient flexibility, making it difficult to harvest fruits that are heavily obscured by branches and leaves or close to the inner side of the branches and trunks; (2) Single robotic arms have low harvesting efficiency and are difficult to cover densely distributed fruit areas inside and outside the entire fruit tree; (3) Rigid robotic arms rely on high-precision positioning when avoiding obstacles, and the vibration generated during movement will cause the end effector to drift; (4) The environmental perception system is singular, the availability of GPS signals in densely planted orchards is low, single sensor positioning is easily affected by signal interference in complex orchard terrain, and the positioning error of relying solely on IMU is relatively high. Therefore, it is urgent to develop a high-efficiency and intelligent apple harvesting robot system to improve the operating performance of harvesting robots in complex environments. Summary of the Invention

[0003] The purpose of this invention is to address the problems existing in the prior art by providing a rigid-flexible heterogeneous four-arm intelligent apple-picking robot system.

[0004] The technical solution to achieve the purpose of this invention is as follows: an intelligent apple-picking robot system with a rigid-flexible heterogeneous robotic arm working collaboratively, comprising: a mobile platform, and a picking mechanism, a control system, and a multimodal sensor system installed on the mobile platform; the picking mechanism achieves division of labor in picking through the collaboration of a rigid arm and a flexible arm, including a rigid robotic arm component and a flexible robotic arm component, which are heterogeneous and complementary in structure, driving method, and operating characteristics; the control system includes a power supply box and a control box for powering and controlling the system; the multimodal sensor system is used to achieve "global-local" environmental perception, accurate environmental mapping, target positioning, and monitoring of the robotic arm's end effector status.

[0005] Furthermore, the mobile platform includes a front housing, a rear housing, a support plate connecting the front and rear housings, and a tracked traveling device located at the bottom of the front and rear housings, wherein the tracked traveling device has a ground clearance of ≥500mm.

[0006] Furthermore, an ultrasonic ranging sensor is installed at the front end of the supporting plate, i.e., at the front end of the mobile platform in the direction of travel, to enable autonomous obstacle avoidance. An embedded picking and collecting box is provided in the middle of the supporting plate, and an infrared sensor is installed at the opening of the picking and collecting box to detect whether the collecting box is full.

[0007] Furthermore, the rigid robotic arm assembly includes multiple sets of six-degree-of-freedom rigid robotic arms, mounted on both sides of the front end of the mobile platform, and each driven by an independent motor; the maximum extension radius of the six-degree-of-freedom rigid robotic arm is ≥0.85m, and the end-effector load range is 3kg-5kg; the end effector of the six-degree-of-freedom rigid robotic arm integrates a rigid gripper and a depth camera; a film pressure sensor is installed on the gripping surface of the rigid gripper, and the depth camera is mounted on a camera bracket, which is fixed to the end of the six-degree-of-freedom rigid robotic arm by a flange; the robotic arm body is equipped with a position sensor, a limit switch, and a force-sensitive resistor collision sensor. When the force-sensitive resistor collision sensor detects a preset collision force that lasts for a preset duration or longer, the six-degree-of-freedom rigid robotic arm immediately performs an emergency retraction action, forming a collision protection mechanism for the robotic arm.

[0008] Furthermore, the flexible robotic arm assembly includes multiple sets of pneumatically driven flexible robotic arms, which are installed on both sides of the rear end of the mobile platform and are driven by independent motors. The maximum extension length of the pneumatically driven flexible robotic arm is ≥0.9m, and the end effector is ≥0.8kg. The end effector of the pneumatically driven flexible robotic arm integrates a depth camera and a flexible gripper, and the gripping surface of the flexible gripper is equipped with a film pressure sensor.

[0009] Furthermore, the multimodal sensor system includes: a "global-local" vision sensor system consisting of a binocular camera fixed to the front end of the harvesting platform and a depth camera that moves with the end of the robotic arm; and a tightly coupled positioning system including a lidar, an inertial navigation module (IMU), and a differential GNSS-RTK module.

[0010] Among them, the binocular camera provides a wide-range global scene perception, while the depth camera enables precise local positioning of the target.

[0011] The tightly coupled positioning system is used to obtain the optimal pose of the picking robot. The lidar serves as the main sensor to construct a three-dimensional point cloud map of the orchard and estimate the pose of the picking robot. The inertial navigation module (IMU) and the differential GNSS-RTK module serve as auxiliary sensors to improve the mapping and positioning accuracy and correct the lidar error.

[0012] Furthermore, the control system is rigidly mounted below the support plate of the mobile platform and located in the longitudinal space formed between the front and rear housings. It includes a power supply box and a control box separated by a partition. The power supply box integrates a power system to provide stable power at different voltage levels to the various execution units and sensors of the entire harvesting robot system. The control box integrates a core control unit to receive data from various sensors, achieve multi-sensor data fusion, plan paths, and generate control commands according to a dynamic priority scheduling mechanism to coordinate the movements of the multiple joints and end effectors of the rigid and flexible robotic arm components, so as to achieve collaborative harvesting by the rigid and flexible arms.

[0013] Furthermore, the step of generating control commands based on a dynamic priority scheduling mechanism to coordinate the operation of the rigid and flexible robotic arm components, thereby achieving collaborative harvesting between the rigid and flexible arms, specifically includes:

[0014] (1) Apple detection and positioning;

[0015] Based on deep learning visual models, the binocular image stream from the binocular camera and the depth image stream from the depth camera are processed to achieve accurate detection and spatial localization of the target apple.

[0016] (2) Target occlusion rate calculation;

[0017] The apple mask is extracted using semantic segmentation technology, and the percentage of obscured pixels is calculated.

[0018] Occlusion rate = 1 - (Number of visible pixels / Total number of pixels)

[0019] (3) Robotic arm reachability score;

[0020] Calculate the reachability of the robotic arm's current pose to the target apple's position. A reachability score is 1, and an unreachable score is 0.

[0021] (4) Dynamic priority generation;

[0022] Construct a target priority scoring model to calculate the apple priority score for each target:

[0023] Target priority score = [60% × (1 - occlusion rate) + 40% × robotic arm reachability score] × robotic arm reachability score

[0024] (5) Based on the target priority score, sort all target apples in descending order to generate a dynamic picking priority list;

[0025] (6) Based on the dynamic picking priority list, generate control instructions to coordinate the operation of the rigid robotic arm component and the flexible robotic arm component: For target apples with a target priority score greater than the first preset threshold, generate the first control instruction to control the rigid robotic arm component to pick the target apples; for target apples with a target priority score less than the first preset threshold but greater than the second preset threshold, generate the second control instruction to control the flexible robotic arm component to pick the target apples, wherein the second preset threshold is less than the first preset threshold; for target apples with a target priority score less than the second preset threshold, generate the third control instruction to not pick them.

[0026] Furthermore, the deep learning vision model is an improved YOLOv11 model, specifically including: based on the YOLOv11 network architecture, embedding a hybrid local attention mechanism, i.e., MLCA, into the backbone network; using a grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure to perform lightweight reconstruction of the neck; designing a group normalized shared-scale detection head, and achieving multi-scale occlusion feature fusion through a shared normalization layer;

[0027] Furthermore, the embedding of the hybrid local attention mechanism (MLCA) in the backbone network specifically involves introducing the MLCA mechanism into the backbone network and embedding it between the original C3k2 layers to form a stacked feature enhancement structure.

[0028] The lightweight reconstruction of the neck using grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure is specifically: replacing the standard convolutional layer with the GSConv module and constructing the VoV-GSCSP cross-stage pathway to replace the traditional CSP module.

[0029] The design of the group-normalized shared-scaled detection head includes three stages:

[0030] Group normalization layers (GN) are used on standard convolutions and depthwise separable convolutions in the detection branches;

[0031] The design incorporates a shared decoupling module. Feature maps of different scales are passed through a 1×1 convolutional module with group normalization layers to unify the feature map dimensions. Subsequently, the independent task branches in the original detection head of the YOLOv11 network architecture are merged into two parallel shared convolutional modules. One module is a concatenation of two 3×3 group normalized standard convolutions, and the other is a concatenation of two 3×3 group normalized depthwise separable convolutions and a 1×1 group normalized standard convolution. The output is then decoupled through two 1×1 convolutional layers, restoring the decoupling between the classification and localization tasks.

[0032] Multiple scale-up layers are integrated into the localization prediction branch to achieve scaling of features at different scales.

[0033] Compared with the prior art, the significant advantages of this invention are:

[0034] (1) Improved Harvesting Coverage and Flexibility: Existing technologies mostly use single rigid or flexible robotic arms. Rigid arms have poor flexibility, while flexible arms have limited load capacity, making it difficult to harvest fruits from different locations on a single fruit tree. This invention innovatively adopts a rigid-flexible heterogeneous robotic arm structure. A six-degree-of-freedom rigid robotic arm is installed on both sides of the front end of the mobile platform. Its maximum extension radius is ≥0.85m, and the maximum end-effector load range is 3-5kg. It is driven by an independent motor, and the end effector integrates a rigid gripper, force sensor, and depth camera. With its high precision and large load capacity, it is responsible for harvesting fruits that are close to the mobile platform and have less obstruction. A pneumatically driven flexible robotic arm is installed on both sides of the rear end of the mobile platform. Its maximum extension length is ≥0.9m, the maximum bending angle in one direction is >200°, and the maximum end-effector load capacity is ≥0.8kg. It adopts a three-chamber parallel pneumatic structure and dynamically adjusts the air pressure from 0.1 to 0.5MPa to achieve closed-loop control of deformation. With high flexibility, it can reach deep into areas obstructed by branches and leaves to harvest fruits that are far away and severely obstructed. The two work together to achieve full coverage of picking on a single fruit tree, effectively solving the problem of missed picking due to shading on the inside of the fruit tree, and greatly improving the picking coverage and flexibility.

[0035] (2) Significantly improved harvesting efficiency: Traditional single-arm harvesting methods are inefficient when dealing with densely distributed fruit areas on both the inside and outside of the entire fruit tree. This invention utilizes multi-arm collaborative operation to harvest multiple target fruits simultaneously, significantly increasing the harvesting quantity per unit time. Simultaneously, it achieves accurate apple detection and spatial localization by processing binocular image streams based on a deep learning visual model, extracts apple masks using semantic segmentation technology to calculate occlusion rates, and evaluates target accessibility using an inverse kinematics solver based on the current pose of the robotic arm. A dynamic priority scoring model with a fusion weight of 60% occlusion rate and 40% accessibility is constructed to prioritize high-scoring targets and avoid ineffective actions, resulting in a significantly improved harvesting efficiency compared to a single robotic arm.

[0036] (3) Enhanced Positioning Stability and Accuracy: Existing robotic arms are prone to end-effector positioning drift under travel vibrations, and single-sensor positioning in orchard environments is easily interfered with. This invention employs a tightly coupled positioning system in a multi-sensor fusion system. This system uses a lidar as the primary sensor to construct a 3D point cloud map of the orchard and estimate the harvesting robot's pose. An inertial navigation module (IMU) and a differential GNSS-RTK module serve as auxiliary sensors to correct errors. The multi-sensor cooperation effectively suppresses end-effector positioning drift caused by travel vibrations, improving positioning stability and accuracy. By installing a binocular camera at the front of the mobile platform's support plate, a wide-area global scene perception is achieved, while depth cameras at the ends of the four robotic arms enable precise local positioning of the target. This constitutes a "global-local" visual sensor system, enabling accurate identification and positioning of the target fruit, providing reliable assurance for efficient harvesting. Furthermore, this multi-sensor fusion strategy effectively overcomes signal interference in GPS-denied environments. In densely planted orchards and adverse weather conditions, the collaborative operation of lidar, IMU, and differential GNSS-RTK modules achieves centimeter-level self-positioning accuracy across all weather conditions.

[0037] (4) The apple picking robot system based on the collaborative operation of four rigid-flexible heterogeneous robotic arms proposed in this invention improves picking efficiency in many ways by organically combining a mobile platform, picking mechanism, control system, and multi-sensor fusion system. The rigid-flexible heterogeneous four robotic arms, combined with a dynamic picking priority scheduling mechanism, achieve efficient division of labor for fruit picking in complex environments; the multi-sensor fusion system solves the problems of positioning drift and signal interference; the tracked mobile platform, in conjunction with various sensors, enhances the robot's environmental adaptability and autonomous operation capabilities. The overall solution effectively solves the problems of missed picking due to shading inside the fruit tree, low efficiency of a single robotic arm, positioning drift of the robotic arm, and low efficiency of multi-sensor collaboration in the prior art, providing a technical solution for efficient and intelligent apple picking operations.

[0038] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the overall structure of an intelligent apple-picking robot system with a rigid-flexible heterogeneous robotic arm working collaboratively in one embodiment.

[0040] Figure 2 This is a flowchart of the dynamic picking priority algorithm of an apple picking robot system in one embodiment.

[0041] Figure 3 This is a flowchart illustrating the harvesting process of an apple-picking robot system in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0044] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0045] In one embodiment, combined Figure 1 This invention provides an intelligent apple-picking robot system with a rigid-flexible heterogeneous robotic arm working collaboratively. The system includes a mobile platform, a picking mechanism, a control system, and a multimodal sensor system mounted on the mobile platform. The picking mechanism achieves division of labor in picking through the collaboration of a rigid arm and a flexible arm, comprising a rigid robotic arm component and a flexible robotic arm component, which are heterogeneous and complementary in structure, drive method, and operational characteristics. The control system includes a power supply box and a control box for power supply and control of the system. The multimodal sensor system is used to achieve "global-local" environmental perception, precise environmental mapping, target localization, and monitoring of the robotic arm's end effector status.

[0046] Furthermore, in one embodiment, the mobile platform includes a front box, a rear box, a support plate 4 connecting the front and rear boxes, and a tracked traveling device 7 located at the bottom of the front and rear boxes. The tracked traveling device has a ground clearance of ≥500mm to adapt to the rugged terrain of the orchard.

[0047] More preferably, in some embodiments, an ultrasonic ranging sensor 8 is installed at the front end of the supporting plate, i.e., at the front end of the mobile platform in the direction of travel, for autonomous obstacle avoidance. An embedded picking and collecting box 3 is provided in the middle of the supporting plate, and an infrared sensor 6 is installed at the opening of the picking and collecting box for detecting whether the collecting box is full.

[0048] Furthermore, in one embodiment, the rigid robotic arm assembly includes multiple sets of six-degree-of-freedom rigid robotic arms, mounted on both sides of the front end of the mobile platform, and each driven by an independent motor; the maximum extension radius of the six-degree-of-freedom rigid robotic arm is ≥0.85m, and the end-effector load range is 3kg-5kg; the end effector of the six-degree-of-freedom rigid robotic arm integrates a rigid gripper 9 and a depth camera 11; a film pressure sensor 15 is installed on the gripping surface of the rigid gripper 9, and the depth camera 11 is mounted on a camera bracket 12, which is fixed to the end of the six-degree-of-freedom rigid robotic arm 1 by a flange 13; the robotic arm body is equipped with a position sensor, a limit switch, and a force-sensitive resistor collision sensor 10 to ensure that the multiple robotic arms move within a safe working range. When the force-sensitive resistor collision sensor detects a preset collision force (5N) that lasts for a preset duration (≥10ms), the six-degree-of-freedom rigid robotic arm immediately performs an emergency retraction action to form a robotic arm collision protection.

[0049] Preferably, the rigid robotic arm assembly includes two sets of six-degree-of-freedom rigid robotic arms 1, symmetrically mounted on both sides of the front end of the mobile platform.

[0050] Preferably, the six-degree-of-freedom rigid robotic arm is the Universal Robots UR5e collaborative robot arm, with a maximum extension radius of 0.85m, an end-effector load capacity of 5kg, and a repeatability of ±0.03mm.

[0051] Preferably, the rigid gripper 9 is an EGP 40 motorized parallel two-finger gripper with a maximum opening distance of 80mm and a maximum gripping force of 50N. The force-sensitive collision sensor 10 is an OnRobot HEX six-dimensional torque sensor with a range of Fx, Fy, Fz: ±100N, Tx, Ty, Tz: ±5Nm. The depth camera 11 is a RealSense D455 with a resolution of 1280x720@30fps, a depth measurement range of 0.4m-6.0m, and a field of view (FOV) of 86°(H)×57°(V).

[0052] Furthermore, in one embodiment, the flexible robotic arm assembly includes multiple sets of pneumatically driven flexible robotic arms, installed on both sides of the rear end of the mobile platform, and each driven by an independent motor; the maximum extension length of the pneumatically driven flexible robotic arm is ≥0.9m, the end effector is ≥0.8kg, and the end effector of the pneumatically driven flexible robotic arm integrates a depth camera and a flexible gripper 14, with a film pressure sensor installed on the gripping surface of the flexible gripper.

[0053] Preferably, the flexible robotic arm assembly includes two sets of pneumatically driven flexible robotic arms 2, which are symmetrically installed on both sides of the rear end of the mobile platform.

[0054] Preferably, the pneumatically driven flexible robotic arm 2 uses a Festo DMSP series pneumatic muscle mechanism, with a maximum extension length of approximately 0.9m, a maximum bending angle in one direction >200°, and a maximum end-effector load capacity of 0.8kg. It employs a three-chamber parallel air circuit, with each chamber having a diameter of approximately 30mm and a length of approximately 850mm. The working air pressure range is controlled within 0.1-0.5MPa, and each chamber is controlled by an independent solenoid valve, achieving an air pressure control accuracy of ±0.01MPa. The flexible gripper uses a soft silicone rubber biomimetic three-finger flexible gripper, exhibiting excellent flexibility and adaptability. The membrane pressure sensor 15 uses a FlexiForce A201 thin-film pressure sensor with a range of 0-100N.

[0055] Furthermore, in one embodiment, the multimodal sensor system includes: a "global-local" vision sensor system consisting of a binocular camera 16 fixed to the front end of the harvesting platform and a depth camera that moves with the end of the robotic arm; and a tightly coupled positioning system including a lidar 17, an inertial navigation module IMU 18 and a differential GNSS-RTK module.

[0056] Among them, the binocular camera provides a wide-range global scene perception, while the depth camera enables precise local positioning of the target.

[0057] The tightly coupled positioning system is used to obtain the optimal pose of the picking robot. The lidar serves as the main sensor to construct a three-dimensional point cloud map of the orchard and estimate the pose of the picking robot. The inertial navigation module (IMU) and the differential GNSS-RTK module serve as auxiliary sensors to improve the mapping and positioning accuracy and correct the lidar error.

[0058] Here, the binocular camera 16 provides global scene perception with a 120° horizontal field of view in an eye-on-hand mode, while the depth camera 11 at the end of the robotic arm achieves local positioning of the target in an eye-on-hand mode.

[0059] Preferably, in some embodiments, the tightly coupled positioning system is used to obtain the optimal pose of the harvesting robot, and the specific process includes:

[0060] (1) Construct a tightly coupled LiDAR-INS-GPS data model, and use this model to fuse LiDAR 3D point cloud data, IMU data, and GPS satellite positioning data to obtain the fused pose of the harvesting robot; the tightly coupled LiDAR-INS-GPS data model is as follows:

[0061] min T ∑ρ(||z lidar -T·ρ'|| 2 )+λ||q IMU -q RTK ||

[0062] In the formula, ρ is the Huber loss function for suppressing outlier interference, T is the transformation matrix, ρ' is the voxelization feature extraction operation of the lidar point cloud, and q IMU q RTK These are inertial navigation IMU data and GPS satellite positioning data, respectively. lidar The data is LiDAR point cloud data, where λ is the weighting coefficient and z is the weighting coefficient. lidar -T·ρ represents the environmental constraint residual between the LiDAR data after pose transformation and the fusion model;

[0063] (2) For inertial navigation IMU data, IMU pose estimation is performed based on historical inertial navigation IMU data. Then, it is determined whether the satellite signal missing time is less than a preset threshold. If so, IMU pose optimization is performed. Otherwise, IMU pose estimation is performed based on historical inertial navigation IMU data.

[0064] (3) For GPS satellite positioning data, determine whether the number of visible GPS satellites is higher than the preset threshold. If so, perform differential RTK pose calculation on the GPS satellite positioning data. Otherwise, continue to collect GPS satellite positioning data until the number of visible GPS satellites meets the preset threshold.

[0065] (4) The fused pose, the optimized IMU pose, and the pose calculated by differential RTK are respectively associated with the three sensors: lidar, IMU, and RTK; the following descriptions are simplified using the three sensors: lidar, IMU, and RTK.

[0066] Establish a similarity vector S = [s] for quantifying the differences in poses among LiDAR, IMU, and RTK. 12 ,s 13 ,s 23 ], s 12 s 13 s 23 These represent the pairwise similarity between the three poses;

[0067] (5) Assign optimal weights to the three poses using subjective and objective weighting methods as in (4), specifically including:

[0068] (5-1) Preset the subjective weights of lidar, IMU, and RTK respectively.

[0069] (5-2) Real-time calculation of objective weights for lidar, IMU, and RTK:

[0070]

[0071] in,

[0072] or

[0073] In the formula, i,j=1,2,3 correspond to the three sensors: lidar, IMU, and RTK, respectively. Let σ be the mean similarity between sensor i, sensor j and the other two sensors, respectively. i σ j Let σ′ be the observation variances of sensor i and sensor j, respectively; i σ′ j These are the observation variance correction terms for sensor i and sensor j, respectively; is the objective weight of sensor i; k is the adjustment coefficient, which is adaptively adjusted according to the environment;

[0074] (5-3) Solve for the optimal weights using the divergence minimization algorithm;

[0075] By the following formula:

[0076]

[0077] Find the optimal weight And satisfy

[0078] (6) Calculate the optimal pose P based on the optimal weights final :

[0079]

[0080] In the formula, P Lidar P IMU P RTK These are the fused pose, the optimized IMU pose, and the pose calculated by differential RTK, respectively.

[0081] (7) Determine whether the target difference ΔW satisfies the condition. If so, then P final The optimal pose of the picking robot is transmitted to the output layer, and the process ends; otherwise, output correction is performed and (8) is executed.

[0082] (8) Determine whether RTK is invalid based on DS evidence theory. If invalid, enforce it. And recalculate the optimal pose P final Then return to execute (7).

[0083] Furthermore, in one embodiment, the control system is rigidly mounted below the support plate of the mobile platform and located in the longitudinal space formed between the front and rear housings. It includes a power supply box and a control box 5 separated by a partition. The power supply box integrates a power system to provide stable power at different voltage levels to the various execution units and sensors of the entire harvesting robot system. The control box integrates a core control unit to receive data from various sensors, achieve multi-sensor data fusion, plan paths, and generate control commands according to a dynamic priority scheduling mechanism to coordinate the movements of the multiple joints and end effectors of the rigid and flexible robotic arm components, so as to achieve collaborative harvesting by the rigid and flexible arms.

[0084] Preferably, in some embodiments, the power supply box integrates a multi-voltage power supply module to provide the execution unit with a 24V / 400W motor drive voltage and a 5V sensor operating voltage; the control box is equipped with an industrial control computer, which realizes multi-arm collaborative control operation based on the ROS (Robot Operating System) architecture, and receives multi-sensor data at a frequency of 1kHz through the CAN bus protocol and coordinates the actions of multiple robotic arms.

[0085] Preferably, in some embodiments, the generation of control commands based on a dynamic priority scheduling mechanism to coordinate the operation of the rigid and flexible robotic arm components is implemented to achieve collaborative harvesting between the rigid and flexible arms. Figure 2 Specifically, it includes:

[0086] (1) Apple detection and positioning;

[0087] Based on deep learning visual models, the binocular image stream from the binocular camera and the depth image stream from the depth camera are processed to achieve accurate detection and spatial localization of the target apple.

[0088] (2) Target occlusion rate calculation;

[0089] The apple mask is extracted using semantic segmentation technology, and the percentage of obscured pixels is calculated.

[0090] Occlusion rate = 1 - (Number of visible pixels / Total number of pixels)

[0091] (3) Robotic arm reachability score;

[0092] Calculate the reachability of the robotic arm's current pose to the target apple's position. A reachability score is 1, and an unreachable score is 0.

[0093] (4) Dynamic priority generation;

[0094] Construct a target priority scoring model to calculate the apple priority score for each target:

[0095] Target priority score = [60% × (1 - occlusion rate) + 40% × robotic arm reachability score] × robotic arm reachability score

[0096] (5) Based on the target priority score, sort all target apples in descending order to generate a dynamic picking priority list;

[0097] (6) Based on the dynamic picking priority list, generate control instructions to coordinate the operation of the rigid robotic arm component and the flexible robotic arm component: For target apples with a target priority score greater than the first preset threshold, generate the first control instruction to control the rigid robotic arm component to pick the target apples; for target apples with a target priority score less than the first preset threshold but greater than the second preset threshold, generate the second control instruction to control the flexible robotic arm component to pick the target apples, wherein the second preset threshold is less than the first preset threshold; for target apples with a target priority score less than the second preset threshold, generate the third control instruction to not pick them.

[0098] Preferably, the first preset threshold is 0.82 and the second preset threshold is 0.4.

[0099] Preferably, in some embodiments, the deep learning visual model is an improved YOLOv11 model, specifically including: based on the YOLOv11 network architecture, embedding a hybrid local attention mechanism, i.e., MLCA, into the backbone network; using a grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure to perform lightweight reconstruction of the neck; designing a group normalized shared-scale detection head, and achieving multi-scale occlusion feature fusion through a shared normalization layer.

[0100] The embedding of the hybrid local attention mechanism (MLCA) in the backbone network specifically involves introducing the MLCA mechanism into the backbone network and embedding it between the original C3k2 layers to form a stacked feature enhancement structure.

[0101] The lightweight reconstruction of the neck using grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure is specifically: replacing the standard convolutional layer with the GSConv module and constructing the VoV-GSCSP cross-stage pathway to replace the traditional CSP module.

[0102] The design of the group-normalized shared-scaled detection head includes three stages:

[0103] Group normalization layers (GN) are used on standard convolutions and depthwise separable convolutions in the detection branches;

[0104] The design incorporates a shared decoupling module. Feature maps of different scales are passed through a 1×1 convolutional module with group normalization layers to unify the feature map dimensions. Subsequently, the independent task branches in the original detection head of the YOLOv11 network architecture are merged into two parallel shared convolutional modules. One module is a concatenation of two 3×3 group normalized standard convolutions, and the other is a concatenation of two 3×3 group normalized depthwise separable convolutions and a 1×1 group normalized standard convolution. The output is then decoupled through two 1×1 convolutional layers, restoring the decoupling between the classification and localization tasks.

[0105] Multiple scale layers are integrated into the localization prediction branch to achieve scaling of features at different scales.

[0106] Combination Figure 3 The apple picking robot system of this invention performs the picking operation according to the following steps:

[0107] (1) System initialization: The power supply box starts the power supply module to supply power to each sensor and actuator. The picking system performs a self-test process. The motors of each joint of the rigid arm perform the return action in sequence. The pneumatic drive unit of the flexible arm performs a pressure pipeline sealing test.

[0108] (2) Map building and global path planning: The lidar 17 and the inertial navigation module 18 collect orchard data and build a three-dimensional point cloud map of the orchard based on visual SLAM. The multi-source positioning data is integrated to establish the global coordinate system of the orchard and identify the row and column coordinates of the fruit trees. The global path planning of the robot is generated. That is, after the robot enters the orchard, it first circles around a single fruit tree 360° to pick the target apples in the field of vision, and then moves to the next fruit tree to circle around and pick.

[0109] (3) Target detection and priority scheduling: The picking robot walks along the map trajectory and global path planning. The binocular camera 16 uses the improved YOLOv11 model to process the binocular image stream in real time and detect the apple target in the field of view. When the target appears, the picking robot stops, the binocular camera 16 saves the current frame image, and calculates the spatial coordinates of the target fruit by combining the point cloud data of the depth camera 11 and the scanning results of the lidar 17. The robot generates a dynamic picking priority list by using semantic segmentation technology to calculate the occlusion rate, inverse kinematics to solve the reachability of the robotic arm, and weight calculation of priority scores.

[0110] (4) Division of Labor for Picking and Autonomous Obstacle Avoidance: Based on a dynamic picking priority list generated according to the target location and degree of occlusion, the rigid-flexible heterogeneous four robotic arms implement division of labor for picking. When a robotic arm cannot reach the target, its priority score is 0. When the target is reachable, if the occlusion rate is ≤30%, the rigid robotic arm will pick the target. That is, the rigid arm handles apple targets with a score ≥0.82, and the flexible arm handles targets with a score between 0.4 and 0.82. Targets with a score lower than 0.4 are not picked. High-scoring targets are prioritized. When the scores are the same, the system adopts a random selection strategy to perform the picking operation. During the movement of the picking robot, the ultrasonic ranging sensor 8 detects obstacles in real time and feeds back the distance to the obstacle to the control system. The detection distance is 0.2m-5m. When the distance is less than 1 meter, the picking robot stops. If the obstacle disappears within 8 seconds, the picking robot continues to walk along the original route. Otherwise, it turns left or right in place to bypass the obstacle and continues to walk.

[0111] (5) Fruit collection and cyclic operation: After the robotic arm grabs the apple, it moves to the top of the collection box 3, places the apple in the box, and repeats the operation until all the apples in the current field of vision are picked; the infrared sensor 6 above the collection box 3 monitors the height of the fruit pile in the collection box in real time. When the box is full, it triggers an audible and visual alarm and stops picking; the picking robot moves forward and picks according to the global path plan generated based on the orchard 3D point cloud map until the picking of the entire orchard is completed.

[0112] As a specific example, this invention is illustrated and verified. The apple-picking robot system of this invention was tested in an apple orchard with a row spacing of 4 meters and a tree spacing of 2 meters. The test platform used an AGV chassis with a maximum travel speed of 1.2 m / s, equipped with an NVIDIA Jetson AGX Orin computing unit. Test results showed that the rigid arm's average time to pick outer apples was 2.1 seconds per apple, with a repeatability accuracy of ±0.03 mm; the flexible arm's average time to pick inner apples was 3.8 seconds per apple. The collaborative operation of the rigid and flexible robotic arms achieved 100% picking coverage on a single tree, a significant 37% improvement compared to a single-arm system. Regarding the picking success rate, the average success rate of the rigid and flexible robotic arms on a single tree for each task reached 87.2%.

[0113] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An intelligent apple-picking robot system with a rigid-flexible heterogeneous robotic arm working collaboratively, comprising: A mobile platform, and a harvesting mechanism, a control system, and a multimodal sensor system installed on the mobile platform; characterized in that the harvesting mechanism achieves division of labor harvesting through the cooperation of a rigid arm and a flexible arm, including a rigid robotic arm assembly and a flexible robotic arm assembly, which are heterogeneous and complementary in structure, drive method, and operation characteristics; the control system includes a power supply box and a control box for power supply and control of the system; the multimodal sensor system is used to achieve "global-local" environmental perception, accurate environmental mapping, target positioning, and monitoring of the robotic arm end state.

2. The intelligent apple-picking robot system with rigid-flexible heterogeneous robotic arms working collaboratively according to claim 1, characterized in that, The mobile platform includes a front box, a rear box, a support plate connecting the front and rear boxes, and a tracked traveling device located at the bottom of the front and rear boxes, wherein the tracked traveling device has a ground clearance of ≥500mm.

3. The intelligent apple-picking robot system with rigid-flexible heterogeneous robotic arms working collaboratively according to claim 2, characterized in that, The front end of the support plate, which is the front end of the mobile platform in the direction of travel, is equipped with an ultrasonic ranging sensor for autonomous obstacle avoidance. The middle of the support plate is equipped with an embedded picking and collecting box, and the opening of the picking and collecting box is equipped with an infrared sensor for detecting whether the collecting box is full.

4. The intelligent apple-picking robot system with rigid-flexible heterogeneous robotic arms working collaboratively according to claim 1, characterized in that, The rigid robotic arm assembly includes multiple sets of six-degree-of-freedom rigid robotic arms, mounted on both sides of the front end of the mobile platform, each driven by an independent motor; the maximum extension radius of the six-degree-of-freedom rigid robotic arm is ≥0.85m, and the end-effector load range is 3kg-5kg; the end effector of the six-degree-of-freedom rigid robotic arm integrates a rigid gripper and a depth camera; a film pressure sensor is installed on the gripping surface of the rigid gripper, and the depth camera is mounted on a camera bracket, which is fixed to the end of the six-degree-of-freedom rigid robotic arm via a flange; the robotic arm body is equipped with a position sensor, limit switch, and force-sensitive resistor collision sensor. When the force-sensitive resistor collision sensor detects a preset collision force that lasts for a preset duration, the six-degree-of-freedom rigid robotic arm immediately performs an emergency retraction action, forming a collision protection mechanism for the robotic arm.

5. The intelligent apple-picking robot system with rigid-flexible heterogeneous robotic arms working collaboratively according to claim 1, characterized in that, The flexible robotic arm assembly includes multiple sets of pneumatically driven flexible robotic arms, which are installed on both sides of the rear end of the mobile platform and are driven by independent motors. The maximum extension length of the pneumatically driven flexible robotic arm is ≥0.9m and the end-effector load is ≥0.8kg. The end effector of the pneumatically driven flexible robotic arm integrates a depth camera and a flexible gripper. The gripping surface of the flexible gripper is equipped with a film pressure sensor.

6. The intelligent apple-picking robot system with rigid-flexible heterogeneous robotic arms working collaboratively according to claim 2, characterized in that, The multimodal sensor system includes: a "global-local" vision sensor system consisting of a binocular camera fixed to the front end of the harvesting platform and a depth camera that moves with the end of the robotic arm; and a tightly coupled positioning system including a lidar, an inertial navigation module (IMU), and a differential GNSS-RTK module. Among them, the binocular camera provides a wide-range global scene perception, while the depth camera enables precise local positioning of the target. The tightly coupled positioning system is used to obtain the optimal pose of the picking robot. The lidar serves as the main sensor to construct a three-dimensional point cloud map of the orchard and estimate the pose of the picking robot. The inertial navigation module (IMU) and the differential GNSS-RTK module serve as auxiliary sensors to improve the mapping and positioning accuracy and correct the lidar error.

7. The intelligent apple-picking robot system with rigid-flexible heterogeneous robotic arms working collaboratively according to claim 6, characterized in that, The control system is rigidly mounted below the support plate of the mobile platform and located in the longitudinal space formed between the front and rear housings. It includes a power supply box and a control box separated by a partition. The power supply box integrates a power system to provide stable power at different voltage levels to the various execution units and sensors of the entire harvesting robot system. The control box integrates a core control unit to receive data from various sensors, achieve multi-sensor data fusion, plan paths, and generate control commands according to a dynamic priority scheduling mechanism to coordinate the movements of the multiple joints and end effectors of the rigid and flexible robotic arm components, so as to achieve collaborative harvesting by the rigid and flexible arms.

8. The intelligent apple-picking robot system with rigid-flexible heterogeneous robotic arms working collaboratively according to claim 7, characterized in that, The dynamic priority scheduling mechanism generates control commands to coordinate the operation of the rigid and flexible robotic arm components, enabling collaborative harvesting between the rigid and flexible arms. Specifically, this includes: (1) Apple detection and positioning; Based on deep learning visual models, the binocular image stream from the binocular camera and the depth image stream from the depth camera are processed to achieve accurate detection and spatial localization of the target apple. (2) Target occlusion rate calculation; The apple mask is extracted using semantic segmentation technology, and the percentage of obscured pixels is calculated. Occlusion rate = 1 - (Number of visible pixels / Total number of pixels) (3) Robotic arm reachability score; Calculate the reachability of the robotic arm's current pose to the target apple's position. A reachability score is 1, and an unreachable score is 0. (4) Dynamic priority generation; Construct a target priority scoring model to calculate the apple priority score for each target: Target priority score = [60% × (1 - occlusion rate) + 40% × robotic arm reachability score] × robotic arm reachability score (5) Based on the target priority score, sort all target apples in descending order to generate a dynamic picking priority list; (6) Based on the dynamic picking priority list, generate control instructions to coordinate the operation of the rigid robotic arm component and the flexible robotic arm component: For target apples with a target priority score greater than the first preset threshold, generate the first control instruction to control the rigid robotic arm component to pick the target apples; for target apples with a target priority score less than the first preset threshold but greater than the second preset threshold, generate the second control instruction to control the flexible robotic arm component to pick the target apples, wherein the second preset threshold is less than the first preset threshold; for target apples with a target priority score less than the second preset threshold, generate the third control instruction to not pick them.

9. The intelligent apple-picking robot system with rigid-flexible heterogeneous robotic arms working collaboratively according to claim 8, characterized in that, The deep learning visual model is an improved YOLOv11 model, specifically including: based on the YOLOv11 network architecture, embedding a hybrid local attention mechanism (MLCA) in the backbone network; using a grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure to perform lightweight reconstruction of the neck; designing a group normalized shared-scale detection head, and achieving multi-scale occlusion feature fusion through a shared normalization layer.

10. The intelligent apple-picking robot system with rigid-flexible heterogeneous robotic arms working collaboratively according to claim 9, characterized in that, The embedding of the hybrid local attention mechanism (MLCA) in the backbone network specifically involves introducing the MLCA mechanism into the backbone network and embedding it between the original C3k2 layers to form a stacked feature enhancement structure. The lightweight reconstruction of the neck using grouped sparse convolution GSConv and VoV-GSCSP cross-stage structure is specifically: replacing the standard convolutional layer with the GSConv module and constructing the VoV-GSCSP cross-stage pathway to replace the traditional CSP module. The design of the group-normalized shared-scaled detection head includes three stages: Group normalization layers (GN) are used on standard convolutions and depthwise separable convolutions in the detection branches; The design incorporates a shared decoupling module. Feature maps of different scales are passed through a 1×1 convolutional module with group normalization layers to unify the feature map dimensions. Subsequently, the independent task branches in the original detection head of the YOLOv11 network architecture are merged into two parallel shared convolutional modules. One module is a concatenation of two 3×3 group normalized standard convolutions, and the other module is a concatenation of two 3×3 group normalized depthwise separable convolutions and a 1×1 group normalized standard convolution. The output is then decoupled through two 1×1 convolutional layers, restoring the decoupling between the classification and localization tasks. Multiple scale layers are integrated into the localization prediction branch to achieve scaling of features at different scales.

Images