A multi-modal perception and force-position collaborative control based lossless fruit picking manipulator

Abstract

The application discloses a nondestructive fruit picking manipulator based on multi-modal perception and force-position collaborative control. In view of the problems of a large visual blind area and easy damage to fruit flesh of the existing equipment, the system is composed of an executing mechanism, a visual positioning component, a dual-mode sensing component and a main control system. The executing mechanism comprises a telescopic arm, a rotating wrist and a V-shaped flexible gripper; the visual component adopts an Eye-in-Hand configuration front-mounted RGB-D depth camera, and realizes high-precision three-dimensional position solution and maturity identification of fruits under complex tree crowns in combination with a neural network; the dual-mode sensing component is composed of a position sensor and a pressure sensor. During operation, the main control system plans a trajectory based on a visual coordinate, and guides the gripper to approach through monitoring of displacement of the position sensor; during grabbing, the pressure sensor feeds back contact force in real time, and triggers a force limiting self-locking when reaching a nondestructive safety threshold. The application constructs a triple control system of "visual guidance-displacement closed loop-torque limiting", effectively eliminates visual dead angles, realizes adaptive flexible nondestructive clamping, and greatly reduces picking damage rate.

Description

Technical Field

[0001] This invention belongs to the field of agricultural equipment automation and agricultural robot technology, specifically relating to an intelligent non-destructive fruit picking robot that integrates visual image recognition technology and a dual-mode sensing force-position collaborative control mechanism. Background Technology

[0002] Fruit harvesting is the most time-consuming and labor-intensive part of the orchard production chain. With the shortage of agricultural labor and the sharp rise in labor costs, the development and popularization of automated harvesting robots has become an urgent need for the development of modern agriculture.

[0003] However, natural orchard environments are typically unstructured, and existing robotic arms exhibit serious bottlenecks in practical applications. First, traditional vision systems often employ a fixed global viewpoint from the base, resulting in large blind spots when facing dense tree canopies. They are also susceptible to interference from strong sunlight, backlighting, or dappled shadows, leading to low target recognition rates and large spatial positioning errors. Second, most fruits, such as peaches, apples, and citrus, have fragile skins and are easily damaged. Existing grasping actuators lack high-precision force feedback and flexible adaptive control, making them prone to insufficient grasping force leading to fruit slippage, or excessive grasping force causing mechanical damage to the fruit flesh, severely impacting the marketability and shelf life of harvested fruit.

[0004] Given the current limitations of technology in terms of visual positioning accuracy and non-destructive gripping control, there is an urgent need to develop a new type of harvesting robot that can penetrate deep into the canopy for precise target capture and has anthropomorphic flexible grasping capabilities. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a non-destructive fruit picking robot based on multimodal perception and force-position coordinated control. It achieves high-precision three-dimensional positioning in complex environments through front-end hand-eye vision and adaptive closed-loop control of grasping displacement and clamping force through rear dual-mode sensors, thereby achieving efficient and non-destructive picking.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A non-destructive fruit-harvesting robot based on multimodal perception and force-position coordinated control includes a robot actuator, a visual recognition and spatial positioning component, a dual-mode sensing feedback component, and a central control system. The visual recognition and spatial positioning component is positioned at the end of the robot actuator for image acquisition, maturity assessment, and three-dimensional pose calculation of the target fruit. The dual-mode sensing feedback component is located at the rear of the robot and at the transmission nodes to monitor telescopic displacement and clamping force. The central control system is responsible for coordinating data calculation and closed-loop control.

[0008] Furthermore, the robotic arm actuator includes a base, a telescopic arm, a rotating wrist, and an externally clamped V-shaped flexible gripper. The telescopic arm provides axial feed for deep penetration into the tree canopy; the rotating wrist provides rotational freedom for attitude adjustment; and the V-shaped flexible gripper has a highly elastic cushioning pad inside, providing embracing for irregularly shaped fruits.

[0009] Furthermore, the visual recognition and spatial positioning component adopts a high-definition RGB-D depth camera and is attached to the side of the hand to form an eye-in-hand system; the edge computing module with embedded lightweight instance segmentation neural network realizes anti-interference recognition of natural ambient light and foliage occlusion, and outputs target coordinates (X, Y, Z).

[0010] Furthermore, the dual-mode sensing feedback component consists of a position sensor and a pressure sensor. These two sensors respectively construct the "position closed loop" and "torque closed loop" of the system's motion, triggering a force-limiting self-locking mechanism when the gripper clamps the fruit to a preset non-damaging force threshold.

[0011] Compared with the prior art, the present invention has the following significant advantages:

[0012] Eliminating visual blind spots and achieving extremely high positioning accuracy: Employing a hand-eye vision-oriented front-facing configuration, the camera extends deep into the fruit tree along with the telescopic arm, avoiding obstruction from external branches and leaves. Combined with a deep learning-based instance segmentation algorithm, it achieves high-precision 3D reconstruction of the geometric centroid of the target fruit and intelligent identification of its maturity.

[0013] Dual-mode force-position coordination achieves absolutely damage-free harvesting: An innovative triple closed-loop control system is introduced, combining "displacement monitoring for approach + pressure monitoring for clamping." Position sensors ensure millimeter-level spatial trajectory tracking; pressure sensors provide high-frequency tactile feedback, stopping the drive as soon as the contact force reaches a set safety threshold, completely eliminating excessive pressure damage to the fruit pulp from rigid machinery.

[0014] The mechanism combines rigidity and flexibility, offering strong adaptability: the long-stroke telescopic arm ensures sufficient working depth, the rotating wrist provides a flexible gripping angle, and the V-shaped two-finger gripper, combined with a highly elastic biomimetic layer, can firmly hold fruits of different diameters, meeting the diverse field harvesting needs of orchards. Attached Figure Description

[0015] Figure 1 This is a side view of the harvesting robot provided in an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of the isometric three-dimensional structure of the harvesting robot provided in an embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram of the front-end structure of the harvesting robot provided in an embodiment of the present invention;

[0018] The following are the reference numerals: 1. Robotic arm actuator; 11. Base; 12. Telescopic arm; 13. Rotating wrist; 14. Flexible gripper; 2. Visual recognition and spatial positioning component; 21. RGB-D depth camera; 22. Edge computing module; 3. Dual-mode sensing feedback component; 31. Position sensor; 32. Pressure sensor. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and preferred embodiments.

[0020] Reference Figures 1 to 3 This invention provides a non-destructive fruit-picking robot based on multimodal sensing and force-position coordinated control. The device is spatially divided into a front-end sensing and execution area and a rear-end power control area.

[0021] Regarding the actuator, the main body of the robotic arm actuator 1 is a long-stroke telescopic arm 12. The outer cylinder of this telescopic arm 12 is fixed to the rear base 11, and the internal piston rod is hydraulically or electrically driven to extend and retract. At the front end of the telescopic arm 12, a flange connects to a rotating wrist 13, which can rotate ±180° in both directions via a built-in stepper motor. A flexible gripper 14 is fixed to the end of the rotating wrist 13. Figure 3 As can be seen from the front view, the flexible gripper 14 is a typical V-shaped two-finger clamping structure, and its inner wall is covered with silicone material with low Shore hardness.

[0022] The deployment of the vision and positioning system is one of the core aspects of this invention. To address the issue of field-of-view occlusion, the RGB-D depth camera 21 is not mounted on the rear base, but rather uses an eye-in-hand strategy, compactly mounted on the side support of the rotating wrist 13 or the flexible gripper 14. During operation, the RGB-D depth camera 21 extends into the tree canopy along with the robotic arm. The edge computing module 22 runs improved vision algorithms such as YOLO or Mask R-CNN to segment the fruit edges after removing ambient light interference, and uses depth information to calculate the three-dimensional coordinates of the target's centroid in the camera coordinate system. Subsequently, a homogeneous transformation matrix is ​​used to map these coordinates into control parameters in the robotic arm's base coordinate system.

[0023] The realization of non-destructive force control relies on the dual-mode sensing feedback component 3. High-precision grating rulers or position sensors 31 are configured at the power source of the base 11 and the telescopic arm 12 to output the spatial coordinates of the robotic arm in real time. On the force-bearing side of the opening and closing drive cylinder or connecting rod of the flexible gripper 14, miniature strain gauge pressure sensors 32 are arranged in series.

[0024] The specific harvesting process demonstrates the superiority of the force-position coordination algorithm:

[0025] Once the system locks onto the ripe fruit, the central control system plans a smooth B-spline motion trajectory. The telescopic arm 12 begins to extend, and the position sensor 31 verifies the actual displacement in real time, forming a closed-loop control that guides the front-end mechanism to weave through complex tree branches without interference.

[0026] Once the flexible gripper 14 encloses the target fruit, it executes a closing command. At this point, the gripper's fingertips contact the fruit peel, and the contact stress begins to rise non-linearly. The pressure sensor 32 collects this stress signal at a frequency in the hundreds of hertz range and feeds it back to the control system. When the stress value reaches the fruit's preset "safe yield threshold" (for example, the threshold for picking peaches differs from that for picking apples, typically falling within a few Newtons), the system immediately issues an interrupt command, and the gripper actuator locks. This mechanism is like giving a robotic hand "tactile sensation," ensuring the gripping force is precisely controlled—"too much and it breaks; too little and it slips."

[0027] Finally, the robotic arm retracts as a whole, using axial tension to separate the fruit stem, completing a standard, efficient, and damage-free harvesting process.

[0028] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or process transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related agricultural automation fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A non-destructive fruit-picking robotic arm based on multimodal sensing and force-position coordinated control, characterized in that, It includes a robotic arm actuator (1), a vision recognition and spatial positioning component (2), a dual-mode sensing feedback component (3), and a central control system; The robotic arm actuator (1) is used to perform multi-degree-of-freedom spatial motion and target grasping actions; The visual recognition and spatial positioning component (2) is located near the end effector at the front end of the robotic arm actuator (1) and is used to collect image features, identify maturity and calculate three-dimensional spatial pose of target fruit in natural tree canopy environment. The dual-mode sensing feedback component (3) is located at the rear and transmission node of the manipulator actuator (1) and is used to acquire the manipulator's motion stroke data and clamping contact force data in real time. The central control system is connected to the robotic arm actuator (1), the visual recognition and spatial positioning component (2), and the dual-mode sensing feedback component (3) respectively. It is used to plan the motion trajectory based on the visual calculation data and implement force-position coordinated closed-loop control according to the sensing feedback data.

2. The non-destructive fruit-picking robot based on multimodal sensing and force-position coordinated control according to claim 1, characterized in that, The robotic arm actuator (1) includes a base (11), a telescopic arm (12), a rotating wrist (13), and a flexible gripper (14). The rear end of the telescopic arm (12) is connected to the base (11) of the machine body and is driven by an internal hydraulic cylinder or linear servo motor to achieve axial extension and retraction. The rotating wrist (13) is rotatably connected to the front end of the telescopic arm (12) and has the freedom to rotate around the axis; The flexible gripper (14) is installed at the front end of the rotating wrist (13) and has an external V-shaped two-finger gripping structure. A highly elastic biomimetic buffer silicone layer is attached to its inner gripping surface.

3. The non-destructive fruit-picking robot based on multimodal sensing and force-position coordinated control according to claim 2, characterized in that, The visual recognition and spatial positioning component (2) includes an RGB-D depth camera (21) and an embedded edge computing module (22) fixedly mounted on the lateral support of the rotating wrist (13). The RGB-D depth camera (21) adopts an eye-in-hand configuration and extends into the canopy with the telescopic arm (12) to collect color image sequences and depth point cloud data with an unobstructed view.

4. The non-destructive fruit-picking robot based on multimodal sensing and force-position coordinated control according to claim 2, characterized in that, The dual-mode sensing feedback component (3) includes a position sensor (31) and a pressure sensor (32). The position sensor (31) is located at the coupling point between the base (11) and the telescopic arm (12) and is used to detect the real-time linear displacement of the telescopic arm (12) and the rotational angular displacement of the rotating wrist (13) at high frequency. The pressure sensor (32) is installed in the drive link or hydraulic control circuit of the flexible gripper (14) to monitor the dynamic contact resistance when the flexible gripper (14) closes to grasp the fruit in real time.

5. A non-destructive fruit-picking robot based on multimodal sensing and force-position coordinated control according to claim 5, characterized in that, The central control system incorporates a constant force limiting control algorithm. When the flexible gripper (14) closes and contacts the fruit peel, the pressure sensor (32) feeds back the dynamic contact resistance to the central control system. If the contact resistance reaches the preset non-destructive flexible gripping threshold, the central control system immediately cuts off the closing drive signal of the flexible gripper (14) and performs mechanical self-locking.

6. The control method for the harvesting robot according to any one of claims 1 to 6, characterized in that, The following work steps are included: S1. Perception and positioning: Drive the robotic arm to approach the work area, the RGB-D depth camera (21) performs visual scanning of the canopy, and the embedded edge computing module (22) identifies the ripe fruit and outputs the three-dimensional spatial coordinates of the target and the optimal grasping posture angle; S2. Position and posture closed-loop approach: The central control system plans the obstacle avoidance motion trajectory based on three-dimensional spatial coordinates and drives the telescopic arm (12) to extend towards the target; during this process, the position sensor (31) provides real-time feedback of the telescopic displacement to form a position closed loop, guiding the flexible gripper (14) to smoothly reach the target airspace. S3. Force-controlled flexible picking: The rotating wrist (13) rotates and aligns according to the optimal grasping posture angle; the flexible gripper (14) performs grasping under the closing command, while the pressure sensor (32) monitors the contact force at a millisecond frequency; when the contact force reaches the safety threshold, it triggers the stop closing, and then the robot arm retracts as a whole or the wrist twists to break the fruit stem, thus completing the non-destructive picking.

Images