Automatic areca nut picking robot system and control method thereof

Abstract

The invention relates to the technical field of areca-nut picking, in particular to an automatic areca-nut picking robot system, and adopts the technical scheme that fruit recognition is performed by fusing multispectral stereoscopic vision and three-dimensional laser radar point cloud data and adopting a deep learning network; the recognition rate and the three-dimensional positioning precision of the areca fruits in a complex illumination and branch and leaf shielding environment are remarkably improved; a four-wheel independent drive and active suspension mobile platform design is adopted, and a multi-sensor fusion tight coupling positioning algorithm is combined, so that the influence of uneven ground of an areca nut garden on platform stability and positioning precision is effectively overcome, and the maneuverability and operation continuity of a robot system in an unstructured environment are ensured; a special end effector based on combination of a seven-degree-of-freedom redundant mechanical arm, flexible clamping and rotary cutting is designed, in cooperation with a compliant control strategy based on force perception, lossless grabbing and precise cutting of areca nuts are achieved, and mechanical damage to the fruits and plants in the harvesting process is reduced to the maximum extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of betel nut picking, and in particular to a betel nut automatic picking robot system and a control method thereof. BACKGROUND

[0002] In the agricultural automation and intelligent equipment industry, robot technology is gradually replacing traditional manual labor to achieve efficient, low-cost and all-weather precision work. Among them, the automatic harvesting system for high-value economic crops has become an important direction of modern agricultural robot research.

[0003] Among them, betel nut as an important tropical economic crop, its fruit harvesting has long relied on manual climbing operation, which has problems of high labor intensity, high safety risk and low harvesting efficiency. The development of betel nut automatic picking robot system aims to realize accurate identification and non-destructive picking of fruits through the integration of mechanical arm, visual recognition and autonomous navigation technology.

[0004] In the prior art, the robot picking of high and tall tree crops such as betel nut still faces many technical bottlenecks. The visual system has insufficient recognition accuracy and positioning accuracy of betel nut fruits under complex natural light and branch and leaf shielding conditions, the motion planning of the mechanical arm is difficult to adapt to the randomness of the spatial distribution of the fruits and the unstructured environment of the plant structure, and the stability and positioning accuracy of the robot mobile platform in the uneven terrain of the betel nut plantation are limited, resulting in low picking success rate, long operation cycle, easy damage to fruits or plants. Therefore, it is urgent to develop a robot system and its control method that can adapt to the characteristics of betel nut planting environment and realize efficient and reliable picking. SUMMARY

[0005] The present application relates to the technical field of betel nut picking, and in particular to a betel nut automatic picking robot system and a control method thereof.

[0006] To achieve the above object, the present application provides the following technical scheme: an automatic areca nut picking robot system, comprising a mobile carrying platform, a perception and positioning module, a central decision and control module, and a multi-degree-of-freedom picking execution mechanism; the mobile carrying platform is used for independently traveling in an areca nut plantation and carrying other functional modules; the perception and positioning module is integrated on the mobile carrying platform and is used for collecting environmental data in real time and completing self-positioning and fruit identification; the central decision and control module is electrically connected with the mobile carrying platform, the perception and positioning module, and the multi-degree-of-freedom picking execution mechanism, and is used for processing perception information, planning a global operation path, deciding a picking sequence, and generating a control instruction; the multi-degree-of-freedom picking execution mechanism is installed on the mobile carrying platform and is used for receiving the instruction of the central decision and control module and executing approaching, grabbing, and separating actions on a target areca nut fruit; the mobile carrying platform adopts a chassis structure with four independent driving and independent steering wheels, each driving wheel is equipped with a hub motor and an optical encoder, and a chassis suspension system adopts a main active suspension based on a magneto-rheological damper; the perception and positioning module comprises a main visual perception unit and an auxiliary environmental perception unit, the main visual perception unit is composed of two multispectral cameras and a three-dimensional laser radar, the two multispectral cameras are installed in a stereo configuration, and the auxiliary environmental perception unit comprises four millimeter wave radars installed around the platform; the central decision and control module comprises a hardware computing platform and a software algorithm system running thereon, and the software algorithm system comprises an environmental modeling and understanding submodule, a task planning submodule, and a motion control submodule.

[0007] Further, the mobile carrying platform further integrates a global positioning unit based on multi-sensor fusion, the unit comprising a dual-antenna global satellite navigation system receiver, an inertial measurement unit, and a wheel speed odometer; the global positioning unit obtains three-dimensional position, attitude, and speed information of the mobile carrying platform in the field coordinate system by solving through a tight coupling Kalman filtering algorithm, fusing global satellite navigation system raw observation values, inertial measurement unit angular velocity and acceleration data, and wheel speed odometer displacement increments.

[0008] Further, the environmental modeling and understanding submodule executes the following processing flow: performing radiation correction and geometric correction on a stereo image sequence; extracting a candidate fruit region in the image based on a deep learning convolutional neural network, outputting a pixel-level segmentation mask and existence confidence of the fruit; performing outlier filtering and ground point segmentation on three-dimensional laser radar point cloud data; registering multiple frames of point cloud data to a unified coordinate system through an iterative closest point algorithm; performing multi-source data fusion, correlating a two-dimensional boundary box of a fruit identified in an image and a corresponding cluster in three-dimensional point cloud data, and calculating a coordinate of a target areca nut fruit in three-dimensional space, a fruit stem connection point coordinate, and a fruit maturity estimation value by using a stereo vision triangulation principle and point cloud spatial information.

[0009] Further, the task planning sub-module adopts a hierarchical planning strategy: the top layer is global path planning, based on the starting point of the mobile platform, the target work area and the static obstacle information in the environment model, an optimal path is generated by A-star search algorithm, which is weighted by energy consumption and time cost; the middle layer is picking sequence planning, for a single palm tree, according to the three-dimensional coordinates of all the fruits identified, the total length of the robot arm motion path is minimized and the number of picking action switching is minimized, and the optimal picking sequence is solved by simulated annealing algorithm; the bottom layer is real-time obstacle avoidance and re-planning, during the movement of the mobile platform or the motion of the robot arm, the millimeter wave radar data and the environment model are continuously monitored and updated, once a dynamic obstacle is detected or the planned path is blocked, the local path re-planning algorithm is started immediately.

[0010] Further, the motion control sub-module includes a platform motion controller and a robot arm motion controller; the platform motion controller receives the path point sequence output by the global path planning, calculates the target speed and steering angle of the four wheel hub motors through the model predictive control algorithm, and adjusts the magnetorheological damper current of the active suspension in real time according to the platform pitch and roll attitude angles fed back by the inertial measurement unit; the robot arm motion controller is designed for a seven-degree-of-freedom redundant robot arm, which receives the spatial coordinates of the target fruit and the coordinates of the fruit stem connection point, and based on the current joint angles of the robot arm, a set of joint angle vectors that can reach the target position is solved through inverse kinematics, and from multiple solutions, the one with the highest operation dexterity index and the highest joint motion smoothness is selected as the final solution, and a quintic polynomial interpolation method with time parameter is used to generate smooth trajectories for each joint.

[0011] Further, the robot arm motion controller introduces an external force observer based on the torque sensor during trajectory tracking, when the contact force between the end effector and the fruit or branch exceeds the preset safety threshold, a pause command is triggered immediately and reported to the central decision module.

[0012] Further, the multi-degree-of-freedom picking execution mechanism is composed of a seven-degree-of-freedom redundant robot arm, a special end effector and a force sensing system; each joint of the robot arm is driven by a frameless torque motor, and is integrated with an absolute optical encoder and a harmonic reducer; the special end effector includes two flexible clamping fingers and a micro rotary cutter; the force sensing system includes a six-axis force torque sensor installed on the wrist of the robot arm.

[0013] Further, the workflow of the special end effector is as follows: the robot motion controller controls the robot motion, so that the end effector approaches the target fruit along the planned trajectory until the preset grasping position; the central decision and control module instructs the flexible clamping fingers to close, and the closing process is divided into two stages, the first stage is the approach stage until the contact pressure detected by the pressure-sensitive conductive rubber reaches the first threshold, and the second stage is the holding stage, the proportional integral derivative controller is dynamically adjusted according to the Z-axis direction force feedback of the force torque sensor to adjust the servo motor current of the clamping fingers; the micro rotary cutter is started to rotate at a speed of 5000 revolutions per minute while advancing 2 millimeters along the fruit stem direction; after cutting is completed, the robot carries the fruit back to the top of the transport box to release the fruit.

[0014] A control method of an automatic areca nut picking robot system, according to an automatic areca nut picking robot system, comprising: system initialization, starting each sensor and actuator to complete self-checking and receiving a work task area map; moving the bearing platform to the work point of the first target areca nut tree according to the global path planning result; the perception and positioning module scans the current areca nut tree in all directions, the environment modeling and understanding submodule constructs a three-dimensional environment model containing fruit position, fruit stem information and obstacles; the task planning submodule calculates the optimal picking sequence of the current tree according to the environment model; for each target fruit in the sequence, the motion control submodule plans the robot motion trajectory and controls the multi-degree-of-freedom picking execution mechanism to complete the approach, grasping and cutting work of the fruit; after the work of a single tree is completed, it is checked whether there are still fruits to be picked, and the work process is repeated or entered into the next target tree accordingly.

[0015] Further, it also includes: continuously monitoring the state and external environment during the whole work process, and executing shutdown, alarm or avoidance action according to the preset safety protocol when encountering emergency or failure.

[0016] Compared with the prior art, the beneficial effects of the present application are: 1. By fusing multi-spectral stereo vision and three-dimensional laser radar point cloud data, and using a deep learning network for fruit recognition, the recognition rate and three-dimensional positioning accuracy of areca nuts in complex lighting and branch and leaf shielding environments are significantly improved, laying a reliable sensing foundation for subsequent accurate picking.

[0017] 2. The mobile platform design adopts four-wheel independent drive and active suspension, combined with a tightly coupled positioning algorithm based on multi-sensor fusion, effectively overcoming the influence of uneven ground in areca nut plantations on platform stability and positioning accuracy, ensuring the mobility and work continuity of the robot system in unstructured environments.

[0018] 3. A dedicated end effector based on a seven-degree-of-freedom redundant robotic arm combined with flexible gripping and rotary cutting was designed. With the help of a force-sensing compliant control strategy, it was able to achieve non-destructive gripping and precise cutting of areca nuts, minimizing mechanical damage to the fruit and plant during the harvesting process.

[0019] 4. A hierarchical intelligent decision-making and control system was constructed, from global path planning to local harvesting sequence decision-making, and then to real-time motion control and obstacle avoidance. This system enables the system to adapt to the randomness of the spatial distribution of areca nuts and optimize the operation process, thereby significantly improving the success rate and overall efficiency of the entire harvesting operation. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall technical architecture of an automatic areca nut harvesting robot system according to the present invention; Figure 2 This is a schematic diagram illustrating the core principle framework of the multi-source sensing data fusion and three-dimensional fruit localization of the present invention; Figure 3 This is a logical flowchart of the hierarchical intelligent decision-making and task planning of the present invention; Figure 4 This is a schematic diagram illustrating the multi-level interaction relationship and data flow between the mobile platform and the harvesting execution mechanism of the present invention; Figure 5 This is a schematic diagram illustrating the principle framework of the flexible gripping and precise cutting operation process of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments. Example 1

[0022] like Figure 1 As shown, an automated areca nut harvesting robot system includes a mobile support platform, a sensing and positioning module, a central decision-making and control module, and a multi-degree-of-freedom harvesting execution mechanism.

[0023] The mobile platform is the physical carrier and base of the whole system, which is responsible for the autonomous and stable movement in the uneven terrain of the areca plantation. The perception and localization module is fixedly installed on the upper structure of the mobile platform, and its core responsibility is to continuously collect the environmental information around the robot, and at the same time to complete the accurate positioning of itself in the coordinate system of the plantation and the identification of the target areca fruit. The central decision and control module is the brain of the system, which establishes a two-way communication connection with the mobile platform, the perception and localization module, and the multi-degree-of-freedom picking execution mechanism through electrical lines. This module receives the raw data from the perception module, processes, analyzes and decides, and finally generates the instruction sequence for controlling the movement of the mobile platform and the action of the mechanical arm picking. The multi-degree-of-freedom picking execution mechanism is mechanically installed in the front or top working area of the mobile platform, which directly receives the instructions from the central decision and control module and accurately executes a series of continuous actions such as spatial approach, physical grasping and fruit stem separation of the target areca fruit.

[0024] The specific technical implementation of the mobile carrying platform adopts an advanced chassis structure with four-wheel independent driving and independent steering. Each driving wheel is integrated with a high-power-density wheel hub motor and a high-resolution optical encoder. This design enables the rotation speed and steering angle of each wheel to be independently and accurately controlled. The suspension system of the chassis is not a traditional passive suspension, but an active suspension system based on a magneto-rheological damper. The magneto-rheological damper is filled with magneto-rheological fluid, the viscosity of which can change instantaneously and reversibly with the change of coil current, thereby achieving active control of the damping coefficient. The motion control submodule in the central decision and control module receives platform attitude data from the inertial measurement unit in real time, mainly the pitch angle and roll angle. When the platform is detected to have a tendency to bounce due to terrain undulations, the motion control submodule will dynamically calculate and output adjustment current to the four magneto-rheological dampers according to the preset control algorithm, to actively suppress the vibration of the vehicle body and maintain platform stability. The mobile carrying platform also integrates a high-performance global positioning unit based on multi-sensor fusion. The unit hardware includes a dual-antenna global satellite navigation system receiver, an inertial measurement unit containing a three-axis gyroscope and a three-axis accelerometer, and four wheel speed odometers embedded in the wheel hub motors. The core algorithm of the global positioning unit is a tightly coupled Kalman filter. This algorithm does not directly use the position results calculated by the global satellite navigation system, but deeply fuses the raw pseudorange and carrier phase observations of the global satellite navigation system, the three-axis angular velocity and three-axis linear acceleration raw data output by the inertial measurement unit, and the displacement increments converted from the wheel rotation angles provided by the wheel speed odometers. Through this tightly coupled method, the Kalman filter can effectively smooth the jumps in the global satellite navigation system signal caused by tree shade obstruction, and correct the drift error accumulated over time by the inertial measurement unit, ultimately outputting high-precision, high-refresh-rate three-dimensional position coordinates, three-dimensional attitude angles, and three-dimensional velocity vectors of the mobile carrying platform in the garden coordinate system.

[0025] The perception and localization module is composed of two parts: the main visual perception unit and the auxiliary environment perception unit. The main visual perception unit is the core of the entire system's perception ability, which is composed of two high-resolution multispectral cameras and a three-dimensional laser radar. The two multispectral cameras are installed in a strict stereo configuration, with their optical axes parallel and a precisely calibrated baseline distance. Their working wavebands cover the visible light and near-infrared spectral range. This configuration enables the system to simultaneously acquire color and near-infrared image sequences of the area around the ensete tree and provides a basis for subsequent stereo vision calculations. The three-dimensional laser radar is installed near the cameras, and its scanning mirror rotates at a specific frequency, emitting laser beams to the forward fan-shaped area and receiving return signals, thereby obtaining high-density three-dimensional point cloud data of the surrounding environment. Each frame of point cloud data contains the three-dimensional coordinates and reflectivity information of thousands of points. The auxiliary environment perception unit is composed of four millimeter wave radars, which are installed in the front, rear, left, and right directions of the mobile platform. Millimeter wave radars can effectively detect obstacles on the path of travel, regardless of their material, and can simultaneously measure the radial distance, azimuth angle, and relative speed of obstacles relative to the platform, providing key information for real-time obstacle avoidance.

[0026] The hardware foundation of the central decision and control module is a powerful heterogeneous computing platform. This platform typically integrates a multi-core general-purpose processor as the core of task scheduling and logical control, as well as a high-performance parallel computing accelerator for processing computationally intensive tasks such as deep learning inference and point cloud processing. On this hardware platform, three core submodules that make up the software algorithm system are running: the environment modeling and understanding submodule, the task planning submodule, and the motion control submodule. The environment modeling and understanding submodule is the end of perception information processing, which is responsible for receiving and analyzing raw data streams from multispectral cameras, three-dimensional laser radars, and millimeter wave radars. The task planning submodule plays the role of a decision maker, generating a global navigation path for the mobile carrier platform and a sequence of picking operations for the multi-degree-of-freedom picking execution mechanism based on the structured environment information output by the environment modeling and understanding submodule and the system's preset picking task goals. The motion control submodule is the bridge between decision and execution, translating the abstract path and sequence output by the task planning submodule into servo control signals that the mobile carrier platform's wheel hub motors, steering servos, and multi-degree-of-freedom picking execution mechanism's joint motors can directly understand and execute.

[0027] The data processing flow of the environment modeling and understanding submodule is a concentrated embodiment of the system's perception intelligence, Figure 2A complete framework from multi-source data to 3D localization of fruits is depicted. The processing flow starts with pre-processing of image sequences captured by a stereo multispectral camera. Pre-processing includes radiometric correction to eliminate the image response difference caused by the sensor itself and inconsistent lighting conditions, and geometric correction to correct lens distortion and ensure the accurate geometric correspondence between image pixels and the physical world. The pre-processed images are sent to a pre-trained deep learning convolutional neural network for fruit recognition. This network is trained on a dataset containing tens of thousands of images annotated with the location of areca nuts and their peduncles. The network can output two key pieces of information: one is the pixel-level segmentation mask of the fruits, which accurately outlines the contour of each fruit in the image; the other is the existence confidence of each fruit, which is used to filter out possible false positives. At the same time of image processing, the raw point cloud data captured by the 3D laser radar also undergoes a series of processing. First, outlier points are filtered out to remove isolated points caused by noise; then ground points are segmented to separate the points belonging to the ground, and the remaining points represent objects of interest such as trees and fruits. In order to construct a continuous and consistent environment model, the system will register the continuous multi-frame point cloud data to the same global coordinate system through the iterative closest point algorithm. Finally, the most critical step of multi-source data fusion is performed. The system associates and matches the 2D bounding box of each fruit identified by the deep learning network in the image with the corresponding point cloud cluster in the 3D point cloud. Using the principle of stereo vision triangulation and combining the accurate depth information provided by the matched point cloud cluster, the system can calculate the accurate coordinates of the target areca nut in 3D space. Further, by analyzing the image texture and point cloud geometric features near the peduncle connection point, the coordinates of the peduncle connection point can be estimated. In addition, multispectral information, especially the reflectance in the near-infrared band, can be used to estimate the maturity of the fruit. The entire fusion process finally outputs a set of core information for each identified fruit, including 3D spatial coordinates, peduncle connection point coordinates, and maturity estimates.

[0028] As Figure 3As shown, the task planning sub-module adopts a hierarchical progressive planning strategy. The topmost layer is global path planning. This planner is based on the starting position coordinates of the mobile platform, the boundary coordinates of the task area map, and the static obstacle polygon information identified in the environment model. An improved A-star search algorithm is used. The improved algorithm integrates the energy consumption model and the time cost model of the mobile platform into the heuristic function and the cost function based on the traditional A-star algorithm. The core cost function can be expressed as: the total cost is the weighted sum of the path length cost and the energy consumption cost. The algorithm finally searches for a global path that achieves the weighted optimal path in terms of energy consumption and operation time from the starting point to the ending point. The middle layer is picking sequence planning, which is performed for all identified fruits on a single areca tree. The planner takes the three-dimensional coordinates of all fruits as input, and its optimization goal is to minimize the total motion path length of the robot arm when picking all fruits according to the sequence, and to minimize the number of action switching times when the robot arm needs to be adjusted significantly between fruits. This is a typical combinatorial optimization problem, and the system uses the simulated annealing algorithm to solve it. The simulated annealing algorithm introduces a gradually decreasing "temperature" parameter to accept slightly worse solutions than the current solution with a certain probability, effectively jumping out of the local optimum and finding a picking sequence close to the global optimum. The bottom layer is real-time obstacle avoidance and re-planning. This is a continuously running daemon process. During the movement of the mobile platform along the global path or the movement of the robot arm according to the picking sequence, the process continuously monitors the real-time data stream from the millimeter wave radar and the updates of the obstacle map by the environment modeling sub-module. Once a dynamic obstacle is detected entering the planned path, or the originally clear path is blocked by a newly appearing static obstacle, the real-time obstacle avoidance algorithm will be triggered immediately. The algorithm is usually based on local planning methods such as dynamic window method or artificial potential field method, and can calculate a new local path that can safely bypass the obstacle and rejoin the original global path in a very short time.

[0029] The motion control sub-module is specifically divided into two parts: platform motion controller and mechanical arm motion controller. The platform motion controller receives the global path planning output from the task planning sub-module, which is usually a series of dense path point sequences. Inside the controller, a model predictive control algorithm is used. In each control cycle, the algorithm predicts the state trajectory of the platform in the future short period of time according to the kinematic and dynamic models of the mobile platform, and solves the optimal target speed sequence and steering angle sequence of the four hub motors in the future several control cycles through optimization calculation, but only executes the instructions of the first control cycle, and then re-predicts and optimizes in the next cycle to form a rolling optimization mechanism. At the same time, the controller reads the platform pitch angle and roll angle feedback from the inertial measurement unit in real time. When these attitude angles exceed the preset stability threshold, the controller will calculate and output the corresponding control current to the four magnetorheological dampers according to the attitude error, to actively adjust the suspension stiffness and keep the platform level. The mechanical arm motion controller is tailor-made for the seven-degree-of-freedom redundant mechanical arm. It receives the single target fruit spatial coordinates and fruit stem connection point coordinates from the task planning sub-module. The primary task of the controller is to convert the target pose of the end effector into the target angle of the seven joints of the mechanical arm through inverse kinematics. Since it is a seven-degree-of-freedom redundant mechanical arm, theoretically there are infinitely many joint angle solutions for the same end target pose. The controller needs to select an optimal solution from the infinite solutions. The selection criteria are usually based on two indicators: one is the operation dexterity indicator, which reflects the speed transmission performance of the mechanical arm end at the current position, and the larger the value, the better the flexibility; the second is the joint motion smoothness, which usually wants the adjacent joint angles to change smoothly to avoid sharp jumps. The system selects a solution with the highest overall score from the feasible solutions obtained by numerical methods as the final joint target. After determining the joint target, the controller uses a quintic polynomial interpolation method with time parameters to generate a position, velocity, and acceleration continuous and smooth motion trajectory for each joint between the current position and the target position. During the trajectory tracking execution process, the controller continuously monitors the readings of the six-axis force torque sensor installed on the wrist of the mechanical arm. By constructing an external force observer, the real external force generated by the contact between the end effector and the external environment can be estimated in real time. Once any direction of the contact force exceeds the safety threshold preset for protecting the fruit and the plant, the controller will immediately trigger an emergency pause command to stop all joint motion, and report this abnormal event to the central decision module for high-level decision making.

[0030] The multi-degree-of-freedom picking execution mechanism is the final execution unit that directly interacts with the betel nut fruit, such as Figure 4As shown, it consists of three main parts: a seven-degree-of-freedom redundant manipulator, a special end-effector, and a force sensing system. The seven joints of the manipulator are directly driven by high dynamic response frameless torque motors, each of which is integrated with a high precision absolute optical encoder for position feedback and equipped with a harmonic reducer to provide high torque output. The special end-effector is mounted on the end-wrist of the manipulator through a standard flange interface. It contains two key active components: two flexible gripping fingers and a miniature rotary cutter. The inner surface of the flexible gripping fingers is covered with a layer of pressure sensitive conductive rubber material, the resistance of which changes continuously with the change of the pressure, thus being able to sense the contact state and gripping force size with the fruit. The miniature rotary cutter is driven by a miniature DC servo motor, and its cutting blade is made of super-hard ceramic material, which has extremely high hardness and wear resistance. Its rotational speed can be precisely stepless regulated by adjusting the pulse width duty cycle of the input signal. The core of the force sensing system is a high precision six-axis force torque sensor installed between the end-wrist of the manipulator and the end-effector. The sensor can simultaneously measure the forces in three orthogonal directions and the torques around the three axes acting on the end-effector, providing comprehensive force feedback information for compliant control.

[0031] The working flow of the special end-effector to complete a picking operation is a fine control process. As shown in Fig. 2, the working flow is divided into three stages: fruit detection, fruit gripping, and fruit cutting. In the fruit detection stage, the end-effector is first moved to the fruit detection position, and then the fruit is detected by the force sensing system. If the fruit is detected, the end-effector will be moved to the fruit gripping position, and then the fruit will be gripped by the flexible gripping fingers. If the fruit is not detected, the end-effector will be moved to the fruit cutting position, and then the fruit will be cut by the miniature rotary cutter. In the fruit gripping stage, the end-effector is first moved to the fruit gripping position, and then the fruit is gripped by the flexible gripping fingers. In the fruit cutting stage, the end-effector is first moved to the fruit cutting position, and then the fruit is cut by the miniature rotary cutter. Figure 5As shown, the data collected by the six-axis force / torque sensor is used to control the flexible gripper fingers and the micro-rotary cutter to achieve the fruit picking. The process starts with the robot motion controller controlling the 7-DOF robot arm to move its end-effector along a pre-planned smooth trajectory from the initial position to a pre-set picking position near the target fruit. This pre-set position is usually right in front of the fruit and ensures that the gripper fingers can easily close from both sides to grasp the fruit. Upon reaching the pre-set position, the central decision and control module sends a command to the end-effector to initiate the closing action of the flexible gripper fingers. The closing action is carefully designed in two stages. The first stage is the fast approach stage, in which the two gripper fingers are driven by the servo motors to close inward rapidly until the pressure-sensitive conductive rubber on the inner surface detects a contact pressure with the fruit surface that reaches a pre-set first threshold. This threshold is low and only serves to confirm the contact. Once the first threshold is reached, the system switches to the second stage, the fine gripping stage. In this stage, the system no longer uses position as the control target but switches to force control mode. The controller uses the Z-axis force feedback from the six-axis force / torque sensor mounted on the wrist as the control variable and uses a high-performance proportional-integral-derivative controller to dynamically adjust the current of the servo motor that drives the gripper fingers. The goal of the proportional-integral-derivative controller is to stabilize the actual gripping force precisely within a higher second threshold range. This second threshold is determined through extensive experiments and ensures that the fruit will not slip during the subsequent cutting and moving process, and absolutely guarantees that there will be no pressure marks or mechanical damage to the outer skin of the betel nut due to excessive gripping force. After confirming that the gripping force has stabilized within the target range for a short period of time, the central decision and control module sends a command to start the micro-rotary cutter. The super-hard ceramic blade in the cutter is rapidly accelerated to a stable speed of 5000 rpm per minute under the drive of the micro-dc servo motor. At the same time, the control of a joint in the wrist of the robot arm or the entire robot arm is carried out to make a small linear motion, so that the rotating blade accurately advances a stroke of 2 millimeters along the direction of the fruit stem. This combination of rotation and advancement can cleanly and neatly cut off the fruit stem and complete the separation. After the cutting action is completed, the robot motion controller takes over again to control the robot arm to carry the picked fruit along a pre-planned recovery trajectory to the upper side of the transport box installed on the mobile platform. Finally, the central decision and control module instructs the flexible gripper fingers to fully open and release the fruit, allowing it to fall into the transport box, completing the complete picking cycle of a single fruit.

[0032] The central decision and control module is also responsible for coordinating the orderly operation of the entire system, and its control logic is embodied in a systematic operation method. This method begins with the system initialization phase, in which the mobile carrier platform, the perception and positioning module, and all sensors and actuators of the multi-degree-of-freedom picking execution mechanism are sequentially powered on and start, and a strict self-checking program is executed to check whether the communication, power supply, sensor readings, and actuator response are normal. After passing the self-checking, the system loads the global map of the task area from the external server or the built-in memory. Then, the system enters the autonomous navigation phase. The mobile carrier platform starts to autonomously navigate to the pre-calculated optimal work point near the first target areca tree in the task area according to the global path planning result calculated by the task planning sub-module. This optimal work point usually takes into account the workspace range of the robotic arm, the optimal observation angle of the perception module, and the stability of the platform itself. After reaching the work point, the system triggers the perception scanning phase. The perception and positioning module starts to scan the current areca tree in all directions, and the multi-spectral camera and three-dimensional laser radar collect data. The environment modeling and understanding sub-module then works to build a three-dimensional environment model containing the precise positions of all fruits on the tree, their respective fruit stem information, and the positions of surrounding obstacles such as the trunk and branches. Subsequently, the task decision phase begins. The task planning sub-module, based on the just-built environment model, calls its internal simulated annealing algorithm to calculate and output the optimal picking sequence for the current tree. Then, the system enters the core picking execution phase. For each target fruit in the optimal picking sequence, the motion control sub-module will sequentially plan the robotic arm motion trajectory and control the multi-degree-of-freedom picking execution mechanism to strictly perform a series of work actions such as approaching, grabbing, and cutting, until the fruit is successfully picked and placed in the transport box. During the picking execution of a single tree, the system continuously checks and iterates. After picking a fruit in the sequence, the system checks whether there are still un-picked and identified fruits on the current tree. If there are, it repeats the task decision phase and the picking execution phase for the next fruit. If all identified fruits on the current tree have been picked, the system controls the mobile carrier platform to travel to the next target tree according to the global path planning, and repeats the entire phase from perception scanning to picking execution. This large cycle will continue until all designated areca trees in the entire task area have completed the harvesting work, or the pre-set work termination condition is reached. Throughout the entire operation process, there is continuous system monitoring and safety guardianship. The system continuously monitors its state parameters such as battery power, motor temperature, and computing load, and also monitors changes in the external environment through the perception module.Once any emergency situation is encountered, such as detection of a person approaching, failure of a system component, or force sensor feedback exceeding a limit that fails to recover in time, the system will automatically execute a series of safeguards ranging from pausing, alerting, to emergency shutdown, active avoidance, and the like, in accordance with a pre-set multi-level safety protocol, to ensure the safety of the equipment and the environment.

[0033] As can be seen from the above extremely detailed embodiments, the system realizes full automation and intelligence from perception, decision-making to execution through highly coordinated modular design. The mobile bearing platform provides a stable and reliable mobile foundation; the perception and positioning module provides accurate environmental perception and fruit positioning; the central decision and control module provides intelligent path planning and action sequence decision-making; and the multi-degree-of-freedom picking execution mechanism provides compliant and accurate physical interaction capabilities. These four parts are organically combined to form a robot system that can adapt to the complex and unstructured environment of the betel nut garden and efficiently complete the automated picking task.

[0034] The above specific embodiments are only several preferred embodiments of the present application, and based on the technical solutions of the present application and the related inspiration of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A betel nut automatic picking robot system, characterized by: The application relates to a mobile picking robot for autonomous picking of areca nuts in areca nut plantations, comprising a mobile carrying platform, a perception and positioning module, a central decision and control module and a multi-degree-of-freedom picking execution mechanism; the mobile carrying platform is used for autonomously moving in the areca nut plantation and carrying other functional modules; the perception and positioning module is integrated on the mobile carrying platform and is used for collecting environmental data in real time and completing self-positioning and fruit identification; the central decision and control module is electrically connected with the mobile carrying platform, the perception and positioning module and the multi-degree-of-freedom picking execution mechanism, and is used for processing perception information, planning a global operation path, deciding a picking sequence and generating a control instruction; The multi-degree-of-freedom picking execution mechanism is installed on the mobile carrying platform and is used for receiving the instruction of the central decision and control module and executing approaching, grabbing and separating actions on target areca nuts; the mobile carrying platform adopts a chassis structure with four-wheel independent driving and independent steering, each driving wheel is provided with a hub motor and an optical encoder, and a main suspension system of the chassis adopts a main active suspension based on a magneto-rheological damper; the perception and positioning module comprises a main visual perception unit and an auxiliary environmental perception unit; the main visual perception unit is composed of two multispectral cameras and a three-dimensional laser radar, the two multispectral cameras are installed in a stereo configuration mode, and the auxiliary environmental perception unit comprises four millimeter wave radars installed around the platform; the central decision and control module comprises a hardware computing platform and a software algorithm system running thereon, and the software algorithm system comprises an environmental modeling and understanding submodule, a task planning submodule and a motion control submodule.

2. The automatic areca nut picking robot system according to claim 1, characterized in that: The mobile carrying platform is also integrated with a global positioning unit based on multi-sensor fusion, the unit comprises a double-antenna global satellite navigation system receiver, an inertial measurement unit and a wheel speed odometer; the global positioning unit obtains three-dimensional position, attitude and speed information of the mobile carrying platform in a field coordinate system through a tight coupling Kalman filtering algorithm, and fuses global satellite navigation system raw observation values, inertial measurement unit angular velocity and acceleration data and wheel speed odometer displacement increments.

3. The automatic areca nut picking robot system according to claim 1, wherein, The environmental modeling and understanding submodule executes the following processing flow: performing radiation correction and geometric correction on a stereo image sequence; extracting a candidate fruit region in the image based on a deep learning convolutional neural network, and outputting a pixel-level segmentation mask and an existence confidence of the fruit; performing outlier filtering and ground point segmentation on three-dimensional laser radar point cloud data; registering multiple frames of point clouds to a unified coordinate system through an iterative closest point algorithm; performing multi-source data fusion, associating a two-dimensional boundary box of the identified fruit in the image with a corresponding cluster in the three-dimensional point cloud, and calculating the coordinates of the target areca nut in the three-dimensional space, the coordinates of the fruit stem connection point and the maturity estimation value of the fruit by using a stereo vision triangulation principle and point cloud spatial information.

4. The automatic areca nut picking robot system according to claim 1, wherein, The task planning sub-module adopts a hierarchical planning strategy: the top layer is global path planning, based on the starting point of the mobile platform, the target work area and the static obstacle information in the environment model, an optimal path is generated by A-star search algorithm, which is weighted by energy consumption and time cost; the middle layer is picking sequence planning, for a single palm tree, according to the three-dimensional coordinates of all the fruits identified, the total length of the robot arm motion path is minimized and the number of picking action switching is minimized, and the optimal picking sequence is solved by simulated annealing algorithm; the bottom layer is real-time obstacle avoidance and re-planning, during the movement of the mobile platform or the robot arm, the millimeter wave radar data and the environment model are continuously monitored and updated, once a dynamic obstacle is detected or the planned path is blocked, the local path re-planning algorithm is started immediately.

5. The automatic betel nut picking robot system according to claim 1, wherein: The motion control sub-module includes platform motion controller and robot arm motion controller; the platform motion controller receives the path point sequence output by the global path planning, calculates the target speed and steering angle of the four wheel hub motors through model predictive control algorithm, and adjusts the magnetorheological damper current of the active suspension in real time according to the platform pitch and roll attitude angles fed back by the inertial measurement unit; the robot arm motion controller is designed for a seven-degree-of-freedom redundant robot arm, which receives the spatial coordinates of the target fruit and the coordinates of the fruit stem connection point, and solves a set of joint angle vectors that can reach the target position based on the current joint angles of the robot arm through inverse kinematics, selects the one with the maximum operation dexterity and the highest joint motion smoothness from multiple solutions as the final solution, and generates smooth trajectories for each joint using a quintic polynomial interpolation method with time parameters.

6. The automatic betel nut picking robot system according to claim 5, wherein: The robot arm motion controller introduces an external force observer based on torque sensors during trajectory tracking, which triggers a pause command and reports to the central decision module when the contact force between the end effector and the fruit or branch exceeds the preset safety threshold.

7. The automatic betel nut picking robot system according to claim 1, wherein: The multi-degree-of-freedom picking execution mechanism is composed of a seven-degree-of-freedom redundant robot arm, a special end effector and a force sensing system; each joint of the robot arm is driven by a frameless torque motor, and is integrated with an absolute optical encoder and a harmonic reducer; the special end effector includes two flexible clamping fingers and a miniature rotary cutter; the force sensing system includes a six-axis force torque sensor installed on the wrist of the robot arm.

8. The automatic betel nut picking robot system according to claim 7, wherein, The working process of the special end effector is as follows: the robot arm motion controller controls the movement of the robot arm, making the end effector approach the target fruit along the planned trajectory until the preset grasping position; the central decision and control module instructs the flexible clamping fingers to close, which is divided into two stages, the first stage is the approach stage until the contact pressure detected by the pressure-sensitive conductive rubber reaches the first threshold, and the second stage is the holding stage, which dynamically adjusts the servo motor current of the clamping fingers according to the Z-axis direction force feedback by the force torque sensor; The miniature rotary cutter is started to rotate at a speed of 5000 revolutions per minute while advancing 2 millimeters along the fruit stem direction; after cutting, the robot arm carries the fruit back to the top of the transport box and releases the fruit.

9. A control method of an automatic areca nut picking robot system according to any one of claims 1 to 8, wherein Comprise: The system is initialized, and each sensor and actuator is started to complete self-checking and receive a task area map. The mobile platform autonomously navigates to the working point of the first target areca tree according to the global path planning result. The perception and positioning module performs omnidirectional scanning on the current areca tree, and the environment modeling and understanding submodule constructs a three-dimensional environment model containing fruit position, fruit stem information and obstacles. The task planning submodule calculates the optimal picking sequence of the current tree according to the environment model. For each target fruit in the sequence, the motion control submodule plans the motion trajectory of the mechanical arm and controls the multi-degree-of-freedom picking execution mechanism to complete the fruit approaching, grabbing and cutting work. After the work of a single tree is completed, it is checked whether there are still fruits to be picked, and the corresponding repetition or next target tree work process is entered.

10. The control method of an automatic betel nut picking robot system according to claim 9, wherein, Also includes: During the whole operation process, the state of itself and the external environment are continuously monitored, and when an emergency or failure occurs, the preset safety protocol is executed to stop, alarm or avoid action.

Images