An ackerman mobile picking robot based on dual vision cooperation

Abstract

The application is suitable for the technical field of agricultural automation equipment, and provides an Ackerman mobile picking robot based on double-vision cooperation, which comprises an Ackerman chassis and further comprises: a six-degree-of-freedom mechanical arm, a shearing end effector being arranged on the six-degree-of-freedom mechanical arm; a global vision module, the global vision module being installed at the front end of the Ackerman chassis through a two-dimensional holder; a hand-eye vision module, the hand-eye vision module being fixedly installed on the shearing end effector; and a control unit, the control unit being integrated in an industrial computer. The device has efficiency and precision, has strong error compensation capacity, has a reasonable structure, is good in adaptability, and has strong anti-interference capacity.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of agricultural automation equipment, and particularly relates to an Ackerman mobile picking robot based on double-vision cooperation. BACKGROUND

[0002] Fruit picking is an important link in agricultural production, and at present, most orchards still rely on manual picking, which has problems of high labor intensity, low efficiency, high cost and the like. With the development of automation technology, picking robots have become a research hotspot.

[0003] The existing picking robot technical solutions mainly have the following deficiencies: first, the robot with a fixed base lacks mobility and has a limited working range. Second, the robot with a mobile platform mostly uses a single vision sensor. If only a global camera is used, the image is prone to shaking when the vehicle moves because the global camera is fixed on the mobile chassis, and the long-distance recognition and positioning accuracy is low, which cannot meet the requirements of accurate picking; if only an eye-in-hand camera is used, the mechanical arm needs to perform a large range of blind scanning to find the target, and the search efficiency is extremely low. Third, many mobile platforms use a differential chassis structure, and the kinematic model and coordinated control of the mechanical arm are relatively simple, but in the complex orchard terrain, the Ackerman chassis has an advantage in terms of turning structure and better stability. However, the non-omnidirectional motion characteristics of the Ackerman chassis will cause the parking pose to have lateral and heading deviations, and the traditional control method cannot enable the mechanical arm to directly perform accurate positioning based on the vision information before moving, and there is currently a lack of effective solutions to compensate for the error introduced by the specific motion model.

[0004] Therefore, there is an urgent need for a picking robot that can balance wide-range and efficient search, high-precision positioning picking, and effectively adapt to unstructured orchard terrain and compensate for the movement deviation. SUMMARY

[0005] The application aims to solve the problems in the background.

[0006] The application is implemented as follows: an Ackerman mobile picking robot based on double-vision cooperation, comprising an Ackerman chassis, and further comprising:

[0007] a six-degree-of-freedom mechanical arm, a base of the six-degree-of-freedom mechanical arm being fixedly installed on a vehicle frame of the Ackerman chassis, and a shearing end effector being arranged on the six-degree-of-freedom mechanical arm, the shearing end effector being used for picking fruits and shearing fruit stems;

[0008] a global vision module, the global vision module being installed at a front end of the Ackerman chassis through a two-dimensional holder, the global vision module being used for collecting color images and depth information of the front environment, and the two-dimensional holder being used for keeping the posture of the global vision module stable;

[0009] a hand-eye vision module fixedly installed on a shearing end effector of the six-degree-of-freedom robot arm, used for collecting images of target fruits at close range;

[0010] a control unit integrated in the industrial computer and in communication connection with the controller of the Ackerman chassis, the controller of the six-degree-of-freedom robot arm, the controller of the two-dimensional holder, the global vision module and the hand-eye vision module, used for receiving data collected by each module and controlling through closed-loop feedback.

[0011] In a further technical solution, the Ackerman chassis adopts a four-wheel structure, the front wheels are steering wheels, and the rear wheels are driving wheels, powered by a lithium battery pack.

[0012] In a further technical solution, the shearing end effector is an electric shearing device with a jaw structure and a shearing structure, and a built-in pressure sensor is used to grab fruits and cut off the fruit stems.

[0013] In a further technical solution, the two-dimensional holder is fixedly installed at the front end of the Ackerman chassis through a support, the horizontal rotation angle range is 0-360°, and the pitch angle range is -30° to +90°; the two-dimensional holder is provided with an IMU (Inertial Measurement Unit) sensor and an attitude sensor, used for monitoring the attitude angle change of the two-dimensional holder in real time.

[0014] In a further technical solution, the industrial computer is installed in a waterproof electrical box on the Ackerman chassis.

[0015] In a further technical solution, the control method of the Ackerman mobile picking robot comprises the following steps:

[0016] Step 1: global coarse positioning and chassis guidance;

[0017] After initializing the system, the global vision module scans the working area under the drive of the two-dimensional holder; during the movement of the Ackerman chassis, the industrial computer continuously reads the IMU sensor data of the Ackerman chassis and the two-dimensional holder, dynamically adjusts the servo of the two-dimensional holder through the PID control algorithm, compensates for the shaking of the Ackerman chassis, and maintains the stable observation posture of the global vision module; the global vision module continuously collects images, identifies the fruit target through the image recognition algorithm, and calculates the first coordinate P_global of the target fruit relative to the coordinate system {O_global} of the global vision module;

[0018] Step 2: coordinate conversion and path planning;

[0019] The industrial computer converts P_global from the {O_global} coordinate system directly to the coordinate system {O_base} of the six-degree-of-freedom robot arm according to the conversion matrix obtained in advance through calibration, to obtain a second coordinate P_approx for the pre-positioning of the six-degree-of-freedom robot arm; at the same time, P_global is converted to the coordinate system {O_akm} of the Ackermann chassis to obtain a third coordinate P_akm, and then the target pose (X_target, Y_target, θ_target) to be moved by the Ackermann chassis is calculated by taking the third coordinate P_akm as input; at the same time, the controller of the six-degree-of-freedom robot arm receives the P_approx coordinate, calculates the target pose (X_gripper, Y_gripper, θ_gripper) of the shear end effector, and controls the six-degree-of-freedom robot arm to start the pre-positioning movement towards the target fruit;

[0020] Step 3: Chassis movement

[0021] The Ackermann chassis receives the instruction and moves to the target pose.

[0022] Step 4: Hand-eye fine calibration

[0023] The six-degree-of-freedom robot arm is controlled to move, the hand-eye vision module immediately captures a high-definition image, a precise positioning of the fruit is performed by using an image processing algorithm, and the fourth coordinate P_precise of the fruit relative to the base coordinate system {O_base} of the robot arm is calculated in combination with the known hand-eye relationship, so as to correct all the accumulated errors in the previous stage, and the precise calibration of the pre-positioning coordinate P_approx of the six-degree-of-freedom robot arm is realized through closed-loop feedback, and the target precise pose (X_gripper_approx, Y_gripper_approx, θ_gripper_approx) to be moved by the shear end effector is calculated.

[0024] Step 5: Picking execution

[0025] The six-degree-of-freedom robot arm plans a collision-free path from (X_gripper, Y_gripper, θ_gripper) to (X_gripper_approx, Y_gripper_approx, θ_gripper_approx), and the shear end effector completes the shearing action.

[0026] Step 6: Loop decision

[0027] After the picking is completed, the control unit decides whether to continue picking other fruits of the current plant or to control the Ackermann chassis to move to the next global target point.

[0028] The embodiment of the present application provides an Ackerman mobile picking robot based on double-vision cooperation, which has the following beneficial effects:

[0029] (1) Efficiency and accuracy: the global vision module is used for target search and coarse positioning in a large range and high efficiency, and the Ackerman chassis is guided to move; the hand-eye vision module is used for final positioning in a short distance and millimeter-level accuracy, and the picking action is guided, so that the contradiction between search and positioning is perfectly solved.

[0030] (2) Strong error compensation capability: the precise positioning process of the hand-eye vision module is creatively used as the closed-loop feedback of the whole system, so that the parking position error of the Ackerman chassis due to the non-omnidirectional motion model, the calibration error of the vision system and other random disturbances in the orchard environment can be effectively compensated, and the robustness and picking success rate of the system are greatly improved.

[0031] (3) Reasonable structure and good adaptability: the Ackerman chassis is more suitable for outdoor uneven terrain movement, the arrangement mode of the double-vision module fully plays the respective advantages, and the whole system architecture is optimized for the modern agricultural application scene, so that the practicality is high.

[0032] (4) Strong anti-interference capability: the two-dimensional holder based on the PID algorithm is introduced, so that the shaking and impact of the Ackerman chassis during movement on the unstructured terrain are effectively inhibited, the stable image can be obtained during movement, and the overall robustness of the system in the complex environment is improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A structure schematic diagram of the Ackerman mobile picking robot based on double-vision cooperation is provided for the embodiment of the present application;

[0034] Figure 2 A side view of the Ackerman mobile picking robot based on double-vision cooperation is provided for the embodiment of the present application;

[0035] Figure 3 A top view of the Ackerman mobile picking robot based on double-vision cooperation is provided for the embodiment of the present application;

[0036] Figure 4 A coordinate conversion relationship schematic diagram of the Ackerman mobile picking robot based on double-vision cooperation is provided for the embodiment of the present application;

[0037] Figure 5 A work flow diagram of the double-vision cooperation control method of the Ackerman mobile picking robot based on double-vision cooperation is provided for the embodiment of the present application.

[0038] In the drawings: Akkerman chassis 1; six degrees of freedom mechanical arm 2; hand-eye vision module 3; shearing end effector 4; global vision module 5; two-dimensional gimbal 6; industrial computer 7. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0040] The specific implementation of the present application is described in detail below in combination with specific embodiments.

[0041] As shown in the drawings, an Akkerman mobile picking robot based on double vision cooperation provided by an embodiment of the present application comprises an Akkerman chassis 1, and further comprises: Figures 1-3 A six degrees of freedom mechanical arm 2, a base of the six degrees of freedom mechanical arm 2 is fixedly installed on a vehicle frame of the Akkerman chassis 1, a shearing end effector 4 is arranged on the six degrees of freedom mechanical arm 2, and the shearing end effector 4 is used for grabbing fruits and cutting off fruit stems;

[0042] A global vision module 5, the global vision module 5 is installed at a front end of the Akkerman chassis 1 through a two-dimensional gimbal 6, the global vision module 5 is used for collecting color images and depth information of a front environment, and the two-dimensional gimbal 6 is used for keeping a posture of the global vision module 5 stable;

[0043] A hand-eye vision module 3, the hand-eye vision module 3 is fixedly installed on the shearing end effector 4 of the six degrees of freedom mechanical arm 2, and is used for collecting target fruit images at a close distance;

[0044] A control unit, the control unit is integrated in an industrial computer 7, is in communication connection with a controller of the Akkerman chassis 1, a controller of the six degrees of freedom mechanical arm 2, a controller of the two-dimensional gimbal 6, the global vision module 5 and the hand-eye vision module 3 respectively, is used for receiving data collected by each module, and controls through closed-loop feedback.

[0045] In the embodiment of the present application, the control unit dynamically adjusts the two-dimensional gimbal 6 through a PID control algorithm according to data of an attitude sensor and an inertial measurement unit sensor built in the two-dimensional gimbal 6, so as to keep the posture of the global vision module 5 stable when the Akkerman chassis 1 moves and shakes; the global vision module 5 is a vision sensor with depth perception function, a depth vision field angle of the global vision module 5 is greater than that of the hand-eye vision module 3, so as to realize target search in a wider range.

[0046]

[0047] ​As a preferred embodiment of the present application, the Ackerman chassis 1 adopts a four-wheel structure, the front wheels are steering wheels, the rear wheels are driving wheels, and is powered by a lithium battery pack.

[0048] As a preferred embodiment of the present application, the hand-eye vision module 3 is fixed above the shearing end effector 4 by a light support, and the optical axis direction is at a certain angle with the working direction of the shearing end effector 4 to avoid blocking.

[0049] As a preferred embodiment of the present application, the shearing end effector 4 is an electric shearing device, which has a jaw structure and a shearing structure, and is provided with a built-in pressure sensor for grabbing fruits and shearing off fruit stems.

[0050] As a preferred embodiment of the present application, the two-dimensional gimbal 6 is fixedly installed at the front end of the Ackerman chassis 1 by a support, and has a horizontal rotation angle range of 0-360° and a pitch angle range of -30° to +90°. The two-dimensional gimbal 6 is provided with an IMU (Inertial Measurement Unit) sensor and an attitude sensor for monitoring the attitude angle change of itself in real time.

[0051] As a preferred embodiment of the present application, the industrial computer 7 is installed in a waterproof electrical box on the Ackerman chassis 1.

[0052] As shown in Figure 5 As a preferred embodiment of the present application, the control method of the Ackerman mobile picking robot comprises the following steps:

[0053] Step 1: global coarse positioning and chassis guidance;

[0054] After initializing the system, the global vision module 5 is driven by the two-dimensional gimbal 6 to scan the working area. During the movement of the Ackerman chassis 1, the industrial computer 7 continuously reads the IMU sensor data of the Ackerman chassis 1 and the two-dimensional gimbal 6, dynamically adjusts the servo of the two-dimensional gimbal 6 through the PID control algorithm, compensates for the shaking of the Ackerman chassis 1, and maintains the stable observation posture of the global vision module 5. The industrial computer 7 runs a target detection algorithm (all the algorithms used are prior art, which will not be specifically stated here), identifies the ripe fruits in the image, and calculates the three-dimensional coordinates P_global of the fruit target in the coordinate system {O_global} of the global vision module 5 through the principle of visual ranging by using the intrinsic parameters of the global vision module 5 and the pre-calibrated extrinsic parameters (first coordinates).

[0055] Step 2: coordinate conversion and path planning;

[0056] The industrial computer 7 converts P global from the {O global} coordinate system to the coordinate system {O base} of the six-degree-of-freedom robot arm 2 according to the conversion matrix obtained in advance through calibration, to obtain the coordinate P approx (second coordinate) for the pre-positioning of the six-degree-of-freedom robot arm 2. The system starts a parallel processing mechanism: P global is converted to the coordinate system {O akm} of the Ackermann chassis 1 to obtain the coordinate P akm (third coordinate), and then the path planning module takes P akm as input to calculate the target pose (X target, Y target, θ target) that the Ackermann chassis 1 needs to move, which ensures that the fruit is located in the best region of the workspace of the six-degree-of-freedom robot arm 2; at the same time, the controller of the six-degree-of-freedom robot arm 2 receives the P approx coordinate and calculates the target pose (X gripper, Y gripper, θ gripper) of the shearing end effector 4 to control the six-degree-of-freedom robot arm 2 to start the pre-positioning movement to the approximate position of the target fruit, which significantly shortens the time of subsequent fine positioning.

[0057] Step 3: chassis movement;

[0058] The Ackermann chassis 1 receives the instruction and moves to the target pose.

[0059] Step 4: fine hand-eye calibration;

[0060] Due to uneven ground and motion control errors, there is a deviation between the actual stopping pose of the Ackermann chassis 1 and the target pose. At this time, the six-degree-of-freedom robot arm 2 has already reached the target approximate region, and the hand-eye vision module 3 immediately takes a high-definition image, uses a fine image processing algorithm to accurately position the fruit, and combines the known hand-eye relationship (obtained through hand-eye calibration) to calculate the precise coordinate P precise (fourth coordinate) of the fruit relative to the robot arm base coordinate system {O base}. This coordinate P precise corrects all the previous accumulated errors and realizes accurate correction of the pre-positioning coordinate P approx of the six-degree-of-freedom robot arm 2 through closed-loop feedback, and calculates the target precise pose (X gripper approx, Y gripper approx, θ gripper approx) of the movement of the shearing end effector 4.

[0061] Step 5: picking execution;

[0062] The six-DOF robotic arm 2 plans a collision-free path from (X_gripper, Y_gripper, θ_gripper) to (X_gripper_approx, Y_gripper_approx, θ_gripper_approx), the scissor end effector 4 completes the shearing action, and the sensor confirms the successful picking.

[0063] Step 6: Loop decision making;

[0064] After picking is completed, the control unit decides whether to continue picking other fruits of the current plant or to control the Ackerman chassis 1 to move to the next global target point.

[0065] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A control method of an Ackerman mobile picking robot based on binocular vision cooperation, characterized in that, The Ackerman mobile picking robot comprises an Ackerman chassis, further comprising: a six-degree-of-freedom mechanical arm, a base of the six-degree-of-freedom mechanical arm being fixedly installed on a frame of the Ackerman chassis, and a shearing end effector being arranged on the six-degree-of-freedom mechanical arm, the shearing end effector being used for picking fruits and shearing off fruit stems; a global vision module, the global vision module being installed at a front end of the Ackerman chassis through a two-dimensional holder, the global vision module being used for collecting color images and depth information of a front environment, and the two-dimensional holder being used for keeping a posture of the global vision module stable; a hand-eye vision module, the hand-eye vision module being fixedly installed on the shearing end effector of the six-degree-of-freedom mechanical arm, and being used for collecting target fruit images at a close distance; a control unit, the control unit being integrated in an industrial computer, and being in communication connection with a controller of the Ackerman chassis, a controller of the six-degree-of-freedom mechanical arm, a controller of the two-dimensional holder, the global vision module and the hand-eye vision module, respectively, the control unit being used for receiving data collected by each module and being controlled through closed-loop feedback; the control method of the Ackerman mobile picking robot comprises the following steps: Step 1: global coarse positioning and chassis guiding; after initializing the system, the global vision module scans a working area under the driving of the two-dimensional holder; during the movement of the Ackerman chassis, the industrial computer continuously reads IMU sensor data of the Ackerman chassis and the two-dimensional holder, dynamically adjusts a rudder of the two-dimensional holder through a PID control algorithm, compensates for the shaking of the Ackerman chassis, maintains a stable observation posture of the global vision module, and continuously collects images, identifies fruit targets through an image recognition algorithm, and calculates a first coordinate P_global of the target fruit relative to a coordinate system {O_global} of the global vision module; Step 2: coordinate conversion and path planning; the industrial computer directly converts P_global from the coordinate system {O_global} to a coordinate system {O_base} of the six-degree-of-freedom mechanical arm through a conversion matrix obtained through calibration in advance, obtains a second coordinate P_approx for the pre-positioning of the six-degree-of-freedom mechanical arm, simultaneously converts P_global to a coordinate system {O_akm} of the Ackerman chassis to obtain a third coordinate P_akm, and then takes the third coordinate P_akm as input to calculate a target pose (X_target, Y_target, θ_target) that the Ackerman chassis needs to move; simultaneously, the controller of the six-degree-of-freedom mechanical arm receives the P_approx coordinate, calculates a target pose (X_gripper, Y_gripper, θ_gripper) of the shearing end effector, and controls the six-degree-of-freedom mechanical arm to start pre-positioning movement towards the target fruit; Step 3: chassis movement; the Ackerman chassis receives instructions and moves to the target pose; Step 4: hand-eye fine calibration; The six-degree-of-freedom robot arm is controlled to move, the hand-eye vision module immediately captures a high-definition image, an image processing algorithm is used to accurately locate the fruit, and the known hand-eye relationship is combined to calculate the fourth coordinate P_precise of the fruit relative to the base coordinate system {O_base} of the robot arm, thereby correcting all the accumulated errors in the previous stage, and through closed-loop feedback, the six-degree-of-freedom robot arm is accurately corrected to the predetermined coordinate P_approx of the robot arm, and the target accurate pose (X_gripper_approx, Y_gripper_approx, θ_gripper_approx) of the shearing end effector is calculated; Step 5: picking execution; The six-degree-of-freedom robot arm plans a collision-free path from (X_gripper, Y_gripper, θ_gripper) to (X_gripper_approx, Y_gripper_approx, θ_gripper_approx), and the shearing end effector completes the shearing action; Step 6: loop decision; After picking, the control unit decides whether to continue picking other fruits of the current plant or to control the Ackerman chassis to move to the next global target point.

2. The control method of the Ackerman mobile picking robot based on binocular vision cooperation according to claim 1, characterized in that, The Ackerman chassis adopts a four-wheel structure, the front wheels are steering wheels, the rear wheels are driving wheels, and the lithium battery pack is used for power supply.

3. The control method of the Ackerman mobile picking robot based on binocular vision cooperation according to claim 1, characterized in that, The shearing end effector is an electric shearing device, which has a jaw structure and a shearing structure, and is provided with a built-in pressure sensor for grabbing the fruit and shearing the fruit stem.

4. The control method of the Ackerman mobile picking robot based on binocular vision cooperation according to claim 1, characterized in that, The two-dimensional gimbal is fixedly installed at the front end of the Ackerman chassis through a support, the horizontal rotation angle range is 0-360°, and the pitch angle range is -30° to +90°. The two-dimensional gimbal is provided with an IMU sensor and an attitude sensor for monitoring the attitude angle change of the two-dimensional gimbal in real time.

5. The control method of the Ackerman mobile picking robot based on binocular vision synergy according to claim 1, characterized in that, The industrial computer is installed in a waterproof electrical box on the Ackerman chassis.

Images