Robot for fruit recognition and picking based on vision and method for using the robot

Abstract

The present invention relates to a robot for fruit picking based on visual recognition, comprising a mobile trolley, a six-degree-of-freedom manipulator, an industrial computer, a fruit storage box, a gripper, and a camera; the six-degree-of-freedom manipulator, the industrial computer, and the fruit storage box are all carried on the mobile trolley; the gripper and the camera are both installed at the end of the six-degree-of-freedom manipulator; through the above scheme, target fruits can be automatically recognized and picked without manual intervention, thereby improving picking efficiency and productivity; through visual recognition, target fruits can be accurately identified, thereby improving picking accuracy and efficiency; the picked target fruits can be placed in the fruit storage box, thereby facilitating continuous picking.

Description

Technical Field

[0001] The present invention relates to the field of robotics, and more particularly to a robot for visually identifying and picking fruits and a method for using the robot. Background Art

[0002] With the rapid development of my country's economy, labor costs in all walks of life continue to rise, especially in the field of agricultural production. The labor cost of picking agricultural products is relatively high, which requires modern agriculture to continue to develop in the direction of mechanization and intelligence.

[0003] As for fruit picking, most of the existing fruit picking robots are semi-automated, with low working efficiency and difficult to promote and apply on a large scale.

[0004] Therefore, there is still a need to improve the existing fruit picking robots to solve the problem of low efficiency of the existing fruit picking robots. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a robot for visually identifying and picking fruits and a method for using the robot in response to the above-mentioned defects of the prior art.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A robot for fruit recognition and picking based on vision is constructed, comprising a mobile trolley, a six-degree-of-freedom manipulator, an industrial computer, a fruit storage box, a gripper, and a camera; the six-degree-of-freedom manipulator, the industrial computer, and the fruit storage box are all mounted on the mobile trolley; the gripper and the camera are both mounted at the end of the six-degree-of-freedom manipulator; wherein,

[0008] The industrial computer is used to generate a planned path based on the current position of the mobile cart, the current position to be picked, a preset orchard map, and path planning rules; the starting point of the planned path is the current position of the mobile cart, and the end point is the current position to be picked;

[0009] The mobile cart is used to move along the planned path to the current picking position;

[0010] The camera is used to capture images of fruit trees at the current location to be picked;

[0011] The industrial computer is used to input the fruit tree image into a preset target fruit recognition model to obtain a target fruit recognition result;

[0012] The industrial computer is used to obtain the position information of the target fruit if there is a target fruit, and adjust the posture of the six-degree-of-freedom manipulator according to the position information of the target fruit until the target fruit enters the gripping range of the gripper;

[0013] The gripper is used to grip the target fruit; the industrial computer is used to adjust the posture of the six-degree-of-freedom manipulator until the target fruit gripped by the gripper enters the storage area of ​​the fruit storage box;

[0014] The clamping claws are used to release the clamped target fruit;

[0015] The industrial computer is used to determine whether there is a next position to be picked. If so, the next position to be picked is set as the current position to be picked, and jump to the step of generating a planned path based on the current position of the mobile vehicle, the current position to be picked, the preset orchard map and the path planning rules until there is no next position to be picked.

[0016] A method for using a robot is also constructed, using the above-mentioned robot for visually identifying and picking fruits; the method for using the robot comprises the following steps:

[0017] Generate a planned path based on the current position of the mobile cart, the current position to be picked, a preset orchard map, and path planning rules; the starting point of the planned path is the current position of the mobile cart, and the end point is the current position to be picked;

[0018] Move to the current picking position along the planned path;

[0019] Take an image of the fruit tree at the current location to be picked;

[0020] Input the fruit tree image into the preset target fruit recognition model to obtain the target fruit recognition result;

[0021] If there is a target fruit, obtain the location information of the target fruit and adjust the posture of the six-degree-of-freedom manipulator according to the location information of the target fruit until the target fruit enters the gripping range of the gripper;

[0022] Clamping the target fruit, and adjusting the posture of the six-degree-of-freedom manipulator until the target fruit clamped by the clamp enters the storage area of ​​the fruit storage box;

[0023] Release the trapped target fruit;

[0024] Determine whether there is a next position to be picked. If so, set the next position to be picked as the current position to be picked, and jump to the step of generating a planned path based on the current position of the mobile car, the current position to be picked, the preset orchard map and the path planning rules until there is no next position to be picked.

[0025] The beneficial effects of the present invention are: it can automatically identify target fruits and pick them without manual intervention, thereby improving picking efficiency and productivity; through visual recognition, it can accurately identify target fruits, thereby improving picking accuracy and efficiency; the picked target fruits can be placed in a fruit storage box, thereby facilitating continuous picking. The specific usage process is as follows: according to the current position of the mobile car, the current position to be picked, the preset orchard map and the path planning rules, a planned path is generated, and then, the mobile car moves to the current position to be picked, and then, the image of the fruit tree at the current position to be picked is captured, and then, the image of the fruit tree is input into the preset target fruit recognition model to obtain the target fruit recognition result, and then, if there is target fruit, the position information of the target fruit is obtained, and the posture of the six-degree-of-freedom manipulator is adjusted according to the position information of the target fruit until the target fruit enters the clamping range of the clamp, and then, the target fruit is clamped, and the posture of the six-degree-of-freedom manipulator is adjusted until the target fruit clamped by the clamp enters the storage area of ​​the fruit storage box, and then, the clamped target fruit is released, and then, it is determined whether there is a next position to be picked. If so, the next position to be picked is set as the current position to be picked, and the step of generating a planned path according to the current position of the mobile car, the current position to be picked, the preset orchard map and the path planning rules is jumped to, until there is no next position to be picked. In the above scheme, the height and end position of the six-degree-of-freedom manipulator are determined each time picking, which can increase the success rate of picking. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive efforts.

[0027] Figure 1 This is an axonometric diagram of a robot for visually recognizing and picking fruits provided in the first embodiment of the present invention. DETAILED DESCRIPTION

[0028] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the following will be a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work shall fall within the scope of protection of the present invention.

[0029] Example 1

[0030] The embodiment of the present invention provides a robot that can recognize and pick fruits based on vision, such as Figure 1As shown, it includes a mobile car 10, a six-degree-of-freedom manipulator 11, an industrial computer 12, a fruit storage box 13, a gripper 14, and a camera 15; the six-degree-of-freedom manipulator 11, the industrial computer 12, and the fruit storage box 13 are all mounted on the mobile car 10; the gripper 14 and the camera 15 are both installed at the end of the six-degree-of-freedom manipulator 11; wherein,

[0031] The industrial computer 12 is used to generate a planned path based on the current position of the mobile cart 10, the current position to be picked, a preset orchard map, and path planning rules; the starting point of the planned path is the current position of the mobile cart 10, and the end point is the current position to be picked;

[0032] A mobile cart 10 is used to move along a planned path to the current picking position;

[0033] Camera 15, used to capture images of fruit trees at the current location to be picked;

[0034] The industrial computer 12 is used to input the fruit tree image into a preset target fruit recognition model to obtain a target fruit recognition result;

[0035] The industrial computer 12 is used to obtain the position information of the target fruit if there is a target fruit, and adjust the posture of the six-degree-of-freedom manipulator 11 according to the position information of the target fruit until the target fruit enters the gripping range of the gripper 14;

[0036] The gripper 14 is used to grip the target fruit; the industrial computer 12 is used to adjust the posture of the six-degree-of-freedom manipulator 11 until the target fruit gripped by the gripper 14 enters the storage area of ​​the fruit storage box 13;

[0037] The clamping claw 14 is used to release the clamped target fruit;

[0038] The industrial computer 12 is used to determine whether there is a next position to be picked. If so, the next position to be picked is set as the current position to be picked, and jump to the step of generating a planned path based on the current position of the mobile cart 10, the current position to be picked, the preset orchard map and the path planning rules, until there is no next position to be picked.

[0039] The robot provided in this embodiment can automatically identify target fruits and pick them without manual intervention, thereby improving picking efficiency and productivity; through visual recognition, it can accurately identify target fruits, thereby improving picking accuracy and efficiency; the picked target fruits can be placed in the fruit storage box 13, facilitating continuous picking. The specific usage process is to generate a planned path according to the current position of the mobile cart 10, the current position to be picked, the preset orchard map and the path planning rules, and then move to the current position to be picked along the planned path, and then take the image of the fruit tree at the current position to be picked, and then input the image of the fruit tree into the preset target fruit recognition model to obtain the target fruit recognition result, and then, if there is target fruit, obtain the position information of the target fruit, and adjust the posture of the six-degree-of-freedom manipulator 11 according to the position information of the target fruit until the target fruit enters the clamping range of the clamp 14, and then clamp the target fruit, and adjust the posture of the six-degree-of-freedom manipulator 11 until the target fruit clamped by the clamp 14 enters the storage area of ​​the fruit storage box 13, and then release the clamped target fruit, and then determine whether there is a next position to be picked. If so, set the next position to be picked as the current position to be picked, and jump to the step of generating a planned path according to the current position of the mobile cart 10, the current position to be picked, the preset orchard map and the path planning rules, until there is no next position to be picked. In the above scheme, the height and the position of the end of the six-degree-of-freedom manipulator 11 are determined each time the picking is performed, which can increase the success rate of picking.

[0040] Optionally, the mobile car 10 is also equipped with a laser radar 16; the end of the six-degree-of-freedom manipulator 11 is also equipped with a pressure sensor 17 electrically connected to the industrial computer 12; wherein,

[0041] The laser radar 16 is used to collect three-dimensional point cloud data of the orchard; the mobile vehicle 10 is also used to drive the laser radar 16 to move in the orchard;

[0042] The industrial computer 12 is used to generate an orchard map based on the three-dimensional point cloud data; wherein the orchard map includes location information of the starting point and the end point;

[0043] The pressure sensor 17 is used to detect the pressure exerted on the clamping jaw 14;

[0044] The industrial computer 12 is used to determine whether the pressure is greater than a preset value. If so, it is determined that the picking is successful and the next step is executed.

[0045] The robot described above uses a laser radar 16 visual SLAM system to construct an orchard map. Compared to visual sensors, the laser radar 16 has high ranging accuracy and is less susceptible to external interference (such as changes in lighting and viewing angle), greatly improving the accuracy of the resulting map point cloud. Furthermore, ranging via the laser radar 16 facilitates autonomous navigation while the mobile cart 10 is moving. Furthermore, the pressure sensor 17 enables real-time assessment of the harvested fruit, avoiding misjudgments caused by emptying the fruit and placing it in the fruit storage bin 13, thus ensuring higher utilization of the fruit storage bin 13.

[0046] Example 2

[0047] An embodiment of the present invention provides a method for using a robot, based on the robot for visually identifying and picking fruit as described in Example 1. The method for using the robot comprises the following steps:

[0048] Generate a planned path based on the current position of the mobile cart 10, the current position to be picked, a preset orchard map, and path planning rules; the starting point of the planned path is the current position of the mobile cart 10, and the end point is the current position to be picked;

[0049] Move to the current picking position along the planned path;

[0050] Take an image of the fruit tree at the current location to be picked;

[0051] Input the fruit tree image into the preset target fruit recognition model to obtain the target fruit recognition result;

[0052] If there is a target fruit, obtain the position information of the target fruit, and adjust the posture of the six-degree-of-freedom manipulator 11 according to the position information of the target fruit until the target fruit enters the gripping range of the gripper 14;

[0053] Clamp the target fruit and adjust the posture of the six-degree-of-freedom manipulator 11 until the target fruit clamped by the gripper 14 enters the storage area of ​​the fruit storage box 13;

[0054] Release the trapped target fruit;

[0055] Determine whether there is a next position to be picked. If so, set the next position to be picked as the current position to be picked, and jump to the step of generating a planned path based on the current position of the mobile cart 10, the current position to be picked, the preset orchard map and the path planning rules, until there is no next position to be picked.

[0056] The method of use provided in this embodiment generates a planned path based on the current position of the mobile cart 10, the current position to be picked, a preset orchard map, and path planning rules, and then moves to the current position to be picked along the planned path. Then, an image of the fruit tree at the current position to be picked is captured, and then the image of the fruit tree is input into a preset target fruit recognition model to obtain a target fruit recognition result. Then, if there is target fruit, the position information of the target fruit is obtained, and the posture of the six-degree-of-freedom manipulator 11 is adjusted according to the position information of the target fruit until the target fruit enters the clamping range of the clamp 14. Then, the target fruit is clamped, and the posture of the six-degree-of-freedom manipulator 11 is adjusted until the target fruit clamped by the clamp 14 enters the storage area of ​​the fruit storage box 13. Then, the clamped target fruit is released. Then, it is determined whether there is a next position to be picked. If so, the next position to be picked is set as the current position to be picked, and the step of generating a planned path based on the current position of the mobile cart 10, the current position to be picked, the preset orchard map, and path planning rules is skipped until there is no next position to be picked. In the above scheme, the height and the position of the end of the six-degree-of-freedom manipulator 11 are determined each time the picking is performed, which can increase the success rate of picking.

[0057] Optionally, the location information of the target fruit includes its own information and spacing information, wherein the own information is the coordinates of its own center point, and the spacing information is the distance to adjacent fruits;

[0058] The steps for obtaining the location information of the target fruit include:

[0059] Based on the deep learning algorithm, the detection frame of each target fruit is identified and the center point coordinates of each detection frame are obtained; the center point coordinates of the detection frame are the center point coordinates of the fruit itself;

[0060] Based on the center point coordinates of each detection frame, the distance between adjacent detection frames is obtained; wherein the distance between adjacent detection frames is the distance information;

[0061] The step of adjusting the posture of the six-degree-of-freedom manipulator 11 according to the position information of the target fruit until the target fruit enters the gripping range of the gripper 14 includes:

[0062] Determine whether the distance between adjacent detection frames is less than the thickness of the claws of the clamping jaws 14;

[0063] If it is less than, the end rotation angle of the six-degree-of-freedom manipulator 11 is adjusted according to a preset angle; wherein the preset angle is the angle between the center point connection line of two adjacent detection frames and the horizontal plane.

[0064] In the above scheme, the distance between the target fruits is calculated based on the deep learning algorithm, and the rotation angle of the six-degree-of-freedom manipulator 11 is obtained. When two target fruits are very close to each other, the gripper 14 can rotate a certain angle with the six-degree-of-freedom manipulator 11 so as to pick the target fruits without damaging them, and the target fruits will not be pushed up by the gripper 14, causing the coordinates to change.

[0065] Optionally, after inputting the fruit tree image into a preset target fruit recognition model and obtaining a target fruit recognition result, the method further includes: if the target fruit is not found, controlling the six-degree-of-freedom manipulator 11 to move backward and / or upward until the target fruit is detected. This configuration ensures that the six-degree-of-freedom manipulator 11 can pick the target fruit every time, thereby improving picking efficiency.

[0066] Optionally, before the step of generating a planned path based on the current position of the mobile vehicle 10, the current position to be picked, a preset orchard map, and path planning rules, the following steps may be further included:

[0067] Control the movement of the six-degree-of-freedom manipulator 11 to obtain different preset groups of first pose coordinates and different preset groups of second pose coordinates; the first pose information is the pose coordinates of the six-degree-of-freedom manipulator 11, and the second pose information is the pose coordinates of the calibration plate in the coordinate system of the camera 15;

[0068] Based on inverse kinematics, the coordinates corresponding to the first pose of the preset group are solved. One-to-one correspondence with the second pose coordinates of the preset group in, is the transformation matrix from the end effector coordinate system of the six-degree-of-freedom manipulator 11 to the base coordinate system of the six-degree-of-freedom manipulator 11, is the conversion matrix from the calibration plate coordinate system to the camera 15 coordinate system;

[0069] Based on the formula Solution Among them, P b represents the coordinates of the base coordinate system of the six-degree-of-freedom manipulator 11, P t Represents the coordinates of the calibration plate coordinate system, P b With P t All are fixed values.

[0070] Optional, based on formula Solution The steps include:

[0071] based on Will Convert to AX=XB, where P t The inverse matrix of B is X is

[0072] Solve X based on the Tsai-Lenz algorithm.

[0073] Among them, there are at least 13 sets of first pose coordinates and second pose coordinates. are all transformation matrices corresponding to the first set of pose coordinates, and so on. are all transformation matrices corresponding to the second set of pose coordinates. The above settings implement calibration between the 6-DOF manipulator 11 and the camera 15 to facilitate conversion between coordinate systems, thereby accurately obtaining the coordinates of the end effector of the 6-DOF manipulator 11 and the coordinates of the camera 15 image, providing accurate positioning information for subsequent control and operation.

[0074] Optionally, after the step of clamping the target fruit, the method further comprises:

[0075] Detecting the pressure on the clamping jaw 14;

[0076] Determine whether the pressure is greater than the preset value;

[0077] If yes, it is determined that the picking is successful and the next step is executed.

[0078] Through the above arrangement, the picking effect of target fruits can be judged in real time, and misjudgment caused by placing the fruits into the fruit storage box 13 after they are picked can be avoided, thereby ensuring a higher utilization rate of the fruit storage box 13.

[0079] Optionally, the step of adjusting the posture of the six-degree-of-freedom manipulator 11 until the target fruit clamped by the gripper 14 enters the storage area of ​​the fruit storage box 13 includes:

[0080] Based on the artificial potential field method, the industrial computer 12 serving as a fixed obstacle adjusts the posture of the six-degree-of-freedom manipulator 11 until the target fruit clamped by the gripper 14 enters the storage area of ​​the fruit storage box 13 .

[0081] The artificial potential field method can be used to implement path planning for obstacle avoidance of the six-degree-of-freedom manipulator 11, thereby preventing the arm of the six-degree-of-freedom manipulator 11 from colliding with the industrial computer 12 during the process of storing the target fruit.

[0082] Optionally, before the step of generating a planned path based on the current position of the mobile vehicle 10, the current position to be picked, a preset orchard map, and path planning rules, the robot use method further includes:

[0083] Drive the laser radar 16 to move in the orchard and collect 3D point cloud data of the orchard;

[0084] Generate an orchard map based on the three-dimensional point cloud data; wherein the orchard map includes the location information of the starting point and the end point;

[0085] Collect training images and corresponding training labels; the training images contain target fruits and interference backgrounds;

[0086] The YOLOv5 network is trained based on the training images and corresponding training labels to obtain the target fruit recognition model.

[0087] The above setup enables the collection of orchard point cloud data and map generation based on LiDAR 16, as well as the training of a target fruit recognition model. This provides important technical support for subsequent orchard operations and fruit picking, such as automated navigation, target positioning, and recognition. This improves the efficiency and accuracy of orchard management and operations, and enhances the robustness of the target fruit recognition model.

[0088] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for using a robot, applied to a robot that recognizes and picks fruit based on vision, wherein the robot comprises a mobile cart, a six-degree-of-freedom manipulator, an industrial computer, a fruit storage box, a gripper, and a camera; the six-degree-of-freedom manipulator, the industrial computer, and the fruit storage box are all mounted on the mobile cart; the gripper and the camera are both mounted on the end of the six-degree-of-freedom manipulator; and the method is characterized in that: The robot usage method comprises the following steps: Generate a planned path based on the current position of the mobile cart, the current position to be picked, a preset orchard map, and path planning rules; the starting point of the planned path is the current position of the mobile cart, and the end point is the current position to be picked; Move to the current picking position along the planned path; Take an image of the fruit tree at the current location to be picked; Input the fruit tree image into the preset target fruit recognition model to obtain the target fruit recognition result; If there is a target fruit, obtain the position information of the target fruit, and adjust the posture of the six-degree-of-freedom manipulator according to the position information of the target fruit until the target fruit enters the gripping range of the gripper; Clamping the target fruit, and adjusting the posture of the six-degree-of-freedom manipulator until the target fruit clamped by the clamp enters the storage area of ​​the fruit storage box; Release the trapped target fruit; Determine whether there is a next position to be picked, and if so, set the next position to be picked as the current position to be picked, and jump to the step of generating a planned path according to the current position of the mobile vehicle, the current position to be picked, the preset orchard map and the path planning rules, until there is no next position to be picked; The location information of the target fruit includes its own information and spacing information. The own information is the coordinates of its own center point, and the spacing information is the distance to the adjacent fruits. The step of obtaining the location information of the target fruit comprises: Based on the deep learning algorithm, the detection frame of each target fruit is identified and the center point coordinates of each detection frame are obtained; the center point coordinates of the detection frame are the center point coordinates of the fruit itself; Based on the center point coordinates of each detection frame, the distance between adjacent detection frames is obtained; wherein the distance between adjacent detection frames is the distance information; The step of adjusting the posture of the six-degree-of-freedom manipulator according to the position information of the target fruit until the target fruit enters the gripping range of the gripper comprises: Determining whether a distance between adjacent detection frames is less than a thickness of the gripper; If it is less than, the end rotation angle of the six-degree-of-freedom manipulator is adjusted according to a preset angle; wherein the preset angle is the angle between the line connecting the center points of two adjacent detection frames and the horizontal plane.

2. A method for using a robot according to claim 1, characterized in that: After the step of inputting the fruit tree image into a preset target fruit recognition model to obtain the target fruit recognition result, the method further includes: if there is no target fruit, controlling the six-degree-of-freedom manipulator to move backward and / or upward until the target fruit is detected.

3. A method for using a robot according to claim 1, characterized in that: Before the step of generating a planned path based on the current position of the mobile vehicle, the current position to be picked, a preset orchard map, and path planning rules, the method further includes: Controlling the movement of the six-degree-of-freedom manipulator to obtain different preset groups of first pose coordinates and different preset groups of second pose coordinates; the first pose information is the pose coordinates of the six-degree-of-freedom manipulator, and the second pose information is the pose coordinates of the calibration plate in the coordinate system of the camera; Based on inverse kinematics, the coordinates corresponding to the first pose of the preset group are solved. , corresponding to the second pose coordinates of the preset group ;in, is the transformation matrix from the end effector coordinate system of the six-degree-of-freedom manipulator to the base coordinate system of the six-degree-of-freedom manipulator, is the conversion matrix from the calibration plate coordinate system to the camera coordinate system; Based on the formula , solve ,in, represents the coordinates of the base coordinate system of the six-degree-of-freedom manipulator, represents the coordinates of the calibration plate coordinate system, and All are fixed values.

4. A method for using a robot according to claim 3, characterized in that: The formula based , solve The steps include: based on ,Will Convert to AX=XB, where for The inverse matrix of , B is , X is ; Solve X based on the Tsai-Lenz algorithm.

5. A method for using a robot according to claim 1, characterized in that: After the step of clamping the target fruit, the method further comprises: detecting the pressure exerted on the clamping jaws; Determine whether the pressure is greater than the preset value; If yes, it is determined that the picking is successful and the next step is executed.

6. A method for using a robot according to claim 1, characterized in that: The step of adjusting the posture of the six-degree-of-freedom manipulator until the target fruit clamped by the gripper enters the storage area of ​​the fruit storage box includes: Based on the artificial potential field method, the industrial computer serving as a fixed obstacle adjusts the posture of the six-degree-of-freedom manipulator until the target fruit clamped by the gripper enters the storage area of ​​the fruit storage box.

7. A method for using a robot according to claim 1, characterized in that: Before the step of generating a planned path based on the current position of the mobile vehicle, the current position to be picked, the preset orchard map, and the path planning rules, the robot use method further includes: Drive the LiDAR to move in the orchard and collect 3D point cloud data of the orchard; Generate an orchard map based on the three-dimensional point cloud data; wherein the orchard map includes the location information of the starting point and the end point; Collect training images and corresponding training labels; the training images contain target fruits and interference backgrounds; The YOLOv5 network is trained based on the training images and corresponding training labels to obtain the target fruit recognition model.

Images