Mobile two-arm robotic system for performing grafts on plants and method of implementation

Abstract

The present invention relates to a mobile two-arm robotic system for performing grafts on plants comprising a navigation system (2), a perception system (3), a central control module (9) and a two-arm robotic handler (4) comprising a first arm with a parallel clamp (5) comprising two gripping fingers (15), a second arm with a multipurpose clamp (6) comprising cutter fingers with a blade (17) to perform cuts on the plants (12, 13) and a press area (19), to handle a graft clip (14) intended to be positioned on the cut plants (12, 13).

Description

[0001] DOUBLE-ARM MOBILE ROBOTIC SYSTEM FOR GRAFTING PLANTS AND PROCEDURE FOR GRAFTING

[0002] OBJECT OF THE INVENTION

[0003] The present invention relates to a robotic system that incorporates a navigation system, a perception system, a double-arm manipulator system with gripping and cutting pliers, a grafting clip and a series of control modules of the previous systems, intended for the automatic grafting of some variety plants onto other rootstock plants for the generation of new plants.

[0004] The invention also relates to the method of performing grafts using the aforementioned double-arm mobile robotic system.

[0005] BACKGROUND OF THE INVENTION

[0006] As in any technological sector, the use of automated manipulators for mechanical activities is widespread in sectors related to agriculture and gardening.

[0007] Thus, processes requiring large amounts of labor have been the first to require this implementation, due to its rapid payback.

[0008] Automated techniques using robots have also been implemented in processes focused on plant reproduction through grafting.

[0009] Document CN109792888 relates to an apple-picking robot comprising an automatic mobile platform with a robot body, the upper portion of which is provided with binocular vision, and a fruit-picking system. A central control unit controls all elements of the robot. Furthermore, the invention describes a method of operating the robot for harvesting. The robot has strong adaptability, flexible movement, safety, reliability, and a compact structure, and has apple-picking and graded harvesting capabilities. Document CN11 1149534 discloses a two-armed mobile robot for pruning branches of fruit trees. The robot comprises tracked driving devices for moving with broad maneuverability, providing a high capacity to cross obstacles and navigate through difficult terrain.It also incorporates a machine body with pruning systems, a control system, and a visual recognition system, so that the control system recognizes the branches that need pruning and the pruning systems perform the pruning automatically and intelligently.

[0010] Document CN1 13924871 A relates to a strip-array tree-cutting robot, which can perform automatic cutting by automatically loading seedling plates and combining machine vision technology to identify roots and stems, thereby greatly freeing up labor. To this end, it comprises an autonomous driving vehicle (AGV) with a cutting unit and a camera for obtaining an image with spatial absolute position coordinate information. The AGV is provided with a power supply unit and a navigation unit for guiding the cutting robot to a position. The device comprises a cutting holder and a multifunctional gripper, and is used to hold the axes and cut them at appropriate positions using machine vision technology for the identification of the rootstock of the axis and being able to achieve automatic cutting.

[0011] Document US2023263100A1 discloses a machine equipped with a pair of robotic arms, a conveyor belt that supplies plants to be grafted, a camera system, and machining devices that shape the ends of the stems to be grafted. The arms operate independently, without interacting with each other.

[0012] However, none of the documents found in the state of the art are focused on providing an automatic solution related to plant varieties with a certain characteristic intended to be grafted onto another type of rootstock for the generation and growth of the new plant.

[0013] DESCRIPTION OF THE INVENTION

[0014] The invention relates to a mobile, double-arm robotic system for grafting plants. The plants must already have cotyledonous leaves or a cotyledonary node in the stem, such that an epicotyl (the area above) and a hypocotyl (the area below) are defined.

[0015] The robotic system comprises the following elements:

[0016] - a mobile platform with a plurality of wheels individually controlled by a first control module,

[0017] - a navigation system connected to the first control module,

[0018] - a perception system connected to the central control module,

[0019] - a double-arm robotic manipulator located on the mobile platform and controlled by a second control module,

[0020] - a central control module comprising a computer and a WiFi module for communication with the outside, connected to the first control module and the second control module.

[0021] The two arms of the robotic manipulator are differentiated to perform different functions.

[0022] Thus, on the one hand, a first arm comprises a parallel clamp with two gripping fingers that can incorporate teeth to make the gripping of the plants more effective.

[0023] On the other hand, a second arm comprises a multi-tool pliers that in turn incorporates a first cutting finger, comprising a blade and a gripping area, and a second cutting finger comprising a slit and also a gripping area.

[0024] In this way, when the multi-tool pliers are closed, the blade and the groove join together complementarily, inserting the blade into the groove to perform cutting functions on plants.

[0025] Furthermore, the gripping areas also join together to perform grafting clip manipulation functions, grasping it, opening it to position it on the cut plants, and then releasing it to leave it as part of the graft.

[0026] Furthermore, the clamping areas can have a rectangular configuration, with the blade positioned at a certain angle relative to the longer sides. This allows the cut to be made not transversely, but rather at an angle, allowing the cut sections of the stems intended for grafting to come into contact, allowing for more effective grafting. This angle is preferably 45 or 60 degrees.

[0027] Preferably, the perception system comprises a color camera, a time-of-flight camera, a pan-tilt unit and a processing module.

[0028] The invention also focuses on a method for grafting plants comprising cotyledonous leaves or a cotyledonary node that define an epicotyl and a hypocotyl using the robotic system described above.

[0029] The procedure comprises the following steps: a) Detecting, using the perception system, a variety plant and a rootstock plant of certain dimensions in terms of the diameter and length of the corresponding hypocotyls; b) Positioning the robotic system, using the navigation system, in a location accessible to the different plants with the forceps; c) Selecting a rootstock gripping point on the epicotyl of the rootstock plant and positioning the parallel forceps over the rootstock gripping point; d) Selecting a rootstock cutting point on the hypocotyl of the rootstock plant and positioning the multi-tool forceps over the rootstock cutting point; e) Closing the parallel forceps to hold the rootstock plant; f) Closing the multi-tool forceps to cut the rootstock plant; g) Opening the multi-tool forceps to release the already cut rootstock plant; h) Withdrawing the arm of the parallel forceps and opening it to dispose of the cut plant;(i) Select a variety gripping point above the epicotyl of the variety plant and position the parallel pliers over the variety gripping point; (j) Select a variety cutting point on the epicotyl of the variety plant and position the multi-tool pliers over the variety cutting point; (k) Close the parallel pliers to hold the variety plant; (l) Close the multi-tool pliers to make a cut section in the variety plant; (m) Open the multi-tool pliers to release the already cut variety plant; (n) Grab the grafting clip with the gripping area of ​​the multi-tool pliers; (o) Position the epicotyl of the variety plant aligned on the hypocotyl of the rootstock plant with the cut sections facing each other; (p) Place the grafting clip at the junction of the epicotyl of the variety plant with the hypocotyl of the rootstock plant; (q) Open the multi-tool pliers to release the grafting clip, holding the two plants;r) Open the parallel clamp to release the variety plant.;

[0030] The robotic system can be located in proximity to a rootstock plant and a variety plant to perform the graft, with both plants within reach of the robotic arm or be in proximity to each of them, having to move to position itself in proximity to each of the plants as necessary.

[0031] This procedure would be repeated to perform as many grafts as necessary.

[0032] In the described procedure, preferably the hypocotyls of the selected rootstock and variety plants have diameters that differ less than 10% in size and the lengths are at least 5 cm.

[0033] Preferably, the pattern cutting point is located at least one centimeter below the cotyledonary leaves or the cotyledonary node and the pattern gripping point is located at least two centimeters above the cotyledonary leaves or the cotyledonary node.

[0034] Also preferably, the variety cut point is located at least one centimeter below the upper limit of the epicotyl and the variety grip point is located at least two centimeters above the epicotyl.

[0035] Also preferably, measurements are taken by synchronizing the images from the color camera and the time-of-flight camera.

[0036] DESCRIPTION OF THE DRAWINGS

[0037] To complement the description being made and in order to help better understand the characteristics of the invention, in accordance with a preferred example of practical embodiment thereof, a set of drawings is attached as an integral part of said description, in which the following has been represented for illustrative and non-limiting purposes: Figure 1.- Represents a general view of the double-arm mobile robotic system for grafting plants.

[0038] Figure 2.- Represents the robotic arm locating the origin of the reference system of the RRF robotic system and the reference system of the PRF workspace perception system.

[0039] Figure 3.- Represents the parallel gripper of the double-arm robotic manipulator.

[0040] Figure 4.- Represents the multi-tool grafting gripper of the double-arm robotic manipulator.

[0041] Figure 5.- Represents the first cutting finger of the multi-tool grafting pliers that has a blade and a gripping area.

[0042] Figure 6.- Represents a second cutting finger of the multi-tool grafting pliers that has a slit where the cutting blade of the first cutting finger fits completely when the multi-tool pliers are closed and a gripping area.

[0043] Figure 7.- Represents a rootstock plant and a variety plant.

[0044] Figure 8.- Represents a pattern plant with the corresponding cutting and gripping points, as well as the parallel and multi-tool pliers used for gripping and cutting, respectively.

[0045] Figure 9.- Represents a variety of plant with the corresponding cutting and gripping points, as well as the tweezers used for cutting and gripping.

[0046] Figure 10.- Represents a graft clip.

[0047] Figure 11 .- Represents the union of the hypocotyl of the rootstock plant with the epicotyl of the variety plant by means of the graft clip.

[0048] PREFERRED EMBODIMENT OF THE INVENTION

[0049] A preferred embodiment of the dual-arm robotic system for grafting plants according to the present invention is described below with the aid of Figures 1 to 11. Figure 1 represents a general view of the dual-arm robotic system comprising the following elements:

[0050] - a mobile platform (1);

[0051] - a navigation system (2);

[0052] - a perception system (3);

[0053] - a double-arm robotic manipulator (4);

[0054] - a parallel gripper (5), installed on one of the robot arms;

[0055] - a multi-tool gripper (6), installed on the other arm of the robot; and

[0056] - a central control module (9).

[0057] The mobile platform (1) has four individually controlled wheels and incorporates a first control module (7) based on a computer.

[0058] The navigation system (2) focuses on environmental perception. It includes a LiDAR, a color camera, a time-of-flight camera, a GPS, and a processing module implemented in ROS (Robot Operating System). All of these elements are installed on the front of the mobile platform (1).

[0059] The perception system (3) is focused on the perception of the workspace for grafting by detecting the plants and identifying the cutting and grasping points. It consists of a pan-tilt unit, a color camera, a time-of-flight camera, and a processing module implemented in ROS, which is responsible for synchronizing the images from the color camera and the time-of-flight camera, and for recording the data provided by both cameras, so that the information is in the same reference frame and there is a direct correspondence between the color information and the spatial information. The origin of the PRF reference system, shown in an overview in Figure 2, is located at the midpoint of the frontal plane of the time-of-flight camera. The color camera is placed above the time-of-flight camera so that their optical axes are parallel and their frontal planes are as aligned as possible.The assembly consisting of the color camera and the time-of-flight camera are installed in the pan-tilt unit, on a profile that is fixed to the front midpoint of the base of the double-arm robotic manipulator (4), point at which the origin of the reference system of the RRF robotic system is located, also represented in a general view in figure 2. The double-arm robotic manipulator (4) has 7 degrees of freedom in each arm and a second integrated control module (8). The central control module (9), implemented in ROS, generates the collision-free trajectories of the arms and communicates with the second control module (8) of the robot for the execution of the movements.

[0060] The parallel gripper (5) is shown in a general view in figure 3. It is installed on one of the robot arms and has two gripping fingers (15) in the shape of an isosceles trapezoidal prism, and positioned so that the rectangular faces corresponding to the larger bases face each other in parallel. In addition, these larger bases have a series of millimetric teeth (16) to facilitate the gripping of the epicotyl of the plants (12, 13).

[0061] The multi-tool gripper (6) is shown in a general view in figure 4. It is installed on the other arm of the robot. This multi-tool gripper (6) has two cutting fingers (10, 11 ): a first cutting finger (10), which incorporates a blade (17) and a second cutting finger (11), which incorporates a slit (18), for cutting the hypocotyl and epicotyl of the plants (12, 13). Each of the cutting fingers (10, 11 ) also incorporates a gripping area (19) at the most protruding end, for holding a grafting clip (14).

[0062] The blade (17) is made of stainless steel and is placed on the inner face of the first cutting finger (10), forming a 45° angle. e or 60 ewith respect to the longitudinal axis of said finger (10) as can be seen in figure 5. The slit (18) of the second cutting finger (11) is shown in figure 6 and is complementary to the blade (17) so that it fits when the multi-tool clamp (6) is closed.

[0063] The central control module (9) is coupled to the rear of the mobile platform (1) and consists of a computer and a WiFi module for communication with the outside.

[0064] In addition, the robotic system also incorporates a set of batteries, not shown in the figures, which is coupled to the mobile platform (1) to provide the required energy autonomy.

[0065] In order to control the double-arm robotic system of the invention, a user, through a remote interface, such as a computer or a tablet, indicates the location of the work area where the plants (12, 13) are located and the number of grafts to be performed. This interface communicates with the central control module (9), through the WiFi module for communication with the outside, and initializes the robotic system. Next, the central control module (9), by means of the first control module (7), and with the information provided by the navigation system (2), positions the robotic system in such a way that the plants (12, 13) are within the robot's manipulation space. The navigation system (2) is responsible for detecting possible obstacles in the robot's environment and with this information, the first control module (7) calculates the robot's trajectory from the current position to the working position, avoiding collisions with the detected obstacles.

[0066] Once the robot has reached the working position, the perception system (3), by means of the processing module it incorporates, selects a rootstock plant (13) and a variety plant (12) that have a hypocotyl of diameter (D) and a length (Lh), as indicated in figure 7, where (Lh) is at least 5 cm. To carry out this selection, the perception system (3) uses at least one image from the color camera to detect the plants (12, 13) and the different elements that compose it (cotyledonous leaves, cotyledonary node, hypocotyl, epicotyl and first true leaves) and at least one point cloud provided by the time-of-flight camera and acquired synchronously with the image from the color camera to estimate the diameter (D) and length (Lh) of the hypocotyls of the plants (12, 13).The processing module implemented in ROS is responsible for this synchronous data acquisition and for recording the data provided by both cameras, ensuring that the information is in the same reference frame and that there is a direct correspondence between the color and spatial information. In this case, it is the color information that is transferred to the time-of-flight camera's reference frame. Therefore, each point in the 3D cloud has an associated reflectance level.

[0067] The detection of the plants (12, 13) is carried out in the processing module implemented in ROS, from the registered color image, first applying a real-time convolutional neural network, such as YOLO (You Only Look Once), and then a semantic segmentation at the pixel level. With the detected pixels of the hypocotyls and epicotyls, and using the 3D cloud points associated with these pixels, the length (Lh) of the hypocotyls and the diameter (D) of the hypocotyls and epicotyls are calculated, selecting a rootstock plant (13) whose hypocotyl has a diameter (D) equal to the epicotyl of the variety plant (12), and that both the rootstock plant (13) and the variety plant (12) have a hypocotyl with a length (Lh) of at least 5 cm.

[0068] Once a rootstock plant (13) and a variety plant (12) have been selected, the corresponding grasping and cutting points are also calculated for each of them. In the rootstock plant (13), shown in a general view in Figure 8, the cutting point (PCp) is located on the hypocotyl, one centimeter below the cotyledonous leaves or the cotyledonary node, and the grasping point (PAp) is located on the epicotyl, 2 cm above the cotyledonous leaves or the cotyledonary node. In the variety plant (12), shown in a general view in Figure 9, the cutting point (PCv) is located on the epicotyl, at least one centimeter below its upper limit, and the grasping point (PAv) is located on the stem, above the epicotyl.The processing module communicates all these positions to the central control module (9), which is responsible for transforming the coordinates of these points from the PRF reference system to the RRF reference system using a homogeneous transformation matrix, and for planning the trajectories of the manipulator arms (4), so that the arm with the parallel gripper (5) goes to the gripping point (PAp) and the arm with the multi-tool gripper (6) goes to the cutting point (PCp), avoiding collisions between the arms. To generate the trajectories, the central control module (9) uses a planning algorithm, such as the RTT-Connect algorithm.

[0069] Once the trajectories have been calculated, the central control module (9) sends the commands for the execution of the trajectories to the second control module (8), as well as the commands for closing the grippers (5, 6). In this way, the gripping at point (PAp) is produced with the parallel gripper (5) and the cutting at point (PCp) of the pattern plant (13) with the blade (17) of the multi-tool gripper (6).

[0070] The central control module (9) then calculates the trajectory of the arm holding the top of the cut rootstock (13) by the epicotyl so that it places it aside, and sends the corresponding commands to the second control module (8) to execute the action. Again, the trajectory of the arm is generated using a planning algorithm, such as the RTT-Connect algorithm.

[0071] Next, the trajectories of the manipulator arms are also planned, so that the arm with the parallel gripper (5) goes to the gripping point (PAv) and the arm with the multi-tool gripper (6) goes to the cutting point (PCv), avoiding collisions between the arms.

[0072] The central control module (9) sends the commands for the execution of the trajectories to the second control module (8), as well as the commands for closing the pliers (5, 6). In this way, the grip at point (PAv) is produced with the parallel pliers (5) and the cut at point (PCv) of the variety plant (12) with the multi-tool pliers (6). Once this step is completed, the central control module (9) sends the command for opening the multi-tool pliers (6) to the second control module (8). The perception system (3) detects, from the recorded color image, the graft clip (14), shown in an overview in Figure 10, first by applying a real-time convolutional neural network, such as YOLO (You Only Look Once), and then a semantic segmentation at the pixel level. With the detected pixels, and using the 3D cloud points associated with these pixels, the holding position of the graft clip is calculated (14).The processing module communicates this position to the central control module (9), which is responsible for transforming the coordinates of this point from the PRF reference system to the RRF reference system using a homogeneous transformation matrix, and for planning the trajectory of the arm with the multi-tool gripper (6) so that it positions the gripping area (19) at the calculated point. Once the trajectory has been calculated, using a planning algorithm such as the RTT-Connect algorithm, the central control module (9) sends the corresponding commands to the second control module (8) to execute the movement, as well as the command for closing the multi-tool gripper (6). In this way, the multi-tool gripper (6) holds the graft clip (14) with the gripping area (19).Next, the central control module (9) calculates, using the same method mentioned above, the trajectory of the arm holding the upper part of the variety plant (12) that has been cut previously by the epicotyl so that it places it on top of the hypocotyl of the rootstock plant (13), so that the epicotyl of the variety plant (12) is completely aligned with the hypocotyl of the rootstock plant (13) (figure 11). In the same way, it calculates the trajectory of the arm with the multi-tool pliers (6) so that it places the grafting clip (14) at the junction of the epicotyl of the variety plant (12) with the hypocotyl of the rootstock plant (13). Once the trajectories have been generated, the central control module (9) sends the corresponding commands to the second control module (8) to execute the movements, as well as the command for opening the multi-tool pliers (6).In this way, the grafting clip (14) is placed at the junction of the epicotyl of the variety plant (12) with the hypocotyl of the rootstock plant (13). Finally, the central control module (9) sends the command to the second control module (8) to open the parallel clamp (5).

[0073] To start with another graft, the robotic system selects again, through the processing module of the perception system (3), another rootstock plant (13) and another variety plant (12) that have a hypocotyl with a diameter (D) and a length (Lh) of at least 5 cm, and the steps described are repeated.

Claims

1. Mobile double-arm robotic system for grafting plants comprising: - a mobile platform (1) with a plurality of wheels individually controlled by a first control module (7), - a navigation system (2) connected to the first control module (7), - a double-arm robotic manipulator (4) located on the mobile platform (1) and controlled by a second control module (8), - a central control module (9) comprising a computer and a WiFi module for communication with the outside, connected to the first control module (7) and the second control module (8), - a perception system (3) connected to the central control module (9), the robotic system being characterized in that the double-arm robotic manipulator (4) comprises: - a first arm with a parallel gripper (5) comprising two gripping fingers (15), - a second arm with a multi-tool clamp (6) comprising: - a first cutting finger (10), comprising a blade (17) and a gripping area (19), and - a second cutting finger (11), comprising a slit (18) and a gripping area (19), where, when the multi-tool clamp (6) is closed: - the blade (17) and the slit (18) are configured to join complementarily, the blade (17) being inserted into the slit (18) and thus being able to make cuts in the plants (12, 13), and - the gripping areas (19) are configured to join together complementarily in order to manipulate a grafting clip (14) intended to be positioned on the cut plants (12, 13).

2. The robotic system of claim 1, wherein the gripping fingers (15) comprise teeth (1).

3. The robotic system of claim 1, wherein the gripping areas (19) have a rectangular configuration and the blade (17) has an inclination with respect to the longer sides.

4. The robotic system of claim 3, wherein the inclination is selectable between 45 and 60 degrees.

5. The robotic system of claim 1, wherein the perception system (3) comprises a color camera, a time-of-flight camera, a pan-tilt unit, and a processing module.

6. Method for grafting plants comprising cotyledonous leaves or a cotyledonary node defining an epicotyl and a hypocotyl by means of the robotic system of the preceding claims, characterized in that it comprises the following steps: a) Detecting, by means of the perception system (3), a variety plant (12) and a rootstock plant (13) of certain dimensions in terms of the diameter (D) and the length (Lh) of the corresponding hypocotyls; b) Positioning the robotic system, by means of the navigation system (2), in a place accessible to the different plants (12, 13) with the forceps (5, 6); c) Selecting a rootstock grip point (PAp) on the epicotyl of the rootstock plant (13) and positioning the parallel forceps (5) over the rootstock grip point (PAp); d) Select a pattern cutting point (PCp) on the hypocotyl of the pattern plant (13) and position the multi-tool forceps (6) over the pattern cutting point (PCp);e) Close the parallel pliers (5) to hold the rootstock plant (13); f) Close the multi-tool pliers (6) to make a cut section in the rootstock plant (13); g) Open the multi-tool pliers (6) to release the already cut rootstock plant (13); h) Withdraw the arm of the parallel pliers (5) and open it to get rid of the cut plant; i) Select a variety gripping point (PAv) above the epicotyl of the variety plant (12) and position the parallel pliers (5) over the variety gripping point (PAv); j) Select a variety cutting point (PCv) on the epicotyl of the variety plant (12) and position the multi-tool pliers (6) over the variety cutting point (PCv); k) Close the parallel pliers (5) to hold the variety plant (12); l) Close the multi-tool pliers (6) to make a cut section in the variety plant (12); m) Open the multi-tool pliers (6) to release the already cut variety plant (12);n) Grasp the grafting clip (14) with the gripping area (19) of the multi-tool pliers (6); o) Position the epicotyl of the variety plant (12) aligned on the hypocotyl of the rootstock plant (13) with the cut sections facing each other; p) Place the grafting clip (14) at the junction of the epicotyl of the variety plant (12) with the hypocotyl of the rootstock plant (13); q) Open the multi-tool pliers (6) to release the grafting clip (14) holding the two plants (12, 13); r) Open the parallel pliers (5) to release the variety plant (12).

7. The method of grafting plants according to claim 6, wherein the hypocotyls of the selected plants (12, 13) have lengths (Lh) of at least 5 cm and have diameters (D) that differ in size by less than 10%.

8. The method for grafting plants according to claim 6, wherein the pattern cutting point (PCp) is located at least one centimeter below the cotyledonous leaves or the cotyledonary node of the pattern plant (13) and the pattern gripping point (PAp) is located at least two centimeters above the cotyledonous leaves or the cotyledonary node.

9. The method for grafting plants according to claim 6, wherein the variety cutting point (PCv) is located at least one centimeter below the upper limit of the epicotyl of the variety plant (12) and the variety gripping point (PAv) is located at least two centimeters above the epicotyl.

10. The method for grafting plants according to claims 7 to 9, wherein the measurements are taken by synchronizing the images of the color camera and the time-of-flight camera of the perception system (3).

Images