A method and system for seedling transplantation based on dual-arm collaborative operation

Abstract

This invention provides a seedling transplantation method and system based on dual-arm collaborative operation, belonging to the field of seedling transplantation. The method includes: conveying a seedling to a location to be identified and acquiring a first image of the seedling; performing target detection on the first image based on a target detection network to determine root detection boxes and stem detection boxes in the first image; determining a gripping point based on the root and stem detection boxes in the first image and controlling a first robotic arm to grip the seedling at the gripping point; acquiring a second image of the seedling on the first robotic arm; performing target detection on the second image based on a target detection network to determine root and stem detection boxes in the second image; determining a cutting point based on the root and stem detection boxes in the second image and controlling a second robotic arm to cut off the stem of the seedling on the first robotic arm at the cutting point; and controlling the first robotic arm to plant the root of the seedling into a petri dish. This invention improves the efficiency and survival rate of seedling transplantation.

Description

Technical Field

[0001] This invention relates to the field of seedling transplantation, and in particular to a seedling transplantation method and system based on dual-arm collaborative operation. Background Technology

[0002] With the rapid development of science and technology, Phalaenopsis orchid tissue culture technology has gradually matured. As an asexual reproduction technique, it has many advantages, such as short cultivation cycle, high survival rate, high yield, and reduced production costs, and has a promising future. Currently, while Phalaenopsis orchid tissue culture technology is quite mature, the transplantation of seedlings is still mainly done manually, especially the root cutting and transplanting. Furthermore, the transplantation of Phalaenopsis orchid seedlings requires a sterile environment, and excessive manual intervention may increase the chance of seedling disease, resulting in a lower survival rate and causing economic and human losses. Therefore, it is necessary to design a complete system based on dual-machine collaborative operation to fully automate the operations of Phalaenopsis orchid seedling delivery, target identification, grasping, cutting, planting, and related path planning.

[0003] Existing methods generally focus on a specific part of the overall system, such as optimizing target detection models to improve the detection accuracy of roots, stems, and leaves of seedlings; or using different trajectory planning algorithms to make the trajectory of the robotic arm more consistent with actual needs. However, in the field of plant seedling transplantation, existing methods rarely design an entire transplantation system for Phalaenopsis orchid seedlings, and there are no mature seedling transplantation systems available. The automation level of related methods is also low, requiring significant manual intervention. Summary of the Invention

[0004] The purpose of this invention is to provide a seedling transplantation method and system based on dual-arm collaborative operation, which can improve the efficiency of seedling transplantation and the survival rate of seedlings.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A seedling transplanting method based on dual-arm collaborative operation includes:

[0007] The seedlings are transported to the location to be identified, and the first image of the seedlings is captured.

[0008] The first image is used to perform object detection based on an object detection network to determine the root detection box and stem detection box in the first image; the object detection network is obtained by pre-training ShuffleNet v2-YOLOv5 with a training sample set; the training sample set includes multiple training sample images containing annotation information of seedling roots, stems, leaves and tubers.

[0009] The gripping point is determined based on the root detection box and stem detection box in the first image, and the first robotic arm is controlled to grip the seedling at the gripping point.

[0010] Acquire a second image of the seedling on the first robotic arm;

[0011] The second image is subjected to target detection based on the target detection network to determine the root detection box and the stem detection box in the second image;

[0012] The cutting point is determined based on the root detection box and stem detection box in the second image, and the second robotic arm is controlled to cut off the stem of the seedling on the first robotic arm at the cutting point.

[0013] The first robotic arm is controlled to plant the roots of the seedlings into the petri dish.

[0014] Optionally, the gripping point is determined based on the root detection box and the stem detection box in the first image, specifically including:

[0015] Determine whether both root detection boxes and stem detection boxes exist simultaneously in the first image. If only root detection boxes exist, control the first robotic arm to plant the roots of the seedling into the petri dish; if only stem detection boxes exist, discard the seedling.

[0016] If both root detection boxes and stem detection boxes exist simultaneously, then the largest connected component in the root detection box is determined.

[0017] The radius of the circumcircle of the root detection frame is determined based on the width and height of the root detection frame.

[0018] Based on the radius of the circumcircle of the root detection frame and the maximum connected region, determine the fitted straight line of the connected region;

[0019] The circumcircle of the root detection frame is determined based on the coordinates of the center point of the root detection frame and the radius of the circumcircle of the root detection frame.

[0020] Based on the connected component fitting line and the circumcircle of the root detection box, determine the x-axis coordinates of the first intersection point and the second intersection point of the connected component fitting line and the circumcircle.

[0021] Based on the fitted straight line of the connected component, the x-axis coordinate of the first intersection point, and the x-axis coordinate of the second intersection point, determine the y-axis coordinates of the first intersection point and the second intersection point;

[0022] Based on the x-axis coordinates of the first intersection point, the y-axis coordinates of the first intersection point, the x-axis coordinates of the second intersection point, the y-axis coordinates of the second intersection point, and the stem detection frame, the gripping point in the camera coordinate system is determined, and the gripping point in the camera coordinate system is converted into the gripping point in the first robotic arm base coordinate system.

[0023] Optionally, determining the largest connected component in the root detection frame specifically includes:

[0024] The root detection box is sequentially subjected to Gaussian filtering for noise reduction, binarization, dilation, and contour extraction to determine the largest connected component within the root detection box.

[0025] Optionally, the connected component fitting line is determined based on the radius of the circumcircle of the root detection frame and the maximum connected component, specifically including:

[0026] For any two target pixels in the maximum connected region, a straight line is determined based on the two target pixels, and the distance from the remaining pixels in the maximum connected region to the straight line is determined; the remaining pixels are the pixels in the maximum connected region other than the two target pixels.

[0027] The total distance between two target pixels is determined based on the radius of the circumcircle of the root detection box and the distances of the remaining pixels to the straight line; the connected component line is the straight line formed by the two target pixels with the smallest total distance in the largest connected component.

[0028] Optionally, the gripping point in the camera coordinate system is determined based on the x-axis coordinates of the first intersection point, the y-axis coordinates of the first intersection point, the x-axis coordinates of the second intersection point, the y-axis coordinates of the second intersection point, and the stem detection box, specifically including:

[0029] Based on the x-axis coordinate and y-axis coordinate of the first intersection point, determine the distance between the first intersection point and the stem detection frame;

[0030] Based on the x-axis coordinates and y-axis coordinates of the second intersection point, determine the distance between the second intersection point and the stem detection frame;

[0031] If the distance between the first intersection point and the stem detection frame is less than the distance between the second intersection point and the stem detection frame, then the first intersection point is the clamping point; otherwise, the second intersection point is the clamping point.

[0032] Optionally, a cutting point is determined based on the root detection box and stem detection box in the second image, and the second robotic arm is controlled to cut off the stem of the seedling on the first robotic arm at the cutting point, specifically including:

[0033] For the nth cut, determine whether both root detection boxes and stem detection boxes exist simultaneously in the second image at the nth cut. If only root detection boxes exist, control the first robotic arm to plant the roots of the seedling into the petri dish; if only stem detection boxes exist, discard the seedling.

[0034] If both root detection frames and stem detection frames exist, the cutting point is determined based on the root detection frames and the stem detection frames.

[0035] Based on the cutting point, the second robotic arm is controlled to cut off the stem of the seedling on the first robotic arm, and the image of the seedling on the first robotic arm is re-acquired to obtain the second image at the (n+1)th cut, and the (n+1)th cut is performed; 0 <n。

[0036] Optionally, the RRT algorithm is used to plan the paths of the first robotic arm and the second robotic arm, so as to control the first robotic arm to grasp the seedling at the gripping point and control the second robotic arm to cut off the stem of the seedling on the first robotic arm at the cutting point.

[0037] Optionally, controlling the first robotic arm to plant the roots of the seedling into the petri dish specifically includes:

[0038] Transfer the empty petri dish to the planting location;

[0039] The first robotic arm is controlled to plant the roots of the seedlings into the pre-set seedling positions in the culture dish, and the number of seedlings in the culture dish is counted.

[0040] Determine whether the number of seedlings in the culture dish is equal to the preset maximum value. If so, transfer the culture dish to the preset culture position and transfer the empty culture dish to the planting position. Otherwise, continue to control the first robotic arm to plant the roots of the seedlings into the culture dish.

[0041] To achieve the above objectives, the present invention also provides the following solution:

[0042] A seedling transplanting system based on dual-arm collaborative operation includes: a first camera, a second camera, a first robotic arm, a second robotic arm, and a processor; the processor is connected to the first camera, the second camera, the first robotic arm, and the second robotic arm respectively; the second camera is mounted on the second robotic arm.

[0043] The first camera is used to capture a first image of the seedling and send it to the processor;

[0044] The processor is used to perform target detection on the first image based on the target detection network to determine the root detection box and stem detection box in the first image, determine the gripping point based on the root detection box and stem detection box in the first image, and control the first robotic arm to grip the seedling at the gripping point; the target detection network is obtained by pre-training ShuffleNet v2-YOLOv5 with a training sample set; the training sample set includes multiple training sample images containing annotation information of seedling roots, stems, leaves, and tubers.

[0045] The second camera is used to capture a second image of the seedling on the first robotic arm and send it to the processor;

[0046] The processor is further configured to perform target detection on the second image based on a target detection network to determine root detection boxes and stem detection boxes in the second image, determine cutting points based on root detection boxes and stem detection boxes in the second image, control the second robotic arm to cut off the stem of the seedling on the first robotic arm at the cutting points, and control the first robotic arm to plant the roots of the seedling into a petri dish.

[0047] Optionally, both the first camera and the second camera are industrial cameras.

[0048] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0049] This invention first transports the seedling to the identification location, determines the root and stem detection boxes in the first image based on a target detection network, then determines the gripping point based on the root and stem detection boxes in the first image, and controls the first robotic arm to grip the seedling at the gripping point. Next, a second image of the seedling on the first robotic arm is acquired, and the root and stem detection boxes in the second image are determined based on the target detection network. A cutting point is then determined based on the root and stem detection boxes in the second image, and the second robotic arm is controlled to cut off the stem of the seedling on the first robotic arm at the cutting point. Finally, the first robotic arm is controlled to plant the root of the seedling into a petri dish. Throughout the entire transplanting process, the transport, identification, gripping, cutting, and planting of the seedling are fully automated, saving significant manpower and resources. It also avoids the problem of the sterile environment being disrupted during transplanting due to human intervention, improving the transplanting efficiency of the seedlings, reducing the disease rate of the seedlings, and increasing the survival rate of the seedlings. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of the seedling transplanting method based on dual-arm cooperative operation according to the present invention;

[0052] Figure 2 This is a flowchart illustrating the overall process of seedling transplantation.

[0053] Figure 3A diagram showing the eye positioned outside the hand;

[0054] Figure 4 A diagram illustrating the marking of the eye on the hand;

[0055] Figure 5 This is a schematic diagram of the seedling transplanting system based on dual-arm collaborative operation according to the present invention.

[0056] Symbol explanation:

[0057] First camera-1, first calibration plate-2, first robotic arm-3, first robotic arm end effector-4, first robotic arm base-5, second camera-6, second calibration plate-7, second robotic arm end effector-8, second robotic arm-9, second camera base-10, processor-11. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] The purpose of this invention is to provide a seedling transplantation method and system based on dual-arm collaborative operation, which realizes the complete automation of a series of processes such as identification, picking, cutting and planting of Phalaenopsis seedlings without human intervention.

[0060] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] Seedling transplantation consists of several operational steps: target recognition, seedling gripping, seedling cutting, seedling planting, and path planning for each step. Seedling target recognition identifies different parts of the seedling to locate it in space, ensuring the smooth operation of subsequent steps. Seedling gripping, based on seedling recognition, involves path planning for the robotic arm to grip the seedling in a suitable position while ensuring collision-free operation and respecting the arm's degrees of freedom, thus facilitating subsequent cutting. Seedling cutting involves recognizing the pose of the gripped seedling using a depth camera, rotating it to a suitable position for cutting, and then cutting the seedling using a robotic arm with a scissor-like end effector. Seedling planting involves sequentially planting the seedlings from the robotic arm into petri dishes, with the petri dishes being replaced via a conveyor belt.

[0062] Example 1

[0063] like Figure 1 As shown, this embodiment provides a seedling transplantation method based on dual-arm cooperative operation, including:

[0064] S1: The seedling is transported to the location to be identified, and a first image of the seedling is acquired. In this embodiment, a first image of the seedling is acquired using a first camera. The seedling is a Phalaenopsis orchid seedling.

[0065] Phalaenopsis orchid seedlings placed in a specific location are transported to the image acquisition position via a conveyor belt. This image acquisition position is within the field of view of the first camera. When one Phalaenopsis orchid seedling has been transplanted, the next seedling is transported to the designated position via the conveyor belt, ensuring the continuity of the operation.

[0066] S2: Perform target detection on the first image based on the target detection network to determine the root detection box and stem detection box in the first image.

[0067] The object detection network was trained on ShuffleNet v2-YOLOv5 using a training sample set. The training sample set included multiple training sample images with annotations for the roots, stems, leaves, and tubers of the seedlings.

[0068] S3: Determine the gripping point based on the root detection box and stem detection box in the first image, and control the first robotic arm to grip the seedling at the gripping point.

[0069] Furthermore, S3 specifically includes:

[0070] (31) Determine whether a root detection box and a stem detection box exist simultaneously in the first image.

[0071] (32) If only the root detection frame exists, control the first robotic arm to plant the roots of the seedling into the petri dish. If only the stem detection frame exists, discard the seedling.

[0072] (33) If both root detection frames and stem detection frames exist simultaneously, the largest connected region within the root detection frame is determined. That is, if both the root region and a separate black tuber region are detected simultaneously, the calculation of the fixed point begins. At this time, the seedling is transported to the designated location via a conveyor belt. Since the seedling is not yet in space, a suitable clamping point can be found by determining its two-dimensional information.

[0073] Specifically, the root detection box is sequentially subjected to Gaussian filtering for noise reduction, binarization, dilation, and contour extraction to determine the largest connected component within the root detection box.

[0074] (34) Determine the radius of the circumcircle of the root detection frame based on its width and height: Where r is the radius of the circumcircle, ws h is the width of the root detection box. s The height of the root detection frame.

[0075] (35) Determine the fitted straight line of the connected region based on the radius of the circumcircle of the root detection frame and the maximum connected region.

[0076] Specifically, for any two target pixels in the maximum connected component, a straight line is determined based on the two target pixels, and the distances from the remaining pixels in the maximum connected component to the straight line are determined. The remaining pixels are all pixels in the maximum connected component other than the two target pixels.

[0077] The total distance between the two target pixels is determined based on the radius of the circumcircle of the root detection box and the distances of the remaining pixels to the straight line. According to the M-estimator algorithm, to prevent insufficient robustness to interference points, the formula ∑ρ(d) is used to calculate the total distance between the two target pixels. d is the distance from any other pixel to the line, r is the radius of the circumcircle, and C = 1.345.

[0078] A connected component line is the straight line formed by the two target pixels with the smallest total distance in the largest connected component. The expression for a connected component line is:

[0079]

[0080] Here, (x1,y1) and (x2,y2) are the coordinates of the two target pixels with the smallest total distance in the largest connected region.

[0081] (36) Determine the circumcircle of the root detection frame based on the coordinates of the center point of the root detection frame and the radius of the circumcircle of the root detection frame:

[0082] (xx s ) 2 +(yy s ) 2 =r' 2 ;

[0083] Among them, (x s ,y s ) represents the coordinates of the center point of the root detection box, and r' is the radius of the circumscribed circle when ∑ρ(d) is minimized.

[0084] (37) Based on the connected component fitting line and the circumcircle of the root detection box, determine the x-axis coordinates of the first intersection point and the second intersection point of the connected component fitting line and the circumcircle.

[0085] Specifically, we combine the expression for the fitted straight line of the connected components and the expression for the circumcircle of the root detection box:

[0086]

[0087] make C'=ky2-(k) 2 x1-ky s -x s Therefore, the equation is a quadratic equation in one variable. Solving for the equation yields the x-coordinates of the first intersection point I1 and the second intersection point I2:

[0088]

[0089] (38) Based on the fitted straight line of the connected domain, the x-axis coordinate of the first intersection point and the x-axis coordinate of the second intersection point, determine the y-axis coordinate of the first intersection point and the y-axis coordinate of the second intersection point.

[0090] Substituting the x-coordinates of the first intersection point and the second intersection point into the expression for fitting a straight line within the connected domain, we obtain the y-coordinates of the first intersection point and the second intersection point.

[0091] (39) Based on the x-axis coordinates of the first intersection point, the y-axis coordinates of the first intersection point, the x-axis coordinates of the second intersection point, the y-axis coordinates of the second intersection point, and the stem detection frame, determine the gripping point in the camera coordinate system, and convert the gripping point in the camera coordinate system into the gripping point in the first robotic arm base coordinate system.

[0092] Specifically, the distance between the first intersection point and the stem detection frame is determined based on the x-axis and y-axis coordinates of the first intersection point. The distance between the second intersection point and the stem detection frame is determined based on the x-axis and y-axis coordinates of the second intersection point. If the distance between the first intersection point and the stem detection frame is less than the distance between the second intersection point and the stem detection frame, then the first intersection point is the clamping point; otherwise, the second intersection point is the clamping point.

[0093] Furthermore, the coordinates of the gripping point are converted to coordinates in the coordinate system of the robotic arm base using the transformation matrix obtained from hand-eye calibration. The angle θ1 between the gripping line and the horizontal axis of the image is then calculated using the following formula, thereby controlling the end effector of the first robotic arm to adjust the gripping angle and make the posture of the first robotic arm gripping the Phalaenopsis seedling more suitable:

[0094]

[0095] The gripping point coordinates are then transmitted to the first robotic arm, and the desired pose for moving the first robotic arm to the designated position is determined. Next, the RRT algorithm is used to plan the gripping path from the robotic arm. The designated position is a spatial location that meets various requirements, including minimizing robotic arm collisions, facilitating the first robotic arm's movements, remaining within the field of view of the second robotic arm's depth camera and industrial camera, and facilitating subsequent cutting operations. Based on the planned path, the first robotic arm is controlled to complete the gripping operation and move to the specific position with a certain pose, thus fixing the orchid in space.

[0096] S4: Acquire a second image of the seedling on the first robotic arm. In this embodiment, a second image of the seedling is acquired using a second camera mounted on the second robotic arm. The second camera is a Hikvision MV-CE060-10UC camera.

[0097] S5: Perform target detection on the second image based on the target detection network to determine the root detection box and stem detection box in the second image.

[0098] S6: Determine the cutting point based on the root detection box and stem detection box in the second image, and control the second robotic arm to cut off the stem of the seedling on the first robotic arm at the cutting point.

[0099] Furthermore, S6 specifically includes:

[0100] (61) For the nth cut, determine whether the second image at the nth cut contains both a root detection box and a stem detection box. If only the root detection box exists, control the first robotic arm to plant the roots of the seedling into the petri dish. If only the stem detection box exists, discard the seedling.

[0101] (62) If both root detection frames and stem detection frames exist simultaneously, the cutting point is determined based on the root detection frames and stem detection frames. The calculation method for the cutting point is the same as that for the gripping point, and will not be repeated here. Then, the angle between the cutting line and the horizontal axis of the image is calculated, and the end gripper of the second robotic arm is controlled to adjust the cutting angle so that the second robotic arm completes the cutting task of the Phalaenopsis seedling in a more suitable pose.

[0102] (63) Based on the cutting point, control the second robotic arm to cut off the stem of the seedling on the first robotic arm, and re-acquire the image of the seedling on the first robotic arm to obtain the second image at the (n+1)th cutting, and perform the (n+1)th cutting; 0 <n。

[0103] In this embodiment, the RRT algorithm is used to plan the paths of the first robotic arm and the second robotic arm, so as to control the first robotic arm to grasp the seedling at the gripping point, and control the second robotic arm to cut off the stem of the seedling on the first robotic arm at the cutting point.

[0104] The second robotic arm is used to cut the Phalaenopsis orchid seedlings. Its end features a scissor-like design for easy severing of the orchid's roots. The purpose of cutting is to remove excess rootstock. This excess rootstock is detrimental to both transplanting and the seedling's growth. Furthermore, a sterilized experimental platform is placed beneath the robotic arm to collect the rootstock that detaches from the orchid seedling after the second robotic arm's cutting. This rootstock is discarded after the operation is complete.

[0105] To better understand the solution of this invention, the cutting steps are further described below:

[0106] (601) The spatial position and depth information of the gripping point of the first robotic arm are obtained through the depth camera on the second robotic arm. After calculating the corresponding cutting point, the first robotic arm is set as an obstacle (to prevent collision between the two arms), and the path of the second robotic arm is planned using the RRT algorithm to find the optimal path. If a path that will not cause a collision can be found, the second robotic arm is moved to the designated position according to the trajectory, and the end effector is controlled to complete the cutting operation. If the planned trajectory does not meet the conditions, the first robotic arm is controlled to change its spatial pose, and the seedling information is re-obtained through the two types of cameras to calculate the cutting point, plan the path, etc., until a path that meets the conditions is found.

[0107] (602) After a cutting operation, the depth camera and industrial camera on the second robotic arm are used to obtain information about the seedlings on the first robotic arm, and it is determined whether the conditions for continuing cutting are met. The possible situations are as follows:

[0108] ① If roots and stems are detected simultaneously, it indicates that the seedlings need to be further cut.

[0109] ② If the roots and stems of the seedling are not detected simultaneously, the spatial pose of the six-DOF first robotic arm is adjusted to prevent seedlings that still require cutting from being mistakenly identified as having been cut after the first cut due to the roots and stems being obscured by the leaves of the Phalaenopsis seedling (at this time, the first robotic arm is still holding the seedling). A second camera scan is then used for detection. If both roots and stems are detected simultaneously at this point, it indicates that the seedling needs further cutting. Otherwise, the seedling does not need cutting, and the cutting operation is complete.

[0110] (603) When it is determined that the seedling needs to be cut, the new cutting point is recalculated, the path of the second robotic arm is replanned through the RRT algorithm, and the cutting operation is performed again.

[0111] (604) Repeat steps (602) and (603) until the seedlings no longer need to be cut. At this point, the entire cutting operation is complete.

[0112] (605) Move the second robotic arm to the set position to avoid collision with the first robotic arm in the following steps. The cutting operation is completed entirely on the experimental table to ensure that all the seedlings being cut land on the experimental table.

[0113] As a preferred embodiment, the end of the first robotic arm is made of a material with good light transmittance, such as an acrylic sheet, thereby improving the accuracy and efficiency of cutting point identification.

[0114] S7: Control the first robotic arm to plant the roots of the seedling into the petri dish.

[0115] After cutting the Phalaenopsis orchid seedlings, they need to be planted into appropriate culture dishes. This part is done by the first robotic arm. The portion of the Phalaenopsis orchid seedling that remains fixed to the first robotic arm after cutting (the part without roots) needs to be transplanted into the culture dish. Phalaenopsis orchid seedlings that were cut by the second robotic arm and fell onto the experimental table need to be discarded. By planning the path of the first robotic arm, the seedling is planted into the culture dish in a suitable position, thus completing the transplantation of one Phalaenopsis orchid seedling.

[0116] If the trajectory planning fails to meet the conditions during the planting process, manual intervention can be adopted: determine the pose of several key points in the trajectory, such as the initial gripping point, several intermediate points, the final planting point, etc. The first robotic arm passes through each two adjacent points in a straight line to ensure that no collision occurs and the pose can be achieved.

[0117] The above method only completes the transplantation of one seedling. To achieve full automation of the entire Phalaenopsis seedling transplantation process, the entire process involves planning the trajectory of planting seedlings at different locations within the same petri dish. The planting locations can be pre-designed, each planting point labeled, and the path planned by the robotic arm to plant the seedling at the center of each planting location in the petri dish can be pre-planned, thus planting multiple seedlings one by one in their respective locations within the petri dish. In this way, multiple seedlings can be transplanted into the petri dish.

[0118] In a preferred embodiment, an empty petri dish is transported to the planting position. The first robotic arm is controlled to plant the roots of seedlings into a pre-set seedling position within the petri dish, and the number of seedlings in the petri dish is counted. It is determined whether the number of seedlings in the petri dish equals a preset maximum value. If so, the petri dish is transported to the pre-set cultivation position, and the empty petri dish is transported to the planting position; otherwise, the first robotic arm continues to plant the roots of seedlings into the petri dish.

[0119] Specifically, a conveyor belt transports the seedling trays to the planting location. The number of plantings in an empty tray is counted, starting from 0. Each time a Phalaenopsis orchid seedling is planted, the count is incremented by 1. When the count reaches 13, it is reset to zero, and the tray is photographed using an industrial camera to confirm that the planting requirements have been met. Furthermore, the trays are placed on the conveyor belt; once planting is complete, the tray is transported away by the conveyor belt, and a new empty tray is delivered. The trays are placed in a fixed position.

[0120] As a specific implementation method, a rotating device can be used to rotate the petri dish 90° each time a seedling is planted, thereby achieving planting of seedlings at the four edges and the center of the four sides, reducing the computational load to some extent. When replacing empty petri dishes, an industrial camera can be used to photograph the planted petri dishes. The captured images are then fed into a ShuffleNet v2-YOLOv5 target recognition network with ShuffleNetv2 as the backbone to identify Phalaenopsis orchid seedlings. If the detection is successful, a detection bounding box is generated in the original image. If the number of identified seedlings is 13, the empty petri dish is replaced.

[0121] After one Phalaenopsis orchid seedling is transplanted, a conveyor belt transports the next seedling to the designated location, ensuring the continuity of seedlings. For example... Figure 2 The diagram shown is a flowchart of the overall process for seedling planting.

[0122] Since the present invention employs dual robotic arm collaborative operation, it is necessary to calibrate the first robotic arm (slave robotic arm) and the second robotic arm (master robotic arm) separately. That is, before acquiring the first image of the seedling, the seedling transplantation method based on dual-arm collaborative operation further includes: S0: calibrating the first camera and the second camera.

[0123] Specifically, the calibration of the first robotic arm adopts a "eye outside the hand" calibration method for hand-eye calibration, such as... Figure 3 As shown. The first camera 1 is a Hikvision MV-CE060-10UC color industrial camera. The first calibration plate 2 is fixed on the first robotic arm 3.

[0124] Since the teach pendant can display accurate information about the robotic arm's pose in space, it can be used to position the first robotic arm 3 at a first position and a second position in space, thus obtaining the pose information of these two positions. Using the pose information from these two positions, the rotation matrix from the first camera 1 to the base 5 of the first robotic arm is calculated using the following formula:

[0125]

[0126] The above formula can be simplified to:

[0127] in, This is the transformation matrix from the world coordinate system to the 4-coordinate system at the end of the first robotic arm. Let be the transformation matrix from the coordinate system of the robot arm base to the coordinate system of the robot arm end effector at the first position point. This is the transformation matrix from the camera coordinate system to the robot arm base coordinate system. Let be the transformation matrix from the world coordinate system to the camera coordinate system at the first position point. Let be the transformation matrix from the coordinate system of the robot arm base to the coordinate system of the robot arm end effector at the second position point. This is the transformation matrix from the world coordinate system to the camera coordinate system at the second position point.

[0128] Further simplification yields: A·X = X·B;

[0129] in,

[0130] Solving for matrix X yields the rotation matrix from the camera coordinate system to the robot arm base coordinate system. After obtaining the target coordinates from the camera, the rotation matrix from the camera coordinate system to the robot arm base coordinate system, obtained through hand-eye calibration, can transform the target coordinates from the camera coordinate system to the robot arm base coordinate system.

[0131] Because the second robotic arm uses two cameras—one for "eye on the hand" and one for "eye outside the hand"—two different hand-eye calibration methods are employed. The eye-on-hand calibration method is as follows: Figure 4 As shown.

[0132] This invention uses a ToF depth camera to acquire a second image of the seedling on the first robotic arm, performing hand-eye calibration in an eye-to-hand manner. In this calibration method, a second camera 6 is fixed to the end of the robotic arm, ensuring that their pose relationship remains constant. The pose relationship between the second camera 6 and the end of the second robotic arm 8 is determined through two movements. A second calibration plate 7 is positioned within the field of view of the second camera 6.

[0133] During calibration, the position P of the object in the image is first obtained. imgThe coordinates of the point in the camera coordinate system are determined by the intrinsic and extrinsic parameters obtained from camera calibration. in, This is the transformation matrix from the world coordinate system to the camera coordinate system. Controlling the robotic arm to grasp an object also requires solving the transformation between the camera coordinate system and the coordinate system of the second robotic arm's base (10). That is... This is the transformation matrix from the coordinate system of the robotic arm's end effector to the coordinate system of the robotic arm's base. Let be the transformation matrix from the camera coordinate system to the robot arm's end effector coordinate system, where It is known that the solution can be obtained by moving the robotic arm to any first and second position in space. The solution process is as follows:

[0134]

[0135] The above equation can be simplified to:

[0136] in, This is the transformation matrix from the world coordinate system to the robot arm base coordinate system. Let be the transformation matrix from the coordinate system of the robot arm's end effector to the coordinate system of the robot arm's base at the first position point. Let be the transformation matrix from the coordinate system of the robot arm's end effector to the coordinate system of the robot arm's base at the second position point.

[0137] Further simplification yields: A·X = X·B;

[0138] in,

[0139] Solving for matrix X yields the transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system.

[0140] This invention achieves completely automated transplantation of Phalaenopsis seedlings, eliminating human intervention in all operations including seedling transport, target identification, grasping, cutting, and planting. Multiple robotic arms can be used within the same space to complete the transplantation task, significantly improving efficiency, meeting practical production needs, and saving substantial manpower and resources. It also avoids the disruption of the sterile environment during transplantation caused by human intervention. Therefore, it reduces the disease rate of Phalaenopsis seedlings and greatly improves their survival rate.

[0141] Example 2

[0142] In order to perform the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a seedling transplantation system based on dual-arm collaborative operation is provided below.

[0143] like Figure 5As shown, the seedling transplanting system based on dual-arm collaborative operation provided in this embodiment includes: a first camera 1, a second camera 6, a first robotic arm 3, a second robotic arm 9, and a processor 11.

[0144] The processor 11 is connected to the first camera 1, the second camera 6, the first robotic arm 3, and the second robotic arm 9. The second camera 6 is mounted on the second robotic arm 9.

[0145] The first camera 1 is used to capture a first image of the seedling and send it to the processor 11.

[0146] The processor 11 is used to perform target detection on the first image based on the target detection network to determine the root detection box and the stem detection box in the first image, determine the gripping point based on the root detection box and the stem detection box in the first image, and control the first robotic arm 3 to grip the seedling at the gripping point.

[0147] The object detection network was trained on ShuffleNet v2-YOLOv5 using a training sample set. The training sample set included multiple training sample images containing annotation information for the roots, stems, leaves, and tubers of seedlings.

[0148] The second camera 6 is used to capture a second image of the seedling on the first robotic arm 3 and send it to the processor 11.

[0149] The processor 11 is further configured to perform target detection on the second image based on the target detection network to determine the root detection box and stem detection box in the second image, determine the cutting point based on the root detection box and stem detection box in the second image, control the second robotic arm 9 to cut off the stem of the seedling on the first robotic arm 3 at the cutting point, and control the first robotic arm 3 to plant the root of the seedling into the petri dish.

[0150] Compared with the prior art, the seedling transplanting system based on dual-arm collaborative operation provided in this embodiment has the same beneficial effects as the seedling transplanting method based on dual-arm collaborative operation provided in Embodiment 1, and will not be repeated here.

[0151] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0152] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A seedling transplanting method based on the cooperative work of two arms, characterized by, The seedling transplanting method based on the cooperation of the two arms comprises: delivering the seedling to a to-be-identified position and collecting a first image of the seedling; performing target detection on the first image based on a target detection network to determine a root detection frame and a stem detection frame in the first image; the target detection network is obtained by pre-training a ShuffleNet v2-YOLOv5 using a training sample set; the training sample set includes multiple training sample images containing root, stem, leaf and tuber annotation information of the seedling; determining a clamping point according to the root detection frame and the stem detection frame in the first image, and controlling a first arm to clamp the seedling at the clamping point; collecting a second image of the seedling on the first arm; performing target detection on the second image based on the target detection network to determine a root detection frame and a stem detection frame in the second image; determining a cutting point according to the root detection frame and the stem detection frame in the second image, and controlling a second arm to cut the stem of the seedling on the first arm at the cutting point; delivering an empty culture dish to a planting position; controlling the first arm to plant the root of the seedling into the culture dish at a pre-set seedling position, and counting the number of seedlings in the culture dish; when the first arm plants the root of the seedling into the culture dish at the pre-set seedling position, a rotating device is used, and the culture dish is rotated by 90° every time a seedling is planted, so as to plant seedlings at four edge positions and four center positions; when the empty culture dish is replaced, an industrial camera is used to take a photo of the planted culture dish, and the photographed image is input into a ShuffleNet v2-YOLOv5 target recognition network with ShuffleNet v2 as the backbone network for phalaenopsis seedling recognition; after successful detection, a detection frame is generated in the original image; it is judged whether the number of seedlings in the culture dish is equal to a pre-set maximum value; if yes, the culture dish is delivered to a pre-set culture position, and an empty culture dish is delivered to the planting position; otherwise, the first arm continues to plant the root of the seedling into the culture dish. In the method, each position for planting seedlings is designed in advance, each planting point is labeled, and the path planning of the first arm for planting seedlings into the center of each planting position in the culture dish is planned, so as to plant multiple seedlings into each position in the culture dish, and then transplant multiple seedlings into the culture dish.

2. The seedling transplanting method based on the cooperative work of the two arms according to claim 1, characterized by, Determining the clamping point according to the root detection frame and the stem detection frame in the first image specifically comprises: judging whether the root detection frame and the stem detection frame exist simultaneously in the first image; if only the root detection frame exists, controlling the first arm to plant the root of the seedling into the culture dish; if only the stem detection frame exists, discarding the seedling; if the root detection frame and the stem detection frame exist simultaneously, determining the largest connected domain in the root detection frame; determining the radius of the circumscribed circle of the root detection frame according to the width and height of the root detection frame; determining a connected domain fitting straight line according to the radius of the circumscribed circle of the root detection frame and the largest connected domain; According to the center point coordinates of the root detection frame and the radius of the circumscribed circle of the root detection frame, a circumscribed circle of the root detection frame is determined; determining a first intersection point of the straight line fitted according to the connected domain and a circumscribed circle of the root detection frame x and a second intersection point of the straight line fitted according to the connected domain and the circumscribed circle of the root detection frame x and a second intersection point of the straight line fitted according to the connected domain and the circumscribed circle of the root detection frame fitting straight lines to the connected domains, the x axis coordinates of the first intersection point and the x axis coordinates of the second intersection point, determining the y axis coordinates of the first intersection point and the y axis coordinates of the second intersection point; According to the first intersection point x The axis coordinates, the first intersection point y The axis coordinates, the second intersection point x The axis coordinates, the second intersection point y The axis coordinates and the stem detection frame, determine the clamping point in the camera coordinate system, and convert the clamping point in the camera coordinate system into the clamping point in the first mechanical arm base coordinate system.

3. The method for seedling transplanting based on the cooperative work of the two arms according to claim 2, characterized in that, The maximum connected domain in the root detection frame is determined, specifically including: The root detection frame is sequentially subjected to Gaussian filter denoising, binarization, dilation and contour line extraction processing to determine the maximum connected domain in the root detection frame.

4. The method for seedling transplanting based on the cooperative work of the two arms according to claim 2, characterized in that, According to the radius of the circumscribed circle of the root detection frame and the maximum connected domain, a connected domain fitting straight line is determined, specifically including: For any two target pixel points in the maximum connected domain, a straight line is determined according to the two target pixel points, and the distances of the remaining pixel points in the maximum connected domain to the straight line are determined; the remaining pixel points are the pixel points in the maximum connected domain except the two target pixel points; According to the radius of the circumscribed circle of the root detection frame and the distances of the remaining pixel points to the straight line, the total distance of the two target pixel points is determined; the connected domain fitting straight line is the straight line formed by the two target pixel points with the smallest total distance in the maximum connected domain.

5. The method for seedling transplanting based on the cooperative work of the two arms according to claim 2, characterized in that, According to the first intersection point x The axis coordinates, the first intersection point y The axis coordinates, the second intersection point x The axis coordinates, the second intersection point y The axis coordinates and the stem detection frame, determine the clamping point in the camera coordinate system, specifically including: According to the first intersection point x The axis coordinate and the first intersection point y The axis coordinate, and determine the distance between the first intersection point and the stem detection frame According to the second intersection point x The axis coordinate and the second intersection point y The axis coordinate, the distance between the second intersection point and the stem detection frame is determined If the distance between the first intersection point and the stem detection frame is less than the distance between the second intersection point and the stem detection frame, the first intersection point is the clamping point, otherwise the second intersection point is the clamping point.

6. The method for seedling transplanting based on the cooperative work of the two arms according to claim 1, characterized in that, According to the root detection frame and the stem detection frame in the second image, a cutting point is determined, and the second mechanical arm is controlled to cut off the stem of the seedling on the first mechanical arm at the cutting point, specifically including: For the first cutting, it is determined whether the root detection frame and the stem detection frame exist simultaneously in the second image at the first cutting. n For the second cutting, it is determined whether the root detection frame and the stem detection frame exist simultaneously in the second image at the second cutting. n If only the root detection frame exists, the first mechanical arm is controlled to plant the root of the seedling into the culture dish; if only the stem detection frame exists, the seedling is discarded. If the root detection frame and the stem detection frame exist at the same time, a cutting point is determined according to the root detection frame and the stem detection frame; According to the cutting point degree, the second mechanical arm cuts the stem of the seedling on the first mechanical arm, and re-acquires the image of the seedling on the first mechanical arm, to obtain the second image at the first cutting, and the first cutting is performed. n +1stcutting; 0 n +1stcutting; 0 n .

7. The method for seedling transplanting based on the cooperative work of the two arms according to claim 1, characterized in that, The RRT algorithm is used to plan the paths of the first mechanical arm and the second mechanical arm to control the first mechanical arm to clamp the seedling at the clamping point and control the second mechanical arm to cut off the stem of the seedling on the first mechanical arm at the cutting point.

8. A seedling transplanting system based on the cooperative work of two arms, characterized by, The seedling transplanting system based on double-arm cooperative operation includes a first camera, a second camera, a first mechanical arm, a second mechanical arm and a processor; the processor is connected with the first camera, the second camera, the first mechanical arm and the second mechanical arm respectively; the second camera is arranged on the second mechanical arm; The first camera is used to collect a first image of a seedling and send it to the processor; The processor is used to perform target detection on the first image based on a target detection network to determine a root detection frame and a stem detection frame in the first image, determine a clamping point according to the root detection frame and the stem detection frame in the first image, and control the first mechanical arm to clamp the seedling at the clamping point; the target detection network is obtained by training ShuffleNet v2-YOLOv5 using a training sample set; the training sample set includes multiple training sample images containing root, stem, leaf and tuber annotation information of seedlings; The second camera is used to collect a second image of a seedling on the first mechanical arm and send it to the processor; The processor is used to perform target detection on the first image based on a target detection network to determine a root detection frame and a stem detection frame in the first image, determine a clamping point according to the root detection frame and the stem detection frame in the first image, and control the first mechanical arm to clamp the seedling at the clamping point; the target detection network is obtained by training ShuffleNet v2-YOLOv5 using a training sample set; the training sample set includes multiple training sample images containing root, stem, leaf and tuber annotation information of seedlings; The processor is further configured to perform target detection on the second image based on a target detection network to determine a root detection frame and a stem detection frame in the second image, determine a cutting point according to the root detection frame and the stem detection frame in the second image, control the second mechanical arm to cut off the stem of the seedling on the first mechanical arm at the cutting point, control the first mechanical arm to plant the root of the seedling into a culture dish, specifically, transfer an empty culture dish to a planting position, control the first mechanical arm to plant the root of the seedling into the culture dish at a preset seedling position, and count the number of seedlings in the culture dish; when the first mechanical arm plants the root of the seedling into the culture dish at the preset seedling position, a rotating device is used, and the culture dish is rotated by 90° after planting each seedling, so as to plant seedlings at four edge positions and four center positions; when the empty culture dish is replaced, an industrial camera is used to take a photo of the culture dish after planting, the photographed image is input into a ShuffleNet v2-YOLOv5 target recognition network with ShuffleNetv2 as a backbone network for phalaenopsis seedling recognition, a detection frame is generated in the original image after successful detection; it is judged whether the number of seedlings in the culture dish is equal to a preset maximum value, if yes, the culture dish is transferred to a preset culture position, and the empty culture dish is transferred to the planting position, otherwise, the first mechanical arm is controlled to continue to plant the root of the seedling into the culture dish; wherein, each position for planting seedlings is designed in advance, each planting point is labeled, and a path planning for the first mechanical arm to plant the seedling into the center of each planting position in the culture dish is planned, so as to plant multiple seedlings into each position in the culture dish one by one, and then transplant multiple seedlings into the culture dish.

9. The seedling transplanting system based on the cooperative work of the two arms according to claim 8, characterized in that, The first camera and the second camera are both industrial cameras.

Images