A smart harvesting method, device and storage medium

Abstract

This invention provides an intelligent harvesting method, device, and storage medium, relating to the field of intelligent control technology. Based on a harvesting robot including dual robotic arms, the method includes: inputting acquired crop images into a trained crop detection model to determine the crop target and its position information; determining the crop target's first and second target pose information in the coordinate systems of the two robotic arms' end effectors; controlling the first robotic arm to move to the first target position based on the first target pose information; converting the first arm segment pose information of all segments of the first robotic arm into the second arm segment pose information in the coordinate system of the second robotic arm's base; controlling the second robotic arm to move to the second target position based on the second target pose information and the second arm segment pose information; and controlling the dual robotic arms to harvest the crop target. This invention uses obstacle avoidance control with dual robotic arms to harvest crops, improving the accuracy and safety of crop harvesting.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and more specifically, to an intelligent harvesting method, device, and storage medium. Background Technology

[0002] With the rapid development of modern agriculture, crops are increasingly adopting large-scale, standardized, and industrialized planting models, such as greenhouse cultivation. In this model, planting crops in rows with uniform spacing significantly reduces the complexity of automated harvesting and improves the feasibility of intelligent harvesting.

[0003] Currently, most existing technologies use computer vision-based harvesting robots to achieve intelligent harvesting. The harvesting robot acquires images captured by a camera, locates the crops through computer vision, and then controls the robotic arm to harvest the crops. Among these, harvesting robots include single-arm robots and multi-arm robots.

[0004] On the one hand, when using a single-arm robotic robot to harvest crops, the harvesting basket is usually fixed below the gripper on the robotic arm to hold the harvested crops. However, this method cannot guarantee accurate alignment between the crops and the basket, causing crops to easily fall out of the basket, resulting in poor harvesting accuracy. On the other hand, when using a multi-arm robotic robot to harvest crops, one arm is typically used to cut the crops, while another arm carries the harvesting basket to hold the harvested crops. However, multiple robotic arms working together are prone to collisions, leading to poor safety. Summary of the Invention

[0005] The problem addressed by this invention is how to improve the accuracy and safety of crop harvesting.

[0006] To address the above problems, the present invention provides an intelligent harvesting method, device, and storage medium.

[0007] In a first aspect, the present invention provides an intelligent harvesting method based on a harvesting robot, the harvesting robot comprising a first robotic arm and a second robotic arm, wherein a holding device is installed at the end of the first robotic arm and a cutting device is installed at the end of the second robotic arm, the intelligent harvesting method comprising:

[0008] Acquire images of the crops to be identified;

[0009] The crop image is input into a trained crop detection model to determine the crop targets in the crop image and the location information of the crop targets;

[0010] The position information is transformed to determine the first target pose information of the crop target in the coordinate system of the first robotic arm end effector and the second target pose information of the crop target in the coordinate system of the second robotic arm end effector.

[0011] Based on the first target pose information, control the first robotic arm to move to the first target point, and obtain the first arm segment pose information of each arm segment of the first robotic arm;

[0012] Based on the fourth transformation matrix between the base coordinate systems of the two robotic arms, all the first arm segment pose information is converted into the second arm segment pose information in the base coordinate system of the second robotic arm, and collision objects are created or updated according to the second arm segment pose information.

[0013] Based on the second target pose information, the second robotic arm is controlled to perform obstacle avoidance movement against the collision object until the second robotic arm moves to the second target point;

[0014] The shearing device is controlled to cut the stem of the crop target, so that the crop target falls into the holding device.

[0015] Optionally, before controlling the first robotic arm to move to the first target position based on the first target pose information, the intelligent harvesting method further includes:

[0016] Use the Docker engine to set up a first container for the ROS node of the first robotic arm and a second container for the ROS node of the second robotic arm.

[0017] A first motion control command is generated based on the first target pose information, and a second motion control command is generated based on the second target pose information.

[0018] The first motion control command is converted into a first JSON string, and the second motion control command is converted into a second JSON string;

[0019] Send the first JSON string to the first container, and send the second JSON string to the second container;

[0020] The first container is used to parse the first JSON string and control the movement of the first robotic arm based on the parsing result; the second container is used to parse the second JSON string and control the movement of the second robotic arm based on the parsing result.

[0021] Optionally, at the hardware level, the host of the harvesting robot, the second robotic arm, and the first robotic arm are communicatively connected to the same network segment;

[0022] At the software level, the host is equipped with an Ethernet bridge for connecting the network segment and a Docker bridge connected to the first bridge. The Docker bridge is connected to the first container and the second container respectively.

[0023] Optionally, before inputting the crop image into the trained crop detection model, the intelligent harvesting method further includes:

[0024] Acquire multiple training images including the crop target;

[0025] The crop targets in each of the training images are labeled to obtain labeled training images;

[0026] Image data augmentation is performed based on the labeled training images, and a training set is generated based on the augmentation results.

[0027] The YOLOv7-tiny model is trained using the training set to obtain the trained crop detection model.

[0028] Optionally, the image data augmentation based on the labeled training images includes:

[0029] The colors of the labeled training images are varied by normal randomness in the HSV color gamut, and the absolute value of a normal random number is used as the standard deviation to perform Gaussian blur transformation on the labeled training images to obtain the first processed image.

[0030] The first processed image is expanded outward into a square, and the edge pixel values ​​of the first processed image are used to fill the expanded new region with normals to obtain the second processed image.

[0031] The second processed image is randomly rotated to obtain the third processed image;

[0032] Generate a new labeled training image based on the third processed image.

[0033] Optionally, randomly rotating the second processed image to obtain the third processed image includes:

[0034] The image center point is selected based on the labeled training image or the first processed image;

[0035] The amplitude is determined based on the side length of the labeled training image or the first processed image, and a random argument is selected with an average value between 0° and 360° to determine four initial points;

[0036] Using the four initial points as center points, four final points are randomly determined based on a normal distribution, and the quadrilateral formed by the four final points is used as the third processed image.

[0037] Optionally, generating a new labeled training image based on the third processed image includes:

[0038] Map the corner coordinates of the four final points to the required image size of the labeled new training image to determine the perspective transformation matrix;

[0039] The second processed image is transformed according to the perspective transformation matrix, and the labeled regions in the second processed image are also transformed to obtain the new labeled training image.

[0040] Optionally, the harvesting robot further includes a camera mounted at the end of the second robotic arm; before performing coordinate system transformation on the position information, the intelligent harvesting method further includes:

[0041] The calibration plate is fixed at a position outside the harvesting robot. The second robotic arm is moved and the camera is controlled to take pictures of the calibration plate at multiple different positions. The first transformation matrix from the camera coordinate system to the end coordinate system of the second robotic arm is determined based on the spatial position of the calibration plate and the image position of the calibration plate in the multiple first calibration plate images.

[0042] The calibration plate is fixed on the first robotic arm. The first robotic arm is moved and the camera is controlled to capture images of the calibration plate at multiple different positions. A second transformation matrix from the end coordinate system of the first robotic arm to the coordinate system of the calibration plate is determined based on the spatial position of the calibration plate and the corresponding image position of the calibration plate in the second calibration plate image. A third transformation matrix from the camera coordinate system to the coordinate system of the first robotic arm base is determined based on the second transformation matrix.

[0043] The fourth transformation matrix from the second robotic arm base coordinate system to the first robotic arm base coordinate system is determined based on the first transformation matrix and the third transformation matrix.

[0044] The fifth transformation matrix from the camera coordinate system to the first robotic arm end coordinate system is determined based on the first transformation matrix and the fourth transformation matrix.

[0045] Secondly, the present invention provides an intelligent harvesting device based on a harvesting robot, the harvesting robot comprising a first robotic arm and a second robotic arm, wherein a holding device is installed at the end of the first robotic arm and a cutting device is installed at the end of the second robotic arm, the intelligent harvesting device comprising:

[0046] The acquisition module is used to acquire images of the crops to be identified.

[0047] The recognition module is used to input the crop image into a trained crop detection model to determine the crop targets in the crop image and the location information of the crop targets;

[0048] The first control module is used to perform coordinate system transformation on the position information, determine the first target pose information of the crop target in the coordinate system of the first robotic arm end effector, and the second target pose information of the crop target in the coordinate system of the second robotic arm end effector; control the first robotic arm to move to the first target point according to the first target pose information, and obtain the first arm segment pose information of each arm segment of the first robotic arm;

[0049] The second control module is used to convert all the first arm segment pose information into the second arm segment pose information in the second arm base coordinate system based on the fourth transformation matrix between the base coordinate systems of the two robotic arms; create or update collision objects according to the second arm segment pose information; and control the second robotic arm to perform obstacle avoidance movement for the collision object according to the second target pose information until the second robotic arm moves to the second target point.

[0050] The third control module is used to control the shearing device to cut the stem of the crop target, so that the crop target falls into the holding device.

[0051] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the intelligent harvesting method as described in the first aspect.

[0052] The beneficial effects of the intelligent harvesting method, device, and storage medium of the present invention are as follows: Crop harvesting is based on a harvesting robot including two robotic arms. The second robotic arm has a shearing device at its end for cutting the stems of the crops; the first robotic arm has a holding device at its end for catching fallen crops. Through the coordinated movement of the two robotic arms, the harvested crops can accurately fall into the holding device, improving harvesting accuracy. Images of the crops to be identified are acquired by capturing images with a depth camera. These images are then input into a trained crop detection model for target identification and localization, outputting the position information of the crop target in the camera coordinate system. Through coordinate system transformation, the position information is converted into first target pose information reachable by the first robotic arm and second target pose information reachable by the second robotic arm. Based on the first target pose information, the end effector of the first robotic arm moves to the first target point, and then the pose information of each segment of the first robotic arm is collected. The pose information of the first robotic arm segment is converted into the pose information of the second robotic arm segment in the second robotic arm's base coordinate system using a fourth transformation matrix between the base coordinate systems of the two robotic arms. This determines the pose of each segment of the first robotic arm in the second robotic arm's base coordinate system, and the fourth transformation matrix can be pre-calibrated to determine its exact form. Collision objects are created or updated based on the pose information of the second robotic arm segment. When controlling the movement of the second robotic arm based on the pose information of the second target, the second robotic arm is controlled to avoid collisions with these objects, thus preventing collisions during the coordinated movement of the two robotic arms. When the second robotic arm reaches the second target point, the shearing device is controlled to cut the stem of the crop target, causing the crop target to fall accurately into the container, achieving precise harvesting. Furthermore, obstacle avoidance control of the two robotic arms effectively prevents collisions, improving the safety of the harvesting process. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating an intelligent harvesting method according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of information interaction for an intelligent harvesting method according to an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the generated image selection box according to an embodiment of the present invention;

[0056] Figure 4 This is a flowchart illustrating the image data augmentation process according to an embodiment of the present invention.

[0057] Figure 5 This is a schematic diagram of the structure of an intelligent harvesting device according to an embodiment of the present invention. Detailed Implementation

[0058] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0059] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0060] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0061] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0062] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0063] like Figure 1 and Figure 2 As shown in the figure, an intelligent harvesting method provided by an embodiment of the present invention is based on a harvesting robot, which includes a first robotic arm and a second robotic arm. The end of the first robotic arm is equipped with a holding device, and the end of the second robotic arm is equipped with a cutting device.

[0064] Specifically, the shearing device can be a clamp, such as the Dahuan AG95 series clamp, with blades installed on the inside of the clamp to cut the stems of the crop. The receiving device is used to collect the cut crop and can include containers such as harvesting baskets, harvesting frames, and harvesting towers.

[0065] The intelligent harvesting method includes:

[0066] S110, acquire the image of the crop to be identified.

[0067] Specifically, images of crops can be taken using depth cameras on harvesting robots.

[0068] S120, input the crop image into the trained crop detection model to determine the crop targets in the crop image and the location information of the crop targets.

[0069] Specifically, a trained crop detection model is used to identify crop targets in crop images and determine the location of the crop targets in the crop images.

[0070] For example, the bounding rectangle of the crop target can be determined, and the bounding rectangle is used as the ROI to crop a sub-image. The sub-image undergoes preprocessing such as grayscale conversion and Gaussian blurring. The Sobel operator is then used for image edge extraction and threshold binarization. After the closing operation, contour searching is performed on the obtained binary edge map. Then, all contours are traversed. If the IoU between the bounding rectangle of a contour and the sub-image rectangle is greater than a given threshold and is the largest, then this contour is considered a detection result contour. Three points are randomly selected within the detection result contour, and the depth information of the corresponding positions on the original image is recorded, and their average value is taken. The centroid of the detection result contour is then taken. Using the horizontal coordinate of the centroid on the original image, the top coordinate of the entire sub-image on the original image, and the previously obtained average depth value, combined with camera intrinsic parameters, the coordinates of the sub-image in the camera coordinate system are obtained. This can be calculated using relevant calculation tools in the RealSense SDK; the calculation steps will not be elaborated here.

[0071] S130, perform coordinate system transformation on the position information to determine the first target pose information of the crop target in the coordinate system of the first robotic arm end effector and the second target pose information of the crop target in the coordinate system of the second robotic arm end effector.

[0072] Specifically, after determining the position information of the crop coordinates in the camera coordinate system, this position information is converted into target pose information that the robotic arm can accurately reach. To simplify the name and facilitate description, the first robotic arm will be referred to as Arm 1 and the second robotic arm as Arm 2 in the following text. For example, the end coordinate system of the first robotic arm will be referred to as the end coordinate system of Arm 1 and the end coordinate system of the second robotic arm will be referred to as the end coordinate system of Arm 2.

[0073] For example, the transformation matrix from the camera coordinate system to the arm 1 base coordinate system can be determined in advance through calibration. And the first transformation matrix from the camera coordinate system to the end coordinate system of arm 2. Record the pose of arm 2 during calibration with the "eye outside the hand" setting to obtain the transformation matrix from the end-effector coordinate system to the base coordinate system at the calibration instant. .

[0074] From a simple coordinate transformation, we get

[0075] ,

[0076] in, Let P represent the pose matrix of the fixed point P in the coordinate system at the end of arm 2 during calibration. This represents the first transformation matrix from the camera coordinate system to the coordinate system at the end of arm 2. This represents the pose matrix of a fixed point P in the camera coordinate system.

[0077] It should be noted that the position coordinates during the acquisition process and the quaternions during the data processing process can be converted into pose matrices and then subjected to coordinate transformation. The process of converting to pose matrices is an existing technology and will not be elaborated here.

[0078] This allows us to obtain the transformation from the camera coordinate system to the coordinate system at the end of arm 2. We can also...

[0079] ,

[0080] ,

[0081] in, This represents the pose matrix of fixed point P in the arm 1 base coordinate system. This represents the third transformation matrix from the camera coordinate system to the arm 1 base coordinate system. Let P represent the pose matrix of a fixed point P in the camera coordinate system. This represents the pose matrix of fixed point P in the arm 2 base coordinate system. Let be the transpose of the coordinate transformation matrix from the end-effector coordinate system of arm 2 to the base coordinate system of arm 2. This represents the pose matrix of fixed point P in the coordinate system at the end of arm 2.

[0082] This allows us to obtain the coordinate transformation matrix between the two arm bases. ,

[0083] The matrix obtained in this way is, in principle, similar to the temporary matrix. Irrelevant This represents the fourth transformation matrix from the coordinate system of arm 2 to the coordinate system of arm 1.

[0084] Using this matrix It can realize the transformation from the camera coordinate system to the coordinate system at the end of arm 1. ,

[0085] in, This represents the pose matrix of fixed point P in the coordinate system at the end of arm 1. This represents the coordinate transformation matrix from the coordinate system at the end of arm 1 to the coordinate system at the base of arm 1. This represents the fourth transformation matrix from the coordinate system of arm 2 to the coordinate system of arm 1. This represents the coordinate transformation matrix from the end-effector coordinate system of arm 2 to the base coordinate system of arm 2. This represents the first transformation matrix from the camera coordinate system to the coordinate system at the end of arm 2. This represents the pose matrix of a fixed point P in the camera coordinate system.

[0086] The position information is converted into a pose matrix of the crop target in the camera coordinate system. The above coordinate transformation matrix is ​​used to determine the first target pose information of the crop target in the coordinate system of the first robotic arm end effector and the second target pose information in the coordinate system of the second robotic arm end effector.

[0087] S140, control the first robotic arm to move to the first target point according to the first target pose information, and obtain the first arm segment pose information of each arm segment of the first robotic arm.

[0088] Specifically, the Moveit! software package can be used to plan the movements of the first robotic arm based on the first target pose information, and control the first robotic arm to move to the corresponding first target point. After the first robotic arm moves to the first target point, the pose information of each segment of the first robotic arm is obtained.

[0089] S150, based on the fourth transformation matrix between the base coordinate systems of the two robotic arms, convert all the first arm segment pose information into the second arm segment pose information in the base coordinate system of the second robotic arm, and create or update collision objects according to the second arm segment pose information.

[0090] Specifically, the first arm segment pose information is converted into the second arm segment pose information in the second robotic arm base coordinate system using the aforementioned calibrated fourth transformation matrix. It is then determined whether this type of information is being received for the first time. If so, a collision object is created and added to the collision detection scene object. Otherwise, the original arm segment in the collision detection scene is removed, and then Moveit! collision objects are added to it again based on the new arm segment pose obtained, thereby completing the update of the collision detection scene.

[0091] S160, based on the second target pose information, control the second robotic arm to perform obstacle avoidance movement against the collision object until the second robotic arm moves to the second target point.

[0092] Specifically, the Moveit! software package is used to plan the movements of the second robotic arm based on the pose information of the second target and the collision object, and the second robotic arm is controlled to perform obstacle avoidance movements for the collision object until the second robotic arm reaches the corresponding second target point.

[0093] It is understandable that the second robotic arm can be controlled to move to the second target position based on the second target pose information first, and then the first robotic arm can be controlled to avoid obstacles. The control process is basically the same, so it will not be described in detail here.

[0094] S170, control the shearing device to cut the stem of the crop target, so that the crop target falls into the holding device.

[0095] Specifically, the grippers can be controlled to close and cut the stem of the crop target, causing the crop target to fall into a container directly below it. This intelligent harvesting method can be used to harvest crops such as bell peppers, apples, and tomatoes.

[0096] In this embodiment, a harvesting robot with dual robotic arms is used for harvesting crops. The end effector of the second robotic arm is equipped with a shearing device for cutting the stems of the crops; the end effector of the first robotic arm is equipped with a container for catching fallen crops. Through the coordinated movement of the two robotic arms, the harvested crops can accurately fall into the container, improving harvesting accuracy. Images of the crops to be identified are acquired. These images are captured by a depth camera and input into a trained crop detection model for target identification and localization, outputting the crop target's position information in the camera coordinate system. Through coordinate system transformation, the position information is converted into first target pose information reachable by the first robotic arm and second target pose information reachable by the second robotic arm. Based on the first target pose information, the end effector of the first robotic arm moves to the first target point, and then the first segment pose information of each segment of the first robotic arm is collected. The pose information of the first robotic arm segment is converted into the pose information of the second robotic arm segment in the second robotic arm's base coordinate system using a fourth transformation matrix between the base coordinate systems of the two robotic arms. This determines the pose of each segment of the first robotic arm in the second robotic arm's base coordinate system, and the fourth transformation matrix can be pre-calibrated to determine its exact form. Collision objects are created or updated based on the pose information of the second robotic arm segment. When controlling the movement of the second robotic arm based on the pose information of the second target, the second robotic arm is controlled to avoid collisions with these objects, thus preventing collisions during the coordinated movement of the two robotic arms. When the second robotic arm reaches the second target point, the shearing device is controlled to cut the stem of the crop target, causing the crop target to fall accurately into the container, achieving precise harvesting. Furthermore, obstacle avoidance control of the two robotic arms effectively prevents collisions, improving the safety of the harvesting process.

[0097] Optionally, before controlling the first robotic arm to move to the first target position based on the first target pose information, the intelligent harvesting method further includes:

[0098] Use the Docker engine to set up a first container for the ROS (Robot Operating System) node of the first robotic arm and a second container for the ROS node of the second robotic arm;

[0099] A first motion control command is generated based on the first target pose information, and a second motion control command is generated based on the second target pose information.

[0100] The first motion control instruction is converted into a first JSON (JavaScript Object Notation) string, and the second motion control instruction is converted into a second JSON string;

[0101] Send the first JSON string to the first container, and send the second JSON string to the second container;

[0102] The first container is used to parse the first JSON string and control the movement of the first robotic arm based on the parsing result; the second container is used to parse the second JSON string and control the movement of the second robotic arm based on the parsing result.

[0103] Specifically, the harvesting robot's host ROS sends a first JSON string to a first container. The first container parses the first JSON string and, based on the parsing result, controls the end effector of the first robotic arm to move to the first target point. It then obtains the first segment pose information of each segment of the first robotic arm and feeds it back to the harvesting robot's host ROS. The harvesting robot's host ROS converts the first segment pose information into the second segment pose information in the second robotic arm's base coordinate system and sends it along with the second JSON string to the second container. The second container creates or updates collision objects based on the second segment pose information, parses the second JSON string, and controls the second robotic arm to perform obstacle avoidance movements for the collision objects based on the parsing result. When the end effector of the second robotic arm reaches the second target point, it sends a completion signal back to the harvesting robot ROS. The harvesting robot ROS then controls the shearing device to cut the stems of the target crop, completing the harvest.

[0104] At the underlying design level, each data frame is in JSON format with newline characters. Firstly, using JSON makes it easy to organize data, allowing for convenient packaging and retrieval of the required data. Secondly, JSON will unescape newline characters that appear in the data, ensuring that no newline characters appear in the JSON string, thus enabling the newline character at the end of the data frame to function as a data frame delimiter.

[0105] Due to the characteristics of TCP (Transmission Control Protocol) sockets in data transmission, some messages may remain in the underlying buffer after each message reception, making it impossible to guarantee that the program will fully receive the message. Based on this, we designed a data reception scheme on the program side to ensure that each data frame is fully received. On the program side, we set up a buffer. Each time a message is received, the message content is first appended to the buffer. After the message reception is complete, this buffer is searched. If a newline character is found in the buffer, the buffer is split according to the newline character, resulting in a split list; otherwise, parsing ends, and the program waits for the next message reception. If the split list is obtained, it is traversed, and each element is parsed using JSON. If parsing is successful, the data frame is valid, and the instructions are executed; if parsing fails, it is ignored. Specifically, when parsing the last element in the table, if successful, the buffer is set to empty after executing the instruction; if unsuccessful, the buffer is set to the value of that element before parsing ends.

[0106] It should be noted that in existing technologies, dual-arm harvesting robots can only connect to one arm at a time in the underlying ROS driver. If ROS drivers for two arms are enabled, errors will occur due to the use of nodes with the same name. The gripper does not require complex planning using the Moveit! package; simply publishing command messages under the corresponding ROS topic is sufficient to drive the gripper.

[0107] In this optional embodiment, a container is maintained for each robotic arm's ROS node using the Docker engine, isolated from the ROS environment of the picking robot host, to avoid the problem of duplicate node names, thereby enabling the driving of the two robotic arms.

[0108] Optionally, at the hardware level, the host of the harvesting robot, the second robotic arm, and the first robotic arm are communicatively connected to the same network segment;

[0109] At the software level, the host is equipped with an Ethernet bridge for connecting the network segment and a Docker bridge connected to the first bridge. The Docker bridge is connected to the first container and the second container respectively.

[0110] Optionally, before inputting the crop image into the trained crop detection model, the intelligent harvesting method further includes:

[0111] S210, acquire multiple training images including the crop target.

[0112] Specifically, training images can be obtained through online searches, purchasing and photographing images from supermarkets, or contacting farmers to take photos.

[0113] S220, the crop targets in each of the training images are labeled to obtain labeled training images.

[0114] Specifically, the LabelMe software can be used to label the crop targets in each training image using polygon annotation, thus obtaining image-labeled data.

[0115] S230, perform image data augmentation based on the labeled training images, and generate a training set based on the data augmentation results.

[0116] Specifically, image data augmentation increases the amount of training images to improve model training performance.

[0117] S240, The YOLOv7-tiny model is trained using the training set to obtain the trained crop detection model.

[0118] Specifically, the YOLOv7-tiny model is used for training, and as a trained crop detection model, it can meet the harvesting scenarios where the accuracy requirements are high and the speed requirements are relatively relaxed.

[0119] Optionally, such as Figure 3 and Figure 4 As shown, the image data augmentation based on the labeled training images includes:

[0120] The colors of the labeled training images are varied randomly in the HSV color gamut using a normal distribution, and the absolute value of a normal random number is used as the standard deviation to perform a Gaussian blur transformation on the labeled training images to obtain the first processed image.

[0121] Specifically, after color modification and Gaussian blur transformation of the image, it can be used to train the model. This allows the trained crop detection model to withstand the influence of ambient light, growth status, camera stability, and other factors, without being significantly affected by these variables.

[0122] The first processed image is expanded outward into a square, and the edge pixel values ​​of the first processed image are used to fill the expanded new region with normals to obtain the second processed image.

[0123] Specifically, the side length of the square can be the length of the diagonal of the original image (i.e., the first processed image or the labeled training image). d Expanding and normal filling can prepare the image for subsequent rotation.

[0124] The second processed image is randomly rotated to obtain the third processed image;

[0125] Generate a new labeled training image based on the third processed image.

[0126] Specifically, the image can also be flipped, with a 50% probability of flipping and a 50% probability of not flipping. Since horizontal and vertical flips can be obtained by rotation operations, and considering that random image rotation has already been performed, we will only consider the case of horizontal flipping here.

[0127] In this optional embodiment, new labeled training images are generated after preprocessing such as color change, image expansion, and image rotation, which can expand the amount of training data and improve the training accuracy of the model.

[0128] Optionally, randomly rotating the second processed image to obtain the third processed image includes:

[0129] The image center point is selected based on the labeled training image or the first processed image.

[0130] Specifically, for images where the original image is a square, the normal distribution is directly used. Simply select a point near the center of the image. d This represents the length of the diagonal of the original image; for images where the original image is rectangular, it represents the length of the shorter side. d 1. In the direction of normality Randomly select the coordinates in this direction, on the longer side d In addition to using this normally distributed random value, a mean can also be added in the second direction. Random numbers, i.e., 0 to 10 ... d 2- d The coordinates of the image center point are denoted as any value between 1 and 2, to ensure uniform selection along the longer side. .

[0131] The amplitude is determined based on the side length of the labeled training image or the first processed image, and a random argument is selected with an average value between 0° and 360° to determine four initial points.

[0132] Specifically, the shorter side of the original image can be... (If the original image is a square, then it is the side length of the original image) The amplitude is calculated as (times), with the average value between 0° and 360° selected as the random amplitude. θAnd determine the points in the four directions based on the selected argument angle. , k The values ​​are 0, 1, 2, and 3.

[0133] Using the four initial points as center points, four final points are randomly determined based on a normal distribution, and the quadrilateral formed by the four final points is used as the third processed image.

[0134] Specifically, taking these four points as the center points, the normal distribution is used again. Randomly select points nearby. d 1 represents the length of the shorter side of the original image. These four final points... , k Taking the values ​​0, 1, 2, and 3, the resulting quadrilateral approximates a square; this is the region of the desired output image, i.e., the third processed image. For example... Figure 3 As shown, Figure 3 Region A in the middle is the probability cloud for selecting the center point of the image, and the four regions B are the probability clouds for the four corner points of the required output image. The quadrilateral formed by the four corner points is the region box of the required output image, that is, the region box of the third processed image.

[0135] In this optional embodiment, if the final point or the initial point exceeds the image boundary, a new point that does not exceed the boundary is selected. This process simulates effects such as stretching and perspective transformation.

[0136] Optionally, generating a new labeled training image based on the third processed image includes:

[0137] Map the corner coordinates of the four final points to the required image size of the labeled new training image to determine the perspective transformation matrix;

[0138] The second processed image is transformed according to the perspective transformation matrix, and the labeled regions in the second processed image are also transformed to obtain the new labeled training image.

[0139] Specifically, assume the coordinates of the corner points of the four final points are respectively ( x 0, y 0), ( x 1, y 1), ( x 2, y 2) and ( x 3. y 3) The required image size for the new labeled training images is ( w , h Then the perspective transformation matrix M Can be recorded as ,

[0140] So through ,

[0141] The perspective transformation matrix can then be calculated. M The various elements.

[0142] Using perspective transformation matrix M The image with the expanded border (i.e., the second processed image) is transformed, and its annotations are also transformed, thus obtaining a pair of image-annotation data.

[0143] Optionally, the harvesting robot further includes a camera mounted at the end of the second robotic arm; before performing coordinate system transformation on the position information, the intelligent harvesting method further includes:

[0144] The calibration plate is fixed at a position outside the harvesting robot. The second robotic arm is moved and the camera is controlled to take pictures of the calibration plate at multiple different positions. The first transformation matrix from the camera coordinate system to the end coordinate system of the second robotic arm is determined based on the spatial position of the calibration plate and the image position of the calibration plate in the multiple first calibration plate images.

[0145] Specifically, during the "eye-on-hand" calibration, the calibration plate is fixed, the robotic arm is moved, and coordinate information is collected. The transformation matrix from the coordinate system at the end of arm 2 to the coordinate system at the base of arm 2 is denoted as... The transformation matrix from the calibration board coordinate system to the camera coordinate system is: The first transformation matrix from the camera coordinate system to the end coordinate system of arm 2 is: Let's assume a fixed point in space. P Its pose matrix in the calibration plate coordinate system is denoted as Its pose matrix in the arm 2 base coordinate system is denoted as From a simple coordinate transformation, we have ,

[0146] After moving the robotic arm, obtain the transformation matrix from the end-effector coordinate system of arm 2 to the base coordinate system of arm 2. Transformation from the new calibration plate coordinate system to the camera coordinate system Since the base, calibration plate, and selection points remain unchanged, there are still... ,

[0147] Combining the two equations, we can deduce the following result with point... P irrelevant equations ,

[0148] If the matrices in the equation are non-singular, then the first transformation matrix from the camera coordinate system to the end coordinate system of arm 2 can be obtained by solving this equation. In actual calibration, multiple points are usually selected to suppress the influence of errors and avoid singular solutions.

[0149] The calibration plate is fixed on the first robotic arm. The first robotic arm is moved and the camera is controlled to capture images of the calibration plate at multiple different positions. A second transformation matrix from the coordinate system of the first robotic arm end effector to the coordinate system of the calibration plate is determined based on the spatial position of the calibration plate and the corresponding image position of the calibration plate in the second calibration plate image. A third transformation matrix from the camera coordinate system to the coordinate system of the first robotic arm base is determined based on the second transformation matrix.

[0150] Specifically, during the "eye outside the hand" calibration, the calibration plate is fixed on arm 1, and the camera is fixed on arm 2. The robotic arm containing the calibration plate is moved to collect coordinate information. The transformation matrix from the coordinate system at the end of arm 1 to the coordinate system at the base of arm 1 is denoted as... The transformation from the calibration board coordinate system to the camera coordinate system is as follows: The third transformation matrix from the camera coordinate system to the arm 1 base coordinate system is: Let the second transformation matrix from the coordinate system at the end of arm 1 to the coordinate system of the calibration plate be... Let's assume a fixed point in space. P Its pose matrix in the camera coordinate system is denoted as Its pose matrix in the arm 1 base coordinate system is denoted as From a simple coordinate transformation, we have ,

[0151] Let represent the transformation matrix from the coordinate system at the base end of arm 1 to the coordinate system at the end of arm 1. The inverse matrix.

[0152] Similarly, after moving the robotic arm, a new transformation matrix from the calibration plate coordinate system to the camera coordinate system is obtained. The transformation matrix from the new arm 1 base end coordinate system to the arm 1 end coordinate system. ,have ,

[0153] The following equations are obtained by combining the points: P irrelevant equations ,

[0154] If the matrices in the equation are non-singular, then the second transformation matrix from the coordinate system at the end of arm 1 to the coordinate system of the calibration plate can be obtained by solving this equation. This matrix can be used to calculate the third transformation matrix from the camera coordinate system to the arm 1 base coordinate system. :

[0155] ,

[0156] Similarly, in actual calibration, multiple points are usually selected to suppress the influence of errors and avoid singular solutions.

[0157] The fourth transformation matrix from the second robotic arm base coordinate system to the first robotic arm base coordinate system is determined based on the first transformation matrix and the third transformation matrix.

[0158] The fifth transformation matrix from the camera coordinate system to the first robotic arm end coordinate system is determined based on the first transformation matrix and the fourth transformation matrix.

[0159] Specifically, the process of determining the fourth transformation matrix has been described above and will not be repeated here.

[0160] ,

[0161] in, This represents the fifth transformation matrix from the camera coordinate system to the coordinate system at the end of arm 1. This represents the coordinate transformation matrix from the coordinate system at the end of arm 1 to the coordinate system at the base of arm 1. This represents the fourth transformation matrix from the coordinate system of arm 2 to the coordinate system of arm 1. This represents the coordinate transformation matrix from the end-effector coordinate system of arm 2 to the base coordinate system of arm 2. This represents the first transformation matrix from the camera coordinate system to the end coordinate system of arm 2.

[0162] like Figure 5 As shown in the figure, an intelligent harvesting device provided by an embodiment of the present invention is based on a harvesting robot. The harvesting robot includes a first robotic arm and a second robotic arm. A holding device is installed at the end of the first robotic arm, and a cutting device is installed at the end of the second robotic arm. The intelligent harvesting device includes:

[0163] The acquisition module is used to acquire images of the crops to be identified.

[0164] The recognition module is used to input the crop image into a trained crop detection model to determine the crop targets in the crop image and the location information of the crop targets;

[0165] The first control module is used to perform coordinate system transformation on the position information, determine the first target pose information of the crop target in the coordinate system of the first robotic arm end effector, and the second target pose information of the crop target in the coordinate system of the second robotic arm end effector; control the first robotic arm to move to the first target point according to the first target pose information, and obtain the first arm segment pose information of each arm segment of the first robotic arm;

[0166] The second control module is used to convert all the first arm segment pose information into the second arm segment pose information in the second arm base coordinate system based on the fourth transformation matrix between the base coordinate systems of the two robotic arms; create or update collision objects according to the second arm segment pose information; and control the second robotic arm to perform obstacle avoidance movement for the collision object according to the second target pose information until the second robotic arm moves to the second target point.

[0167] The third control module is used to control the shearing device to cut the stem of the crop target, so that the crop target falls into the holding device.

[0168] The intelligent harvesting device in this embodiment is used to implement the intelligent harvesting method described above. Its advantages over the prior art are the same as the advantages of the intelligent harvesting method over the prior art, and will not be repeated here.

[0169] Optionally, the intelligent harvesting device further includes a processing module, which is used to: set up a first container for the ROS node of the first robotic arm and a second container for the ROS node of the second robotic arm using a Docker engine; generate a first motion control command based on the first target pose information and generate a second motion control command based on the second target pose information; convert the first motion control command into a first JSON string and convert the second motion control command into a second JSON string; send the first JSON string to the first container and send the second JSON string to the second container; wherein the first container is used to parse the first JSON string and control the movement of the first robotic arm based on the parsing result; the second container is used to parse the second JSON string and control the movement of the second robotic arm based on the parsing result.

[0170] Optionally, at the hardware level, the host of the harvesting robot, the second robotic arm, and the first robotic arm are communicatively connected to the same network segment; at the software level, the host is equipped with an Ethernet bridge for connecting the network segment, and a Docker bridge connected to the first bridge, the Docker bridge being connected to the first container and the second container respectively.

[0171] Optionally, the intelligent harvesting device further includes a training module, which is used to: acquire multiple training images including the crop target; label the crop target in each of the training images to obtain labeled training images; perform image data augmentation based on the labeled training images, and generate a training set based on the data augmentation results; and train the YOLOv7-tiny model using the training set to obtain the trained crop detection model.

[0172] Optionally, the training module is specifically used to: change the color of the labeled training image in the HSV color gamut using a normal random distribution, and perform a Gaussian blur transformation on the labeled training image using the absolute value of a normal random number as the standard deviation to obtain a first processed image; expand the first processed image outward into a square, and fill the expanded new region with the normal using the edge pixel values ​​of the first processed image to obtain a second processed image; randomly rotate the second processed image to obtain a third processed image; and generate a new labeled training image based on the third processed image.

[0173] Optionally, the training module is further configured to: select an image center point based on the labeled training image or the first processed image; determine the amplitude based on the side length of the labeled training image or the first processed image, select a random argument angle with an average value between 0° and 360°, and determine four initial points; and randomly determine four final points based on a normal distribution, using the four initial points as center points respectively, and use the quadrilateral formed by the four final points as the third processed image.

[0174] Optionally, the training module is further configured to: map the corner coordinates of the four final points to the required image size of the labeled new training image to determine the perspective transformation matrix; transform the second processed image according to the perspective transformation matrix, and transform the labeled region in the second processed image to obtain the labeled new training image.

[0175] Optionally, the harvesting robot further includes a camera mounted on the end of the second robotic arm; the intelligent harvesting device further includes a calibration module, which is used to: fix a calibration plate at a position outside the harvesting robot; move the second robotic arm and control the camera to photograph the calibration plate at multiple different positions; determine a first transformation matrix from the camera coordinate system to the coordinate system at the end of the second robotic arm based on the spatial position of the calibration plate and the image position of the calibration plate in multiple photographed first calibration plate images; fix the calibration plate on the first robotic arm; move the first robotic arm and control the camera to photograph the calibration plate at multiple positions. Based on the spatial position of the calibration board and its corresponding image position in the second calibration board image, a second transformation matrix is ​​determined from the first robotic arm end-effector coordinate system to the calibration board coordinate system. Then, a third transformation matrix is ​​determined from the camera coordinate system to the first robotic arm base coordinate system based on the second transformation matrix. Finally, a fourth transformation matrix is ​​determined from the second robotic arm base coordinate system to the first robotic arm base coordinate system based on the first and third transformation matrices. Finally, a fifth transformation matrix is ​​determined from the camera coordinate system to the first robotic arm end-effector coordinate system based on the first and fourth transformation matrices.

[0176] An electronic device provided by an embodiment of the present invention includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to implement the intelligent harvesting method described above when the computer program is executed.

[0177] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the intelligent harvesting method described above.

[0178] The present invention will now describe electronic devices that can serve as servers or clients of the present invention, which are examples of hardware devices that can be applied to various aspects of the present invention. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0179] Electronic devices include a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0180] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0181] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A smart harvesting method, characterized in that, Based on a harvesting robot, the harvesting robot includes a first robotic arm and a second robotic arm. The first robotic arm has a container installed at its end, and the second robotic arm has a shearing device installed at its end. The intelligent harvesting method includes: Acquire images of the crops to be identified; The crop image is input into a trained crop detection model to determine the crop targets in the crop image and the location information of the crop targets; The position information is transformed to determine the first target pose information of the crop target in the coordinate system of the first robotic arm end effector and the second target pose information of the crop target in the coordinate system of the second robotic arm end effector. Based on the first target pose information, control the first robotic arm to move to the first target point, and obtain the first arm segment pose information of each arm segment of the first robotic arm; Based on the fourth transformation matrix between the base coordinate systems of the two robotic arms, all the first arm segment pose information is converted into the second arm segment pose information in the base coordinate system of the second robotic arm, and collision objects are created or updated according to the second arm segment pose information. Based on the second target pose information, the second robotic arm is controlled to perform obstacle avoidance movement against the collision object until the second robotic arm moves to the second target point; The shearing device is controlled to cut the stem of the crop target, so that the crop target falls into the holding device.

2. The intelligent harvesting method according to claim 1, characterized in that, Before controlling the first robotic arm to move to the first target position based on the first target pose information, the intelligent harvesting method further includes: Use the Docker engine to set up a first container for the ROS node of the first robotic arm and a second container for the ROS node of the second robotic arm. A first motion control command is generated based on the first target pose information, and a second motion control command is generated based on the second target pose information. The first motion control command is converted into a first JSON string, and the second motion control command is converted into a second JSON string; Send the first JSON string to the first container, and send the second JSON string to the second container; The first container is used to parse the first JSON string and control the movement of the first robotic arm based on the parsing result; the second container is used to parse the second JSON string and control the movement of the second robotic arm based on the parsing result.

3. The intelligent harvesting method according to claim 2, characterized in that, At the hardware level, the main unit of the harvesting robot, the second robotic arm, and the first robotic arm are communicatively connected to the same network segment; At the software level, the host is equipped with an Ethernet bridge for connecting the network segment and a Docker bridge connected to the Ethernet bridge. The Docker bridge is connected to the first container and the second container respectively.

4. The intelligent harvesting method according to claim 1, characterized in that, Before inputting the crop image into the trained crop detection model, the intelligent harvesting method further includes: Acquire multiple training images including the crop target; The crop targets in each of the training images are labeled to obtain labeled training images; Image data augmentation is performed based on the labeled training images, and a training set is generated based on the augmentation results. The YOLOv7-tiny model is trained using the training set to obtain the trained crop detection model.

5. The intelligent harvesting method according to claim 4, characterized in that, The image data augmentation based on the labeled training images includes: The colors of the labeled training images are varied by normal randomness in the HSV color gamut, and the absolute value of a normal random number is used as the standard deviation to perform Gaussian blur transformation on the labeled training images to obtain the first processed image. The first processed image is expanded outward into a square, and the edge pixel values ​​of the first processed image are used to fill the expanded new region with normals to obtain the second processed image. The second processed image is randomly rotated to obtain the third processed image; Generate a new labeled training image based on the third processed image.

6. The intelligent harvesting method according to claim 5, characterized in that, The step of randomly rotating the second processed image to obtain the third processed image includes: The image center point is selected based on the labeled training image or the first processed image; The amplitude is determined based on the side length of the labeled training image or the first processed image, and a random argument is selected with an average value between 0° and 360° to determine four initial points; Using the four initial points as center points, four final points are randomly determined based on a normal distribution, and the quadrilateral formed by the four final points is used as the third processed image.

7. The intelligent harvesting method according to claim 6, characterized in that, The step of generating a new labeled training image based on the third processed image includes: Map the corner coordinates of the four final points to the required image size of the labeled new training image to determine the perspective transformation matrix; The second processed image is transformed according to the perspective transformation matrix, and the labeled regions in the second processed image are also transformed to obtain the new labeled training image.

8. The intelligent harvesting method according to any one of claims 1 to 7, characterized in that, The harvesting robot also includes a camera mounted at the end of the second robotic arm; before performing coordinate system transformation on the position information, the intelligent harvesting method further includes: The calibration plate is fixed at a position outside the harvesting robot. The second robotic arm is moved and the camera is controlled to take pictures of the calibration plate at multiple different positions. The first transformation matrix from the camera coordinate system to the end coordinate system of the second robotic arm is determined based on the spatial position of the calibration plate and the image position of the calibration plate in the multiple first calibration plate images. The calibration plate is fixed on the first robotic arm. The first robotic arm is moved and the camera is controlled to capture images of the calibration plate at multiple different positions. A second transformation matrix from the end coordinate system of the first robotic arm to the coordinate system of the calibration plate is determined based on the spatial position of the calibration plate and the corresponding image position of the calibration plate in the second calibration plate image. A third transformation matrix from the camera coordinate system to the coordinate system of the first robotic arm base is determined based on the second transformation matrix. The fourth transformation matrix from the second robotic arm base coordinate system to the first robotic arm base coordinate system is determined based on the first transformation matrix and the third transformation matrix. The fifth transformation matrix from the camera coordinate system to the first robotic arm end coordinate system is determined based on the first transformation matrix and the fourth transformation matrix.

9. A smart harvesting device, characterized in that, A harvesting robot is used to implement the intelligent harvesting method as described in any one of claims 1 to 8. The harvesting robot includes a first robotic arm and a second robotic arm. A holding device is installed at the end of the first robotic arm, and a cutting device is installed at the end of the second robotic arm. The intelligent harvesting device includes: The acquisition module is used to acquire images of the crops to be identified. The recognition module is used to input the crop image into a trained crop detection model to determine the crop targets in the crop image and the location information of the crop targets; The first control module is used to perform coordinate system transformation on the position information, determine the first target pose information of the crop target in the coordinate system of the first robotic arm end effector, and the second target pose information of the crop target in the coordinate system of the second robotic arm end effector; control the first robotic arm to move to the first target point according to the first target pose information, and obtain the first arm segment pose information of each arm segment of the first robotic arm; The second control module is used to convert all the first arm segment pose information into the second arm segment pose information in the second arm base coordinate system based on the fourth transformation matrix between the base coordinate systems of the two robotic arms; create or update collision objects according to the second arm segment pose information; and control the second robotic arm to perform obstacle avoidance movement for the collision object according to the second target pose information until the second robotic arm moves to the second target point. The third control module is used to control the shearing device to cut the stem of the crop target, so that the crop target falls into the holding device.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the intelligent harvesting method as described in any one of claims 1 to 8.

Images