Shearing method, apparatus, robot and storage medium

Abstract

This application relates to a pruning method, apparatus, robot, and storage medium. Applied to a robot, the robot includes: a workbench for placing plants, a first depth camera mounted on top of the workbench, a robot workstation, a robotic arm mounted on the robot workstation, and a second depth camera and an end effector mounted on the robotic arm. The method includes: acquiring a first image of the plant taken by the first depth camera toward the workbench; determining, based on the first image, whether there are branches to be pruned; determining, when branches to be pruned are determined to exist, the average depth of the plant; controlling the movement of the robotic arm to adjust the observation height of the second depth camera; acquiring a second image of the plant taken by the second depth camera; determining, based on the second image, the branches to be pruned; and controlling the end effector to prune the branches to be pruned. This invention enables efficient and high-precision pruning.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a shearing method, apparatus, robot, and storage medium. Background Technology

[0002] When plant branches reach a certain length, they need to be harvested. Currently, the branches to be harvested are identified manually, and then manually cut. This cutting method is inefficient, labor-intensive, and not accurate enough.

[0003] Therefore, improving the efficiency and accuracy of shearing is an urgent problem to be solved. Summary of the Invention

[0004] Therefore, it is necessary to provide a high-efficiency and high-precision shearing method, apparatus, robot, storage medium, and computer program product to address the aforementioned technical problems.

[0005] In a first aspect, this application provides a shearing method applied to a shearing robot, the shearing robot comprising: a workbench for placing plants, a first depth camera disposed on top of the workbench, a robot workstation, a robotic arm disposed on the robot workstation, and a second depth camera and an end effector disposed on the robotic arm, the method comprising:

[0006] Acquire a first image of the plant taken by the first depth camera facing the workbench;

[0007] Determine whether there are branches to be cut based on the first image;

[0008] When it is determined that there are branches to be pruned, the average depth of the plant is determined;

[0009] The robotic arm is moved according to the average depth, so that the second depth camera adjusts its observation height under the action of the robotic arm, and the observation height of the second depth camera matches the average depth.

[0010] Acquire a second image taken by the second depth camera facing the plant;

[0011] The branches of the plant to be cut are determined based on the second image;

[0012] The end effector is controlled to cut the branch to be cut.

[0013] In one embodiment, determining whether there are branches to be cut based on the first image includes:

[0014] The first image is identified based on the first neural network model to determine whether there are branches to be pruned. The first neural network model is obtained by training the first image set. Any image in the first image set has labeled branches to be pruned. It is understood that the labeling can be done by users or by machines, both are feasible, and no limitation is made here.

[0015] In one embodiment, after determining the branch of the plant to be pruned based on the second image, the method further includes:

[0016] The first position and first posture of the end effector are determined based on the position of the branch to be cut.

[0017] The robotic arm is moved according to the first position and the first posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector.

[0018] The first position is near the branch to be cut. When the end effector moves to the first position and the attitude of the end effector is adjusted to the first attitude, the branch to be cut is located within the cutting area of ​​the end effector.

[0019] In one embodiment, controlling the movement of the robotic arm based on the first position and the first posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector, includes:

[0020] The robotic arm is controlled to move, causing the end effector to move to the first position under the drive of the robotic arm, and the posture of the end effector is adjusted to the first posture; when the distance sensor of the end effector detects that the branch to be cut is within a first preset distance, it is determined that the end effector can detect the branch to be cut within the end effector cutting area, and the distance sensor is set on the end effector to detect the distance between the object in the cutting area of ​​the end effector and the distance sensor.

[0021] In one embodiment, controlling the movement of the robotic arm based on the first position and the first posture, so that the end effector can detect the branch to be cut within the end effector cutting area, includes:

[0022] The robotic arm is controlled to move, causing the end effector to move to the first position under the drive of the robotic arm, and the attitude of the end effector is adjusted to the first attitude; when the distance sensor of the end effector does not detect the branch to be cut, the robotic arm is controlled to move, causing the end effector to adjust its position and / or attitude according to a preset adjustment strategy under the drive of the robotic arm; when the distance sensor of the end effector detects the branch to be cut within a first preset distance, it is determined that the end effector can detect the branch to be cut within the cutting area of ​​the end effector.

[0023] In one embodiment, the method further includes:

[0024] After the end effector adjusts its position and / or posture according to a preset adjustment strategy under the drive of the robotic arm, when the distance sensor of the end effector does not detect the branch to be cut, the worktable is triggered to rotate, and the worktable drives the plant to rotate to the second position;

[0025] In the second position, the third position and the second posture are determined based on the position of the branch to be cut;

[0026] The robotic arm is moved according to the third position and the second posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector.

[0027] The second position can be the position of the plant after the workbench has rotated randomly for a few seconds and then stopped rotating; the second position can also be the position of the plant after the workbench has rotated by a preset angle (such as rotating the workbench 10 degrees clockwise or 20 degrees counterclockwise); the second position can also be the position of the plant after the workbench has rotated in other ways and then stopped. The rotation method of the workbench can be preset as needed and is not limited here.

[0028] It should be noted that after the plant's position changes to the second position, in order to ensure that the end effector can perform the cutting operation on the branch to be cut, it is necessary to redetermine the position the end effector has moved to (i.e., the third position) and the pose of the end effector (i.e., the second pose). The third position is near the branch to be cut. When the end effector moves to the third position and adjusts its pose to the second pose, the branch to be cut is located within the cutting area of ​​the end effector.

[0029] In one embodiment, after controlling the movement of the robotic arm according to the first position and the first posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector, and before controlling the end effector to cut the branch to be cut, the method further includes:

[0030] Control the end effector to move along the branch to be cut toward the main trunk of the plant;

[0031] When the end effector moves along the branch to be cut toward the main trunk of the plant, if the force on the end effector along the extension direction of the branch to be cut does not exceed a first threshold, the end effector is controlled to move along the branch to be cut toward the main trunk of the plant a second preset distance.

[0032] When the end effector moves along the branch to be cut toward the main trunk of the plant, if the force on the end effector along the extension direction of the branch to be cut exceeds a first threshold, the end effector is controlled to move along the branch to be cut away from the main trunk of the plant a third preset distance.

[0033] Control the end effector to swing back and forth along a direction perpendicular to the branch to be cut;

[0034] The end effector is controlled to clamp the branch to be cut.

[0035] The second preset distance can be preset according to the type of plant and the density of its branches and leaves. For example, the second preset distance can be 5 mm, 8 mm, or 15 mm, etc. The specific distance can be preset according to the actual situation and is not limited here.

[0036] In one embodiment, after controlling the end effector to clamp the branch to be cut, and before controlling the end effector to cut the branch to be cut, the method further includes:

[0037] Control the end effector to move a fourth preset distance away from the main stem of the plant.

[0038] In one embodiment, after controlling the end effector to cut the branch to be cut, the method further includes:

[0039] Control the end effector to move away from the main stem of the plant;

[0040] When the end effector is subjected to a force toward the trunk exceeding a second threshold, the end effector is controlled to cut the branch to be cut.

[0041] Secondly, this application provides a shearing device applied to a shearing robot, the shearing robot comprising: a workbench for placing plants, a first depth camera disposed on the top of the workbench, a robot workstation, a robotic arm disposed on the robot workstation, and a second depth camera and an end effector disposed on the robotic arm, the shearing comprising:

[0042] The first acquisition module is used to acquire a first image of the plant taken by the first depth camera facing the workbench;

[0043] The first determining module is used to determine whether there are branches to be cut based on the first image;

[0044] The second determining module is used to determine the average depth of the plant when it is determined that there are branches to be cut.

[0045] The first control module is used to control the movement of the robotic arm according to the average depth, so that the second depth camera adjusts its observation height under the drive of the robotic arm, and the observation height of the second depth camera matches the average depth.

[0046] The second acquisition module is used to acquire a second image taken by the second depth camera towards the plant;

[0047] The third determining module is used to determine the branches of the plant to be cut based on the second image;

[0048] The shearing module is used to control the end effector to shear the branch to be sheared.

[0049] Thirdly, this application provides a robot for shearing, the robot comprising: a workbench for placing plants, a first depth camera disposed on top of the workbench, a robot workstation, a robotic arm disposed on the robot workstation, and a second depth camera and an end effector disposed on the robotic arm, the computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0050] Acquire the first image of the plant taken by the first depth camera facing the workbench;

[0051] Determine whether there are branches to be cut based on the first image;

[0052] When it is determined that there are branches to be pruned, the average depth of the plant is determined;

[0053] The robotic arm is moved according to the average depth, so that the second depth camera adjusts its observation height under the action of the robotic arm, and the observation height of the second depth camera matches the average depth.

[0054] Acquire a second image taken by the second depth camera facing the plant;

[0055] The branches of the plant to be cut are determined based on the second image;

[0056] The end effector is controlled to cut the branch to be cut.

[0057] Fourthly, this application also provides a computer-readable storage medium for use in a robot, the shearing robot comprising: a workbench for placing plants, a first depth camera disposed on top of the workbench, a robot workstation, a robotic arm disposed on the robot workstation, and a second depth camera and an end effector disposed on the robotic arm, the computer-readable storage medium storing a computer program thereon, the computer program performing the following steps when executed by a processor:

[0058] Acquire a first image of the plant taken by the first depth camera facing the workbench;

[0059] Determine whether there are branches to be cut based on the first image;

[0060] When it is determined that there are branches to be pruned, the average depth of the plant is determined;

[0061] The robotic arm is moved according to the average depth, so that the second depth camera adjusts its observation height under the action of the robotic arm, and the observation height of the second depth camera matches the average depth.

[0062] Acquire a second image taken by the second depth camera facing the plant;

[0063] The branches of the plant to be cut are determined based on the second image;

[0064] The end effector is controlled to cut the branch to be cut.

[0065] Fifthly, this application also provides a computer program product applied to a robot, the shearing robot comprising: a workbench for placing plants, a first depth camera disposed on top of the workbench, a robot workstation, a robotic arm disposed on the robot workstation, and a second depth camera and an end effector disposed on the robotic arm, the computer program product comprising a computer program that, when executed by a processor, performs the following steps:

[0066] Acquire a first image of the plant taken by a first depth camera facing the workbench;

[0067] Determine whether there are branches to be cut based on the first image;

[0068] When it is determined that there are branches to be pruned, the average depth of the plant is determined;

[0069] The robotic arm is moved according to the average depth, so that the second depth camera adjusts its observation height under the action of the robotic arm, and the observation height of the second depth camera matches the average depth.

[0070] Acquire a second image taken by the second depth camera facing the plant;

[0071] The branches of the plant to be cut are determined based on the second image;

[0072] The end effector is controlled to cut the branch to be cut.

[0073] The aforementioned pruning method, apparatus, computer equipment, and storage medium utilize a pruning robot to prune the branches to be pruned, thereby improving pruning efficiency. A first depth camera acquires the average depth of the branches to be pruned and the plant. Then, a robotic arm moves a second depth camera to match its observation height with the average depth of the plant. Based on the second image captured by the second depth camera, the branches to be pruned are determined, and an end effector performs the pruning operation. This process enables precise positioning of the branches to be pruned, improving pruning accuracy. Attached Figure Description

[0074] Figure 1 This is a diagram illustrating the application environment of the shearing method in one embodiment;

[0075] Figure 2A This is a flowchart illustrating the cutting method in one embodiment;

[0076] Figure 2B A schematic diagram of plants identified from an image captured by a first depth camera;

[0077] Figure 2C A schematic diagram of plants identified from another image captured by a first depth camera;

[0078] Figure 3 This is a flowchart illustrating the cutting method in one embodiment;

[0079] Figure 4A This is a schematic diagram showing the relative position of the end effector on the robotic arm and the branch to be cut in one embodiment;

[0080] Figure 4B for Figure 4A A schematic diagram showing the relative position of the end effector to the branch to be cut after it has moved.

[0081] Figure 4C for Figure 4B A schematic diagram showing the relative position of the end effector to the branch to be cut after it has moved.

[0082] Figure 4D for Figure 4C A schematic diagram showing the relative position of the end effector to the branch to be cut after it has moved.

[0083] Figure 5A This is a schematic diagram showing the relative position of the end effector on the robotic arm and the branch to be cut in one embodiment;

[0084] Figure 5B for Figure 5A A schematic diagram showing the relative position of the end effector to the branch to be cut after it has moved.

[0085] Figure 6A This is a schematic diagram showing the relative position of the end effector on the robotic arm and the branch to be cut in one embodiment;

[0086] Figure 6B for Figure 6A A schematic diagram showing the relative position of the end effector to the branch to be cut after it has moved.

[0087] Figure 6C for Figure 6B A schematic diagram showing the relative position of the end effector to the branch to be cut after it has moved.

[0088] Figure 6D for Figure 6C A schematic diagram showing the relative position of the end effector to the branch to be cut after it has moved.

[0089] Figure 6E for Figure 6D A schematic diagram showing the relative position of the end effector to the branch to be cut after it has moved.

[0090] Figure 7 This is a flowchart illustrating the cutting method in one embodiment;

[0091] Figure 8A This is a schematic diagram showing the relative position of the end effector on the robotic arm and the branch to be cut in one embodiment;

[0092] Figure 8B for Figure 8A A schematic diagram showing the relative position of the end effector to the branch to be cut after it has moved.

[0093] Figure 8C for Figure 8B A schematic diagram of the end effector swinging in the image;

[0094] Figure 8D for Figure 8C A schematic diagram of the end effector cutting the branch to be cut;

[0095] Figure 9A This is a schematic diagram of the movement of the end effector on the robotic arm after performing a cutting operation on the branch to be cut in one embodiment.

[0096] Figure 9B for Figure 9A A schematic diagram showing that the branch to be cut was not severed after the end effector performed the cutting operation on the branch to be cut;

[0097] Figure 9C for Figure 9A A schematic diagram showing the branch being cut off after the end effector performs a cutting operation on the branch to be cut.

[0098] Figure 10 This is a structural block diagram of the shearing device in one embodiment;

[0099] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0101] The cutting method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, the shearing system 103 communicates with the controller 102 via a network. The data storage system 101 can store the data that the controller 102 needs to process. The data storage system 101 can be integrated into the controller 102, or it can be placed in the cloud or on another network server. This application provides a shearing method applied to a shearing robot. The shearing robot includes: a workbench 111 for placing a plant 110, a first depth camera 109 mounted on the top of the workbench 111, a robot workstation 104, a robotic arm mounted on the robot workstation 104, and a second depth camera 105 and an end effector 106 mounted on the robotic arm. The shearing method includes the following steps: acquiring a first image of the plant 110 taken by the first depth camera 109 toward the workbench 111; determining whether there are branches to be sheared based on the first image; determining the average depth of the plant 110 when it is determined that there are branches to be sheared; controlling the movement of the robotic arm based on the average depth, so that the observation height of the second depth camera 105 is adjusted under the drive of the robotic arm, so that the observation height of the second depth camera 105 matches the average depth; acquiring a second image of the plant 110 taken by the second depth camera 106; determining the branches of the plant 110 to be sheared based on the second image; and controlling the end effector to shear the branches to be sheared. This shearing method can also be applied to scenarios where the shearing robot interacts with the controller 102. For example, the first image captured by the first depth camera 109 can be uploaded to the controller 102, which will then recognize the image and return the result to the shearing robot. The harvesting robot will then move its harvesting mechanism to the shearing area based on the recognition result and adjust the position and orientation of the end effector. Finally, it will control the shearing mechanism of the end effector to cut the branch to be sheared. In some embodiments, the controller 102 can also be integrated into the shearing robot; this is feasible and not limited here. In some possible embodiments, the controller 102 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0102] In one embodiment, as shown in Figure 2, a cutting method is provided, which is applied to... Figure 1 The following steps, 201 to 207, are illustrated using a shearing robot as an example:

[0103] 201. Obtain the first image of the plant taken by the first depth camera facing the workbench.

[0104] For example, with Figure 1 For example, in some embodiments, the first depth camera 109 takes a first image of the plant 110 facing the worktable 111.

[0105] 202. Determine whether there are branches to be cut based on the first image.

[0106] In some embodiments, a neural network model can be used to perform image recognition on the first image to determine whether there are branches to be pruned. The neural network model can be obtained by training a model on an image set, which can be obtained in advance by taking pictures of the plant placed on the workbench using a first depth camera, and then manually labeling the images to indicate whether there are branches to be pruned. Understandably, the neural network model can also be trained in other ways, which are not limited here.

[0107] In one embodiment, plants in the first image are identified to obtain... Figure 2B In another embodiment, plants in another first image are identified to obtain... Figure 2C The branches to be pruned can be predefined. For example, the length of the branch to be pruned can be defined as greater than or equal to 8 centimeters. If a branch longer than 8 centimeters is detected, that branch can be identified as the branch to be pruned. After identification, if... Figure 2B If all the branches are less than 8 centimeters in length, then Figure 2B The corresponding plant does not have any branches to be pruned. If Figure 2C If the plant in the image has branches longer than 8 centimeters, then... Figure 2C The corresponding plant has branches that need to be pruned.

[0108] Specifically, to avoid missing any branches to be pruned, the criteria for determining the existence of branches to be pruned in this step can be appropriately relaxed. For example, if the plant is identified as a specified type, it is determined that there is a branch to be pruned; if the plant is not identified as a specified type, it is determined that there is no branch to be pruned. The criteria for determining whether there is a branch to be pruned can be set as needed and are not limited here.

[0109] 203. When it is determined that there are branches to be pruned, determine the average depth of the plant.

[0110] In some implementations, a first depth camera can be used to capture images of the plants and obtain depth information at various points, wherein the distance from the worktable to the top camera is known, the depth information is retained at points between the worktable surface and the top camera, and the average depth of these points is calculated as the average depth of the plants.

[0111] 204. Control the movement of the robotic arm according to the average depth, so that the observation height of the second depth camera is adjusted under the drive of the robotic arm, so that the observation height of the second depth camera matches the average depth.

[0112] In some implementations, the observation height of the second depth camera on the robotic arm can be determined based on the average depth. For example, the observation height of the second depth camera = the height difference between the first depth camera and the robot's workbench base - the average depth + a constant. The constant can be set empirically, ensuring that the second depth camera can capture a complete image of the plant at that observation height. It is understood that the constant may not be exactly the same for different types of plants, and the specific value of the constant can be determined based on factors such as the plant species; no specific limitation is made here.

[0113] 205. Obtain the second image taken by the second depth camera towards the plant.

[0114] It should be noted that the first depth camera views the plant from above, while the second depth camera observes the plant from the side.

[0115] 206. Determine the branches of the plant to be cut based on the second image.

[0116] The branch information obtained by photographing the plant from the side is relatively accurate. The branch to be cut can be determined based on the second image captured by the second depth camera. The location of the branch to be cut can be determined from the second image, and based on that location, the position and orientation of the end effector can be determined. The end effector's shearing structure is then controlled to move to the vicinity of the branch to be cut, ensuring that the branch is within the shearing area of ​​the shearing structure. Sensors can be used to sense and determine whether the branch to be cut is within the shearing structure.

[0117] 207. Control the end effector to cut the branch to be cut.

[0118] After the branch to be cut is located in the shearing area of ​​the shearing structure, the clamping structure of the end effector can be controlled to clamp the branch to be cut, and then the shearing structure is used to cut the branch to be cut. It should be noted that the clamping structure and the shearing structure are not limited to the structures shown in the accompanying drawings of this application, and can also be other structures that can achieve the same function.

[0119] The above-described shearing method utilizes a shearing robot to cut the branches to be cut, thereby improving shearing efficiency. A first depth camera acquires the average depth of the branches to be cut and the plant. Then, the robotic arm moves a second depth camera to match the observation height of the second depth camera with the average depth of the plant. Based on the second image captured by the second depth camera, the branches to be cut are determined, and the end effector performs the shearing operation on the branches to be cut. Through the above operation, the branches to be cut can be accurately located, thus improving the shearing accuracy.

[0120] In one embodiment, such as Figure 3 As shown, a cutting method is provided, which is applied to... Figure 1 Taking a shearing robot as an example, this embodiment describes in detail how to adjust the end effector after determining the branch to be sheared from the second image. The shearing method provided in this embodiment includes the following steps 301 to 309, wherein:

[0121] 301. Obtain the first image of the plant taken by the first depth camera facing the workbench.

[0122] 302. Determine whether there are branches to be cut based on the first image.

[0123] 303. When it is determined that there are branches to be pruned, determine the average depth of the plant.

[0124] 304. Control the movement of the robotic arm according to the average depth, so that the observation height of the second depth camera is adjusted under the drive of the robotic arm, so that the observation height of the second depth camera matches the average depth.

[0125] 305. Obtain the second image taken by the second depth camera towards the plant.

[0126] 306. Determine the branches of the plant to be cut based on the second image.

[0127] It should be noted that steps 301 to 306 are the same as steps 201 to 206 in the previous embodiments. For the specific implementation process, please refer to the description of steps 201 to 206 in the previous embodiments. For the sake of brevity, they will not be repeated here.

[0128] 307. Determine the first position and first attitude of the end effector based on the position of the branch to be cut.

[0129] The first position is near the branch to be cut. When the end effector moves to the first position and the attitude of the end effector is adjusted to the first attitude, the branch to be cut is located within the cutting area of ​​the end effector.

[0130] In some implementations, key points can be obtained based on the location of the branch to be cut, where the key points can be the intersection of the branch to be cut and the main trunk; based on the key points, the growth direction of the branch to be cut is simulated to obtain simulation results, and based on the simulation results, the first position and first pose of the end effector are determined.

[0131] It should be noted that if the end effector is difficult to position and assume the first position and first posture due to obstruction by other objects, the worktable can be rotated to re-determine the first position and first posture.

[0132] 308. Control the movement of the robotic arm according to the first position and the first posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector.

[0133] In some possible implementations, the robotic arm can be controlled to move, causing the end effector to move to a first position and adjust its posture to a first posture. When the end effector's distance sensor detects the presence of a branch to be cut within a first preset distance, it is determined that the end effector can detect the branch to be cut within its cutting area. The distance sensor can be set to detect the distance between the object within the end effector's cutting area and the distance sensor. See also Figures 4A to 4D In one embodiment, the initial position between the end effector 400 and the branch to be pruned is as follows: Figure 4A As shown, the cutting plane of the branch to be cut is perpendicular to the end effector 400, as indicated by M in the figure. A distance sensor 401 is mounted on the end effector 400. Figure 4A In the middle, the distance sensor 401 cannot detect the branch M to be cut. Based on the first position and the first attitude, the end effector, driven by the robotic arm, is... Figure 4A Move to the position shown Figure 4B As shown in the image, after the end effector rotates clockwise, it gets closer to the branch to be pruned. Figure 4C As shown, in Figure 4C At the position shown, distance sensor 401 can detect the branch M to be cut. Since the branch M is far from distance sensor 401, it is determined that the branch M is not within the cutting area of ​​end effector 400. Therefore, the end effector 400 is controlled to move along a direction perpendicular to the branch to be cut, until it moves to the position shown. Figure 4D If the distance sensor 401 detects a branch to be cut within a specified distance at the position shown, it can be determined that a branch to be cut has been detected within the cutting area of ​​the end effector.

[0134] 309. Control the end effector to cut the branch to be cut.

[0135] After the branch to be cut is located in the shearing area of ​​the shearing structure, the clamping structure of the end effector can be controlled to clamp the branch to be cut, and then the shearing structure is used to cut the branch to be cut.

[0136] The above-mentioned shearing method improves shearing efficiency and accuracy by controlling the movement of the robotic arm so that the end effector can detect the branch to be sheared within the shearing area of ​​the end effector.

[0137] In some possible embodiments, the robotic arm is controlled to move, causing the end effector to move to a first position and adjust its posture to a first posture. When the distance sensor of the end effector does not detect the branch to be cut, the robotic arm can be controlled to move, causing the end effector to adjust its position and / or posture according to a preset adjustment strategy. When the distance sensor of the end effector detects the presence of the branch to be cut within a first preset distance, it is determined that the end effector can detect the branch to be cut within its cutting area.

[0138] like Figure 5A and Figure 5B As shown, according to a preset adjustment strategy, the end effector 500 can be controlled to move upwards. The distance sensor 501 detects the plant N to be cut and controls the end effector 500 to move towards the branch N to be cut, so that the branch N to be cut is located within the cutting area of ​​the end effector 500. The preset adjustment strategy may include the end effector moving in different directions; the specific movement is not limited in this application and can be limited as needed.

[0139] by Figures 6A to 6E For example, when the end effector 600 does not detect the branch X to be pruned, if the preset adjustment strategy is to control the end effector to rotate counterclockwise, the relative position of the end effector 600 and the branch X to be pruned will change from... Figure 6A The position shown becomes Figure 6B The location is shown. Distance sensor 601 did not detect the branch X to be pruned. Based on the prediction adjustment strategy, the end sensor is controlled to move to the left and clockwise by a preset angle to obtain the desired position. Figure 6C Then, according to the preset adjustment strategy, the end sensor moves clockwise and to the right a preset distance to obtain... Figure 6D The relative position of the end effector 500 and the branch X to be cut is shown. Further, according to a preset adjustment strategy, the end effector 600 continues to move clockwise. After the distance sensor 601 detects the branch X to be cut, it moves towards the branch X perpendicular to it, so that the branch X is located within the cutting area of ​​the end effector 600. Figure 6E As shown.

[0140] In some possible implementations, after controlling the movement of the robotic arm according to the first position and the first attitude, so that the end effector can detect the branch to be cut within the end effector's shearing area, and before controlling the end effector to cut the branch to be cut, such as Figure 7 As shown, the cutting method may also include steps 701 to 705:

[0141] 701. Control the end effector to move along the branch to be cut toward the main stem of the plant.

[0142] It should be noted that in some embodiments, when the first position and first orientation of the end effector are far from the root of the branch to be cut, it is desirable to cut the branch as close as possible to the root of the branch to be cut, such as... Figure 8A As shown, in this embodiment, the branch to be cut is already within the cutting area of ​​the end effector.

[0143] 702. When the end effector moves along the branch to be cut toward the main stem of the plant, if the force on the end effector along the extension direction of the branch to be cut does not exceed the first threshold, the end effector is controlled to move along the branch to be cut toward the main stem of the plant a second preset distance.

[0144] When the end effector moves a specified distance along the branch to be cut toward the plant's trunk, or when the end effector receives a force along the extension direction of the branch to be cut, the relative positions of the end effector 803 with the branch to be cut 801 and the main stem 802 of the plant are as follows: Figure 8B As shown.

[0145] 703. When the end effector moves along the branch to be cut toward the main trunk of the plant, if the force on the end effector along the extension direction of the branch to be cut exceeds the first threshold, the end effector is controlled to move along the branch to be cut away from the main trunk of the plant by a third preset distance.

[0146] The third preset distance can be 0 or a pre-specified value. The specific value can be determined based on experience and is not limited here.

[0147] 704. Control the end effector to swing back and forth along the direction perpendicular to the branch to be cut. For example... Figure 8C As shown.

[0148] 705. Control the end effector to clamp the branch to be cut.

[0149] Then the end effector performs the cutting operation on the branch to be cut, such as Figure 8D As shown.

[0150] It should be noted that, after controlling the end effector to clamp the branch to be cut, and before controlling the end effector to cut the branch to be cut, the cutting method may further include: controlling the end effector to move a fourth preset distance away from the main stem of the plant. The fourth preset distance can be set as needed. By controlling the end effector to move away from the main stem of the plant, the branch to be cut can be tensioned, which is beneficial for the cutting operation.

[0151] In some possible embodiments, due to factors such as the branch to be cut being relatively thick or other errors occurring during the operation, the end effector may not completely sever the branch after performing the cutting operation. Specifically, after completing the cutting operation, the end effector moves along the growth direction of the plant to be cut, such as... Figure 9A As shown, the sensor on the end sensor detects whether the branch to be cut receives a force greater than a preset value along the direction of the branch to be cut toward the main trunk. If so, the branch to be cut is not severed. Figure 9B As shown, the end effector can be controlled to perform a second cutting operation on the branch to be cut. Otherwise, the branch to be cut has not been severed. Figure 9C As shown, the end effector moves the cut branch to a designated position, then controls the clamping module to release, placing the cut branch in the designated location. If the number of cuts is less than the preset number, the branch is cut, indicating that the cutting is complete. If the preset number of cuts is reached but the branch is still not cut, the cutting process stops, indicating that the cutting is incomplete.

[0152] Based on the same inventive concept, this application also provides a cutting device. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more cutting device embodiments provided below can be found in the limitations of the cutting method above, and will not be repeated here.

[0153] In one embodiment, such as Figure 10 As shown, a shearing device 1000 is provided, including: a first acquisition module 1001, a first determination module 1002, a first control module 1003, a second determination module 1004, a second acquisition module 1005, a third determination module 1006, and a shearing module 1007.

[0154] The first acquisition module 1001 is used to acquire the first image of the plant taken by the first depth camera facing the workbench.

[0155] The first determining module 1002 is used to determine whether there are branches to be cut based on the first image.

[0156] The second determining module 1004 is used to determine the average depth of the plant when it is determined that there are branches to be cut.

[0157] The first control module 1003 is used to control the movement of the robotic arm according to the average depth, so that the second depth camera adjusts its observation height under the drive of the robotic arm, and the observation height of the second depth camera matches the average depth.

[0158] The second acquisition module 1005 is used to acquire a second image taken by the second depth camera facing the plant.

[0159] The third determining module 1006 is used to determine the branches of the plant to be cut based on the second image.

[0160] The shearing module 1007 is used to control the end effector to shear the branches to be sheared.

[0161] For specific limitations in the example, please refer to the limitations on the cutting method mentioned above, which will not be repeated here.

[0162] In one embodiment, the first determining module is specifically used to identify the first image based on the first neural network model to determine whether there are branches to be cut; the first neural network model is obtained by training the first image set, and any image in the first image set has been labeled by the user with branches to be cut.

[0163] In one embodiment, the shearing device may further include a fourth determining module and a second control module. The fourth determining module is used to determine a first position and a first orientation of the end effector based on the position of the branch to be sheared. The second control module is used to control the movement of the robotic arm based on the first position and the first orientation, so that the end effector can detect the branch to be sheared within its shearing area.

[0164] In one embodiment, the second control module is specifically used to control the movement of the robotic arm, so that the end effector moves to a first position under the drive of the robotic arm and adjusts the posture of the end effector to a first posture. When the distance sensor of the end effector detects the presence of a branch to be cut within a first preset distance, it is determined that the end effector can detect the branch to be cut within the cutting area of ​​the end effector. The distance sensor is set on the end effector to detect the distance between the object in the cutting area of ​​the end effector and the distance sensor.

[0165] In one embodiment, in controlling the movement of the robotic arm according to the first position and the first posture so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector, the second control module is further configured to control the movement of the robotic arm so that the end effector moves to the first position under the drive of the robotic arm and adjusts the posture of the end effector to the first posture. When the distance sensor of the end effector does not detect the branch to be cut, the robotic arm is controlled to move so that the end effector adjusts the position and / or posture of the end effector according to a preset adjustment strategy under the drive of the robotic arm. When the distance sensor of the end effector detects the presence of the branch to be cut within a first preset distance, it is determined that the end effector can detect the branch to be cut within the shearing area of ​​the end effector.

[0166] In one embodiment, the shearing device may further include a third control module and a fifth determining module. The third control module is configured to, after the end effector adjusts its position and / or orientation according to a preset adjustment strategy under the drive of the robotic arm, trigger a worktable rotation when the distance sensor of the end effector does not detect the branch to be sheared. The worktable then rotates the plant to a second position. The fifth determining module is configured to, at the second position, determine a third position and a second orientation based on the position of the branch to be sheared. The third control module is further configured to control the movement of the robotic arm based on the third position and the second orientation, so that the end effector can detect the branch to be sheared within its shearing area.

[0167] In one embodiment, the shearing device may further include a fourth control module, configured to: control the end effector to move along the branch to be sheared toward the main stem of the plant after the second control module controls the movement of the robotic arm according to the first position and the first posture, so that the end effector can detect the branch to be sheared within the shearing area of ​​the end effector, and before controlling the end effector to shear the branch to be sheared; and control the end effector to move along the branch to be sheared toward the main stem of the plant a second preset distance if the force on the end effector along the extension direction of the branch to be sheared does not exceed a first threshold when the end effector moves along the branch to be sheared toward the main stem of the plant; control the end effector to move along the branch to be sheared away from the main stem of the plant a third preset distance if the force on the end effector along the extension direction of the branch to be sheared exceeds the first threshold when the end effector moves along the branch to be sheared toward the main stem of the plant; control the end effector to swing back and forth in a direction perpendicular to the branch to be sheared; and control the end effector to clamp the branch to be sheared.

[0168] In one embodiment, after the end effector clamps the branch to be cut, and before the end effector cuts the branch to be cut, the fourth control module can also be used to control the end effector to move a fourth preset distance away from the main stem of the plant.

[0169] In one embodiment, after the end effector cuts the branch to be cut, the fourth control module is further configured to control the end effector to move away from the main stem of the plant; and to control the end effector to cut the branch to be cut when the force on the end effector toward the main stem exceeds a second threshold.

[0170] Each module in the aforementioned shearing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0171] In one embodiment, a robot 1100 is provided, which may include a server, the internal structure of which can be shown in the diagram below. Figure 11 As shown, the robot 1100 includes a processor 1102, memory, and a network interface 1108 connected via a system bus 1101. The processor 1102 provides computational and control capabilities. The memory of the robot 1100 includes a non-volatile storage medium 1103 and internal memory 1107. The non-volatile storage medium 1103 stores an operating system 1104, a computer program 1105, and a database 116. The internal memory 1107 provides an environment for the operation of the operating system 1104 and the computer program 1105 stored in the non-volatile storage medium 1103. The database 1106 of the robot 1100 stores data. The network interface 1108 of the robot 1100 is used to communicate with external terminals via a network connection. When the computer program 1105 is executed by the processor 1102, it implements a cutting method.

[0172] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the robot to which the present application is applied. A specific robot may include more or fewer parts than shown in the figure, or combine certain parts, or have different part arrangements.

[0173] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0174] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0175] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory, magnetic variable memory, ferroelectric memory, phase change memory, graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A cutting method, characterized in that, Applied to a robot, the robot comprising: a workbench for placing plants, a first depth camera disposed on top of the workbench, a robot workstation, a robotic arm disposed on the robot workstation, and a second depth camera and an end effector disposed on the robotic arm, the method comprising: A first image of the plant is acquired by the first depth camera, which is directed toward the workbench; wherein the first depth camera views the plant from above. Determine whether there are branches to be cut based on the first image; When it is determined that there are branches to be pruned, the average depth of the plant is determined; The robotic arm is moved according to the average depth, so that the second depth camera adjusts its observation height under the action of the robotic arm, and the observation height of the second depth camera matches the average depth; wherein, the observation height of the second camera = the height difference between the first depth camera and the robot workbench base - the average depth + a constant, and the constant is determined according to the type of plant and is used to adjust the observation height to capture a complete image of the plant. A second image is acquired by the second depth camera, which is directed toward the plant; wherein the second depth camera observes the plant from the side. The branches of the plant to be cut are determined based on the second image; The end effector is controlled to cut the branch to be cut.

2. The method according to claim 1, characterized in that, The step of determining whether there are branches to be cut based on the first image includes: The first image is identified based on the first neural network model to determine whether there are branches to be cut. The first neural network model is obtained by training the model on the first image set, where any image in the first image set has a labeled branch to be cut.

3. The method according to claim 1, characterized in that, After determining the branch of the plant to be pruned based on the second image, the method further includes: The first position and first posture of the end effector are determined based on the position of the branch to be cut; The robotic arm is moved according to the first position and the first posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector.

4. The method according to claim 3, characterized in that, The step of controlling the movement of the robotic arm based on the first position and the first posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector, includes: Control the movement of the robotic arm so that the end effector moves to the first position under the drive of the robotic arm, and adjusts the posture of the end effector to the first posture; When the distance sensor of the end effector detects that the branch to be cut is within a first preset distance, it is determined that the branch to be cut can be detected within the cutting area of ​​the end effector. The distance sensor is set on the end effector to detect the distance between the object in the cutting area of ​​the end effector and the distance sensor.

5. The method according to claim 4, characterized in that, The step of controlling the movement of the robotic arm based on the first position and the first posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector, includes: Control the movement of the robotic arm so that the end effector moves to the first position under the drive of the robotic arm, and adjusts the posture of the end effector to the first posture; When the distance sensor of the end effector does not detect the branch to be cut, the robotic arm is controlled to move, so that the end effector adjusts its position and / or attitude according to a preset adjustment strategy under the drive of the robotic arm. When the distance sensor of the end effector detects that the branch to be cut is within a first preset distance, it is determined that the end effector can detect the branch to be cut within the cutting area of ​​the end effector.

6. The method according to claim 5, characterized in that, The method further includes: After the end effector adjusts its position and / or posture according to a preset adjustment strategy under the drive of the robotic arm, when the distance sensor of the end effector does not detect the branch to be cut, the worktable is triggered to rotate, and the worktable drives the plant to rotate to the second position; In the second position, the third position and second posture of the end effector are determined according to the position of the branch to be cut; The robotic arm is moved according to the third position and the second posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector.

7. The method according to any one of claims 3 to 6, characterized in that, After controlling the movement of the robotic arm according to the first position and the first posture, so that the end effector can detect the branch to be cut within the shearing area of ​​the end effector, and before controlling the end effector to cut the branch to be cut, the method further includes: Control the end effector to move along the branch to be cut toward the main trunk of the plant; When the end effector moves along the branch to be cut toward the main trunk of the plant, if the force on the end effector along the extension direction of the branch to be cut does not exceed a first threshold, the end effector is controlled to move along the branch to be cut toward the main trunk of the plant a second preset distance. When the end effector moves along the branch to be cut toward the main trunk of the plant, if the force on the end effector along the extension direction of the branch to be cut exceeds a first threshold, the end effector is controlled to move along the branch to be cut away from the main trunk of the plant a third preset distance. Control the end effector to swing back and forth along a direction perpendicular to the branch to be cut; The end effector is controlled to clamp the branch to be cut.

8. The method according to claim 7, characterized in that, After the end effector is controlled to clamp the branch to be cut, and before the end effector is controlled to cut the branch to be cut, the method further includes: Control the end effector to move a fourth preset distance away from the main stem of the plant.

9. The method according to any one of claims 1 to 6, characterized in that, After controlling the end effector to cut the branch to be cut, the method further includes: Control the end effector to move away from the main stem of the plant; When the end effector is subjected to a force toward the trunk exceeding a second threshold, the end effector is controlled to cut the branch to be cut.

10. A shearing device, characterized in that, The application is to a robot, the robot comprising: a workbench for placing plants, a first depth camera mounted on top of the workbench, a robot workstation, a robotic arm mounted on the robot workstation, and a second depth camera and an end effector mounted on the robotic arm, wherein the shearing includes: The first acquisition module is used to acquire a first image of the plant taken by the first depth camera facing the workbench; wherein the first depth camera is looking down at the plant from above. The first determining module is used to determine whether there are branches to be cut based on the first image; The second determining module is used to determine the average depth of the plant when it is determined that there are branches to be cut. The first control module is used to control the movement of the robotic arm according to the average depth, so that the second depth camera adjusts its observation height under the drive of the robotic arm, and the observation height of the second depth camera matches the average depth; wherein, the observation height of the second camera = the height difference between the first depth camera and the robot workbench base - the average depth + a constant, and the constant is determined according to the type of plant, and is used to adjust the observation height to capture a complete image of the plant; The second acquisition module is used to acquire a second image taken by the second depth camera facing the plant; wherein the second depth camera observes the plant from the side; The third determining module is used to determine the branches of the plant to be cut based on the second image; The shearing module is used to control the end effector to shear the branch to be sheared.

11. A robot comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images