A pushing, prodding and grasping method and device under the condition of sand burial

Abstract

The present invention discloses a pushing and picking method and device under the condition of sand burial, which relates to the field of robot grasping. The method includes the following steps: collecting point cloud data of the scene through a camera, using a point cloud instance segmentation method to segment the point cloud data of the target, calculating the main direction and normal vector of the target point cloud, determining a pushing action according to the main direction of the target point cloud through the normal vector direction and the target point cloud, pushing the target out of the sand by executing the pushing action, and then executing a picking action to pick out the target from the sand. By first pushing the grasping target out of the sand through the pushing action, the present invention increases the sensing data of the target to be grasped, which is beneficial to executing the picking action and realizing the target grasping under the condition of sand burial.

Description

Technical Field

[0001] The present invention relates to the field of robot grasping, and in particular, to a pushing and grasping method and device in the case of sand burial. Background Art

[0002] Robot grasping is an important basic problem in robot technology and the basis of robot operation. Currently, most robot grasping methods adopt data-driven and learning-based methods, which can achieve good results when facing a single object. However, in the case of multiple objects stacked or an object buried in sand, due to the reduction of sensing information about the grasping target, direct grasping cannot be performed, and other methods are needed to assist in solving the grasping problem.

[0003] Most of the existing studies mainly focus on the stacking of cluttered objects. For example, the paper "Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning" published by Zeng in the "International Conference on Intelligent Robots and Systems" learns the pushing and grasping cooperation strategy through reinforcement learning. By pushing away the objects affecting grasping, the sensing information of the grasping target is increased, and finally the grasping action is completed. The paper "A Planning Framework for Non-Prehensile Manipulation under Clutter and Uncertainty" published by Zhang in "Autonomous Robots" combines object detection and grasping detection, can infer the occlusion relationship between objects in a cluttered environment, and plan the grasping order according to the occlusion relationship, and remove other objects blocking the target object to achieve the grasping of a specific target object in a cluttered environment.

[0004] These existing methods can only solve the problem of difficult grasping in a cluttered environment and cannot solve the problem of grasping an object buried in sand.

[0005] Therefore, those skilled in the art are committed to developing a pushing and grasping method and device in the case of sand burial. Summary of the Invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is to provide a pushing and grasping method in a sand burial scenario, solve the problem of less sensing data of the target object in the sand burial scenario, and improve the success rate of target grasping.

[0007] To achieve the above object, the present invention provides a pushing and grasping method under the condition of sand burial, and the method includes the following steps:

[0008] S101: Collect the point cloud data of the scene through a camera;

[0009] S103: Use the point cloud instance segmentation method to segment the point cloud data of the target;

[0010] S105: Calculate the main direction and normal vector of the target point cloud, and determine the pushing action according to the main direction of the target point cloud, through the normal vector direction and the target point cloud;

[0011] S107: Execute the pushing action to push the target out of the sand;

[0012] S109: Execute the grasping action to grab the target out of the sand.

[0013] Further, in the step S105, use the PCA principal component analysis method to calculate the main direction of the target point cloud and determine the normal vector direction of the target point cloud.

[0014] Further, in the step S105, the pushing action is determined by the pushing start position and the pushing end position, and the pushing start position and the pushing end position are determined by the coordinates of the pushing start position and the pushing end position.

[0015] Further, in the coordinate system formed by the three main directions of the point cloud, the x and y coordinates of the pushing start position are the maximum and minimum points of the point cloud in the x and y directions offset by a first distance along the direction away from the target, the x and y coordinates of the pushing end position are the x and y coordinates of the center position of the point cloud, and the z coordinates of the pushing start position and the pushing end position are the z coordinate of the center of the point cloud offset by a second distance along the normal direction of the normal vector.

[0016] Further, in the step S107, the pushing direction of the pushing action is set to any one of the four directions combined by the positive and negative x and y directions of the point cloud.

[0017] Further, in the step S107, the pushing action is executed by the robot through a two-finger gripper. When the robot executes the pushing action, the two-finger gripper tilts by a predetermined angle in the pushing direction.

[0018] Further, in the step S107, when the robot executes the pushing action, the two-finger gripper moves to the pushing start position in an inclined posture, moves above the pushing end position from the pushing start position, and completes the pushing action.

[0019] Further, before the step S109, the following judgment steps are further included:

[0020] Collect the scene point cloud data, and judge whether it meets the predetermined conditions for performing the grasping action. If the judgment result is that the predetermined conditions for the grasping action are met, then execute the step S109. Otherwise, execute the steps S103 - S107.

[0021] Further, the judgment result of whether the predetermined conditions for performing the grasping action are met is output by a grasping pose detection network. The grasping pose detection network is pre-trained, and the training data for the pre-training includes the scene point cloud data.

[0022] On the other hand, the present invention also provides a pushing and grasping device under the condition of sand burial. The device includes:

[0023] A collection module, which collects the point cloud data of the scene through the camera;

[0024] A calculation module, which segments the point cloud data of the target and determines the pushing action;

[0025] A judgment module, which judges whether the predetermined conditions for performing the grasping action are met and outputs the judgment result;

[0026] An execution module, which executes the pushing action and the grasping action to grab the target from the sand;

[0027] The device adopts the pushing and grasping method under the condition of sand burial to complete the grasping of the target under the condition of sand burial.

[0028] In a preferred embodiment of the present invention, for the target object in the sand burial scene, the present invention first pushes the target to be grabbed out of the sand through a pushing action, increases the acquisition of sensing data of the target to be grabbed, and through multiple acquisitions and analyses of the target point cloud data, is conducive to performing the grasping action, realizes the target grasping under the condition of sand burial, and improves the target grasping success rate.

[0029] The following will further illustrate the concept, specific structure and technical effects generated by the present invention in conjunction with the drawings, so as to fully understand the purpose, features and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of the pushing and grasping method under the condition of sand burial in a preferred embodiment of the present invention;

[0031] Figure 2Schematic diagram of a coordinate system established based on three main directions of point cloud in a preferred embodiment of the present invention;

[0032] Figure 3 Schematic diagram of the process of a robot pushing and prodding a stone in a preferred embodiment of the present invention;

[0033] Figure 4 Schematic diagram of the module composition of a pushing, prodding and grasping device under the condition of sand burial in a preferred embodiment of the present invention. Detailed implementation manners

[0034] The following introduces multiple preferred embodiments of the present invention with reference to the accompanying drawings of the specification, making its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0035] In the accompanying drawings, components with the same structure are denoted by the same numerical reference signs, and components with similar structures or functions are denoted by similar numerical reference signs. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of some parts in the drawings is appropriately exaggerated.

[0036] As Figure 1 shown, a pushing, prodding and grasping method under the condition of sand burial provided by an embodiment of the present invention includes the following steps:

[0037] S101: Collect point cloud data of the scene through a camera;

[0038] S103: Segment the point cloud data of the target using a point cloud instance segmentation method;

[0039] S105: Calculate the main direction and normal vector of the target point cloud, and determine a pushing and prodding action based on the main direction of the target point cloud, the normal vector direction and the target point cloud;

[0040] S107: Execute the pushing and prodding action to push the target out of the sand;

[0041] S109: Execute a grasping action to grab the target out of the sand.

[0042] When specifically executing step S105, the PCA (Principal Component Analysis) principal component analysis method is used to calculate the main direction of the target point cloud, determine the normal vector direction of the target point cloud, and the determined pushing and prodding action is determined by the pushing and prodding start position and the pushing and prodding end position.

[0043] When determining the starting position and the ending position of the pushing and prodding, a coordinate system formed by the three main directions of the point cloud is used for determination. Among them, the x and y coordinates of the starting position of the pushing and prodding are the maximum and minimum points of the point cloud in the x and y directions offset by a first distance along the direction away from the target, the x and y coordinates of the ending position of the pushing and prodding are the x and y coordinates of the center position of the point cloud, and the z coordinates of the starting position and the ending position of the pushing and prodding are the z coordinate of the center of the point cloud offset by a second distance along the normal direction of the normal vector.

[0044] When performing step S107, the pushing direction of the pushing and prodding action is randomly selected from any of the four directions combined by the positive and negative x and y directions of the point cloud, and the robot executes this pushing and prodding action through a two-finger gripper. When the robot executes the pushing action, the two-finger gripper tilts at a predetermined angle in the pushing direction to improve the pushing effect. When the robot executes the pushing and prodding action, the two-finger gripper moves to the starting position of the pushing and prodding in an inclined posture, and then moves from the starting position of the pushing and prodding above the ending position of the pushing and prodding, thereby completing the entire pushing and prodding action.

[0045] Before the step S109, the following judgment steps are further included:

[0046] Collect the scene point cloud data again, and judge whether it meets the predetermined conditions for performing the grasping action. If the judgment result is that it meets the predetermined conditions for the grasping action, then execute the step S109. Otherwise, continue to execute steps S103 - S107 to continue pushing the target to be grasped out of the sand until the grasping target can be grasped from the sand.

[0047] The above judgment steps are completed by using a grasping pose detection network. This grasping pose detection network has been trained in advance, and the training data includes scene point cloud data. According to these point cloud data, the judgment result output by the grasping pose detection network is whether the grasping operation can be performed.

[0048] The present invention provides the following two preferred embodiments. Embodiment 1 is a pushing and prodding grasping method under the condition of sand burial, and Embodiment 2 is a pushing and prodding grasping device under the condition of sand burial. These two preferred embodiments are introduced in detail below.

[0049] Embodiment 1

[0050] As Figure 1 shown, this embodiment provides a pushing and prodding grasping method under the condition of sand burial, which specifically includes the following steps:

[0051] 1) Collect the point cloud information of the grasping scene through a camera: The grasping target is a white stone, and half of the volume of the stone is buried in the sand.

[0052] 2) Use the point cloud instance segmentation method to segment the point cloud of the grasping target: Use the color region segmentation method to segment the point cloud of the stone and the point cloud of the sand scene.

[0053] 3) Calculate the main direction and normal vector of the target point cloud. Based on the three main directions of the point cloud, determine the pushing and prodding actions through the normal vector direction and the target point cloud.

[0054] Use the PCA principal component analysis method to calculate the three main directions of the point cloud, and establish a coordinate system based on the three main directions. As Figure 2 shown, the z-axis direction is the normal vector direction;

[0055] Calculate the coordinates of the center position of the point cloud cen = (x c , y c , z c );

[0056] Determine the pushing and prodding actions through the normal vector direction and the target point cloud. The pushing and prodding actions are determined by the starting position and the ending position of the pushing and prodding;

[0057] Calculate the maximum and minimum values of the x and y coordinates of the point cloud in the main direction coordinate system and denote them as x min , x max , y min , y max ;

[0058] The x and y coordinates of the starting position of the pushing and prodding are the maximum and minimum value points of the point cloud in the x and y directions, offset by a certain length d1 along the direction away from the target. The x and y coordinates of the ending position of the pushing and prodding are the x and y coordinates of the center position of the point cloud. The z coordinates of the starting position and the ending position of the pushing and prodding are the z coordinate of the center of the point cloud, offset by a certain length d2 along the normal direction of the normal vector.

[0059] There are four pushing and prodding directions along the positive and negative x and y directions, obtaining the coordinates of four starting positions of the pushing and prodding (x min - d1, y c , z c + d2), (x max + d1, y c , z c + d2), (x c , y max + d1, z c + d2), (x c , y min - d1, z c + d2), and the coordinate of one ending position of the pushing and prodding (x c , y c , z c + d2)

[0060] 4) Execute the pushing and prodding actions through the two-finger gripper of the robot to push the grasped target out of the sand.

[0061] Randomly select one of the positive and negative x and y pushing and pulling directions, and obtain the corresponding starting position and ending position of the pushing and pulling. When pushing, the gripper tilts 45 degrees in the pushing direction, so that the gripper is inserted under the object in an inclined posture to improve the pushing effect.

[0062] The specific pushing and pulling action is that the gripper moves to the starting position of the pushing and pulling in an inclined posture, moves to the ending position of the pushing and pulling, and finally moves above the ending position of the pushing and pulling to push the stone out of the sand, completing the pushing and pulling action, as Figure 3 shown.

[0063] 5) Re-collect the point cloud of the grasping scene, judge whether it can be grasped. If it cannot be grasped, continue to execute the pushing and pulling action. If it can be grasped, execute the grasping action to grab the target object out of the sand. The above judgment process uses a grasping pose detection network to judge whether the stone can be grasped. If so, the grasping pose detection network outputs a judgment result that it can be grasped. Otherwise, it outputs a judgment result that it cannot be grasped.

[0064] Embodiment 2

[0065] As Figure 4 shown, this embodiment provides a pushing and pulling grasping device under the condition of sand burial. The device includes the following modules:

[0066] The acquisition module acquires the point cloud data of the corresponding scene through a camera;

[0067] The calculation module segments the point cloud data of the target to be grasped and determines the pushing and pulling action;

[0068] The judgment module judges whether the predetermined conditions for executing the grasping action are satisfied and outputs a judgment result;

[0069] The execution module executes the pushing and pulling action and the grasping action to grab the target to be grasped out of the sand.

[0070] When the device executes the specific grasping action, it adopts the specific method described in Embodiment 1 to complete the grasping of the target under the condition of sand burial.

[0071] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations according to the concept of the present invention without creative labor. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. A pushing, prodding and grasping method under the condition of sand burial, characterized in that The method includes the following steps: S101: Collect point cloud data of the scene through a camera; S103: Use a point cloud instance segmentation method to segment the point cloud data of the target; S105: Calculate the main direction and normal vector of the target point cloud. According to the main direction of the target point cloud, determine a pushing and prodding action through the normal vector direction and the target point cloud; S107: Execute the pushing and prodding action to push the target out of the sand; S109: Execute a grasping action to grab the target out of the sand; Wherein, in step S105, use the PCA principal component analysis method to calculate the main direction of the target point cloud and determine the normal vector direction of the target point cloud; specifically, By segmenting the point cloud data of the target, use the PCA principal component analysis method to calculate the three main directions of the point cloud data, and establish a coordinate system according to the three main directions, and the z-axis direction is the normal vector direction; In step S105, the pushing and prodding action is determined by a pushing and prodding start position and a pushing and prodding end position, and the pushing and prodding start position and the pushing and prodding end position are determined by the coordinates of the pushing and prodding start position and the pushing and prodding end position; In the coordinate system formed by the three main directions of the point cloud, the x and y coordinates of the pushing and prodding start position are the maximum and minimum points of the point cloud in the x and y directions offset by a first distance along the direction away from the target, and the x and y coordinates of the pushing and prodding end position are the x and y coordinates of the center position of the point cloud, and the z coordinates of the pushing and prodding start position and the pushing and prodding end position are the z coordinate of the center of the point cloud offset by a second distance along the normal direction of the normal vector.

2. The method according to claim 1, wherein In step S107, the pushing direction of the pushing and prodding action is set to any one of the four directions combined by the positive and negative x and y directions of the point cloud.

3. The method according to claim 2, wherein In step S107, the pushing and prodding action is executed by a robot through a two-finger gripper. When the robot executes the pushing and prodding action, the two-finger gripper is tilted by a predetermined angle in the pushing direction.

4. The method according to claim 3, wherein In step S107, when the robot executes the pushing and prodding action, the two-finger gripper moves to the pushing and prodding start position in an inclined posture, moves above the pushing and prodding end position, and completes the pushing and prodding action.

5. The method according to claim 1, wherein Before step S109, the following judgment step is also included: Collect the scene point cloud data, judge whether the predetermined conditions for executing the grasping action are satisfied. If the judgment result is that the predetermined conditions for the grasping action are satisfied, then execute step S109. Otherwise, execute steps S103 - S107.

6. The method according to claim 5, wherein The judgment result of judging whether the predetermined conditions for executing the grasping action are satisfied is output by a grasping pose detection network, and the grasping pose detection network is pre-trained, and the training data of the pre-training includes the scene point cloud data.

7. A pushing, prodding and grasping device under the condition of being buried by sand and soil, characterized in that, The device includes: An acquisition module that collects the point cloud data of the scene through the camera; A calculation module that segments the point cloud data of the target and determines the pushing and prodding action; A judgment module that judges whether the predetermined conditions for performing the grasping action are satisfied and outputs the judgment result; An execution module that performs the pushing and grasping actions to grab the target from the sand; The device uses the method described in any one of claims 5-6 to complete the grasping of the target in the case of being buried in sand.

Images