A 6-DOF grasping posture generation method for robots in cluttered environments

Abstract

The present invention belongs to the field of robot grasping and operation technology, and discloses a 6-DOF grasping posture generation method for a robot in a cluttered environment, including S1: training data set generation; S2: neural network training; S3: conversion of prediction results to a grasping posture homogeneous matrix. The present invention is an end-to-end 6-DOF grasping posture generation method, that is, the input scene point cloud directly outputs a diverse and dense successful grasping posture of each object in the scene. The diversity of postures can ensure that the robot can still meet the grasping requirements when considering obstacle avoidance and kinematic constraints; during specific execution, the grasping posture with the highest probability of the pointing point predicted by the network and for which the robot has a kinematic solution is selected for execution; there is no need for a long sampling process and grasping posture evaluation process, and it is more universal than the 3‑4-DOF grasping posture; the proposed new grasping posture representation method only needs to predict g i =(q i ,a i ,θ i ,d i ) four quantities, and does not require orthogonal constraints between vectors, which is more conducive to the learning of neural networks.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot grasping and operation, and in particular relates to a method for generating a 6-DOF grasping posture of a robot in a cluttered environment. Background Art

[0002] Robotic grasping technology has been studied in the field of robotics for decades. As the most fundamental form of interaction between robots and other objects, it forms the cornerstone for more complex tasks, such as manipulation. Previously, robots required a known, single target object in a simple, unobstructed environment (a single-color background with no obstacles). However, with the recent development of deep learning technology and the increase in computing power, the demands on robots for tasks are becoming increasingly stringent. Current robotic grasping technology is evolving towards handling more complex tasks, requiring robots to be able to adaptively grasp arbitrary, unseen objects in a cluttered environment.

[0003] To simplify the task, many studies have opted to find grasping posture configurations with 3 or 4 degrees of freedom [1,2]. This restricts the grasping posture to be perpendicular to the tabletop, representing the grasping posture configuration as or . A commonly used dataset for this approach is the Cornell Grasping Dataset, where objects in the scene are labeled with rectangular bounding boxes to represent the grasping posture. RGB images are typically used as input to a neural network, which outputs a oriented rectangular bounding box representing the grasping posture. This problem is often formulated as a regression problem, similar to object detection evaluation methods, using the intersection-over-union (IOU) ratio between the predicted bounding box and the labeled bounding box as the evaluation criterion. Due to its limited degrees of freedom, this grasping method often encounters difficulties in cluttered scenes, as randomly stacked objects require additional degrees of freedom to grasp (making grasping from a vertical direction impossible). Furthermore, the grasping configuration represented by the rectangle is often subjective, and using IOU as an evaluation criterion cannot truly reflect the quality of the grasping configuration.

[0004] Another commonly used method is the sampling-based GPD[3,4] 6-DOF grasping posture detection method, which directly detects grasping postures in point cloud scenes within a cluttered workspace. This method involves the following steps: 1. Randomly selecting points in the point cloud scene, typically sampling several thousand points to ensure that a sufficient number of candidate grasping postures can be detected; 2. Establishing a coordinate system at the sampled points; 3. Performing grasping posture detection based on specific grasping posture search rules to obtain a set of candidate grasping postures; 4. Evaluating the grasping postures in the candidate grasping set according to the force closure principle or GWS.

[0005] Grasping posture requirements: The robot gripper does not collide with the point cloud, and the gripper closed area contains at least one point in the point cloud.

[0006] Coordinate system establishment: For each sampling point, a coordinate system is established based on the feature vector.

[0007]

[0008] in represents the unit normal vector at point q, B r (p) represents the point cloud of the area with radius r at point p.

[0009] Search rules: Perform a two-dimensional grasping posture search, translate and rotate the grasping posture along the Y axis of the established coordinate system and around the Z axis of the coordinate system respectively, and add the grasping posture that meets the requirements as a candidate to the candidate set.

[0010] The sampling and search of this method is very time-consuming, generally taking several seconds to complete. When only single-view scene information can be obtained, it is difficult to ensure that the searched gripper does not collide with the point cloud. Therefore, it is often necessary to perform three-dimensional reconstruction of scene information obtained from multiple perspectives to ensure that the searched grasping posture meets the requirements. Summary of the Invention

[0011] The purpose of the present invention is to provide a method for generating a 6-DOF grasping posture of a robot in a cluttered environment to solve the above-mentioned technical problems.

[0012] To solve the above technical problems, the specific technical solution of the method for generating a 6-DOF grasping posture of a robot in a cluttered environment of the present invention is as follows:

[0013] A method for generating a 6-DOF grasping posture of a robot in a cluttered environment comprises the following steps:

[0014] S1: training dataset generation;

[0015] S2: neural network training;

[0016] S3: The prediction result is converted to the grasping posture homogeneous matrix.

[0017] Furthermore, the S1 includes the following specific steps:

[0018] The synthetic object is sampled by grasping posture in a simulation environment. The simulation scene contains a table with 1 to 12 objects randomly scattered on the table, which is statically stable under the action of gravity. Using antipodal sampling, any point on the object mesh surface is sampled as the first contact point of the parallel gripper. The normal vector at this point and the point can obtain a cone. A line is sampled inside the cone, and the intersection of this line and the object mesh is the second contact point. Then, uniform rotation sampling is performed based on the center point between the two contact points and the straight line. The postures that collide with the gripper and the postures that have no intersection with the space between the gripper fingers are removed. The above steps are repeated until enough grasping posture candidates are obtained. The obtained candidate postures are evaluated in the simulation environment. The gripper closes the fingers in each candidate posture until the force threshold is reached or the fingers are completely closed. Finally, a shaking action is performed: the gripper first moves up and down along its approach direction, and then rotates around a line parallel to the finger movement joint axis. The success of the grasp is recorded by testing whether the object is still in contact with the two fingers, and the grasping posture set G = {g1…g n}, g=(q,a,θ,d).

[0019] Furthermore, the S2 includes the following specific steps:

[0020] S21: Data preprocessing:

[0021] Perform grasping posture mapping on the dataset generated by S1: the objects in the dataset are all meshes, and the desktop mesh scene point cloud is obtained from a single perspective For each point, mark whether it is a successful pointing point where q j ∈P is the pointing point of the successful grasping posture of the object mesh, r is the maximum radius, P + ={p i |s i =1} represents the set of pointing points,

[0022] S22: Constructing neural network structure:

[0023] A U-shaped neural network is constructed based on the PointNet++ network structure. The input of the network is 20,000 points randomly selected from a single-view scene. The output is the predicted grasping posture of 2048 points, namely g = (q, a, θ, d). The network contains four prediction heads, which are implemented through one-dimensional convolution to predict the output of each point: whether it is a pointing point that has been successfully grasped. Approach vector at point The rotation angle of the grab direction vector and the offset between the hand grabbing origin and the pointing point

[0024] S23: Geometric loss calculation and data training:

[0025] For each point, whether it is a successful grasp point is predicted, using the binary 2 cross entropy loss function, and backpropagating the error l of the k points with the largest error bce,k , select 5 3D points above the gripper To represent the gripper posture, these points are rotated and translated according to the true value and the predicted value to calculate the geometric loss:

[0026]

[0027] Only perform error backpropagation on the grasping poses that are predicted to be successful grasping of the pointing point:

[0028]

[0029] The total error is l = αl bce,k +βl g ;

[0030] The Adam optimizer is used for parameter optimization, and the point cloud coordinates are converted to the camera coordinate system. During the training process, a data augmentation algorithm is added to add Gaussian noise to the point cloud to make the algorithm more robust.

[0031] Furthermore, the S3 includes the following specific steps:

[0032] According to the definition of rotation and translation, the predicted output g = (q, a, θ, d) is transformed into a homogeneous matrix:

[0033] g=(R g ,t g )∈SE(3);R g =[bca]; t g =-da,

[0034] Where c = a × b, R g ,t g are the rotation matrix and translation vector of the gripper respectively, d is the offset distance from the pointing point q to the gripper coordinate origin o, q is a (x, y, z) three-dimensional point in the scene point cloud, vector b is the movement direction of the parallel gripper finger, the direction only needs to be specified manually, because the gripper is essentially a symmetrical structure, the specified direction only needs to conform to the right-hand coordinate system, vector a is the approach direction of the gripper, vector b is represented by the desktop normal vector, when the desktop normal vector n is not parallel to a, n and a can construct a plane, the normal vector of the plane is l, then there is a vector angle θ between b and l, b is expressed as When n is parallel to a, define the angle θ as the angle between the X-axis of the desktop coordinate system and b.

[0035] Furthermore, the vector a is a predicted result, and only a vector product calculation is required. If they are parallel, 1-abs(n·a)<ε, where ε is a small quantity, which is 1e-6.

[0036] The present invention provides a 6-DOF grasping posture generation method for a robot in a cluttered environment, which has the following advantages: the present invention is an end-to-end 6-DOF grasping posture generation method, that is, the input scene point cloud directly outputs a diverse and dense successful grasping posture of each object in the scene. The diversity of postures can ensure that the robot can still meet the grasping requirements when considering obstacle avoidance and kinematic constraints. During specific execution, the grasping posture with the highest probability of the pointing point predicted by the network and for which the robot has a kinematic solution is selected for execution. The present invention does not require a long sampling process and a grasping posture evaluation process, and is more versatile than a grasping posture with 3-4 degrees of freedom. The new grasping posture representation method proposed in the present invention only needs to predict g i =(q i ,a i ,θ i ,d i ) four quantities, and does not require orthogonal constraints between vectors, which is more conducive to the learning of neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 : Schematic diagram of the desktop coordinate system when the desktop normal vector is not parallel to the gripper approach vector;

[0038] Figure 2 : Schematic diagram of the desktop coordinate system when the desktop normal vector is parallel to the gripper approach vector;

[0039] Figure 3 : Schematic diagram of the positions of five geometric points on the gripper;

[0040] Figure 4 : Dataset scene example, schematic diagram of multiple static and stable objects randomly placed on the table. DETAILED DESCRIPTION

[0041] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a 6-DOF grasping posture generation method for a robot in a cluttered environment in conjunction with the accompanying drawings.

[0042] The present invention proposes a new grasping posture representation method: the pointing point grasping posture is represented by g = (q, a, θ, d), and the specific meanings of each symbol are as follows:

[0043] The approach vector of the gripper is used as the Z coordinate system of the gripper. g A successful grasping posture Z can be observed gThe axis always points to the object like a pointer, so the pointing point can be found on the object. Conversely, a grasping posture that cannot see the pointing point is usually not a high-quality grasping posture. Therefore, the grasping postures of successful grasps can be mapped to their respective pointing points q, that is, the grasping posture g = (q, a, θ, d) is expressed as a homogeneous matrix of rotation and translation:

[0044] g=(R g ,t g )∈SE(3);R g =[bca]; t g =-da,

[0045] Where c = a × b, R g ,t g They are the rotation matrix and translation vector of the gripper, d is the offset distance from the point q (q is actually a point in the scene point cloud, which is an x, y, z three-dimensional coordinate point) to the gripper coordinate origin o, vector b is the movement direction of the parallel gripper fingers (the direction only needs to be specified manually, because the gripper is essentially a symmetrical structure, and the specified direction only needs to conform to the right-hand coordinate system in the end), and vector a is the gripper approach direction. For learning algorithms that directly use vector A total of 6 quantities are required for prediction, and orthogonal constraints between a and b need to be added, which is not conducive to learning. The present invention proposes a new representation method for vector b, which is represented by the desktop normal vector. When the desktop normal vector n is not parallel to a (the unit vector a is the result of the prediction, and only the vector product calculation needs to be performed. If they are parallel, 1-abs(n·a)<ε, ε is a small quantity and can generally be 1e-6), n and a can construct a plane, and the normal vector of the plane is l (the normal vector can be easily obtained by fitting the plane algorithm, because the assumption is that the object is on a horizontal table, and the desktop coordinate system can also be pre-specified such as Figure 1 , as shown in 2, the normal vector direction can be arbitrarily specified as long as it finally conforms to the right-hand coordinate system direction), then there is a vector angle θ between b and l, and b can be expressed as When n is parallel to a, define the angle θ as the angle between the X-axis of the desktop coordinate system and b.

[0046] A method for generating a 6-DOF grasping posture of a robot in a cluttered environment according to the present invention comprises the following steps:

[0047] S1: training dataset generation;

[0048] Since it is difficult to obtain the grasping posture of large-scale real objects, the present invention uses a simulation environment to sample the grasping posture of synthetic objects. Figure 4As shown in the figure, the simulation scene contains a table with 1 to 12 objects randomly scattered on the table, which is statically stable under the action of gravity. Using antipodal sampling, any point on the object mesh surface is sampled as the first contact point of the parallel gripper, and the normal vector at this point and the point can be used to obtain a cone. A line is sampled within the cone, and the intersection of this line and the object mesh is the second contact point. Then, uniform rotation sampling is performed based on the center point between the two contact points and the straight line, and the postures that collide with the gripper and the postures that do not intersect with the space between the gripper fingers are removed. The above steps are repeated until enough grasping posture candidates are obtained. The obtained candidate postures are evaluated in the simulation environment, and the gripper closes the fingers in each candidate posture until the force threshold is reached or the fingers are completely closed. Finally, a shaking action is performed: the gripper first moves up and down along its approach direction, and then rotates around a line parallel to the finger movement joint axis. The success of the grasp is recorded by testing whether the object is still in contact with the two fingers, and the grasping posture set G = {g1…g n}, g=(q,a,θ,d).

[0049] S2: neural network training;

[0050] S21: Data preprocessing:

[0051] Perform grasping posture mapping on the dataset generated above: the objects in the dataset are all meshes, and the desktop mesh scene point cloud is obtained from a single perspective For each point, mark whether it is a successful pointing point where q j ∈P is the pointing point of the successful grasping posture of the object mesh, r is the maximum radius, P + ={p i |s i =1} represents the set of pointing points,

[0052] S22: Constructing neural network structure:

[0053] The present invention is a deep learning method that requires a neural network to process point cloud data. The PointNet++ network structure is one of the most outstanding network structures for extracting point cloud features at this stage. Therefore, the present invention constructs a U-shaped neural network based on the PointNet++ network structure. The input of the network is 20,000 points randomly selected from a single-view scene, and the output is the predicted grasping posture of 2048 points, namely g = (q, a, θ, d). The network contains four prediction heads, which are implemented through one-dimensional convolution to predict the output of each point: whether it is a pointing point that has been successfully grasped Approach vector at point The rotation angle of the grab direction vector and the offset between the hand grabbing origin and the pointing point

[0054] S23: Geometric loss calculation and data training:

[0055] For each point, whether it is a successful grasp point is predicted, using the binary 2 cross entropy loss function, and backpropagating the error l of the k points with the largest error bce,k .like Figure 3 As shown, select 5 3D points above the gripper To represent the gripper posture, these points are rotated and translated according to the true value and the predicted value to calculate the geometric loss:

[0056]

[0057] Only perform error backpropagation on the grasping poses that are predicted to be successful grasping of the pointing point:

[0058]

[0059] The total error is l = αl bce,k +βl g .

[0060] Specifically, the present invention uses the Adam optimizer for parameter optimization, the point cloud coordinates are converted to the camera coordinate system, and a data enhancement algorithm is added during the training process to add Gaussian noise to the point cloud to make the algorithm more robust, so that it will have better performance when migrating to the point cloud data scene of the real sensor.

[0061] S3: The prediction result is converted into a homogeneous matrix of the grasping posture;

[0062] According to the definition of rotation and translation, the predicted output g = (q, a, θ, d) is transformed into a homogeneous matrix:

[0063] g=(R g ,t g )∈SE(3);R g =[bca]; t g =-da.

[0064] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A method for generating 6-DOF grasping postures of a robot in a cluttered environment, characterized in that: The steps include: S1: training dataset generation; Grasping gestures of synthetic objects are sampled using a simulation environment. The simulation scene consists of a table with 1 to 12 objects randomly scattered on the table, which is statically stable under the action of gravity. Using antipodal sampling, an arbitrary point on the surface of the object mesh is sampled as the first contact point of the parallel gripper. The normal vectors at this point and the point can obtain a cone. A straight line is sampled inside the cone, and the intersection of this line and the object mesh is the second contact point. Then, uniform rotation sampling is performed based on the center point between the two contact points and the straight line. The postures that collide with the gripper and the postures that have no intersection with the space between the gripper fingers are removed. The above steps are repeated until enough grasping posture candidates are obtained. The obtained candidate postures are evaluated in a simulation environment. The gripper closes the fingers in each candidate posture until the force threshold is reached or the fingers are completely closed. Finally, a shaking action is performed: the gripper first moves up and down along its approach direction, and then rotates around a straight line parallel to the finger movement joint axis. The success of the grasp is recorded by testing whether the object is still in contact with the two fingers, and the grasping posture set G = {g1…g n }, g = (q, a, θ, d); S2: neural network training; S21: Data preprocessing: Perform grasping posture mapping on the dataset generated by S1: the objects in the dataset are all meshes, and the desktop mesh scene point cloud is obtained from a single perspective For each point, mark whether it is a successful grasping point where q j ∈P is the pointing point of the successful grasping posture of the object mesh, r is the maximum radius, P + ={p i |s i =1} represents the set of pointing points, S22: Constructing neural network structure: A U-shaped neural network is constructed based on the PointNet++ network structure. The input of the network is 20,000 points randomly selected from a single-view scene. The output is the predicted grasping posture of 2048 points, namely g = (q, a, θ, d). The network contains four prediction heads, which are implemented through one-dimensional convolution to predict the output of each point: whether it is a pointing point that has been successfully grasped. Approach vector at point The rotation angle of the grab direction vector and the offset between the hand grabbing origin and the pointing point S23: Geometric loss calculation and data training: For each point, whether it is a successful grasp point is predicted, using the binary cross entropy loss function, and backpropagating the error l of the k points with the largest error bce,k , select 5 3D points above the gripper To represent the gripper posture, these points are rotated and translated according to the true value and the predicted value to calculate the geometric loss: Only perform error backpropagation on the grasping poses that are predicted to be successful grasping of the pointing point: The total error is l = αl bce,k +βl g ; The Adam optimizer is used for parameter optimization. The point cloud coordinates are converted to the camera coordinate system. During the training process, a data augmentation algorithm is added to the point cloud to add Gaussian noise to make the algorithm more robust. S3: The prediction result is converted into a homogeneous matrix of the grasping posture; According to the definition of rotation and translation, the predicted output g = (q, a, θ, d) is transformed into a homogeneous matrix: g=(R g ,t g )∈SE(3);R g =[b c a];t g =-da, Where c = a × b, R g ,t g are the rotation matrix and translation vector of the gripper respectively, d is the offset distance from the pointing point q to the gripper coordinate origin o, q is a (x, y, z) three-dimensional point in the scene point cloud, vector b is the movement direction of the parallel gripper finger, the direction only needs to be specified manually, because the gripper is essentially a symmetrical structure, the specified direction only needs to conform to the right-hand coordinate system, vector a is the approach direction of the gripper, vector b is represented by the desktop normal vector, when the desktop normal vector n is not parallel to a, n and a can construct a plane, the normal vector of the plane is l, then there is a vector angle θ between b and l, b is expressed as When n is parallel to a, define the angle θ as the angle between the X-axis of the desktop coordinate system and b.

2. The method for generating 6-DOF grasping postures of a robot in a cluttered environment according to claim 1, characterized in that: The vector a is the predicted result, and only the vector product calculation needs to be performed. If they are parallel, 1-abs(n·a)<ε, where ε is a small quantity, which is 1e-6.

Images