Device and method for establishing robot grasping data set based on teaching and deep learning

Abstract

The present invention discloses a device and method for establishing a robot grasping data set based on teaching and deep learning, and relates to the field of robots. The method comprises: step 1, using a teaching pendant to teach grasping postures on a target object, extracting grasping part point clouds from a scene according to the grasping space, and manually scoring and labeling data while collecting data; step 2, using the collected label data to quickly train a six-degree-of-freedom grasping posture evaluation network; step 3, using continuous frame shooting to quickly collect point cloud data of an actual grasping scene; step 4, sampling six-degree-of-freedom grasping postures on the actual grasping scene point cloud, and using the trained six-degree-of-freedom grasping posture evaluation network to evaluate and score each sampled posture, thereby completing the automatic generation of labels for the scene point cloud. The present invention solves the technical difficulties of high hardware cost and time cost in the previous generation and establishment of six-degree-of-freedom grasping data sets.

Description

Technical Field

[0001] The present invention relates to the field of robots, and in particular to a device and method for establishing a robot grasping data set based on teaching and deep learning. Background Art

[0002] The robot's disordered grasping of arbitrary objects is a classic research problem in the field of robotics. The robot predicts the six-degree-of-freedom grasping posture in the action space based on the visual sensor input. The robot generally uses a two-finger gripper to perform the grasping action on the object. Most of the prediction methods for the six-degree-of-freedom grasping posture are data-driven. Through an end-to-end neural network model, the sensor information in the form of point clouds is input and the prediction of the grasping posture is output.

[0003] Arbitrary object grasping pose estimation algorithms based on machine learning are often data-driven and require a large amount of labeled data to train network models. Establishing a dataset that can be used to train the network is often a very labor-intensive task, and real data is noisy. In most cases, manual intervention is required for data labeling, and there is a lot of repetitive work in the process of dataset generation.

[0004] Grasping datasets can be divided into three categories according to the generation and annotation methods: manually annotated data, data collected by grasping experiments, and simulation data. For manually annotated grasping datasets, such as the Cornell and VMRD datasets, they are all based on RGB images. Each image has a manually annotated rectangular box to represent the grasping configuration. Since it is difficult to manually annotate a large number of 6-DoF grasping poses on the scene point cloud, the manually annotated datasets are all based on RGB images and are only suitable for 4-DoF grasping pose detection algorithm training.

[0005] Datasets based on robot grasping experiments often require a large number of real-machine experiments, which is not only time-consuming but also labor-intensive. Levine et al. used 14 robots to conduct grasping experiments simultaneously. The entire data set collection process conducted more than 800,000 grasping attempts. Multiple robots grasped synchronously, and then automatically labeled the grasped data based on the feedback results. Tsinghua University has also established a data set based on real-machine experiments. The data is collected by a robot through repeated grasping. The collected data includes multiple experimental data such as vision and touch.

[0006] Since data collection in real experiments is labor-intensive and the data annotation process is slow, most studies use grasping mechanics analysis on the 3D model of the object in a simulation environment to generate grasping data. Nvidia's research team used Graspit! grasping simulation engine to perform millions of grasping simulations on the 3D models of multiple objects. Since simulation can adjust physical parameters such as friction coefficient, these parameters can be adjusted to test the stability of grasping. Therefore, a large amount of grasping data with quantitative grasping quality indicators can be generated based on the grasping results.

[0007] In addition to using simulation physics engines, more researchers directly perform 6-DoF grasping posture sampling on the 3D model of the object, and then determine the grasping quality by analyzing the grasping force closure, so as to score and annotate the sampled grasping posture. Ten, Fang, LH, Ni and others have established relevant physical calculation models, established their own simulation data sets through analysis, and proposed their own grasping quality quantitative indicators. Through model-based mechanical analysis, the candidate grasping postures can be evaluated, and feasible grasping postures can be selected from them, so as to annotate the grasping labels. The object models used in these studies are obtained from scans of known objects in life, and most of them are from general object model data sets, such as the YCB data set and the BigBIRD data set. Therefore, due to the limited number of available models, the types of objects contained in the data sets established by these studies are also limited, and there are differences between the simulation data and the information obtained by the actual sensor, which leads to the decline in the performance of the network trained based on the simulation data set in real experiments.

[0008] In summary, there are two main problems with previous robot grasping datasets: first, it is difficult to obtain a large number of data samples based on real data, and the data collection cost and time overhead are very alarming; second, the grasping dataset established based on the simulation environment needs to generate data individuals through simulated grasping, and the model trained based on simulation data is difficult to migrate to the real environment, resulting in a decrease in network performance. In addition, the process of generating a dataset based on a simulation environment is relatively slow, requiring thousands of sets of simulated grasping to collect data, and a complete three-dimensional model of the object is required. However, in many application scenarios, the three-dimensional model of the object is difficult to obtain quickly and completely. Therefore, it is difficult to quickly collect and establish a six-degree-of-freedom grasping dataset based on a real environment.

[0009] In summary, technicians in this field are committed to developing a device and method for establishing a robot grasping data set based on teaching and deep learning. Summary of the invention

[0010] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the time cost and time cost of real data collection are high, the real environment migration of simulation data is poor, and the network performance is reduced.

[0011] To achieve the above object, the present invention provides a method for establishing a robot grasping data set based on teaching and deep learning, the method comprising:

[0012] Step 1: Use the teaching pendant to teach the grasping posture on the target object, extract the grasped part point cloud from the scene according to the grasping space, collect data and perform manual scoring while marking the label data;

[0013] Step 2: Use the collected label data to quickly train the six-degree-of-freedom grasping posture evaluation network;

[0014] Step 3: Use continuous frame shooting to quickly collect point cloud data of the actual captured scene;

[0015] Step 4: Sample the six-degree-of-freedom grasping posture on the actual grasping scene point cloud, and use the trained six-degree-of-freedom grasping posture evaluation network to evaluate and score each sampled posture, thereby completing the automatic generation of scene point cloud labels.

[0016] Furthermore, in step 1, data expansion is performed on the collected teaching posture, and the expansion method is: after teaching a grasping posture, an appropriate posture transformation is performed on the posture, including rotation around the grasping axis and displacement in a direction perpendicular to the grasping axis. After a small amount of posture transformation, the transformed posture has the same grasping score as the taught posture, and when excessive posture transformation is performed, the grasping score of the transformed posture should be reduced.

[0017] Furthermore, the six-degree-of-freedom grasping posture evaluation network is a point cloud-based network model, and the internal network modules include PointNet and VoxNet.

[0018] Furthermore, the grasped part point cloud needs to be processed before being input into the six-degree-of-freedom grasping posture evaluation network training, including:

[0019] Step 1.1: Grab the point cloud of the part and perform voxel filtering, and filter out the middle part;

[0020] Step 1.2: Perform manual feature selection, including calculating point normal vector, point curvature, and spatial distribution parameters. The spatial distribution parameters refer to the ratio of the distance of a single point to the nearest fingertip to the width of the object. The point cloud finally input to the network contains 8-dimensional features.

[0021] Step 1.3: After feature extraction, the point cloud is input into PointNet to extract 128-dimensional geometric features. The point cloud of the grasped part will also be voxelized. The grasping space will be voxelized into a 16×16×40 grid, and the size of the value represented by the voxel block will be determined according to the occupancy of the point. The four-dimensional variable obtained by voxelization is input into VoxNet; the geometric features and distribution features will be input into the fully connected perceptron to obtain 128-dimensional mixed features, and finally through another fully connected perceptron, the quality score of the corresponding six-degree-of-freedom grasping posture is obtained according to the extracted geometric features, distribution features, and mixed features.

[0022] Furthermore, the tag generation method in step 4 includes:

[0023] Step 4.1: Obtain a multi-view scene and collect the point cloud of the scene using continuous frames;

[0024] Step 4.2: For each collected scene point cloud individual, perform six-degree-of-freedom grasping pose sampling based on the point cloud normal vector;

[0025] Step 4.3: For each grasping posture with a grasping part and no collision, it is scored through the "six-degree-of-freedom grasping posture evaluation network" to achieve automatic generation of six-degree-of-freedom grasping posture labels.

[0026] Furthermore, the sampling of the point cloud normal vector determines the direction of the grasping coordinate axis of the grasping posture in the opposite direction of the point normal vector. The grasping posture center starts from the origin position and gradually moves backward along the main axis direction until there is no contact with the point cloud and the grasped part has a point cloud. Each grasping posture obtained by sampling is rotated along the main axis direction to expand the sampling data.

[0027] Furthermore, the grasping teaching pendant is a robot two-finger claw model with an arucomarker, and the grasping part point cloud is collected by a camera sensor. During the teaching process, the two fingertips of the teaching pendant are respectively located at the two ends of the object. There are at least two camera sensors, and the cameras are arranged on both sides of the object. The arrangement of the cameras should ensure that the point cloud information of the two opposite surfaces of the object can be obtained.

[0028] On the other hand, the present invention also discloses a robot grasping data set establishment device based on manual teaching and deep learning, comprising:

[0029] A grasping teaching pendant is used to teach grasping postures on target objects;

[0030] Multiple camera sensors are used to collect point cloud information of the scene. The cameras are arranged in a way that ensures that point cloud information of two sides of the target object is obtained;

[0031] A six-degree-of-freedom grasping posture evaluation network is used to score and evaluate the specified six-degree-of-freedom grasping position, the graspable part point cloud of the grasping posture is input into the six-degree-of-freedom grasping posture evaluation network, the grasping posture score is output, and the six-degree-of-freedom grasping posture evaluation network is trained;

[0032] The six-degree-of-freedom grasping data label generation module samples the six-degree-of-freedom grasping postures on the grasped scene point cloud, and uses the trained six-degree-of-freedom grasping posture evaluation network to evaluate and score each sampled posture, thereby completing the automatic generation of labels for the scene point cloud.

[0033] In another embodiment, the six-degree-of-freedom grasping posture evaluation network construction method is:

[0034] Step 1.1: Filter the point cloud voxels of the captured part and filter out the middle part;

[0035] Step 1.2: Perform manual feature selection, including calculating point normal vector, point curvature, and spatial distribution parameters. The spatial distribution parameters refer to the ratio of the distance of a single point to the nearest fingertip to the width of the object. The point cloud finally input to the network contains 8-dimensional features.

[0036] Step 1.3: After feature extraction, the point cloud will be input into PointNet to extract 128-dimensional geometric features. The point cloud of the grasped part will also be voxelized. The grasping space will be voxelized into a 16×16×40 grid, and the size of the value represented by the voxel block will be determined according to the occupancy of the point. The four-dimensional variable obtained by voxelization is input into VoxNet; the geometric features and distribution features will be input into the fully connected perceptron to obtain 128-dimensional mixed features, and finally through another fully connected perceptron, the quality score of the corresponding six-degree-of-freedom grasping posture is obtained according to the extracted geometric features, distribution features, and mixed features.

[0037] In another embodiment, the six-degree-of-freedom grasping data label generation module samples the six-degree-of-freedom grasping posture on the grasping scene point cloud. The sampling criterion is to determine the grasping coordinate axis direction of the grasping posture in the opposite direction of the point normal vector. The grasping posture center starts from the origin position and gradually moves backward along the main axis direction until there is no contact with the point cloud and the grasped part has a point cloud. Each sampled grasping posture is rotated along the main axis direction to expand the sampling data.

[0038] The technical effects of the present invention are:

[0039] Without the need for a simulation environment or any three-dimensional model of an object, the grasping data can be quickly collected on any object based on manual teaching, and the six-degree-of-freedom grasping data with scores can be annotated on any scene point cloud through the grasping data label generation method specified in this solution (invention). The technical difficulties of high hardware cost and time cost in the previous generation of six-degree-of-freedom grasping data sets have been solved. In addition, the hardware equipment required by the present invention is relatively low in cost, and the method can be applied to various laboratory and factory application environments.

[0040] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a preferred embodiment of the method for establishing the present invention;

[0042] Figure 2 It is a robot two-finger claw model;

[0043] Figure 3 It is a schematic diagram of the arrangement of camera sensors;

[0044] Figure 4 This is a schematic diagram of the teaching pendant grabbing an object;

[0045] Figure 5 It is a six-degree-of-freedom grasping posture evaluation network model. DETAILED DESCRIPTION

[0046] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0047] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0048] The present invention provides a complete technical solution for quickly establishing a grasping data set based on manual teaching. The solution includes equipment: a grasping teaching device and a camera sensor, a grasping labeling method based on manual teaching, an algorithm model is a six-degree-of-freedom grasping posture evaluation network, and a method: a six-degree-of-freedom grasping data label generation method.

[0049] The grasping teaching pendant is a robot two-finger claw model with an aruco marker, which is mainly assembled from a glass fiber disk and a copper column. The teaching pendant is used for people to hold and teach the corresponding grasping posture on the target object.

[0050] A grasping and labeling method based on manual teaching is proposed. This method specifies the way of grasping by the teaching pendant, the expansion method of grasping and labeling data, and the corresponding arrangement of hardware sensors. This method collects data based on the grasping teaching pendant and trains a six-degree-of-freedom grasping posture evaluation network.

[0051] The six-degree-of-freedom grasping posture evaluation network inputs the graspable part point cloud corresponding to the grasping posture, outputs the grasping posture score, and is used to score and evaluate the specified six-degree-of-freedom grasping posture. The algorithm model is a coupling of the classic network model. The algorithm process includes manual feature selection and extraction of the grasping part point cloud, multi-network feature extraction and fusion. The algorithm model finally scores whether the object part of the gripping part of the gripper is suitable for stable grasping based on the geometric features of the graspable part point cloud and the distribution features between the gripper fingertips, thereby evaluating the feasibility score of the six-degree-of-freedom grasping posture.

[0052] A method for generating six-degree-of-freedom grasping data labels. This method describes how to label grasping pose labels on point clouds, and scores the labeled labels based on a six-degree-of-freedom grasping pose evaluation network. This method is used to establish a data set, and the method outputs scene point cloud data with grasping labels, which can be used to train related grasping pose prediction networks.

[0053] The technical contents of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, the method for establishing a robot crawling data set specifically includes:

[0055] Step 1: Use the teaching pendant to teach the grasping posture on the target object, collect data, and manually score and annotate the data;

[0056] Step 2: Use the collected labeled data to quickly train the six-DOF grasping posture evaluation network;

[0057] Step 3: Use continuous frame shooting to quickly collect point cloud data of the actual capture scene;

[0058] Step 4: Sample the six-degree-of-freedom grasping postures on the grasped scene point cloud, and use the trained six-degree-of-freedom grasping posture evaluation network to evaluate and score each sampled posture, thereby completing the automatic generation of labels for the scene point cloud.

[0059] like Figure 2As shown, in one embodiment, the grasping teaching pendant is a robot two-finger claw model with an aruco marker, which is mainly assembled from a glass fiber disk and a copper column. The teaching pendant is used for people to hold and teach the corresponding grasping posture on the target object.

[0060] like Figure 3 As shown, the continuous frame shooting adopts an RGBD camera sensor, and other sensors that can be used for taking pictures can also be used. The target object is placed flat on a platform such as a desktop. No less than two RGBD camera sensors should be arranged next to the platform to collect point cloud information of the scene. The arrangement of the cameras should ensure that the point cloud information of two opposite surfaces (two sides or front and back) of the object can be obtained, so that when the partial point cloud can be captured for feature extraction later, more complete geometric features can be obtained, thereby improving the accuracy of the method.

[0061] Specifically, the operator holds the grasping teaching pendant and teaches the feasible grasping posture and the infeasible grasping posture on the target object. The camera sensor obtains the grasping six-degree-of-freedom posture in real time through the Aruco Marker on the teaching pendant. During a single teaching process, the operator should record the point cloud information of the current scene before teaching to ensure that the teaching process does not cause occlusion of the field of view of key information.

[0062] As shown in the figure, during the teaching process, the operator should place the two fingertips of the teaching pendant at the two ends of the object, so as to ensure that the fingertips of the robot grasp the object in the camera's field of view when grasping the two-finger grasping model, and there will be no loss of visual information. When the operator teaches the target grasping posture of the teaching pendant, the posture of the teaching pendant is recorded. Each teaching action can record the posture of the teaching pendant, and extract the point cloud of the grasped part from the scene according to the grasping space. The operator should score the grasping posture while recording the posture, marking 2 points for feasible postures and 0 points for infeasible grasping postures.

[0063] In another embodiment, step 1 includes: after teaching a grasping posture, performing an appropriate posture transformation on the posture, including rotating around the grasping axis and displacing in a direction perpendicular to the grasping axis. After a small amount of posture transformation, the transformed posture has the same grasping score as the taught posture, and after excessive posture transformation, the score of the transformed posture is less than the taught posture score.

[0064] Specifically, each teaching action should be expanded to ensure an efficient data collection process. The expansion basis is that after teaching a grasping posture, the posture is transformed appropriately, including rotation around the grasping axis and displacement perpendicular to the grasping axis. After a small amount of posture transformation, this method believes that the transformed posture has the same grasping score as the taught posture. However, when excessive posture transformation is performed, the execution of the current grasping posture of the task of this method will have a greater impact on the original posture of the object, and the grasping may fail. The grasping score corresponding to the grasping posture is reduced by 1. For example, after teaching a grasping with a score of 2, if the grasping posture taught in the data expansion stage has undergone excessive posture transformation, the score is reduced by 1, and the grasping posture obtained by transformation is 1, which represents a satisfactory grasping, that is, the grasping may be successful, but the grasping may be unstable or cause excessive transformation of the original posture of the object. Using this method for data collection, more than 20K labeled data individuals can be collected within 2 hours.

[0065] A grasping and labeling method based on manual teaching is proposed. This method specifies the way of grasping by the teaching pendant, the expansion method of grasping and labeling data, and the corresponding arrangement of hardware sensors. This method collects data based on the grasping teaching pendant and trains a six-degree-of-freedom grasping posture evaluation network.

[0066] like Figure 5 As shown, in another embodiment, the grasped part point cloud needs to be processed before inputting into the six-degree-of-freedom grasping posture evaluation network training, including:

[0067] Step 1.1: Grab the point cloud of the part and perform voxel filtering, and filter out the middle part;

[0068] Step 1.2: Perform manual feature selection, including calculating point normal vector, point curvature, and spatial distribution parameters. The spatial distribution parameters refer to the ratio of the distance of a single point to the nearest fingertip to the width of the object. The point cloud finally input to the network contains 8-dimensional features.

[0069] Step 1.3: After feature extraction, the point cloud is input into PointNet to extract 128-dimensional geometric features. The point cloud of the grasped part will also be voxelized. The grasping space will be voxelized into a 16×16×40 grid, and the size of the value represented by the voxel block will be determined according to the occupancy of the point. The four-dimensional variable obtained by voxelization is input into VoxNet; the geometric features and distribution features will be input into the fully connected perceptron to obtain 128-dimensional mixed features, and finally through another fully connected perceptron, the quality score of the corresponding six-degree-of-freedom grasping posture is obtained according to the extracted geometric features, distribution features, and mixed features.

[0070] Specifically, Figure 5As shown, the "six-degree-of-freedom grasping posture evaluation network" is a point cloud-based network model, and the internal network modules include PointNet and VoxNet. The grasped part of the point cloud needs to be processed before being input into the network, including voxel filtering and filtering out the middle part, because the present invention believes that the quality of grasping is largely only related to the geometric features and distribution of the part of the object in contact with the fingertips. Filtering out the invalid parts can greatly improve the speed of network training and increase the evaluation accuracy. Next, the algorithm will perform manual feature selection, including calculating the point normal vector (3D variable), point curvature (1D variable), and spatial distribution parameters (1D variable), where the spatial distribution parameters refer to the ratio of the distance of a single point to the nearest fingertip to the width of the object. The point cloud finally input into the network contains 8-dimensional features.

[0071] After feature extraction, the point cloud will be input into PointNet to extract 128-dimensional geometric features. The point cloud of the grasping part will also be voxelized. The grasping space will be voxelized into a 16×16×40 grid, and the size of the value represented by the voxel block will be determined according to the point occupancy. The four-dimensional variable obtained by voxelization is input into VoxNet to extract the 128-dimensional distribution features of the grasping part between the fingertips. The geometric features and distribution features will be input into the fully connected perceptron to obtain 128-dimensional mixed features, and finally through another fully connected perceptron, the quality score of the corresponding six-degree-of-freedom grasping posture is obtained based on the extracted geometric features, distribution features, and mixed features.

[0072] In this example, the "six-degree-of-freedom grasping posture evaluation network" will be trained based on the labeled grasping part point cloud data taken by the "grasping and labeling method based on manual teaching". Since the present invention limits the receptive field of the network and performs sufficient manual feature selection and extraction, the individual data is small, so the network model training is very fast and the network training can be completed within 2 hours.

[0073] In another embodiment, the step 3 performs six-degree-of-freedom grasping posture sampling based on the point cloud normal vector. The sampling criterion is to determine the grasping coordinate axis direction of the grasping posture in the opposite direction of the point normal vector. The grasping posture center starts from the origin position and gradually moves backward along the main axis direction until there is no contact with the point cloud and the grasping part has a point cloud. Each grasping posture obtained by sampling is rotated along the main axis direction to expand the sampling data. For each grasping posture with a grasping part and no collision, any number of randomly placed objects are arranged on a rotating table.

[0074] Specifically, any number of randomly placed objects are arranged on a rotating desktop. The camera sensors are arranged in the form of a graph to obtain scenes from multiple perspectives. The desktop is rotated, and the camera uses continuous frames to collect point clouds of the scene, thereby obtaining multi-angle scene data and improving data collection efficiency.

[0075] For each collected scene point cloud individual, six-degree-of-freedom grasping posture sampling is performed based on the point cloud normal vector. The sampling criterion is to determine the grasping coordinate axis direction of the grasping posture in the opposite direction of the point normal vector. The grasping posture center starts from the origin position and gradually moves backward along the main axis direction until there is no contact with the point cloud and the grasped part has a point cloud. Each sampled grasping posture is rotated along the main axis direction to expand the sampling data. For each grasping posture with a grasping part and no collision, it is scored through the "six-degree-of-freedom grasping posture evaluation network" to achieve the automatic generation of six-degree-of-freedom grasping posture labels.

[0076] In order to solve the problem that the six-degree-of-freedom robot grasping data set is difficult to quickly generate based on real data, the method of manual teaching is used to quickly collect data and label the six-degree-of-freedom grasping posture for the target object, and the invention introduces a six-degree-of-freedom grasping posture evaluation neural network to directly evaluate and score the six-degree-of-freedom grasping posture based on real data. The present invention proposes a method for automatically generating six-degree-of-freedom grasping posture labels based on real data, which can directly label grasping posture labels on a large number of scene point clouds actually collected. Compared with the previous grasping data set generation method, the efficiency of data set generation is significantly improved, the cost and time of data collection are reduced, and the data set does not require any simulated virtual data. The network trained based on the data set established by the invention can be well migrated to actual application scenarios.

[0077] In this example, the grasping part point cloud with manual annotation scores is collected according to the "grasping annotation method based on manual teaching", which can be used to train the network algorithm model "six-degree-of-freedom grasping posture evaluation network" proposed in this scheme. The model inputs the point cloud of the grasping part, extracts the morphological features of the grasped part of the object, and determines the feasibility of grasping according to the relative posture of the object and the gripper, and finally outputs the grasping score.

[0078] In another embodiment, the present invention provides a complete system for quickly establishing a grasping data set based on manual teaching. The system includes a grasping teaching device and an image collector. The teaching device is used to teach the grasping posture on the target object during training, and perform manual scoring and annotating data at the same time; the image collector is used to collect scene point cloud information and quickly collect point cloud data of the actual grasping scene by continuous frame shooting. The system trains a six-degree-of-freedom grasping posture evaluation network through annotated data and scene point cloud information, and inputs the point cloud information of the actual grasping scene into the six-degree-of-freedom grasping posture evaluation network, evaluates and scores each sampled posture, thereby completing the automatic generation of scene point cloud labels.

[0079] In one embodiment, the grasping teaching device is a robot two-finger claw model with an aruco marker, which is mainly composed of a glass fiber disk and a copper column. The teaching device is used for people to hold and teach the corresponding grasping posture on the target object. The image acquisition device uses an RGBD camera sensor, and other sensors that can be used for taking pictures can also be used. The target object is placed flat on a platform such as a desktop. There should be no less than 2 RGBD camera sensors next to the platform to collect point cloud information of the scene, such as Figure 3 As shown in the reference example, the cameras should be arranged in a way that ensures that the point cloud information of two opposite surfaces (two sides or the front and back) of the object can be obtained, so that when the partial point cloud can be captured for feature extraction later, more complete geometric features can be obtained, thereby improving the accuracy of the method.

[0080] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for establishing a robot grasping data set based on teaching and deep learning, characterized in that: The method comprises: Step 1: Use the teaching pendant to teach the grasping posture on the target object, extract the grasped part point cloud from the scene according to the grasping space, collect data and perform manual scoring while marking the label data; Step 2: Use the collected label data to quickly train the six-degree-of-freedom grasping posture evaluation network; Step 3: Use continuous frame shooting to quickly collect point cloud data of the actual captured scene; Step 4: Sampling six-degree-of-freedom grasping postures on the actual grasping scene point cloud, and using the trained six-degree-of-freedom grasping posture evaluation network to evaluate and score each sampled posture, thereby completing the automatic generation of scene point cloud labels; The six-degree-of-freedom grasping posture evaluation network is a point cloud-based network model, and its internal network modules include PointNet and VoxNet; The grasped part point cloud needs to be processed before inputting into the six-degree-of-freedom grasping posture evaluation network training, including: Step 1.1: Grab the point cloud of the part and perform voxel filtering, and filter out the middle part; Step 1.2: Perform manual feature selection, including calculating point normal vector, point curvature, and spatial distribution parameters. The spatial distribution parameters refer to the ratio of the distance of a single point to the nearest fingertip to the width of the object. The point cloud finally input to the network contains 8-dimensional features. Step 1.3: After feature extraction, the point cloud is input into PointNet to extract 128-dimensional geometric features. The point cloud of the grasped part will also be voxelized. The grasping space will be voxelized into a 16×16×40 grid, and the size of the value represented by the voxel block will be determined according to the occupancy of the point. The four-dimensional variable obtained by voxelization is input into VoxNet; the geometric features and distribution features will be input into the fully connected perceptron to obtain 128-dimensional mixed features, and finally through another fully connected perceptron, the quality score of the corresponding six-degree-of-freedom grasping posture is obtained according to the extracted geometric features, distribution features, and mixed features.

2. The method for establishing a robot grasping data set based on teaching and deep learning as claimed in claim 1, characterized in that: In the step 1, data expansion is performed on the collected teaching posture, and the expansion method is: after teaching a grasping posture, an appropriate posture transformation is performed on the posture, including rotation around the grasping axis and displacement in a direction perpendicular to the grasping axis. After a small amount of posture transformation, the transformed posture has the same grasping score as the taught posture, and when excessive posture transformation is performed, the grasping score of the transformed posture should be reduced.

3. The method for establishing a robot grasping data set based on teaching and deep learning as claimed in claim 1, characterized in that The label generation method in step 4 includes: Step 4.1: Obtain a multi-view scene and collect the point cloud of the scene using continuous frames; Step 4.2: For each collected scene point cloud individual, perform six-degree-of-freedom grasping pose sampling based on the point cloud normal vector; Step 4.3: For each grasping posture with grasping parts and no collision, score it through the "six-degree-of-freedom grasping posture evaluation network" to achieve automatic generation of six-degree-of-freedom grasping posture labels.

4. The method for establishing a robot grasping data set based on teaching and deep learning as described in claim 3 is characterized in that The sampling of the point cloud normal vector determines the direction of the grasping coordinate axis of the grasping posture in the opposite direction of the point normal vector. The grasping posture center starts from the origin position and gradually moves backward along the main axis direction until there is no contact with the point cloud and the grasped part has a point cloud. Each grasping posture obtained by sampling is rotated along the main axis direction to expand the sampling data.

5. The method for establishing a robot grasping data set based on teaching and deep learning as claimed in claim 1, characterized in that The grasping teaching pendant is a robot two-finger claw model with an arucomarker. The grasping part point cloud is collected by a camera sensor. During the teaching process, the two fingertips of the teaching pendant are respectively located at the two ends of the object. There are at least two camera sensors, and the cameras are arranged on both sides of the object. The arrangement of the cameras should ensure that the point cloud information of the two opposite sides of the object can be obtained.

6. A robot grasping data set establishment device based on teaching and deep learning, characterized in that include: A grasping teaching pendant is used to teach grasping postures on target objects; Multiple camera sensors are used to collect point cloud information of the scene. The cameras are arranged in a way that ensures that point cloud information of two sides of the target object is obtained; A six-degree-of-freedom grasping posture evaluation network is used to score and evaluate the specified six-degree-of-freedom grasping position, the graspable part point cloud of the grasping posture is input into the six-degree-of-freedom grasping posture evaluation network, the grasping posture score is output, and the six-degree-of-freedom grasping posture evaluation network is trained; The six-degree-of-freedom grasping data label generation module samples the six-degree-of-freedom grasping postures on the grasped scene point cloud, and uses the trained six-degree-of-freedom grasping posture evaluation network to evaluate and score each sampled posture, thereby completing the automatic generation of labels for the scene point cloud.

7. The device for establishing a robot grasping data set based on teaching and deep learning as claimed in claim 6, characterized in that The six-degree-of-freedom grasping posture evaluation network construction method is: Step 1.1: Filter the point cloud voxels of the captured part and filter out the middle part; Step 1.2: Perform manual feature selection, including calculating point normal vector, point curvature, and spatial distribution parameters. The spatial distribution parameters refer to the ratio of the distance of a single point to the nearest fingertip to the width of the object. The point cloud finally input to the network contains 8-dimensional features. Step 1.3: After feature extraction, the point cloud will be input into PointNet to extract 128-dimensional geometric features. The point cloud of the grasped part will also be voxelized. The grasping space will be voxelized into a 16×16×40 grid, and the size of the value represented by the voxel block will be determined according to the occupancy of the point. The four-dimensional variable obtained by voxelization is input into VoxNet; the geometric features and distribution features will be input into the fully connected perceptron to obtain 128-dimensional mixed features, and finally through another fully connected perceptron, the quality score of the corresponding six-degree-of-freedom grasping posture is obtained according to the extracted geometric features, distribution features, and mixed features.

8. The device for establishing a robot grasping data set based on teaching and deep learning as claimed in claim 6, characterized in that The six-degree-of-freedom grasping data label generation module samples the six-degree-of-freedom grasping posture on the grasping scene point cloud. The sampling criterion is to determine the grasping coordinate axis direction of the grasping posture in the opposite direction of the point normal vector. The grasping posture center starts from the origin position and gradually moves backward along the main axis direction until there is no contact with the point cloud and the grasped part has a point cloud. Each sampled grasping posture is rotated along the main axis direction to expand the sampling data.

Images