Grabbing method based on six-degree-of-freedom GraspNet pose generation
By building an RGB-D visual grasping system and Mask R-CNN instance segmentation, combined with six-degree-of-freedom GraspNet pose generation, the problem of inconvenient grasping in existing technologies is solved, and efficient and accurate grasping of any object is achieved autonomously.
Patent Information
- Application Number
- CN202512038771.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-05-15
AI Technical Summary
Existing six-DOF pose generation methods for grasping objects cannot efficiently interact with them, resulting in inconvenient grasping.
An RGB-D vision grasping system was built, using Mask R-CNN for instance segmentation and object detection. Combined with intrinsic and extrinsic parameter calibration, multiple candidate grasping poses were generated through a six-DOF GraspNet and evaluated and filtered to generate high-precision point cloud information to assist the end effector in autonomous grasping.
It enables autonomous grasping of any object in unstructured and complex environments, improves the success rate and accuracy of grasping, and provides a convenient method for object interaction.
Smart Images
Figure CN122049040A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and in particular relates to a method for pose generation and capture based on six-degree-of-freedom GraspNet. Background Technology
[0002] Currently, six-DOF pose generation methods for grasping objects can be mainly categorized into reinforcement learning-based methods, pose estimation-based methods, methods combining visual and language models, and geometry optimization-based methods. Simply using six-DOF pose generation methods alone cannot efficiently interact with the object being grasped. Therefore, six-DOF pose generation grasping methods that combine object detection and semantic segmentation can better interact with the desired object, providing a more convenient methodological foundation for users to grasp objects. Summary of the Invention
[0003] The present invention aims to solve at least one of the technical problems existing in the prior art.
[0004] This invention provides a method for grasping based on six-DOF GraspNet pose generation, the method comprising:
[0005] Build an RGB-D visual grasping system: Perform intrinsic and extrinsic parameter calibration on the camera, and obtain the intrinsic and extrinsic parameter matrices based on the intrinsic and extrinsic parameter calibrations respectively;
[0006] Mask R-CNN is constructed and used for color image instance segmentation, outputting object category and object pixel information. The Mask R-CNN network is used to achieve parallel object detection and semantic segmentation. The Mask R-CNN network includes a backbone network, RPN network, RoI Align network and FCN network connected in sequence. The FCN network includes a mask branch network, which is used to generate category boxes and mask information for objects at the same time.
[0007] Target point cloud extraction: The dataset is trained using the Mask R-CNN network, and the target point cloud of the object is extracted by combining the intrinsic parameter matrix, depth image, color image and object pixel information;
[0008] Six-DOF GraspNet pose generation and evaluation: Input the target object point cloud into the six-DOF GraspNet pose generation network to generate multiple candidate grasping poses, and then filter and evaluate them by evaluating the network and threshold.
[0009] Furthermore, the extrinsic parameter calibration installation package is placed in the workspace for compilation. After successful compilation, the end effector and camera are tested in sequence. After successful testing, the calibration board is placed, the end effector position is repeatedly moved, and the corresponding extrinsic parameter matrix data is read in sequence.
[0010] Furthermore, Mask R-CNN instance segmentation uses the Faster R-CNN algorithm for object detection and the FCN algorithm for the corresponding semantic segmentation of objects.
[0011] Furthermore, after instance segmentation, the mask information of the items is saved in the corresponding JSON format or binary image format.
[0012] Furthermore, after training the dataset using the Mask R-CNN network, the corresponding weight file is generated. Using the weight file of the final round, the types and corresponding paths in the Python file for prediction execution are modified, and an npy file containing depth image information is added to the code.
[0013] Furthermore, the six-DOF GraspNet pose generation network uses a variational autoencoder to augment the pose of the point cloud information.
[0014] Furthermore, the six-DOF GraspNet pose generation network uses PointNet++ to capture, classify, and score the point cloud information.
[0015] This invention provides a method for grasping objects based on six-DOF GraspNet pose generation. This method effectively generates multiple candidate poses through the construction of an RGB-D visual grasping system, instance segmentation using Mask R-CNN, point cloud extraction, and pose generation and evaluation using 6-DOF GraspNet. This facilitates subsequent autonomous grasping operations by the end effector, enabling the autonomous grasping of arbitrary objects in unstructured and complex environments. Compared with existing technologies, this invention solves the technical problem of inefficient interaction with the grasped object. Attached Figure Description
[0016] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.
[0017] Figure 1 A flowchart illustrating the method for pose generation and grasping based on six-DOF GraspNet is shown.
[0018] Figure 2 A schematic diagram of the Aruco code is shown;
[0019] Figure 3 A schematic diagram of the Mask R-CNN network structure is shown.
[0020] Figure 4 A schematic diagram of the instance segmentation effect is shown;
[0021] Figure 5 The binary image corresponding to the object is shown;
[0022] Figure 6 The depth map corresponding to the object is shown;
[0023] Figure 7 The mask diagram corresponding to the object is shown;
[0024] Figure 8 The point cloud extraction process for the item is shown;
[0025] Figure 9 The pose sampling structure diagram is shown;
[0026] Figure 10 The image shows the pose generation of different objects. Figure 10 (a) The pose of the corresponding box is captured and generated. Figure 10 (b) The pose of the cup is captured and a graph is generated. Detailed Implementation
[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0029] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0030] like Figure 1 As shown in the figure, a method for grasping based on six-DOF GraspNet pose generation is provided according to a specific embodiment of the present invention. The method includes:
[0031] Build an RGB-D visual grasping system: perform intrinsic and extrinsic parameter calibration on the camera, and obtain the intrinsic and extrinsic parameter matrices based on the intrinsic and extrinsic parameter calibration;
[0032] Mask R-CNN is constructed and used for color image instance segmentation, outputting object category and object pixel information. The Mask R-CNN network is used to achieve parallel object detection and semantic segmentation. The Mask R-CNN network includes a backbone network, RPN network, RoI Align network and FCN network connected in sequence. The FCN network includes a mask branch network, which is used to generate category boxes and mask information for objects at the same time.
[0033] Target point cloud extraction: The dataset is trained using the Mask R-CNN network, and the target point cloud of the object is extracted by combining the intrinsic parameter matrix, depth image, color image and object pixel information;
[0034] Six-DOF GraspNet pose generation and evaluation: Input the target object point cloud into the six-DOF GraspNet pose generation network to generate multiple candidate grasping poses, and then filter and evaluate them by evaluating the network and threshold.
[0035] This configuration provides a method for grasping based on six-DOF GraspNet pose generation. This method effectively generates multiple candidate poses by building an RGB-D visual grasping system, performing instance segmentation with Mask R-CNN, extracting point clouds, and generating and evaluating poses with 6-DOF GraspNet. This is beneficial for subsequent autonomous grasping operations by the end effector, enabling the autonomous grasping of any object in an unstructured and complex environment.
[0036] First, in this invention, an RGB-D visual grasping system is built: the camera is calibrated using standard intrinsic and extrinsic parameters, and the intrinsic and extrinsic parameter matrices are obtained based on the intrinsic and extrinsic parameter calibrations, respectively.
[0037] The key technologies included in the vision grasping system are intrinsic parameter calibration, extrinsic parameter calibration, transformation of rotation matrices in different coordinate systems, and data format conversion to guide the motion of the end effector. Intrinsic and extrinsic parameter calibration ensures that the obtained intrinsic and extrinsic parameter matrices have minimal errors.
[0038] Preferably, the camera intrinsic parameter calibration environment mainly involves installing the ROS operating system and the camera_calibration package on Ubuntu 18.04, and then configuring the corresponding software packages. The calibration checkerboard used is an 8×6 108mm grid. 2 The calibration board, and the calibrated intrinsic parameter matrix K are as follows, in millimeters (mm):
[0039]
[0040] Extrinsic parameter calibration requires the installation of the corresponding extrinsic parameter calibration package. This package includes software packages such as visip, aruco_ros, and easy_handeye. Place all packages in the workspace and compile them. After successful compilation, test the end effector and camera sequentially. After successful testing, place the calibration board on the table and repeatedly move the end effector position, reading the corresponding extrinsic parameter matrix data sequentially. Aruco codes are required during extrinsic parameter calibration, and these codes must be consistent with those in the aruco_ros package. Figure 2 As shown.
[0041] Enter the compilation space easy_handeye / easy_handeye / launch, and perform camera extrinsic parameter calibration on the file corresponding to launch. During the calibration process, the robot (end effector) changes angle and ensures that the calibration plate does not disappear from the field of view. Multiple sets of extrinsic parameter matrix data are collected for calculation.
[0042] Preferably, 17 sets of extrinsic parameter matrix data are collected and averaged. After extrinsic parameter calibration, the transformation matrix T (i.e., the extrinsic parameter matrix) from the camera coordinate system to the robot end effector coordinate system is finally obtained, as shown below:
[0043]
[0044] The extrinsic parameter matrix is used for coordinate transformation of the end effector (manipulator) after pose generation, so as to facilitate the subsequent autonomous grasping operation of the end effector.
[0045] Furthermore, in this invention, Mask R-CNN is constructed and used for color image instance segmentation, outputting object type and object pixel information. The Mask R-CNN network comprises a backbone network, an RPN network, a RoIAlign network, and an FCN network connected sequentially. The FCN network includes a mask branch network used to simultaneously generate category boxes and mask information for objects. This Mask R-CNN network is used to achieve parallel object detection and semantic segmentation. To achieve simultaneous object detection and semantic segmentation, and to extract clearer point cloud information of objects, this invention uses a Mask R-CNN-based instance segmentation network to output object type and object pixel information.
[0046] The Mask R-CNN instance segmentation method mainly combines two algorithms: Faster R-CNN and FCN. The Faster R-CNN algorithm is used for object detection, and the FCN algorithm is used for semantic segmentation of objects. Therefore, the Mask R-CNN algorithm realizes parallel object detection and semantic segmentation, enabling instance segmentation of objects. It can output the corresponding object categories and perform pixel-level segmentation of each object.
[0047] Instance segmentation, as the foundation for subsequent crawling, is a crucial step in the entire process. In this invention, Mask R-CNN adds a branch network to Faster R-CNN's object detection, enabling parallel object detection and semantic segmentation. The entire Mask R-CNN network mainly consists of four parts: the backbone network, the RPN network, the RoIAlign network, and the FCN network. These four main neural networks are connected to achieve the final instance segmentation process, grouping similar items together and different items for instance segmentation.
[0048] The mask branch network added to Faster R-CNN is as follows: Figure 3 As shown, this network can generate both category boxes and masks for items simultaneously.
[0049] Preferably, experiments involving instance segmentation of images captured by cameras other than the dataset, such as... Figure 4 As shown, the cup and box instances were segmented accordingly, including cups of the same type, cups and boxes of different types, etc.
[0050] Regarding the instance segmentation results, corresponding detections were performed at different locations in multiple images to ensure that complete instance segmentation could be performed at each location when captured by the camera. The training and test data in the experiment are shown in Table 1. Judging from the test image effects and success rate, the instance segmentation effect of the present invention is relatively good.
[0051] Table 1. Analysis of Instance Segmentation and Detection Results
[0052]
[0053] Furthermore, in this invention, target point cloud extraction is performed: the dataset is trained using a Mask R-CNN network, and the target point cloud of the object is extracted by combining the intrinsic parameter matrix, depth image, color image and object pixel information.
[0054] As a specific embodiment of the present invention, object detection and segmentation in a home environment are performed. Considering that identical items may not exist in the dataset, a custom dataset is created. Objects are categorized into two types based on their type, shape, and size: cups and boxes. This primarily includes different shaped cups and instance segmentation between different cups and boxes, fully demonstrating the characteristics of instance segmentation. Instance segmentation is performed on both items of the same and different types. Furthermore, each item has its own unique grasping characteristics; therefore, these two types of items were primarily chosen for the experiment. Instance segmentation mainly combines object detection and semantic segmentation, improving upon previous single object detection and semantic segmentation networks and further enhancing the sophistication of image recognition. Based on the characteristics of instance segmentation, the MS COCO dataset is selected for detection. To improve the generalization of object representation, 2000 images of objects at different locations and under different lighting conditions are created as images in the dataset.
[0055] The instance segmentation in the previous step obtained the mask pixel information corresponding to the object from the color image. By combining the intrinsic parameter matrix in the aforementioned calibration and the depth image obtained from the camera, the point cloud map corresponding to the target object can be obtained.
[0056] After instance segmentation, the mask information of the items is saved in either JSON format or binary image format. Here, the binary image format is primarily used for subsequent calculations. The binary image of the target object is shown below. Figure 5 As shown, the main feature is to clearly separate the object from the background and to distinguish it into black and white.
[0057] After training the dataset using the Mask R-CNN network, the corresponding weight file is generated. Using the weight file of the final round, the categories in the prediction execution Python file are changed to the category names in the training set, and the paths involved are changed to the corresponding paths. An npy file for generating image information is added to the code for use in subsequent point cloud generation.
[0058] Preferably, the color depth image captured by the RealSense D435i camera is converted to grayscale, such as... Figure 6 And generate a mask image, such as Figure 7 As shown, the generated depth map makes it easier to see the depth of the corresponding object. The depth information is saved in .npy data format because the input data required for subsequent crawling requires a resolution of 640×480. Therefore, the data format saved here corresponds one-to-one with the required format for subsequent crawling applications.
[0059] In instance segmentation, point clouds of the desired target object can be extracted from different types of items or items of the same type. You only need to select the corresponding item type name in the category. Figure 8 The process of extracting the cup and the box from the box is given, and a visualization of the target point cloud is shown.
[0060] This invention combines depth images to extract point cloud information of the target object, providing a clearer point cloud basis for subsequent end effector grasping, making the grasping pose more accurate and the precision higher.
[0061] Furthermore, in this invention, pose generation and evaluation using a six-DOF GraspNet are performed: the point cloud of the target object is input into a six-DOF GraspNet pose generation network to generate multiple candidate grasping poses, which are then screened and evaluated using an evaluation network and a threshold.
[0062] To achieve autonomous, unstructured grasping tasks, target point clouds extracted from instance segmentation are input into a six-degree-of-freedom (6DOF) neural network. This 6DOF neural network primarily employs a variational autoencoder to augment the point cloud information with poses, and PointNet++ is used to classify and score the point cloud information for grasping purposes. Finally, grasping poses with different scores are output. By using the scores and thresholds of the grasping poses, a preliminary assessment can be made of which poses have a high success rate in grasping.
[0063] The variational autoencoder, primarily within the framework of the 6-DOF GraspNet pose generation network, aims to generate as many poses as possible, ensuring poses are distributed across various regions. This maximizes the coverage of pose generation within the field of view, providing more possibilities for subsequent grasping and indirectly increasing the grasping success rate. It also enriches the grasping locations, offering more possibilities for object grasping. Furthermore, this algorithm optimizes, refines, and applies maximum likelihood estimation, Gaussian mixture modeling, and inference algorithms, significantly increasing the number of corresponding generated samples. The grasping generated patterns are roughly as follows: Figure 9 As shown.
[0064] PointNet++ demonstrates outstanding advantages in neural network evaluation, capable of evaluating and scoring corresponding poses while efficiently completing tasks, greatly facilitating subsequent crawling.
[0065] For example, the pose of the cup and box is captured and an image is generated. Figure 10 As shown, the candidates to be crawled are different for each calculation, and they will be recalculated based on the actual input data.
[0066] This invention aims to enable autonomous grasping of everyday objects in complex environments. It proposes a grasping method based on six-DOF GraspNet pose generation. By building an RGB-D visual grasping system, performing instance segmentation using Mask R-CNN, extracting point clouds, and generating and evaluating poses using 6-DOF GraspNet, the density of the extracted point clouds is effectively improved. Multiple candidate poses are generated through the GraspNet six-DOF pose generation network. The obtained poses can be transformed into corresponding coordinates for subsequent autonomous grasping operations by the end effector. This allows users to grasp target objects more quickly based on different grasping angles, thereby completing the autonomous grasping task of any object in unstructured complex environments.
[0067] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for grasping pose generation based on six-DOF GraspNet, characterized in that, The method for capturing pose generation based on six-degree-of-freedom GraspNet includes: Build an RGB-D visual grasping system: perform intrinsic and extrinsic parameter calibration on the camera, and obtain the intrinsic and extrinsic parameter matrices based on the intrinsic and extrinsic parameter calibration; Mask R-CNN is constructed and used for color image instance segmentation, outputting object category and object pixel information. The Mask R-CNN network is used to achieve parallel object detection and semantic segmentation. The Mask R-CNN network consists of a backbone network, RPN network, RoIAlign network and FCN network connected in sequence. The FCN network includes a mask branch network, which is used to generate category boxes and mask information for objects at the same time. Target point cloud extraction: The dataset is trained using the Mask R-CNN network, and the target point cloud of the object is extracted by combining the intrinsic parameter matrix, depth image, color image and object pixel information; Six-DOF GraspNet pose generation and evaluation: Input the target object point cloud into the six-DOF GraspNet pose generation network to generate multiple candidate grasping poses, and then filter and evaluate them by evaluating the network and threshold.
2. The method of generating grasps based on a 6-DOF GraspNet pose according to claim 1, wherein, The extrinsic parameter calibration installation package is placed in the workspace and compiled. After successful compilation, the end effector and camera are tested in sequence. After successful testing, the calibration board is placed, the end effector position is moved repeatedly, and the corresponding extrinsic parameter matrix data is read in sequence.
3. The method for capturing pose generation based on six-degree-of-freedom GraspNet according to claim 1, characterized in that, Mask R-CNN instance segmentation uses the Faster R-CNN algorithm for object detection and the FCN algorithm for the corresponding semantic segmentation of objects.
4. The method for capturing pose generation based on six-DOF GraspNet according to claim 1, characterized in that, After instance segmentation, the mask information of the items is saved in the corresponding JSON format or binary image format.
5. The method for grasping based on six-degree-of-freedom GraspNet pose generation according to claim 1, characterized in that, After training the dataset using the Mask R-CNN network, the corresponding weight file is generated. Using the weight file of the final round, the types and corresponding paths in the Python file for prediction execution are modified, and an npy file containing depth image information is added to the code.
6. The method for grasping based on six-degree-of-freedom GraspNet pose generation according to claim 1, characterized in that, The six-DOF GraspNet pose generation network uses a variational autoencoder to augment the pose of point cloud information.
7. The method for capturing pose generation based on six-DOF GraspNet according to claim 1, characterized in that, The six-DOF GraspNet pose generation network uses PointNet++ to capture, classify, and score point cloud information.