A robot grasping detection method based on multimodal visual information fusion

Through the multimodal visual information fusion method, the YOLO network and the convolutional neural network are used to generate stable grasping postures, solving the problem of unstable grasping in complex environments and achieving efficient interaction between the robotic arm and the environment.

CN115861999BActive Publication Date: 2025-03-21NORTHWESTERN POLYTECHNICAL UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202211212605.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-30
Publication Date
2025-03-21
Estimated Expiration
2042-09-30

AI Technical Summary

Technical Problem

The prior art is difficult to generate a stable grab pose in complex environments, and cannot fully utilize the global point cloud information of objects, resulting in unstable grabbing and inability to apply to complex scenarios.

Method used

RGB images and depth images are obtained through the depth camera, the target object is detected using the YOLO network, the three-dimensional coordinates of the bounding box are obtained in combination with the depth information, the point cloud of the target object is segmented and extracted, voxelized downsampling, filtering and clustering are performed, the point cloud center of mass and normal vector direction are calculated, candidate grab poses are generated, and the scores of each pose are calculated through the convolutional neural network, and the global and local point cloud information are fused to select the highest quality grab pose.

Benefits of technology

Generate a stable grasping posture in complex environments, improve the interaction ability between the robot arm and the environment, and ensure the stability and accuracy of the grasping operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115861999B_ABST
    Figure CN115861999B_ABST
Patent Text Reader

Abstract

The present invention relates to a robot grasping detection method based on multimodal visual information fusion, comprising: obtaining RGB images and depth images through a depth camera; using deep learning YOLO to detect the external rectangular frame of the target object; combining the depth image to obtain the bounding box of the target object; segmenting and extracting the point cloud of the target object; processing the point cloud, successively performing downsampling, point cloud filtering and point cloud clustering operations; calculating the centroid of the target point cloud and using the PCA algorithm to calculate the principal component direction; randomly and uniformly sampling the target point cloud to generate candidate grasping postures; encoding the grasping candidate internal point cloud into a multi-channel image, and using a convolutional neural network to predict the score; fusing the global point cloud information and the local point cloud information, and selecting the highest quality grasping posture as the execution posture through weighted summation. The method makes full use of the global and local information of the color image, the depth image and the target object point cloud, and improves the interaction ability of the robot arm with the environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of robot target object grasping, and in particular is a grasping posture detection method based on the fusion of image and point cloud visual information. Background Art

[0002] With the development of artificial intelligence, robots are widely used in the industrial and service sectors. Among them, the grasping operation of robots plays an important role, effectively avoiding the numerous pick-and-place tasks of humans. Compared with the previous grasping of fixed objects or grasping at fixed positions, it is of great significance for robots to grasp specified objects autonomously and accurately. The key is to know the objects in the scene when operating autonomously, and how to detect the grasping posture under different visual information. Therefore, target recognition and grasping posture estimation are research hotspots in the field of robotics.

[0003] Deep learning technology is an important technical method in the direction of target recognition and grasping posture estimation. YOLO, as a real-time single-stage target detection network, has been widely used in the field of robot vision detection. For 6DoF grasping posture detection, the traditional grasping posture estimation method generates the grasping posture based on the object posture of the known CAD model, but it is not easy to obtain an accurate model of the object in reality. At present, the model-free grasping posture detection process is to first sample and generate grasping candidates and then evaluate them. For example, the grasping posture generation network based on point clouds (A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt, "Grasp pose detection in point clouds," The International Journal of Robotics Research (IJRR), 2017) directly judges the grasping quality based on the internal point cloud of the grasping closure. The advantage is that the amount of data is greatly reduced and the generalization is strong. However, since the global point cloud information of the object is not fully utilized, a stable grasp cannot be generated. At the same time, the type of the grasped object is not known, and it cannot be directly applied in complex scenes. The fusion extraction of image and point cloud visual information features is of utmost importance for robot autonomous operation. Summary of the invention

[0004] Technical issues to be solved

[0005] In order to avoid the shortcomings of the prior art, the present invention provides a robot grasping detection method based on multimodal visual information fusion, which can realize the generation of stable grasping posture for specified target objects in complex environments, make full use of color images, depth images and global and local information of target object point clouds, and improve the interaction ability of the robot arm with the environment.

[0006] Technical Solution

[0007] A robot grasping detection method based on multimodal visual information fusion, characterized by the following steps:

[0008] Step 1: Obtain RGB image and depth image containing the target object through the depth camera;

[0009] Step 2: Use the deep learning YOLO network to detect the bounding rectangle of the target object in the RGB image;

[0010] Step 3: Use the depth information corresponding to the rectangular box to obtain the three-dimensional coordinates of the bounding box of the target object in the camera coordinate system;

[0011] Step 4: Segment and extract the point cloud set of the target object according to the corner point information of the target object's bounding box;

[0012] Step 5: Perform voxel downsampling, point cloud filtering, and point cloud clustering on the segmented point cloud to obtain the ideal point cloud of the target object;

[0013] Step 6: Calculate the centroid of the target ideal point cloud and use the PCA algorithm to calculate the point cloud normal vector direction as the target grasping approach reference direction, which is used as the global information of the target object point cloud;

[0014] Step 7: Randomly and uniformly sample the target point cloud to generate candidate grasping postures, and expand more grasping postures through rotation and translation;

[0015] Step 8: Encode the internal point cloud of the grasp candidate into a compressed multi-channel image and use a convolutional neural network to calculate the score of each grasp candidate;

[0016] Step 9: Calculate the Euclidean distance between the grasp candidate center point and the object point cloud centroid, as well as the angle between the grasp approach direction and the grasp approach reference direction. By calculating the grasp candidate score, the sum of the Euclidean distance and the weighted score of the angle, select the grasp with the highest quality as the robot's grasp target posture, effectively integrating the global point cloud and local point cloud information of the target object.

[0017] A further technical solution of the present invention: the camera described in step 1 is a depth camera or a binocular camera, which can be fixed at the end of the robotic arm and move with the robotic arm, or can be completely fixed.

[0018] A further technical solution of the present invention: The deep learning YOLO network described in step 2 is based on the object category of the rectangular box of the object in the RGD image, the relative ratio of the center point coordinates to the image width and height, and the relative ratio of the height width to the image width and height as labels, trains the YOLO target object detection model, and obtains the pixel coordinates corresponding to the corner points of the target bounding box.

[0019] Further technical solution of the present invention: Step 3 is specifically: based on the target rectangular frame in the RGB image, by fusing the maximum and minimum values ​​of the depth information of the area where the rectangular frame is located in the depth image, the geometric information of the bounding box of the target object in the camera coordinate system and the corner point coordinates in the camera coordinate system are calculated.

[0020] A further technical solution of the present invention: The segmentation and extraction of the point cloud described in step 4 is based on the bounding box information, that is, the coordinate range of the target object in the camera coordinate system along the three axes of x, y, and z, and a magnification factor λ is added to ensure that all point clouds of the target object are retained, and then the target point cloud is segmented and extracted using a point cloud pass filter.

[0021] A further technical solution of the present invention: the point cloud extracted by segmentation is processed in step 5, the point cloud voxelization and downsampling operation is used to reduce the amount of point cloud data, the point cloud filtering can adopt radius filtering or statistical filtering to remove outliers, and the point cloud clustering operation is used to obtain the largest cluster point set to obtain the point cloud of the target object.

[0022] A further technical solution of the present invention: The global information of the target object point cloud C in step 6 includes the centroid P cen (x cen ,y cen ,z cen ) and the grasping approach direction v3, where the point cloud set C contains n points, expressed as:

[0023] C=(c1,c2,...,c n )

[0024] Centroid P cen is the average value of all point coordinates in the point cloud set, and the calculation formula is:

[0025]

[0026] The grasping approach direction is obtained by calculating the eigenvalues ​​λ1, λ2, λ3 of the point cloud covariance matrix S and the corresponding eigenvectors v1, v2, v3 through the PCA algorithm, where λ1≥λ2≥λ3; the direction with the smallest variance is the normal vector direction of the target point cloud, which is used as the reference direction for grasping approach, that is, v3; the point cloud covariance is calculated as follows:

[0027]

[0028] A further technical solution of the present invention is as follows: the random sampling of the target point cloud to establish a grasping coordinate system in step 7 is to take each sampling point as the center of a sphere, and to form a point cloud set of the sampling point by a number of points closest to the surrounding points, and to calculate the homogeneous coordinate matrix of the point cloud set by using the PCA algorithm, thereby establishing the grasping coordinate system of the point;

[0029] In order to generate more grasping candidate postures, the grasping candidate postures are expanded by adding translation along the y-axis of the current grasping coordinate system and adding rotation around the z-axis, while continuously translating along the x-axis so that the x-coordinate value meets the critical value that the grasping posture does not collide with the target point cloud;

[0030] There are two conditions that need to be met to capture candidate poses:

[0031] (1) The grasping posture is located so that it does not contain the target point cloud, that is, the robot arm gripper does not collide with the target point cloud;

[0032] (2) The target point cloud is contained in the closed area of ​​the grasping posture, that is, the target point cloud is contained inside the gripper;

[0033] The generated grasping candidate postures are screened according to the above conditions to remove invalid grasping candidates.

[0034] A further technical solution of the present invention: The input of the convolutional neural network for predicting the grasping candidate scores described in step 8 is to project each generated valid grasping candidate internal point cloud along the three coordinate axis directions of the grasping coordinate system, and compress it into a depth map according to the coordinate value in this direction, that is, only retain the depth information along the current coordinate axis, and at the same time map the normal vector of each point on the point cloud surface in this direction to an RGB type normal feature map; by generating a depth feature map and a normal feature map, they are used to extract local surface features and geometric features of the point cloud, and the above-mentioned depth map and normal feature map are used together as the input of the convolutional neural network.

[0035] A further technical solution of the present invention: the candidate center point p is captured in step 9 g (x g ,y g ,z g ) and the object point cloud centroid p cen (x cen ,y cen ,z cen The calculation formula of the Euclidean distance d is:

[0036]

[0037] where d max It is the reference value for grabbing the maximum value of the candidate center point and the object point cloud centroid;

[0038] The angle calculation method between the grasping approach direction and the grasping approach reference direction in step 9 is to use the grasping approach direction vector v approach (x a ,y a ,z a ) and the grasping approach reference direction vector Calculate the angle α, where the vector v approachBoth v and v3 are unit vectors, and the calculation formula is:

[0039]

[0040]

[0041] That is, the value range of α is [0,90];

[0042] The process of calculating the final crawl quality score S described in step 9 is to normalize first and then weight, and the calculation formula is:

[0043]

[0044] Among them, γ, λ and η are the weighting coefficients of network score, distance and angle respectively, and their value range is [0,1). max It is the reference value for the maximum value of the crawling network prediction score.

[0045] Beneficial Effects

[0046] The present invention provides a robot grasping detection method based on multimodal visual information fusion, which can realize the generation of stable grasping posture for a specified target object in a complex environment, make full use of color images, depth images and global and local information of the target object point cloud, and improve the interaction ability of the robot arm with the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0048] Figure 1 The figure is an overall flow chart of the method disclosed in the present invention.

[0049] Figure 2 This is the YOLO target detection result diagram.

[0050] Figure 3 Schematic diagram of the grabbing coordinate system for calculating the target point cloud using the PCA algorithm.

[0051] Figure 4 Schematic diagram of the grabbing coordinate system of the sampling points.

[0052] Figure 5 The point cloud (feature) encoding map inside the gripper: (a) depth feature map; (b) normal feature map. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0054] like Figure 1 As shown, it is an overall flow chart of the method of the present invention. The present invention discloses a robot grasping detection method based on multimodal visual information fusion, which mainly includes nine steps: step 1, obtaining RGB images and depth images through a depth camera; step 2, using deep learning YOLO to detect the external rectangular frame of the target object; step 3, combining the depth image to obtain the bounding box of the target object; step 4, segmenting and extracting the point cloud of the target object; step 5, processing the point cloud, successively performing downsampling, point cloud filtering and point cloud clustering operations; step 6, calculating the center of mass of the target point cloud and using the PCA algorithm to calculate the direction of the principal component; step 7, randomly and uniformly sampling the target point cloud to generate candidate grasping postures; step 8, encoding the internal point cloud of the grasping candidate into a multi-channel image, and using a convolutional neural network to predict the score; step 9, fusing the global point cloud information and the local point cloud information, and selecting the highest quality grasping posture as the execution posture through weighted summation.

[0055] The implementation of the present invention requires a depth camera or a binocular camera and a GPU. In the specific implementation process, a notebook with a Geforce 1060 GPU and a realsense d435i depth camera are used.

[0056] The method disclosed in the present invention specifically comprises the following steps:

[0057] Step 1: Acquire an RGD image and a depth image containing a target object through a depth camera; the camera is a depth camera or a binocular camera, which can be fixed at the end of a robotic arm and move with the robotic arm, or can be completely fixed.

[0058] Step 2, based on RGB image, such as Figure 2 As shown, the target object is detected using deep learning YOLO detection, and the output is the circumscribed rectangular frame information of the target object.

[0059] Specifically, it is the ratio of the horizontal and vertical coordinates of the center point in the camera coordinate system to the width and height of the input image px, py, and the ratio of the width and height of the rectangular box to the width and height of the input image pw, ph, where the input is a fixed-size image with a width of w and a height of h. The above information is used to calculate the coordinates of the four corner points of the rectangular box p tl (x1,y1),p tr (x2,y2),p bl(x3,x3), p br (x4,y4), the calculation formula is:

[0060]

[0061]

[0062]

[0063]

[0064] Step 3: Find the maximum value z of the depth information within the target's bounding rectangle in the depth image max With the minimum value z min , in order to comprehensively calculate the three-dimensional coordinates of the eight corner points of the bounding box of the target object in the camera coordinate system.

[0065] Step 4: Segment and extract the point cloud set of the target object according to the corner point information of the bounding box of the target object;

[0066] Specifically, according to the coordinate range of the target object along the x, y, and z axes in the camera coordinate system, a magnification factor λ is added to ensure that all point clouds of the target object are retained, and then the point cloud conditional filter is used to segment and extract the target point cloud set.

[0067] Step 5: voxelize and downsample the point cloud, filter the point cloud to remove outliers, and cluster the point cloud to obtain the ideal point cloud of the target object. The point cloud voxelization and downsampling operation is used to reduce the amount of point cloud data, the point cloud filtering can use radius filtering or statistical filtering to remove outliers, and the point cloud clustering operation is used to obtain the largest cluster point set to obtain the point cloud of the target object;

[0068] Step 6: Calculate the centroid of the target ideal point cloud and use the PCA algorithm to calculate the homogeneous coordinate matrix of the point cloud. Take the direction with the smallest variance as the target point cloud grab approach direction, and establish the reference coordinate system of the target point cloud as follows: Figure 3 As shown, the center of mass and grasp approach direction are used as the global information of the target point cloud;

[0069] The global information of the target object point cloud C includes the centroid P cen (x cen ,y cen ,z cen ) and the grasping approach direction v3, where the point cloud set C contains n points, expressed as:

[0070] C=(c1,c2,...,c n )

[0071] Centroid P cen is the average value of all point coordinates in the point cloud set, and the calculation formula is:

[0072]

[0073] The grasping approach direction is obtained by calculating the eigenvalues ​​λ1, λ2, λ3 (where λ1≥λ2≥λ3) of the point cloud covariance matrix S and the corresponding eigenvectors v1, v2, v3 through the PCA algorithm. The direction with the smallest variance is the normal vector direction of the target point cloud, which is used as the reference direction for grasping approach (i.e. v3). The point cloud covariance is calculated as follows:

[0074]

[0075] Step 7: Randomly and uniformly sample the target point cloud to generate candidate grasping postures, and expand more grasping postures through rotation and translation.

[0076] Taking each sampling point as the center of the sphere, a point cloud set of the sampling point is formed by several points closest to the surrounding area. The PCA algorithm is used to calculate the homogeneous coordinate matrix of the point cloud set, thereby establishing the grabbing coordinate system of the point.

[0077] Specifically, each sampling point is taken as the sphere center, and the nearest points around it constitute the point cloud set C of the sampling point. sample ; Taking sampling point p sample (x s ,y s ,z s ) as the center, and use the PCA algorithm to calculate the eigenvalue λ of the covariance of the local point cloud set s1 ,λ s2 ,λ s3 With the feature vector v s1 ,v s2 ,v s3 , thereby establishing the grasping coordinate system g(p sample ,v s1 ,v s2 ,v s3 );like Figure 4 shown.

[0078] In order to generate more grasping candidate postures, the grasping candidate postures are expanded by adding translation along the y-axis of the current grasping coordinate system and adding rotation around the z-axis, while continuously translating along the x-axis so that the x-coordinate value meets the critical value that the grasping posture does not collide with the target point cloud.

[0079] There are two conditions that need to be met to capture candidate poses:

[0080] (1) The grasping posture is located in a position that does not contain the target point cloud, that is, the robot arm gripper does not collide with the target point cloud.

[0081] (2) The target point cloud is contained in the closed area of ​​the grasping posture, that is, the target point cloud is contained inside the gripper.

[0082] The generated grasping candidate postures are screened according to the above conditions, and invalid grasping candidates are removed to obtain a grasping candidate set G, which contains n grasping postures g.

[0083] The input of the convolutional neural network for predicting the grasping candidate scores described in step 8 is to project each generated valid grasping candidate internal point cloud along the three coordinate axis directions of the grasping coordinate system, and compress it into a depth map according to the coordinate value in this direction, that is, only retain the depth information along the current coordinate axis, and at the same time map the normal vector of each point on the point cloud surface in this direction to an RGB type normal feature map. By generating a depth feature map and a normal feature map, the local surface features and geometric features of the point cloud are extracted, and the above-mentioned depth map and normal feature map are used together as the input of the convolutional neural network. Figure 5 shown.

[0084] The calculation formula of the Euclidean distance d between the grasped candidate center point and the centroid of the object point cloud described in step 9 is:

[0085]

[0086] where d max It is the reference value for grabbing the maximum value of the candidate center point and the object point cloud centroid.

[0087] The angle between the grasping approach direction and the PCA main direction is calculated by using the grasping approach direction vector A(x a ,y a ,z a ) and the PCA principal component direction vector B(x b ,y b ,z b ) to calculate the angle, where vectors A and B are both unit vectors, and the calculation formula is:

[0088]

[0089]

[0090] That is, the value range of α is [0,90]

[0091] The process of calculating the final crawl quality score S described in step 9 is to normalize first and then weight. The calculation formula is:

[0092]

[0093] Among them, γ, λ and η are the weighting coefficients of network score, distance and angle respectively, and their value range is [0,1). max It is the reference value for the maximum value of the crawling network prediction score.

[0094] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A robot grasping detection method based on multimodal visual information fusion, characterized in that Here are the steps: Step 1: Obtain RGB image and depth image containing the target object through the depth camera; Step 2: Use the deep learning YOLO network to detect the bounding rectangle of the target object in the RGB image; Step 3: Use the depth information corresponding to the rectangular box to obtain the three-dimensional coordinates of the bounding box of the target object in the camera coordinate system; Step 4: Segment and extract the point cloud set of the target object according to the corner point information of the target object's bounding box; Step 5: Perform voxel downsampling, point cloud filtering, and point cloud clustering on the segmented point cloud to obtain the ideal point cloud of the target object; Step 6: Calculate the centroid of the target ideal point cloud and use the PCA algorithm to calculate the point cloud normal vector direction as the target grasping approach reference direction, which is used as the global information of the target object point cloud; Step 7: Randomly and uniformly sample the target point cloud to generate candidate grasping postures, and expand more grasping postures through rotation and translation; Step 8: Encode the internal point cloud of the grasp candidate into a compressed multi-channel image and use a convolutional neural network to calculate the score of each grasp candidate; Step 9: Calculate the Euclidean distance between the grasp candidate center point and the object point cloud centroid, as well as the angle between the grasp approach direction and the grasp approach reference direction. By calculating the grasp candidate score, the sum of the Euclidean distance and the weighted score of the angle, select the grasp with the highest quality as the robot's grasp target posture, effectively integrating the global point cloud and local point cloud information of the target object.

2. The robot grasping detection method based on multimodal visual information fusion according to claim 1 is characterized in that: The camera described in step 1 is a depth camera or a binocular camera, which can be fixed at the end of the robotic arm and move with the robotic arm, or can be completely fixed.

3. The robot grasping detection method based on multimodal visual information fusion according to claim 1 is characterized in that: The deep learning YOLO network described in step 2 is based on the object category of the rectangular box of the object in the RGD image, the relative ratio of the center point coordinates to the image width and height, and the relative ratio of the height width to the image width and height as labels, trains the YOLO target object detection model, and obtains the pixel coordinates corresponding to the corner points of the target bounding box.

4. The robot grasping detection method based on multimodal visual information fusion according to claim 1 is characterized in that: Step 3 is as follows: Based on the target rectangular frame in the RGB image, the maximum and minimum values ​​of the depth information of the area where the rectangular frame is located in the depth image are fused to calculate the geometric information of the bounding box of the target object in the camera coordinate system and the coordinates of the corner points in the camera coordinate system.

5. The robot grasping detection method based on multimodal visual information fusion according to claim 1 is characterized in that: The point cloud of the target object segmented and extracted in step 4 is based on the bounding box information, that is, the coordinate range of the target object along the x, y, and z axes in the camera coordinate system, and the magnification factor λ is added to ensure that all point clouds of the target object are retained, and then the target point cloud is segmented and extracted using a point cloud pass filter.

6. The robot grasping detection method based on multimodal visual information fusion according to claim 1 is characterized in that: The point cloud extracted by segmentation is processed as described in step 5. The point cloud voxelization and downsampling operation is used to reduce the amount of point cloud data. The point cloud filtering can use radius filtering or statistical filtering to remove outliers. The point cloud clustering operation is used to obtain the largest cluster point set to obtain the point cloud of the target object.

7. The robot grasping detection method based on multimodal visual information fusion according to claim 1 is characterized in that: The global information of the target object point cloud C described in step 6 includes the centroid P cen (x cen ,y cen ,z cen ) and the grasping approach direction v3, where the point cloud set C contains n points, expressed as: C=(c1,c2,...,c n ) Centroid P cen is the average value of all point coordinates in the point cloud set, and the calculation formula is: The grasping approach direction is obtained by calculating the eigenvalues ​​λ1, λ2, λ3 of the point cloud covariance matrix S and the corresponding eigenvectors v1, v2, v3 through the PCA algorithm, where λ1≥λ2≥λ3; the direction with the smallest variance is the normal vector direction of the target point cloud, which is used as the reference direction for grasping approach, that is, v3; the point cloud covariance is calculated as follows:

8. The robot grasping detection method based on multimodal visual information fusion according to claim 1 is characterized in that: The random sampling of the target point cloud to establish the grasping coordinate system in step 7 is to take each sampling point as the sphere center, and form a point cloud set of the sampling point by a number of points closest to the surrounding points, and use the PCA algorithm to calculate the homogeneous coordinate matrix of the point cloud set, thereby establishing the grasping coordinate system of the point; In order to generate more grasping candidate postures, the grasping candidate postures are expanded by adding translation along the y-axis of the current grasping coordinate system and adding rotation around the z-axis, while continuously translating along the x-axis so that the x-coordinate value meets the critical value that the grasping posture does not collide with the target point cloud; There are two conditions that need to be met to capture candidate poses: (1) The grasping posture is located so that it does not contain the target point cloud, that is, the robot arm gripper does not collide with the target point cloud; (2) The target point cloud is contained in the closed area of ​​the grasping posture, that is, the target point cloud is contained inside the gripper; The generated grasping candidate postures are screened according to the above conditions to remove invalid grasping candidates.

9. The robot grasping detection method based on multimodal visual information fusion according to claim 1 is characterized in that: The input of the convolutional neural network described in step 8 is to project each generated valid grasping candidate internal point cloud along the three coordinate axes of the grasping coordinate system, and compress it into a depth map according to the coordinate value in the coordinate axis direction, that is, only retain the depth information along the current coordinate axis, and at the same time map the normal vector of each point on the point cloud surface in the coordinate axis direction to an RGB type normal feature map; by generating a depth feature map and a normal feature map, the local surface features and geometric features of the point cloud are extracted, and the above-mentioned depth map and normal feature map are used together as the input of the convolutional neural network.

10. The robot grasping detection method based on multimodal visual information fusion according to claim 1, characterized in that: The candidate center point p described in step 9 is grasped g (x g ,y g ,z g ) and the object point cloud centroid p cen (x cen ,y cen ,z cen The calculation formula of the Euclidean distance d is: where d max It is the reference value for grabbing the maximum value of the candidate center point and the object point cloud centroid; The angle calculation method between the grasping approach direction and the grasping approach reference direction in step 9 is to use the grasping approach direction vector v approach (x a ,y a ,z a ) and the grasping approach reference direction vector Calculate the angle α, where the vector v approach Both v and v3 are unit vectors, and the calculation formula is: That is, the value range of α is [0,90]; The process of calculating the final crawl quality score S described in step 9 is to normalize first and then weight, and the calculation formula is: Among them, γ, λ and η are the weighting coefficients of network score, distance and angle respectively, and their value range is [0,1). max It is the reference value for the maximum value of the crawling network prediction score.

Citation Information

Patent Citations

  • Robot grabbing detection method based on instance segmentation under single-view-angle point cloud

    CN110363815A

  • Seven-degree-of-freedom grabbing posture detection method based on RGB image and depth image

    CN114140418A