A multimodal feature fusion method for simultaneous segmentation and detection of grasping pose

Abstract

The present invention provides a multimodal feature fusion method for simultaneous segmentation and detection of grasping posture, and relates to the technical field of robot grasping posture detection. The present invention designs a fusion module with collaborative guidance of point cloud images, which can not only fuse point cloud and image features, but also highlight key feature pairs and suppress features of non-interested areas. The prediction process of the algorithm starts with foreground point segmentation of the fused features, and then designs a seed point sampling algorithm that simultaneously considers semantic information, distance information, and grasping confidence to generate higher quality seed points. The grasping posture is then predicted based on the selected seed points. The present invention can make full use of point cloud and image information, and promote the performance of segmentation and grasping detection through multimodal features to serve application scenarios such as service robots and industrial robots.

Description

Technical Field

[0001] The present invention relates to the technical field of robot grasping posture detection, and in particular to a multimodal feature fusion method for simultaneously segmenting and detecting grasping posture. Background Art

[0002] Robotic visual grasp detection is a crucial research area in robotics. It involves the interaction and manipulation of robots with the real world and is an essential component of robotic applications, attracting widespread attention in recent years. Existing grasp pose detection methods are primarily categorized into planar grasp detection and 6-DOF grasp detection. Planar grasp detection methods typically predict a grasping rectangle corresponding to an object on an RGB image. This 2D rectangle requires the robotic arm to approach the object from a top-down perspective, limiting its application scenarios. 6-DOF grasp detection typically utilizes a 3D point cloud of the scene to predict a more flexible multi-degree-of-freedom (6-DOF) grasp pose. This grasping approach allows the robotic gripper to approach the object from any direction and holds great promise for future applications. 6-DOF grasp detection primarily utilizes neural networks to extract geometric structural features from 3D point clouds and then decode and predict high-level features. RGB images contain rich appearance and texture information, which can assist in tasks such as object segmentation and improve the robustness of grasp detection. However, existing 6-DOF grasp detection models often neglect the use of image information during feature extraction, hindering further improvements in model accuracy.

[0003] The paper "Wang C, Xu D, Zhu Y, et al. Densefusion: 6d object pose estimation by iterative dense fusion [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 3343-3352." proposed a method using dense fusion of RGB and depth data to estimate the 6D pose of objects in the scene. Subsequently, the paper "He, Yisheng, et al." Ffb6d: A full flow bidirectional fusion network for 6d pose estimation." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021." improved DenseFusion and proposed a full-flow bidirectional fusion network called FFB6D. The FFB6D network consists of two branches, one is the RGB branch and the other is the depth branch. The two branches share some convolutional layers to capture the correlation between image and depth. Then, the feature maps of the two branches are fused into a unified feature space. Next, the bidirectional manifold alignment module is used to align the features of the two branches to fully utilize the correlation between RGB and depth. The papers "Qi, Charles Ruizhongtai, et al." Pointnet++: Deep hierarchical feature learning on pointsets in a metric space." Advances in neural information processing systems 30 (2017)." and "Fang, Hao-Shu, et al." Graspnet-1 billion: A large-scale benchmark for general object grasping." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020." propose using the farthest point sampling algorithm to downsample the point cloud in three-dimensional space to generate seed points. This method can fully cover the entire point cloud space and well preserve the geometric features of the scene.

[0004] While the paper "Wang C, Xu D, Zhu Y, et al. Densefusion: 6D object pose estimation by iterative dense fusion [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 3343-3352." can fuse point cloud and image features, the algorithm only uses a simple feature concatenation step and does not highlight the key features of the point cloud and image feature pairs. The paper "He, Yisheng, et al. "Ffb6d: A full flow bidirectional fusion network for 6D pose estimation." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021." performs feature fusion at each stage of encoding and decoding point cloud and image features. However, as the image size gradually decreases during encoding, the alignment between the point cloud and image is disrupted, potentially leading to incorrect feature matching and fusion. The seed point sampling algorithm in the literature "Qi, Charles Ruizhongtai, et al." Pointnet++: Deephierarchical feature learning on point sets in a metric space." Advances inneural information processing systems 30 (2017).", "Fang, Hao-Shu, et al." Graspnet-1 billion: A large-scale benchmark for general object grasping." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020." only uses distance metric for sampling. Although the farthest point sampling algorithm can well cover the entire three-dimensional space, it also includes background points and points with low grasping confidence. In summary, the current algorithm cannot fully fuse point cloud-image feature pairs and generate higher quality seed points. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper provides a multimodal feature fusion method for simultaneous segmentation and grasp pose detection. This method achieves efficient fusion of point cloud and image feature pairs, simultaneously performing foreground-background segmentation and grasp pose detection. It also combines the semantic information of the segmentation results with the distance information between point clouds to generate higher-quality grasp seed points.

[0006] A multimodal feature fusion method for simultaneous segmentation and detection of grasping posture includes the following steps:

[0007] Step 1: Use a two-stream backbone network to extract point cloud and image features respectively;

[0008] Specifically, given a frame of aligned RGB-D image, the depth map is first converted into a 3D point cloud using camera intrinsic parameters, and then a two-stream backbone network is used to extract geometric features and color features from the point cloud stream and image stream respectively;

[0009] The point cloud stream uses the PointNet++ backbone network to extract point-by-point geometric features from the 3D point cloud image. The PointNet++ backbone network consists of four aggregate abstraction (SA) layers and four feature propagation (FP) layers. The SA layers downsample the point cloud and extract local features, while the feature propagation (FP) layers upsample the point features to restore the point cloud to its original size. The point cloud stream inputs the network with an N×3 point cloud data size. After feature extraction by the PointNet++ backbone network, the network outputs an N×512 point cloud data size.

[0010] The image stream uses a ResNet residual network (ResNet) to extract pixel-level color features from RGB images. The ResNet consists of four convolutional and deconvolutional layers. Each convolutional layer reduces the input feature map by half to expand the receptive field and extract image features. The deconvolutional layer restores the high-level image feature map to the same size (H×W) as the original image. The image data input to the image stream is H×W×3. After feature extraction by the residual network, the network outputs data of H×W×128.

[0011] After the feature extraction of the above two-stream backbone network, the point-by-point geometric features and pixel-by-pixel image features are obtained, respectively. point and F image express;

[0012] Step 2: Design a collaborative guidance point cloud-image feature fusion module to perform point cloud-image feature fusion;

[0013] The specific design steps of the collaboratively guided point cloud-image feature fusion module are as follows:

[0014] Step S1: Design two parallel fully connected layers (FC) to compress the number of channels of point cloud and image features respectively;

[0015] Step S2: Perform feature map splicing operation to splice the point cloud with compressed channel number and image features;

[0016] Step S3: After the splicing operation, a fully connected layer is used to fuse the spliced ​​features again to form the basis for collaborative guidance;

[0017] Step S4: A max pooling layer and a sigmoid activation function are used to normalize the fused features to obtain attention scores representing the importance of point cloud and image features.

[0018] Step S5: After obtaining the attention scores, use the attention scores to analyze the original features, that is, the features F output by the two-stream backbone network. point and F image Perform multiplication and weighting operations to suppress features in areas of non-interest and highlight important features;

[0019] Step S6: Re-join the weighted feature pairs and fuse them again through a new fully connected layer to obtain the final fusion feature F fusion ;

[0020] The input of the point cloud-image feature fusion module is the point cloud feature F point and image features F image , their sizes are N×512 and N×128 respectively. After the point cloud-image feature fusion module, the output fusion feature size F fusion It is N×512.

[0021] The specific formula for point cloud-image feature fusion is:

[0022]

[0023] Where AttenS is the attention score representing the importance of each feature pair, F fusion Represents the final fused features, σ represents the Sigmoid activation function, Max() represents the maximum pooling operation, MLP() represents the fully connected layer, represents the feature concatenation operation commonly used in neural networks, Represents a multiplication operation.

[0024] Step 3: Design an efficient seed point sampling algorithm to generate seed points:

[0025] Step 3.1: Perform foreground-background segmentation; fusion feature F fusion N×512 is used as input, and MLP outputs a classification score of N×2, where 2 represents the probability of the point belonging to the foreground and background. When the foreground probability is large, the point is considered to be a foreground point, and Focal Loss is used as the loss function for the task;

[0026] Step 3.2: Design a sampling algorithm that considers distance, semantics, and importance, namely, a seed point sampling algorithm, to obtain seed points;

[0027] Step 3.2.1: Use the farthest point sampling algorithm FPS to downsample the points in the foreground area; assuming that K foreground points are obtained by segmentation, the algorithm uses farthest point sampling to generate λn candidate points from the K foreground points;

[0028] Step 3.2.2: Use the attention score AttenS of the point cloud-image feature fusion module in step 2 to sort the candidate points; λn candidate points are obtained in the previous step, and then the λn candidate points are sorted according to the attention score AttenS.

[0029] Step 3.2.3: Select the top n points with the highest attention scores among the candidate points as seed points;

[0030] Step 4: Predict the 6-DOF grasping pose based on the seed points generated in step 3;

[0031] Use the Approach network and Operation network to predict the corresponding approach direction, plane rotation angle, grasp width, depth, and grasp pose score to obtain the final 6-DOF grasp pose;

[0032] The beneficial effects of adopting the above technical solution are:

[0033] The present invention provides a multimodal feature fusion method for simultaneous segmentation and detection of grasping posture. In view of the problem that the previous 6D grasping detection algorithm ignores image information, the performance of the grasping posture detection algorithm is improved. A fusion module with collaborative guidance of point cloud image is designed. The module can not only fuse point cloud and image features, but also highlight the key feature pairs and suppress features of non-interest areas. The prediction process of the algorithm starts with foreground point segmentation of the fused features. Then we design a seed point sampling algorithm that simultaneously considers semantic information, distance information, and grasping confidence to generate higher quality seed points. The grasping posture is then predicted based on the selected seed points. The present invention can make full use of point cloud and image information, and promote the performance of segmentation and grasping detection through multimodal features to serve application scenarios such as service robots and industrial robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1This is a network flow chart in an embodiment of the present invention;

[0035] Figure 2 This is a diagram of a fusion module for collaborative guidance of point cloud images in an embodiment of the present invention;

[0036] Figure 3 This is the pseudo code of the seed point sampling algorithm in the embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0038] Figure 1 This is the main flow chart of the technical solution of the present invention. The prediction process of the algorithm starts with the segmentation of the foreground points of the fused features, and then the grasping pose is predicted based on the selected seed points. The algorithm we proposed consists of a feature fusion module guided by the collaborative work of point cloud and image, an efficient seed point sampling algorithm and a local feature aggregation module. Figure 1 As shown in FIG, the present invention proposes a multimodal feature fusion method for simultaneously segmenting and detecting grasping posture, comprising the following steps:

[0039] Step 1: Use a two-stream backbone network to extract point cloud and image features respectively;

[0040] Specifically, given a frame of aligned RGB-D image, the depth map is first converted into a 3D point cloud using camera intrinsic parameters, and then a two-stream backbone network is used to extract geometric features and color features from the point cloud stream and image stream respectively;

[0041] The point cloud stream uses the PointNet++ backbone network to extract point-by-point geometric features from the 3D point cloud image. This example uses a segmented version of PointNet++. The PointNet++ backbone network includes four aggregate abstraction (SA) layers and four feature propagation (FP) layers. The SA layers downsample the point cloud and extract local features, while the feature propagation (FP) layers upsample the point features to restore the point cloud to its original size. The point cloud stream inputs the network with an N×3 point cloud data size. After feature extraction by the PointNet++ backbone network, the network outputs an N×512 point cloud data size.

[0042] The image stream uses the residual network ResNet to extract pixel-level color features in the RGB image. The residual network ResNet includes four convolutional layers and deconvolution layers. Each convolutional layer reduces the feature map input to the layer by half to expand the receptive field and extract image features. The deconvolution layer is used to restore the size of the high-level image feature map to the same size H×W as the original image. The image data of the image stream input network is H×W×3. After feature extraction by the residual network, the network output data is H×W×128.

[0043] After the feature extraction of the above two-stream backbone network, the point-by-point geometric features and pixel-by-pixel image features are obtained, respectively. point and F image express;

[0044] Step 2: Design a collaborative guidance point cloud-image feature fusion module to perform point cloud-image feature fusion;

[0045] Point clouds and images have different data formats. A simple and direct fusion method is to concatenate the XYZ channels with the RGB channels and then extract features from the concatenated data. However, this fusion method has gradually been replaced by other better fusion strategies, such as DenseFusion and FFB6D.

[0046] Achieving efficient fusion is difficult because different points have different importance. Points that are not in the region of interest may interfere with the prediction results. On the contrary, points with high grasp confidence will improve the accuracy of the prediction. The collaborative guided feature fusion module we designed is as follows: Figure 2 As shown, it can capture the key information of each data modality.

[0047] The specific design steps of the collaboratively guided point cloud-image feature fusion module are as follows:

[0048] Step S1: Design two parallel fully connected layers to compress the number of channels of point cloud and image features respectively;

[0049] Step S2: Perform feature map splicing operation to splice the point cloud with compressed channel number and image features;

[0050] Step S3: After the splicing operation, a fully connected layer is used to fuse the spliced ​​features again to form the basis for collaborative guidance;

[0051] Step S4: A maximum pooling layer and a sigmoid activation function are used to normalize the fused features to obtain the attention scores representing the importance of the feature pairs.

[0052] Step S5: After obtaining the attention scores, use the attention scores to analyze the original features, that is, the features F output by the two-stream backbone network. point and F image Perform multiplication and weighting operations to suppress features in areas of non-interest and highlight important features;

[0053] Step S6: Re-join the weighted feature pairs and fuse them again through a new fully connected layer to obtain the final fusion feature F fusion ;

[0054] The input of the point cloud-image feature fusion module is the point cloud feature F point and image features F image , their sizes are N×512 and N×128 respectively. After the point cloud-image feature fusion module, the output fusion feature size F fusion It is N×512.

[0055] The attention mechanism is used to suppress interference and highlight important feature pairs; for each geometric feature and its color feature, a fully connected layer is used to compress their channels and connect them together. Then we use a fully connected layer to fuse the spliced ​​information to form the basis for collaborative guidance of point cloud and image. To obtain the attention score, we use maximum pooling and sigmoid function to normalize the feature pairs. Finally, we use the attention score to reweight the original point cloud-image feature pairs and splice them together again to obtain the final fused features. The specific formula for point cloud-image feature fusion is:

[0056]

[0057] Where AttenS is the attention score representing the importance of each feature pair, F fusion Represents the final fused features, σ represents the Sigmoid activation function, Max() represents the maximum pooling operation, MLP() represents the fully connected layer, represents the feature concatenation operation commonly used in neural networks, Represents a multiplication operation.

[0058] Step 3: Design an efficient seed point sampling algorithm to generate high-quality seed points:

[0059] Step 3.1: Perform foreground-background segmentation; There are many unseen object categories in unstructured environments. Foreground point segmentation is insensitive to object categories and can help the robot obtain point-by-point semantic labels, including desktop (background) and object (foreground). Our segmentation modeling is implemented through a multi-layer perceptron (MLP) network. Specifically, the algorithm fuses the feature F fusionN×512 is used as input, MLP outputs classification scores N×2, and Focal Loss is used as the loss function for the task;

[0060] Farthest Point Sampling (FPS) or uniform sampling are usually used to generate seed points that cover the entire space as much as possible. However, these methods only utilize the distance information between points and do not fully utilize the semantic information and attribute characteristics of the points such as importance. Therefore, we propose a simple but effective sampling algorithm that considers distance, semantics and importance at the same time. Figure 3 shown.

[0061] Step 3.2: Design a sampling algorithm that considers distance, semantics, and importance, namely, a seed point sampling algorithm, to obtain seed points;

[0062] Step 3.2.1: Use the farthest point sampling algorithm FPS to downsample the points in the foreground area; assuming that K foreground points are obtained by segmentation, the algorithm uses farthest point sampling to generate λn candidate points from the K foreground points, so that these candidate points can cover the entire foreground area.

[0063] Step 3.2.2: Use the attention score AttenS of the point cloud-image feature fusion module in step 2 to sort the candidate points; λn candidate points are obtained in the previous step, and then the λn candidate points are sorted according to the attention score AttenS.

[0064] Step 3.2.3: Select the top n points with the highest attention scores among the candidate points as seed points; therefore, we select seed points with better properties and consider multimodal information, and the features of these seed points are n×512.

[0065] Step 4: Predict the 6-DOF grasping pose based on the seed points generated in step 3;

[0066] For grasp pose prediction, we use the Approach Network and Operation Network proposed in the paper "Fang, Hao-Shu, et al. "Graspnet-1billion: A large-scale benchmark for general object grasping." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020." to predict the corresponding approach direction, plane rotation angle, grasp width, depth, and grasp pose score to obtain the final 6-DOF grasp pose;

[0067] Due to the efficient fusion of point cloud-image feature pairs and the ability to generate higher quality grasping seed points, this patent achieves superior performance compared to previous algorithms.

[0068] In order to verify that the algorithm can achieve good detection accuracy, the embodiment of the present invention was tested on the GraspNet dataset. The experimental equipment platform is NVIDIA RTX3090 GPU and Intel(R) Xeon(R) Platinum 8260C CPU@2.30GHz. seen , AP similar , AP novel In these scenarios, we achieved grasp detection accuracies of 67.73%, 57.58%, and 23.40%, respectively. These results outperform other grasp pose detection algorithms, demonstrating that our algorithm can effectively handle grasping scenarios on cluttered desktops. We conducted comparative experiments with the fusion algorithm and seed point sampling algorithm, respectively, against DenseFusion and FPS, achieving improvements of 1.29% and 1.47%, respectively. Furthermore, to demonstrate the algorithm's effectiveness in real-world grasping, the network was tested in real-world grasping scenarios. The grasp accuracy rates for single object scenes and cluttered scenes were 82% and 70.4%, respectively, meeting the operational requirements of practical systems.

[0069] The above description is merely a preferred embodiment of the present disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multimodal feature fusion method for simultaneous segmentation and detection of grasping posture, characterized in that: The following steps are involved: Step 1: Use a two-stream backbone network to extract point cloud and image features respectively; Step 2: Design a collaborative guidance point cloud-image feature fusion module to perform point cloud-image feature fusion; The input of the point cloud-image feature fusion module is the point cloud feature F point and image features F image , their sizes are N×512 and N×128 respectively. After the point cloud-image feature fusion module, the output fusion feature size F fusion N × 512; The specific design steps of the collaboratively guided point cloud-image feature fusion module are as follows: Step S1: Design two parallel fully connected layers (FCs) to compress the number of channels of point cloud and image features respectively; Step S2: Perform feature map splicing operation to splice the point cloud with compressed channel number and image features; Step S3: After the splicing operation, a fully connected layer is used to fuse the spliced ​​features again to form the basis for collaborative guidance; Step S4: A max pooling layer and a sigmoid activation function are used to normalize the fused features to obtain attention scores representing the importance of point cloud and image features. Step S5: After obtaining the attention scores, use the attention scores to analyze the original features, that is, the features F output by the two-stream backbone network. point and F image Perform multiplication and weighting operations to suppress features in areas of non-interest and highlight important features; Step S6: Re-join the weighted feature pairs and fuse them again through a new fully connected layer to obtain the final fusion feature F fusion ; The specific formula for point cloud-image feature fusion is: Among them, AttenS is the abbreviation of Attention Scores, which represents the attention score of each feature to its importance. fusion Represents the final fused features, σ represents the Sigmoid activation function, Max() represents the maximum pooling operation, MLP() represents the fully connected layer, represents the feature concatenation operation in the neural network, Represents a multiplication operation; Step 3: Design a seed point sampling algorithm to generate seed points; The step 3 specifically includes the following steps: Step 3.1: Perform foreground-background segmentation; fusion feature F fusion N×512 is used as input, and MLP outputs a classification score of N×2, where 2 represents the probability of the point belonging to the foreground and background. If the foreground probability is greater than the background probability, the point is considered to be a foreground point and belongs to the foreground area. Focal Loss is used as the loss function for the task; Step 3.2: Design a sampling algorithm that considers distance, semantics, and importance, namely, a seed point sampling algorithm, to obtain seed points; Step 3.2.1: Use the farthest point sampling algorithm FPS to downsample the points in the foreground area; assuming that K foreground points are obtained by segmentation, the algorithm uses farthest point sampling to generate λn candidate points from the K foreground points; Step 3.2.2: Use the attention score AttenS of the point cloud-image feature fusion module in step 2 to sort the candidate points; in the previous step, λn candidate points are obtained, and then the λn candidate points are sorted according to the attention score AttenS; Step 3.2.3: Select the top n points with the highest attention scores among the candidate points as seed points; Step 4: Predict the 6-DOF grasping pose based on the seed points generated in step 3.

2. A multimodal feature fusion method for simultaneous segmentation and detection of grasping posture according to claim 1, characterized in that: Specifically, step 1 is as follows: given a frame of aligned RGB-D image, the depth map is first converted into a 3D point cloud map using camera intrinsic parameters, and then a two-stream backbone network is used, i.e., the point cloud stream and the image stream are used to extract geometric features and color features respectively.

3. A multimodal feature fusion method for simultaneous segmentation and detection of grasping posture according to claim 2, characterized in that: The point cloud stream uses the PointNet++ backbone network to extract point-by-point geometric features in the 3D point cloud image. The PointNet++ backbone network includes four aggregate abstraction SA layers and four feature propagation FP layers. The aggregate abstraction SA layers downsample the point cloud and extract local features, while the feature propagation FP layers upsample the point features and restore the point cloud to its original size. The point cloud stream inputs the network with point cloud data of N×3. After feature extraction by the PointNet++ backbone network, the network outputs data of N×512. The image stream uses the residual network ResNet to extract pixel-level color features from the RGB image. The residual network ResNet contains four convolutional layers and deconvolution layers. Each convolutional layer reduces the feature map input to the layer by half to expand the receptive field and extract image features. The deconvolution layer is used to restore the size of the high-level image feature map to the same size H×W as the original image. The image data input to the network of the image stream is H×W×3. After the feature extraction of the residual network, the output data of the network is H×W×128. After the feature extraction of the above two-stream backbone network, the point-by-point geometric features and pixel-by-pixel image features are obtained, respectively. point and F image express.

4. The multimodal feature fusion method for simultaneous segmentation and detection of grasping posture according to claim 1, characterized in that: Specifically, step 4 uses the Approach network and the Operation network to predict the corresponding approach direction, plane rotation angle, grasp width, depth, and grasp pose score to obtain the final 6-DOF grasp pose.

Images