A 6D pose estimation method for robot grasping tasks

Abstract

The present invention discloses a 6D pose estimation method applied to robot grasping tasks, which belongs to the technical field of 6D pose estimation, and includes the following steps: obtaining an RGB image and a depth image of a scene through a depth camera, obtaining a target mask through the RGB image, and obtaining sampling points from the target mask; constructing a 6D pose estimation network model; defining a loss function for 6D pose estimation, optimizing the loss function through back propagation, and training the 6D pose estimation network model according to the optimized loss function; inputting the RGB image, the depth image and the sampling points into the trained 6D pose estimation network model, obtaining the 6D pose estimation result of each sampling point regression and the confidence of each sampling point; taking the 6D pose estimation result of the sampling point with the highest confidence as the final prediction result, and completing the 6D pose estimation. The present invention solves the problem of unbalanced feature fusion strategy in the existing 6D pose estimation method based on RGB-D data.

Description

Technical Field

[0001] The present invention belongs to the technical field of 6D pose estimation, and in particular relates to a 6D pose estimation method applied to robot grasping tasks. Background Art

[0002] 6D pose estimation is a technology for estimating the position and attitude of a target. In robot grasping tasks based on 6D pose estimation, 6D pose estimation can help robots accurately perceive the position and attitude of objects, thereby achieving precise grasping, which is crucial for robots to reliably grasp objects in complex environments. However, there are still many difficulties and challenges in the current 6D pose estimation: complex environmental conditions in actual production, such as occlusion, sensor noise, the influence of complex lighting, and the challenges of weak texture and symmetry of objects, the robustness and accuracy of 6D pose estimation are not high. Therefore, how to improve the robustness and accuracy of the target's 6D pose estimation and provide the robot with an accurate 6D pose of the target to achieve reliable and safe grasping will promote the development of robotics technology and promote the progress of intelligent manufacturing. The robustness and accuracy of 6D pose estimation provide the robot with an accurate 6D pose of the target to achieve reliable and safe grasping.

[0003] Existing 6D pose estimation methods are divided into methods based on RGB data and methods based on RGB-D data. Due to the complementarity between RGB information and depth information, the methods based on RGB-D data are more robust and accurate, but these two types of information are heterogeneous features and require appropriate feature fusion methods to fuse them. In existing methods based on RGB-D data, the feature fusion method is not effective enough and the network tends to tilt toward some objects in the dataset. Summary of the invention

[0004] The present invention provides a 6D pose estimation method applied to robot grasping tasks, which solves the problem of unbalanced feature fusion strategy in the existing 6D pose estimation method based on RGB-D data.

[0005] In order to solve the above technical problems, the technical solution of the present invention is: a 6D pose estimation method applied to robot grasping tasks, comprising the following steps:

[0006] S1. Obtain an RGB image and a depth image of the scene through a depth camera, obtain a target mask through the RGB image, and obtain sampling points from the target mask;

[0007] S2, build a 6D pose estimation network model;

[0008] S3. Define the loss function of 6D pose estimation, optimize the loss function through back propagation, and train the 6D pose estimation network model according to the optimized loss function;

[0009] S4, input the RGB image, the depth image and the sampling points into the trained 6D pose estimation network model to obtain the 6D pose estimation result of each sampling point regression and the confidence of each sampling point;

[0010] S5. The 6D pose estimation result of the sampling point regression with the highest confidence is taken as the final prediction result to complete the 6D pose estimation.

[0011] The beneficial effects of the present invention are as follows: the present invention performs channel fusion through the strategy of dimension increase followed by dimension reduction to solve the problem of feature imbalance obtained by the existing channel fusion strategy of continuous dimension increase. The channel fusion strategy realizes balanced feature expression, and at the same time restores the information lost in the dimension reduction process through spatial fusion to resist occlusion interference. In addition, feature normalization is used in the geometric feature extractor to improve the distribution of geometric features, solve the problem that the shared multi-layer perceptron cannot effectively capture the relationship between inputs, and enhance the network's local perception ability of point clouds, thereby improving the accuracy of pose estimation and the robustness to occlusion, and solving the problem of feature fusion strategy imbalance in the existing 6D pose estimation method based on RGB-D data.

[0012] Furthermore, the 6D pose estimation network model in step S2 includes an RGB pixel-level feature extractor, a geometric feature extractor, an RGB-D channel-space fusion module and a decoder;

[0013] The RGB pixel-level feature extractor is connected to the RGB-D channel-space fusion module and the decoder respectively; the geometric feature extractor is connected to the RGB-D channel-space fusion module and the decoder respectively; the RGB-D channel-space fusion module is connected to the decoder.

[0014] Furthermore, the RGB pixel-level feature extractor includes a backbone network ResNet18, a pyramid scene analysis network, a first shared perceptron, a second shared perceptron, and a first LayerNorm normalization layer connected in sequence;

[0015] The geometric feature extractor includes a third shared perceptron, a second LayerNorm normalization layer, a fourth shared perceptron, a third LayerNorm normalization layer, a fifth shared perceptron and a fourth LayerNorm normalization layer connected in sequence;

[0016] The RGB-D channel-space fusion module includes a channel fusion submodule and a space fusion submodule; the channel fusion submodule is connected to the space fusion submodule; the channel fusion submodule includes a fifth shared perceptron, a sixth shared perceptron and a channel attention mechanism connected in sequence; the space fusion submodule includes a seventh shared perceptron;

[0017] The decoder includes a first branch, a second branch and a third branch; the first branch, the second branch and the third branch each have four shared perceptrons connected in sequence.

[0018] Furthermore, the loss function expression of 6D pose estimation in step S3 is:

[0019]

[0020]

[0021] Among them, L represents the loss function of 6D pose estimation, N represents the number of sampling points, and c k represents the confidence of the kth sampling point, L ADD(-S),k represents the pose loss of the kth sampling point regression, w represents the penalty coefficient, M represents the number of vertices randomly sampled on the object CAD model, R k The rotation matrix representing the pose of the kth sampling point regression, t k represents the translation vector of the k-th sampling point regression pose, R represents the rotation matrix of the real pose, t represents the translation vector of the real pose, p i represents the i-th vertex, ||·|| represents the norm, p j represents the j-th vertex, and min(·) represents the minimum value.

[0022] Furthermore, the specific steps of step S4 are:

[0023] S41, extracting pixel-level features from the RGB image and the sampling points through an RGB pixel-level feature extractor to obtain RGB shared features and RGB pixel-level features;

[0024] S42, extracting geometric features from the depth image and the sampling points through a geometric feature extractor to obtain a first geometric feature, a second geometric feature, and a third geometric feature;

[0025] S43, after performing a feature splicing operation on the RGB pixel-level feature and the second geometric feature, the feature is input into an RGB-D channel-space fusion module for feature fusion to obtain a channel-space fusion feature;

[0026] S44, after performing feature splicing operation on RGB shared features, RGB pixel-level features, first geometric features, second geometric features, third geometric features and channel-space fusion features, input them into the decoder for decoding to obtain the 6D pose estimation result of each sampling point regression and the confidence of each sampling point.

[0027] Furthermore, the specific steps of step S41 are:

[0028] S411, input the RGB image into the backbone network ResNet18 to perform preliminary feature extraction to obtain preliminary features;

[0029] S412, the preliminary features are passed through a pyramid scene analysis network to obtain an RGB feature map;

[0030] S413, obtaining RGB features corresponding to the RGB feature map according to the sampling points;

[0031] S414, inputting the RGB features into the first shared perceptron in sequence to perform feature extraction to obtain RGB shared features;

[0032] S415, inputting the RGB shared features into the second shared perceptron and the first LayerNorm normalization layer in sequence, performing RGB pixel-level feature extraction, and obtaining RGB pixel-level features.

[0033] The beneficial effects of the above further scheme are: the LayerNorm normalization layer adjusts the RGB pixel-level feature distribution to prepare for subsequent feature fusion.

[0034] Furthermore, the specific steps of step S42 are:

[0035] S421, converting the depth image into a point cloud, and obtaining a point cloud corresponding to the sampling point;

[0036] S422, inputting the point cloud corresponding to the sampling point into the third shared perceptron and the second LayerNorm normalization layer in sequence to extract geometric features, and obtaining a first geometric feature;

[0037] S423, inputting the first geometric feature into the fourth shared perceptron and the third LayerNorm normalization layer in sequence to extract the geometric feature, so as to obtain the second geometric feature;

[0038] S424: Input the second geometric feature into the fifth shared perceptron and the fourth LayerNorm normalization layer in sequence to extract the geometric feature, so as to obtain the third geometric feature.

[0039] The beneficial effects of the above further scheme are as follows: the LayerNorm normalization layer enhances the local feature extraction capability of the geometric feature extractor and improves the occlusion robustness of the network. At the same time, the third geometric feature obtained supplements the small object feature and improves the accuracy of small object pose estimation.

[0040] Furthermore, the specific steps of step S43 are:

[0041] S431, performing a feature splicing operation on the RGB pixel-level feature and the second geometric feature to obtain a splicing feature;

[0042] S432, inputting the concatenated features into the fifth shared perceptron and the sixth shared perceptron in sequence for feature extraction to obtain first channel features;

[0043] S433, passing the first channel feature through the channel attention mechanism to obtain a channel attention feature;

[0044] S434, multiplying the channel attention feature by the first channel feature to obtain a second channel feature;

[0045] S435, performing a transposition operation on the second channel feature to obtain a first spatial feature;

[0046] S436, extracting the first spatial feature through the seventh shared perceptron to obtain a second spatial feature;

[0047] S437. Perform a transposition operation on the second spatial feature to obtain a channel-space fusion feature.

[0048] The beneficial effect of the above further scheme is: channel fusion is performed by first increasing the dimension and then reducing the dimension to balance the feature expression, and at the same time, the information lost in the dimensionality reduction process is restored through spatial fusion to resist occlusion interference.

[0049] Furthermore, the specific steps of step S44 are:

[0050] S441, performing a feature splicing operation on the RGB shared feature, the RGB pixel-level feature, the first geometric feature, the second geometric feature, the third geometric feature, and the channel-space fusion feature to obtain a decoder input feature;

[0051] S442, input the decoder input features to the first branch, the second branch and the third branch of the decoder respectively, and obtain the 6D pose estimation result of each sampling point regression and the confidence of each sampling point. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention is a flow chart of a 6D pose estimation method for robot grasping tasks.

[0053] Figure 2 It is a schematic diagram of the structure of the 6D pose estimation network model of the present invention.

[0054] Figure 3 It is a schematic diagram of the structure of the RGB pixel-level feature extractor and the geometric feature extractor of the present invention.

[0055] Figure 4 It is a structural schematic diagram of the RGB-D channel-space fusion module of the present invention.

[0056] Figure 5 It is a schematic diagram of the structure of the decoder of the present invention.

[0057] Figure 6 This is a visualization comparison chart of the pose estimation results of the method of the present invention and DenseFusion. DETAILED DESCRIPTION

[0058] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

[0059] Example 1

[0060] like Figure 1 As shown, the present invention provides a 6D pose estimation method applied to robot grasping tasks, comprising the following steps:

[0061] S1. Obtain an RGB image and a depth image of the scene through a depth camera, obtain a target mask through the RGB image, and obtain sampling points from the target mask;

[0062] S2, build a 6D pose estimation network model;

[0063] S3. Define the loss function of 6D pose estimation, optimize the loss function through back propagation, and train the 6D pose estimation network model according to the optimized loss function;

[0064] S4, input the RGB image, the depth image and the sampling points into the trained 6D pose estimation network model to obtain the 6D pose estimation result of each sampling point regression and the confidence of each sampling point;

[0065] S5. The 6D pose estimation result of the sampling point regression with the highest confidence is taken as the final prediction result to complete the 6D pose estimation.

[0066] The 6D pose estimation network model in step S2 includes an RGB pixel-level feature extractor, a geometric feature extractor, an RGB-D channel-space fusion module, and a decoder;

[0067] The RGB pixel-level feature extractor is connected to the RGB-D channel-space fusion module and the decoder respectively; the geometric feature extractor is connected to the RGB-D channel-space fusion module and the decoder respectively; the RGB-D channel-space fusion module is connected to the decoder.

[0068] The RGB pixel-level feature extractor includes a backbone network ResNet18, a pyramid scene analysis network, a first shared perceptron, a second shared perceptron, and a first LayerNorm normalization layer connected in sequence;

[0069] The geometric feature extractor includes a third shared perceptron, a second LayerNorm normalization layer, a fourth shared perceptron, a third LayerNorm normalization layer, a fifth shared perceptron and a fourth LayerNorm normalization layer connected in sequence;

[0070] The RGB-D channel-space fusion module includes a channel fusion submodule and a space fusion submodule; the channel fusion submodule is connected to the space fusion submodule; the channel fusion submodule includes a fifth shared perceptron, a sixth shared perceptron and a channel attention mechanism connected in sequence; the space fusion submodule includes a seventh shared perceptron;

[0071] The decoder includes a first branch, a second branch and a third branch; the first branch, the second branch and the third branch each have four shared perceptrons connected in sequence.

[0072] like Figure 2 As shown in the figure, the RGB image and depth image of the scene are obtained through the depth camera, and the resolution is 480×640. During the test, the RGB image is semantically segmented to obtain the target mask mask, the bounding box bbox (i.e., the object bounding box) corresponding to the mask, and the target category class, and random sampling points are obtained from the mask. During training, random sampling points are obtained from the mask label, and the number of sampling points is 500.

[0073] The loss function expression of 6D pose estimation in step S3 is:

[0074]

[0075]

[0076] Among them, L represents the loss function of 6D pose estimation, N represents the number of sampling points, and c k represents the confidence of the kth sampling point, L ADD(-S),k represents the pose loss of the kth sampling point regression, w represents the penalty coefficient, M represents the number of vertices randomly sampled on the object CAD model, R k The rotation matrix representing the pose of the kth sampling point regression, t k represents the translation vector of the k-th sampling point regression pose, R represents the rotation matrix of the real pose, t represents the translation vector of the real pose, p i represents the i-th vertex, ||·|| represents the norm, p jrepresents the j-th vertex, and min(·) represents the minimum value.

[0077] In this embodiment 1, R and t together constitute the real pose, which respectively describe the pose (orientation) and position of the object in the camera coordinate system. Meanwhile, the vertex is a point on the CAD model of the object, and each vertex has a coordinate represented by a three-dimensional vector (x, y, z).

[0078] The specific steps of step S4 are:

[0079] S41, extracting pixel-level features from the RGB image and the sampling points through an RGB pixel-level feature extractor to obtain RGB shared features and RGB pixel-level features;

[0080] S42, extracting geometric features from the depth image and the sampling points through a geometric feature extractor to obtain a first geometric feature, a second geometric feature, and a third geometric feature;

[0081] S43, after performing a feature splicing operation on the RGB pixel-level feature and the second geometric feature, the feature is input into an RGB-D channel-space fusion module for feature fusion to obtain a channel-space fusion feature;

[0082] S44, after performing feature splicing operation on RGB shared features, RGB pixel-level features, first geometric features, second geometric features, third geometric features and channel-space fusion features, input them into the decoder for decoding to obtain the 6D pose estimation result of each sampling point regression and the confidence of each sampling point.

[0083] like Figure 3 As shown in (a), the specific steps of step S41 are:

[0084] S411, input the RGB image into the backbone network ResNet18 to perform preliminary feature extraction to obtain preliminary features;

[0085] S412, the preliminary features are passed through a pyramid scene analysis network to obtain an RGB feature map;

[0086] S413, obtaining RGB features corresponding to the RGB feature map according to the sampling points;

[0087] S414, inputting the RGB features into the first shared perceptron in sequence to perform feature extraction to obtain RGB shared features;

[0088] S415, inputting the RGB shared features into the second shared perceptron and the first LayerNorm normalization layer in sequence, performing RGB pixel-level feature extraction, and obtaining RGB pixel-level features.

[0089] Figure 3In it, PSPNet represents the pyramid scene analysis network, Shared-MLP represents the shared perceptron, and LayerNorm represents the LayerNorm normalization layer.

[0090] In this embodiment 1, the RGB image The preliminary features are obtained by extracting the preliminary features in the backbone network ResNet18. The pyramid scene analysis network consists of four adaptive average pooling layers with scales of 1×1, 2×2, 3×3, and 6×6 and one 1×1 convolution layer. The preliminary features are compressed by the convolution layer after four poolings, and then upsampled and stacked to obtain the RGB feature map. Then, the RGB features corresponding to the RGB feature map are obtained according to the sampling points.

[0091] The convolution kernels of the first shared perceptron and the second shared perceptron are 64 and 128 respectively. Therefore, the RGB feature Get RGB shared features through the first shared perceptron RGB shared features After passing through the second shared perceptron and the first LayerNorm normalization layer, RGB pixel-level features are obtained.

[0092] like Figure 3 As shown in (b), the specific steps of step S42 are:

[0093] S421, converting the depth image into a point cloud, and obtaining a point cloud corresponding to the sampling point;

[0094] S422, inputting the point cloud corresponding to the sampling point into the third shared perceptron and the second LayerNorm normalization layer in sequence to extract geometric features, and obtaining a first geometric feature;

[0095] S423, inputting the first geometric feature into the fourth shared perceptron and the third LayerNorm normalization layer in sequence to extract the geometric feature, so as to obtain the second geometric feature;

[0096] S424: Input the second geometric feature into the fifth shared perceptron and the fourth LayerNorm normalization layer in sequence to extract the geometric feature, so as to obtain the third geometric feature.

[0097] In this embodiment 1, the convolution kernel numbers of the third shared perceptron, the fourth shared perceptron, and the fifth shared perceptron are 64, 128, and 256, respectively. Input into the geometric feature extractor to obtain the first geometric features Second geometric feature and the third geometric feature

[0098] like Figure 4 As shown, the specific steps of step S43 are:

[0099] S431, performing a feature splicing operation on the RGB pixel-level feature and the second geometric feature to obtain a splicing feature;

[0100] S432, inputting the concatenated features into the fifth shared perceptron and the sixth shared perceptron in sequence for feature extraction to obtain first channel features;

[0101] S433, passing the first channel feature through the channel attention mechanism to obtain a channel attention feature;

[0102] S434, multiplying the channel attention feature by the first channel feature to obtain a second channel feature;

[0103] S435, performing a transposition operation on the second channel feature to obtain a first spatial feature;

[0104] S436, extracting the first spatial feature through the seventh shared perceptron to obtain a second spatial feature;

[0105] S437. Perform a transposition operation on the second spatial feature to obtain a channel-space fusion feature.

[0106] In this embodiment 1, the convolution kernels of the fifth shared perceptron, the sixth shared perceptron, and the seventh shared perceptron are 512, 256, and 256, respectively. The channel attention mechanism consists of a pooling operation, a convolution with a convolution kernel of 128, a ReLU activation function, a convolution with a convolution kernel of 256, a Sigmoid activation function, and a dot product operation.

[0107] like Figure 5 As shown, the specific steps of step S44 are:

[0108] S441, performing a feature splicing operation on the RGB shared feature, the RGB pixel-level feature, the first geometric feature, the second geometric feature, the third geometric feature, and the channel-space fusion feature to obtain a decoder input feature;

[0109] S442, input the decoder input features to the first branch, the second branch and the third branch of the decoder respectively, and obtain the 6D pose estimation result of each sampling point regression and the confidence of each sampling point.

[0110] In this embodiment 1, the number of convolution kernels of the four-layer shared perceptron of the first branch of the decoder is 384, 256, 128 and 4 respectively, the number of convolution kernels of the four-layer shared perceptron of the second branch is 384, 256, 128 and 3 respectively, and the number of convolution kernels of the four-layer shared perceptron of the third branch is 384, 256, 128 and 1 respectively. The first branch, the second branch and the third branch respectively output the rotation matrix, the translation vector and the confidence of the 6D pose regressed by each sampling point, and the one with the highest confidence c is selected. m The 6D pose estimation result R of the sampling point regression at m and t m as the final prediction result.

[0111] Example 2

[0112] In order to verify the effectiveness of the method of the present invention, this Example 2 conducted experiments on the LineMOD dataset, using 13 weakly textured objects for the experiment, of which the 8th and 9th objects were symmetrical objects, each of which contained more than 1,200 RGB-D images with a resolution of 480×640, and each object had about 130 training images and about 1,100 test images. In addition, the Occlusion LineMOD dataset was used to test the robustness of network occlusion.

[0113] In this embodiment 2, the average distance error ADD is used to evaluate the estimated 6D pose.

[0114] For asymmetric objects, the actual pose T and the predicted pose Convert M vertices into the real point cloud and predicted point cloud in the camera coordinate system Calculate the real point cloud and predicted point cloud The average distance between corresponding points err ADD , the calculation formula is:

[0115]

[0116] in, The rotation matrix of the predicted pose, The translation vector representing the predicted pose.

[0117] For symmetrical objects, calculate the real point cloud and the predicted point cloud The average distance between the closest points err ADD-S , the calculation formula is:

[0118]

[0119] The final statistical test sample satisfies err ADDThe percentage of <0.1d is used as the evaluation index, where d represents the model radius, that is, the maximum distance between two points on the object, which is used to indicate the size of the object.

[0120] In this embodiment 2, the effectiveness of each module is verified. As shown in Table 1, the ADD(-S) index performance of each improvement on the LineMOD dataset is shown in Table 2. The ADD(-S) index performance of each improvement on the Occlusion LineMOD dataset is shown in Table 2. Among them, the ADD(-S) index indicates that the err ADD The percentage <0.1d is the success rate.

[0121] Table 1

[0122]

[0123]

[0124] Table 2

[0125]

[0126] In Tables 1 and 2, DF represents the existing pose estimation network DenseFusion, LN represents only the first two levels of geometric features, CF represents the channel fusion submodule, SF represents the spatial fusion submodule, and LN3 represents the addition of the third level of geometric features.

[0127] Tables 1 and 2 respectively show the performance of various improvements on the LineMOD dataset and the Occlusion LineMOD dataset. It can be seen that the various improvements of the present invention gradually improve the performance.

[0128] In the geometric feature extractor, the present invention introduces a LayerNorm layer for channel normalization, which scales the eigenvalues ​​according to the mean and variance of the features, and amplifies the differences in eigenvalues ​​between regions. After this feature scaling process, the geometric features show obvious regional changes, and the eigenvalues ​​of some areas on the object become higher, and some areas become lower. The geometric feature extractor can better perceive the differences between different areas of the object, and the local perception ability of the geometric feature extractor is enhanced, and it can perceive specific information. According to the DF+LN column in Table 2, it can be seen that the pose estimation accuracy of 8 objects has been improved to varying degrees, indicating that after the improvement improves the local perception ability of the geometric feature extractor, even if some areas are blocked, the captured local features can also perform pose estimation better.

[0129] Secondly, in order to improve the disadvantage of imbalanced expression of high-dimensional fusion features, we use the method of first increasing the dimension and then reducing the dimension to perform feature fusion on the basis of DF+LN. In addition, before feature fusion, in order to adjust the RGB features to a distribution suitable for feature fusion, the LayerNorm layer is also used to normalize the channels of the RGB features. From the DF+LN+CF column in Table 1, it can be seen that on the LineMOD dataset, except for object 10, the performance of each object is relatively balanced, indicating that the channel fusion submodule can effectively solve the problem of feature imbalance caused by single dimensionality increase.

[0130] However, it can be seen from Table 2 that the pose estimation accuracy of objects 6, 8, 9, and 10 decreases in the Occlusion LineMOD dataset. This is because in the case of occlusion, the features after dimensionality reduction lack necessary information.

[0131] In order to make up for the missing information in the dimensionality reduction process, based on DF+LN+CF, we add a spatial fusion submodule SF after the channel fusion submodule to perform one-dimensional convolution in the spatial dimension. This convolution process enables the features to be fully connected in the spatial dimension, that is, the sampling points interact with each other, and in the case of occlusion, the missing information is restored from other sampling points to resist the interference caused by occlusion. As can be seen from Table 2, on the Occlusion LineMOD dataset, the pose estimation accuracy of the eight objects has been improved to varying degrees.

[0132] For small objects, the features contained in the RGB image are often not obvious enough. Therefore, in order to improve the accuracy of pose estimation of small objects, a layer of shared perceptron is added to the geometric feature extractor to enhance the geometric features and supplement the feature information of small objects. As can be seen from Table 1, after enhancing the geometric features, the accuracy of these three objects is improved by 3.7%, 7.6% and 2.7% respectively.

[0133] This Example 2 also compares the method of the present invention with the existing methods. As shown in Table 3, it is a comparison of the ADD(-S) index performance of the method of the present invention and other networks on the LineMOD dataset.

[0134] Table 3

[0135]

[0136] Table 3 summarizes the performance of various methods on the LineMOD dataset and the years in which they were proposed. It can be seen that the present invention is superior to other methods.

[0137] Figure 6This is a visualization comparison chart of the pose estimation results of the method of the present invention and DenseFusion, where the 3D bounding box of the true pose is green. Each of the 13 objects in the LineMOD dataset has a pair of visualization results. The first row is the 3D bounding box of the 6D pose estimation result of DenseFusion (red), and the second row is the 3D bounding box of the 6D pose estimation result of the method of the present invention (red).

[0138] Figure 6 The pose estimation results all satisfy err ADD <0.1d threshold requirement, but by comparing the red 3D bounding box in the second row, it can be seen that the 3D bounding box of the 6D pose estimation result of the method of the present invention is closer to the 3D bounding box of the real pose in most cases than that of DenseFusion. It can be seen that the 6D pose estimation result predicted by the method of the present invention is more accurate.

[0139] This Example 2 also counts the performance of various different methods on the Occlusion LineMOD dataset and the years in which they were proposed, as shown in Table 4, which is a comparison of the ADD(-S) index performance of the method of the present invention and other networks on the Occlusion LineMOD dataset.

[0140] Table 4

[0141]

[0142] It can be seen from Table 4 that the occlusion robustness of the method of the present invention is better than that of other methods.

[0143] In summary, the present invention performs channel fusion through the strategy of dimension increase followed by dimension reduction to solve the problem of feature imbalance obtained by the existing channel fusion strategy of continuous dimension increase. The channel fusion strategy realizes balanced feature expression and restores the information lost in the dimension reduction process through spatial fusion to resist occlusion interference. In addition, feature normalization is used in the geometric feature extractor to improve the distribution of geometric features, solve the problem that the shared multi-layer perceptron cannot effectively capture the relationship between inputs, and enhance the network's local perception ability of point clouds, thereby improving the accuracy of pose estimation and robustness to occlusion, and solving the problem of feature fusion strategy imbalance in the existing 6D pose estimation method based on RGB-D data.

Claims

1. A 6D pose estimation method for robot grasping tasks, characterized in that: The following steps are involved: S1. Obtain an RGB image and a depth image of the scene through a depth camera, obtain a target mask through the RGB image, and obtain sampling points from the target mask; S2, build a 6D pose estimation network model; S3. Define the loss function of 6D pose estimation, optimize the loss function through back propagation, and train the 6D pose estimation network model according to the optimized loss function; S4, input the RGB image, the depth image and the sampling points into the trained 6D pose estimation network model to obtain the 6D pose estimation result of each sampling point regression and the confidence of each sampling point; S5, taking the 6D pose estimation result of the sampling point regression with the highest confidence as the final prediction result, and completing the 6D pose estimation; The 6D pose estimation network model in step S2 includes an RGB pixel-level feature extractor, a geometric feature extractor, an RGB-D channel-space fusion module, and a decoder; The RGB pixel-level feature extractor is connected to the RGB-D channel-space fusion module and the decoder respectively; the geometric feature extractor is connected to the RGB-D channel-space fusion module and the decoder respectively; the RGB-D channel-space fusion module is connected to the decoder; The RGB pixel-level feature extractor includes a backbone network ResNet18, a pyramid scene analysis network, a first shared perceptron, a second shared perceptron, and a first LayerNorm normalization layer connected in sequence; The geometric feature extractor includes a third shared perceptron, a second LayerNorm normalization layer, a fourth shared perceptron, a third LayerNorm normalization layer, a fifth shared perceptron and a fourth LayerNorm normalization layer connected in sequence; The RGB-D channel-space fusion module includes a channel fusion submodule and a space fusion submodule; the channel fusion submodule is connected to the space fusion submodule; the channel fusion submodule includes a fifth shared perceptron, a sixth shared perceptron and a channel attention mechanism connected in sequence; the space fusion submodule includes a seventh shared perceptron; The decoder includes a first branch, a second branch and a third branch; the first branch, the second branch and the third branch each have four shared perceptrons connected in sequence.

2. The 6D pose estimation method for robot grasping tasks according to claim 1, characterized in that: The loss function expression of 6D pose estimation in step S3 is: Among them, L represents the loss function of 6D pose estimation, N represents the number of sampling points, and c k represents the confidence of the kth sampling point, L ADD(-S),k represents the pose loss of the kth sampling point regression, w represents the penalty coefficient, M represents the number of vertices randomly sampled on the object CAD model, R k The rotation matrix representing the pose of the kth sampling point regression, t k represents the translation vector of the k-th sampling point regression pose, R represents the rotation matrix of the real pose, t represents the translation vector of the real pose, p i represents the i-th vertex, ||·|| represents the norm, p j represents the j-th vertex, and min(·) represents the minimum value.

3. The 6D pose estimation method for robot grasping tasks according to claim 1, characterized in that: The specific steps of step S4 are: S41, extracting pixel-level features from the RGB image and the sampling points through an RGB pixel-level feature extractor to obtain RGB shared features and RGB pixel-level features; S42, extracting geometric features from the depth image and the sampling points through a geometric feature extractor to obtain a first geometric feature, a second geometric feature, and a third geometric feature; S43, after performing a feature splicing operation on the RGB pixel-level feature and the second geometric feature, the feature is input into an RGB-D channel-space fusion module for feature fusion to obtain a channel-space fusion feature; S44, after performing feature splicing operation on RGB shared features, RGB pixel-level features, first geometric features, second geometric features, third geometric features and channel-space fusion features, input them into the decoder for decoding to obtain the 6D pose estimation result of each sampling point regression and the confidence of each sampling point.

4. The 6D pose estimation method for robot grasping tasks according to claim 3, characterized in that: The specific steps of step S41 are: S411, input the RGB image into the backbone network ResNet18 to perform preliminary feature extraction to obtain preliminary features; S412, the preliminary features are passed through a pyramid scene analysis network to obtain an RGB feature map; S413, obtaining RGB features corresponding to the RGB feature map according to the sampling points; S414, inputting the RGB features into the first shared perceptron in sequence to perform feature extraction to obtain RGB shared features; S415, inputting the RGB shared features into the second shared perceptron and the first LayerNorm normalization layer in sequence, performing RGB pixel-level feature extraction, and obtaining RGB pixel-level features.

5. The 6D pose estimation method for robot grasping tasks according to claim 3, characterized in that: The specific steps of step S42 are: S421, converting the depth image into a point cloud, and obtaining a point cloud corresponding to the sampling point; S422, inputting the point cloud corresponding to the sampling point into the third shared perceptron and the second LayerNorm normalization layer in sequence to extract geometric features, and obtaining a first geometric feature; S423, inputting the first geometric feature into the fourth shared perceptron and the third LayerNorm normalization layer in sequence to extract the geometric feature, so as to obtain the second geometric feature; S424: Input the second geometric feature into the fifth shared perceptron and the fourth LayerNorm normalization layer in sequence to extract the geometric feature, so as to obtain the third geometric feature.

6. The 6D pose estimation method for robot grasping tasks according to claim 3, characterized in that: The specific steps of step S43 are: S431, performing a feature splicing operation on the RGB pixel-level feature and the second geometric feature to obtain a splicing feature; S432, inputting the concatenated features into the fifth shared perceptron and the sixth shared perceptron in sequence for feature extraction to obtain first channel features; S433, passing the first channel feature through the channel attention mechanism to obtain a channel attention feature; S434, multiplying the channel attention feature by the first channel feature to obtain a second channel feature; S435, performing a transposition operation on the second channel feature to obtain a first spatial feature; S436, extracting the first spatial feature through the seventh shared perceptron to obtain a second spatial feature; S437. Perform a transposition operation on the second spatial feature to obtain a channel-space fusion feature.

7. The 6D pose estimation method for robot grasping tasks according to claim 3, characterized in that: The specific steps of step S44 are: S441, performing a feature splicing operation on the RGB shared feature, the RGB pixel-level feature, the first geometric feature, the second geometric feature, the third geometric feature, and the channel-space fusion feature to obtain a decoder input feature; S442, input the decoder input features to the first branch, the second branch and the third branch of the decoder respectively, and obtain the 6D pose estimation result of each sampling point regression and the confidence of each sampling point.

Images