Target pose estimation method based on multimodal feature fusion deep neural network
By constructing a deep neural network that fuses multimodal features, the problem of using only single-modal features in existing methods is solved, and pose estimation with higher accuracy is achieved.
Patent Information
- Application Number
- CN202410242373.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-04
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-03-04
AI Technical Summary
Existing pose estimation methods based on deep neural networks typically use only one modal feature, either image information or depth information, which fails to fully extract multimodal information, resulting in low pose recognition accuracy.
A multimodal feature fusion deep neural network is constructed. Features of RGB images and depth images are extracted by color feature extraction module and depth feature extraction module respectively, and multi-level fusion is performed by feature fusion module. Finally, regression prediction is performed using multi-layer perceptron.
It improves the accuracy and robustness of pose estimation, enabling accurate pose estimation under occlusion and environmental changes.
Smart Images

Figure CN118247346B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of target pose estimation, specifically relating to a method for estimating the pose of a target object by constructing a deep neural network that fuses multimodal features. Background Technology
[0002] With the development of intelligent manufacturing technology, more and more industrial robots are undertaking complex tasks such as sorting and loading / unloading in industrial production. These tasks rely heavily on accurate pose estimation of objects in randomly stacked scenarios. Six-DOF pose estimation refers to determining the position (including three dimensions of displacement) and orientation (including three dimensions of rotation) of an object in three-dimensional space. Efficient and accurate object pose estimation algorithms play a crucial role in industrial automation, such as robotic arm grasping, target detection and tracking, and automatic obstacle avoidance. The successful application of Convolutional Neural Networks (CNNs) deep learning in visual recognition has prompted research into how to utilize CNNs for deep learning to better acquire features from RGB-D images for six-DOF pose estimation. Currently, some existing deep neural network-based pose estimation methods only use image or depth information as a single modality for pose prediction, failing to learn the rich geometric and appearance features of objects. Other methods, while using both image and depth modal information, employ feature extraction methods that cannot fully extract local and global features from multimodal information, hindering the deep neural network's ability to learn multimodal features for pose prediction. For example, the DenseFusion pose estimation method proposed by Chen Wang et al. (DenseFusion: 6D Object Pose Estimation by Iterative DenseFusion, 2018, IEEE Conference on Computer Vision and Pattern Recognition) is one such method. This method uses PoseCNN to process RGB images to obtain color features and PointNet to process deep point clouds to obtain depth features. At the pixel level, the color features and corresponding depth features are concatenated to obtain fused features. Finally, a multilayer perceptron is used for regression to obtain the 6-DOF pose. However, the feature extraction method used by DenseFusion does not extract sufficiently rich multimodal features, which is not conducive to the network using multimodal features for pose prediction, and its pose recognition accuracy needs improvement. Summary of the Invention
[0003] To address the problems and needs existing in the background technology, this invention aims to propose a target pose estimation method based on a deep neural network with multimodal feature fusion. This invention can simultaneously learn features from color and depth information, enabling the deep learning network to more comprehensively utilize the rich appearance and geometric information of the object, resulting in advantages such as high pose estimation accuracy and strong robustness.
[0004] This invention provides a target pose estimation method based on a deep neural network with multimodal feature fusion, the technical solution of which is as follows:
[0005] S1: Obtain the target category and color and depth information belonging to the same target from the RGB and depth images captured by the camera. Convert the color and depth information of all pixels of the same target into the original point cloud data of the target and record the target category information. After preprocessing the original point cloud data corresponding to all targets, obtain the corresponding preprocessed point cloud and the corresponding target category information, thereby obtaining the training set.
[0006] S2: Construct a position estimation network and a pose estimation network. The position estimation network and the pose estimation network have the same structure. Both the position estimation network and the pose estimation network include a color feature extraction module, a depth feature extraction module, a feature fusion module and a regression prediction module. The color feature extraction module and the depth feature extraction module are connected to the feature fusion module, and the feature fusion module is connected to the regression prediction module.
[0007] S3: Train the position estimation network and pose estimation network separately using the training set to obtain the trained position estimation network and pose estimation network respectively;
[0008] S4: Input the target point cloud to be estimated and the corresponding target category information into the trained position estimation network and pose estimation network to obtain the three-degree-of-freedom position and three-degree-of-freedom pose of the target pose, thereby obtaining the target estimated pose.
[0009] In step S1, the preprocessing of the original point cloud data corresponding to all targets includes standardization, normalization, and downsampling.
[0010] The color feature extraction module includes dense convolutional blocks and transition layers. The input to the color feature extraction module is the color information in the point cloud data, which serves as the input to the first dense convolutional block. The first dense convolutional block is connected to the second dense convolutional block after passing through the transition layer. The output of the second dense convolutional block serves as the output of the color feature extraction module.
[0011] Both the first and second dense convolutional blocks contain three dense convolutional layers. In the first dense convolutional block, the input of the first dense convolutional block is used as the input of the first dense convolutional layer. The input of the first dense convolutional layer and the output of the first dense convolutional layer are concatenated to serve as the input of the second dense convolutional layer. The input of the second dense convolutional layer and the output of the second dense convolutional layer are concatenated to serve as the input of the third dense convolutional layer. The input of the third dense convolutional layer and the output of the third dense convolutional layer are concatenated to serve as the output of the first dense convolutional block.
[0012] The transition layer comprises a batch normalization function, a ReLU activation function, and a convolutional layer connected in sequence.
[0013] The deep feature extraction module includes two multi-layer neighborhood aggregation modules and two global feature feedback modules. The input to the deep feature extraction module is the positional information in the point cloud data, which serves as the input point cloud for the first multi-layer neighborhood aggregation module. The first multi-layer neighborhood aggregation module outputs a first downsampled point cloud and its corresponding depth features, which are then used as the input for the second multi-layer neighborhood aggregation module. The second multi-layer neighborhood aggregation module outputs a second downsampled point cloud and its corresponding depth features. In the first global feature feedback module, weighted interpolation and convolution are performed based on the first downsampled point cloud and its corresponding depth features, as well as the second downsampled point cloud and its corresponding depth features, to obtain the updated depth features corresponding to the first downsampled point cloud. In the second global feature feedback module, weighted interpolation and convolution are performed based on the input point cloud of the deep feature extraction module, as well as the first downsampled point cloud and its corresponding updated depth features, to obtain the updated depth features corresponding to the input point cloud of the deep feature extraction module, which are then used as the output of the deep feature extraction module.
[0014] In the first multi-layer neighborhood aggregation module, the input point cloud of the depth feature extraction module is first downsampled to obtain the first downsampled point cloud. For each point in the first downsampled point cloud, the points in the input point cloud of the depth feature extraction module are queried in two spherical neighborhoods with different radii to obtain two spherical neighborhood point sets. Then, the two spherical neighborhood point sets of the point are convolved to extract the features of the point at two scales. The features of the point at the two scales are then concatenated to obtain the depth features of the point. All points in the first downsampled point cloud are traversed to obtain the depth features corresponding to the first downsampled point cloud.
[0015] In the second multi-layer neighborhood aggregation module, the input point cloud of the second multi-layer neighborhood aggregation module is first downsampled to obtain the second downsampled point cloud. For each point in the second downsampled point cloud, the points of the point in the input point cloud of the second multi-layer neighborhood aggregation module with two different radii are queried to obtain two spherical neighborhood point sets. Then, the two spherical neighborhood point sets of the point and the depth features corresponding to the points in the point sets are concatenated and then convolved to extract the features of the point at two scales. The features of the point at two scales are then concatenated to obtain the depth features of the point. All points in the second downsampled point cloud are traversed to obtain the depth features corresponding to the second downsampled point cloud.
[0016] In the first global feature feedback module, for each point in the first downsampled point cloud, the three closest points to that point are found in the second downsampled point cloud, and the reciprocal of the square of the distances between these three points is used as the corresponding weights. Then, the depth features corresponding to these three points are multiplied by their weights and summed to obtain the fitted depth features. The depth features of that point in the first downsampled point cloud are concatenated with the fitted depth features and then convolved to obtain the updated depth features of that point in the first downsampled point cloud. The updated depth features of the first downsampled point cloud are obtained by traversing all points in the first downsampled point cloud.
[0017] In the second global feature feedback module, for each point in the input point cloud of the deep feature extraction module, the three closest points to that point are found in the first downsampled point cloud, and the reciprocal of the square of the distances between these three points is used as the corresponding weights. Then, the updated depth features corresponding to these three points are multiplied by their weights and summed to obtain the fitted depth features. The fitted depth features are concatenated with the input point cloud of the deep feature extraction module and then convolved to obtain the updated depth features of that point in the input point cloud of the deep feature extraction module. The updated depth features corresponding to the input point cloud of the deep feature extraction module are obtained by traversing all points in the input point cloud of the deep feature extraction module.
[0018] In the feature fusion module, the color feature information extracted by the color feature extraction module, the depth feature information extracted by the depth feature extraction module, and the target category information corresponding to the point cloud are concatenated together to obtain a multimodal feature tensor, which is then used as input. After passing through 5 convolutional layers, the output is fused feature information containing multiple levels and multiple modalities.
[0019] The regression prediction module is a multilayer perceptron.
[0020] In S3, the Euclidean distance between the predicted position and the true position is used as the position loss function during the training process of the position estimation network; the geodesic distance between the rotation matrix corresponding to the predicted posture and the rotation matrix corresponding to the true posture is used as the posture loss function during the training process of the attitude estimation network.
[0021] Compared with existing technologies, this invention provides a method for pose estimation that integrates color and depth modal features, which has the following advantages:
[0022] (1) The present invention designs a dense convolutional block for extracting color information features from point cloud data. For color information, the dense convolutional block concatenates the output of each dense convolutional layer with the output of the previous dense convolutional layer, so that the network can learn local and global features at the same time, promote the transmission and sharing of information, and facilitate the full perception of color information.
[0023] (2) The present invention designs a multi-layer neighborhood aggregation module and a global feature feedback module to extract depth features. For the location information in point cloud data, the multi-layer neighborhood aggregation module introduces the concept of multi-scale aggregation, which can extract depth features at different scales. The global feature feedback module uses the nearest neighbor interpolation method to feed the global features back to the input point cloud, making the final depth features more representative.
[0024] (3) This invention designs a feature extraction module corresponding to two different modal information in point cloud data, which can be used for targeted learning. A feature fusion module is designed for the obtained multimodal features, which consists of multiple convolutional layers and can form multi-level multimodal fusion features. Compared with pixel-level feature fusion, feature fusion at the point cloud level can more comprehensively perceive and understand the appearance and geometric features of objects in the scene. At the same time, two networks are designed to predict position and pose respectively, which increases the focus of the network and improves the accuracy of pose estimation. Attached Figure Description
[0025] Figure 1 This is a schematic diagram showing the composition of the position estimation network and attitude estimation network proposed in this invention.
[0026] Figure 2 This is a detailed structural diagram of the position estimation network and attitude estimation network proposed in this invention.
[0027] Figure 3 This is a flowchart of a target pose estimation method based on a multimodal feature fusion deep neural network according to the present invention.
[0028] Figure 4 This is a flowchart illustrating the model training process for the position estimation network and pose estimation network proposed in this invention.
[0029] Figure 5The images show a comparison of the pose estimation results of the present invention method and the DenseFusion method for targets in the same scene. (a) is a comparison of the actual pose of the drill with the pose estimation results of the DenseFusion method and the present invention method for the drill. (b) is a comparison of the actual pose of the can with the pose estimation results of the DenseFusion method and the present invention method for the can.
[0030] Figure 6 This is a comparison chart of the threshold-accuracy curves of the method of this invention and the DenseFusion method. Detailed Implementation
[0031] like Figure 1 , Figure 2 As shown, this invention proposes a target pose estimation method based on a multimodal feature fusion deep neural network. This method requires the object's color and depth information, as well as the corresponding object category information, as inputs to the pose estimation network. This method can simultaneously learn features from color and depth information, making more comprehensive use of the object's rich appearance and shape geometry information, and can still perform pose estimation under partial occlusion and environmental changes.
[0032] like Figure 3 and Figure 4 As shown, the present invention includes the following steps:
[0033] S1: From the RGB and depth images captured by the camera, instance segmentation is performed on the RGB images, and the results are mapped to the depth map to obtain the target category and color and depth information of pixels belonging to the same target. The color and depth information of all pixels belonging to the same target in the RGB and depth images are converted into the raw point cloud data of the target and the target category information is recorded. After preprocessing the raw point cloud data corresponding to all targets, the corresponding preprocessed point cloud and the corresponding target category information are obtained, thus obtaining two parts: the training set and the test set. The preprocessing of the raw point cloud data corresponding to all targets includes standardization, normalization, and downsampling of the point cloud using the farthest point sampling method. In this embodiment, RGB and depth images from the YCB-Video dataset are used directly. This dataset contains 92 video sequences, totaling 133,827 frames, including 21 objects selected from the YCB object set, with 6D pose annotations.
[0034] The Mask R-CNN method is used to segment RGB images, and the results are mapped to a depth map to obtain the target category and color and depth information of the same target. The color and depth information of all pixels belonging to the same target in the RGB and depth images are converted into point cloud data of the target and the target category information is recorded. The point corresponding to each pixel contains XYZ position information and RGB color information. Thus, the point cloud data of all targets in the RGB image and the target category are obtained. The position and color information of each target point cloud data are standardized and normalized respectively. Then, the farthest point sampling method is used to downsample the point cloud. The preprocessed point cloud data and the target categories corresponding to the point cloud are used as a dataset, which is divided into two parts: a training set and a test set. 80 video sequences are used as the training dataset, and the remaining 12 video sequences are used as the test dataset.
[0035] S2: As Figure 1 and Figure 2 As shown, the present invention constructs a position estimation network and a pose estimation network. The position estimation network and the pose estimation network have the same structure. Both the position estimation network and the pose estimation network include a color feature extraction module, a depth feature extraction module, a feature fusion module and a regression prediction module. The color feature extraction module and the depth feature extraction module are connected to the feature fusion module, and the feature fusion module is connected to the regression prediction module.
[0036] The color feature extraction module extracts global and local color features. It includes dense convolutional blocks and transition layers. The input to the color feature extraction module is the color information from the preprocessed point cloud, which serves as the input to the first dense convolutional block. This first dense convolutional block is connected to the second dense convolutional block after passing through the transition layer. The output of the second dense convolutional block is the output of the color feature extraction module, i.e., a multi-scale color feature tensor. Both the first and second dense convolutional blocks contain three dense convolutional layers. In the first dense convolutional block, the input of the first dense convolutional block serves as the input of the first dense convolutional layer. The input and output of the first dense convolutional layer are concatenated to serve as the input of the second dense convolutional layer. The input and output of the second dense convolutional layer are then concatenated to serve as the input of the third dense convolutional layer. Finally, the input and output of the third dense convolutional layer are concatenated to serve as the output of the first dense convolutional block. The transition layer consists of sequentially connected batch normalization, ReLU activation functions, and convolutional layers, used to control the number of feature channels extracted.
[0037] Specifically:
[0038] First, the color information P in the point cloud color A convolution operation is performed using a 1×1 kernel to obtain the initial color feature as F0;
[0039] Next, the initial color features F0 are input into the first dense convolutional block to obtain richer and more advanced color features. One of the dense convolutional blocks contains three dense convolutional layers. In each dense convolutional layer, the input color features are convolved, and the result is concatenated with the output of the previous layer to form the color feature F extracted by that layer. i Output:
[0040] F i =Concat(F i-1 Conv 3×3 (Conv 1×1 (F i-1 i = 1, 2, 3
[0041] Where Fi-1 is the output of the previous layer, Conv 1×1 (),Conv 3×3 () represent convolution operations using convolution kernels of size 1×1 and 3×3 respectively, and Concat() represents concatenation operations along the feature dimension. The color feature extracted by the first dense convolution block is F3.
[0042] Then, the features extracted from the first dense convolutional block are processed using a transition layer, which includes batch normalization of the input, ReLU activation function, and finally convolution operation with kernel size of 1×1 to control the number of feature channels extracted, reduce computational burden, and help extract features of different scales at different levels. The output is F′0.
[0043] Finally, the color features F′0 processed by the transition layer are input into the second dense convolutional block, whose structure is exactly the same as the first dense convolutional block. The output of each dense convolutional layer is F. i F' (i = 1, 2, 3) is used as the output of the dense convolutional block F3' as the final extracted color modality feature F. color .
[0044] The deep feature extraction module is used to extract global and local feature information of positional information in the point cloud. It includes two multi-layer neighborhood aggregation modules and two global feature feedback modules. The input point cloud of the deep feature extraction module is used as the input point cloud of the first multi-layer neighborhood aggregation module. The first multi-layer neighborhood aggregation module outputs a first downsampled point cloud and its corresponding depth features, which are used as the input of the second multi-layer neighborhood aggregation module. The second multi-layer neighborhood aggregation module outputs a second downsampled point cloud and its corresponding depth features. In the first global feature feedback module, weight interpolation and convolution are performed based on the first downsampled point cloud and its corresponding depth features, as well as the second downsampled point cloud and its corresponding depth features, to obtain the updated depth features corresponding to the first downsampled point cloud. In the second global feature feedback module, weight interpolation and convolution are performed based on the input point cloud of the deep feature extraction module, as well as the first downsampled point cloud and its corresponding updated depth features, to obtain the updated depth features corresponding to the input point cloud of the deep feature extraction module, which are then used as the output of the deep feature extraction module.
[0045] In the first multi-layer neighborhood aggregation module, the input point cloud of the depth feature extraction module is downsampled using the farthest point sampling method to obtain the first downsampled point cloud. For each point in the first downsampled point cloud, the points in the input point cloud of the depth feature extraction module are queried in two spherical neighborhoods with different radii. The number of points in the spherical neighborhoods with different radii is different, so two spherical neighborhood point sets are obtained. Then, the two spherical neighborhood point sets of the point are convolved to extract the features of the point at two scales. The features of the point at the two scales are then concatenated to obtain the depth features of the point. All points in the first downsampled point cloud are traversed to obtain the depth features corresponding to the first downsampled point cloud.
[0046] Specifically, the module first processes the input point cloud P depth The first downsampled point cloud P is obtained by downsampling using the farthest point sampling method. depth Then, using the concept of multi-scale aggregation, the first downsampled point cloud P is processed sequentially according to two different scales. depth Aggregation and feature extraction operations are performed. The multi-scale aggregation method first utilizes the spherical neighborhood query method for the first downsampled point cloud P. depth For each point in the input point cloud P, with that point as the center and radii of 0.1m and 0.2m respectively, [the following points are defined]. depth Find the 32 and 64 nearest points to the first downsampled point cloud P. depth For each point in the vector, add a new dimension under that point, and then convert the input point cloud P into a vector. depth The corresponding 32 points and 64 points found are combined to obtain the aggregated point cloud P in the newly added dimension. i(i = 1, 2), containing local information around each sampling point, will yield aggregated point clouds P1 and P2 at two scales. For the aggregated point clouds, three convolutional layers with kernel size of 1×1 are used for convolution to obtain the feature representation F. i depth (i = 1, 2), and finally the features F at both scales. i depth (i=1,2) concatenation yields the first downsampled point cloud P. depth The deep features F′ after multi-scale aggregation depth .
[0047] In the second multi-layer neighborhood aggregation module, the input point cloud (i.e., the first downsampled point cloud) of the second multi-layer neighborhood aggregation module is first downsampled using the farthest point sampling method to obtain the second downsampled point cloud. For each point in the second downsampled point cloud, the points of the point in the input point cloud of the second multi-layer neighborhood aggregation module are queried in two spherical neighborhoods with different radii. The number of points in the spherical neighborhoods with different radii is different, so two spherical neighborhood point sets are obtained. Then, the two spherical neighborhood point sets of the point and the depth features corresponding to the points in the point sets are concatenated and then convolved to extract the features of the point at two scales. The features of the point at the two scales are then concatenated to obtain the depth features of the point. All points in the second downsampled point cloud are traversed to obtain the depth features corresponding to the second downsampled point cloud.
[0048] Specifically, the structure of the second multi-layer neighborhood aggregation module is the same as that of the first multi-layer neighborhood aggregation module, but the input at this time is not only the first downsampled point cloud P. depth ′, and the first downsampled point cloud P depth The corresponding deep feature F′ depth .
[0049] For P depth The second downsampled point cloud P is obtained by performing downsampling. depth Then, using the concept of multi-scale aggregation, for the second downsampled point cloud P depth For each point in the graph, with that point as the center and radii of 0.2m and 0.4m respectively, in the point cloud P... depth Find the 32 and 64 nearest points to the given point in the second downsampled point cloud P. depth Each point in the matrix is combined to obtain the aggregated point cloud P. i ′(i=1,2).
[0050] The following operations differ from the first multi-layer neighborhood aggregation module. Given P... depth ′Mid-distance P depth The positions of the 32 and 64th nearest sampling points in P depth The point cloud depth features F′depth The depth features corresponding to these 32 points and 64 points are found separately and then aggregated to obtain point clouds P at different scales. i The initial point cloud depth features of point P (i = 1, 2) are then stitched together. i In the spatial coordinate dimension of ′, a point cloud P containing both positional information and depth features is obtained. i (i=1,2), finally, features are extracted from P1″ and P2″ at the two scales, and then the features are concatenated to obtain the second downsampled point cloud P. depth "Richer and more abstract deep features F after multi-scale aggregation" depth .
[0051] In the first global feature feedback module, for each point in the first downsampled point cloud, the three closest points to that point are found in the second downsampled point cloud, and the reciprocal of the square of the distances between these three points is used as the corresponding weights. Then, the depth features corresponding to these three points are multiplied by their weights and summed to obtain the fitted depth features. The depth features of that point in the first downsampled point cloud are concatenated with the fitted depth features and then convolved to obtain the updated depth features of that point in the first downsampled point cloud. The process is repeated for all points in the first downsampled point cloud to obtain the updated depth features corresponding to the first downsampled point cloud.
[0052] Specifically, the first global feature feedback module mainly includes nearest neighbor weight interpolation and convolution operations. Firstly, it performs a subsampled point cloud P... depth "Find the cloud P that is the first downsampled point" depth For each point in the triangle, calculate P from the three nearest points. depth "From these three points to P" depth The distance corresponding to the point in F' is used as the weight, with the reciprocal of the square of the distance as the weight. depth The corresponding deep features are multiplied by their respective weights and then summed to obtain the result from F″. depth Feedback from the first downsampled point cloud P depth The depth feature F′ of the corresponding point in ′ 3NN Then F′ 3NN With P depth Its original depth feature F′ depth After concatenation, a convolution operation is performed, and the data is fed into two convolutional layers with kernel size of 1×1 to extract features, resulting in the first downsampled point cloud P. depth The new depth feature representation F' 3NN-depth .
[0053] In the second global feature feedback module, for each point in the input point cloud of the deep feature extraction module, the three closest points to that point are found in the first downsampled point cloud. The reciprocal of the square of the distance between these three points is used as the corresponding weight. Then, the updated depth features corresponding to these three points are multiplied by their weights and summed to obtain the fitted depth features. The fitted depth features are concatenated with the input point cloud of the deep feature extraction module and then convolved to obtain the updated depth features of that point in the input point cloud of the deep feature extraction module. The process is repeated for all points in the input point cloud of the deep feature extraction module to obtain the updated depth features corresponding to the input point cloud of the deep feature extraction module.
[0054] Specifically, a second global feature feedback module with the same structure as the first global feature feedback module is used to process the first downsampled point cloud P. depth The new depth features F' obtained 3NN-depth Input point cloud P depth Direction propagation, using nearest neighbor interpolation, yields the result from point cloud P. depth Feedback from the point cloud P depth depth features F 3NN At this point, since point cloud P has not been extracted using convolutional layers before... depth The features, so the obtained depth features F 3NN The point cloud P is obtained by feeding it into a convolutional layer to extract features. depth The final deep modal feature representation F depth .
[0055] The feature fusion module will integrate the color modality feature F color and depth modal features F depth And the target category information corresponding to the point cloud is used as input. First, the color modality feature F color and depth modal features F depth The target category information corresponding to the point cloud is concatenated to obtain the multimodal feature tensor F0. The target category information is stored in one-hot encoding and contains the category information of each point in the point cloud. Then, the multimodal feature tensor F0 is fed into 5 sequentially connected convolutional layers. Each convolutional layer uses a 1×1 convolutional kernel. After all convolutional operations, the output is the fused feature information F containing multi-level multimodal features. final .
[0056] The regression prediction module is a multilayer perceptron, which uses the multilayer perceptron to perform regression prediction on the fused features to obtain the target position and pose.
[0057] S3: Train the position estimation network and pose estimation network separately using the training set. After training for a specified number of rounds, retain the optimal model file to obtain the trained position estimation network and pose estimation network respectively.
[0058] In S3, the Euclidean distance between the predicted position and the true position is used as the position loss function during the training of the position estimation network; the geodesic distance between the rotation matrix corresponding to the predicted pose and the rotation matrix corresponding to the true pose is used as the pose loss function during the training of the pose estimation network.
[0059] Specifically, for the location loss function, since the location information in the point cloud data has been standardized and normalized, the regression prediction results need to be destandardized and denormalized to obtain the complete location coordinates. The Euclidean distance between the predicted and actual locations is used as the location loss function L. t :
[0060]
[0061] in, These are the predicted location coordinates. These are the actual position coordinates, and ||·||2 represents the 2-norm of the vector;
[0062] For the pose loss function, the multilayer perceptron regression yields a predicted pose in axis-angle representation. In axis-angle representation, each pose has a unique rotation axis and rotation angle, which helps the network learn a more stable and reliable pose representation. Furthermore, axis-angle representation provides an intuitive model of pose, making it easier to understand the pose parameters learned by the network.
[0063] In the axis-angle representation, a 3-degree-of-freedom vector is used. Indicates a The unit vector is the axis of rotation, and the rotation angle is... The pose is given by ||·||², where ||·||² represents the 2-norm of the vector. The rotation matrix R used for coordinate transformation of the pose can be expressed as a vector with axis angles. After mathematical transformation, we obtain:
[0064]
[0065] Among them, I 3×3 It is a 3x3 identity matrix, R * It is an antisymmetric matrix.
[0066]
[0067] The attitude estimate obtained from regression Attitude loss function L r The calculation is based on the predicted attitude expressed in axis angles. The corresponding rotation matrix R and the axis angle represent the true pose. The corresponding rotation matrix R real The geodesic distance between them, i.e.:
[0068]
[0069] Here, arccos(·) is the inverse trigonometric function of arccosine, and trace(·) represents the trace of a matrix, which is the sum of the elements on the main diagonal of the input matrix.
[0070] Geodesic distance is a concept that describes the shortest path between two points on a surface. As a loss function, it makes the optimization process more intuitive, and geodesic distance usually provides better numerical stability.
[0071] After training for a specified 60 epochs, retain the optimal model file.
[0072] S4: Input the target point cloud to be estimated and the corresponding target category information into the trained position estimation network and pose estimation network to obtain the three-degree-of-freedom position and three-degree-of-freedom pose of the target, thereby obtaining the estimated target pose in the camera coordinate system. In specific implementation, the test set is input into the trained position estimation network and pose estimation network.
[0073] To verify the practical effect of the target pose estimation method based on multimodal feature fusion deep neural network disclosed in this invention, tests were conducted on a test dataset, and the test results were compared with the DenseFusion method:
[0074] Select a scenario from the test dataset (e.g.) Figure 5 To visually demonstrate the pose estimation results, the results of the two methods are represented by shaded areas and white bounding boxes, and plotted on the original scene image. In this scene, the drill is largely unobstructed, while the can has some occlusion. The prediction results are as follows: Figure 5 (a) and Figure 5 As shown in (b), it can be seen that the DenseFusion method and the method of the present invention have roughly the same predicted pose for the electric drill, and both are relatively accurate. However, for the can with some obstruction, the predicted pose of the method of the present invention is closer to the actual pose than that of the DenseFusion method.
[0075] The average distance (AD) between the actual and predicted values of the object's point cloud is used as the pose estimation evaluation metric. A target object is randomly selected from the test set, and the point cloud Ω at the origin of the camera coordinate system and its corresponding true pose are obtained. The point cloud contains m points, and the true position coordinates in the camera coordinate system are t. real The true rotation matrix R real Then, the estimated position coordinates t are obtained by the method of this invention and the DenseFusion method, respectively.pred and the rotation matrix estimate R pred And calculate the average distance (AD):
[0076]
[0077] Where ||·||2 represents the 2-norm of the vector.
[0078] If the average distance is less than a given threshold, the estimated pose of the target object is considered accurate. The average distance of all objects in the test set is calculated, and a threshold-accuracy curve is plotted with the threshold as the horizontal axis and the pose estimation accuracy at different thresholds as the vertical axis. The maximum threshold is set to 0.1m. The threshold-accuracy curves for the DenseFusion method and the method of this invention are shown below. Figure 6 (a) and Figure 6 As shown in (b), it can be seen that, under different thresholds, the prediction accuracy of the method of this invention is generally better than that of the DenseFusion method. Calculate the area under this curve ( Figure 6 The proportion of the shaded area to the total area is used as the accuracy of the final pose estimation method, where the total area is calculated by multiplying the maximum threshold by the maximum accuracy (i.e., 1).
[0079] The accuracy of prediction was calculated by applying the method of this invention and the DenseFusion method to all objects in the test set, and the results are shown in Table 1.
[0080] Table 1 compares the accuracy of the method of this invention and the DenseFusion method on the YCB test dataset.
[0081] Pose estimation methods accuracy Method of the present invention 79.2% DenseFusion method 74.7%
[0082] As can be seen from the results in Table 1, compared with the DenseFusion method, the target pose estimation method based on multimodal feature fusion deep neural network of the present invention can learn richer depth and color modal features through its feature extraction module, which enables the deep neural network to make fuller use of multimodal features for pose estimation and improves the accuracy.
[0083] For all canned objects in the test set (with some occlusion), the method of this invention and the DenseFusion method were used to predict and calculate the accuracy. The results are shown in Table 2:
[0084] Table 2 compares the accuracy of the method of this invention and the DenseFusion method in predicting canned objects in the YCB test dataset.
[0085]
[0086] As can be seen from the results in Table 2, compared with the DenseFusion method, the target pose estimation method based on multimodal feature fusion deep neural network of the present invention can more accurately identify the local features of the target and can also perform pose estimation well when there is a certain occlusion between objects.
[0087] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.
Claims
1. A target pose estimation method based on a multi-modal feature fusion deep neural network, characterized in that, Includes the following steps: S1: Obtain the target category and color and depth information belonging to the same target from the RGB and depth images captured by the camera. Convert the color and depth information of all pixels of the same target into the original point cloud data of the target and record the target category information. After preprocessing the original point cloud data corresponding to all targets, obtain the corresponding preprocessed point cloud and the corresponding target category information, thereby obtaining the training set. S2: Construct a position estimation network and a pose estimation network. The position estimation network and the pose estimation network have the same structure. Both the position estimation network and the pose estimation network include a color feature extraction module, a depth feature extraction module, a feature fusion module and a regression prediction module. The color feature extraction module and the depth feature extraction module are connected to the feature fusion module, and the feature fusion module is connected to the regression prediction module. In the feature fusion module, the color feature information extracted by the color feature extraction module, the depth feature information extracted by the depth feature extraction module, and the target category information corresponding to the point cloud are concatenated together to obtain a multimodal feature tensor, which is then used as input. After passing through 5 convolutional layers, the output is fused feature information containing multiple levels and multiple modalities. The color feature extraction module includes a dense convolutional block and a transition layer. The input of the color feature extraction module is the color information in the point cloud data, which is used as the input of the first dense convolutional block. The first dense convolutional block is connected to the second dense convolutional block after passing through the transition layer. The output of the second dense convolutional block is used as the output of the color feature extraction module. The deep feature extraction module includes two multi-layer neighborhood aggregation modules and two global feature feedback modules. The input to the deep feature extraction module is the positional information in the point cloud data, which serves as the input point cloud for the first multi-layer neighborhood aggregation module. The first multi-layer neighborhood aggregation module outputs a first downsampled point cloud and its corresponding depth features, which are then used as the input for the second multi-layer neighborhood aggregation module. The second multi-layer neighborhood aggregation module outputs a second downsampled point cloud and its corresponding depth features. In the first global feature feedback module, weighted interpolation and convolution are performed based on the first downsampled point cloud and its corresponding depth features, as well as the second downsampled point cloud and its corresponding depth features, to obtain the updated depth features corresponding to the first downsampled point cloud. In the second global feature feedback module, weighted interpolation and convolution are performed based on the input point cloud of the deep feature extraction module, as well as the first downsampled point cloud and its corresponding updated depth features, to obtain the updated depth features corresponding to the input point cloud of the deep feature extraction module, which are then used as the output of the deep feature extraction module. S3: Train the position estimation network and pose estimation network separately using the training set to obtain the trained position estimation network and pose estimation network respectively; S4: Input the target point cloud to be estimated and the corresponding target category information into the trained position estimation network and pose estimation network to obtain the three-degree-of-freedom position and three-degree-of-freedom pose of the target pose, thereby obtaining the target estimated pose. 2.The target pose estimation method based on the multi-modal feature fusion deep neural network according to claim 1, characterized in that, In step S1, the preprocessing of the original point cloud data corresponding to all targets includes standardization, normalization, and downsampling. 3.The target pose estimation method based on the multi-modal feature fusion deep neural network according to claim 1, characterized in that, Both the first and second dense convolutional blocks contain three dense convolutional layers. In the first dense convolutional block, the input of the first dense convolutional block is used as the input of the first dense convolutional layer. The input of the first dense convolutional layer and the output of the first dense convolutional layer are concatenated to serve as the input of the second dense convolutional layer. The input of the second dense convolutional layer and the output of the second dense convolutional layer are concatenated to serve as the input of the third dense convolutional layer. The input of the third dense convolutional layer and the output of the third dense convolutional layer are concatenated to serve as the output of the first dense convolutional block. 4.The target pose estimation method based on the multi-modal feature fusion deep neural network according to claim 1, characterized in that, The transition layer comprises a batch normalization function, a ReLU activation function, and a convolutional layer connected in sequence.
5. The target pose estimation method based on a multimodal feature fusion deep neural network according to claim 1, characterized in that, In the first multi-layer neighborhood aggregation module, the input point cloud of the depth feature extraction module is first downsampled to obtain the first downsampled point cloud. For each point in the first downsampled point cloud, the points in the input point cloud of the depth feature extraction module are queried in two spherical neighborhoods with different radii to obtain two spherical neighborhood point sets. Then, the two spherical neighborhood point sets of the point are convolved to extract the features of the point at two scales. The features of the point at the two scales are then concatenated to obtain the depth features of the point. All points in the first downsampled point cloud are traversed to obtain the depth features corresponding to the first downsampled point cloud.
6. The target pose estimation method based on a multimodal feature fusion deep neural network according to claim 1, characterized in that, In the second multi-layer neighborhood aggregation module, the input point cloud of the second multi-layer neighborhood aggregation module is first downsampled to obtain the second downsampled point cloud. For each point in the second downsampled point cloud, the points of the point in the input point cloud of the second multi-layer neighborhood aggregation module with two different radii are queried to obtain two spherical neighborhood point sets. Then, the two spherical neighborhood point sets of the point and the depth features corresponding to the points in the point sets are concatenated and then convolved to extract the features of the point at two scales. The features of the point at two scales are then concatenated to obtain the depth features of the point. All points in the second downsampled point cloud are traversed to obtain the depth features corresponding to the second downsampled point cloud.
7. The target pose estimation method based on a multimodal feature fusion deep neural network according to claim 1, characterized in that, In the first global feature feedback module, for each point in the first downsampled point cloud, the three closest points to that point are found in the second downsampled point cloud, and the reciprocal of the square of the distances between these three points is used as the corresponding weights. Then, the depth features corresponding to these three points are multiplied by their weights and summed to obtain the fitted depth features. The depth features of that point in the first downsampled point cloud are concatenated with the fitted depth features and then convolved to obtain the updated depth features of that point in the first downsampled point cloud. The updated depth features of the first downsampled point cloud are obtained by traversing all points in the first downsampled point cloud.
8. The target pose estimation method based on a multimodal feature fusion deep neural network according to claim 1, characterized in that, In the second global feature feedback module, for each point in the input point cloud of the deep feature extraction module, the three closest points to that point are found in the first downsampled point cloud, and the reciprocal of the square of the distances between these three points is used as the corresponding weights. Then, the updated depth features corresponding to these three points are multiplied by their weights and summed to obtain the fitted depth features. The fitted depth features are concatenated with the input point cloud of the deep feature extraction module and then convolved to obtain the updated depth features of that point in the input point cloud of the deep feature extraction module. The updated depth features corresponding to the input point cloud of the deep feature extraction module are obtained by traversing all points in the input point cloud of the deep feature extraction module.
Citation Information
Patent Citations
Object six-degree-of-freedom pose estimation method based on color and depth information fusion
CN111179324A
Space target relative pose measurement method and system based on laser radar variable scale observation
CN116593990A
