A six-degree-of-freedom pose estimation method for objects in occluded and truncated scenes by learning multiple geometric features
Through the combination of multi-geometric feature learning and deep convolutional neural network, the features of color images and point cloud data are extracted, and the fusion characteristics of full-stream two-way fusion network are solved, and the pose estimation problem of the existing technology in complex backgrounds and occlusion scenarios is achieved, achieving stronger versatility and robustness.
Patent Information
- Application Number
- CN202210629967.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-06
- Publication Date
- 2025-04-25
- Estimated Expiration
- 2042-06-06
AI Technical Summary
Existing six-degree-of-freedom pose estimation methods for objects do not perform well in complex backgrounds and severe occlusion or truncation scenarios, and deep learning methods are not binding enough in these scenarios.
Using multi-geometric feature learning method, the appearance features are extracted from color images through deep convolutional neural networks, and the depth images are converted into point cloud data to extract geometric features from them. Then, the full-stream bidirectional fusion network is used to converge appearance and geometric features to generate fusion features. These features are input to the instance-level semantic segmentation module and the geometric representation prediction module for implementing the six-degree-of-freedom pose estimation of an object.
The performance of object position estimation in occlusion and truncation scenarios is improved, the universality and robustness of the method are enhanced, and it can better adapt to different application objects and complex lighting conditions.
Smart Images
Figure CN115147599B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an object pose estimation method in the technical field of computer vision and industrial automation, and in particular to an object six-degree-of-freedom pose estimation method for multi-geometric feature learning in occluded and truncated scenes. Background Art
[0002] Six-degree-of-freedom object pose estimation is a fundamental problem in many visual application fields, such as robot manipulation, autonomous driving, augmented reality, etc. It is defined as detecting the three-dimensional direction and position of the target object in the image in the camera coordinate system, usually represented by a rigid transformation from the object's own coordinate system to the camera coordinate system. In specific application scenarios, an ideal six-degree-of-freedom object pose estimation solution should be able to handle objects of various geometric shapes and surface textures, and be applicable to complex scenarios such as occlusion, truncation, and lighting changes.
[0003] Depending on whether the 3D model of the target object is known, the 6DOF pose estimation methods can be divided into instance-level (known) and general-level (unknown); according to the different modes of perception data, these methods can be divided into RGB-based, RGBD-based, point cloud-based, etc. In the field of robotic grasping, the 3D model of the target object is often easier to obtain, and with the popularity of consumer-grade RGBD sensors, instance-level object 6DOF pose estimation based on RGBD data currently has great research and application value.
[0004] Early methods for estimating the 6DOF pose of objects can be divided into two categories: one is to build the connection between the object model and the image through artificially designed features (such as SIFT, ORB, etc.), and the other is to use template matching technology to find the closest pose, but these methods are very sensitive to complex backgrounds and occlusions. In recent years, deep learning technology has been gradually introduced into this field and has achieved significant performance improvements. The 6DOF pose estimation of objects is decomposed into regressing the key points of the object through a neural network and solving the PnP or ICP problem. However, these methods perform poorly in the case of severe occlusion or truncation because the key points do not provide sufficient constraints. Summary of the invention
[0005] In order to address the shortcomings of traditional pose estimation methods, such as being sensitive to scene clutter and appearance changes, and the lack of constraints in current key point regression methods based on deep learning in severe occlusion and truncation scenes, the present invention proposes a six-degree-of-freedom pose estimation method for objects based on multi-geometric feature learning for occluded and truncated scenes, so as to address the above-mentioned shortcomings of existing pose estimation methods.
[0006] The technical solution of the present invention comprises the following steps:
[0007] Step 1) collecting a color image containing a target object and its corresponding depth image, extracting the appearance features of the scene from the color image using a specific deep convolutional neural network, converting the depth image into point cloud data, and extracting the geometric features of the scene from the point cloud data using a specific point cloud segmentation network;
[0008] In the step 1),
[0009] The specific deep convolutional neural network uses ResNet-34 and PSPNet. The pre-trained ResNet-34 and PSPNet are used to extract feature maps of size (H, W, C / 2) from color images of size (H, W, 3).
[0010] The specific point cloud segmentation network uses RandLA-Net; N pixels are randomly sampled from a depth image of size (H, W), the N sampled points are converted into point cloud data, and then input into RandLA-Net to obtain geometric features of dimension C / 2.
[0011] Among them, H and W represent the height and width of the input image respectively, C represents the dimension of the final generated fusion feature, and N represents the number of pixels randomly sampled from the input image.
[0012] Step 2) In the process of extracting appearance features and geometric features in step 1), the appearance features and geometric features extracted from the color image and point cloud data are fused using the full-flow bidirectional fusion network FFBFusion. The fusion process is divided into point-to-pixel and pixel-to-point local learnable fusion.
[0013] Step 3) aligning and splicing the appearance features and geometric features extracted in step 1) point by point to obtain fusion features of several sampling points in the scene;
[0014] In step 3), the fusion feature obtained by aligning and splicing the two features is N C-dimensional feature vectors f i (N sampling points that fuse appearance features and geometric features).
[0015] Step 4) Input the fused features into the instance-level semantic segmentation module, and use the trained instance-level semantic segmentation module to achieve segmentation prediction of different instances of multiple types of objects in the scene;
[0016] In step 4), the instance-level semantic segmentation module includes semantic segmentation and three-dimensional center point prediction, wherein the semantic segmentation is used to distinguish objects of different categories, and the three-dimensional center point prediction is used to distinguish different instances of objects of the same category;
[0017] The training phase of semantic segmentation constructs the following loss function:
[0018] L sem= -α(1-c i ) γ log(c i )
[0019] Among them, α is the balance parameter; γ is the modulation parameter; c i is the probability that the i-th sampling point belongs to the corresponding true class label;
[0020] The following loss function is constructed during the training phase of 3D center point prediction:
[0021]
[0022] Among them, O j represents the set of sampling points corresponding to the jth instance, j = (1, 2, ..., J), J is the number of all object instances detected in the scene; |O j | represents the number of N sampling points belonging to the jth instance; p i represents the position of the i-th sampling point in the camera coordinate system, i = (1, 2, ..., N); of i Represents point p i The center point bias of the three-dimensional center point prediction, of i * For point p i The true center point offset.
[0023] Step 5) Input the fused features into the geometric representation prediction module (shared MLP), and use the trained geometric representation prediction module to regress the key point bias, three-dimensional edge vector and dense symmetric points of the object;
[0024] In the step 5):
[0025] The N C-dimensional feature vectors f i Input the geometric representation prediction module to regress the key point bias, which is the sampling point p i Point to the 3D key point set of the instance in the camera coordinate system Vector of i,l ;kp l represents the position of the lth key point obtained by clustering in the camera coordinate system, l = (1, 2, ..., L), L represents the number of predefined 3D key points on the instance; the corresponding geometric representation prediction module adopts the following loss function in the training stage:
[0026]
[0027] Among them, i,l Represents the predicted value of the key point bias; Represents the true value of the key point bias: kp′l represents the lth key point predefined in the object coordinate system, l = (1, 2, ..., L), L represents the number of predefined three-dimensional key points in the instance, R is the rotation matrix, and t is the displacement vector.
[0028] The N C-dimensional feature vectors f i The input geometry representation prediction module regresses to obtain M edge vectors, Specifically: for object instance O j The point set {p i |p i ∈O j}, the geometric representation prediction module predicts all three-dimensional edge vectors ed at each sampling point m,i ; The corresponding training stage establishes the following loss function:
[0029]
[0030] Among them, ed m,i Denotes the geometric representation prediction module at point p i The mth edge vector predicted at , m∈M; Represents the mth real edge vector predefined on the instance.
[0031] For instance O j The point set {p i |p i ∈O j}, and each point p is regressed through the geometric representation prediction module i The symmetric point offset (Δx i ,Δy i ,Δz i ) T , where the specified symmetry plane is obtained by calculating the object's three-dimensional model through the SymSeg algorithm; the corresponding training stage establishes the following loss function:
[0032]
[0033] in, and Represents the true symmetry point offset.
[0034] Step 6) for each object instance sampling point set obtained in step 4), combined with the point cloud data obtained in step 1), the position of each key point in the camera coordinate system is calculated from the corresponding key point offset, and the symmetric point of the object surface point is obtained from the symmetric point prediction, and then the three-dimensional coordinates of each edge vector are obtained through statistical analysis;
[0035] In the step 6), for the object instance O jThe point set {p i |p i ∈O j}: Calculate the position kp of the 3D key point of each sampling point in the camera coordinate system based on the predicted key point bias l,i :kp l,i =p i +ofi i,l ; Then the object instance O is transformed into j Predicted 3D key points kp l,i and the three-dimensional edge vector ed m,i Clustering to get the optimal key point kp l and edge vector ed m ; The predicted dense symmetric points are expressed as in, Point p i The x-axis, y-axis, and z-axis coordinates in the camera coordinate system;
[0036] Thus we get a set of key points Edge vector collection and dense sets of symmetric points
[0037] Step 7) According to the correspondence between the key points predefined in the object coordinate system and the key points predicted in the camera coordinate system, the initial pose {R init |t init};
[0038] In step 7), the following least squares fitting is used to solve the predefined key point set in the object coordinate system: To the set of key points predicted in the camera coordinate system The pose transformation {R init |t init}:
[0039]
[0040] Where R is the rotation matrix and t is the displacement vector;
[0041] Among them, the set of key points predefined in the object coordinate system The three-dimensional model of the object is calculated by the FPS (Farthest pointsampling) algorithm;
[0042] Step 8) Define the difference equations about the pose transformation for the key point set, edge vector set and dense symmetric point set obtained in step 6) respectively, and construct a nonlinear optimization problem, starting from the initial pose {R init |t init}Starting from the Gauss-Newton method, the final position of the object is obtained by iterative optimization {R final |t final}.
[0043] The step 8) is specifically as follows:
[0044] 8.1) Three geometric representations kp for key points, edge vectors and dense symmetric points l ,ed m ,q i , respectively construct the following three difference equations about the rotation matrix R and the displacement vector t
[0045]
[0046] in, is the value of the normal vector of the specified symmetry plane of the object in the object coordinate system, and are the real edge vectors The key points corresponding to both ends;
[0047] 8.2) Construct the following nonlinear optimization equation, starting from the initial pose {R init |t init}Starting from the Gauss-Newton method, the final position of the object is obtained by iterative optimization {R final |t final}:
[0048]
[0049] Among them, σ edg and σ sym is a hyperparameter describing the importance of edge vectors and symmetric points, ρ() is the German-Mcclure function, β kpt , β edg and β sym are the hyperparameters corresponding to the three geometric representations.
[0050] The beneficial effects of the present invention are:
[0051] Compared with the traditional object six-degree-of-freedom pose estimation method, the present invention is more versatile and robust. Traditional methods based on local descriptors perform well only when the object has rich texture information, and template matching methods are more sensitive to the clutter and appearance changes of the scene; the method of the present invention utilizes the inherent feature learning and representation capabilities of the deep learning model to achieve the goal of adapting to different application objects, and by rendering the target object in different scenes and different lighting conditions, the model's anti-interference ability can be improved.
[0052] The present invention achieves the goal of improving the pose estimation performance in occlusion and truncation scenarios. The present invention uses a deep neural network to predict multiple geometric representations of the target object, wherein the introduction of edge vectors allows the network to pay more attention to the local spatial relationship of the object, and dense symmetric correspondence provides a large number of geometric constraints on the symmetric planes of the target object, thereby enabling the model to adapt to severe occlusion and truncation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 The six-degree-of-freedom pose estimation flow chart of the object proposed by the present invention;
[0054] Figure 2 This is the visualization result of the present invention on the Occlusion LineMOD dataset;
[0055] Figure 3 This is the visualization result of the present invention on the Truncation LineMOD dataset. DETAILED DESCRIPTION
[0056] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.
[0057] like Figure 1 As shown, the embodiments of the method of the present invention are as follows:
[0058] 1) Collect an RGBD image containing the target object, use a specific deep convolutional neural network to extract the appearance features of the scene from the color image, convert the corresponding depth data into point cloud data, and use a specific point cloud segmentation network to extract the geometric features of the scene from the point cloud data.
[0059] like Figure 1 As shown in the figure, a color image of size (480,640,3) containing the target object and a corresponding depth image of size (480,640) are collected, and 12,800 pixels are randomly sampled from the depth image to be converted into point cloud data containing scene geometry information.
[0060] 2) In the process of extracting features from different modal data through two networks in step 1), the full-flow bidirectional fusion network is used to locally fuse appearance features and geometric features at different levels.
[0061] The appearance feature map of (480, 640, 64) is extracted using ResNet-34 and PSPNet, and the geometric features of (12800, 64) are extracted using RandLA-Net.
[0062] 3) Align and concatenate the two features extracted in step 1) point by point to obtain the fused features of several pixel points in the scene, which are input into the geometric representation prediction module and the instance-level semantic segmentation module respectively.
[0063] The appearance feature with the corresponding size of (12800, 64) is obtained from the appearance feature map through the saved sampling point position, and the two are concatenated to obtain the fusion feature of (12800, 128).
[0064] Both the geometric representation prediction module and the instance-level semantic segmentation module use a multi-layer perceptron model consisting of three one-dimensional convolutional layers, but differ in the output dimension of the last layer.
[0065] 4) Use the trained geometric representation prediction module to regress object key point biases, three-dimensional edge vectors and dense symmetric correspondences, and use the trained instance-level semantic segmentation module to realize segmentation prediction of different instances of multiple types of objects in the scene.
[0066] The farthest point algorithm is used to select 8 key points on the surface of the 3D model of each target object. The network predicts the 3D offset vectors from each point to the key points in the camera coordinate system. The edge vectors are taken from the graph model composed of the above 8 key points, with a total of 28.
[0067] The SymSeg algorithm is used to segment the symmetry plane of the object's three-dimensional model, and then a point in the camera coordinate system is converted to the object coordinate system to obtain its symmetric point about the symmetry plane. Finally, the symmetric point is converted to the camera coordinate system as the symmetric point.
[0068] 5) For the pixel point set of each instance of the target object obtained in step 4), combined with the corresponding depth data, the position of each key point in the camera coordinate system is calculated from the corresponding key point offset, and the symmetric point of the object surface point is obtained from the symmetric correspondence prediction, and then the three-dimensional coordinates of each edge vector are obtained through statistical analysis.
[0069] 6) According to the correspondence between the predefined key points and the predicted key points of the target object, the initial pose is solved using the least squares fitting method.
[0070] 7) For each key point, each edge vector and dense symmetric point obtained in step 5), define the differential equation for posture transformation, and construct a nonlinear optimization problem. Starting from the initial posture, the final posture of the object is obtained by iterative optimization using the Gauss-Newton method.
[0071] In order to better demonstrate the advantages of the present invention, the present invention uses the LineMOD dataset to train the network model. The dataset consists of 13 video sequences, each sequence takes about 1,200 images for a low-texture object, and the ratio of training to testing is 1:5. The present invention also renders 20,000 synthetic images for each object for training.
[0072] like Figure 2 and Figure 3 As shown in the figure, the Occlusion LineMOD and Truncation LineMOD datasets are used to verify the performance of the model in occlusion and truncation scenarios respectively. The following two indicators, ADD (for asymmetric objects) and ADD-S (for symmetric objects), are examined in the quantitative comparative analysis:
[0073]
[0074] Among them, {R * |t *} represents the true pose, {R|t} represents the predicted pose, p represents the point on the object surface Ω, m represents the number of points used in the calculation, and ADD or ADD-S value less than 10% of the diameter of the object's three-dimensional model is considered to be a correct estimate.
[0075] As can be seen from the following Tables 1 and 2, the present invention is in a leading position in terms of the ADD(S) index for pose estimation of objects with different texture patterns in occlusion and truncation scenarios.
[0076] Table 1 Quantitative comparison results of the embodiments on the Occlusion LineMOD dataset
[0077]
[0078] Table 2 Quantitative comparison results of the embodiments on the Truncation LineMOD dataset.
[0079]
Claims
1. A six-degree-of-freedom pose estimation method for objects based on multi-geometric feature learning for occluded and truncated scenes, characterized in that: The following steps are involved: 1) Collect a color image containing the target object and its corresponding depth image, use a specific deep convolutional neural network to extract the appearance features of the scene from the color image, convert the depth image into point cloud data, and use a specific point cloud segmentation network to extract the geometric features of the scene from the point cloud data; 2) In the process of extracting appearance features and geometric features in step 1), the appearance features and geometric features extracted from the color image and point cloud data are fused using a full-flow bidirectional fusion network; 3) Aligning and splicing the appearance features and geometric features extracted in step 1) point by point to obtain fusion features in the scene; 4) Input the fused features into the instance-level semantic segmentation module, and use the trained instance-level semantic segmentation module to realize the segmentation prediction of different instances of multi-type objects in the scene; 5) Input the fused features into the geometric representation prediction module, and use the trained geometric representation prediction module to regress the key point bias, three-dimensional edge vector and dense symmetric points of the object; 6) For each object instance sampling point set obtained in step 4), combined with the point cloud data obtained in step 1), the position of each key point in the camera coordinate system is calculated from the corresponding key point offset, and the symmetric point of the object surface point is obtained from the symmetric point prediction, and then the three-dimensional coordinates of each edge vector are obtained through statistical analysis; 7) According to the correspondence between the key points predefined in the object coordinate system and the key points predicted in the camera coordinate system, the initial pose {R init |t init }; 8) Define the difference equations about the pose transformation for the key point set, edge vector set and dense symmetric point set obtained in step 6) respectively, and construct a nonlinear optimization problem, starting from the initial pose {R init |t init }Starting from the Gauss-Newton method, the final position of the object is obtained by iterative optimization {R final |t final }.
2. According to the method for object six-degree-of-freedom pose estimation based on multi-geometric feature learning for occluded and truncated scenes according to claim 1, in the step 1), The specific deep convolutional neural network uses ResNet-34 and PSPNet. The pre-trained ResNet-34 and PSPNet are used to extract feature maps of size (H, W, C / 2) from color images of size (H, W, 3). The specific point cloud segmentation network uses RandLA-Net; N pixels are randomly sampled from a depth image of size (H, W), the N sampled points are converted into point cloud data, and then input into RandLA-Net to obtain geometric features of dimension C / 2.
3. The method for object six-degree-of-freedom pose estimation based on multi-geometric feature learning for occluded and truncated scenes according to claim 1, characterized in that: In step 3), the fusion feature obtained by aligning and splicing the two features is N C-dimensional feature vectors f i .
4. The method for object six-degree-of-freedom pose estimation based on multi-geometric feature learning for occluded and truncated scenes according to claim 1, characterized in that: In step 4), the instance-level semantic segmentation module includes semantic segmentation and three-dimensional center point prediction, wherein the semantic segmentation is used to distinguish objects of different categories, and the three-dimensional center point prediction is used to distinguish different instances of objects of the same category; The training phase of semantic segmentation constructs the following loss function: L sem =-α(1-c i ) γ log(c i ) Among them, α is the balance parameter; γ is the modulation parameter; c i is the probability that the i-th sampling point belongs to the corresponding true class label; The following loss function is constructed during the training phase of 3D center point prediction: Among them, O j represents the set of sampling points corresponding to the jth instance, j = (1, 2, ..., J), J is the number of all object instances detected in the scene; |O j | represents the number of N sampling points belonging to the jth instance; p i represents the position of the i-th sampling point in the camera coordinate system, i = (1, 2, ..., N); of i Represents point p i The center point bias of the three-dimensional center point prediction, For point p i The true center point offset.
5. The method for object six-degree-of-freedom pose estimation based on multi-geometric feature learning for occluded and truncated scenes according to claim 1, characterized in that: In the step 5): The N C-dimensional feature vectors f i The input geometry representation prediction module is regressed to obtain the key point bias. The corresponding geometry representation prediction module adopts the following loss function in the training phase: Among them, i,l represents the predicted value of the key point bias; p i Represents the position of the i-th sampling point in the camera coordinate system; Represents the true value of the key point bias: Among them, kp′ l represents the lth key point predefined in the object coordinate system, l = (1, 2, ..., L), L represents the number of predefined three-dimensional key points in the instance, R is the rotation matrix, and t is the displacement vector.
6. The method for object six-degree-of-freedom pose estimation based on multi-geometric feature learning for occluded and truncated scenes according to claim 1, characterized in that: In the step 5): The N C-dimensional feature vectors f i The input geometry representation prediction module regresses to obtain M edge vectors, Specifically: For the object instance O j The point set {p i |p i ∈O j }, the geometric representation prediction module predicts all three-dimensional edge vectors ed at each sampling point m,i ; The corresponding training stage establishes the following loss function: Among them, ed m,i Denotes the geometric representation prediction module at point p i The mth edge vector predicted at , m∈M; Represents the mth real edge vector predefined on the instance.
7. The method for object six-degree-of-freedom pose estimation based on multi-geometric feature learning for occluded and truncated scenes according to claim 1, characterized in that: In the step 5), For instance O j The point set {p i |p i ∈O j }, and each point p is regressed through the geometric representation prediction module i The symmetric point offset (Δx i ,Δy i ,Δz i ) T , where the specified symmetry plane is obtained by calculating the object's three-dimensional model through the SymSeg algorithm; the corresponding training stage establishes the following loss function: in, and Represents the true symmetry point offset.
8. The method for object six-degree-of-freedom pose estimation based on multi-geometric feature learning for occluded and truncated scenes according to claim 1, characterized in that: In the step 6), for the object instance O j The point set {p i |p i ∈O j }: Calculate the position kp of the 3D key point of each sampling point in the camera coordinate system according to the predicted key point offset l,i :kp l,i =p i +of i,l ; Then the object instance O is transformed into j Predicted 3D key points kp l,i and the three-dimensional edge vector ed m,i Clustering to get the optimal key point kp l and edge vector ed m ; The predicted dense symmetric points are expressed as in, Point p i The x-axis, y-axis, and z-axis coordinates in the camera coordinate system; Thus we get a set of key points Edge vector collection and dense sets of symmetric points 9. The method for object six-degree-of-freedom pose estimation based on multi-geometric feature learning for occluded and truncated scenes according to claim 1, characterized in that: In step 7), the following least squares fitting is used to solve the predefined key point set in the object coordinate system: To the set of key points predicted in the camera coordinate system The pose transformation {R init |t init }: Where R is the rotation matrix and t is the displacement vector; Among them, the set of key points predefined in the object coordinate system The three-dimensional model of the object is calculated using the FPS algorithm.
10. The method for object six-degree-of-freedom pose estimation based on multi-geometric feature learning for occluded and truncated scenes according to claim 1, characterized in that: The step 8) is specifically as follows: 8.1) Three geometric representations kp for key points, edge vectors and dense symmetric points l ,ed m ,q i , respectively construct the following three difference equations about the rotation matrix R and the displacement vector t in, is the value of the normal vector of the specified symmetry plane of the object in the object coordinate system, and are the real edge vectors The key points corresponding to both ends; p i represents the position of the i-th sampling point in the camera coordinate system; kp′ l Represents the lth key point predefined in the object coordinate system; 8.2) Construct the following nonlinear optimization equation, starting from the initial pose {R init |t init }Starting from the Gauss-Newton method, the final position of the object is obtained by iterative optimization {R final |t final }: Among them, σ edg and σ sym is a hyperparameter describing the importance of edge vectors and symmetric points, ρ() is the German-Mcclure function, β kpt , β edg and β dym are the hyperparameters corresponding to the three geometric representations.