End-to-end pose estimation method for hinged object
Through a single-stage, end-to-end pose estimation method, semantic segmentation and point cloud feature extraction model are used, combined with clustering and alignment algorithms, the problem of pose estimation complexity of articulated objects is solved, and efficient and accurate 6D poses are achieved. It is estimated that it is suitable for real-time robot crawling tasks.
Patent Information
- Application Number
- CN202411201557.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to accurately estimate the position information of articulated objects, making it difficult for robots to accurately perceive and manipulate when grasping and operating, especially in complex situations under different assembly methods and environments.
Using a single-stage, end-to-end pose estimation method, a three-dimensional point cloud is extracted from RGB-D pictures through a semantic segmentation model, and a neural network model and prediction head are used to extract component categories, centroid offsets and normalize parts. Estimation of coordinate space regions, combining clustering algorithms and umeyama algorithms to restore the pose and size of each component.
A high-estimation rate and high accuracy of articulated objects 6D pose estimation is achieved, reducing calculation costs, suitable for robotic real-time grabbing tasks, avoiding damage to subtle joints.
Smart Images

Figure CN119068161A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of robot grasping and manipulation, and in particular relates to an end-to-end pose estimation method for an articulated object. Background Art
[0002] Before robots can perform grasping and manipulation tasks, it is crucial to accurately estimate the pose information of an object. Existing research has made significant progress in the pose estimation of a single rigid body. However, for non-rigid objects, such as articulated objects with multiple moving parts, the estimation of their pose information remains a challenge due to their complex physical properties. For example, articulated objects are composed of multiple rigid moving parts, making their perception and manipulation particularly complex. Articulated objects contain articulated objects, and inaccurate perception of articulated objects may cause the robot to damage delicate joints. Therefore, it is necessary to improve the robot's perception and estimation capabilities of articulated objects in order to enhance the robot's manipulation capabilities.
[0003] Currently, there are still significant challenges in perceiving articulated objects, including: 1) Variations in articulated objects within the same category. New articulated objects often do not have accurate 3D CAD models and need to be generalized within the category. For example, to estimate the handles of different bucket types, it is necessary to find a shared representation that can be generalized across various instances. 2) Variations in articulated objects across category environments. Articulated objects of one category have huge differences in the part environments of different object categories. Unlike category-level rigid body pose methods that deal with a single, consistent shape, articulated objects have multiple moving parts, resulting in differences in various environments. Therefore, even parts of the same instance may be assembled differently from other rigid body parts. For example, a hinged lid can be part of a laptop or a trash can. This difference also complicates the estimation of the pose and size of the target articulated object to be identified. Summary of the invention
[0004] In order to solve the problem that the time cost of joint object pose estimation in the prior art is large and cannot meet the robot's real-time pose estimation for grasping, the present invention provides a single-stage, real-time category-level joint object 6D pose estimation method for robot grasping scenarios, which has a high estimation rate and a high pose estimation accuracy. Compared with other objects, articulated objects are composed of multiple component instances. An articulated object needs to estimate the pose of each component instance. The traditional pose estimation method can only estimate the overall pose of the object as a whole. The method of the present invention directly estimates the component category, center of mass offset and NPCS representation on the point cloud through a unified network model; then, the estimated component category label is first used to classify the point cloud according to different component categories, and then the clustering algorithm is used based on the predicted center of mass offset to distinguish the points belonging to the same component instance. The design of component category and center of mass offset estimation enables the method of the present invention to effectively distinguish different component instance point clouds; finally, the final pose and size of each component instance are restored by aligning the NPCS area of each instance with the real point cloud.
[0005] Specifically, the present invention adopts the following technical solutions:
[0006] The present invention provides an end-to-end pose estimation method for an articulated object, the method having the following technical features, comprising the following steps: step S1, using a semantic segmentation model to segment RGB-D image information of a target articulated object from an input RGB-D picture, and using camera intrinsic parameters to map the RGB-D image information back to a three-dimensional space to obtain a three-dimensional point cloud of the target articulated object; step S2, inputting the three-dimensional point cloud of the target articulated object into a pose estimation model, thereby obtaining a pose estimation of an articulated component in the target articulated object, the pose estimation model comprising a point cloud feature extraction neural network model and three prediction heads, wherein step S2 comprises the following sub-steps: step S2-1, The three-dimensional point cloud of the target articulated object is input into the point cloud feature extraction neural network model to obtain a corresponding feature map; step S2-2, the feature map is input into the prediction heads respectively, and the semantic label, center offset, and normalized part coordinate space area coordinates of each point in the three-dimensional point cloud of the target articulated object are predicted by the three prediction heads respectively; step S2-3, based on the semantic label and the center offset, the three-dimensional point cloud of each component of the target articulated object is obtained by a clustering algorithm; step S2-4, the three-dimensional point cloud of the component is aligned with the normalized part coordinate space area coordinates to obtain the pose estimation of the component, which includes the pose estimation of the articulated component.
[0007] The end-to-end pose estimation method for articulated objects provided by the present invention may also have such technical features, wherein the pose estimation model is a pre-trained YOEO (You Only Estimate Once), the point cloud feature extraction neural network model is RandLa-Net, and the prediction head is a multi-layer perceptron.
[0008] The end-to-end pose estimation method for articulated objects provided by the present invention may also have such a technical feature, wherein, in step S2-3, the three-dimensional point cloud of each component of the target articulated object is obtained based on the semantic label and center offset of each point through the Meanshift clustering algorithm, and in step S2-4, the three-dimensional point cloud of each component is aligned with the normalized part coordinate space area coordinates through the Umeyama algorithm.
[0009] The end-to-end pose estimation method for articulated objects provided by the present invention may also have such a technical feature, wherein the training of the pose estimation model includes the following steps: step SA1, initializing the model parameters of the pose estimation model; step SA2, dividing the data set into a training set, a validation set, and a test set, and further dividing the test set and the validation set into subsets of object categories that have been seen and not seen in the training set, respectively; step SA3, inputting the training set into the pose estimation model in batches to obtain output results; step SA4, combining the output results with the true value to calculate the value of the loss function; step SA5, judging whether a predetermined iteration stop condition is reached, stopping the iteration when the judgment is yes, and obtaining the trained pose estimation model; step SA6, when step SA5 judges as no, calculating the gradient of the loss function, and updating the model parameters based on the gradient, and returning to step SA2.
[0010] The end-to-end pose estimation method for articulated objects provided by the present invention may also have such a technical feature, wherein, in step SA3, the three prediction heads of the pose estimation model respectively output the semantic label, center offset, and normalized part coordinate space region coordinates of each point in the three-dimensional point cloud, and in step SA4, Focal Loss is used as the loss function for the prediction head of semantic prediction, L1 Loss is used as the loss function for the prediction head of center offset, and for the prediction head of normalized part coordinate space region coordinates, the coordinates of the input three-dimensional point cloud are divided into multiple classes by discretization, and Softmax cross-entropy loss is used as the loss function, and the loss functions corresponding to the three prediction heads are multiplied by the corresponding weights to obtain the final loss function.
[0011] The end-to-end pose estimation method for an articulated object provided by the present invention may also have such a technical feature, wherein, in step SA5, the iteration stopping condition is that the loss after smoothing processing no longer continues to decrease or the number of training rounds reaches 20 epochs.
[0012] The end-to-end pose estimation method for articulated objects provided by the present invention may also have such a technical feature, wherein the data set is a GAPartNet data set, and the test set and the verification set respectively contain ten types of component categories.
[0013] The end-to-end pose estimation method for articulated objects provided by the present invention may also have such a technical feature, wherein, in step S1, the semantic segmentation model is Grounded-SAM.
[0014] Function and Effect of the Invention
[0015] According to the end-to-end pose estimation method for articulated objects of the present invention, semantic labels, centroid offsets and NPCS regional representations are simultaneously estimated directly on the three-dimensional point cloud of the target articulated object through a unified network model, so that points belonging to the same component instance can vote for the same centroid, and a clustering algorithm is used to distinguish points belonging to the same component instance based on the predicted centroid offset, and then the final posture and size of each component are restored by aligning the NPCS regional representation of each component instance with the real point cloud, so that accurate 6D pose estimation of all articulated parts in the target articulated object can be obtained. In addition, since the method of the present invention is a single-stage, end-to-end method, it is not necessary to first segment multiple component instances in the point cloud and then perform pose estimation like the existing multi-stage method, so the method of the present invention has a relatively low computational cost.
[0016] The method of the present invention can be applied to robot grasping scenarios, in which single-stage, real-time, category-level 6D pose estimation of articulated objects is achieved, allowing the robot to better distinguish instances of articulated components, thereby better locating and grasping articulated components, avoiding damage to their subtle joints, and enabling the robot to have higher performance in corresponding real-time tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 is a flow chart of an end-to-end pose estimation method for an articulated object in an embodiment of the present invention;
[0018] Figure 2 is a network structure diagram of a posture estimation model in an embodiment of the present invention;
[0019] Figure 3 This is an example of the result of performing pose estimation on an articulated object using the method of an embodiment of the present invention. Figure 1 ;
[0020] Figure 4 This is an example of the result of performing pose estimation on an articulated object using the method of an embodiment of the present invention. Figure 2 ;
[0021] Figure 5 This is an example of the result of performing pose estimation on an articulated object using the method of an embodiment of the present invention. Figure 3 . DETAILED DESCRIPTION
[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the end-to-end pose estimation method for articulated objects of the present invention is specifically described below in combination with embodiments and drawings.
[0023] <Example>
[0024] Figure 1 4 is a flow chart of the end-to-end pose estimation method for an articulated object in this embodiment.
[0025] like Figure 1 As shown, the method comprises the following steps:
[0026] Step S1, using a semantic segmentation model to segment the RGB-D information of the target articulated object from the input RGB-D image, and using the camera intrinsic parameters to map the RGB-D information back to the three-dimensional space to obtain the 3D point cloud of the target articulated object.
[0027] Step S2, inputting the 3D point cloud of the target articulated object into a component pose estimation model, thereby obtaining an estimated pose of the articulated component in the target articulated object.
[0028] The above steps will be described in detail below.
[0029] Step S1, using a semantic segmentation model to segment the RGB-D information of the target articulated object from the input RGB-D image, and using the camera intrinsic parameters to map the segmented RGB-D information of the target articulated object back to the three-dimensional space to obtain the 3D point cloud of the target articulated object.
[0030] The target articulated object refers to an object including at least one articulated component, such as a bucket with an articulated handle, a laptop computer with a flip cover, and the like.
[0031] In step S1, an RGB-D image (depth image containing color information) is obtained by taking a picture with an RGB-D camera. The model of the RGB-D camera is RealSense TM Depth Camera D435i. The semantic segmentation model is Grounded-SAM.
[0032] Step S2, inputting the 3D point cloud of the target articulated object into a pose estimation model, thereby obtaining a pose estimation of the articulated component in the target articulated object.
[0033] In step S2, the pose estimation model is a YOEO model, which includes a pre-trained point cloud feature extraction neural network model and three different prediction heads. In this embodiment, the point cloud feature extraction neural network model uses RandLa-Net. The three prediction heads all use multi-layer perceptrons (MLP).
[0034] Step S2 includes the following sub-steps:
[0035] Step S2-1, input the 3D point cloud of the target articulated object into the pre-trained point cloud feature extraction neural network model RandLa-Net to obtain the corresponding feature map.
[0036] The 3D point cloud of the target articulated object segmented by step S2 is recorded as (N, C in ), where N is the number of points in the point cloud, which is 20000 in this embodiment; C in is the input feature dimension, considering the xyz coordinates of the point cloud and the RGB information of the point cloud, that is, the feature dimension is 6. in ) Input the RandLa-Net model and obtain the corresponding feature map (N, 64).
[0037] In step S2-2, the feature map is input into three prediction heads respectively, and the semantic label, center offset (center of mass offset), and normalized part coordinate space (NPCS) area coordinates of each point in the corresponding 3D point cloud are predicted by the three prediction heads respectively.
[0038] Figure 2 It is a processing flow chart of the pose estimation model in this embodiment. From the figure, we can see the network structure of the YOEO model and its processing flow for the input feature map.
[0039] Step S2-3, based on the semantic label and center offset of each point, the point cloud segmentation result of each component of the target articulated object is obtained by using the Meanshift clustering algorithm, that is, the 3D point cloud of each component is segmented from the 3D point cloud of the target articulated object.
[0040] Step S2-4, aligning the segmented 3D point cloud of each component with the NPCS area coordinates using the Umeyama algorithm, thereby obtaining a pose estimate of each component, including a pose estimate of the articulated component.
[0041] The pre-training process of the above pose estimation model (YOEO model) includes the following steps:
[0042] Step SA1, initialize the model parameters (network parameters) of the pose estimation model, generally using normal distribution random initialization.
[0043] Step SA2: divide the data set into a training set, a validation set, and a test set, and further subdivide the test set and the validation set into two subsets: a subset of object categories that have been seen in the training set and a subset of object categories that have not been seen in the training set.
[0044] In this embodiment, the GAPartNet dataset is used, which contains 3D point clouds and annotations of various parts. The annotations include the NPCS region coordinates of each point in the 3D point cloud, the corresponding part category, and the corresponding part instance label. Both the test set and the validation set contain ten part categories.
[0045] Among them, the subset of object categories that have been seen in the training set is mainly used to evaluate the accuracy of the pose estimation model for the pose estimation of known articulated objects; the subset of object categories that have not been seen in the training set is used to test the generalization ability of the pose estimation model for the pose estimation of unknown articulated objects.
[0046] Step SA3: Input the training set into the pose estimation model in batches to obtain the output results, wherein the three prediction heads of the pose estimation model respectively output the semantic label, center offset, and NPCS area coordinates of each point in the 3D point cloud.
[0047] In step SA3, the training set is input in batches, and the batch size is set to 4.
[0048] Step SA4, combining the output result with the true value to calculate the value of the loss function to determine the gap between the output of the pose estimation model and the true value.
[0049] In step SA4, when calculating the value of the loss function, different loss functions are used for the three different prediction heads. Focal Loss is used as the loss function for the semantic prediction prediction head; L1 Loss is used as the loss function for the centroid offset prediction prediction head; for the NPCS coordinate prediction prediction head, the 0-1 coordinates of the input 3D point cloud are discretized into 100 classes with a step of 0.01, so that the regression problem becomes a classification problem, and Softmaxcross-entropy loss is used as the loss function. Finally, the loss functions corresponding to the three prediction heads are multiplied by the corresponding weights to obtain the final loss function.
[0050] Step SA5, determine whether a predetermined iteration stop condition is reached, where the loss after smoothing no longer decreases or the number of training rounds reaches 20 epochs. If the judgment in step SA5 is yes, the iteration stops and a trained pose estimation model is obtained.
[0051] Step SA6, when the judgment in step SA5 is no, calculate the gradient of the final loss function, and update the model parameters of the pose estimation model based on the calculated gradient, and return to step SA2, that is, input the next batch of training set data into the pose estimation model after updating the parameters, and continue the next round of training.
[0052] Figures 3 to 5 They are examples of the results of position estimation of an articulated object using the method of this embodiment. Figures 1 to 3 .
[0053] Figures 3 to 5 As shown, through the method of this embodiment, the hinged parts are accurately identified in the picture, for example Figure 3 The handle of the bucket was correctly identified and the 6D pose of the articulated part was obtained (the rectangular box in the figure).
[0054] Functions and Effects of the Embodiments
[0055] According to the end-to-end pose estimation method for articulated objects provided by this embodiment, semantic labels, centroid offsets, and NPCS regional representations are simultaneously estimated directly on the three-dimensional point cloud of the target articulated object through a unified network model, so that points belonging to the same component instance can vote for the same centroid, and a clustering algorithm is used based on the predicted centroid offset to distinguish points belonging to the same component instance, and then the final posture and size of each component are restored by aligning the NPCS regional representation of each component instance with the real point cloud, so that accurate 6D pose estimation of all articulated parts in the target articulated object can be obtained. In addition, since the method of this embodiment is a single-stage, end-to-end method, it is not necessary to first segment multiple component instances in the point cloud and then perform pose estimation like the existing multi-stage method, so the computational cost of the method of this embodiment is relatively low.
[0056] The method of this embodiment can be applied to robot grasping scenarios, in which single-stage, real-time, category-level 6D pose estimation of articulated objects is implemented, allowing the robot to better distinguish instances of articulated parts, thereby better locating and grasping articulated parts, avoiding damage to their delicate joints, and enabling the robot to have higher performance in corresponding real-time tasks.
[0057] The inventors deployed the above-mentioned pose estimation model trained with synthetic data in a real-world environment, providing real-time visual feedback at 200 Hz, enabling the physical Kinova robot to interact with unseen joint objects with high precision, and also demonstrating that the method of the embodiment has high utility and effectiveness in practical applications.
[0058] The above embodiments are only used to illustrate the specific implementation of the present invention, and the present invention is not limited to the description scope of the above embodiments. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. An end-to-end pose estimation method for an articulated object, characterized in that: The following steps are involved: Step S1, using a semantic segmentation model to segment the RGB-D image information of the target articulated object from the input RGB-D image, and using the camera intrinsic parameters to map the RGB-D image information back to the three-dimensional space to obtain the three-dimensional point cloud of the target articulated object; Step S2, inputting the three-dimensional point cloud of the target articulated object into a pose estimation model, thereby obtaining a pose estimation of the articulated component in the target articulated object, wherein the pose estimation model includes a point cloud feature extraction neural network model and three prediction heads, Wherein, step S2 includes the following sub-steps: Step S2-1, inputting the three-dimensional point cloud of the target articulated object into the point cloud feature extraction neural network model to obtain a corresponding feature map; Step S2-2, inputting the feature map into the prediction head, and respectively predicting the semantic label, center offset, and normalized part coordinate space region coordinates of each point in the three-dimensional point cloud of the target articulated object through the three prediction heads; Step S2-3, obtaining a three-dimensional point cloud of each component of the target articulated object by a clustering algorithm based on the semantic label and the center offset; Step S2-4, aligning the three-dimensional point cloud of the component with the normalized part coordinate space region coordinates, thereby obtaining a pose estimate of the component, including a pose estimate of the hinged component.
2. The end-to-end pose estimation method for an articulated object according to claim 1, characterized in that: in, The pose estimation model is the pre-trained YOEO, the point cloud feature extraction neural network model is RandLa-Net, and the prediction head is a multi-layer perceptron.
3. The end-to-end pose estimation method for an articulated object according to claim 2, characterized in that: in, In step S2-3, a three-dimensional point cloud of each component of the target articulated object is obtained by using a Meanshift clustering algorithm based on the semantic label and center offset of each point. In step S2-4, the three-dimensional point cloud of each component is aligned with the normalized part coordinate space region coordinates by using the Umeyama algorithm.
4. The end-to-end pose estimation method for an articulated object according to claim 2, Features: The training of the pose estimation model includes the following steps: Step SA1, initializing model parameters of the pose estimation model; Step SA2, dividing the data set into a training set, a validation set, and a test set, and further dividing the test set and the validation set into subsets of object categories seen and unseen in the training set, respectively; Step SA3, inputting the training set into the pose estimation model in batches to obtain output results; Step SA4, combining the output result with the true value to calculate the value of the loss function; Step SA5, determining whether a predetermined iteration stop condition is reached, and stopping the iteration when the condition is yes, to obtain the trained posture estimation model; Step SA6, when the judgment in step SA5 is no, calculate the gradient of the loss function, update the model parameters based on the gradient, and return to step SA2.
5. The end-to-end pose estimation method for an articulated object according to claim 4, characterized in that: in, In step SA3, the three prediction heads of the pose estimation model respectively output the semantic label, center offset, and normalized part coordinate space region coordinates of each point in the three-dimensional point cloud. In step SA4, Focal Loss is used as the loss function for the prediction head of semantic prediction, L1 Loss is used as the loss function for the prediction head of center offset, and for the prediction head of normalized part coordinate space region coordinates, the coordinates of the input three-dimensional point cloud are divided into multiple classes by discretization, and Softmax cross-entropy loss is used as the loss function. The loss functions corresponding to the three prediction heads are multiplied by the corresponding weights to obtain the final loss function.
6. The end-to-end pose estimation method for an articulated object according to claim 5, characterized in that: in, In step SA5, the iteration stopping condition is that the loss after smoothing no longer decreases or the number of training rounds reaches 20 epochs.
7. The end-to-end pose estimation method for an articulated object according to claim 4, characterized in that: in, The data set is the GAPartNet data set. The test set and the validation set respectively contain ten component categories.
8. The end-to-end pose estimation method for an articulated object according to claim 1, characterized in that: in, In step S1, the semantic segmentation model is Grounded-SAM.
