Class-level object 6D attitude estimation method based on shape prior
Through the 6D pose estimation method of class-level objects based on shape priors, the shape prior point cloud is generated and processed by using the RetinaNet and pointnet networks, and the problem of the traditional pose estimation network training for each object is solved, achieving efficient object pose recognition.
Patent Information
- Application Number
- CN202411964344.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-29
AI Technical Summary
Traditional pose estimation networks train a network individually for each object, which is time-consuming and unfavorable for actual use.
A 6D pose estimation method of class-level objects based on shape priors is used to detect objects through the RetinaNet network to generate local three-dimensional point clouds of objects, and a shape prior generator is trained using a pointnet encoder and decoder to generate a shape prior point cloud. Then, the shape prior point cloud and the object local point cloud are input to the pose prediction network, and a prior point cloud shape change and correspondence relationship are generated, and the actual 6D pose of the object is calculated through the Umeyama algorithm.
Predicting the 6D pose of a type of object in the same network improves the efficiency of object pose recognition and does not require the object's CAD model to be obtained.
Smart Images
Figure CN119904512A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of deep learning technology, and in particular relates to a 6D posture estimation method for class-level objects based on shape prior. Background Art
[0002] Posture estimation technology can help industrial robots identify the position and posture of objects, so as to accurately grasp, assemble or operate items. For example, on an assembly line, robots can automatically adjust the grasping angle through posture estimation technology to improve operation accuracy.
[0003] However, the traditional posture estimation network trains a separate network for each object for posture estimation, which is very time-consuming and not conducive to practical use. Summary of the invention
[0004] The purpose of the present invention is to provide a 6D pose estimation method for class-level objects based on shape prior to solve the above-mentioned technical problems.
[0005] In order to solve the above technical problems, the specific technical solution of a class-level object 6D posture estimation method based on shape prior of the present invention is as follows:
[0006] A method for estimating 6D pose of class-level objects based on shape priors comprises the following steps:
[0007] Step 1: Detect all objects that need to be predicted in the image based on the RetinaNet network and obtain the object mask map.
[0008] Further generate a local three-dimensional point cloud of the object;
[0009] Step 2: Train a shape prior generator based on a pointnet-like encoder, a fully connected layer decoder, and an existing 3D point cloud dataset;
[0010] Step 3: Input the shape prior point cloud, the segmented object local RGB image, and the object local point cloud into the posture prediction network to generate the shape change of the prior point cloud and the correspondence with the object local point cloud;
[0011] Step 4: Deform the prior point cloud according to the shape change and rotation change of the prior point cloud to obtain the normalized point cloud of the object;
[0012] Step 5: Use the Umeyama algorithm to align the normalized point cloud and the actual local point cloud of the object to obtain the actual 6D posture of the object.
[0013] Furthermore, in step 1, the original RGB image is input into the RetinaNet network to obtain the bounding box of the object to be detected and the object mask; the original RGB image is further cropped into a local RGB image I0 of the object, the depth of each pixel corresponding to the object mask is obtained from the depth information, and then the local object point cloud P0 is calculated according to the intrinsic parameter K of the camera.
[0014] Furthermore, the step 2 includes the following specific steps:
[0015] Step 2.1: Select the point cloud data of the object type to be detected from the shapeNet dataset, and scale the selected point cloud data to ensure that the diagonal length of its bounding box is 1 and the direction is consistent;
[0016] Step 2.2: Input the preprocessed point cloud into the encoder network to obtain the latent space vector z. Assume that the input point cloud is P, which contains n points. Input the n*3 point cloud into a shared MLP layer to obtain an n*128 feature vector. After passing through the maximum pooling layer, an intermediate feature vector z0 is generated. Then, z0 is concatenated with the previously obtained feature vector to generate an n*256 feature vector. After passing through a shared MLP layer, a maximum pooling layer, and a fully connected layer, a latent space vector z is generated. Step 2.3: Input the latent space vector z into the decoder to restore it to a three-dimensional point cloud. The decoder consists of three fully connected layers.
[0017] Step 2.4: Let the input point cloud be P, and the output point cloud after the network be P1, then the training loss function is:
[0018]
[0019] Where p is a point in point cloud P, and p1 is a point in point cloud P1;
[0020] Step 2.5: After the training converges, the point clouds in the training set are input into the encoder network to obtain the corresponding latent space vectors, and then these latent space vectors are averaged to generate the average latent space vector z ave , then z ave Input decoder network generates shape prior point cloud P r .
[0021] Furthermore, the step 3 includes the following specific steps:
[0022] Step 3.1: Extract the n*64 feature vector z corresponding to the local image of the object from the RetinaNet network I , extracted from the feature map with a larger resolution, where n is the number of points in the local point cloud of the object, that is, only the pixels with the object mask of 1 are extracted, and the local point cloud of the object and the prior point cloud are input into two MLP layers respectively to obtain the n*64 local point cloud feature vector z of the objectp0 and m*64 image feature vector z pr , m is the number of points in the prior point cloud;
[0023] Step 3.2: Transform the feature vector z I 、z p0 Concatenate and pass through an MLP layer and an average pooling layer to generate a global feature vector z ave1 , and then z ave1 、z I 、z p0 、z pr Concatenate into the final feature vector z fin ;
[0024] Step 3.3: The final feature vector z fin After an MLP layer, the prior point cloud shape change P is generated rc , P rc It is a vector of m*3, where m is the number of points in the prior point cloud;
[0025] Step 3.4: The final feature vector z fin After an MLP layer, the corresponding P of the local point cloud of the object is generated rh , P rh It is an n*m vector, where m is the number of points in the prior point cloud and n is the number of points in the local point cloud of the object.
[0026] Furthermore, the step 4 deforms the prior point cloud to obtain the normalized point cloud of the object:
[0027] P nocs =P rh *(P rc +P r )
[0028] Where P nocs is the normalized point cloud of the object.
[0029] The 6D posture estimation method of a class-level object based on shape prior of the present invention has the following advantages: The 6D posture estimation method of a class-level object based on shape prior of the present invention can predict the 6D posture of a class of objects in the same network, and does not need to obtain the CAD model of the object when it is actually used. It improves the efficiency of robot recognition of the position and posture of the object. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is the algorithm flow chart of the present invention;
[0031] Figure 2 Schematic diagram of posture estimation. DETAILED DESCRIPTION
[0032] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a class-level object 6D posture estimation method based on shape prior in the present invention in conjunction with the accompanying drawings.
[0033] like Figure 1 Figure 2 As shown, a method for estimating 6D posture of a class-level object based on shape prior of the present invention comprises the following steps:
[0034] Step 1: Detect all objects that need to be predicted in the image based on the RetinaNet network, obtain the object mask map, and further generate the local 3D point cloud of the object.
[0035] The original RGB image is input into the RetinaNet network to obtain the bounding box and mask of the object to be detected. The original RGB image is further cropped into the local RGB image I0 of the object. The depth of each pixel corresponding to the object mask can be obtained from the depth information, and then the local object point cloud P0 can be calculated based on the camera's intrinsic parameter K.
[0036] Step 2: Train a shape prior generator based on a pointnet-like encoder, a fully connected layer decoder, and an existing 3D point cloud dataset.
[0037] Step 2.1: Select the point cloud data of the object type to be detected from the shapeNet dataset, and scale the selected point cloud data to ensure that the diagonal length of its bounding box is 1 and the direction is consistent.
[0038] Step 2.2: Input the preprocessed point cloud into the encoder network to obtain the latent space vector z. Assume that the input point cloud is P, which contains n points. Input the n*3 point cloud into a shared MLP layer (fully connected layer) to obtain an n*128 feature vector. After passing through the maximum pooling layer, an intermediate feature vector z0 is generated. Then, z0 is concatenated with the previously obtained feature vector to generate an n*256 feature vector. After passing through a shared MLP layer, a maximum pooling layer, and a fully connected layer, a latent space vector z is generated.
[0039] Step 2.3: Input the latent space vector z into the decoder to restore it to a three-dimensional point cloud. The decoder consists of three fully connected layers.
[0040] Step 2.4: Let the input point cloud be P, and the output point cloud after the network be P1, then the training loss function is:
[0041]
[0042] Where p is a point in point cloud P, and p1 is a point in point cloud P1.
[0043] Step 2.5: After the training converges, the point clouds in the training set are input into the encoder network to obtain the corresponding latent space vectors, and then these latent space vectors are averaged to generate the average latent space vector z ave . Then z ave Input decoder network generates shape prior point cloud P r .
[0044] Step 3: Input the shape prior point cloud, the segmented local RGB image of the object, and the local point cloud of the object into the posture prediction network to generate the shape change of the prior point cloud and the correspondence with the local point cloud of the object.
[0045] Step 3.1: Extract the n*64 feature vector z corresponding to the local image of the object from the RetinaNet network I (Extracted from the feature map with a larger resolution), where n is the number of points in the local point cloud of the object, that is, only pixels with an object mask of 1 are extracted. The local point cloud of the object and the prior point cloud are input into two MLP layers respectively to obtain the n*64 local point cloud feature vector z of the object p0 and m*64 image feature vector z pr , m is the number of points in the prior point cloud.
[0046] Step 3.2: Transform the feature vector z I 、z p0 Concatenate and pass through an MLP layer and an average pooling layer to generate a global feature vector z ave1 , and then z ave1 、z I 、z p0 、z pr Concatenate into the final feature vector z fin .
[0047] Step 3.3: The final feature vector z fin After an MLP layer, the prior point cloud shape change P is generated rc , P rc It is an m*3 vector, where m is the number of points in the prior point cloud.
[0048] Step 3.4: The final feature vector z fin After an MLP layer, the corresponding P of the local point cloud of the object is generated rh , P rh It is an n*m vector, where m is the number of points in the prior point cloud and n is the number of points in the local point cloud of the object.
[0049] Step 4: Deform the prior point cloud according to the shape change and rotation change of the prior point cloud to obtain the normalized point cloud of the object.
[0050] P nocs =P rh*(P rc +P r )
[0051] Where P nocs is the normalized point cloud of the object.
[0052] Step 5: Use the Umeyama algorithm to align the normalized point cloud and the actual local point cloud of the object to obtain the actual 6D posture of the object.
[0053] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
Claims
1. A class-level object 6D pose estimation method based on shape prior, characterized in that: The steps include: Step 1: Detect all objects that need to be predicted in the image based on the RetinaNet network, obtain the object mask map, and further generate the local 3D point cloud of the object; Step 2: Train a shape prior generator based on a pointnet-like encoder, a fully connected layer decoder, and an existing 3D point cloud dataset; Step 3: Input the shape prior point cloud, the segmented object local RGB image, and the object local point cloud into the posture prediction network to generate the shape change of the prior point cloud and the correspondence with the object local point cloud; Step 4: Deform the prior point cloud according to the shape change and rotation change of the prior point cloud to obtain the normalized point cloud of the object; Step 5: Use the Umeyama algorithm to align the normalized point cloud and the actual local point cloud of the object to obtain the actual 6D posture of the object.
2. The method for class-level object 6D pose estimation based on shape prior according to claim 1, characterized in that: In the step 1, the original RGB image is input into the RetinaNet network to obtain the object frame and the object mask to be detected; the original RGB image is further cropped into a local RGB image I0 of the object, the depth of each pixel corresponding to the object mask is obtained from the depth information, and then the local object point cloud P0 is calculated according to the intrinsic parameter K of the camera.
3. The method for class-level object 6D pose estimation based on shape prior according to claim 1, characterized in that: The step 2 comprises the following specific steps: Step 2.1: Select the point cloud data of the object type to be detected from the shapeNet dataset, and scale the selected point cloud data to ensure that the diagonal length of its bounding box is 1 and the direction is consistent; Step 2.2: Input the preprocessed point cloud into the encoder network to obtain the latent space vector z. Assume that the input point cloud is P, which contains n points. Input the n*3 point cloud into a shared MLP layer to obtain an n*128 feature vector. After passing through the maximum pooling layer, an intermediate feature vector z0 is generated. Then, z0 is concatenated with the previously obtained feature vector to generate an n*256 feature vector. After passing through a shared MLP layer, a maximum pooling layer, and a fully connected layer, a latent space vector z is generated. Step 2.3: Input the latent space vector z into the decoder to restore it to a three-dimensional point cloud. The decoder consists of three fully connected layers. Step 2.4: Let the input point cloud be P, and the output point cloud after the network be P1, then the training loss function is: Where p is a point in point cloud P, and p1 is a point in point cloud P1; Step 2.5: After the training converges, the point clouds in the training set are input into the encoder network to obtain the corresponding latent space vectors, and then these latent space vectors are averaged to generate the average latent space vector z ave , then z ave Input decoder network generates shape prior point cloud P r .
4. The method for class-level object 6D pose estimation based on shape prior according to claim 1, characterized in that: The step 3 comprises the following specific steps: Step 3.1: Extract the n*64 feature vector z corresponding to the local image of the object from the RetinaNet network I , extracted from the feature map with a larger resolution, where n is the number of points in the local point cloud of the object, that is, only the pixels with the object mask of 1 are extracted, and the local point cloud of the object and the prior point cloud are input into two MLP layers respectively to obtain the n*64 local point cloud feature vector z of the object p0 and m*64 image feature vector z pr , m is the number of points in the prior point cloud; Step 3.2: Transform the feature vector z I 、z p0 Concatenate and pass through an MLP layer and an average pooling layer to generate a global feature vector z ave1 , and then z ave1 、z I 、z p0 、z pr Concatenate into the final feature vector z fin ; Step 3.3: The final feature vector z fin After an MLP layer, the prior point cloud shape change P is generated rc , P rc It is a vector of m*3, where m is the number of points in the prior point cloud; Step 3.4: The final feature vector z fin After an MLP layer, the corresponding P to the local point cloud of the object is generated rh , P rh It is an n*m vector, where m is the number of points in the prior point cloud and n is the number of points in the local point cloud of the object.
5. The method for class-level object 6D pose estimation based on shape prior according to claim 1, characterized in that: The step 4 deforms the prior point cloud to obtain the normalized point cloud of the object: P nocs =P rh *(P rc +P r ) Where P nocs is the normalized point cloud of the object.