A Category-Level Pose Estimation Algorithm Based on Point Transformer Networks

Abstract

This invention discloses a category-level pose estimation algorithm based on a Point Transformer network, relating to the field of pose estimation for 3D objects. The algorithm includes the following steps: Step 1, image acquisition; Step 2, point cloud instance segmentation and point cloud filtering to obtain point cloud data, and the filtered point cloud data is denoted as the workpiece point cloud; Step 3, translation and scaling estimation to obtain a preprocessed point cloud; Step 4, constructing a point cloud reconstruction network, using the preprocessed point cloud and template point cloud as input, extracting pose feature vectors and shape feature vectors, and fusing them to obtain a fused feature vector and a reconstructed point cloud; Step 5, iterative reconstruction of the point cloud; Step 6, point cloud registration, using the ICP algorithm to register the reconstructed point cloud and the template point cloud to obtain an estimated value of the rotation matrix; Step 7, obtaining a category-level pose estimation model through network training; Step 8, model prediction.

Description

Technical Field

[0001] This invention relates to the field of pose estimation for three-dimensional objects, and more particularly to a category-level pose estimation algorithm based on a Point Transformer network. Background Technology

[0002] Object pose estimation, which aims to estimate the position and orientation of a target object relative to a camera, has always been a key problem in 3D computer vision research. Object pose estimation is closely related to rapidly developing technologies such as robot grasping, autonomous driving, and augmented reality. In recent years, with the continued in-depth application of deep learning techniques to object pose estimation, instance-level six-degree-of-freedom (6DOF) pose estimation has made significant progress. However, most instance-level 6DOF pose estimation algorithms require an accurate CAD model of the target object in advance. This prerequisite largely limits the application scenarios of instance-level 6DOF pose estimation algorithms because, considering the diversity of models and the cost of modeling, it is often difficult or even impossible to obtain a CAD model of the target object. Against this backdrop, category-level object pose estimation has begun to attract increasing attention from researchers.

[0003] In category-level object pose estimation tasks, it is necessary to predict the position, orientation, and size of 9 degrees of freedom (9DoF), including rotation R∈SO(3), translation T∈R^3, and object size S∈R^3. Strictly speaking, category-level object pose estimation should be called 9DoF object pose estimation. For instance-level pose estimation, classic algorithms usually estimate the object's pose by establishing a correspondence between RGB or RGBD images and the CAD model of the target object. However, for category-level object pose estimation, there is no object CAD model that can be directly used, which makes it difficult for the algorithm to directly establish the correspondence. In addition, different objects within the same category often have significant differences in shape, color, size, and texture, that is, different objects within the same category have intra-class deformation. This situation is especially common in industrial metal parts, where the same type of industrial metal parts often have multiple specifications, materials, and textures. Therefore, traditional instance-level pose estimation methods are difficult to apply directly to category-level problems because category-level pose estimation tasks require modeling the intra-class deformation of different instances, rather than simply matching a specific CAD model. These characteristics place higher demands on the generalization performance of category-level pose estimation algorithms.

[0004] On the other hand, compared to pose estimation tasks for common household items, pose estimation tasks for industrial metal parts have unique characteristics. First, the RGB images of industrial metal parts are significantly affected by factors such as the direction and intensity of light incidence. Even slight changes in lighting conditions can cause noticeable changes in the RGB images of industrial metal parts. Second, the high reflectivity of the depth map of industrial parts makes it prone to depth loss in reflective areas. These characteristics place higher demands on the robustness of pose estimation algorithms for industrial metal parts. In addition, industrial metal parts are usually machined on lathes, and their structures are mostly composed of simple geometric structures such as cylinders and spheres. Therefore, in terms of geometry, industrial metal parts are more regular than common household items.

[0005] In their Chinese invention patent application "A Method and Apparatus for Object Pose Estimation Based on Bidirectional RGB-D Feature Fusion" (application number CN202211419607.4), Meng Qiwei et al. projected RGB and point cloud features onto a standard space using a bidirectional feature fusion module to achieve alignment, complementarity, and mutual enhancement, thereby improving the saliency and representativeness of the final RGB-D features. However, this method is still an instance-level pose estimation method, requiring an accurate 3D model of the object as a priori condition, which limits its application scenarios to some extent.

[0006] Therefore, those skilled in the art are dedicated to developing a new category-level pose estimation algorithm to address the aforementioned shortcomings in existing technologies. Summary of the Invention

[0007] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to complete the pose estimation of an object without requiring an accurate 3D model of the object, thereby improving the generalization ability of the pose estimation algorithm.

[0008] To achieve the above objectives, this invention provides a category-level pose estimation algorithm based on Point Transformer networks, comprising the following steps: Step 1: Perform image acquisition. Acquire RGB images and corresponding depth images of multiple types of workpieces under different shooting angles and lighting conditions, and align the acquired RGB images and depth images. Step 2: Perform point cloud instance segmentation and point cloud filtering. The segmentation of the multi-category workpieces in the depth image is achieved by segmenting the multi-category workpieces in the RGB image, obtaining point cloud data for the multi-category workpieces. The point cloud data is then filtered to remove outliers, resulting in higher-quality point cloud data. This filtered point cloud data is denoted as the workpiece point cloud. ; Step 3: Perform translation and scaling estimation, and then convert the workpiece point cloud into a single image. With template point cloud Feature vectors are extracted using the PointNet++ network. as well as Then as well as After concatenation, average pooling of the feature vectors and information extraction by a multilayer perceptron (MLP) are performed to obtain the estimated translation vector. and scaling For the point cloud of the workpiece Processing is performed to restore the workpiece point cloud. The translation and scaling are used to obtain the preprocessed point cloud. : Step 4: Construct a point cloud reconstruction network using the preprocessed point cloud. and the template point cloud As input, extract the preprocessed point cloud. pose feature vector and the template point cloud Shape feature vector The attitude feature vector and the shape feature vector go through Cross-attention network performs feature fusion to obtain fused feature vector. Furthermore, the fused feature vector The reconstructed point cloud is obtained after pooling and multilayer perceptron (MLP) processing. ; Step 5: Perform iterative reconstruction of the point cloud. During network training, the reconstructed point cloud... As a new preprocessed point cloud, it is compared with the pose feature vector. The shape feature vector Together, they are input again into the point cloud reconstruction network to regenerate the reconstructed point cloud. ; Step 6: Perform point cloud registration. Use the ICP algorithm to register the reconstructed point cloud with the template point cloud to obtain the estimated value of the rotation matrix. Step 7: Obtain a category-level pose estimation model through network training. The network training includes two completely decoupled parts: a lightweight network based on PointNet++ and a point cloud reconstruction network. The training set data is input into the network training and trained separately. Step 8: Perform model prediction. Use the trained category-level pose estimation model for inference. Input test set data and output the iteratively reconstructed point cloud and estimated pose information. Calculate the mAP and FPS of the category-level pose estimation model on the test set data. The reconstructed point cloud, template point cloud and workpiece point cloud can be visualized through the Open3D library to intuitively view the point cloud reconstruction effect.

[0009] Furthermore, the point cloud instance segmentation in step 2 includes the following sub-steps: Step 2.1: Perform instance segmentation on the RGB image using a two-dimensional image instance segmentation algorithm to obtain the mask of each of the multi-category workpieces in the RGB image; Step 2.2: Use the mask to extract the depth values ​​of each of the multi-category workpieces in the corresponding depth image; Step 2.3: Using the camera's intrinsic and extrinsic parameters, convert the depth value into the point cloud data of the corresponding workpiece.

[0010] Furthermore, the point cloud filtering in step 2 uses the remove_statistical_outlier() function in the Open3D library to remove outliers from the point cloud data using statistical methods.

[0011] Furthermore, in step 4, the Point Transformer block is used to process the preprocessed point cloud. Downsampling and feature extraction are performed, including the following sub-steps: Step 4.1: Use the farthest point sampling algorithm to sample the preprocessed point cloud. 512 points were sampled in the middle; Step 4.2: Calculate the feature vectors corresponding to the 512 points using the Point Transformer network. The feature vectors are 128-dimensional. Step 4.3: After two downsampling and feature vector calculations, a 128-dimensional global feature vector is obtained. The global feature vector is copied 128 times and concatenated with the local feature vectors corresponding to 256 points. After fusion by a multilayer perceptron (MLP), a preliminary fused feature vector is obtained. Step 4.4: Then, the coordinates of the 256 points are used as position codes. After being encoded by a multilayer perceptron (MLP), a 128-dimensional feature vector containing the position information of the point cloud is obtained. Step 4.5: The fused feature vector and the feature vector containing point cloud position information are fused through a multilayer perceptron (MLP) to obtain the 128-dimensional pose feature vector. .

[0012] Further, in step 4, for the template point cloud The shape feature vector can be obtained by using the same operations as steps 4.1 to 4.5. .

[0013] Furthermore, after obtaining the pose feature vector and the shape feature vector Then, through the feature mapping function, the pose feature vector is... Mapping the shape feature vector to a key vector K and a value vector V, Mapped to query vector Q; Subtract the key vector K from the query vector Q and encode the position using the feature mapping function. Adding them together and then fusing them through two MLP layers yields the unnormalized attention vector: Among them, position encoding It can be represented as , , , , Both are feature mapping functions; The unnormalized attention vector output is then passed through the Softmax function to obtain the attention vector. : The reconstructed point cloud is obtained by performing matrix multiplication between the attention vector and the value vector V, followed by aggregation of point cloud features using a feature mapping function. : in, , Both are feature mapping functions. The pose feature vector Mapped to the value vector V, The matrix product of the attention vector and the value vector V is mapped to the reconstructed point cloud. .

[0014] Furthermore, in step 5, the point cloud iterative reconstruction can be iterated several times; considering both the real-time performance of the algorithm and the accuracy of point cloud reconstruction, the specific number of reconstruction iterations is set to three. After three iterations, three reconstructed point clouds can be obtained: , , .

[0015] Furthermore, in the point cloud registration in step 6, the reconstructed point cloud is taken. With template point cloud ICP registration is performed to obtain an estimate of the rotation matrix. : use Reconstructing point clouds Perform rotational transformation to obtain the reconstructed point cloud in standard pose. And the reconstructed point cloud under the standard pose. Calculate its in The coordinate range in three directions can be used to obtain the dimensions of the multi-category workpieces in three directions.

[0016] Furthermore, in step 7, the loss function of the PointNet++-based lightweight network can be expressed as: in, To estimate the loss for the translation vector, To estimate the loss for scaling, L2 regression loss was used. The hyperparameter settings of the lightweight network based on PointNet++ are as follows: the workpiece point cloud and the template point cloud are uniformly sampled to 1024 points, the Adam optimizer is used for 30 rounds of training, the size of each batch is set to 16, the learning rate is initialized to 0.005, and adjusted by the cosine annealing algorithm.

[0017] Furthermore, in step 7, the loss function of the point cloud reconstruction network can be expressed as: in, , as well as The point cloud losses for the three reconstructions are respectively calculated using the chamfer distance as the loss function. For example, the calculation formula is as follows: in, This is the function for calculating chamfer distance. It is the truth value of the rotation matrix; The hyperparameters of the point cloud reconstruction network are set as follows: the workpiece point cloud and the template point cloud are uniformly sampled to 1024 points, the Adam optimizer is used for 50 rounds of training, the size of each batch is set to 16, the learning rate is initialized to 0.001, and the learning rate is adjusted by the cosine annealing algorithm.

[0018] The class-level pose estimation algorithm based on Point Transformer network provided by this invention has at least the following technical effects: 1. The technical solution provided by this invention completely decouples the translation vector and scaling estimation tasks from the rotation estimation tasks. By performing estimations through two separate networks, the difficulty of network estimation is reduced, and the mutual coupling and interference of a single network estimating multiple tasks are avoided. 2. The technical solution provided by this invention employs Point Transformer as the backbone network for point cloud reconstruction. Point Transformer networks can fully utilize global information when processing point cloud data, possessing advantages such as position invariance and applicability to irregular data. Therefore, it can effectively extract the required point cloud features in point cloud processing tasks. Furthermore, in the feature fusion module, the cross-attention mechanism of Point Transformer is utilized for feature fusion. Compared to traditional Transformer networks, Point Transformer can consider both global and local feature information, resulting in better feature fusion. 3. The technical solution provided by this invention proposes an iterative method for reconstructing target point clouds, which significantly improves the quality of point cloud reconstruction. Furthermore, the existing point cloud reconstruction network is reused in the iterative reconstruction process without increasing the network's complexity. Iterative reconstruction transforms point cloud reconstruction from a single-stage process to a multi-stage process, and the reconstruction quality gradually improves with each iteration. 4. The technical solution provided by this invention, through the improvements described above, enables the model to possess strong robustness and generalization ability, and improves the accuracy of category-level attitude estimation for metal workpiece point clouds. Furthermore, compared to instance-level attitude estimation algorithms, the technical solution provided by this invention does not require an accurate workpiece point cloud model for attitude estimation; it only requires a template point cloud of a specific category to which the workpiece point cloud belongs. This significantly improves the algorithm's generalization ability and reduces its usage cost.

[0019] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of a preferred embodiment of the category-level pose estimation algorithm of the present invention; Figure 2 for Figure 1 The diagram shows the structure of the translation and scaling estimation network in the embodiment shown. Figure 3 for Figure 1 The network structure diagram of the point cloud reconstruction network for shape and pose feature extraction in the illustrated embodiment; Figure 4 for Figure 1The illustrated embodiment shows a feature fusion network structure based on the Point Transformer cross-attention mechanism. Detailed Implementation

[0021] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0022] This invention provides a category-level pose estimation algorithm for industrial metal parts based on Point Transformer networks. It does not require an accurate 3D model of the object; it only requires a template point cloud of the same category of objects to complete the pose estimation, thereby improving the generalization ability of the pose estimation algorithm.

[0023] Specifically, firstly, a lightweight PointNet++ network is used to estimate the object's translation vector and scaling scale. Then, the PointNet++ estimation results are used to initially reconstruct the translation and size of the observed point cloud. The resulting point cloud and its corresponding template point cloud are then input into a feature extraction module to extract features from the observed and template point clouds, respectively. Next, the observed and template point cloud features are processed by a point cloud reconstruction module. This module utilizes the cross-attention mechanism of PointTransformer to fuse the shape features of the template point cloud with the pose features of the observed point cloud, reconstructing a point cloud with a pose consistent with the observed point cloud and a shape consistent with the template point cloud. The point cloud reconstruction process can be iterated multiple times to obtain a more accurate reconstructed point cloud. Finally, the ICP algorithm is used to register the reconstructed point cloud and the template point cloud to obtain an estimated rotation matrix. This rotation matrix estimation result is combined with the translation vector and scaling scale initially estimated by PointNet++ to obtain the final pose estimation result.

[0024] Example 1 like Figure 1 As shown, this embodiment of the invention provides a category-level pose estimation algorithm based on Point Transformer networks, including the following steps: Step 1: Perform image acquisition. Collect RGB images and corresponding depth images of various types of workpieces from different shooting angles and under different lighting conditions, and align the acquired RGB images and depth images.

[0025] Specifically, an RGB-D camera was used to acquire RGB images and corresponding depth images under different shooting angles and lighting conditions. The specific operation was as follows: various types of metal workpieces were placed on a worktable, and the RGB-D camera was mounted on the end effector of a robotic arm. The camera position was adjusted by the robotic arm to capture high-quality RGB and depth images. During image acquisition, the intensity of indoor lighting could be adjusted to simulate different lighting conditions, and multi-angle images were acquired by adjusting the robotic arm's posture. Furthermore, the acquired RGB and depth images were aligned using known information such as the camera's intrinsic and extrinsic parameters. This process ensured that the acquired images maintained high quality and diversity under various conditions, thus providing a reliable data foundation for subsequent analysis and processing.

[0026] Step 2: Perform point cloud instance segmentation and point cloud filtering. This involves segmenting multiple workpiece categories in the depth image by segmenting the RGB image, obtaining point cloud data for each category, and then filtering the point cloud data to remove outliers and obtain higher-quality point cloud data. The filtered point cloud data is then denoted as the workpiece point cloud. ; Step 3: Perform translation and scaling estimation. The category-level pose estimation task is decoupled into two parts: translation and scaling estimation and rotation estimation, to improve the efficiency and accuracy of the neural network. Compared to rotation estimation, translation and scaling estimation is simpler for neural networks; therefore, methods such as... Figure 2 The example shown is a lightweight network based on PointNet++ that estimates translation and scaling.

[0027] Specifically, the workpiece point cloud is divided into... With template point cloud Feature vectors are extracted using the PointNet++ network. as well as Then as well as After concatenation, average pooling of the feature vectors and information extraction by a multilayer perceptron (MLP) are performed to obtain the estimated translation vector. and scaling .

[0028] Specifically, PointNet++ is a deep learning model designed for 3D point cloud data. By introducing a hierarchical structure and local feature learning, it overcomes the limitations of the original PointNet network in handling local details and complex geometric structures. Its multi-scale feature extraction and adaptive resolution methods enable PointNet++ to perform exceptionally well when processing point cloud data of varying densities and complexities.

[0029] Obtain the estimated translation vector and scaling Then, the point cloud of the workpiece is analyzed. Process and restore the workpiece point cloud. The translation and scaling are used to obtain the preprocessed point cloud. : Preprocessing point clouds and template point cloud This will become the input for subsequent rotation estimation tasks.

[0030] Step 4: The rotation estimation task is mainly completed by the point cloud reconstruction network and the point cloud registration algorithm. A point cloud reconstruction network is constructed to preprocess the point cloud. and template point cloud As input, extract the preprocessed point cloud. pose feature vector and template point cloud Shape feature vector , attitude feature vector and shape feature vector go through Cross-attention network performs feature fusion to obtain fused feature vector. Furthermore, the feature vectors will be fused. The reconstructed point cloud is obtained after pooling and multilayer perceptron (MLP) processing. ; Step 5: To improve the quality of point cloud reconstruction, iterative point cloud reconstruction is performed. During network training, the reconstructed point cloud is... As a new preprocessed point cloud, it is combined with pose feature vectors Shape feature vector Together, they are input into the point cloud reconstruction network again to regenerate the reconstructed point cloud. This iterative point cloud reconstruction method significantly improves the quality of point cloud reconstruction because the acquired point cloud is a single-view point cloud. Compared to a complete template point cloud, a single-view point cloud can only carry partial pose information, while the reconstructed point cloud is a complete point cloud. During the point cloud reconstruction process, some pose information is recovered. Therefore, if the reconstructed point cloud... Re-inputting the data into the reconstruction network will further improve the reconstruction quality of the point cloud.

[0031] Step 6: Perform point cloud registration. The point cloud reconstruction network does not directly output the rotation matrix. The ICP algorithm is required to register the reconstructed point cloud with the template point cloud to obtain an estimate of the rotation matrix.

[0032] Step 7: Obtain a category-level pose estimation model through network training. Network training consists of two completely decoupled parts: a lightweight network based on PointNet++ and a point cloud reconstruction network, used to estimate translation vectors and scaling. The dataset is divided into training, test, and validation sets in a 7:2:1 ratio. The training set data is input into the network training and trained separately.

[0033] Step 8: Perform model prediction. Use the trained class-level pose estimation model for inference. Input test set data and output iteratively reconstructed point cloud and estimated pose information. Calculate the mAP and FPS of the class-level pose estimation model on the test set data. The reconstructed point cloud, template point cloud and workpiece point cloud can be visualized through the Open3D library to intuitively view the point cloud reconstruction effect.

[0034] Example 2 Based on Example 1, since the acquired RGB image and depth image are aligned, the workpiece in the depth image can be segmented by segmenting the workpiece in the two-dimensional RGB image.

[0035] Specifically, step 2, point cloud instance segmentation, includes the following sub-steps: Step 2.1: Use a two-dimensional image instance segmentation algorithm (such as YOLO v8) to perform instance segmentation on the RGB image to obtain the mask of each multi-category workpiece in the RGB image. Step 2.2: Use a mask to extract the depth values ​​of each multi-category workpiece in the corresponding depth image; Step 2.3: Using the camera's intrinsic and extrinsic parameters, convert the depth values ​​into point cloud data of the corresponding workpiece.

[0036] Specifically, due to errors introduced by instance segmentation algorithms and camera capture processes, the acquired raw workpiece point cloud often contains outliers. To improve point cloud quality, filtering is necessary. Specifically, the `remove_statistical_outlier()` function from the Open3D library is used to remove outliers from the point cloud using statistical methods, thereby obtaining higher-quality workpiece point cloud data. This process ensures the accuracy and reliability of the point cloud data, laying a solid foundation for subsequent analysis and applications.

[0037] Example 3 Based on Example 1 or 2, the extraction of pose and shape features, such as Figure 3 As shown, the preprocessed point cloud and template point cloud First, input the feature extraction module to extract pose and shape features. In step 4, the Point Transformer block is used to preprocess the point cloud. Downsampling and feature extraction are performed, including the following sub-steps: Step 4.1: Use the farthest point sampling algorithm to sample the preprocessed point cloud. 512 points were sampled in the middle; Step 4.2: Calculate the feature vectors corresponding to the 512 points using the Point Transformer network. The feature vectors are 128-dimensional. Step 4.3: After two downsampling and feature vector calculations, a 128-dimensional global feature vector is obtained. The global feature vector is copied 128 times and concatenated with the local feature vectors corresponding to 256 points. After fusion by a multilayer perceptron (MLP), a preliminary fused feature vector is obtained. Step 4.4: Then, the coordinates of the 256 points are used as position codes. After being encoded by a multilayer perceptron (MLP), a 128-dimensional feature vector containing the position information of the point cloud is obtained. Step 4.5: The fused feature vector and the feature vector containing point cloud position information are fused through a multilayer perceptron (MLP) to obtain a 128-dimensional pose feature vector. .

[0038] For template point cloud The shape feature vector can be obtained by using the same operations as steps 4.1 to 4.5. .

[0039] Feature fusion and point cloud reconstruction networks accept pose feature vectors Shape feature vector and preprocessed point clouds As input, the output is a reconstructed point cloud. .

[0040] Specifically, such as Figure 4 As shown, after obtaining the pose feature vector and shape feature vector Afterwards, the pose feature vector is processed by the feature mapping function (MLP). Mapping the shape feature vector to a key vector K and a value vector V, and converting the shape feature vector... Mapped to query vector Q. The key vector K is subtracted from the query vector Q and then encoded using the position mapped by the feature mapping function (MLP). Adding them together and then fusing them through two MLP layers yields the unnormalized attention vector: Among them, position encoding It can be represented as , , , , Both are feature mapping functions; The unnormalized attention vector output is then passed through the Softmax function to obtain the attention vector. : The attention vector and value vector V are multiplied by a matrix, and then the point cloud features are aggregated using a feature mapping function to obtain the reconstructed point cloud. : in, , Both are feature mapping functions. pose feature vector Mapped to a value vector V, The matrix product of the attention vector and the value vector V is mapped to the reconstructed point cloud. .

[0041] Example 4 Based on Examples 1, 2, or 3, in step 5, the point cloud iterative reconstruction can be iterated several times. However, increasing the number of iterations will affect the real-time performance of the algorithm. Considering both the real-time performance and the accuracy of point cloud reconstruction, the specific number of reconstruction iterations is set to three. After three iterations, three reconstructed point clouds can be obtained: , , .

[0042] In the point cloud registration in step 6, the reconstructed point cloud is taken. With template point cloud ICP registration is performed to obtain an estimate of the rotation matrix. : use Reconstructing point clouds Perform rotational transformation to obtain the reconstructed point cloud in standard pose. And reconstruct point clouds under standard pose. Calculate its in The coordinate range in three directions allows us to obtain the dimensions of various types of workpieces in three directions.

[0043] Example 5 Building upon Examples 1, 2, 3, or 4, in step 7, firstly, a lightweight network based on PointNet++ is trained. The loss function of the lightweight network based on PointNet++ can be expressed as: in, To estimate the loss for the translation vector, To estimate the loss for scaling, L2 regression loss was used. The hyperparameter settings for the lightweight network based on PointNet++ are as follows: the workpiece point cloud and the template point cloud are uniformly sampled to 1024 points, the Adam optimizer is used for 30 rounds of training, the batch size is set to 16, the learning rate is initialized to 0.005, and adjusted by the cosine annealing algorithm.

[0044] Then, the point cloud reconstruction network is trained. In step 7, the loss function of the point cloud reconstruction network can be expressed as: in, , as well as The point cloud losses for the three reconstructions are respectively calculated using the chamfer distance as the loss function. For example, the calculation formula is as follows: in, This is the function for calculating chamfer distance. It is the truth value of the rotation matrix; The hyperparameters of the point cloud reconstruction network are set as follows: the workpiece point cloud and the template point cloud are uniformly sampled to 1024 points, the Adam optimizer is used for 50 rounds of training, the size of each batch is set to 16, the learning rate is initialized to 0.001, and adjusted by the cosine annealing algorithm.

[0045] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A category-level pose estimation algorithm based on Point Transformer networks, characterized in that, The algorithm includes the following steps: Step 1: Perform image acquisition. Acquire RGB images and corresponding depth images of multiple types of workpieces under different shooting angles and lighting conditions, and align the acquired RGB images and depth images. Step 2: Perform point cloud instance segmentation and point cloud filtering. The segmentation of the multi-category workpieces in the depth image is achieved by segmenting the multi-category workpieces in the RGB image, obtaining point cloud data for the multi-category workpieces. The point cloud data is then filtered to remove outliers, resulting in higher-quality point cloud data. This filtered point cloud data is denoted as the workpiece point cloud. ; Step 3: Perform translation and scaling estimation, and then convert the workpiece point cloud into a single image. With template point cloud Feature vectors are extracted using the PointNet++ network. as well as Then as well as After concatenation, average pooling of the feature vectors and information extraction by a multilayer perceptron (MLP) are performed to obtain the estimated translation vector. and scaling For the point cloud of the workpiece Processing is performed to restore the workpiece point cloud. The translation and scaling are used to obtain the preprocessed point cloud. : Step 4: Construct a point cloud reconstruction network using the preprocessed point cloud. and the template point cloud As input, extract the preprocessed point cloud. pose feature vector and the template point cloud Shape feature vector The attitude feature vector and the shape feature vector go through Cross-attention network performs feature fusion to obtain fused feature vector. Furthermore, the fused feature vector The reconstructed point cloud is obtained after pooling and multilayer perceptron (MLP) processing. ; Step 5: Perform iterative reconstruction of the point cloud. During network training, the reconstructed point cloud... As a new preprocessed point cloud, it is compared with the pose feature vector. The shape feature vector Together, they are input again into the point cloud reconstruction network to regenerate the reconstructed point cloud. ; Step 6: Perform point cloud registration. Use the ICP algorithm to register the reconstructed point cloud with the template point cloud to obtain the estimated value of the rotation matrix. Step 7: Obtain a category-level pose estimation model through network training. The network training includes two completely decoupled parts: a lightweight network based on PointNet++ and a point cloud reconstruction network. The training set data is input into the network training and trained separately. Step 8: Perform model prediction. Use the trained category-level pose estimation model for inference. Input test set data and output the iteratively reconstructed point cloud and estimated pose information. Calculate the mAP and FPS of the category-level pose estimation model on the test set data. The reconstructed point cloud, template point cloud and workpiece point cloud can be visualized through the Open3D library to intuitively view the point cloud reconstruction effect.

2. The category-level pose estimation algorithm based on Point Transformer network as described in claim 1, characterized in that, Step 2, point cloud instance segmentation, includes the following sub-steps: Step 2.1: Perform instance segmentation on the RGB image using a two-dimensional image instance segmentation algorithm to obtain the mask of each of the multi-category workpieces in the RGB image; Step 2.2: Use the mask to extract the depth values ​​of each of the multi-category workpieces in the corresponding depth image; Step 2.3: Using the camera's intrinsic and extrinsic parameters, convert the depth value into the point cloud data of the corresponding workpiece.

3. The category-level pose estimation algorithm based on Point Transformer network as described in claim 1, characterized in that, The point cloud filtering in step 2 uses the remove_statistical_outlier() function in the Open3D library to remove outliers from the point cloud data using statistical methods.

4. The category-level pose estimation algorithm based on Point Transformer network as described in claim 1, characterized in that, In step 4, the Point Transformer block is used to process the preprocessed point cloud. Downsampling and feature extraction are performed, including the following sub-steps: Step 4.1: Use the farthest point sampling algorithm to sample the preprocessed point cloud. 512 points were sampled in the middle; Step 4.2: Calculate the feature vectors corresponding to the 512 points using the Point Transformer network. The feature vectors are 128-dimensional. Step 4.3: After two downsampling and feature vector calculations, a 128-dimensional global feature vector is obtained. The global feature vector is copied 128 times and concatenated with the local feature vectors corresponding to 256 points. After fusion by a multilayer perceptron (MLP), a preliminary fused feature vector is obtained. Step 4.4: Then, the coordinates of the 256 points are used as position codes. After being encoded by a multilayer perceptron (MLP), a 128-dimensional feature vector containing the position information of the point cloud is obtained. Step 4.5: The fused feature vector and the feature vector containing point cloud position information are fused through a multilayer perceptron (MLP) to obtain the 128-dimensional pose feature vector. .

5. The category-level pose estimation algorithm based on Point Transformer network as described in claim 4, characterized in that, In step 4, for the template point cloud The shape feature vector can be obtained by using the same operations as steps 4.1 to 4.

5. .

6. The category-level pose estimation algorithm based on Point Transformer network as described in claim 5, characterized in that, After obtaining the posture feature vector and the shape feature vector Then, through the feature mapping function, the pose feature vector is... Mapping the shape feature vector to a key vector K and a value vector V, Mapped to query vector Q; Subtract the key vector K from the query vector Q and encode the position using the feature mapping function. Adding them together and then fusing them through two MLP layers yields the unnormalized attention vector: Among them, position encoding It can be represented as , , , , Both are feature mapping functions; The unnormalized attention vector output is then passed through the Softmax function to obtain the attention vector. : The reconstructed point cloud is obtained by performing matrix multiplication between the attention vector and the value vector V, followed by aggregation of point cloud features using a feature mapping function. : in, , Both are feature mapping functions. The pose feature vector Mapped to the value vector V, The matrix product of the attention vector and the value vector V is mapped to the reconstructed point cloud. .

7. The category-level pose estimation algorithm based on Point Transformer network as described in claim 1, characterized in that, In step 5, the point cloud iterative reconstruction can be performed several times. Considering both the real-time performance of the algorithm and the accuracy of point cloud reconstruction, the specific number of reconstruction iterations is set to three. After three iterations, three reconstructed point clouds can be obtained: , , .

8. The category-level pose estimation algorithm based on Point Transformer network as described in claim 7, characterized in that, In the point cloud registration in step 6, the reconstructed point cloud is taken. With template point cloud ICP registration is performed to obtain an estimate of the rotation matrix. : use Reconstructing point clouds Perform rotational transformation to obtain the reconstructed point cloud in standard pose. And the reconstructed point cloud under the standard pose. Calculate its in The coordinate range in three directions can be used to obtain the dimensions of the multi-category workpieces in three directions.

9. The category-level pose estimation algorithm based on Point Transformer network as described in claim 1, characterized in that, In step 7, the loss function of the PointNet++-based lightweight network can be expressed as: in, To estimate the loss for the translation vector, To estimate the loss for scaling, L2 regression loss was used. The hyperparameter settings of the lightweight network based on PointNet++ are as follows: the workpiece point cloud and the template point cloud are uniformly sampled to 1024 points, the Adam optimizer is used for 30 rounds of training, the size of each batch is set to 16, the learning rate is initialized to 0.005, and adjusted by the cosine annealing algorithm.

10. The category-level pose estimation algorithm based on Point Transformer network as described in claim 7, characterized in that, In step 7, the loss function of the point cloud reconstruction network can be expressed as: in, , as well as The point cloud losses for the three reconstructions are respectively calculated using the chamfer distance as the loss function. For example, the calculation formula is as follows: in, This is the function for calculating chamfer distance. It is the truth value of the rotation matrix; The hyperparameters of the point cloud reconstruction network are set as follows: the workpiece point cloud and the template point cloud are uniformly sampled to 1024 points, the Adam optimizer is used for 50 rounds of training, the size of each batch is set to 16, the learning rate is initialized to 0.001, and the learning rate is adjusted by the cosine annealing algorithm.

Images