Posture estimation method based on multi-class object dynamic key point learning and progressive optimization

Abstract

The invention discloses an attitude estimation method based on multi-class object dynamic key point learning and progressive optimization. The attitude estimation method comprises the following steps: carrying out outlier filtering on dense point clouds converted from depth images by adopting 3D image convolution to obtain filtered point clouds; obtaining key points and feature representation of the object by using the filtered point cloud coordinates, the RGB features of the object and the geometric features of the filtered point cloud through a multi-modal key point querier; dense point cloud geometric features are extracted by using a dynamic graph convolutional network, and in combination with local multi-level features based on key points and a feature reconstruction model, the feature expression ability in a complex scene is enhanced by using attention weighted fusion; and finally, through a progressive optimization module, on the basis of a pose residual prediction network and a parameter dynamic updating mechanism, object pose information is output after three times of iterative computation. According to the method, a dynamic key point learning mechanism based on three-dimensional geometric feature guidance is constructed, and the generalization ability and geometric perception precision of attitude estimation are effectively improved by fusing multi-modal perception information and an iterative optimization strategy.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and object pose estimation, and particularly relates to a pose estimation method based on multi-class object dynamic key point learning and progressive optimization. Background Art

[0002] In recent years, as a core function of robot applications, robot grasping highly depends on computer vision technology. Computer vision methods based on deep learning provide a new solution for robot grasping detection, enabling robots to accurately identify grasping targets through visual information. In the key link of visual information processing and grasping pose generation, deep learning-based technologies have shown significant advantages. Especially in the field of six-dimensional object pose estimation, by synchronously acquiring three-dimensional coordinates and normal vectors through RGB-D data, not only can the Cartesian space position be accurately calculated, but also the rotation angle around the axis can be predicted, providing a complete pose parameter matrix for robotic arm motion planning and realizing precise control of robot grasping.

[0003] In complex real-world scenarios, to balance model generalization, current pose estimation is mainly divided into two categories: instance-level and class-level. Although instance-level pose estimation methods can accurately locate known objects through fine-grained modeling, they have obvious limitations: The corresponding-based methods highly depend on object surface texture features. When dealing with textureless or weakly textured objects, 2D-3D key point matching and 3D-3D point cloud registration are difficult, resulting in a significant decrease in estimation accuracy. The template-based methods solve the texture dependence problem, but they need to pre-build a multi-view annotation template library, with high computational complexity, difficult to apply in real time, and poor template generalization ability and system scalability, becoming the core bottleneck for their promotion.

[0004] Class-level 6D pose estimation constructs a class-level canonical representation (such as the NOCS coordinate space), getting rid of the dependence on instance-level CAD models and successfully realizing generalization prediction for unseen instances. However, existing methods have many problems. On the one hand, they rely on shape prior deformation and feature matching, but it is difficult to effectively model the local geometric differences and global structural associations of intra-class instances. Facing objects with large shape variations (such as non-rigid deformation or topological structure changes), the generalization ability is greatly reduced. On the other hand, mainstream methods highly depend on large-scale fully supervised annotation data and often use complex multi-modal feature fusion or prior point cloud matching to improve accuracy, which not only increases the data acquisition cost but also brings a heavy computational burden, limiting real-time applications. In addition, some methods overly rely on specific category assumptions, weakening the cross-category generalization potential. In actual scenarios, object shapes are complex, and it is costly to comprehensively and finely describe all details. Using sparse key points to represent shapes can capture key features in a concise manner, reduce data redundancy, lower computational costs, and effectively handle shape variations, enhancing the model's generalization ability. Summary of the Invention

[0005] For the problems described above and various deficiencies existing in the current technology, the present invention proposes a more generalizable pose estimation method based on multi-class object dynamic key point learning and progressive optimization.

[0006] To achieve the above object, the present invention proposes a pose estimation method based on multi-class object dynamic key point learning and progressive optimization, and the method includes:

[0007] Step 1: Obtain an RGB-D image through a binocular camera or a depth camera, and then use Mask R-CNN to detect and segment the RGB image and the corresponding depth map of the target object based on the RGB-D image and its mask image using the internal and external parameters of the camera.

[0008] Step 2: For the RGB image segmented in the previous step, use lightweight ResNet-18 as the backbone network of PSPNet, fuse multi-scale context information through the pyramid pooling module, restore it to the original resolution through upsampling, and extract the C-dimensional RGB feature F rgb ; Subsequently, convert the depth map into point cloud data containing 1024 points. Adding a small amount of noise information to the original image during the training process can enhance the robustness of the model to complex scenes and images under strong light or weak light interference.

[0009] Step 3: Since the geometric information contained in each point in the point cloud is relatively local, fusing as much multi-scale information as possible for each point can enhance the accuracy of the pose. However, the number of points in the point cloud is too large and there are outlier points. It is necessary to reduce the number of points in the original point cloud while retaining more original object information. Use 3DGCN to calculate 512 points with the most retained information through Euclidean distance and similarity.

[0010] Step 4: Use DGCNN to extract the feature F from the original point cloud dgcnn and the RGB feature F extracted by the PSPNet network rgb are fused through an attention mechanism to obtain a global feature. Based on the above-selected 512 points, a learnable query is set to obtain 100 most information-dense local key points and features. Based on the local key points, reconstruction is performed, and PointNet++ is used to extract the feature F of the point cloud reconstructed by the key points point and then fuse it with the RGB feature F rgb using a self-attention mechanism to fuse again to obtain the local feature F local and finally weighted fusion into the final feature F final and the six-degree-of-freedom information restored from the final feature is obtained through a multi-layer perceptron.

[0011] Step 5: Through the predicted pose information, i.e., the rotation matrix \(R\in SO(3)\), the translation vector \(t\in\mathbb{R}\), 3 and the scaling factor \(s\in\mathbb{R}\); render the point cloud of the object model based on the predicted pose, and calculate the symmetric Chamfer distance between it and the real point cloud as the residual; during the training process, dynamically adjust the number of iterations through the comparison of losses and the gap between the current six-degree-of-freedom information and the label, and use the Adam optimizer to update the pose parameters to solve the least squares solution to update the pose parameters.

[0012] Step 6: According to the relationship between the world coordinate system and the NOCS space, the object in the NOCS space can be transformed into the world coordinate system to obtain the pose estimation of the object. Finally, the reconstructed model will first go through NOCS prediction and then obtain the real pose coordinates based on the NOCS prediction.

[0013] Step 7: When querying key points, the diversity loss \(L\) div is needed to inal constrain the positions of the key points. Based on the final feature \(F\) final and the point cloud \(P\) cd obtained by reconstruction, use the chamfer loss \(L\) optimize to continuously adjust during the iteration process to obtain the optimized point cloud \(P\). optimize For the optimized \(P\), nocs it is also necessary to further project it into the NOCS space and perform detailed adjustment by the loss function \(L\). pose Finally, for the six-degree-of-freedom information obtained from the optimized model, an attitude loss \(L\) total is also needed. Finally, the total loss function is expressed as: \(L\) div =\(\lambda_1L\) cd +\(\lambda_2L\) pose +\(\lambda_3L\) nocs .

[0014] It should be noted that Step 1 mainly includes:

[0015] 1.1: For the RGB-D images obtained by the depth camera, it is necessary to perform label partitioning on the objects existing in the images. The label partitioning includes object category labels and pixel-level mask annotations. Among them, the mask is a binary image, and the annotated area corresponds to the pixel positions of the object in the RGB-D image; then use Mask R-CNN to detect and segment the objects in the RGB-D images according to the corresponding label partitioning.

[0016] 1.2: Obtain the RGB image and depth image of the segmented object, and then use the internal parameters of the camera corresponding to the RGB-D image to convert the depth image into the corresponding point cloud \(P\) o represents the point cloud converted from the depth map, and \(N\) oIndicates the number of points contained in the point cloud; the point cloud transformation formula is N is the number of point clouds.

[0017] 1.3: During the training phase, appropriate Gaussian noise will be added to the RGB images and point clouds of a single object for data augmentation to make the training process more robust.

[0018] More specifically, the ResNet-18 in step 2 as the backbone network structure of the PSPNet network is:

[0019] 2.1: Pyramid pooling module, multiple upsampling operations, and Dropout operations, where the pyramid pooling module contains four parallel branches for pooling, and the output features are compressed by 1×1 convolution and then concatenated, and then restored to the input resolution through three upsamplings;

[0020] More specifically, the methods for obtaining RGB features and point cloud features in step 2 are mainly:

[0021] 2.2: For RGB images and point cloud data, preliminary features are obtained through convolution, pooling, and fully connected operations and used as the input for the next layer, and then through convolution, pooling, and fully connected operations until the final RGB features and point cloud features are obtained using the last layer of the network, and the dimensions of both are the same;

[0022] 2.3: The output of the PSPNet network is 128-dimensional RGB features; the output of the DGCNN network is 128-dimensional point cloud features containing more global information.

[0023] More specifically, the outlier point screening rules in step 3 are:

[0024] 3.1: Extract RGB features from the image, map the RGB information point by point to the point cloud according to the distribution of points in the point cloud, and then use the 3DGCN layer based on the geometric relationship between the point cloud with RGB information and the original point cloud.

[0025] 3.2: Use RGB information and geometric information as the selection conditions. The core of 3DGCN is KNN calculation: for each point, calculate its Euclidean distance and cosine similarity with all other points, and select the nearest 20 neighbors. Calculate the cosine similarity + Euclidean distance of the information contained in the points in the point cloud through 3DGCN, and screen out the 512 points with the highest correlation; the current point P i and the template point P j The Euclidean distance formula is d ij =||P i -P j ||2( is the coordinate of the current point, The cosine similarity calculation formula for the template point coordinates is ( is the feature extracted by 3DGCN).

[0026] 3.3: Score according to the relationship between each point and its adjacent points, and screen 512 points in descending order of the comprehensive score to form a new point cloud P r represents the point cloud after removing outliers, and N r represents the number of points included in the new point cloud;

[0027] More specifically, the process of weighted fusion of the global feature and the local feature in step 4 is as follows:

[0028] 4.1: The network structure of PointNet++ is as follows: the target number of points for farthest point sampling (FPS) is 1000, the spherical neighborhood grouping radii are 0.05m, 0.1m, and 0.2m in sequence, each group contains 32 neighborhood points, and then the feature F is output through MLP point .

[0029] 4.2: Use the point cloud converted from the depth image to extract the feature F through DGCNN dgcnn and the RGB feature F rgb After the attention operation and then splicing to obtain the global feature F that retains the most original information global ; The DGCNN dynamic graph convolution can enhance the use of neighborhood correlation. The attention operation is defined as: where Q, K, and V are the query, key, and value features respectively, generated by linear transformation, and d k is the feature dimension.

[0030] 4.3: A set of learnable key point queryers are set for each object in the model to learn the feature representation of the current object. Use the previously obtained point cloud P r and the RGB feature F rgb and the feature r extracted by PointNet++ for P as the conditions for the key point queryer to learn. Then, based on the cross-attention of Transformer, learn the point and feature distribution in the current queryer, and then calculate the cosine similarity with P r to find 100 key points for reconstruction, and then extract the local structure feature F by PointNet++ point , and the feature dimension here should be the same as the RGB feature F rgb obtained previously; the point cloud feature F point and then the RGB feature F rgbStrengthen the internal feature correlation of each through the self-attention mechanism, and then splice them to enhance the attention weight of the strongly correlated regions between the two in the spatial dimension.

[0031] 4.4: Feature Weighted Fusion Feature weighted fusion is automatically adjusted through model training to enable F local and F global to achieve multi-level feature interaction and construct a multi-modal fusion representation with enhanced geometric structure information; the feature weighted fusion formula is: F final = ω local ·F local + ω global ·F global .

[0032] More specifically, step 5 mainly includes:

[0033] 5.1: Concatenate the global feature F global and the local feature F local obtained through the above steps, and then pass through a multi-layer perceptron to obtain the pose information of the unknown object, and use the least squares method to solve for the rotation matrix of the model relative to the camera coordinate system. The rotation matrix contains the three-dimensional rotation R, three-dimensional translation t, and scaling factor s information of the object. When solving the rotation matrix using the least squares method, the SVD decomposition is used to find the closed-form solution: R = VU T where

[0034] 5.2: Obtaining the three-dimensional rotation R, three-dimensional translation t, and scaling factor s information of the object can obtain the object pose. Add the offset value obtained by passing the model reconstructed using the final feature through a multi-layer perceptron to the final feature, and reconstruct based on the new final feature to obtain new R1, t1, and s1. Here, the pose is iteratively updated through the progressive optimization block until the optimal number of iterations is found or the maximum number of iterations is reached and then stopped, making the model more accurate;

[0035] More specifically, step 6 mainly includes:

[0036] 6.1: According to the relationship between the world coordinate system and the NOCS space, use the true pose information of the object to transform it into the pose in the NOCS space to obtain the representation of the true object in the standard normalized space;

[0037] 6.2: According to its pose information in the NOCS space, a more unified representation of the pose information of different objects can be obtained. After adding constraints for training in this space, project the coordinates in the NOCS space onto the world coordinate system to obtain the true coordinate information.

[0038] More specifically, step 7 mainly includes:

[0039] 7.1: When performing key point queries, in order to prevent the key points queried for each object from clustering or overlapping, it is necessary to constrain the positions of the key points. Therefore, a diversity loss needs to be used, and the expression is as follows:

[0040]

[0041] Among them, N k represents the number of key points, and P k represents the point cloud composed of key points.

[0042] 7.2: Based on the final feature F final The point cloud P final reconstructed is adjusted by the chamfer loss L cd to obtain the optimized point cloud P optimize . The expression of the chamfer loss is:

[0043]

[0044] Among them, P obj represents the true point cloud of the object, which can be derived from the true pose in the dataset. P final is the model reconstructed from the final feature F final ;

[0045] 7.3: To obtain the six-degree-of-freedom information from the model P optimize during the optimization process, it is necessary to further project it into the NOCS space and adjust the details by the loss function L nocs . The expression of this loss is as follows:

[0046]

[0047] Among them represents the model in the NOCS space derived from the true pose, M obj represents the point cloud extracted from the CAD model of the same type as the current object, S gt represents the true scaling factor of the object, R gt is the true rotation matrix information of the object, th2 represents the set threshold, and t gt represents the true translation vector of the object;

[0048] 7.4: In addition, an attitude loss is added to the six-degree-of-freedom information obtained from the optimized model, and the expression is as follows:

[0049] L pose = ||R gt - R||2 + ||t gt - t||2 + ||S gt - S||2;

[0050] Among them, R, t, and S are the predicted pose information obtained based on the optimized reconstruction model;

[0051] 7.5: The final total loss function is expressed as:

[0052] L total =λ1L div +λ2L cd +λ3L pose +λ4L nocs ;

[0053] where λ1, λ2, λ3 and λ4 are the diversity loss L div , chamfer loss L cd , NOCS loss L nocs And the posture loss L pose The weight parameter of .

[0054] The beneficial effects of the present invention are:

[0055] By leveraging the complementary information of multimodal data (RGB and point cloud), the model can effectively capture the geometric structure and texture details of objects. The use of multi-level feature fusion design and a reasonable loss function further improves the robustness and accuracy of the model in complex scenes.

[0056] The model combines the point cloud feature extraction capabilities of PointNet++ and DGCNN, as well as multi-level feature fusion. By effectively integrating local and global features, it fully utilizes the complementary information of RGB and point cloud data, and can achieve high precision in pose estimation and key point prediction tasks. At the same time, the pose optimization module further optimizes the pose estimation results and enhances the model's adaptability to complex scenes. This modular design not only improves the generalization ability, making it suitable for synthetic and real datasets of different scales, but also, through flexible attention mechanisms and optimization strategies, it balances performance while maintaining computational efficiency, providing robustness and scalability for 3D vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0058] Figure 1 This is a flowchart of a posture estimation method based on dynamic key point learning and progressive optimization of multiple types of objects proposed by the present invention;

[0059] Figure 2The framework diagram of a pose estimation method based on the learning and progressive optimization of dynamic key points of multiple types of objects proposed by the present invention;

[0060] Figure 3 The structural diagram of cross-attention for a pose estimation method based on the learning and progressive optimization of dynamic key points of multiple types of objects proposed by the present invention;

[0061] Figure 4 The schematic diagram of the optimization process of the loss function for a pose estimation method based on the learning and progressive optimization of dynamic key points of multiple types of objects proposed by the present invention; Detailed implementation manners

[0062] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than all embodiments. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of the present application. They are only examples of methods consistent with some aspects of the embodiments of the present application detailed in the appended claims.

[0063] Although the logical order is shown in the flowchart and the basic structure of the present invention is illustrated in the framework diagram, it only shows the components related to the present invention.

[0064] Figure 1 It is an overall flowchart of a pose estimation method based on the learning and progressive optimization of dynamic key points of multiple types of objects proposed in an embodiment of the present invention, including:

[0065] Step 1: Obtain an RGB-D image through a binocular camera or a depth camera, and then use Mask R-CNN to detect and segment the RGB image and the corresponding depth map of the target object based on the RGB-D image and its mask image using the internal and external parameters of the camera.

[0066] The said Step 1 includes the following sub-steps:

[0067] 1.1: For the RGB-D image obtained by the depth camera, it is necessary to perform label division on the objects existing in the image. The label division includes object category labels and pixel-level mask annotations. The mask is a binary image, and the annotation area corresponds to the pixel positions of the object in the RGB-D image; then use Mask R-CNN to detect and segment the objects in the RGB-D image according to the corresponding label division.

[0068] 1.2: Obtain the RGB image and the depth image of the segmented object, and then use the internal parameters of the camera corresponding to the RGB-D image to convert the depth image into the corresponding point cloud Po represents the point cloud converted from the depth map, N o represents the number of points contained in the point cloud; the point cloud conversion formula is N is the number of point clouds.

[0069] 1.3: During the training phase, appropriate Gaussian noise will be added to the RGB images and point clouds of a single object for data augmentation to make the training process more robust.

[0070] The specific structural design of the embodiments of this application is as follows Figure 2 shown. When starting model training, relevant parameters are initialized, including the number of key points kpt_num, the number of training epochs epoch, the number of iterations iters, the batch size batch_size, the upper bound of the learning rate max_lr, and the base_lr.

[0071] Step 2: For the RGB image segmented in the previous step, a lightweight ResNet-18 is used as the backbone network of PSPNet. Multiscale context information is fused through the pyramid pooling module, and the original resolution is restored through upsampling to extract the C-dimensional RGB feature F rgb ; Subsequently, the depth map is converted into point cloud data containing 1024 points. Adding a small amount of noise information to the original image during training can enhance the robustness of the model for complex scenes and images under strong or weak light interference.

[0072] The description of the backbone network structure of the PSPNet network in Step 2 is as follows:

[0073] 2.1: PSPNet includes a pyramid pooling module, multiple upsampling operations, and Dropout operations. Among them, the pyramid pooling module contains four parallel branches for pooling. The output features are compressed by 1×1 convolution and then concatenated, and then restored to the input resolution through three upsamplings;

[0074] The methods for obtaining the RGB feature and the point cloud feature in Step 2 include:

[0075] It is obtained by convolution, pooling, and fully connected operations on the RGB image and the point cloud data. The preliminary features are used as the input of the next layer, and then through convolution, pooling, and fully connected operations until the final RGB feature and point cloud feature are obtained using the last layer of the network, and the dimensions of both are the same;

[0076] 2.3: The output of the PSPNet network is a 128-dimensional RGB feature; the output of the DGCNN network is a 128-dimensional point cloud feature containing more global information.

[0077] Step 3: Since the geometric information contained in each point in the point cloud is relatively local, fusing as much multi-scale information as possible for each point can enhance the accuracy of the pose. However, the number of points in the point cloud is extremely large and there are outlier points. It is necessary to reduce the number of points in the original point cloud while retaining more original object information. Use 3DGCN to calculate 512 points with the most retained information through Euclidean distance and similarity.

[0078] The said Step 3 includes the following sub-steps:

[0079] 3.1: Extract RGB features from the picture, map the RGB information point by point into the point cloud according to the distribution of points in the point cloud, and then use the 3DGCN layer based on the geometric relationship between the point cloud with RGB information and the original point cloud.

[0080] 3.2: Use RGB information and geometric information as the selection conditions. The core of 3DGCN is KNN calculation: for each point, calculate its Euclidean distance and cosine similarity with all other points, and select the nearest 20 neighbors. Calculate the cosine similarity + Euclidean distance of the information contained in the points in the point cloud through 3DGCN, and filter out the 512 points with the highest correlation; the current point P i and the template point P j The Euclidean distance formula is d ij = ||P i - P j ||2 ( is the current point coordinate, is the template point coordinate) The cosine similarity calculation formula is ( is the feature extracted by 3DGCN).

[0081] 3.3: Score according to the relationship between each point and its adjacent points, and screen 512 points in descending order of the comprehensive score to form a new point cloud P r represents the point cloud after removing outliers, and N r represents the number of points contained in the new point cloud;

[0082] Step 4: Use DGCNN to extract the feature F dgcnn from the original point cloud and the RGB feature F rgb extracted by the PSPNet network are fused through the attention mechanism to obtain the global feature F global , and the fusion process here is as Figure 3 shown. Based on the above-screened 512 points, set a learnable query to obtain 100 most information-dense local key points and features, reconstruct based on the local key points, and use PointNet++ to extract the feature F pointAnd then with the RGB feature F rgb Use the self-attention mechanism to fuse again to obtain the local feature F local , and finally perform weighted fusion to become the final feature F final , and obtain the six-degree-of-freedom information restored from the final feature through a multi-layer perceptron.

[0083] In this embodiment, the processes of local feature fusion and final feature fusion use self-attention and linear attention, but the structure is the same as Figure 3 similar.

[0084] Step 4 includes the following sub-steps:

[0085] 4.1: The network structure of the PointNet++ is as follows: the target number of points for farthest point sampling (FPS) is 1000, the spherical neighborhood grouping radii are 0.05m, 0.1m, and 0.2m in sequence, each group contains 32 neighborhood points, and then output the feature F through the MLP point .

[0086] 4.2: Use the point cloud converted from the depth image to extract the feature F through DGCNN dgcnn and the RGB feature F rgb After the attention operation and then splicing to obtain the global feature F that retains the most original information global ; The DGCNN dynamic graph convolution can enhance the use of neighborhood correlation. The attention operation is defined as: where Q, K, and V are the query, key, and value features respectively, generated by linear transformation, and d k is the feature dimension.

[0087] In this embodiment, step 4 may include but is not limited to the following step 4.2.1.

[0088] 4.2.1: The core of using the attention mechanism for global feature fusion here lies in multi-head attention. Use the linear layer to calculate Q, K, and V. The specific expressions are:

[0089] Q = LN(F dgcnn );

[0090] K = LN(F rgb );

[0091] V = LN(F rgb );

[0092]

[0093] SA, d, and CA represent self-attention, the distance between points, and cross-attention respectively;

[0094] 4.3: A set of learnable key point queryers are set for each object in the model to learn the feature representation of the current object, using the previously obtained point cloud P r and the RGB feature F rgb and using PointNet++ for P r The extracted features As the condition for the key point queryer to learn, and then based on the cross-attention of the Transformer, the point and feature distributions in the current queryer are learned, and then compared with P r Perform cosine similarity calculation to find 100 key points for reconstruction, and then extract the local structure feature F by PointNet++ point , where the feature dimension should be the same as the previously obtained RGB feature F rgb ; Point cloud feature F point And the RGB feature F again rgb Strengthen the internal feature correlation of each other through the self-attention mechanism, and then splice to enhance the attention weight of the strong correlation area between the two in the spatial dimension.

[0095] In this embodiment, the setting of the initial learnable queryer includes the following sub-steps. Considering that if the initial value is set to be sufficiently scattered and includes enough RGB information and geometric information, a large amount of computing power can be saved in subsequent calculations. Then use cosine similarity to calculate the scores corresponding to each point and select the 100 points with the highest scores as key points.

[0096] 4.4: Feature weighted fusion Feature weighted fusion is automatically adjusted through model training to enable F local and F global Realize multi-level feature interaction and construct a multi-modal fusion representation with enhanced geometric structure information; the feature weighted fusion formula is: F final = ω local ·F local + ω global ·F global .

[0097] In this embodiment, the fusion calculation process of step 4.4 is as follows:

[0098] 4.4.1: For objects of different categories with different shapes and unevenly distributed collected points, we initialize the queryer of each object using the RGB feature of each object and the filtered point cloud information Q learn Represents the initially set query range, and then add the geometric features of the point cloud to guide so that the key points can learn more color and geometric information.

[0099] 4.4.2: After setting the initial value, the formula for learning the feature distribution of the corresponding object is as follows:

[0100] Q = Q learn W Q ;

[0101]

[0102] W Q，K，V is the weight automatically adjusted during the training of the point distribution;

[0103]

[0104] Q obj is the learned query point and the corresponding feature distribution, and d is the distance between the initially defined query point and the learned query point;

[0105] 4.5: Feature weighted fusion Feature weighted fusion is automatically adjusted through model training to enable F local and F global to achieve multi-level feature interaction and construct a multi-modal fusion representation with enhanced geometric structure information; the feature weighted fusion formula is: F final = ω local ·F local + ω global ·F global .

[0106] Step 5: Based on the predicted pose information, i.e., the rotation matrix R ∈ SO(3), the translation vector t ∈ R 3 and the scaling factor s ∈ R; render the point cloud of the object model based on the predicted pose, calculate the symmetric Chamfer distance between it and the real point cloud as the residual; during the training process, dynamically adjust the number of iterations through the comparison of losses and the gap between the current six-degree-of-freedom information and the label, and use the Adam optimizer to update the pose parameters to solve the least squares solution to update the pose parameters.

[0107] The said Step 5 includes the following sub-steps:

[0108] 5.1: Concatenate the global feature F global and the local feature F local obtained through the above steps, and then pass them through a multi-layer perceptron to obtain the pose information of the unknown object, and use the least squares method to solve the rotation matrix of the model relative to the camera coordinate system. The rotation matrix contains the three-dimensional rotation R, three-dimensional translation t, and scaling factor s information of the object. When solving the rotation matrix by the least squares method, use SVD decomposition to find the closed-form solution: R = VU T where

[0109] 5.2: Information on the three-dimensional rotation R, three-dimensional translation t, and scaling factor s of an object can be obtained to get the object's pose. The model reconstructed using the final features is passed through a multi-layer perceptron to obtain the offset value, which is then added to the final features. Based on the new final features, reconstruction is performed to obtain new R1, t1, and s1. Here, the pose is iteratively updated through a progressive optimization block (such as Figure 2 as shown) until the optimal number of iterations is found or the maximum number of iterations is reached and then stopped, making the model more accurate;

[0110] Step 6: According to the relationship between the world coordinate system and the NOCS space, the object in the NOCS space can be transformed into the world coordinate system to obtain the pose estimation of the object. Finally, the reconstructed model will first go through NOCS prediction and then obtain the true pose coordinates based on the NOCS prediction.

[0111] The said Step 6 includes:

[0112] 6.1: According to the relationship between the world coordinate system and the NOCS space, the true pose information of the object is used to transform the pose in the NOCS space to obtain the representation of the true object in the standard normalized space;

[0113] In this embodiment, the transformation process of Step 6.1 is as follows:

[0114] 6.1.1: The purpose of the spatial transformation here is that NOCS normalizes objects of different instances into a unified coordinate system. Regardless of the actual size of the object, it is mapped into a standardized space. This standardization helps to process objects of different scales and shapes. The expression for the transformation is:

[0115]

[0116] R T represents the transpose of the rotation matrix, which is used to rotate points from the world coordinate system to the standard coordinate system of the object; subtracting the translation vector t eliminates the translation of the object in the world coordinate system; and dividing by the scaling factor s completes the normalization in terms of size.

[0117] 6.2: According to the pose information in the NOCS space, a more unified representation of the pose information of different objects can be obtained. After adding constraints for training in this space, the coordinates in the NOCS space are projected into the world coordinate system to obtain the true coordinate information.

[0118] In this embodiment, the process of transforming back to the pose in the world for Step 6.2 is as follows:

[0119] 6.2.1: Through matrix transformation, the finally predicted NOCS pose can be transformed into the world coordinate system through matrix transformation and compared with the actual label to evaluate the quality of the model. The expression is as follows:

[0120] P camera = R·P nocs + t;

[0121] P world = R ext ·P camera + t ext ;

[0122] P nocs is a point in NOCS, and P camera is the corresponding point in the camera coordinate system, R ext and t ext are respectively the rotation matrix and the translation vector of the external camera parameters.

[0123] The loss function used in this embodiment and its use in the model are as Figure 4 shown. Strictly speaking, the chamfer loss is used three times in the model. Once during reconstruction, the angular loss is used to constrain the reconstructed shape. The model restored based on the final features also needs to use the chamfer loss to constrain the shape. Finally, after the progressive optimization is completed, the chamfer loss is also needed to perform the final model comparison to determine whether the current optimization result is closer to the real model.

[0124] The embodiments described in the embodiments of the present invention are for more clearly illustrating the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0125] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than those shown, or combine certain steps, or different steps.

[0126] In several embodiments provided by the present invention, it should be understood that the disclosed modules and methods can be implemented in other ways. For example, the above-described embodiments are merely illustrative. For example, the division of functional modules in the flowchart is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules can be combined, or some features can be ignored.

[0127] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, and the scope of the rights of the embodiments of the present application is not limited thereby. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A pose estimation method based on multi-class object dynamic key point learning and progressive optimization, characterized in that It includes the following steps: Step 1: Obtain an RGB-D image through a binocular camera or a depth camera. Subsequently, use Mask R-CNN to detect and segment the RGB image and the corresponding depth map of the target object based on the RGB-D image and its mask image using the internal and external parameters of the camera. Step 2: For the RGB image segmented in the previous step, a lightweight ResNet-18 is used as the backbone network of the PSPNet. Multiscale context information is fused through the pyramid pooling module, and the original resolution is restored through upsampling to extract the C-dimensional RGB feature F rgb ; Subsequently, the depth map is converted into point cloud data containing 1024 points. Adding a small amount of noise information to the original image during training can enhance the robustness of the model to complex scenes and images under strong or weak light interference. Step 3: Since the geometric information contained in each point in the point cloud is relatively local, fusing as much multi-scale information as possible for each point can enhance the accuracy of the pose. However, the number of points in the point cloud is extremely large and there are outlier points. It is necessary to reduce the number of points in the original point cloud while retaining more original object information. Use 3DGCN to calculate 512 points with the most retained information through Euclidean distance and similarity. Step 4: Use DGCNN to extract the feature F of the original point cloud dgcnn and the RGB feature F extracted by the PSPNet network rgb Fuse them through the attention mechanism to obtain the global feature. Based on the 512 points screened above, set a learnable query to obtain 100 most information-dense local key points and features, reconstruct based on the local key points, and use PointNet++ to extract the feature F of the point cloud reconstructed from the key points point and then the RGB feature F rgb Use the self-attention mechanism to fuse again to obtain the local feature F local , and finally weighted fusion to become the final feature F final , and obtain the six-degree-of-freedom information restored from the final feature through the multi-layer perceptron. Step 5: Through the predicted pose information, i.e., rotation matrix R ∈ SO(3), translation vector t ∈ R 3 and scaling factor s ∈ R; Render the point cloud of the object model based on the predicted pose, calculate the symmetric Chamfer distance between it and the real point cloud as the residual; During the training process, dynamically adjust the number of iterations by comparing the losses and the gap between the current six-degree-of-freedom information and the label, and use the Adam optimizer to update the pose parameters to solve the least-squares solution and update the pose parameters. Step 6: According to the relationship between the world coordinate system and the NOCS space, the object in the NOCS space can be transformed into the world coordinate system to obtain the pose estimation of the object. Finally, the reconstructed model will first go through NOCS prediction and then obtain the true pose coordinates based on the NOCS prediction. Step 7: The diversity loss L is required when querying key points div Constrain the positions of key points based on the final feature F inal The point cloud P obtained by reconstruction final Use the chamfer loss L cd Continuously adjust during the iteration to obtain the optimized point cloud P optimize , For the generated P during optimization optimize It is also necessary to further project it into the NOCS space by the loss function L nocs For detailed adjustment. Finally, an attitude loss L is also required for the six-degree-of-freedom information obtained from the optimized model pose , Finally, the total loss function is expressed as: L total = λ1L div + λ2L cd + λ3L pose + λ4L nocs .

2. The pose estimation method based on multi-class object dynamic key point learning and progressive optimization according to claim 1, wherein The said Step 1 includes the following sub-steps: 1.1: For the RGB-D image obtained by the depth camera, it is necessary to perform label division on the objects existing in the image. The said label division includes object category labels and pixel-level mask annotations. Among them, the mask is a binary image, and the annotation area corresponds to the pixel position of the object in the RGB-D image. Then use Mask R-CNN to detect and segment the objects in the RGB-D image according to the corresponding label division. 1.2: Obtain the RGB image and depth image of the segmented object, and then use the internal parameters of the camera corresponding to the RGB-D image to convert the depth image into the corresponding point cloud P o represents the point cloud converted from the depth map, and N o represents the number of points included in the point cloud; the point cloud conversion formula is N is the number of point clouds. 1.3: During the training phase, appropriate Gaussian noise will be added to the RGB images and point clouds of individual objects for data augmentation to make the training process more robust.

3. The pose estimation method based on multi-class object dynamic key point learning and progressive optimization according to claim 1, wherein The structural description of the ResNet-18 in the said Step 2 as the backbone network of the PSPNet network includes: 2.1: It includes a pyramid pooling module, multiple upsampling operations, and Dropout operations. Among them, the pyramid pooling module contains four parallel branches for pooling. The output features are compressed by 1×1 convolution and then concatenated, and then restored to the input resolution through three upsamplings.

4. The pose estimation method based on multi-class object dynamic key point learning and progressive optimization according to claims 1 and 3, characterized in that, The methods for obtaining RGB features and point cloud features in the said Step 2 include: 2.2: For the RGB image and the point cloud data, preliminary features are obtained through convolution, pooling, and fully connected operations and used as the input for the next layer. Then, through convolution, pooling, and fully connected operations until the final RGB features and point cloud features are obtained using the last layer of the network, and the dimensions of the two are the same. 2.3: The output of the PSPNet network is 128-dimensional RGB features; the output of the DGCNN network is 128-dimensional point cloud features containing more global information.

5. The pose estimation method based on multi-class object dynamic key point learning and progressive optimization according to claim 1, characterized in that The said Step 3 includes the following sub-steps: 3.1: Extract RGB features from the image, map the RGB information point by point to the point cloud according to the distribution of the points in the point cloud, and then use the 3DGCN layer based on the geometric relationship between the point cloud with RGB information and the original point cloud. 3.2: Use RGB information and geometric information as selection conditions. The core of 3DGCN is KNN calculation: for each point, calculate its Euclidean distance and cosine similarity with all other points, and select the 20 nearest neighbors. Calculate the cosine similarity + Euclidean distance of the information contained in each point in the point cloud through 3DGCN, and filter out the 512 points with the highest correlation; the current point P i and the template point P j The Euclidean distance formula is d ij = ||P i - P j ||2( is the current point coordinate, is the template point coordinate) The cosine similarity calculation formula is ( is the feature extracted by 3DGCN). 3.3: Score each point based on its relationship with adjacent points, and screen out 512 points in descending order of the comprehensive score to form a new point cloud P r represents the point cloud after removing outliers, N r represents the number of points included in the new point cloud.

6. The pose estimation method based on multi-class object dynamic key point learning and progressive optimization according to claim 1, characterized in that, The said Step 4 includes the following sub-steps: 4.1: The network structure of the PointNet++ is as follows: the target number of points for farthest point sampling (FPS) is 1000, the spherical neighborhood grouping radii are 0.05m, 0.1m, and 0.2m in sequence, each group contains 32 neighborhood points, and then the feature F is output through MLP point . 4.2: Extract feature F through DGCNN using the point cloud converted from the depth image dgcnn and RGB feature F rgb After the attention operation and then concatenation, the global feature F that retains the most original information is obtained global ; The DGCNN dynamic graph convolution can enhance the use of neighborhood correlation. The attention operation is defined as: where Q, K, and V are the query, key, and value features respectively, generated by linear transformation, and d k is the feature dimension 4.3: A set of learnable key point queryers is set for each object in the model to learn the feature representation of the current object, using the previously obtained point cloud P r and the RGB feature F rgb and using PointNet++ for P r The extracted features As the condition for the key point queryer to learn, and then based on the cross-attention of the Transformer, the point and feature distributions in the current queryer are learned, and then compared with P r Perform cosine similarity calculation to find 100 key points for reconstruction, and then extract the local structure feature F by PointNet++ point , where the feature dimension should be the same as the previously obtained RGB feature F rgb Dimension consistent; point cloud feature F point The RGB feature F is then rgb Strengthen the internal feature correlation of each other through the self-attention mechanism, and then splice to enhance the attention weight of the strong correlation area between the two in the spatial dimension. 4.4: Feature Weighted Fusion Feature weighted fusion is automatically adjusted through model training to enable F local and F global to achieve multi-level feature interaction and construct a multi-modal fusion representation with enhanced geometric structure information; the feature weighted fusion formula is: F final = ω local · F local + ω global · F global .

7. The pose estimation method based on multi-class object dynamic key point learning and progressive optimization according to claim 1, characterized in that The said Step 5 includes the following sub-steps: 5.1: The global feature F obtained through the above steps global and the local feature F local are concatenated and then passed through a multi-layer perceptron to obtain the pose information of the unknown object. The rotation matrix of the model relative to the camera coordinate system is solved using the least squares method. The rotation matrix contains the three-dimensional rotation R, three-dimensional translation t, and scaling factor s information of the object. When solving the rotation matrix using the least squares method, the SVD decomposition is used to obtain the closed-form solution: R = VU T where 5.2: Obtaining the three-dimensional rotation R, three-dimensional translation t, and scaling factor s information of an object can yield the object's pose. The model reconstructed using the final features is passed through a multi-layer perceptron to obtain the offset value, which is then added to the final features. Based on the new final features, reconstruction is performed to obtain new R1, t1, and s1. Here, the pose is iteratively updated through a progressive optimization block until the optimal number of iterations is found or the maximum number of iterations is reached and then stopped, making the model more accurate.

8. The pose estimation method based on multi-class object dynamic key point learning and progressive optimization according to claim 1, characterized in that The said step 6 includes: 6.1: According to the relationship between the world coordinate system and the NOCS space, the true pose information of the object is used to transform the pose in the NOCS space to obtain the representation of the true object in the standard normalized space. 6.2: According to the pose information in the NOCS space, a more unified representation of different object pose information can be obtained. After adding constraints for training in this space, the coordinates in the NOCS space are projected onto the world coordinate system to obtain the true coordinate information.

9. According to the pose estimation method based on multi-class object dynamic key point learning and progressive optimization described in claim 1, the loss function for finally adjusting the true pose described in step 7 of its rights includes: 7.1: When querying key points, in order to prevent the key points queried for each object from clustering or overlapping together, it is necessary to constrain the positions of the key points. Therefore, a diversity loss needs to be used, and the expression is as follows: Among them, N k represents the number of key points, and P k represents the point cloud composed of key points. 7.2: Based on the final feature F final The point cloud P obtained by reconstruction final Passes through the chamfer loss L cd Adjust the reconstructed point cloud to obtain the optimized point cloud P optimize , and the expression of the chamfer loss is: Among them, P obj The real point cloud representing the object can be derived from the real pose in the dataset, and P final is the final feature F final The model obtained by reconstruction. 7.3: From the model P in the optimization process optimize To obtain the six-degree-of-freedom information, it is also necessary to project it into the NOCS space and further adjust the details by the loss function L nocs The loss expression is as follows: Among them represents the model in the NOCS space derived from the true pose, M obj represents the point cloud extracted from the CAD model of the same type as the current object, S gt represents the true scaling factor of the object, R gt is the true rotation matrix information of the object, th2 represents the set threshold, t gt represents the true translation vector of the object. 7.4: In addition, a pose loss is added to the six-degree-of-freedom information obtained from the optimized model, and the expression is as follows: L pose = ||R gt - R||2 + ||t gt - t||2 + ||S gt - S||2; Among them, R, t, and S are the predicted pose information obtained based on the optimized reconstruction model respectively. 7.5: Finally, the total loss function is expressed as: L total = λ1L div+ λ2L cd + λ3L pose + λ4L nocs ; where λ1, λ2, λ3, and λ4 are the weight parameters of the diversity loss L div , chamfer loss L cd , NOCS loss L nocs and pose loss L pose , respectively.

Images