A method for establishing a category-level pose estimation model, a pose estimation method, a computer system and a computer readable storage medium

Abstract

The present application belongs to the field of computer image intelligent recognition, and particularly relates to a category level pose estimation model establishing method, a pose estimation method, a computer system and a computer readable storage medium. The method comprises the following steps: 1) inputting the global features corresponding to the features of the image and the observed point cloud of the training set instance and the features of the category level shape prior point cloud into a deformation module to obtain an instance complete point cloud and point cloud features thereof; fusing the features of the image and the observed point cloud and inputting the same into a reconstruction module containing a pre-trained reconstruction network to obtain a reconstructed point cloud and point cloud features thereof; fusing the instance complete point cloud and the reconstructed point cloud and the point cloud features thereof respectively to obtain respective point relationship features, fusing the point relationship features with average features to obtain pose features, and then obtaining an estimated pose through a regression module; 2) adjusting the parameters of the deformation module, the reconstruction module and the regression module according to the instance complete point cloud, the reconstructed point cloud, the estimated pose and the corresponding true label respectively; and repeating the steps 1) and 2) until the iteration is stopped.

Description

Technical Field

[0001] This invention belongs to the field of intelligent computer image recognition, specifically relating to a method for establishing a category-level pose estimation model, a pose estimation method, a computer system, and a computer-readable storage medium. Background Technology

[0002] The category-level 6D object pose and size estimation task estimates the rotation, translation, and size of unseen object instances of certain categories in a cluttered RGB-D scene. This task plays a crucial role in many applications such as augmented reality, robotic manipulation, and scene understanding. Traditional instance-level pose estimation typically assumes that each object of interest has a 3D CAD model, regressing pose information through instance segmentation and template matching with standard CAD models. However, in reality, there are often cases where the CAD model of the instance object is missing. In contrast, category-level tasks do not require the CAD model of the instance, thus giving them an advantage in this regard.

[0003] Category-level pose estimation places higher demands on adaptability to various shapes and textures within each category. To explicitly address intra-class shape variations, one existing approach utilizes a Shape Prior. This involves using a PointNet autoencoder to extract the prior point cloud for each category as the average of all shapes within that category, thus obtaining the Shape Prior. Before fitting the observed instance, this Shape Prior further deforms the shape and assigns the observed point cloud to the deformed shape model. Then, by aligning it with the observed instance point cloud, the Umeyama algorithm is used to obtain the target's pose and size. This method primarily involves deforming the Shape Prior in the canonical space and obtaining the representation of the observed instance point cloud in the canonical space through the relation matrix A. However, since the observed point cloud often fails to fully encompass the shape features of the instance, using this incomplete observation point cloud for pose estimation results in a lack of information, and errors in the learned relation matrix significantly impact pose estimation. Summary of the Invention

[0004] The purpose of this invention is to provide a method for establishing a category-level pose estimation model, a pose estimation method, a computer system, and a computer-readable storage medium, in order to solve the problems in the prior art where using observed point clouds for pose estimation actually results in a lack of information and the error of the learned relation matrix has a significant impact on pose estimation.

[0005] To achieve the above objectives, this invention provides a method for establishing a category-level pose estimation model, comprising:

[0006] 1) Based on the features extracted from the images and observation point clouds of the target object instances in the training set, the global features of the images and observation point clouds are obtained respectively, and together with the features extracted from the category-level shape prior point cloud of the target object instances, they are input into the deformation module of the model for feature fusion to obtain the complete point cloud of the instance in the normal space and its point cloud features.

[0007] The fused features obtained by fusing features extracted from the images of target object instances in the training set and the observed point clouds are input into the reconstruction module of the model to obtain the reconstructed point cloud and its point cloud features; the reconstruction module contains a pre-trained reconstruction network.

[0008] The complete point cloud of the instance and its point cloud features are fused with the reconstructed point cloud and its point cloud features according to the one-to-one correspondence between the points in the complete point cloud of the instance and the points in the reconstructed point cloud to obtain the point relationship features; the point relationship features are fused with the average features obtained based on the point relationship features to obtain the pose features; the pose features are regressed through the regression module of the model to obtain the estimated pose.

[0009] 2) Adjust the parameters of the deformation module, reconstruction module, and regression module respectively; repeat 1)-2) until the iteration stops to complete the establishment of the category-level pose estimation model.

[0010] Furthermore, the reconstruction network is an encoder, and the pre-training methods for the reconstruction network include:

[0011] a) Input the data from the pre-training samples into the pre-training network. The data in the pre-training samples contains point embeddings corresponding to point clouds of instances from a certain category. The masked point embeddings are processed by the labeling masking module in the pre-training network and then reconstructed by the encoder that needs to be pre-trained. The reconstructed points are processed by the masked point modeling module in the pre-training network to obtain point labels.

[0012] b) The pre-training of the encoder is supervised by the tokenizer module based on the real label corresponding to the data input to the pre-trained network. This process is repeated from a) to b) until the iteration stops to complete the pre-training of the encoder.

[0013] Furthermore, the obtained fused features are input into the reconstruction module of the model to obtain the reconstructed point cloud and its point cloud features in the following ways:

[0014] After the obtained fused features are processed by a multilayer perceptron subnetwork in the reconstruction module, they are then reconstructed by a pre-trained reconstruction network in the reconstruction module to obtain the point cloud features of the reconstructed point cloud; then the point cloud features of the reconstructed point cloud are processed by another multilayer perceptron subnetwork in the reconstruction module to obtain the reconstructed point cloud.

[0015] Furthermore, the global features of the image and the observed point cloud are obtained together with the features extracted from the shape prior point cloud of the target object instance and input into the deformation module for feature fusion. The methods for obtaining the complete point cloud of the instance in the normal space and its point cloud features include:

[0016] After stitching together the global features of the image and the observed point cloud from the input deformation module, as well as the features extracted from the shape prior point cloud of the target object instance, the stitching result is processed by a multilayer perceptron subnetwork in the deformation module to obtain deformation domain features. The deformation domain features are then added element-wise to the features extracted from the shape prior point cloud of the target object instance to obtain the point cloud features of the instance complete point cloud in the normal space. The point cloud features of the instance complete point cloud in the normal space are then processed by another multilayer perceptron subnetwork in the deformation module to obtain the instance complete point cloud in the normal space.

[0017] Furthermore, the methods for obtaining global features of the image and observation point cloud respectively based on features extracted from the target object instance image and observation point cloud in the training set include:

[0018] Average pooling is performed on the features extracted from the image of the target object instance in the training set and the features extracted from the observation point cloud of the target object instance to obtain the global features of the image and the global features of the observation point cloud, respectively.

[0019] Furthermore, the complete point cloud of the instance and its point cloud features are fused with the reconstructed point cloud and its point cloud features according to the one-to-one correspondence between the points in the complete point cloud of the instance and the points in the reconstructed point cloud, respectively, to obtain the point relationship features in the following ways:

[0020] Based on the one-to-one correspondence between the points in the complete point cloud of the instance and the points in the reconstructed point cloud obtained by matching the complete point cloud of the instance and the points in the reconstructed point cloud, the points in the complete point cloud of the instance with each corresponding relationship and the points in the reconstructed point cloud, as well as the point cloud features of these two points, are spliced ​​and fused. Then, a multilayer perceptron sub-network is used to process the splicing and fusion results to obtain the relationship features of each point.

[0021] Furthermore, the methods for adjusting the parameters of the deformation module, reconstruction module, and regression module respectively include:

[0022] The parameters of the deformation module, reconstruction module, and regression module are adjusted based on the differences between the complete point cloud and the corresponding ground truth label, the differences between the reconstructed point cloud and the corresponding ground truth label, and the differences between the estimated pose and the corresponding ground truth label, respectively.

[0023] The above-described technical solution of this invention provides a novel method for establishing a category-level pose estimation model. Its beneficial effects include: when constructing the category-level pose estimation model, it not only utilizes category-level shape prior point clouds but also integrates the global features corresponding to the image of the target object instance and the observed point cloud. Therefore, the complete point cloud of the instance in the canonical space obtained can simultaneously contain the actual features of the target object instance and category-level shape prior features. Furthermore, a pre-trained reconstruction network is used to reconstruct the point cloud based on the actual features of the target object instance. Finally, the pose features used for regression to obtain the estimated pose include the point relationship features (equivalent to features corresponding to the relationship between each point in the point cloud and other points) fused with the complete point cloud representation in the canonical space corresponding to each point and the point cloud reconstruction results corresponding to the actual features, as well as the average features of each point relationship feature (equivalent to features corresponding to the global relationship of all points in the point cloud). Therefore, the features used for regression are richer, especially the point cloud reconstructed by the pre-trained reconstruction network can compensate for the missing information in the observed point cloud when describing the shape features of the instance, thereby improving the accuracy of the pose estimation results output by the constructed model.

[0024] The present invention also provides a pose estimation method, comprising: inputting an image of a target object instance whose pose is to be estimated, an observation point cloud, and a category-level shape prior point cloud into a pose estimation model to obtain a pose estimation result;

[0025] The pose estimation model is constructed using the method described above for establishing a category-level pose estimation model.

[0026] This pose estimation method can achieve the same beneficial effects as the above-mentioned method for establishing category-level pose estimation models.

[0027] The present invention also provides a computer system including a processor for executing executable program instructions, the executable program instructions being executed to implement the pose estimation method described above.

[0028] This computer system can achieve the same beneficial effects as the pose estimation method described above.

[0029] The present invention also provides a computer-readable storage medium storing computer program instructions, which, when executed, implement the above-described pose estimation method.

[0030] This computer-readable storage medium can achieve the same beneficial effects as the pose estimation method described above. Attached Figure Description

[0031] Figure 1 This is a schematic diagram illustrating the structural principle of the category-level pose estimation model established in an embodiment of the method for establishing the category-level pose estimation model of the present invention.

[0032] Figure 2 This is a schematic diagram illustrating the structural principle of the pre-trained network used to pre-train the encoder in the reconstruction module of the category-level pose estimation model, as described in an embodiment of the method for establishing the category-level pose estimation model of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0034] Examples of methods for establishing category-level pose estimation models

[0035] This embodiment presents a technical solution for establishing a category-level pose estimation model. By integrating the actual features of the target object instance, the complete point cloud features of the instance in the canonical space with category-level shape prior features, and the point cloud features reconstructed by the pre-trained reconstruction network based on the actual features of the target object instance, category-level pose estimation is achieved. This not only considers both actual features and category-level shape prior features, but also uses the reconstructed point cloud to compensate for the missing information in the observed point cloud when describing the shape features of the instance, thereby improving the accuracy of the pose estimation results output by the constructed model.

[0036] The method specifically includes:

[0037] 1) Based on the features extracted from the images and observation point clouds of the target object instances in the training set, the global features of the images and observation point clouds are obtained respectively. These features are then input together with the features extracted from the category-level shape prior point cloud of the target object instances into the deformation module of the category-level pose estimation model for feature fusion, so as to obtain the complete point cloud of the instance in the normal space and its point cloud features.

[0038] The fused features obtained by fusing the features extracted from the images of the target object instances in the training set and the observed point clouds are input into the reconstruction module of the category-level pose estimation model to obtain the reconstructed point cloud and its point cloud features; the reconstruction module contains a pre-trained reconstruction network.

[0039] The complete point cloud of the instance and its point cloud features are fused with the reconstructed point cloud and its point cloud features according to the one-to-one correspondence between the points in the complete point cloud of the instance and the points in the reconstructed point cloud to obtain point relationship features; the point relationship features are fused with the average features obtained based on the point relationship features to obtain pose features; the pose features are regressed through the regression module of the category-level pose estimation model to obtain the estimated pose.

[0040] 2) Update the parameters of the deformation module, reconstruction module, and regression module respectively; repeat 1)-2) until the iteration stops to complete the establishment of the category-level pose estimation model.

[0041] Therefore, this method for constructing a category-level pose estimation model not only utilizes category-level shape prior point clouds but also integrates the global features corresponding to the image of the target object instance and the observed point cloud. Thus, the complete point cloud of the instance in the canonical space obtained can simultaneously contain the actual features of the target object instance and category-level shape prior features. On this basis, a pre-trained reconstruction network is used to reconstruct the point cloud based on the actual features of the target object instance. Finally, the pose features used for regression to obtain the estimated pose include the point relationship features fused with the complete point cloud representation of each point in the canonical space and the point cloud reconstruction results, as well as the average features of each point relationship feature. Therefore, the features used for regression are richer, especially the point cloud reconstructed by the pre-trained reconstruction network can make up for the missing information in the observed point cloud when describing the shape features of the instance, thereby improving the accuracy of the pose estimation results output by the constructed model.

[0042] Reference Figure 1 In this embodiment, the CNN module extracts I from the images of the target object instances in the training set. o Extracting features F Io Using two parallel Pointnet++ modules, the observed point cloud P of the target object instances in the training set is obtained respectively. o (Also known as a visible point cloud), category-level shape prior point cloud Q c Extracting features F from (also known as Shape Prior point cloud) Po F Qc In other embodiments, other existing feature extraction modules or feature extraction methods may also be used, and no specific restrictions are imposed.

[0043] Specifically, the methods for obtaining global features of the image and observation point cloud based on features extracted from the images and observation point clouds of the target object instances in the training set include: extracting features F from the images of the target object instances in the training set... Io and the feature F extracted from the observation point cloud of the target object instance. Po Perform average pooling operations separately to obtain the global features f of the image. Io and global features f of the observed point cloud Po This method can be expressed by the following formula:

[0044] f Io =AvgPool(F Io )

[0045] f Po =AvgPool(F Po )

[0046] In the formula, AvgPool represents the average pooling operation, that is... Figure 1 The Average Pooling operation in [the context of the text].

[0047] Then, refer to Figure 1 The global features of the image and observed point cloud are obtained and input together with the features extracted from the shape prior point cloud of the target object instance into the deformation module for feature fusion. The methods to obtain the complete point cloud of the instance in the normal space and its point cloud features include:

[0048] Global features f of the input deformation module image and the observed point cloud Io f Po and the feature F extracted from the shape prior point cloud of the target object instance. Qc ∈R Mxd After point-by-point stitching, the stitching result is processed by a Multi-Layer Perceptron (MLP) sub-network in the deformation module to obtain the deformation domain feature F. D ∈R Mxd ; to transform the feature F of the deformation domain D Features F extracted from the shape prior point cloud of the target object instance Qc Add element by element (i.e.) Figure 1 The matrix addition operation in the model yields the point cloud features F of the complete point cloud in the normalized space. Qv ∈R Mxd ; Point cloud features F of the complete point cloud instance in the canonical space are obtained through another multilayer perceptron subnetwork in the deformation module. Qv After processing, a complete point cloud Q in the normalized space is obtained. v Since the complete point cloud instance in the canonical space in this embodiment is in the form of a 3D point cloud, therefore Q v ∈R Mx3 The method for obtaining the complete point cloud instance and its point cloud features in the above-mentioned normalized space is expressed by the following formula:

[0049] F D =MLP(Concat(F Qc Tile M (f Io Tile M (f Po )))

[0050] Q v =MLP(F Qv ) = MLP(F Qc +F D )

[0051] Where Concat represents feature concatenation, MLP represents multilayer perceptron, and Tile represents feature concatenation. M(·) represents M copies of the eigenvector.

[0052] In this embodiment, the methods for adjusting the parameters of the deformation module, reconstruction module, and regression module respectively include:

[0053] The parameters of the deformation module, reconstruction module, and regression module are adjusted based on the differences between the complete point cloud of the instance and the corresponding ground truth label, the differences between the reconstructed point cloud and the corresponding ground truth label, and the differences between the estimated pose and the corresponding ground truth label.

[0054] Specifically, during the training process (i.e., the model building process described in steps 1-2 above), the deformation module learns by using the true labels corresponding to the complete point clouds of the instances (i.e., annotated instance point clouds P′). v The group-truth method provides guidance and supervision, that is, it updates the parameters of the deformation module based on the difference between the complete point cloud of the instance and the corresponding real label; the loss function used is the chamfer distance loss L. SPDN The specific supervision methods are as follows:

[0055]

[0056] Among them, D cham Indicates the chamfer distance, used to limit Q. v With P′ v The distance between them; M represents the instance complete point cloud Q in the deformed canonical space. v The point cloud P′ with annotation v The number of midpoints Q represents v The i-th point, Represents P′ v The i-th point. Since the method of supervising model training through chamfer distance loss is existing technology, it will not be described in detail here; in addition, in other embodiments, other loss functions can be used to update the parameters of the deformation module based on the difference between the complete point cloud of the instance and the corresponding real label.

[0057] In this embodiment, the reconstruction network uses an encoder; in other embodiments, other neural networks may also be used. The fused features obtained by fusing the features extracted from the images of the target object instances in the training set and the observed point clouds are input into the reconstruction module of the category-level pose estimation model to obtain the reconstructed point cloud and its point cloud features.

[0058] Through the reconstruction module of the category-level pose estimation model (i.e. Figure 1 A multilayer perceptron subnetwork in the Reconstruction module region extracts features F from the image of the target object instance and the observation point cloud, respectively.Io F Po After processing the fused features obtained through fusion, they are then reconstructed using a pre-trained encoder in the reconstruction module to obtain the point cloud features F of the reconstructed point cloud. Pv Then, the point cloud features F of the reconstructed point cloud are processed by another multilayer perceptron subnetwork in the reconstruction module. Pv ∈R Mxd After processing, the reconstructed point cloud P is obtained. v In this embodiment, feature F Io F Po The fusion method is to first process F Io and F Po The points are stitched together and then processed by a multilayer perceptron to enhance the point feature information; other feature fusion methods can also be used in other embodiments; the above-mentioned method of obtaining the reconstructed point cloud and its point cloud features is expressed by the following formula:

[0059] F Pv =Φ(MLP(Concat(F) Io ,F Po )))

[0060] Pv=MLP(Φ(MLP(Concat(F Io ,F Po ))))

[0061] Where Concat represents feature stitching, MLP represents multilayer perceptron, and Φ represents encoder; since the reconstructed point cloud in this embodiment is in the form of a 3D point cloud, P v ∈R Mx3 .

[0062] During the training process (i.e., the model building process described in steps 1-2 above), the reconstruction module learns by using the ground truth labels (i.e., annotated instance point clouds P′) corresponding to the reconstructed point clouds. v The Group-Truth algorithm provides guidance and supervision, meaning that the parameters of the reconstruction module are updated based on the differences between the reconstructed point cloud and the corresponding ground truth labels; the loss function used is the chamfer distance loss L. Recon The specific supervision methods are as follows:

[0063]

[0064] Among them, D cham Indicates the chamfer distance, used to limit P v With P′ v The distance between them; M represents the point cloud P reconstructed by the pre-trained encoder. v The point cloud P′ with annotation v The number of midpoints P represents v The i-th point, Represents P′ v The i-th point. Since the method of supervising model training using chamfer distance loss is existing technology, it will not be described in detail here; in addition, in other embodiments, other loss functions can be used to update the parameters of the reconstruction module based on the difference between the reconstructed point cloud and the corresponding real label.

[0065] Similarly, refer to Figure 1 In step 1), the complete point cloud of the instance and its point cloud features are fused with the reconstructed point cloud and its point cloud features according to the one-to-one correspondence between the points in the complete point cloud of the instance and the points in the reconstructed point cloud. The methods for obtaining the point relationship features include:

[0066] Complete point cloud Q based on the example v With the reconstructed point cloud P v The complete point cloud Q instance obtained by matching between them v Points and reconstructed point cloud P v The one-to-one correspondence between points in the point cloud is used to complete the points in the point cloud for each group of corresponding instances. Points in the reconstructed point cloud After the point cloud features of these two points are stitched together and fused, the stitching and fusion result is then processed by a multilayer perceptron subnetwork to obtain the relationship features F of each point. coor .

[0067] Obtain the relational features F of each point coor Then, the relationship features F of each point are... coor The average feature f obtained based on the relationship characteristics of each point corr The specific methods for fusing to obtain pose features include:

[0068] The relationship features F of each point coor The result of the average pooling operation f corr Relationship characteristics F with each point coor The data is then stitched together, processed through a multilayer perceptron subnetwork, and finally subjected to average pooling to obtain the pose features f. pose The method for obtaining pose features is expressed by the following formula:

[0069] f pose =AvgPool(MLP(Concat(F corr Tile M (f corr ))))

[0070] f corr =AvgPool(F corr)

[0071] In the formula, AvgPool represents the average pooling operation, Concat represents feature concatenation, MLP represents the multilayer perceptron, and Tile represents the Tile network. M (·) represents M copies of the feature vector. Based on this, refer to... Figure 1 In this embodiment, the pose features are obtained through the regression module of the category-level pose estimation model (i.e., Figure 1 The Pose Estimation module (corresponding to the content in the module) uses three parallel multilayer perceptron subnetworks to regress the pose features to obtain the estimated pose, which is expressed by the following formula:

[0072] R,t,s=ρ(MLP(f pose MLP(f) pose ),MLP(f pose )

[0073] Where R, t, and s are the three dimensions of the estimated pose, where R represents the rotation of the object in three-dimensional space, t represents the position of the object in the camera coordinate system, and s is a scaling factor representing the size information of the object; ρ(·) represents the transformation network from 6D representation to 3×3 rotation matrix R, and MLP represents multilayer perceptron.

[0074] During the training process (i.e., the model building process described in steps 1-2 above), the regression module's learning is guided and supervised by the ground truth labels corresponding to the estimated pose (i.e., the annotated Group-Truth pose (R′,t′,s′)). That is, the parameters of the regression module are updated based on the difference between the estimated pose and the corresponding ground truth labels. Its loss function L... pose It can be represented as:

[0075] L pose =D pose (R,t,s,R′,t′,s′)=‖RR′‖2+‖tt′‖2+‖ss′‖2

[0076] In the formula, D pose This indicates attitude loss.

[0077] In addition, in this embodiment, the encoder pre-training method includes:

[0078] a) Input the data from the pre-training samples into the pre-training network. The data in the pre-training samples contains point embeddings corresponding to point clouds of instances from a certain category. The masked point embeddings are obtained by processing the data through the labeling masking module in the pre-training network. The data is then reconstructed by the encoder that needs to be pre-trained. The reconstructed points are processed by the masked point modeling module in the pre-training network to obtain point labels.

[0079] b) The pre-training of the encoder is supervised by the tokenizer module based on the real label corresponding to the data input to the pre-trained network. This process is repeated from a) to b) until the iteration stops to complete the pre-training of the encoder.

[0080] Reference Figure 2 In this embodiment, the pre-trained encoder (reconstruction network) is a Transformer encoder; in step a), the data in the training samples contains point embeddings corresponding to the point clouds of instances of a certain category (i.e., Figure 2 Point Embeddings in this context refers to a given set of shapes. in It is the point embedding obtained by inputting the 3D point cloud of instance i from class c into the Pointnet++ module; these point embeddings are then processed by a label mask in a pre-trained network (i.e., Figure 2 The token masking module processes the data to obtain the masked point embeddings, which are then reconstructed (i.e., the point embeddings are recovered) by a pre-trained encoder to learn the correlations between the point embeddings. Figure 2 In this context, cls refers to the category label. The points reconstructed by the pre-trained encoder (i.e., the point embeddings recovered by the pre-trained encoder) are processed by the mask point modeling module (MPM) in the pre-trained network to obtain point labels. The mask point modeling module is used to pre-train the Transformer encoder. Specifically, the mask point modeling module masks a part of the input point cloud and learns to reconstruct the missing point labels in the mask region, so that the pre-trained Transformer encoder can infer the geometry of the missing part based on the remaining part.

[0081] In step b), the tokenizer (i.e. Figure 2 The Tokenizer module takes point embeddings as input and converts them into discrete point tokens, and supervises the training of the Transformer encoder to recover the original point tokens. This trains the encoder to reconstruct a complete shape point cloud from an incomplete one. Specifically, the tokenizer module supervises the pre-training of the encoder based on the ground truth labels corresponding to the point tokens and the data input to the pre-trained network. Since the specific method of supervising the pre-training of the encoder is existing technology, it will not be elaborated here.

[0082] Example of pose estimation method

[0083] This embodiment provides a pose estimation method, which specifically includes: inputting the image of the target object instance whose pose is to be estimated, the observation point cloud, and the category-level shape prior point cloud into the pose estimation model to obtain the pose estimation result;

[0084] The pose estimation model in this embodiment is constructed using the method for establishing a category-level pose estimation model as described in the above embodiment.

[0085] Since the specific working principle and effect of the pose estimation model constructed in the pose estimation method of this embodiment have been described in detail in the above embodiment of the method for establishing the category-level pose estimation model, they will not be repeated here.

[0086] Computer System Implementation Examples

[0087] This embodiment provides a technical solution for a computer system, which includes a processor storing executable program instructions. These executable program instructions are executed to implement the pose estimation method in the above-described pose estimation method embodiment.

[0088] Since the specific working principle and effect of the computer system in this embodiment have been described in detail in the pose estimation method embodiment, they will not be repeated here.

[0089] Computer-readable storage medium embodiments

[0090] This embodiment provides a technical solution for a computer-readable storage medium storing computer program instructions. When executed, these computer program instructions implement the pose estimation method as described in the above-described pose estimation method embodiment.

[0091] Since the specific working principle and effect of the computer-readable storage medium in this embodiment have been described in detail in the above-described pose estimation method embodiment, they will not be repeated here.

[0092] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or explanatory of the principles of the present invention, and do not constitute a limitation thereof.

Claims

1. A method for establishing a category-level pose estimation model, characterized in that, include: 1) Based on the features extracted from the images and observation point clouds of the target object instances in the training set, the global features of the images and observation point clouds are obtained respectively, and together with the features extracted from the category-level shape prior point cloud of the target object instances, they are input into the deformation module of the model for feature fusion to obtain the complete point cloud of the instance in the normal space and its point cloud features. The fused features obtained by fusing features extracted from the images of target object instances in the training set and the observed point clouds are input into the reconstruction module of the model to obtain the reconstructed point cloud and its point cloud features; the reconstruction module contains a pre-trained reconstruction network. The complete point cloud of the instance and its point cloud features are fused with the reconstructed point cloud and its point cloud features according to the one-to-one correspondence between the points in the complete point cloud of the instance and the points in the reconstructed point cloud to obtain the point relationship features; the point relationship features are then fused with the average features obtained based on the point relationship features to obtain the pose features. The estimated pose is obtained by regressing the pose features through the regression module of the model. 2) Adjust the parameters of the deformation module, reconstruction module, and regression module respectively; repeat 1)-2) until the iteration stops to complete the establishment of the category-level pose estimation model.

2. The method for establishing a category-level pose estimation model according to claim 1, characterized in that, The reconstruction network is an encoder, and the pre-training methods for the reconstruction network include: a) Input the data from the pre-training samples into the pre-training network. The data in the pre-training samples contains point embeddings corresponding to point clouds of instances from a certain category. The masked point embeddings are processed by the labeling masking module in the pre-training network and then reconstructed by the encoder that needs to be pre-trained. The reconstructed points are processed by the masked point modeling module in the pre-training network to obtain point labels. b) The pre-training of the encoder is supervised by the tokenizer module based on the real label corresponding to the data input to the pre-trained network. Repeat a)-b) until the iteration stops to complete the pre-training of the encoder.

3. The method for establishing a category-level pose estimation model according to claim 1 or 2, characterized in that, The methods for inputting the obtained fused features into the reconstruction module of the model to obtain the reconstructed point cloud and its point cloud features include: After the obtained fused features are processed by a multilayer perceptron subnetwork in the reconstruction module, they are then reconstructed by a pre-trained reconstruction network in the reconstruction module to obtain the point cloud features of the reconstructed point cloud; then the point cloud features of the reconstructed point cloud are processed by another multilayer perceptron subnetwork in the reconstruction module to obtain the reconstructed point cloud.

4. The method for establishing a category-level pose estimation model according to claim 1 or 2, characterized in that, The global features of the image and observed point cloud are obtained and input together with features extracted from the shape prior point cloud of the target object instance into the deformation module for feature fusion. The methods to obtain the complete point cloud of the instance in the normal space and its point cloud features include: After stitching together the global features of the image and the observed point cloud from the input deformation module, as well as the features extracted from the shape prior point cloud of the target object instance, the stitching result is processed by a multilayer perceptron subnetwork in the deformation module to obtain deformation domain features. The deformation domain features are then added element-wise to the features extracted from the shape prior point cloud of the target object instance to obtain the point cloud features of the instance complete point cloud in the normal space. The point cloud features of the instance complete point cloud in the normal space are then processed by another multilayer perceptron subnetwork in the deformation module to obtain the instance complete point cloud in the normal space.

5. The method for establishing a category-level pose estimation model according to claim 1 or 2, characterized in that, The methods for obtaining global features of images and observation point clouds based on features extracted from target object instances in the training set include: Average pooling is performed on the features extracted from the image of the target object instance in the training set and the features extracted from the observation point cloud of the target object instance to obtain the global features of the image and the global features of the observation point cloud, respectively.

6. The method for establishing a category-level pose estimation model according to claim 1 or 2, characterized in that, The complete point cloud of the instance and its point cloud features are fused with the reconstructed point cloud and its point cloud features according to the one-to-one correspondence between the points in the complete point cloud of the instance and the points in the reconstructed point cloud. The methods for obtaining the point relationship features include: Based on the one-to-one correspondence between the points in the complete point cloud of the instance and the points in the reconstructed point cloud obtained by matching the complete point cloud of the instance and the points in the reconstructed point cloud, the points in the complete point cloud of the instance with each corresponding relationship and the points in the reconstructed point cloud, as well as the point cloud features of these two points, are spliced ​​and fused. Then, a multilayer perceptron sub-network is used to process the splicing and fusion results to obtain the relationship features of each point.

7. The method for establishing a category-level pose estimation model according to claim 1 or 2, characterized in that, The methods for adjusting the parameters of the deformation module, reconstruction module, and regression module respectively include: The parameters of the deformation module, reconstruction module, and regression module are adjusted based on the differences between the complete point cloud and the corresponding real label, the differences between the reconstructed point cloud and the corresponding real label, and the differences between the estimated pose and the corresponding real label, respectively.

8. A pose estimation method, characterized in that, include: Input the image of the target object instance whose pose is to be estimated, the observation point cloud, and the category-level shape prior point cloud into the pose estimation model to obtain the pose estimation result; The pose estimation model is constructed using the method for establishing a category-level pose estimation model as described in any one of claims 1-7.

9. A computer system comprising a processor, the processor being configured to execute executable program instructions, characterized in that, The executable program instructions are executed to implement the pose estimation method according to claim 8.

10. A computer-readable storage medium storing computer program instructions, characterized in that, The computer program instructions are used to implement the pose estimation method as described in claim 8 when executed by a processor.

Images