A class-level 6D pose estimation method based on geometric information enhancement

Abstract

A category-level 6D pose estimation method based on geometric information enhancement is proposed. This method introduces a normal angle map as an additional depth representation and designs a depth geometric information enhancement module to fuse the geometric features and appearance texture features of the object, thereby enabling the pose estimation model to better perceive the geometric shape information of the object. Furthermore, a category shape prior is introduced and adaptively matched to the geometric deformation of the target instance is used to overcome the impact of large intra-class differences between different objects within the same category on the model's prediction accuracy and generalization. First, instance segmentation is performed on the input single RGB-D image to obtain the target instance information. The normal angle map and the point cloud of the instance in 3D space are calculated by combining the depth map, and the corresponding category shape prior is input according to the instance's category. Based on a multi-branch network structure, the instance's geometric features and category prior features are fused separately to reconstruct the 3D point cloud of the target instance in the normalized object coordinate space. Finally, the pose information of the target object is calculated.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D scene understanding, specifically a category-level 6D pose estimation method based on geometric information enhancement. Background Technology

[0002] 6D pose estimation is an important task in the field of 3D vision, aiming to recover the pose and size of objects in an image in 3D space. Compared to traditional instance-level pose estimation tasks, category-level tasks can predict the pose of different objects within a category without acquiring a 3D model, and have a wider range of applications. However, due to the lack of explicit shape guidance from a 3D model, and the significant differences in appearance and shape among different instances within the same category, category-level pose estimation methods often face challenges in prediction accuracy and generalization.

[0003] The differences between the existing technologies are as follows:

[0004] In comparison with the technology of patent CN114863573A "A Category-Level 6D Pose Estimation Method Based on Monocular RGB-D Images"

[0005] Patent CN114863573A discloses a self-attention module that fuses pixel color features and point cloud geometric features of a target instance in an RGB-D image. This patent utilizes a depth map approach, employing only the point cloud P projected into 3D space. The geometric features of the point cloud are then fused with the pixel appearance features using a traditional attention mechanism. The point cloud features are used as the query tensor Q, and pixel features are used to calculate K and V. The feature encoder involved in this patent consists of three branches: feature extraction.

[0006] This patent relates to a progressive depth geometry enhancement module that fuses the geometric and appearance features of a target instance in an RGB-D image. This patent not only projects the depth map into a 3D point cloud but also calculates a normal angle map as additional local geometric planar features to supplement the depth geometry information. Before applying the attention module, the point cloud geometric features are first enhanced using the normal angle map. The designed cross-attention module uses the enhanced geometric features to calculate the query tensor Q and uses pixel features to calculate the key-value tensors K and V. The feature encoder involved in this patent consists of four feature extraction branches.

[0007] In patent CN114863573A, a FoldingNet point cloud autoencoder is used to generate a category shape prior. The static category shape prior is used to reconstruct the target instance point cloud, and finally, the traditional Umeyama algorithm is used to solve the 6D pose. A combined indirect loss function is designed to supervise the complete point cloud model of the intermediate output of the network and the model in the NOCS coordinate space (bevel distance loss, smoothing L1 loss) to train the pose estimation network.

[0008] In this patent, a point cloud autoencoder based on PointNet is used to generate category shape priors. Furthermore, an adaptive fusion module based on complementary structural differences is designed to dynamically fuse shape priors and instance geometric features to reconstruct the target instance model. Finally, pose-sensitive features are modeled and 6D pose is directly regressed. In addition to the intermediate point cloud output of the indirect supervision network, this patent also introduces a direct pose consistency loss, using normalized pose labels and normalized predicted poses during the loss function calculation.

[0009] In comparison with the technology of patent CN117351078A "Target Size and 6D Attitude Estimation Method Based on Shape Priority"

[0010] Patent CN117351078A discloses a pixel-level dense fusion method that fuses the pixel appearance features and geometric features of a target instance in an RGB-D image. This patent utilizes a depth map approach, projecting it onto a point cloud P in three-dimensional space. Each point in P corresponds to a pixel in image I. The geometric features of each point are fused with its corresponding color features at the pixel level. The feature encoder in this patent consists of three feature extraction branches.

[0011] This patent relates to a progressive depth geometry enhancement module that fuses the geometric and appearance features of a target instance in an RGB-D image. This patent not only projects the depth map into a 3D point cloud but also calculates a normal angle map as additional local geometric planar features to supplement the depth geometry information. Before applying the attention module, the point cloud geometric features are first enhanced using the normal angle map. The designed cross-attention module uses the enhanced geometric features to calculate the query tensor Q and uses pixel features to calculate the key-value tensors K and V. The feature encoder involved in this patent consists of four feature extraction branches.

[0012] Patent CN114863573A uses a UMLP-based point cloud autoencoder to generate category shape priors. It proposes to incorporate prior deformations into image patch I and fuse point cloud features in advance to reconstruct the target instance point cloud. Finally, it uses the traditional Umeyama algorithm to solve the 6D pose. A combined indirect loss function is designed to supervise the complete point cloud model of the intermediate output of the network and the model in the NOCS coordinate space (bevel distance loss, smoothing L1 loss) to train the pose estimation network.

[0013] In this patent, a PointNet-based point cloud autoencoder is used to generate category shape priors. Furthermore, a prior adaptive fusion module based on complementary structural differences is designed. This module utilizes a mutual attention mechanism to complement the structural differences between instance point cloud features and prior point clouds, ultimately modeling pose-sensitive features and directly regressing 6D pose. In addition to the intermediate point cloud output of the indirect supervision network, this patent also introduces a direct pose consistency loss, using normalized pose labels and normalized predicted poses during the loss function calculation. Summary of the Invention

[0014] To address the aforementioned technical problems, this invention proposes a category-level 6D pose estimation method based on geometric information enhancement. This method fully utilizes the geometric shape information of object instances in RGB-D images and the category-shared information provided by category geometric priors, enabling accurate prediction of the 6D pose of objects even when the object's 3D model is unknown.

[0015] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0016] A category-level 6D pose estimation method based on geometric information augmentation includes a preprocessing stage, a feature encoding stage, and a pose estimation stage, as follows:

[0017] 1) Preprocessing stage:

[0018] A series of processes are performed on the input RGB-D image to obtain the appearance and depth geometry information of the object instance, and to obtain the shape prior information of the corresponding category;

[0019] 2) Feature encoding stage:

[0020] The preprocessed input is processed by a multi-branch feature extraction network. The designed deep geometric information enhancement module enhances the network’s perception of the geometric and appearance features of the instance. Furthermore, the class-shared features provided by the class shape prior are used to perform structural complementary fusion with the instance geometric information, thereby effectively reconstructing the 3D point cloud of the target instance in the standardized coordinate space.

[0021] 3) Attitude estimation stage:

[0022] After the feature encoding stage is completed, the dense point cloud features of the object instance in the observation space and the normalized object space are obtained. The correspondence between the two is modeled and the pose-sensitive features are further obtained. Finally, the 6D pose and size of the target instance are obtained by decoding through a multilayer perceptron.

[0023] Furthermore, step 1) the preprocessing stage specifically includes the following steps:

[0024] Step 1.1: Input the RGB image into the existing instance segmentation network Mask-RCNN to obtain the target instance mask, segment the target instance from the RGB-D image, and obtain the instance RGB map denoted as I and the instance depth map denoted as D;

[0025] Step 1.2: Calculate the normal map using the depth map D of the target instance, and calculate the normal vector N(n) for each pixel using the least squares local plane fitting method. x n y n z Then, the angles between the normal vector and the three coordinate axes are calculated using the following formula:

[0026] d x =arccos(N·x),

[0027] d y =arccos(N·y),

[0028] d z =arccos(N·z).

[0029] The angles are normalized to the range of 0-255 to obtain a two-dimensional normal angle map, where the three channels of each pixel are the normalized normal angles.

[0030] Step 1.3: Using the depth map D of the target instance, obtain the three-dimensional coordinates corresponding to each pixel based on the camera intrinsic parameter matrix and the back projection of the depth map, and randomly sample to obtain the three-dimensional spatial point set P of the object;

[0031] Step 1.4: Obtain the category shape prior F of the target instance based on the category information of instance segmentation. pr .

[0032] Furthermore, step 2) of the training phase specifically includes the following steps:

[0033] Step 2.1: Extract the geometric features F of the instance using a multi-branch feature extraction network. nrm F po and image features F rgb Geometric features F of category shape prior pr ;

[0034] Step 2.2: Through the designed depth geometry information enhancement module, local geometric features of the object surface are fused, and the instance appearance features are adaptively adjusted. The output instance appearance features F are then processed. rgb The fusion of point cloud features and geometrically enhanced instance features F is obtained. ins ;

[0035] Step 2.3: Through a mutual attention mechanism, the differences in category shape prior and instance point cloud structure are propagated complementaryly to obtain instance point cloud features that can perceive the overall structural information of the category shape prior, and category geometric features F that can perceive the diverse structural information of instances. cat ;

[0036] Step 2.4: Use the deformation prediction network to obtain the deformation field features F of the instance relative to the class shape prior. D The geometric features of the category are deformed to obtain the complete point cloud features of the target instance. Further, a complete point cloud of the instance is obtained through a multilayer perceptron. Calculate the error in reconstructing the complete point cloud in the example;

[0037] Step 2.5: Concatenate the target instance features with the complete point cloud features along the channel dimension, and then predict the correlation matrix using a multilayer perceptron, as shown in the following formula;

[0038]

[0039] Step 2.6: Based on the correlation matrix and the complete point cloud of the target instance, calculate the reconstruction error of the instance in the normalized coordinate space. Obtain the coordinates Q of the target instance in the normalized coordinate space through matrix multiplication. o and its corresponding depth features

[0040] Q o =A corr ×Q v

[0041]

[0042] Step 2.7: The loss function introduces a smooth L1 reconstruction loss for point clouds in normalized coordinate space:

[0043]

[0044] Where x = (x1, x2, x3) ∈ Q o One of the coordinate prediction values ​​in the representation. Represents the actual value.

[0045] Furthermore, the specific steps in step 2.3 of the feature encoding stage are as follows;

[0046] First, the normal angle map pixel features aligned point-by-point with the observed point cloud are selected. Then, these features are concatenated along the channel dimension. The resulting feature map is then fused through a linear layer, and geometric features F are concatenated along the channel dimension. nrm and F po Local geometric features F on the object surface are fused through linear layers.nrm and F po The resulting point cloud features, which incorporate local geometric information, retain the spatial information of the original point cloud features while enhancing the model's ability to understand the shape and surface structure of objects. The formula is as follows:

[0047] F′ po =Linear(Concat(F po F nrm )

[0048] The point cloud features F′ after point-by-point alignment po The RGB features are mapped to the same feature space through a 1×1 convolution to obtain V. r K r =MLP(F rgb ), Q r =MLP(F′) po Then, the element-wise dot product attention mechanism is applied to calculate the attention weights of the depth features in the RGB features, and this attention weight matrix is ​​applied to the RGB features V. r Above, to highlight the RGB feature parts related to depth information;

[0049] F′ rgb =F rgb +Attention(F′ po W Q F rgb W K F rgb W V ).

[0050] Furthermore, the specific steps of module 2.4 in the training phase are as follows:

[0051] Given the geometric features F of an instance po and category prior geometric features F pr First, perform 1×1 convolutions on each channel to obtain channel compression features. To model structural difference information, the instance features and prior features are concatenated and passed through a linear layer. The resulting features are then average-pooled and concatenated with the original features to enhance the perception of global structure. Finally, a 1×1 convolution is applied to obtain... To achieve the introduction of self-attention and cross-attention of geometric information, a corresponding attention mechanism was designed, utilizing the obtained feature map K. po V po First, the structural relation matrix is ​​obtained by performing matrix multiplication on Q and K. Then, the attention weight matrix is ​​obtained by performing softmax normalization on this matrix.

[0052] A po=softmax(Q) T K po )

[0053] Each element represents the correlation between instance features and global structural features, and the instance features F′ are weighted according to the calculated attention weights. po The weighted features are then fused with the original features to obtain instance features that incorporate complete category shape information, as shown in the following formula:

[0054]

[0055] For the category prior point cloud features, the same attention enhancement is applied to obtain category features that combine instance-specific geometric information.

[0056]

[0057] Through the above calculations, the intra- and inter-modal correlations between each spatial location of the deep features and all locations are obtained, and the information enhancement of key spatial locations is achieved in the fused features, which is used to reconstruct a more accurate standardized coordinate space model.

[0058] Furthermore, in step 2.5 of the feature encoding stage:

[0059] The two branches of the network were trained through backpropagation based on the errors in point cloud reconstruction and coordinate prediction, respectively. The point cloud reconstruction error is as follows:

[0060]

[0061] Q v To reconstruct the predicted values ​​of the point cloud, A standard normalized 3D point cloud model of the object.

[0062] Furthermore, step 3) the attitude estimation stage specifically includes the following steps:

[0063] Step 3.1: Establish a correspondence between the features of the target instance obtained in step 2) in the observation space and the features in the normalized object coordinate space through feature concatenation, and apply a multilayer perceptron to fuse the features, as shown in the following formula;

[0064]

[0065] Step 3.2: Connect F corr The average features of F are used to inject global information into F. corr In the point-to-point correspondence, this correspondence is strengthened by a multilayer perceptron, and the pose-sensitive feature vector F is obtained from the correspondence using average pooling. pose The formula is:

[0066] F pose =AvgPool(MLP(concat(F corr Tile N (F′ corr )))),

[0067] stF′ corr =AvgPool(F corr ).

[0068] Step 3.3: Input the pose features into three parallel multilayer perceptrons, regress the instance pose information [R|t] and size information s respectively, and calculate the pose estimation error. If the model is an inference process, this stage ends; if it is a training process, proceed to step 3.4.

[0069] Step 3.4: Reconstruct the point cloud, the coordinates of the target instance in the normalized coordinate space, and the pose and size of the final output using the target instance output from the intermediate network, and calculate the overall loss;

[0070] Step 3.5: Perform gradient backpropagation of the loss function to optimize model parameters.

[0071] Beneficial effects:

[0072] The method of this invention can predict the 6D pose of a target object from an RGB-D image, overcoming the dependence on the object's 3D model in 6D pose estimation. It proposes a novel category-level pose estimation framework that introduces a normal angle map as an additional depth representation, fully utilizing instance depth geometry, potential geometric-semantic associations, and instance-class structural differences to enhance the exploration of intra-class shape information. This framework can better adapt to changes in the appearance and shape of different objects within the same class in a scene, achieving robust 6D pose estimation. Attached Figure Description

[0073] Figure 1 This is a network structure diagram of the category-level 6D pose estimation method according to an embodiment of the present invention;

[0074] Figure 2 This is a flowchart illustrating the training process of the category-level 6D pose estimation method according to an embodiment of the present invention.

[0075] Figure 3 This is a flowchart illustrating the reasoning process of the category-level 6D pose estimation method according to an embodiment of the present invention. Detailed Implementation

[0076] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0077] The model structure of this invention is as follows Figure 1 The flowchart of the training process is shown below. Figure 2 The flowchart of the reasoning stage is shown below. Figure 3 This paper presents a category-level 6D pose estimation method based on geometric information augmentation, which includes a preprocessing stage, a feature encoding stage, and a pose estimation stage. The stages are as follows:

[0078] 1) Preprocessing stage:

[0079] A series of processes are performed on the input RGB-D image to obtain the appearance and depth geometry information of the object instance, and to obtain the shape prior information of the corresponding category.

[0080] Step 1) The preprocessing stage specifically includes the following steps:

[0081] Step 1.1: Input the RGB image into the existing instance segmentation network Mask-RCNN to obtain the target instance mask, segment the target instance from the RGB-D image, and obtain the instance RGB map denoted as I and the instance depth map denoted as D.

[0082] Step 1.2: Calculate the normal map using the depth map D of the target instance, and calculate the normal vector N(n) for each pixel using the least squares local plane fitting method. x n y n z Then, the angles between the normal vector and the three coordinate axes are calculated using the following formula:

[0083] d x =arccos(N·x),

[0084] d y =arccos(N·y),

[0085] d z =arccos(N·z).

[0086] By normalizing the angles to the range of 0-255, a two-dimensional normal angle map is obtained, where the three channels of each pixel are the normalized normal angles.

[0087] Step 1.3: Using the depth map D of the target instance, obtain the three-dimensional coordinates corresponding to each pixel based on the camera intrinsic parameter matrix and the back projection of the depth map, and randomly sample to obtain the three-dimensional spatial point set P of the object.

[0088] Step 1.4: Obtain the category shape prior F of the target instance based on the category information of instance segmentation. pr .

[0089] 2) Feature encoding stage:

[0090] The preprocessed input is processed by a multi-branch feature extraction network. The designed deep geometric information enhancement module enhances the network’s perception of the geometric and appearance features of the instance. Furthermore, the class-shared features provided by the class shape prior are used to perform structural complementary fusion with the instance geometric information, thereby effectively reconstructing the 3D point cloud of the target instance in the standardized coordinate space.

[0091] Step 2) The training phase specifically includes the following steps:

[0092] Step 2.1: Extract the geometric features F of the instance using a multi-branch feature extraction network. nrm F po and image features F rgb Geometric features F of category shape prior pr .

[0093] Step 2.2: Through the designed depth geometry information enhancement module, local geometric features of the object surface are fused, and the instance appearance features are adaptively adjusted. The output instance appearance features F are then processed. rgb The fusion of point cloud features and geometrically enhanced instance features F is obtained. ins .

[0094] Step 2.3: Through a mutual attention mechanism, the differences in category shape prior and instance point cloud structure are propagated complementaryly to obtain instance point cloud features that can perceive the overall structural information of the category shape prior, and category geometric features F that can perceive the diverse structural information of instances. cat .

[0095] In step 2.3 of the feature encoding stage:

[0096] A deep geometry enhancement mechanism is implemented by first selecting the pixel features of the normal angle map aligned point-by-point with the observed point cloud, then concatenating them with the features in the channel dimension. The resulting feature map is then fused through a linear layer, and finally, geometric features F are concatenated in the channel dimension. nrm and F po Local geometric features F on the object surface are fused through linear layers. nrm and F po This yields point cloud features that incorporate local geometric information. While preserving the original point cloud feature spatial information, the combination of local geometric features (normal vector map features) enhances the model's ability to understand the shape and surface structure of objects. The formula is as follows:

[0097] F′ po =Linear(Concat(F po F nrm )

[0098] In this module, a geometric information perception module was further designed to process the point cloud features F′ after point-by-point alignment. po The RGB features are mapped to the same feature space through a 1×1 convolution to obtain V. r K r =MLP(F rgb ), Q r =MLP(F′) po Then, the element-wise dot product attention mechanism is applied to calculate the attention weights of the depth features in the RGB features, and this attention weight matrix is ​​applied to the RGB features V. r Above, to highlight the RGB feature parts related to depth information.

[0099] F′ rgb =F rgb +Attention(F′ po W Q F rgb W K F rgb W V );

[0100] Step 2.4: Use the deformation prediction network to obtain the deformation field features F of the instance relative to the class shape prior. D The geometric features of the category are deformed to obtain the complete point cloud features of the target instance. Further, a complete point cloud of the instance is obtained through a multilayer perceptron. Calculate the error in reconstructing a complete point cloud instance.

[0101] In step 2.4 of the training phase:

[0102] A shape-prior adaptive fusion module is designed, which adaptively learns the structural relationship between instance point cloud features and category prior features, which is beneficial for constructing their structural difference information at the feature level. The module is specifically designed as follows: Given instance geometric features F po and category prior geometric features F pr First, perform 1×1 convolutions on each channel to obtain channel compression features. Simultaneously, to model structural difference information, the instance features and prior features are concatenated and passed through a linear layer. The resulting features are then average-pooled and concatenated with the original features to enhance the perception of global structure. Finally, a 1×1 convolution is applied to obtain... It incorporates rich structural information from instance and category priors, enabling the introduction of geometric self-attention and cross-attention. To fully utilize the differentiated information of instance and category geometric features, a corresponding attention mechanism is designed. Taking the design of instance point cloud feature attention as an example, it utilizes the obtained feature map K... po Vpo First, the structural relation matrix is ​​obtained by performing matrix multiplication on Q and K. Then, the attention weight matrix is ​​obtained by performing softmax normalization on this matrix.

[0103] A po =softmax(Q) T K po )

[0104] Each element represents the correlation between instance features and global structural features. The instance features F′ are then weighted according to the calculated attention weights. po The weighted features are then fused with the original features to obtain instance features that incorporate complete category shape information, as shown in the following formula:

[0105]

[0106] For the category prior point cloud features, the same attention enhancement is applied to obtain category features that combine instance-specific geometric information.

[0107]

[0108] Through the above calculations, the intra- and inter-modal correlations between each spatial location of the deep features and all locations can be obtained, and the information enhancement of key spatial locations can be achieved in the fused features, which can be used to reconstruct a more accurate standardized coordinate space model.

[0109] Step 2.5: Concatenate the target instance features with the complete point cloud features along the channel dimension, and then predict the correlation matrix using a multilayer perceptron, as shown in the following formula.

[0110]

[0111] In step 2.5 of the feature encoding stage:

[0112] The two branches of the network were trained through backpropagation based on the errors in point cloud reconstruction and coordinate prediction, respectively. The point cloud reconstruction error is as follows:

[0113]

[0114] Q v To reconstruct the predicted values ​​of the point cloud, A standard normalized 3D point cloud model of the object.

[0115] Step 2.6: Based on the correlation matrix and the complete point cloud of the target instance, calculate the reconstruction error of the instance in the normalized coordinate space. Obtain the coordinates Q of the target instance in the normalized coordinate space through matrix multiplication. o and its corresponding depth features

[0116] Q o =A corr ×Q v

[0117]

[0118] Step 2.7: The loss function introduces a smooth L1 reconstruction loss for point clouds in normalized coordinate space:

[0119]

[0120] Where x = (x1, x2, x3) ∈ Q o One of the coordinate prediction values ​​in the representation. Represents the actual value.

[0121] 3) Attitude estimation stage:

[0122] After the feature encoding stage is completed, the dense point cloud features of the object instance in the observation space and the normalized object space are obtained. The correspondence between the two is modeled and the pose-sensitive features are further obtained. Finally, the 6D pose and size of the target instance are obtained by decoding through a multilayer perceptron.

[0123] Step 3) The attitude estimation stage specifically includes the following steps:

[0124] Step 3.1: Establish a correspondence between the features of the target instance obtained in step 2) in the observation space and the features in the normalized object coordinate space through feature concatenation, and apply a multilayer perceptron to fuse the features, as shown in the following formula.

[0125]

[0126] Step 3.2: Connect F corr The average features of F are used to inject global information into F. corr In the point-to-point correspondence, this correspondence is strengthened by a multilayer perceptron, and the pose-sensitive feature vector F is obtained from the correspondence using average pooling. pose The formula is:

[0127] F pose =AvgPool(MLP(concat(F corr Tile N (F′ corr )))),

[0128] stF′ corr =AvgPool(F corr ).

[0129] Step 3.3: Input the pose features into three parallel multilayer perceptrons to regress the instance pose information [R|t] and size information s, and calculate the pose estimation error. If the model is an inference process, this stage ends. If it is a training process, proceed to step 3.4.

[0130] Step 3.4: Reconstruct the point cloud, the coordinates of the target instance in the normalized coordinate space, and the pose and size of the final output using the target instance output from the intermediate network, and calculate the overall loss.

[0131] Step 3.5: Perform gradient backpropagation of the loss function to optimize model parameters.

[0132] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A category-level 6D pose estimation method based on geometric information augmentation, comprising a preprocessing stage, a feature encoding stage, and a pose estimation stage, characterized in that, The stages are as follows: 1) Preprocessing stage: A series of processes are performed on the input RGB-D image to obtain the appearance and depth geometry information of the object instance, and to obtain the shape prior information of the corresponding category; 2) Feature encoding stage: The preprocessed input is processed by a multi-branch feature extraction network. The designed deep geometric information enhancement module enhances the network’s perception of the geometric and appearance features of the instance. Furthermore, the class-shared features provided by the class shape prior are used to perform structural complementary fusion with the instance geometric information, thereby effectively reconstructing the 3D point cloud of the target instance in the standardized coordinate space. Step 2) The training phase specifically includes the following steps: Step 2.1: Extract the geometric features of the instances separately using a multi-branch feature extraction network. , and image features Geometric features of category shape prior ; Step 2.2: Through the designed depth geometry information enhancement module, local geometric features of the object surface are fused, and the instance appearance features are adaptively adjusted. The output instance appearance features are then displayed. By fusing point cloud features, geometrically enhanced instance features are obtained. ; Step 2.3: Through a mutual attention mechanism, the differences in category shape prior and instance point cloud structure are propagated complementaryly to obtain instance point cloud features that can perceive the overall structural information of the category shape prior, as well as category geometric features that can perceive the diverse structural information of instances. ; Step 2.4: Use the deformation prediction network to obtain the deformation field features of the instance relative to the class shape prior. The geometric features of the category are deformed to obtain the complete point cloud features of the target instance. Furthermore, a complete point cloud of the instance is obtained through a multilayer perceptron. Calculate the error in reconstructing the complete point cloud of the instance; Step 2.5: Concatenate the target instance features with the complete point cloud features along the channel dimension, and then predict the correlation matrix using a multilayer perceptron, as shown in the following formula; ； Step 2.6: Based on the correlation matrix and the complete point cloud of the target instance, calculate the reconstruction error of the instance in the normalized coordinate space, and obtain the coordinates of the target instance in the normalized coordinate space through matrix multiplication. and its corresponding depth features , ； ； Step 2.7: The loss function introduces a smooth L1 reconstruction loss for point clouds in normalized coordinate space: ； in One of the coordinate prediction values ​​in the representation. Represents the actual value; 3) Attitude estimation stage: After the feature encoding stage is completed, the dense point cloud features of the object instance in the observation space and the normalized object space are obtained. The correspondence between the two is modeled and the pose-sensitive features are further obtained. Finally, the 6D pose and size of the target instance are obtained by decoding through a multilayer perceptron.

2. The category-level 6D pose estimation method based on geometric information enhancement according to claim 1, characterized in that: Step 1) The preprocessing stage specifically includes the following steps: Step 1.1: Input the RGB image into the existing instance segmentation network Mask-RCNN to obtain the target instance mask, segment the target instance from the RGB-D image, and obtain the instance RGB map denoted as I and the instance depth map denoted as D; Step 1.2: Calculate the normal map using the depth map D of the target instance, and calculate the normal vector of each pixel using the least squares local plane fitting method. Then, the angles between the normal vector and the three coordinate axes are calculated using the following formula: ； The angles are normalized to the range of 0-255 to obtain a two-dimensional normal angle map, where the three channels of each pixel are the normalized normal angles. Step 1.3: Using the depth map D of the target instance, obtain the three-dimensional coordinates corresponding to each pixel based on the camera intrinsic parameter matrix and the back projection of the depth map, and randomly sample to obtain the three-dimensional spatial point set P of the object; Step 1.4: Obtain the category shape prior of the target instance based on the category information of instance segmentation. .

3. The category-level 6D pose estimation method based on geometric information augmentation according to claim 1, characterized in that: The specific steps in step 2.3 of the feature encoding stage are as follows; First, the normal angle map pixel features aligned point-by-point with the observed point cloud are selected. Then, these features are concatenated along the channel dimension. The resulting feature map is then fused through a linear layer, and geometric features are concatenated along the channel dimension. and Local geometric features of an object's surface are fused through linear layers. and The resulting point cloud features, which incorporate local geometric information, retain the spatial information of the original point cloud features while enhancing the model's ability to understand the shape and surface structure of objects. The formula is as follows: ； Point-by-point aligned depth point cloud features The RGB features are mapped to the same feature space through a 1×1 convolution, resulting in... , Then, the element-wise dot product attention mechanism is applied to calculate the attention weights of the depth features in the RGB features, and this attention weight matrix is ​​applied to the RGB features. Above, to highlight the RGB feature parts related to depth information; 。 4. The category-level 6D pose estimation method based on geometric information augmentation according to claim 1, characterized in that: The specific steps of module 2.4 in the training phase are as follows: Given instance geometric features and category prior geometric features First, perform 1×1 convolutions on each channel to obtain the channel compression features. , To model structural difference information, the instance features and prior features are concatenated and passed through a linear layer. The resulting features are then average-pooled and concatenated with the original features to enhance the perception of global structure. Finally, a 1×1 convolution is applied to obtain... To achieve the introduction of self-attention and cross-attention of geometric information, a corresponding attention mechanism was designed, utilizing the obtained feature maps. , First, the structural relation matrix is ​​obtained by performing matrix multiplication on Q and K. Then, the attention weight matrix is ​​obtained by performing softmax normalization on this matrix. ； Each element represents the correlation between instance features and global structural features, and the instance features are weighted according to the calculated attention weights. The weighted features are then fused with the original features to obtain instance features that incorporate complete category shape information, as shown in the following formula: ； For the category prior point cloud features, the same attention enhancement is applied to obtain category features that combine instance-specific geometric information; ； Through the above calculations, the intra- and inter-modal correlations between each spatial location of the deep features and all locations are obtained, and the information enhancement of key spatial locations is achieved in the fused features, which is used to reconstruct a more accurate standardized coordinate space model.

5. The category-level 6D pose estimation method based on geometric information enhancement according to claim 1, characterized in that: In step 2.5 of the feature encoding stage: The two branches of the network were trained through backpropagation based on the errors in point cloud reconstruction and coordinate prediction, respectively. The point cloud reconstruction error is as follows: ； in To reconstruct the predicted values ​​of the point cloud, A standard normalized 3D point cloud model of the object.

6. The category-level 6D pose estimation method based on geometric information augmentation according to claim 1, characterized in that: Step 3) The attitude estimation stage specifically includes the following steps: Step 3.1: Establish a correspondence between the features of the target instance obtained in Step 2) in the observation space and the features in the normalized object coordinate space through feature concatenation. Apply a multilayer perceptron to fuse the features, as shown in the following formula; ； Step 3.2: Connect The average features, injecting global information into In the point-to-point correspondence, this correspondence is strengthened by a multilayer perceptron, and the pose-sensitive feature vector is obtained from the correspondence using average pooling. The formula is: ； Step 3.3: Input the pose features into three parallel multilayer perceptrons, and regress them to obtain the instance pose information. and size information The attitude estimation error is calculated. If the model follows an inference process, this stage ends; otherwise, proceed to step 3.

4. Step 3.4: Reconstruct the point cloud, the coordinates of the target instance in the normalized coordinate space, and the pose and size of the final output using the target instance output from the intermediate network; calculate the overall loss. Step 3.5: Perform gradient backpropagation of the loss function to optimize model parameters.

Images