Category-level 6d object pose estimation method and system based on topology-aware and semantic mamba

Abstract

The application discloses a category-level 6D object pose estimation method and system based on topology perception and semantic Mamba, the method comprising: acquiring an RGB image and point cloud data of an object to be estimated, and performing pretreatment; constructing a feature extraction network comprising a topology feature encoder, a visual feature extractor and an instance adaptive key point detector to generate an initial key point set; constructing a feature aggregation network comprising a local-global feature aggregator to globally optimize key point features to obtain optimized key point features; and inputting the optimized key point features into a pose regression head to predict 6D pose parameters and size information of a target object. By introducing a parameter-free topology feature encoder, topology entropy and Betti curve and other topology features are extracted from point cloud data based on the theory of persistent homology, the generalization ability of the model in a complex geometric structure scene is improved, and the semantic consistency and overall structure constraint ability of key point features are enhanced.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robot perception technology, specifically to a method and system for class-level 6D object pose estimation based on topological perception and semantic Mamba. Background Technology

[0002] Category-level 6D object pose estimation aims to predict the rotation, translation, and size of unseen instances within the same category in 3D space. Unlike instance-level pose estimation, which relies on accurate CAD models, category-level tasks need to handle significant differences in shape, texture, and geometry among objects within a class. Existing category-level 6D pose estimation methods are mainly divided into shape-prior methods and shape-prior-free methods. Shape-prior-free methods have two main approaches: dense correspondence-based and sparse keypoint-based methods. Dense correspondence-based methods attempt to map the observed point cloud to a normalized object coordinate space, but their generalization ability is limited when dealing with objects with drastically different shapes. In recent years, sparse keypoint-based methods (such as AG-Pose) have shown better robustness by learning keypoints with semantic consistency.

[0003] However, existing keypoint-based methods still have significant limitations in practical applications:

[0004] Lack of global topology awareness: Existing methods mostly rely on networks such as PointNet++ to extract local geometric features. These methods are sensitive to local appearance changes, occlusion, and sampling density changes, making it difficult to effectively characterize the global topological structure information contained in point cloud data, such as the connectivity and hole structure of point clouds.

[0005] Limited ability to model global features of key points: In the key point feature aggregation stage, existing technologies usually use simple statistical methods such as global average pooling to summarize features. Such methods are difficult to effectively model the complex spatial relationships and long-range dependencies between key points, and cannot fully reflect the relative layout relationship of key points in the overall spatial structure, thus limiting the model's ability to express the overall geometric structure of the target.

[0006] Insufficient utilization of semantic prior information: Existing methods make low use of semantic priors such as category information during the modeling process, and fail to effectively introduce known category semantics into the feature modeling and pose reasoning process, making it difficult for the model to fully leverage the role of semantic information in narrowing the solution space and enhancing discriminative ability.

[0007] Limited Nonlinear Mapping Capability: With the increasing application of structures like Mamba in point cloud computing, they offer advantages in sequence modeling efficiency and numerical stability. However, due to their structural characteristics, Mamba's ability to model complex nonlinear relationships is relatively insufficient, typically requiring linear layers for feature readout. Because shape variations within the same category are significant, the mapping between point cloud features and 6D pose often exhibits a highly nonlinear relationship, making simple linear readout structures a bottleneck in model expressiveness. Summary of the Invention

[0008] The purpose of this invention is to provide a category-level 6D object pose estimation method and system based on topology awareness and semantic Mamba, so as to solve the problems mentioned in the background art.

[0009] The present invention provides the following technical solution: a category-level 6D object pose estimation method and system based on topology awareness and semantic Mamba, comprising the following operational steps:

[0010] Step S1: Obtain the RGB image and point cloud data of the object to be estimated, and perform preprocessing.

[0011] Preferably, an instance segmentation network such as Mask R-CNN is used to obtain 2D masks of objects in the scene, and the RGB image and depth map of the target region are cropped based on the mask. The RGB image is then adjusted to... Resolution. The cropped region is back-projected into a 3D point cloud based on the depth map and camera intrinsics, and the point cloud is uniformly sampled to contain N=1024 points. The point cloud is then normalized, its center is shifted to the origin, and it is scaled to within a unit sphere. During training, to enhance robustness, Gaussian noise is applied to the point cloud, and color jitter and other perturbations are applied to the RGB image.

[0012] Step S2: Construct a feature extraction network that includes a topological feature encoder, a visual feature extractor, and an instance adaptive keypoint detector to generate an initial set of keypoints.

[0013] Preferably, the topological feature encoder is used to perform continuous homology analysis on point cloud data, extract the topological features of the point cloud and generate a topological feature vector; by using a pre-trained DINOv2 network with frozen parameters to process RGB images, image features with strong semantic information are extracted, and at the same time, the PointNet++ network is used to process the normalized point cloud to extract the local geometric features of the point cloud.

[0014] Visual feature extractor: used to extract RGB image features and geometric features of point clouds; by constructing an Alpha complex on the normalized point cloud, it reflects the shape of the point cloud at different scales; calculates the persistence cohomology of the Alpha complex to obtain a k-dimensional persistence graph; calculates persistence entropy based on the persistence graph to quantify the distribution disorder of the topological feature lifecycle; calculates the Betty curve to describe the evolution of topological features with scale, normalizes the birth and death times of topological features to the [0,1] interval, and counts the number of features present on m uniform sampling points; concatenates the extracted persistence entropy and the Betty curve to form a topological feature vector.

[0015] Instance-Adaptive Keypoint Detector: This network fuses topological feature vectors, RGB image features, and point cloud geometric features to generate an initial set of keypoints. It concatenates and fuses RGB images, geometric features extracted by PointNet++, and topological feature vectors, then inputs these into an instance-adaptive keypoint detection network. This network outputs semantically consistent keypoints and their corresponding feature vectors.

[0016] Step S3: Construct a feature aggregation network containing a local-global feature aggregator to perform global optimization on the key point features and obtain the optimized key point features.

[0017] Preferably, the local-global feature aggregator includes a local feature aggregation module and a global feature modeling module based on semantic bidirectional KMamba. By inputting the initial set of key points into the feature aggregation network, the key point features are globally optimized to obtain optimized key point features.

[0018] Preferably, the steps for processing key points by the global feature modeling module based on semantic bidirectional KMamba include:

[0019] Semantic injection: For the detected K key points and their features, a learnable category token c is generated. This category token c is initialized through a multilayer perceptron and pre-positioned at the beginning of the key point feature sequence to form a semantic perception sequence.

[0020] Hilbert serialization: Sorting K key points using Hilbert space-filling curves;

[0021] Bidirectional Mamba modeling: A state-space model is used to construct a bidirectional path containing forward and reverse paths. The forward path is used to directly apply Mamba blocks to the sorted semantically aware sequence and output forward features. The reverse path is used to flip the feature channels of the input sequence and input them into the Mamba block. Then the output is flipped back to obtain the reverse features.

[0022] Feature fusion: Forward and inverse features are concatenated with the initial input features and fused using a multilayer perceptron. The fused result is then residually connected to the initial input features to generate updated keypoint features containing long-range contextual information.

[0023] Step S4: Input the optimized key point features into the pose regression head to predict the 6D pose parameters and size information of the target object.

[0024] Preferably, by using a GroupKAN network as the readout head to replace the last linear mapping layer of Mamba, the GroupKAN network divides the input feature vector into G subgroups, performs nonlinear mapping on the features within each group using a combination of learnable univariate basis functions, and concatenates the outputs of all groups to obtain the final regression features, regressing the rotation, translation, and size of the object, thus realizing the solution of 6D pose.

[0025] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0026] (1) This invention innovatively introduces a parameterless topological feature encoder, which extracts topological features such as topological entropy and Betti curves from point cloud data based on continuous homology theory. Compared with traditional feature representations that rely on local geometric information, the above-mentioned topological invariants are more robust to the rigid transformation and local deformation of objects, and can effectively characterize the abstract structural features shared by objects of the same category, such as skeleton structure and hole distribution, thereby improving the generalization ability of the model in complex geometric scenarios.

[0027] (2) The semantic bidirectional Mamba modeling mechanism proposed in this invention, combined with the Hilbert curve serialization strategy, effectively solves the adaptation problem of unordered point cloud data in sequence modeling. While maintaining linear computational complexity, it achieves efficient modeling of long-range geometric dependencies between key points. At the same time, by introducing CATToken for explicit semantic information injection, the category prior can be propagated globally, enhancing the semantic consistency of key point features and the overall structural constraint capability.

[0028] (3) Improve the accuracy of nonlinear feature representation and regression: To address the complex feature mapping problem caused by significant changes in shape within categories, this invention uses GroupKAN to replace the traditional linear readout structure in Mamba. Through a learnable activation function combination mechanism, GroupKAN significantly enhances the model's ability to model and fit high-dimensional nonlinear manifolds.

[0029] (4) Efficient multimodal feature fusion: This invention designs a unified feature fusion framework that organically combines texture semantic features extracted by DINOv2, local geometric features extracted by PointNet++, and global topological features extracted by TDA. This multimodal complementary strategy overcomes the limitations of a single modality and can still maintain robust pose estimation performance in scenarios such as occlusion, large differences in object deformation, and no observed instances. Attached Figure Description

[0030] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0031] Figure 1 This is a schematic diagram of the overall framework of the method provided in the embodiments of the present invention;

[0032] Figure 2 This is a schematic diagram of the topology feature encoder and the process of generating continuous graphs and Betty curves provided in the embodiments of the present invention;

[0033] Figure 3 This is a schematic diagram of the specific structure of KMamba provided in the embodiments of the present invention;

[0034] Figure 4 This is a comparison chart of pose estimation results on the REAL275 dataset provided in this embodiment of the invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Combination Figures 1-4 As shown, this invention provides a technical solution: a category-level 6D object pose estimation method based on topology awareness and semantic Mamba, comprising the following steps:

[0037] Step S1: Obtain the RGB image and point cloud data of the object to be estimated and perform preprocessing;

[0038] In this embodiment, an instance segmentation network such as Mask R-CNN is used to obtain 2D masks of objects in the scene, and the RGB image and depth map of the target region are cropped based on the mask. The RGB image is then adjusted to... Resolution. The cropped region is back-projected into a 3D point cloud based on the depth map and camera intrinsics, and the point cloud is uniformly sampled to contain N=1024 points. The point cloud is then normalized, its center is shifted to the origin, and it is scaled to within a unit sphere. During training, to enhance robustness, Gaussian noise is applied to the point cloud, and color jitter and other perturbations are applied to the RGB image.

[0039] Step S2: Construct a feature extraction network that includes a topological feature encoder, a visual feature extractor, and an instance adaptive keypoint detector.

[0040] In this embodiment, the feature extraction network includes a topological feature encoder, a visual feature extractor, and an instance-adaptive keypoint detector, wherein:

[0041] Visual Feature Extractor: A pre-trained DINOv2 network with frozen parameters is used to process RGB images and extract image features with strong semantic information. At the same time, a PointNet++ network is used to process the normalized point cloud and extract the local geometric features of the point cloud.

[0042] Topological Feature Encoder: To capture the stable global structure inherent in an object, this invention introduces a parameterless topological encoder. The specific operation process includes:

[0043] First, an Alpha complex is constructed on the normalized point cloud. The Alpha complex is a collection of simple complexes that can reflect the shape of the point cloud at different scales.

[0044] Secondly, the persistence cohomology of the Alpha complex is calculated to obtain the k-dimensional persistence graph. This embodiment mainly focuses on two dimensions: k=0 (connected components) and k=1 (cycles / holes).

[0045]

[0046] in, These represent topological dimensions such as connected components and rings, respectively. This represents the scale at which the j-th topological feature appears. This indicates the scale at which it disappears.

[0047] Next, based on the continuous graph Calculate the duration entropy Used to quantify the distribution disorder of topological feature lifetimes:

[0048]

[0049] in, The duration of the feature. To prevent the use of tiny constants that are meaningless in logarithms.

[0050] Finally, the Betty curve is calculated to describe the evolution of topological features with scale, and the birth and death times of topological features are normalized to the [0,1] interval, using m uniformly sampled points. The number of features present in the above statistics :

[0051]

[0052] in, These are the values ​​after normalizing the birth and death scales of topological features to [0,1]. The scale for uniform sampling within the interval [0,1] The number of elements in the set is represented by the topological entropy and the Betty curve, which are concatenated to form a topological feature vector. This vector has rotation and translation invariance and can effectively supplement the global structural information ignored by PointNet++.

[0053] Instance-adaptive keypoint detector: RGB features, geometric features extracted by PointNet++, and topological feature vectors are concatenated and fused, and then input into an instance-adaptive keypoint detection network. This network outputs K=16 semantically consistent keypoints and their corresponding feature vectors.

[0054] Step S3: Construct a feature aggregation network that includes local-global feature aggregators.

[0055] In this embodiment, the feature aggregation network includes a local feature aggregation module and a global feature modeling module based on semantic bidirectional KMamba, which is the core construction of this invention. The steps of the global feature modeling module based on semantic bidirectional KMamba in processing key points include:

[0056] Semantic injection: For the detected K key points and their features A learnable category token (CATToken) c is generated. This CATToken is initialized through a multilayer perceptron and pre-positioned at the beginning of the keypoint feature sequence to form a semantic-aware sequence. The role of CATToken is to interact with all keypoints in subsequent sequence modeling, injecting category-level global semantic priors.

[0057] Hilbert Serialization: Since keypoints in a point cloud are spatially unordered, directly processing them as a sequence would destroy the spatial neighborhood structure. This invention utilizes Hilbert space-filling curves to sort the K keypoints. Hilbert curves have good spatial proximity relationships, which can map adjacent points in three-dimensional space to adjacent elements in a one-dimensional sequence, thereby providing a higher quality sequence input for the Mamba model.

[0058] Bidirectional Mamba Modeling: To capture long-range dependencies between key points, this invention employs a state-space model, specifically constructing a bidirectional path that includes forward and reverse paths.

[0059] Forward path: directly processing the sorted semantically perceptual sequence Apply Mamba blocks to output positive features. : .

[0060] Reverse path: To overcome the information lag in unidirectional modeling and the instability caused by token-level order reversal during serialization, the feature channels of the input sequence are first reversed, denoted as... Then input a Mamba block, and finally flip the output back to output the inverted feature. : .

[0061] Feature fusion: combining positive features and reverse features The initial input is concatenated and fused through a multilayer perceptron. The fusion result is then residually connected with the initial input features to generate updated keypoint features containing long-range context information, which is the final enhanced keypoint features.

[0062] Step S4: Input the optimized key point features into the pose regression head to predict the 6D pose parameters and size information of the target object.

[0063] In this embodiment, traditional Mamba typically uses a linear layer at the end, but it often suffers from weak nonlinear modeling capabilities when dealing with complex intra-class shape changes. This invention employs a Grouped Kolmogorov-Arnold Network (GroupKAN) as the readout head, replacing the last linear mapping layer of Mamba.

[0064] For example, GroupKAN divides the input feature vector into... Subgroups. For each group Internal features Instead of using a fixed weight matrix, learnable univariate basis functions are used. and Combinations of these are used for nonlinear mapping:

[0065]

[0066] in, For feature dimension, To output the number of components, the outputs of all groups are concatenated to obtain the final regression features, which regress the object's rotation, translation, and size, thus achieving 6D pose calculation.

[0067] The present invention defines the following training strategy: training and validation are performed on the REAL275, CAMERA25, and HouseCat6D datasets.

[0068] Hardware environment: Using a single NVIDIA RTX 3090 GPU.

[0069] Optimizer: Adam optimizer is used.

[0070] Learning rate scheduling: A cyclic learning rate strategy is adopted, where the learning rate is... arrive The training cycled through the cells for a total of 60 epochs.

[0071] Batch size: Set to 40.

[0072] Number of key points: K is set to 16. Feature dimension d is set to 256.

[0073] Example 2

[0074] This embodiment provides a category-level 6D object pose estimation system based on topology awareness and semantic Mamba, used to implement the method of Embodiment 1. The system includes:

[0075] Data acquisition module: used to connect to an RGB-D camera or read offline datasets, and perform Mask R-CNN detection, image cropping, point cloud back projection, sampling (1024 points) and normalization operations.

[0076] Topological and geometric feature extraction module:

[0077] It incorporates a DINOv2 interface and a PointNet++ subnet to process images and point cloud flows, respectively.

[0078] The embedded topology computation unit is used to execute the Alpha complex construction algorithm and the continuous homology computation algorithm, and outputs the topology entropy value and the Betti curve vector.

[0079] An embedded keypoint detection subnet is used to fuse multimodal features and regress the initial keypoint coordinates.

[0080] Semantic Mamba global modeling module:

[0081] It includes a serialization unit, stores a pre-computed Hilbert curve mapping table, and converts 3D keypoint coordinates into 1D indexes.

[0082] It contains a SemanticMamba network, which consists of multiple stacked MAN blocks. Each block deploys a forward SSM and a reverse channel flip SSM for global context updates of keypoint features.

[0083] By introducing GroupKAN into the Mamba structure, the nonlinear expressive power of the model is improved, thereby enhancing the quality of keypoint features.

[0084] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0085] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A category-level 6D object pose estimation method based on topology awareness and semantic Mamba, characterized by: The following steps are included: Step S1: Obtain the RGB image and point cloud data of the object to be estimated, and perform preprocessing; Step S2: Construct a feature extraction network that includes a topological feature encoder, a visual feature extractor, and an instance adaptive keypoint detector to generate an initial set of keypoints; Step S3: Construct a feature aggregation network containing a local-global feature aggregator to globally optimize the keypoint features and obtain the optimized keypoint features; Step S4: Input the optimized key point features into the pose regression head to predict the 6D pose parameters and size information of the target object.

2. The category-level 6D object pose estimation method based on topology awareness and semantic Mamba as described in claim 1, characterized in that: Step S1 includes: Use the Mask R-CNN instance segmentation network to obtain 2D masks of objects in the scene; Extract the RGB image and depth map of the target area based on the mask; The clipped region is back-projected into a 3D point cloud based on the depth map and camera intrinsic parameters, and the point cloud is uniformly sampled. The point cloud is normalized by shifting its center to the origin and scaling it to a unit sphere.

3. The category-level 6D object pose estimation method based on topology awareness and semantic Mamba as described in claim 2, characterized in that: In step S2: Topological feature encoder: used to perform continuous homology analysis on point cloud data, extract the topological features of the point cloud and generate topological feature vectors; Visual feature extractor: used to extract features from RGB images and geometric features from point clouds; Instance-adaptive keypoint detector: used to fuse topological feature vectors, RGB image features and point cloud geometric features to generate an initial set of keypoints.

4. The category-level 6D object pose estimation method based on topology awareness and semantic Mamba as described in claim 3, characterized in that: The operation of the visual feature extractor includes: processing RGB images using a pre-trained DINOv2 network with frozen parameters to extract image features with strong semantic information, and processing normalized point clouds using a PointNet++ network to extract local geometric features of the point clouds.

5. The category-level 6D object pose estimation method based on topology awareness and semantic Mamba as described in claim 4, characterized in that: The operation of the topology feature encoder includes: An Alpha complex is constructed on a normalized point cloud to reflect the shape of the point cloud at different scales; Calculate the persistence cohomology of the Alpha complex to obtain the k-dimensional persistence graph; The persistence entropy is calculated based on the persistence graph to quantify the distribution disorder of the life cycle of topological features; Calculate the Betty curve to describe the evolution of topological features with scale, normalize the birth and death times of topological features to the [0,1] interval, and count the number of features present on m uniform sampling points; The extracted persistent entropy and Betty curve are concatenated to form a topological feature vector.

6. The category-level 6D object pose estimation method based on topology awareness and semantic Mamba as described in claim 5, characterized in that: The instance-adaptive keypoint detector operates by concatenating and fusing RGB images, geometric features extracted by PointNet++, and topological feature vectors, and inputting them into an instance-adaptive keypoint detection network. This network outputs keypoints with semantic consistency and their corresponding feature vectors.

7. The category-level 6D object pose estimation method based on topology awareness and semantic Mamba as described in claim 6, characterized in that: The local-global feature aggregator in step S3 includes a local feature aggregation module and a global feature modeling module based on semantic bidirectional KMamba. By inputting the initial set of key points into the feature aggregation network, the key point features are globally optimized to obtain the optimized key point features.

8. The category-level 6D object pose estimation method based on topology awareness and semantic Mamba as described in claim 7, characterized in that: The steps for processing key points using the global feature modeling module based on semantic bidirectional KMamba include: Semantic injection: For the detected K key points and their features, a learnable category token c is generated. This category token c is initialized through a multilayer perceptron and pre-positioned at the beginning of the key point feature sequence to form a semantic perception sequence. Hilbert serialization: Sorting K key points using Hilbert space-filling curves; Bidirectional Mamba modeling: A state-space model is used to construct a bidirectional path containing forward and reverse paths. The forward path is used to directly apply Mamba blocks to the sorted semantically aware sequence and output forward features. The reverse path is used to flip the feature channels of the input sequence and input them into the Mamba block. Then the output is flipped back to obtain the reverse features. Feature fusion: The forward and reverse features, along with the initial input features, are concatenated and fused using a multilayer perceptron. The fusion result is then residually connected to the initial input features to generate updated keypoint features containing long-range contextual information.

9. The category-level 6D object pose estimation method based on topology awareness and semantic Mamba as described in claim 8, characterized in that: Step S4 includes: using a GroupKAN network as the readout head to replace the last linear mapping layer of Mamba. The GroupKAN network divides the input feature vector into G subgroups, performs nonlinear mapping on the features within each group using a combination of learnable univariate basis functions, and concatenates the outputs of all groups to obtain the final regression features, regressing the rotation, translation, and size of the object to achieve 6D pose calculation.

10. A category-level 6D object pose estimation system based on topology awareness and semantic Mamba, characterized in that: The method for class-level 6D object pose estimation based on topology awareness and semantic Mamba as described in any one of claims 1-9 includes: Data acquisition module: used to connect to an RGB-D camera or read offline datasets, and perform Mask R-CNN detection, image cropping, point cloud back projection, sampling and normalization operations; Topology and geometric feature extraction module: used to output topological entropy values ​​and Betty curve vectors, as well as to fuse multimodal features and regress the initial keypoint coordinates; Semantic Mamba Global Modeling Module: Used to convert 3D keypoint coordinates into 1D indices and perform global context updates on keypoint features.

Images