Class-level object pose estimation method and system based on region-level shape prior

Abstract

The disclosure provides a category-level object pose estimation method and system based on a region-level shape prior, and relates to the technical field of multi-degree-of-freedom pose estimation. A category-level object 6DoF pose estimation network based on a region-level shape prior is adaptive, extracts common features of all known objects within a category, proposes a region segmentation strategy, independently processes different sub-regions in a feature map, and can extract more critical geometric features within a local region. The structural similarity between the shape prior point cloud and the instance observation point cloud is used to dynamically adapt the category-level shape prior to each specific object instance. A dual-channel architecture is used to enable interaction and information flow between regions. The 6DoF pose estimation of the target object is realized through a deformation network and a matching network. The disclosure enhances the robustness of the network model to shape changes within the category.

Description

Technical Field

[0001] This disclosure relates to the field of multi-degree-of-freedom pose estimation technology, specifically to a category-level object pose estimation method and system based on region-level shape priors. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] In recent years, research on 6DoF (6 Degrees of Freedom) pose estimation for objects has focused on category-level 6DoF pose estimation. Unlike traditional instance-level 6DoF pose estimation, category-level tasks aim to estimate the pose of unknown objects by learning the common features of all objects within a category. The main challenge of this task lies in the significant shape differences between different object instances within the same category. This requires the network model to effectively handle these internal variations while capturing the general characteristics of the category, making it more difficult than instance-level 6DoF pose estimation.

[0004] In the current field of 6DoF pose estimation for class-specific objects, solutions to this challenge remain limited. Existing techniques have proposed shape-prior-based methods, aiming to generate class-specific shape prior point clouds for each object class and deform them to construct a standardized 3D model for a specific object instance. However, due to the static nature of class-specific shape prior point clouds, they cannot be flexibly adjusted for each specific object instance, exhibiting weak adaptability to the diverse shapes of objects within the same class, thus hindering their generalization performance when faced with objects with significant shape differences. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure proposes a category-level object pose estimation method and system based on region-level shape priors. Specifically, it proposes a category-level 6DoF object pose estimation network (PPA-Net) based on patch-wise shape prior adaptation. This network combines region segmentation strategies and shape prior point clouds to dynamically adapt category-level shape priors to specific object instances, while also considering global features. This enables interactions and information flow between regions, achieving 6DoF pose estimation of the target object.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions:

[0007] Category-based object pose estimation methods based on region-level shape priors include:

[0008] Acquire the observation point cloud, RGB image, and category-level shape prior point cloud of the target object instance;

[0009] The observed point cloud, RGB image, and category-level shape prior point cloud are input into the pose estimation network model, and the estimated 6DoF pose of the target object is output.

[0010] In the pose estimation network model, the instance geometric features of the observed point cloud, the instance semantic features of the RGB image, and the prior geometric features of the shape prior point cloud are first extracted. Then, the instance geometric features, instance semantic features, and prior geometric features are input into the dual-channel region segmentation module. The two channels segment the feature maps of each feature in parallel. Each channel then learns the structural similarity between the shape prior point cloud and the instance observed point cloud according to the segmented local region through the region-level shape prior adaptive module. The instance semantic features are then passed to the prior geometric features through structural similarity to generate their respective local prior semantic features. The feature recombination module concatenates the local prior semantic features of the two channels into their respective global feature maps. Finally, the global feature maps of the two channels are fused to obtain the final global prior semantic features. Finally, the global prior semantic features are input into the deformation network and the matching network to estimate the 6DoF pose of the target object.

[0011] According to some embodiments, the present disclosure adopts the following technical solutions:

[0012] Category-based object pose estimation systems based on region-level shape priors include:

[0013] The data acquisition module is used to acquire the observation point cloud, RGB image, and category-level shape prior point cloud of the target object instance;

[0014] The pose estimation module is used to input the observed point cloud, RGB image and category-level shape prior point cloud into the pose estimation network model and output the estimated 6DoF pose of the target object.

[0015] In the pose estimation network model, the instance geometric features of the observed point cloud, the instance semantic features of the RGB image, and the prior geometric features of the shape prior point cloud are first extracted. Then, the instance geometric features, instance semantic features, and prior geometric features are input into the dual-channel region segmentation module. The two channels segment the feature maps of each feature in parallel. Each channel then learns the structural similarity between the shape prior point cloud and the instance observed point cloud according to the segmented local region through the region-level shape prior adaptive module. The instance semantic features are then passed to the prior geometric features through structural similarity to generate their respective local prior semantic features. The feature recombination module concatenates the local prior semantic features of the two channels into their respective global feature maps. Finally, the global feature maps of the two channels are fused to obtain the final global prior semantic features. Finally, the global prior semantic features are input into the deformation network and the matching network to estimate the 6DoF pose of the target object.

[0016] According to some embodiments, the present disclosure adopts the following technical solutions:

[0017] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned category-level object pose estimation method based on region-level shape prior.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the aforementioned category-level object pose estimation method based on region-level shape prior.

[0020] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0021] This disclosure presents a category-level object pose estimation method based on region-level shape priors. It proposes a pose estimation network model, namely, a category-level 6DoF object pose estimation network based on patch-wise shape prior adaptation (PPA-Net). First, a category-level shape prior is generated for each object class to extract common features of all known objects within the class. Second, given the differences in overall shape among objects of the same class, while their local morphology exhibits high similarity, a region segmentation strategy is proposed. By independently processing different sub-regions in the feature map, the network can extract more critical geometric features within local regions. Then, utilizing the structural similarity between the shape prior point cloud and the instance observation point cloud, the category-level shape prior is dynamically adapted to each specific object instance. Furthermore, to balance the focus on global features, a dual-channel architecture is proposed to enable interaction and information flow between regions. Finally, a deformation network and a matching network are used to achieve 6DoF pose estimation of the target object.

[0022] This paper discloses a category-level object pose estimation method based on region-level shape priors. Given that objects of the same category differ in overall shape but exhibit high similarity in their local morphology, this paper proposes a region segmentation strategy to enable PPANet to extract effective information within local regions, thereby enhancing the robustness of the network model to changes in the shape of objects within the same category.

[0023] The class-level object pose estimation method based on region-level shape prior disclosed in this paper reduces computational complexity by dividing the feature map into multiple sub-regions, each of which can be processed independently in parallel with less computational resources, so as to be better applied to real-time scenarios.

[0024] The class-level object pose estimation method disclosed herein utilizes the geometric structural similarity between the class-level prior point cloud and the instance observation point cloud to inject object instance features into the prior point cloud, thereby achieving adaptive adjustment of prior information.

[0025] This disclosure presents a category-level object pose estimation method based on region-level shape priors. It proposes a dual-channel architecture to enable interaction and information flow between different regions, thus considering both global features and practical applications. Experimental results show that the proposed PPA-Net network model outperforms other state-of-the-art methods in pose estimation accuracy on benchmark datasets, demonstrating robustness to intra-class object shape changes. Attached Figure Description

[0026] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0027] Figure 1 This is a diagram showing the overall framework of the PPA-Net network model according to an embodiment of this disclosure.

[0028] Figure 2 This is a structural diagram of the region segmentation module of the dual-channel architecture according to an embodiment of the present disclosure;

[0029] Figure 3 This is a diagram illustrating the region segmentation strategy for the second channel in an embodiment of this disclosure.

[0030] Figure 4 This is a diagram showing the regional-level shape prior adaptive module structure of the dual-channel architecture according to an embodiment of this disclosure.

[0031] Figure 5 This is a feature map processing diagram of the first channel by the feature reconstruction module in an embodiment of this disclosure;

[0032] Figure 6 This is a feature map processing diagram of the second channel by the feature reconstruction module in an embodiment of this disclosure;

[0033] Figure 7 Visualization results of pose estimation of objects in the CAMERA25 dataset in this embodiment of the present disclosure;

[0034] Figure 8 The image shows the pose estimation results of objects in the REAL275 dataset in this embodiment of the present disclosure. Detailed Implementation

[0035] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0037] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0038] Example 1

[0039] One embodiment of this disclosure provides a category-level object pose estimation method based on region-level shape prior. Based on object category-level shape prior knowledge, a novel deep learning framework is proposed for category-level 6DoF object pose estimation. This framework uses the RGB image of the target object as the basis for the estimation. e Object instance observation point cloud C e and category-level shape prior point cloud C s As input, the goal is to estimate the 6DoF pose of the target object. Compared to instance-level tasks, category-level tasks require pose estimation for a class of objects with unknown CAD models, where these object instances have different shapes and sizes. Therefore, category-level object 6DoF pose estimation requires not only estimating the 3D rotation R∈SO(3) and 3D translation, but also... The size of the target object also needs to be estimated.

[0040] First, this disclosure proposes a novel deep learning-based pose estimation network model—the Category-level 6DoF Object Pose Estimation Network Based on Patch-wise Shape Prior Adaptation (PPA-Net). This network model includes a feature extraction module, a dual-channel region segmentation module, a region-level shape prior adaptation module, a feature reconstruction module, and a pose estimation module. The dual-channel region segmentation module includes a first-channel region segmentation module and a second-channel region segmentation module. The first-channel region segmentation module includes a first-region-level shape prior adaptation module, and the second-channel region segmentation module includes a second-region-level shape prior adaptation module.

[0041] Furthermore, obtain the observation point cloud C of the target object instance. e RGB image I e and category-level shape prior point cloud C s PPANet first utilizes a feature extraction module to extract features from the observed point cloud C of object instances. e RGB image of an object instance e and category-level shape prior point cloud C s Extract instance geometric features respectively Instance semantic features and prior geometric features Then, these three features are input into a dual-channel region segmentation module, where the two channels employ different region segmentation strategies to emphasize local features of different regions of the object. Next, within each channel, a region-level shape prior adaptation module is used to dynamically adapt the category-level shape prior information to specific object instances. Subsequently, the features from the two channels are recombined through a feature reorganization module to obtain global prior semantic features. Finally, the 6DoF pose of the target object is estimated using a deformable network and a matching network.

[0042] As one embodiment, the specific implementation process of the category-level object pose estimation method based on region-level shape prior disclosed herein is as follows:

[0043] Step 1: Obtain the observation point cloud C of the target object instance. e RGB image I e and category-level shape prior point cloud C s The observation point cloud C e RGB image I e and category-level shape prior point cloud C s The input is fed into the pose estimation network model, and the feature extraction module extracts the instance geometric features of the observed point cloud. RGB image Instance semantic features and shape prior geometric features of point clouds

[0044] As one example, the feature extraction module uses an existing deep learning feature extraction network to extract three types of feature maps.

[0045] Step 2: Given that different objects within the same category differ in overall shape but exhibit high similarity in their local morphology, this disclosure proposes a region segmentation strategy. This strategy aims to refine the processing of each sub-region in the feature map, improving not only the accuracy of information extraction but also enhancing the network model's ability to perceive details, providing strong support for a deeper understanding of the object's local morphology. Furthermore, to avoid being limited to local features of fixed regions, this disclosure employs a dual-channel architecture, using two different region segmentation methods to enable information interaction between sub-regions. In addition, PPA-Net allows for parallel processing of each sub-region with low computational resource consumption, significantly optimizing the computational complexity of the network model and effectively improving processing efficiency, thus enabling better application in real-time scenarios.

[0046] Specifically, the first step is to extract the instance geometric features of the observed point cloud. RGB image Instance semantic features and shape prior geometric features of point clouds The data is input into the first and second channel region segmentation modules of the dual-channel region segmentation module, respectively. The two channels process the instance geometric features in parallel. Instance semantic features and prior geometric features Where M e =M s =1024, d=64.

[0047] First, such as Figure 2 As shown, the spatial structure of each input feature is reshaped, and the 1024 feature points are rearranged into a 32×32 planar structure. Then, in the first channel region segmentation module, a regular window-style segmentation strategy is adopted to uniformly segment the three input feature maps into four 16×16 local region feature maps in a non-overlapping manner.

[0048] In the second channel region segmentation module, to enhance the interaction and information flow between regions, a moving window-based region segmentation method is adopted, such as... Figure 3 As shown, firstly, the input feature map is segmented into regions. The segmented sub-regions are then partially moved along the vertical and horizontal directions to construct a new feature map layout. Specifically, the top region is moved to the bottom, and the newly obtained left region is moved to the right. Finally, the sub-regions are stitched together to form four 16×16 local region feature maps.

[0049] Step 3: After segmentation by the first channel region segmentation module and the second channel region segmentation module, the two channels respectively obtain the geometric features Z of the object instance. e Instance semantic features F e and prior geometric features Z s The local region feature maps, each with dimensions of 16×16×64, are then flattened into 256×64-dimensional region-level feature vectors, from which the region-level instance geometric features Z are obtained. ei (i = 1, 2, 3, 4), regional instance semantic features F ei (i = 1, 2, 3, 4) and the region-level prior geometric features Z si (i = 1, 2, 3, 4) Considering the overall differences and local similarities in the shapes of similar objects, this disclosure performs shape prior adaptive processing on the local region-level feature maps respectively, aiming to dynamically apply category-level shape prior information to specific object instances, which helps to capture the local structural similarity of objects more accurately.

[0050] like Figure 4 As shown, the regional instance geometric features Z output from the first and second channels are...ei (i = 1, 2, 3, 4), regional instance semantic features F ei (i = 1, 2, 3, 4) and the region-level prior geometric features Z si (i = 1, 2, 3, 4) are input into the shape prior adaptive module of their respective channels, firstly, the regional instance geometric features Z are... ei (i = 1, 2, 3, 4) and the region-level prior geometric features Z si (i = 1, 2, 3, 4) are associated to simulate the category-level shape prior point cloud C. s With instance observation point cloud C e The regional structural similarity between them is represented as The magnitude of element xkl reflects point c. k ∈C s With point c l ∈C e The structural similarity between them. Intuitively speaking, x kl The larger the value, the more it should be from c. l To c k The more semantic features F from the region-level instance are transmitted ei The semantic features of object instances. In summary, guided by structural similarity, the semantic features F of object instances are... e Observing point cloud C from instances using structural similarity X e Propagation to shape prior point cloud C s This allows for adaptive adjustment of the prior point cloud.

[0051] Specifically, this disclosure uses the Transformer architecture to implement the above scheme. The Transformer architecture is able to efficiently handle complex data relationships mainly due to its built-in multi-head attention mechanism. This mechanism can not only achieve efficient mapping between complex data, but also capture and express deep-level relationships between data, especially when processing unordered point cloud data. Therefore, this disclosure utilizes this advantage to learn the structural similarity between two unordered point clouds, and performs adaptive adjustment of the shape prior point cloud based on this similarity for category-level object 6DoF pose estimation.

[0052] Furthermore, the regional-level prior geometric features Z si The region-level instance geometric feature Z of the object instance ei and region-level instance semantic features F ei These are respectively used as the query, key, and value for the multi-head attention module for computation:

[0053]

[0054] in, and All are learning projection matrices, corresponding to the query, key, and value in the Transformer architecture, respectively. d is the dimension of the feature vector, σ(·) represents the standard softmax normalization function, and h = 1, 2, ..., H represents the number of heads in the multi-head attention module.

[0055] In each head of the Transformer network, by associating Q... (h) and K (h) The instance observation point cloud C can be calculated. e With shape prior point cloud C s The structural similarity X within the projected embedding space is then calculated. Then X is compared with V. (h) Multiplication yields region-level prior semantic features. This disclosure uses H attention blocks to comprehensively and accurately model C. s With C e The structural similarity between the points is determined, and the semantic information of the target object is fully transferred to the shape prior point cloud. Then, the output features of the H attention blocks are concatenated:

[0056] Y = Concat(Y) (1) ,Y (2) ,...,Y (H) )

[0057] The concatenated result Y is input into the feedforward network to obtain the adjusted regional-level prior semantic features F. si :

[0058] F si =FFN(Y)

[0059] Where i = 1, 2, 3, 4, corresponding to the four sub-regions in the feature map, and FFN(·) represents the feedforward network.

[0060] In the Transformer-based implementation, a multi-head attention mechanism is used to extract stable similarities from two geometric features, and the representation of prior features is further optimized by introducing semantic features. This adaptive adjustment not only enhances prior knowledge with rich semantic information, but also enables the originally static category-level shape prior to flexibly adapt to changes in various object instances.

[0061] As one embodiment, the shape prior adaptation process performed by the first region-level shape prior adaptation module and the second region-level shape prior adaptation module in the two channels is the same, so it will not be described separately.

[0062] Step 4: Combine the region-level prior semantic features F obtained by the first region-level shape prior adaptation module and the second region-level shape prior adaptation module in the two channels. si The input is fed into the feature reorganization module, which then performs reshaping, splicing, and flattening operations on the feature maps of the two channels to generate global prior semantic features. During this process, the feature map in the second channel is moved in reverse order as described in step 2 to ensure that each sub-region is restored to its initial position in the feature map and aligned with the feature map in the first channel, facilitating feature fusion. After the spatial movement and adjustment in the second channel, each sub-region contains feature information from other sub-regions. Finally, the semantic features output from the two channels are fused, thereby expanding the network model's ability to understand global features.

[0063] Specifically, such as Figure 5 As shown, for the regional prior semantic features output in the first channel, the 256×64-dimensional regional prior semantic features are first reshaped into a 16×16×64-dimensional feature map, then the four local feature maps are stitched together into a 32×32×64-dimensional global feature map, and finally flattened into a 1024×64-dimensional feature vector, which is the global prior semantic feature of the first channel.

[0064] For the region-level prior semantic features output from the shape prior adaptive module in the second channel, first reshaping, splicing, and flattening operations are performed to generate a 32×32×64-dimensional global feature map. Then, spatial positioning is moved and adjusted, and finally spliced ​​to form the global prior semantic features of the second channel. For example... Figure 6 As shown, the right region is moved to the left, and the newly obtained bottom region is moved to the top. This step aims to restore the initial arrangement of each sub-region in the feature map so as to achieve spatial alignment with the feature map in channel one. Subsequently, each sub-region is stitched together into a global feature map of size 32×32×64. Finally, it is flattened into a 1024×64-dimensional feature vector, which is the global prior semantic feature of the second channel.

[0065] Finally, the global prior semantic features from the first channel and the global prior semantic features from the second channel are fused. This process only fuses information without changing the dimensions, generating the final global prior semantic feature F. s .

[0066] Step 5: Finally, through the deformable network U a and matching network U b The 6DoF pose of the target object is estimated. First, the final global prior semantic features F are... s and prior geometric features Z sThe features are concatenated to obtain the adaptively adjusted complete prior features U. s Semantic features F of spliced ​​instances e and instance geometric features Z e To form a complete object instance feature U e :

[0067]

[0068] Then, through the deformable network U a Associating category-level shape priors with target object instances generates an adjusted shape prior point cloud.

[0069] Specifically, the deformable network U a Based on the structural and semantic relationships between shape priors and object instances, the deformation field A from the shape prior point cloud to the instance observation point cloud is calculated. Then, by adjusting the position of each point in the shape prior point cloud, the 3D model of the target object is reconstructed.

[0070] C s ′=C s +A=C s +U a (U e U s )

[0071] Among them, C s ′ represents the deformed shape prior point cloud, i.e., the 3D reconstructed point cloud of the target object, C s Let A be the a priori point cloud of the shape before deformation, and U be the predicted deformation field. a (·) denotes a deformable network.

[0072] Deformation networks primarily utilize point-to-point deformation fields to transform prior shape point clouds into canonical models of target object instances. This process can be viewed as a dynamic adjustment process, adaptively adjusting the prior point cloud based on the difference between the actual observed object shape and prior knowledge to accurately map the actual shape of the target object.

[0073] Next, match network U b In the 3D reconstructed point cloud C of the target object s ′ and the actual observed object instance point cloud C e A precise point-to-point correspondence is established between them, thereby effectively aligning the two point clouds and calculating C. s ′ to C e The corresponding matrix B. Matrix B can reconstruct model C. s The expression is transformed into its representation in the Normalized Object Coordinate Space (NOCS), which is an existing shared canonical representation for all objects in the same category. The matching process is represented as:

[0074] C e =B×C s ′=U b (U e U s )×C s ′

[0075] Among them, U b (·) represents a matching network. The representation of 3D reconstructed point clouds in NOCS, For the corresponding matrix, M e M is the number of points in the observation point cloud of the target object. s It is the number of points in the shape prior point cloud.

[0076] The Bth digit in matrix B i The row represents the observation of the i-th point in the point cloud and C. s The correspondence of all points in ′, B×C s Essentially, it's a weighted summation process; based on the weights in the corresponding matrix B, the reconstructed point cloud C is calculated. s The coordinates of the corresponding points in the matrix are weighted and averaged to calculate their positions in NOCS.

[0077] Finally, by combining the observation point cloud C of the target object... e With C e The algorithm calculates the 6DoF pose and dimensions of the target object using the correspondence-based Umeyama algorithm. It aligns and compares the similarity between two sets of point clouds, aiming to determine the optimal translation, rotation, and scaling parameters by minimizing the root-square deviation between point pairs.

[0078] The core advantage of this shape-prior-based pose estimation method lies in its ability to fully utilize prior knowledge at the category level and accurately capture the spatial layout of the target object through a structure-guided dynamic adaptive process. This not only improves the accuracy of 6DoF pose estimation but also significantly enhances the algorithm model's adaptability to the diversity of shapes within the object class.

[0079] As one embodiment, the core learning objective of the method disclosed herein is to construct an accurate deformation field and corresponding matrix. This disclosure uses the following loss function to train PPA-Net:

[0080] L=λ1L re +λ2L ar +λ3L b +λ4L en

[0081] Among them, L re To rebuild the losses, L arFor deformation field regularization loss, these two factors jointly supervise the deformation field A; L b For the corresponding matrix loss, L en For the entropy regularization loss of the corresponding matrix, these two items jointly supervise the corresponding matrix B, and the coefficients λ1, λ2, λ3, and λ4 are set to 5.0, 0.01, 1.0, and 0.0001, respectively.

[0082] 1) Reconstruction loss L re

[0083] Reconstruction loss L re By minimizing the 3D reconstruction model C s ′ and the real model C of the target object gt The chamfer distance between the two models is increased to make them as close as possible in spatial layout, thereby enabling indirect supervision of the deformation field A.

[0084]

[0085] Where, d CD (·) represents the chamfer distance, and x and y represent the reconstructed model C, respectively. s ′ and the real model C gt The point in the middle, Indicate C s For each point x in ', we need to find C. gt Find the point y that is closest to it, and calculate the square of the Euclidean distance between these two points. Similarly.

[0086] 2) Deformation field regularization loss L ar

[0087] Deformation field regularization loss L ar The aim is to suppress excessive and unreasonable deformations in the deformation field A. For example, it ensures that points representing the "cup handle" in the shape prior point cloud remain at the handle position in the reconstructed model. This regularization method helps maintain the realism of the reconstructed geometry while allowing for a degree of shape adjustment to suit specific detail settings. Its loss function is expressed as:

[0088]

[0089] Among them, M s a represents the number of points in the shape prior point cloud. i Let ||·||2 represent the vector in the deformed field A, and let ||·||2 represent the L2 norm.

[0090] 3) Corresponding matrix loss L b

[0091] Corresponding matrix loss Lb The aim is to optimize the consistency between the NOCS coordinates predicted by the network and the true NOCS coordinates, thereby indirectly supervising the corresponding matrix B. Specifically, this paper uses a modified smoothed L1 loss function:

[0092]

[0093] Among them, C′ e The NOCS representation of network prediction, C NOCS M represents the true NOCS representation. e This represents the number of points in the observed point cloud, where i = 1, 2, 3 represent the three dimensions in the NOCS space, and x i and y i They represent C′ respectively e and C NOCS The coordinates of any point in a certain dimension.

[0094] 4) Entropy regularization loss L corresponding to the matrix en

[0095] The entropy regularization loss L corresponding to the matrix en The aim is to optimize the distribution of the corresponding matrix B and clarify the relationship between each observation point and the deformed prior point cloud C. s The correspondence between points is defined as follows: Specifically, this loss term aims to ensure that each observation point corresponds to a few points in the prior point cloud, avoiding scattered correspondences of multiple points, and the Bth point in matrix B... i The row represents the observation of the i-th point in the point cloud and C. s The correspondences of all points in ′ are defined. Therefore, minimizing the entropy regularization loss L of the correspondence matrix is ​​the key. en This allows each row in the corresponding matrix B to approximate a "peak distribution," which is achieved by minimizing the average cross-entropy:

[0096]

[0097] Among them, M e B is the number of points in the observation point cloud of the target object. ij Let be the element in the i-th row and j-th column of matrix B, representing the relationship between the i-th observation point and the deformed prior point cloud C. s The weight of the correspondence between the j-th points in the array.

[0098] Note: Due to B ij logB ij The term is negative (0 < B) ij ≤1, logB ij <0), to ensure L enFor positive values, a negative sign must be added before the summation term. This results in a larger entropy (and a more uncertain correspondence) leading to a larger loss function value, which corresponds to L... en The purpose is to reduce the uncertainty of the correspondence.

[0099] Experiments and Results

[0100] 1. Experimental setup

[0101] This experiment utilizes the PPA-Net network built with the PyTorch framework and employs the Adam optimizer to ensure efficient and stable training. All experiments were performed on a computer equipped with four NVIDIA RTX 2080 graphics cards. A batch size of 24 was used to balance the amount of data processed per iteration with the utilization of GPU resources. The initial learning rate was set to 1e. -4 When training reaches the 5th round, the learning rate is adjusted to 0.6 times the initial value; when training enters the 10th round, the learning rate is further reduced to 0.3 times the initial value to ensure that the network can converge stably in the later stages of training.

[0102] For generating shape prior point clouds, this experiment utilizes an autoencoder network trained on the ShapeNet dataset. By inputting the average embedding values ​​of object instances of the same category into the trained decoder, shape prior point clouds reflecting the average characteristics of the category are generated. In terms of point cloud data processing, both the observed point clouds of object instances and the category-level shape prior point clouds are uniformly sampled to 1024 points to ensure data consistency and reduce computational complexity. For the input RGB image, this experiment crops it to a size of 192×192 and randomly extracts 1024 points from it through downsampling to meet the network's input requirements.

[0103] 2. Dataset Selection

[0104] To comprehensively evaluate the performance of the PPA-Net model, this paper selects two different datasets for testing: the computer-generated virtual dataset CAMERA25 and the real-world dataset REAL275. The CAMERA25 dataset contains 300,000 synthetic RGB-D images with virtual objects superimposed on real backgrounds. This design aims to simulate complex real-world scenes, providing a diverse learning and testing environment for the network model. This experiment selects 275,000 images from this dataset as the training set and 25,000 images as the test set to comprehensively examine the performance of PPA-Net in virtual scenes.

[0105] The REAL275 dataset fully demonstrates the complexity and diversity of real-world environments, containing 8000 RGB-D images from 18 different scenes. In terms of dataset partitioning, this experiment uses images from 7 scenes (4300 images in total) as the training set to train the PPA-Net model to learn key object features. The validation set consists of 950 images from 5 scenes, used for parameter tuning and optimization during model training. The remaining 6 scenes (containing 2750 images) constitute the test set to comprehensively examine the network model's generalization ability and performance in unknown real-world environments. This experiment adopts the same training strategy as the shape-prior-based benchmark method SPD, training with CAMERA25 synthetic data and REAL275 real data in a 3:1 ratio, while only using the real REAL275 dataset for testing.

[0106] Furthermore, the CAMERA25 and REAL275 datasets together cover six common object categories encountered in daily life, including bottles, bowls, cameras, cans, laptops, and mugs. This breadth of object categories not only helps the model learn more diverse feature information but also allows for a more comprehensive evaluation of PPA-Net's performance across different object categories. Through in-depth analysis and evaluation of these two benchmark sets, this experiment provides a clear understanding of the PPA-Net model's performance in both virtual and real-world environments.

[0107] 3. Evaluation Indicators

[0108] This experiment aims to comprehensively and accurately evaluate the performance of the PPA-Net model. It follows the common evaluation criteria in the field of 6DoF pose estimation for category-level objects and uses the 3D Intersection over Union (3D IoU) evaluation metric and the n°, n cm evaluation metric to quantify the accuracy of the network model.

[0109] 1) 3D IoU Evaluation Metrics

[0110] The 3D IoU evaluation metric measures the degree of overlap between the estimated pose and the 3D bounding box defined by the true pose. Let the predicted 3D bounding box be represented as... If the true 3D bounding box is represented as b, then the 3D IoU is calculated as follows:

[0111]

[0112] in, The volume of the intersection of the predicted 3D bounding box and the ground truth 3D bounding box. The volume is the union of the predicted 3D bounding box and the true 3D bounding box.

[0113] Specifically, the IoU value ranges from [0,1], where 1 represents a perfect match and 0 represents no overlapping region. The prediction is considered accurate when the overlap ratio between the predicted bounding box and the ground truth bounding box exceeds a preset threshold. This experiment employs strict IoU... 50 and IoU 75 The evaluation metrics are 0.5 and 0.75, respectively, representing IoU values ​​between the predicted bounding box and the ground truth bounding box.

[0114] 2) Evaluation indicators for n° and n cm

[0115] The n°, n cm evaluation metrics are used to directly quantify the accuracy of the network model in predicting the rotation and translation of a target object. Let the predicted rotation matrix be... The translation matrix is The actual rotation matrix is ​​R, the translation matrix is ​​T, and the rotation error is calculated as follows:

[0116]

[0117] in, express The transpose of the matrix is ​​given by , trace(·) represents the trace of the matrix, which is the sum of the diagonal elements, and Δθ represents the angle of rotation error.

[0118] Translation error is calculated by measuring the Euclidean distance between the predicted translation matrix and the actual translation matrix:

[0119]

[0120] in, Let T be the predicted translation matrix, T be the true translation matrix, and Δd be the distance of the translation error.

[0121] This experiment evaluates the accuracy of the prediction by setting thresholds for rotation and translation errors. The pose prediction is considered accurate only when the rotation error is within a specific angle range and the translation error is simultaneously within a specific distance range. By setting different thresholds, the experiment allows for flexible evaluation of the network model's performance. This experiment used four threshold combinations: 5°, 2cm, 5°, 5cm, 10°, 2cm, and 10°, 5cm, to comprehensively evaluate the accuracy of the PPA-Net model in the 6DoF pose estimation task for category-level objects.

[0122] Based on the 3D IoU and n°, n cm evaluation metrics, this experiment summarizes the mean average precision (mAP) for all six object categories in the CAMERA25 and REAL275 datasets to provide comprehensive evaluation results. By comparing the mAP values ​​of different methods, the overall performance of various 6DoF pose estimation methods across multiple object categories can be intuitively understood.

[0123] 4. Comparative experiments with state-of-the-art methods on different datasets.

[0124] 1) Results on the CAMERA25 dataset

[0125] Table 1 presents a detailed comparison between the proposed PPA-Net and state-of-the-art 6DoF pose estimation methods for objects on the CAMERA25 dataset. "-" indicates missing data, while CenterSnap-R indicates that the method incorporates iterative pose refinement techniques. Experimental results show that the proposed PPA-Net achieves state-of-the-art accuracy in four of the six evaluation metrics. Specifically, for the n°, n cm evaluation metric, the mAP at the four thresholds of 5°, 2cm, 5°, 5cm, 10°, 2cm, and 10°, 5cm reaches 72.3%, 75.8%, 83.6%, and 88.5%, respectively. Compared to the shape-prior-based baseline method SPD, these represent improvements of 18.0%, 16.8%, 10.3%, and 7.0%, respectively, and improvements of 3.2%, 2.1%, 3.8%, and 5.0%, respectively, compared to the latest method GSNet proposed in 2023. Regarding the 3D IoU evaluation metric, in terms of IoU... 50 and IoU 75 The mAP at the two thresholds reached 92.9% and 88.0%, respectively, representing improvements of 6.1% and 9.0% compared to SAR-Net proposed in 2022. Compared to the SGPA method, PPA-Net achieved better IoU. 75 The mAP at the threshold differs from it by only 0.1%. These experimental results fully validate the high accuracy performance of the proposed PPA-Net on the 6DoF pose estimation task of class-level objects, demonstrating robustness to the diversity of in-class object morphology.

[0126] Table 1. Experimental results on the CAMERA25 dataset.

[0127]

[0128]

[0129] Figure 7This paper presents a detailed visualization of pose estimation results on the CAMERA25 dataset. The green bounding boxes represent the true pose of the target object, while the red bounding boxes represent the pose estimation results of the proposed PPA-Net. To comprehensively evaluate the performance of the algorithm model, this paper selects pose estimation results from four different scenes in the CAMERA dataset, showcasing three representative images from each scene. As can be seen from the images, the red and green bounding boxes are closely aligned, exhibiting a very high degree of overlap. This phenomenon fully demonstrates that the pose estimation results of the PPA-Net network model highly match the real-world situation, demonstrating excellent performance. This not only reflects the accuracy of PPA-Net in 6DoF object pose estimation but also further proves its robustness in handling different scenes and objects.

[0130] 2) Results on the REAL275 dataset

[0131] Table 2 presents a comprehensive comparative analysis of the proposed PPA-Net on the REAL275 dataset with state-of-the-art 6DoF object pose estimation methods. "-" in the table indicates missing data. Experimental results show that the proposed method achieves optimal accuracy in four out of six evaluation metrics. Specifically, in the 3D IoU evaluation metric, when the threshold is set to IoU... 50 and IoU 75 At the specified thresholds, the mAP reached 80.0% and 62.1%, respectively, representing improvements of 2.7% and 8.9% compared to the shape-prior-based benchmark method SPD, and improvements of 0.7% and 6.2% compared to the recently proposed CR-Net. Under the n°, n cm evaluation metric, the mAP at the four thresholds of 5°, 2cm, 5°, 5cm, 10°, 2cm, and 10°, 5cm were 28.7%, 32.7%, 55.5%, and 66.1%, respectively, representing improvements of 9.4%, 11.3%, 12.3%, and 12.0% compared to the shape-prior-based benchmark method SPD. At the 5°, 5cm and 10°, 5cm thresholds, the mAP improved by 3.6% and 1.8% compared to the CenterSnap method with pose iteration refiner proposed in 2022. Compared to other state-of-the-art methods listed in Table 2, the proposed PPA-Net demonstrates a significant accuracy advantage on the real-world REAL275 dataset and does not rely on a time-consuming pose optimizer, validating its real-time pose estimation performance in real-world scenarios.

[0132] Table 2 shows the experimental results on the REAL275 dataset.

[0133]

[0134] The visualization results of object pose estimation in the REAL275 dataset are as follows: Figure 8 As shown, the green bounding box represents the true pose of the target object, and the red bounding box represents the pose estimation result of PPA-Net. To comprehensively evaluate the model performance, pose estimation results from all six scenarios used for testing in the REAL275 dataset were selected, with two representative visualizations chosen for each scenario. By observing the degree of overlap between the red and green bounding boxes in the figures, it can be concluded that good overlap indicates that the pose estimated by PPA-Net is very close to the true pose, while slight deviations in overlap indicate prediction errors. Taking laptops as an example, the pose estimation accuracy of such objects in certain scenarios still needs improvement. This indicates that improving the robustness and adaptability of the network model to changes in the shape of objects within the same class remains a hot issue that urgently needs to be addressed in the field of 6DoF pose estimation for class-level objects.

[0135] The results comparison clearly demonstrates that the proposed 6DoF pose estimation algorithm for class-based objects exhibits superior performance on both synthetic and real-world data. Specifically, in synthetic data testing, the model accurately estimates object poses by precisely capturing common features of similar objects. In real-world data testing, the model successfully overcomes the complexity and variability of real-world scenes, demonstrating excellent robustness. Across various evaluation metrics, PPA-Net significantly outperforms shape-prior-based benchmark methods, showcasing its superior accuracy and stability. In practical applications, the model also demonstrates good generalization ability, providing a reliable and efficient solution for class-based 6DoF pose estimation tasks.

[0136] 5. Ablation test

[0137] To further analyze the effectiveness of the proposed PPA-Net method, ablation comparison experiments were conducted on the CAMERA25 and REAL275 datasets using the 3D IoU and n°,n cm evaluation metrics. The experimental results are shown in Tables 3 and 4. In Table A, the baseline method is represented, which does not employ a dual-channel region segmentation approach and directly performs prior adaptive processing on global features. Table B represents a region segmentation strategy using a single-channel regular window approach, building upon Table A. Table C represents a region segmentation strategy using a dual-channel approach, building upon Table B, and fusing the features from each channel after prior adaptive processing.

[0138] Table 3 Ablation experimental results on the CAMERA25 dataset.

[0139]

[0140]

[0141] Table 4 Ablation experimental results on the RAEL275 dataset.

[0142]

[0143] By comparing the accuracy values ​​of Method B and Method A in Tables 3 and 4, it can be observed that the performance of Method B has been improved across all six evaluation metrics. Specifically, the mAP of Method B on the CAMERA25 dataset using the 5°, 5cm evaluation metric increased from 69.5% to 72.0%; and the mAP of Method B on the REAL275 dataset using the 10°, 5cm evaluation metric increased from 62.9% to 64.4%, validating the effectiveness of the proposed region segmentation strategy that focuses on local similarity. By comparing the accuracy values ​​of Method C and Method B in the table, it can be observed that the performance of Method C has been improved across all evaluation metrics. Specifically, the mAP of Method C on the CAMERA25 dataset using the 5°, 2cm evaluation metric increased from 68.4% to 72.3%; and the mAP of Method B on the REAL275 dataset increased from 60.3% to 62.1%, validating the effectiveness of the dual-channel fusion architecture designed in this paper and demonstrating its adaptability to diverse shapes of objects within the same class.

[0144] Example 2

[0145] One embodiment of this disclosure provides a category-level object pose estimation system based on region-level shape prior, including:

[0146] The data acquisition module is used to acquire the observation point cloud, RGB image, and category-level shape prior point cloud of the target object instance;

[0147] The pose estimation module is used to input the observed point cloud, RGB image and category-level shape prior point cloud into the pose estimation network model and output the estimated 6DoF pose of the target object.

[0148] In the pose estimation network model, the instance geometric features of the observed point cloud, the instance semantic features of the RGB image, and the prior geometric features of the shape prior point cloud are first extracted. Then, the instance geometric features, instance semantic features, and prior geometric features are input into the dual-channel region segmentation module. The two channels segment the feature maps of each feature in parallel. Each channel then learns the structural similarity between the shape prior point cloud and the instance observed point cloud according to the segmented local region through the region-level shape prior adaptive module. The instance semantic features are then passed to the prior geometric features through structural similarity to generate their respective local prior semantic features. The feature recombination module concatenates the local prior semantic features of the two channels into their respective global feature maps. Finally, the global feature maps of the two channels are fused to obtain the final global prior semantic features. Finally, the global prior semantic features are input into the deformation network and the matching network to estimate the 6DoF pose of the target object.

[0149] Example 3: A non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they implement the category-level object pose estimation method based on region-level shape prior.

[0150] Example 4

[0151] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the aforementioned category-level object pose estimation method based on region-level shape prior.

[0152] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A category-level object pose estimation method based on region-level shape prior, characterized in that, include: Acquire the observation point cloud, RGB image, and category-level shape prior point cloud of the target object instance; The observed point cloud, RGB image, and category-level shape prior point cloud are input into the pose estimation network model, and the estimated 6DoF pose of the target object is output. In the pose estimation network model, the instance geometric features of the observed point cloud, the instance semantic features of the RGB image, and the prior geometric features of the shape prior point cloud are first extracted. Then, the instance geometric features, instance semantic features, and prior geometric features are input into the dual-channel region segmentation module. The two channels segment the feature map of each feature in parallel. Each channel learns the structural similarity between the shape prior point cloud and the instance observed point cloud according to the segmented local region through the region-level shape prior adaptive module. The instance semantic features are then passed to the prior geometric features through structural similarity to generate their respective local prior semantic features. The feature recombination module concatenates the local prior semantic features of the two channels into their respective global feature maps. Then, the global feature maps of the two channels are fused to obtain the final global prior semantic features. Finally, the global prior semantic features are input into the deformation network and the matching network to estimate the 6DoF pose of the target object. The pose estimation network model includes a feature extraction module, a dual-channel region segmentation module, a region-level shape prior adaptation module, a feature recombination module, and a pose estimation module. The dual-channel region segmentation module includes a first-channel region segmentation module and a second-channel region segmentation module. The first-channel region segmentation module includes a first-region-level shape prior adaptation module, and the second-channel region segmentation module includes a second-region-level shape prior adaptation module. The instance geometric features, instance semantic features, and prior geometric features are input into the first and second channel region segmentation modules of the dual-channel region segmentation module. In the first channel region segmentation module, a regular window segmentation method is used to uniformly segment the three input feature maps into four local region feature maps in a non-overlapping manner. In the second channel region segmentation module, a moving window segmentation method is used to move the segmented sub-regions in the vertical and horizontal directions, and then stitch the moved sub-regions together to form four local region feature maps of the same size, thus constructing a new feature map layout. The local region feature maps of the two channels after segmentation are subjected to shape prior adaptive processing. A Transformer network is used to simulate the regional structural similarity between the category-level instance observation point cloud and the shape prior point cloud. The regional-level instance geometric features, regional-level instance semantic features, and regional-level prior geometric features are used as queries, keys, and values ​​for the multi-head attention module, respectively. The regional-level instance geometric features and regional-level prior geometric features are associated to calculate the structural similarity between the instance observation point cloud and the shape prior point cloud in the projection embedding space. The structural similarity is multiplied by the regional-level instance semantic features and continuously adjusted to obtain the regional-level prior semantic features.

2. The category-level object pose estimation method based on region-level shape prior as described in claim 1, characterized in that, The shape prior adaptation process performed by the first region-level shape prior adaptation module and the second region-level shape prior adaptation module in both channels is the same.

3. The category-level object pose estimation method based on region-level shape prior as described in claim 1, characterized in that, Two channels are processed through shape prior adaptive processing to obtain regional-level prior semantic features. These regional-level prior semantic features are then input into the feature reorganization module. The feature reorganization module reshapes, splices, and flattens the regional-level prior semantic features output from the first regional-level shape prior adaptive module to generate the global prior semantic features for the first channel. The regional-level prior semantic features output from the second regional-level shape prior adaptive module are first reshaped, spliced, and flattened, then moved and adjusted in space, and then spliced ​​to form the global prior semantic features for the second channel. Finally, the global prior semantic features for the first and second channels are fused to generate the final global prior semantic features.

4. The category-level object pose estimation method based on region-level shape prior as described in claim 1, characterized in that, The final global prior semantic features are concatenated with the prior geometric features to obtain the adaptively adjusted complete prior features. The instance semantic features and instance geometric features are concatenated to obtain the complete object instance features. Then, a deformable network is used to associate the category-level prior features with the object instance features to reconstruct the 3D model of the target object. A matching network is then used to establish a point-to-point correspondence between the 3D model and the actual observed object instance point cloud. The Umeyama algorithm based on the correspondence is used to estimate the 6DoF pose and size of the target object.

5. A category-level object pose estimation system based on region-level shape prior, characterized in that, Specifically, the method for class-level object pose estimation based on region-level shape prior as described in any one of claims 1-4 includes: The data acquisition module is used to acquire the observation point cloud, RGB image, and category-level shape prior point cloud of the target object instance; The pose estimation module is used to input the observed point cloud, RGB image and category-level shape prior point cloud into the pose estimation network model and output the estimated 6DoF pose of the target object. In the pose estimation network model, the instance geometric features of the observed point cloud, the instance semantic features of the RGB image, and the prior geometric features of the shape prior point cloud are first extracted. Then, the instance geometric features, instance semantic features, and prior geometric features are input into the dual-channel region segmentation module. The two channels segment the feature maps of each feature in parallel. Each channel then learns the structural similarity between the shape prior point cloud and the instance observed point cloud according to the segmented local region through the region-level shape prior adaptive module. The instance semantic features are then passed to the prior geometric features through structural similarity to generate their respective local prior semantic features. The feature recombination module concatenates the local prior semantic features of the two channels into their respective global feature maps. Finally, the global feature maps of the two channels are fused to obtain the final global prior semantic features. Finally, the global prior semantic features are input into the deformation network and the matching network to estimate the 6DoF pose of the target object.

6. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the category-level object pose estimation method based on region-level shape prior as described in any one of claims 1-4.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the category-level object pose estimation method based on region-level shape prior as described in any one of claims 1-4.

Images