End-to-end category level object pose estimation method and system based on space-time implicit anchor point query

Abstract

The invention discloses an end-to-end category level object pose estimation method based on space-time implicit anchor point query, which comprises the following steps of: inputting an RGB (Red, Green and Blue) image sequence into a visual encoder, extracting multi-scale semantic features and fusing depth and normal vector geometric features to generate 2.5-dimensional multi-scale features; a 3D implicit anchor queue is established based on the features, and queries in the queue are processed through confidence screening and camera pose offset transformation. Performing modeling on space-time correlation by using multi-head attention, outputting time sequence enhanced query features, and performing geometric perception feature sampling; and finally, mapping the implicit 3D query into 9-degree-of-freedom attitude parameters through bipartite graph matching to realize end-to-end training.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional space perception, and in particular to an end-to-end category-level object pose estimation method and system based on spatiotemporal implicit anchor point query. Background Art

[0002] Category-level object pose estimation is an important research direction in computer vision and robotics. It aims to infer the position and pose of an object in three-dimensional space from a monocular RGB image. Unlike instance-level pose estimation, category-level pose estimation does not rely on a specific CAD model of the object, but instead aims to generalize to different instances of the same category. This capability is crucial in real-world applications, as many scenarios, such as robotic grasping, augmented reality, and autonomous driving, require systems to handle a large number of unknown objects.

[0003] Early object pose estimation methods primarily relied on template matching and geometric features. Template matching methods establish 2D-to-3D correspondences using features such as edges or SIFT and use the PnP algorithm to determine pose. However, these methods perform poorly in the face of occlusion and illumination variations. Point-pair feature (PPF) methods extract surface normals and distance features from point clouds to construct hash tables and estimate pose using a voting mechanism. However, this approach is sensitive to noise and computationally expensive. These early methods typically rely on high-precision depth sensors, are only applicable to static scenes and known models, and have difficulty generalizing to new objects. With the rise of deep learning technology, researchers have begun exploring data-driven pose regression methods. Keypoint detection methods use convolutional neural networks (CNNs) to predict 2D projections of keypoints on the object surface and combine them with the PnP algorithm to determine pose. Direct regression methods use the network to directly output rotation matrices or quaternions, but due to the non-Euclidean nature of rotation representations, these methods have limited regression accuracy. Implicit representation methods, such as Neural Object Coordinate Spaces (NOCS), estimate pose by mapping object surface coordinates to a standard space and aligning them using ICP. However, these methods still rely on CAD models of objects to generate training data, cannot handle shape differences within categories, and NOCS alignment is susceptible to depth estimation errors.

[0004] To overcome the limitations of instance-level pose estimation, researchers have proposed methods for category-level pose estimation based on neural networks. The multi-stage processes of these methods (such as detection, NOCS prediction, depth estimation, and PnP solution) lead to error accumulation and rely on real depth data or synthetic data, which has domain differences from the actual monocular RGB input. The main challenges faced by existing methods include the contradiction between shape generalization and geometric consistency, depth ambiguity of monocular RGB, and temporal separation of dynamic scenes. The geometric differences of objects within the category lead to distortion of the NOCS mapping, and implicit shape encoding is difficult to jointly optimize with the pose parameters. In the absence of real depth input, relying on the monocular depth estimation network, its error will be passed on to subsequent modules. Object motion, occlusion and camera shake in video scenes lead to jitter in single-frame prediction, and there is a lack of effective cross-frame association mechanism.

[0005] The Chinese invention patent with application number 202411318263.7 discloses "Object pose estimation method and device based on single-view reference template", which includes: extracting feature information of a single reference RGB-D image, and generating a reference segmentation mask of a single reference RGB-D image through a preset SAM algorithm; obtaining the target segmentation mask corresponding to the target RGB-D image, and back-projecting it and the reference segmentation mask to obtain a reference point cloud and a target point cloud; performing coarse point cloud matching on the reference point cloud and the target point cloud, establishing a first correspondence matrix between the reference point cloud and the target point cloud, and calculating coarse pose estimation information; based on the coarse pose estimation information, performing fine point cloud matching on the reference point cloud and the target point cloud, establishing a second correspondence matrix between the reference point cloud and the target point cloud, and obtaining fine pose estimation information, thereby obtaining the final pose estimation result of the target to be measured. Summary of the Invention

[0006] To solve the technical problems existing in the background technology, the present invention provides an end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor point query. The technical solution adopted by the present invention is:

[0007] A first aspect of the present invention provides an end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor point query, comprising the following steps:

[0008] A: The RGB image sequence is input into the visual encoder to extract the multi-scale semantic feature pyramid. The semantic features, depth and normal vector geometric features are fused layer by layer through residual connections to generate a 2.5-dimensional multi-scale feature that integrates semantics and geometry.

[0009] B: Based on the output multi-scale features, a 3D implicit anchor queue is established, containing several time steps, each containing a preset number of anchors. The confidence level of the current frame query is screened based on the predicted box overlap ratio, and the queries with the highest scores are pushed to the end of the queue, while queries from previous time steps are removed. A camera pose offset transformation is applied to the historical queries in the queue, unifying the 3D spatial feature representation using LayerNorm.

[0010] C: Based on the 3D implicit anchor queue, using the current frame and historical queries as keys and values, it models the spatiotemporal correlation matrix through multi-head attention and outputs temporally enhanced query features. It also projects the 3D anchors onto the image plane for geometrically perceived feature sampling.

[0011] D: Based on query features enhanced by output timing, implicit 3D queries are mapped to 9-DOF pose parameters through a set prediction mechanism of bipartite graph matching, and end-to-end training is achieved by jointly optimizing multi-task losses.

[0012] As a preferred solution, the method of inputting the RGB image sequence into the visual encoder to extract the multi-scale semantic feature pyramid, and fusing the semantic features, depth and normal vector geometric features layer by layer through residual connection to generate a 2.5-dimensional multi-scale feature that fuses semantics and geometry includes the following:

[0013] A1: A single RGB frame is input into the ViT architecture for block embedding, and a semantic feature pyramid containing multiple levels is output. Features at each level are retained through skip connections to form a semantic feature sequence with decreasing resolution.

[0014] A2: Starting from the highest-level features, the convolutional block of DepthAnythingV2 is used to generate the initial geometric features. After upsampling, the features are added pixel by pixel with the initial features of the same size. Then, cross-modal information is fused through the convolutional layer, and refined depth and normal vector geometric features are output. The above process is repeated until the highest-resolution geometric features are generated.

[0015] A3: Based on the backbone network in step A2, the semantic and geometric features of each layer are spliced ​​along the channel dimension to form a mixed feature block. The number of channels after splicing is reduced to the original dimension through 1×1 convolution, and the multi-scale features [z0,z1,...,z l ], where z0 integrates the richest spatial-semantic-geometric information.

[0016] As a preferred solution, based on the output multi-scale features, a 3D implicit anchor queue containing several time steps is established, with each time step containing a preset number of anchors. The confidence level of the current frame query is screened based on the predicted box overlap ratio, and the queries with the highest scores are pushed to the end of the queue, while queries from historical time steps are removed. The camera pose offset transformation is applied to the historical queries in the queue, and the method of unifying the 3D spatial feature expression through LayerNorm includes the following:

[0017] B1: First, create a queue of 256 3D implicit anchors for three time steps, each initialized to 0. Based on the overlap ratio of the predicted boxes, select the 256 queries with the highest scores and push them to the end of the 3D implicit anchor queue. Meanwhile, remove the 256 queries at time t-3 from the queue.

[0018] B2: Based on the overlap ratio of the predicted boxes, the 256 queries with the highest scores are selected and pushed to the end of the 3D implicit anchor queue. The 256 queries at time t-3 are removed from the queue.

[0019] B3: Apply camera pose offset transformation to historical queries in the queue and use LayerNorm to achieve unified 3D query feature expression.

[0020] As a preferred solution, based on a 3D implicit anchor queue, the current frame and historical queries are used as keys and values, a spatiotemporal correlation matrix is ​​modeled through multi-head attention, and temporally enhanced query features are output. The method of projecting the 3D anchor points onto the image plane for geometrically perceived feature sampling includes the following:

[0021] C1: Using the current frame and the queries in the historical queue as keys and values, multi-head attention calculates the spatiotemporal correlation matrix between the current query and historical queries, and outputs time-enhanced query features;

[0022] C2: Use camera intrinsic parameters to map 3D anchor points to 2D coordinates; perform bilinear interpolation at the corresponding positions in the feature pyramid, and finally predict the sampling offset through a lightweight MLP to achieve adaptive feature positioning.

[0023] As a preferred solution, based on query features enhanced by output timing, implicit 3D queries are mapped to 9-DOF pose parameters through a set prediction mechanism of bipartite graph matching. The method for jointly optimizing multi-task loss to achieve end-to-end training includes the following:

[0024] D1: Build a cost matrix between the predicted query and the true value object based on the Hungarian algorithm. The cost matrix includes classification and pose errors.

[0025] D2: Directly outputs 9DoF parameters through a three-layer forward feedback network, including translation 3D parameters, rotation 3D parameters, and size 3D parameters.

[0026] The second aspect of the present invention provides an end-to-end category-level object pose estimation system based on spatiotemporal implicit anchor point query, the system comprising a multimodal feature implicit fusion module, a dynamic memory queue maintenance module, a time-series-aware feature aggregation module, and a differentiable pose regression module;

[0027] The multimodal feature implicit fusion module is used to input the RGB image sequence into the visual encoder to extract the multi-scale semantic feature pyramid, and fuse the semantic features, depth and normal vector geometric features layer by layer through residual connection to generate a 2.5-dimensional multi-scale feature that integrates semantics and geometry;

[0028] The dynamic memory queue maintenance module is used to establish a 3D implicit anchor queue containing several time steps based on the output multi-scale features, where each time step contains a preset number of anchor points; perform confidence screening on the current frame query based on the predicted box overlap ratio, push the queries with the highest scores to the end of the queue, and remove queries from historical time steps; apply camera pose offset transformation to the historical queries in the queue, and unify the 3D spatial feature expression through LayerNorm;

[0029] The temporal-aware feature aggregation module is used to output temporally enhanced query features based on a 3D implicit anchor queue, using the current frame and historical queries as keys and values, modeling a spatiotemporal correlation matrix through multi-head attention, and projecting the 3D anchor points onto the image plane for geometric-aware feature sampling.

[0030] The differentiable pose regression module is used to map implicit 3D queries into 9-DOF pose parameters based on query features enhanced by output temporal sequence through a set prediction mechanism of bipartite graph matching, and jointly optimize multi-task loss to achieve end-to-end training.

[0031] As a preferred solution, the multimodal feature implicit fusion module includes a multi-scale semantic feature extraction submodule, a residual feature fusion decoding submodule, and a cross-modal channel fusion submodule;

[0032] The multi-scale semantic feature extraction submodule is used to embed the single-frame RGB input into the ViT architecture in blocks, output a semantic feature pyramid containing multiple levels, retain the features of each level through skip connections, and form a semantic feature sequence with decreasing resolution;

[0033] The residual feature fusion decoding submodule is used to generate initial geometric features starting from the highest-level features using the convolution block of DepthAnythingV2, upsample the features and add them pixel by pixel with the initial features of the same size through an addition operation, then perform cross-modal information fusion through the convolution layer, output the refined depth and normal vector geometric features, and repeat the above process until the highest resolution geometric features are generated;

[0034] The cross-modal channel fusion submodule is used to splice the semantic and geometric features of each level along the channel dimension based on the backbone network in the residual feature fusion decoding submodule to form a mixed feature block, and reduce the number of spliced ​​channels to the original dimension through 1×1 convolution, and finally output multi-scale features [z0,z1,...,z l ], where z0 integrates the richest spatial-semantic-geometric information.

[0035] As a preferred solution, the dynamic memory queue maintenance module includes a query initialization submodule, a confidence screening submodule, and a posture compensation encoding submodule;

[0036] The query initialization submodule is used to establish a queue of 256 3D implicit anchor points for three time steps, each of which is initialized to 0; based on the overlap ratio of the predicted boxes, the 256 queries with the highest scores are selected and pushed to the end of the 3D implicit anchor point queue, while the 256 queries at time t-3 are removed from the queue;

[0037] The confidence screening submodule is used to screen based on the overlap ratio of the predicted boxes, select the 256 queries with the highest scores and push them to the end of the 3D implicit anchor queue, and remove the 256 queries at time t-3 from the queue;

[0038] The posture compensation encoding submodule is used to apply camera posture offset transformation to historical queries in the queue, and realize query feature expression in a unified three-dimensional space through LayerNorm.

[0039] As a preferred solution, the temporal-aware feature aggregation module includes a cross-frame attention alignment submodule and a deformable geometric sampling submodule;

[0040] The cross-frame attention alignment submodule is used to use the current frame and the query of the historical queue as the key and value, calculate the spatiotemporal correlation matrix of the current query and the historical query through multi-head attention, and output the query features enhanced by time series;

[0041] The deformable geometry sampling submodule is used to map 3D anchor points to 2D coordinates using camera intrinsic parameters; bilinear interpolation is performed at corresponding positions in the feature pyramid, and finally a lightweight MLP is used to predict the sampling offset to achieve adaptive feature positioning.

[0042] As a preferred solution, the differentiable posture regression module includes a bipartite graph matching submodule and a parameter regression network submodule;

[0043] The bipartite graph matching submodule is used to establish a cost matrix between the predicted query and the true value object based on the Hungarian algorithm, and the cost matrix includes classification and pose errors;

[0044] The parameter regression network submodule is used to directly output 9DoF parameters through a three-layer forward feedback network, where the 9DoF parameters include three-dimensional translation parameters, three-dimensional rotation parameters, and three-dimensional size parameters.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The proposed end-to-end category-level object pose estimation framework demonstrates significant advantages in both technical performance and application value. Through core technological breakthroughs such as implicit multimodal feature fusion, dynamic memory queue maintenance, and differentiable pose regression, it effectively addresses bottlenecks such as error propagation, timing inconsistency, and non-differentiable post-processing in traditional approaches. These technical advantages translate directly into practical application: Using monocular RGB input, the system achieves centimeter-level accuracy without relying on depth sensors or object CAD models. This enables accurate pose identification of unknown parts in industrial robotic grasping scenarios. In augmented reality, it effectively addresses the issue of virtual objects floating within real scenes. The framework's end-to-end architecture significantly simplifies the traditional multi-stage process, achieving over three times the performance of traditional cascaded approaches and meeting the requirements of real-time interaction. By introducing implicit geometric representation and a dynamic memory queue, the system significantly enhances the robustness of pose estimation in occluded scenarios. Furthermore, the differentiable pose regression module eliminates traditional PnP / RANSAC post-processing, enabling the network to optimize geometric consistency end-to-end, simultaneously improving depth prediction accuracy in monocular depth estimation tasks. This is of great significance for obstacle distance perception in autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of an end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor point query provided by this embodiment;

[0048] Figure 2 The overall network structure diagram corresponding to the end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor point query provided by this embodiment. DETAILED DESCRIPTION

[0049] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting the present invention;

[0050] It should be clear that the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the embodiments of the present application.

[0051] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present application. The singular forms "a," "the," and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0052] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0053] In addition, in the description of this application, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship. The present invention is further described below with reference to the accompanying drawings and examples.

[0054] The present invention is further described below with reference to the accompanying drawings and embodiments.

[0055] Example 1

[0056] Please refer to Figure 1 as well as Figure 2 This embodiment provides an end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor point query, comprising the following steps:

[0057] A: The RGB image sequence is input into the visual encoder to extract the multi-scale semantic feature pyramid. The semantic features, depth and normal vector geometric features are fused layer by layer through residual connections to generate a 2.5-dimensional multi-scale feature that integrates semantics and geometry.

[0058] In a specific embodiment, a method for inputting an RGB image sequence into a visual encoder to extract a multi-scale semantic feature pyramid, and fusing semantic features, depth, and normal vector geometric features layer by layer through residual connections to generate a 2.5-dimensional multi-scale feature that fuses semantics and geometry includes:

[0059] A1: A single RGB frame is input into the ViT architecture for block embedding, and a semantic feature pyramid containing multiple levels is output. Features at each level are retained through skip connections to form a semantic feature sequence with decreasing resolution.

[0060] A2: Starting from the highest-level features, the convolutional block of DepthAnythingV2 is used to generate the initial geometric features. After upsampling, the features are added pixel by pixel with the initial features of the same size. Then, cross-modal information is fused through the convolutional layer, and refined depth and normal vector geometric features are output. The above process is repeated until the highest-resolution geometric features are generated.

[0061] A3: Based on the backbone network in step A2, the semantic and geometric features of each layer are spliced ​​along the channel dimension to form a mixed feature block. The number of channels after splicing is reduced to the original dimension through 1×1 convolution, and the multi-scale features [z0,z1,...,z l ], where z0 integrates the richest spatial-semantic-geometric information.

[0062] B: Based on the output multi-scale features, a 3D implicit anchor queue is established, containing several time steps, each containing a preset number of anchors. The confidence level of the current frame query is screened based on the predicted box overlap ratio, and the queries with the highest scores are pushed to the end of the queue, while queries from previous time steps are removed. A camera pose offset transformation is applied to the historical queries in the queue, unifying the 3D spatial feature representation using LayerNorm.

[0063] In a specific embodiment, based on the output multi-scale features, a 3D implicit anchor queue containing several time steps is established, where each time step contains a preset number of anchors; the confidence of the current frame query is screened based on the predicted box overlap ratio, the queries with the highest scores are pushed to the end of the queue, and the queries of the historical time steps are removed; the camera pose offset transformation is applied to the historical queries in the queue, and the method of unifying the three-dimensional spatial feature expression through LayerNorm includes:

[0064] B1: First, create a queue of 256 3D implicit anchors for three time steps, each initialized to 0. Based on the overlap ratio of the predicted boxes, select the 256 queries with the highest scores and push them to the end of the 3D implicit anchor queue. Meanwhile, remove the 256 queries at time t-3 from the queue.

[0065] B2: Based on the overlap ratio of the predicted boxes, the 256 queries with the highest scores are selected and pushed to the end of the 3D implicit anchor queue. The 256 queries at time t-3 are removed from the queue.

[0066] B3: Apply camera pose offset transformation to historical queries in the queue and use LayerNorm to achieve unified 3D query feature expression.

[0067] C: Based on the 3D implicit anchor queue, using the current frame and historical queries as keys and values, it models the spatiotemporal correlation matrix through multi-head attention and outputs temporally enhanced query features. It also projects the 3D anchors onto the image plane for geometrically perceived feature sampling.

[0068] In a specific embodiment, based on a 3D implicit anchor queue, the current frame and historical queries are used as keys and values, a spatiotemporal correlation matrix is ​​modeled through multi-head attention, and temporally enhanced query features are output; and the 3D anchors are projected onto the image plane for geometrically perceived feature sampling. The method includes:

[0069] C1: Using the current frame and the queries in the historical queue as keys and values, multi-head attention calculates the spatiotemporal correlation matrix between the current query and historical queries, and outputs time-enhanced query features;

[0070] C2: Use camera intrinsic parameters to map 3D anchor points to 2D coordinates; perform bilinear interpolation at the corresponding positions in the feature pyramid, and finally predict the sampling offset through a lightweight MLP to achieve adaptive feature positioning.

[0071] D: Based on query features enhanced by output timing, implicit 3D queries are mapped to 9-DOF pose parameters through a set prediction mechanism of bipartite graph matching, and end-to-end training is achieved by jointly optimizing multi-task losses.

[0072] In a specific embodiment, based on query features enhanced by output timing, implicit 3D queries are mapped to 9-DOF pose parameters through a set prediction mechanism of bipartite graph matching, and a method for jointly optimizing multi-task loss to achieve end-to-end training includes:

[0073] D1: Build a cost matrix between the predicted query and the true value object based on the Hungarian algorithm. The cost matrix includes classification and pose errors.

[0074] D2: Directly outputs 9DoF parameters through a three-layer forward feedback network, including translation 3D parameters, rotation 3D parameters, and size 3D parameters.

[0075] Example 2

[0076] This embodiment provides an end-to-end category-level object pose estimation system based on spatiotemporal implicit anchor point query, the system comprising a multimodal feature implicit fusion module, a dynamic memory queue maintenance module, a time-series-aware feature aggregation module, and a differentiable pose regression module;

[0077] The multimodal feature implicit fusion module is used to input the RGB image sequence into the visual encoder to extract the multi-scale semantic feature pyramid, and fuse the semantic features, depth and normal vector geometric features layer by layer through residual connection to generate a 2.5-dimensional multi-scale feature that integrates semantics and geometry;

[0078] The dynamic memory queue maintenance module is used to establish a 3D implicit anchor queue containing several time steps based on the output multi-scale features, where each time step contains a preset number of anchor points; perform confidence screening on the current frame query based on the predicted box overlap ratio, push the queries with the highest scores to the end of the queue, and remove queries from historical time steps; apply camera pose offset transformation to the historical queries in the queue, and unify the 3D spatial feature expression through LayerNorm;

[0079] The temporal-aware feature aggregation module is used to output temporally enhanced query features based on a 3D implicit anchor queue, using the current frame and historical queries as keys and values, modeling a spatiotemporal correlation matrix through multi-head attention, and projecting the 3D anchor points onto the image plane for geometric-aware feature sampling.

[0080] The differentiable pose regression module is used to map implicit 3D queries into 9-DOF pose parameters based on query features enhanced by output temporal sequence through a set prediction mechanism of bipartite graph matching, and jointly optimize multi-task loss to achieve end-to-end training.

[0081] In a specific embodiment, the multimodal feature implicit fusion module includes a multi-scale semantic feature extraction submodule, a residual feature fusion decoding submodule, and a cross-modal channel fusion submodule;

[0082] The multi-scale semantic feature extraction submodule is used to embed the single-frame RGB input into the ViT architecture in blocks, output a semantic feature pyramid containing multiple levels, retain the features of each level through skip connections, and form a semantic feature sequence with decreasing resolution;

[0083] The residual feature fusion decoding submodule is used to generate initial geometric features starting from the highest-level features using the convolution block of DepthAnythingV2, upsample the features and add them pixel by pixel with the initial features of the same size through an addition operation, then perform cross-modal information fusion through the convolution layer, output the refined depth and normal vector geometric features, and repeat the above process until the highest resolution geometric features are generated;

[0084] The cross-modal channel fusion submodule is used to splice the semantic and geometric features of each level along the channel dimension based on the backbone network in the residual feature fusion decoding submodule to form a mixed feature block, and reduce the number of spliced ​​channels to the original dimension through 1×1 convolution, and finally output multi-scale features [z0,z1,...,z l], where z0 integrates the richest spatial-semantic-geometric information.

[0085] In a specific embodiment, the dynamic memory queue maintenance module includes a query initialization submodule, a confidence screening submodule, and a posture compensation encoding submodule;

[0086] The query initialization submodule is used to establish a queue of 256 3D implicit anchor points for three time steps, each of which is initialized to 0; based on the overlap ratio of the predicted boxes, the 256 queries with the highest scores are selected and pushed to the end of the 3D implicit anchor point queue, while the 256 queries at time t-3 are removed from the queue;

[0087] The confidence screening submodule is used to screen based on the overlap ratio of the predicted boxes, select the 256 queries with the highest scores and push them to the end of the 3D implicit anchor queue, and remove the 256 queries at time t-3 from the queue;

[0088] The posture compensation encoding submodule is used to apply camera posture offset transformation to historical queries in the queue, and realize query feature expression in a unified three-dimensional space through LayerNorm.

[0089] In a specific embodiment, the temporal-aware feature aggregation module includes a cross-frame attention alignment submodule and a deformable geometric sampling submodule;

[0090] The cross-frame attention alignment submodule is used to use the current frame and the query of the historical queue as the key and value, calculate the spatiotemporal correlation matrix of the current query and the historical query through multi-head attention, and output the query features enhanced by time series;

[0091] The deformable geometry sampling submodule is used to map 3D anchor points to 2D coordinates using camera intrinsic parameters; bilinear interpolation is performed at corresponding positions in the feature pyramid, and finally a lightweight MLP is used to predict the sampling offset to achieve adaptive feature positioning.

[0092] In a specific embodiment, the differentiable posture regression module includes a bipartite graph matching submodule and a parameter regression network submodule;

[0093] The bipartite graph matching submodule is used to establish a cost matrix between the predicted query and the true value object based on the Hungarian algorithm, and the cost matrix includes classification and pose errors;

[0094] The parameter regression network submodule is used to directly output 9DoF parameters through a three-layer forward feedback network, where the 9DoF parameters include three-dimensional translation parameters, three-dimensional rotation parameters, and three-dimensional size parameters.

[0095] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. An end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor query, characterized by: The following steps are involved: A: The RGB image sequence is input into the visual encoder to extract the multi-scale semantic feature pyramid. The semantic features, depth and normal vector geometric features are fused layer by layer through residual connections to generate a 2.5-dimensional multi-scale feature that integrates semantics and geometry. B: Based on the output multi-scale features, a 3D implicit anchor queue is established, containing several time steps, each containing a preset number of anchors. The confidence level of the current frame query is screened based on the predicted box overlap ratio, and the queries with the highest scores are pushed to the end of the queue, while queries from previous time steps are removed. A camera pose offset transformation is applied to the historical queries in the queue, unifying the 3D spatial feature representation using LayerNorm. C: Based on the 3D implicit anchor queue, it uses the current frame and historical queries as keys and values, models the spatiotemporal correlation matrix through multi-head attention, and outputs temporally enhanced query features; And project the 3D anchor points to the image plane for geometric perception feature sampling; D: Based on query features enhanced by output timing, implicit 3D queries are mapped to 9-DOF pose parameters through a set prediction mechanism of bipartite graph matching, and end-to-end training is achieved by jointly optimizing multi-task losses.

2. The end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor query according to claim 1, characterized in that: The method of inputting the RGB image sequence into the visual encoder to extract the multi-scale semantic feature pyramid, and fusing the semantic features, depth and normal vector geometric features layer by layer through residual connection to generate a 2.5-dimensional multi-scale feature that fuses semantics and geometry includes: A1: A single RGB frame is input into the ViT architecture for block embedding, and a semantic feature pyramid containing multiple levels is output. Features at each level are retained through skip connections to form a semantic feature sequence with decreasing resolution. A2: Starting from the highest-level features, the convolutional block of DepthAnythingV2 is used to generate the initial geometric features. After upsampling, the features are added pixel by pixel with the initial features of the same size. Then, cross-modal information is fused through the convolutional layer, and refined depth and normal vector geometric features are output. The above process is repeated until the highest-resolution geometric features are generated. A3: Based on the backbone network in step A2, the semantic and geometric features of each layer are spliced ​​along the channel dimension to form a mixed feature block. The number of channels after splicing is reduced to the original dimension through 1×1 convolution, and the multi-scale features [z0,z1,...,z l ], where z0 integrates the richest spatial-semantic-geometric information.

3. The end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor query according to claim 1, characterized in that: Based on the output multi-scale features, a 3D implicit anchor queue containing several time steps is established, with each time step containing a preset number of anchors. The confidence level of the current frame query is screened based on the predicted box overlap ratio, and the queries with the highest scores are pushed to the end of the queue, while queries from previous time steps are removed. A camera pose offset transformation is applied to the historical queries in the queue. The methods for unifying the 3D spatial feature expression through LayerNorm include: B1: First, create a queue of 256 3D implicit anchors for three time steps, each initialized to 0. Based on the overlap ratio of the predicted boxes, select the 256 queries with the highest scores and push them to the end of the 3D implicit anchor queue. Meanwhile, remove the 256 queries at time t-3 from the queue. B2: Based on the overlap ratio of the predicted boxes, the 256 queries with the highest scores are selected and pushed to the end of the 3D implicit anchor queue. The 256 queries at time t-3 are removed from the queue. B3: Apply camera pose offset transformation to historical queries in the queue and use LayerNorm to achieve unified 3D query feature expression.

4. The end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor query according to claim 1, characterized in that: Based on the 3D implicit anchor queue, the current frame and historical queries are used as keys and values, and the spatiotemporal correlation matrix is ​​modeled through multi-head attention to output temporally enhanced query features. The method of projecting the 3D anchor point onto the image plane for geometrically perceptual feature sampling includes: C1: Using the current frame and the queries in the historical queue as keys and values, multi-head attention calculates the spatiotemporal correlation matrix between the current query and historical queries, and outputs time-enhanced query features; C2: Use camera intrinsic parameters to map 3D anchor points to 2D coordinates; perform bilinear interpolation at the corresponding positions in the feature pyramid, and finally predict the sampling offset through a lightweight MLP to achieve adaptive feature positioning.

5. The end-to-end category-level object pose estimation method based on spatiotemporal implicit anchor query according to claim 1, characterized in that: Based on query features enhanced by output timing, implicit 3D queries are mapped to 9-DOF pose parameters through a set prediction mechanism of bipartite graph matching. The method of jointly optimizing multi-task loss to achieve end-to-end training includes: D1: Build a cost matrix between the predicted query and the true value object based on the Hungarian algorithm. The cost matrix includes classification and pose errors. D2: Directly outputs 9DoF parameters through a three-layer forward feedback network, including translation 3D parameters, rotation 3D parameters, and size 3D parameters.

6. An end-to-end category-level object pose estimation system based on spatiotemporal implicit anchor query, characterized by: The system includes a multimodal feature implicit fusion module, a dynamic memory queue maintenance module, a time series perception feature aggregation module, and a differentiable posture regression module; The multimodal feature implicit fusion module is used to input the RGB image sequence into the visual encoder to extract the multi-scale semantic feature pyramid, and fuse the semantic features, depth and normal vector geometric features layer by layer through residual connection to generate a 2.5-dimensional multi-scale feature that integrates semantics and geometry; The dynamic memory queue maintenance module is used to establish a 3D implicit anchor queue containing several time steps based on the output multi-scale features, where each time step contains a preset number of anchor points; perform confidence screening on the current frame query based on the predicted box overlap ratio, push the queries with the highest scores to the end of the queue, and remove queries from historical time steps; apply camera pose offset transformation to the historical queries in the queue, and unify the 3D spatial feature expression through LayerNorm; The temporal-aware feature aggregation module is used to model the spatiotemporal correlation matrix based on the 3D implicit anchor queue, using the current frame and historical queries as keys and values, and outputting temporally enhanced query features through multi-head attention. And project the 3D anchor points to the image plane for geometric perception feature sampling; The differentiable pose regression module is used to map implicit 3D queries into 9-DOF pose parameters based on query features enhanced by output temporal sequence, through a set prediction mechanism of bipartite graph matching, and jointly optimize multi-task loss to achieve end-to-end training.

7. The end-to-end category-level object pose estimation system based on spatiotemporal implicit anchor query according to claim 6, characterized in that: The multimodal feature implicit fusion module includes a multi-scale semantic feature extraction submodule, a residual feature fusion decoding submodule, and a cross-modal channel fusion submodule; The multi-scale semantic feature extraction submodule is used to embed the single-frame RGB input into the ViT architecture in blocks, output a semantic feature pyramid containing multiple levels, retain the features of each level through skip connections, and form a semantic feature sequence with decreasing resolution; The residual feature fusion decoding submodule is used to generate initial geometric features starting from the highest-level features using the convolution block of DepthAnythingV2, upsample the features and add them pixel by pixel with the initial features of the same size through an addition operation, then perform cross-modal information fusion through the convolution layer, output the refined depth and normal vector geometric features, and repeat the above process until the highest resolution geometric features are generated; The cross-modal channel fusion submodule is used to splice the semantic and geometric features of each level along the channel dimension based on the backbone network in the residual feature fusion decoding submodule to form a mixed feature block, and reduce the number of spliced ​​channels to the original dimension through 1×1 convolution, and finally output multi-scale features [z0,z1,...,z l ], where z0 integrates the richest spatial-semantic-geometric information.

8. The end-to-end category-level object pose estimation system based on spatiotemporal implicit anchor query according to claim 6, characterized in that: The dynamic memory queue maintenance module includes a query initialization submodule, a confidence screening submodule, and a posture compensation encoding submodule; The query initialization submodule is used to establish a queue of 256 3D implicit anchor points for three time steps, each of which is initialized to 0; based on the overlap ratio of the predicted boxes, the 256 queries with the highest scores are selected and pushed to the end of the 3D implicit anchor point queue, while the 256 queries at time t-3 are removed from the queue; The confidence screening submodule is used to screen based on the overlap ratio of the predicted boxes, select the 256 queries with the highest scores and push them to the end of the 3D implicit anchor queue, and remove the 256 queries at time t-3 from the queue; The posture compensation encoding submodule is used to apply camera posture offset transformation to historical queries in the queue, and realize query feature expression in a unified three-dimensional space through LayerNorm.

9. The end-to-end category-level object pose estimation system based on spatiotemporal implicit anchor query according to claim 6, characterized in that: The temporal perception feature aggregation module includes a cross-frame attention alignment submodule and a deformable geometry sampling submodule; The cross-frame attention alignment submodule is used to use the current frame and the query of the historical queue as the key and value, calculate the spatiotemporal correlation matrix of the current query and the historical query through multi-head attention, and output the query features enhanced by time series; The deformable geometry sampling submodule is used to map 3D anchor points to 2D coordinates using camera intrinsic parameters; bilinear interpolation is performed at corresponding positions in the feature pyramid, and finally a lightweight MLP is used to predict the sampling offset to achieve adaptive feature positioning.

10. The end-to-end category-level object pose estimation system based on spatiotemporal implicit anchor query according to claim 6, characterized in that: The differentiable posture regression module includes a bipartite graph matching submodule and a parameter regression network submodule; The bipartite graph matching submodule is used to establish a cost matrix between the predicted query and the true value object based on the Hungarian algorithm, and the cost matrix includes classification and pose errors; The parameter regression network submodule is used to directly output 9DoF parameters through a three-layer forward feedback network, where the 9DoF parameters include three-dimensional translation parameters, three-dimensional rotation parameters, and three-dimensional size parameters.

Images