Three-dimensional relative attitude estimation method and device, electronic equipment and storage medium

Abstract

The invention provides a three-dimensional relative attitude estimation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a single support image and a query image, and carrying out the measurement depth estimation of the support image based on a measurement depth estimation model pre-trained across a data set, and obtaining a support depth map; semantic features are extracted to obtain a support semantic graph and a query semantic graph; constructing a 2.5 D support grid model based on the support depth map, the support image and the support semantic map; rendering the 2.5 D support grid model from the current relative attitude to obtain a rendered image and a rendered semantic graph, and optimizing the current relative attitude based on the query image, the rendered image, the query semantic graph and the rendered semantic graph to obtain the next relative attitude. And circularly executing the next relative attitude as the current relative attitude until a termination condition is met, and determining the last relative attitude as the three-dimensional relative attitude. According to the invention, training-free generalized attitude estimation based on a single support image is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and three-dimensional perception technology, and in particular to a three-dimensional relative posture estimation method, device, electronic device and storage medium. Background Art

[0002] Three-dimensional object pose estimation is a key technology for enabling interaction with real-world objects, with widespread applications in fields such as computer vision, virtual / augmented reality, robotics, and autonomous driving. This technology can estimate the pose of objects in input RGB (Red-Green-Blue) images, providing key support for applications such as augmented reality positioning, industrial grasping planning, and three-dimensional perception.

[0003] Current mainstream methods primarily rely on CAD models or multi-view annotated data to construct 3D representations of objects. For example, CAD template matching-based methods pre-scan the object to obtain a 3D model, while multi-view support methods reconstruct 3D geometric features from multi-angle images. However, these methods face significant limitations in practical applications: CAD models rely on expensive, specialized scanning equipment, making them difficult to generalize to unknown objects in dynamic scenes; and multi-view methods require intensive image acquisition and pose annotation, resulting in extremely high data preparation costs.

[0004] While the recently emerging single-view relative pose estimation method avoids the need for 3D modeling, it still requires training deep neural networks based on large amounts of annotated data. This leads to a sharp drop in performance in cross-dataset scenarios due to differences in data distribution. Achieving accurate zero-shot pose estimation across datasets, without relying on annotated data or CAD models, using only a single supporting image, has become a key bottleneck restricting the widespread adoption of this technology. To address this issue, a training-free 3D pose estimation solution is urgently needed that can generalize across scenarios using only a single supporting RGB image. Summary of the Invention

[0005] The present invention provides a three-dimensional relative pose estimation method, device, electronic device and storage medium, which are used to solve the defects of the existing technology that three-dimensional pose estimation relies on labeled data or CAD models, resulting in high cost and poor generalization.

[0006] The present invention provides a three-dimensional relative posture estimation method, comprising: Obtain a single support image and a query image, perform metric depth estimation on the support image based on a metric depth estimation model pre-trained across datasets, and obtain a support depth map; Extracting semantic features of the target object in the support image and the query image respectively to obtain a support semantic graph and a query semantic graph; Constructing a 2.5D support mesh model of the target object based on the support depth map, the support image, and the support semantic map; determining an initial relative pose from the plurality of uniformly sampled candidate relative poses, taking the initial relative pose as a current relative pose, rendering the 2.5D support mesh model from the current relative pose to an image plane by a differentiable renderer to obtain a rendered image and a rendered semantic map under the current relative pose; optimizing the current relative pose based on the query image, the rendered image, the query semantic map and the rendered semantic map to obtain a next relative pose; performing rendering and optimization cyclically by taking the next relative pose as the current relative pose until an iteration termination condition is reached, and determining a last relative pose as the three-dimensional relative pose of the target object in the query image.

[0007] According to the three-dimensional relative pose estimation method provided by the application, the optimization of the current relative pose based on the query image, the rendered image, the query semantic map and the rendered semantic map to obtain a next relative pose comprises: determining an optimization loss under the current relative pose based on the query image, the rendered image, the query semantic map and the rendered semantic map; determining a pose gradient by back propagation through a rendering pipeline based on the optimization loss under the current relative pose; optimizing the current relative pose based on the pose gradient to obtain a next relative pose.

[0008] According to the three-dimensional relative pose estimation method provided by the application, the optimization of the current relative pose based on the query image, the rendered image, the query semantic map and the rendered semantic map to obtain a next relative pose comprises: determining an image similarity loss based on the difference between the query image and the rendered image; determining a semantic similarity loss based on the difference between the query semantic map and the rendered semantic map; weighting and summing the image similarity loss and the semantic similarity loss to obtain the optimization loss under the current relative pose.

[0009] According to the three-dimensional relative pose estimation method provided by the application, the initial relative pose is determined based on the following steps: generating a plurality of candidate relative poses by uniform sampling; rendering the 2.5D support mesh model from each candidate relative pose to an image plane by a differentiable renderer to obtain a rendered image and a rendered semantic map under each candidate relative pose; determining an initial loss under each candidate relative pose based on the rendered image and the rendered semantic map under each candidate relative pose, and the query image and the query semantic map. taking the candidate relative pose with the minimum initial loss as the initial relative pose.

[0010] According to the three-dimensional relative pose estimation method provided by the application, the semantic features of the target object in the support image and the query image are extracted respectively to obtain a support semantic graph and a query semantic graph, which comprises: Based on the pre-trained visual model, the semantic features of the target object in the support image and the query image are extracted respectively to obtain high-dimensional support semantic features and high-dimensional query semantic features. Based on the principal component analysis method, the high-dimensional support semantic features and the high-dimensional query semantic features are processed by dimension reduction transformation to obtain the support semantic graph and the query semantic graph.

[0011] According to the three-dimensional relative pose estimation method provided by the application, the 2.5D support grid model of the target object is constructed based on the support depth graph, the support image and the support semantic graph, which comprises: Based on the support depth graph and the camera internal parameter, the 2.5D point cloud data of the target object is constructed. Based on the 2.5D point cloud data of the target object, a 2.5D surface grid is constructed by applying a triangulation algorithm. The support image and the support semantic graph are taken as texture information, which is drawn onto the 2.5D surface grid by texture mapping technology to obtain the 2.5D support grid model of the target object.

[0012] According to the three-dimensional relative pose estimation method provided by the application, the 2.5D support grid model is rendered from the current relative pose to the image plane by the differentiable renderer to obtain the rendered image and the rendered semantic graph under the current relative pose, which comprises: When rendering, the triangular patches of the 2.5D support grid model facing away from the camera direction are removed.

[0013] The application also provides a three-dimensional relative pose estimation device, which comprises: An image acquisition unit is configured to acquire a single support image and a query image, and perform metric depth estimation on the support image based on a metric depth estimation model pre-trained across datasets to obtain a support depth graph. A semantic extraction unit is configured to extract semantic features of a target object in the support image and the query image respectively to obtain a support semantic graph and a query semantic graph. A grid construction unit is configured to construct a 2.5D support grid model of the target object based on the support depth graph, the support image and the support semantic graph. an image rendering unit configured to determine an initial relative pose from the plurality of uniformly sampled candidate relative poses, and render the 2.5D support mesh model from the initial relative pose to an image plane by a differentiable renderer to obtain a rendered image and a rendered semantic map under the initial relative pose; a pose optimization unit configured to optimize the initial relative pose to obtain a next relative pose based on the query image, the rendered image, the query semantic map and the rendered semantic map; a pose determination unit configured to cyclically execute the rendering and the optimization by taking the next relative pose as the initial relative pose until an iteration termination condition is reached, and determine a final relative pose as the three-dimensional relative pose of the target object in the query image.

[0014] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the three-dimensional relative pose estimation method according to any one of the above when executing the computer program.

[0015] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program is executable on a processor to implement the three-dimensional relative pose estimation method according to any one of the above.

[0016] The application further provides a computer program product comprising a computer program, and the computer program is executable on a processor to implement the three-dimensional relative pose estimation method according to any one of the above.

[0017] The three-dimensional relative pose estimation method, device, electronic device and storage medium provided by the application have at least the following beneficial effects: 1) No training and strong generalization ability: No time-consuming neural network model training is required for specific objects, categories or scenes, and no label data with three-dimensional pose true value is required. This training-free feature fundamentally eliminates the dependence of traditional learning-based methods on a large amount of labeled data, and can naturally generalize to new objects and differentiated data set scenarios that have not been seen during the development stage. Experiments show that the performance of the method is better than that of a variety of existing supervised learning methods under strict pose estimation accuracy evaluation standards and challenging cross-dataset tests.

[0018] 2) Only a single support image is required, and the data acquisition cost is low: the method only needs one support image and one query image to complete the relative pose estimation. This greatly reduces the requirement for input support information, avoiding the difficulties and high costs brought by traditional methods of obtaining complete three-dimensional CAD models of objects or collecting and calibrating multiple support images from different perspectives.

[0019] 3) Using 2.5D support mesh model to improve practicability and robustness: By constructing the 2.5D mesh model of the object directly from a single support depth map for subsequent rendering and comparison, the method ingeniously avoids the difficult problem of accurately reconstructing or hallucinating the complete 3D model of the unseen object from single-view information. The 2.5D support mesh model is closer to the direct observation of the original sensor, reducing unnecessary shape inference errors, thereby improving the practicability and robustness to imperfections of input data in practical applications.

[0020] 4) Fusion of semantic information to enhance adaptability to complex situations: The method innovatively integrates the semantic feature map into the rendering and comparison process of the 2.5D support mesh model. Semantic information can provide structural clues independent of object parts, materials or textures, enabling the method to maintain high pose estimation robustness and accuracy when dealing with complex situations such as partial occlusion of the target object, large pose difference between the support and query views, and lack of rich texture of the object itself.

[0021] 5) High-precision "sampling-rendering-optimization" process: The method combines an effective initial pose large-scale sampling strategy (for global search) and a fine iterative optimization framework based on differentiable rendering (for local fine-tuning). This two-stage strategy can effectively avoid falling into local optimal solution, thereby converging to an accurate relative pose result.

[0022] 6) Wide potential application prospects: The method has been fully experimentally verified on the publicly available pose estimation benchmark dataset (such as LineMOD), proving its effectiveness and superiority. Due to its training-free, strong generalization and single support image requirement, the method is particularly suitable for fields such as robot grasping unknown objects, real-time augmented reality scene construction, and other fields that require fast and robust pose perception of temporarily appearing unknown objects in the environment. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0024] Figure 1 is one of the flowcharts of the three-dimensional relative pose estimation method provided by the present application.

[0025] Figure 2 is the flowchart of the initial relative pose determination method provided by the present application.

[0026] Figure 3 is a structural schematic diagram of a three-dimensional relative pose estimation device provided by the present application.

[0027] Figure 4 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0029] In view of the problem in the prior art that three-dimensional pose estimation needs to rely on labeled data or CAD models, resulting in high cost and poor generalization, an embodiment of the present application proposes a three-dimensional relative pose estimation method. In the method, a single support image and a query image are first obtained, a support depth map is obtained by performing metric depth estimation on the support image based on a cross-dataset pre-trained metric depth estimation model, semantic features of target objects in the support image and the query image are extracted respectively to obtain a support semantic map and a query semantic map, a 2.5D support mesh model of the target objects is constructed based on the support depth map, the support image and the support semantic map, the 2.5D support mesh model is rendered from a current relative pose to an image plane by a differentiable renderer to obtain a rendered image and a rendered semantic map under the current relative pose, and the current relative pose is optimized based on the query image, the rendered image, the query semantic map and the rendered semantic map to obtain a next relative pose, and the initial relative pose is determined from a plurality of uniformly sampled candidate relative poses. The rendering and optimization are executed in a loop with the next relative pose as the current relative pose until an iteration termination condition is reached, and the last relative pose is determined as the three-dimensional relative pose of the target object in the query image.

[0030] The method provided by the embodiments of the present invention achieves high-precision pose estimation using only a single RGB image, without relying on multi-view training data or ground-truth depth information. Compared to existing methods (such as supervised training schemes based on CAD models or multi-view RGB(-D) images), the embodiments of the present invention introduce a generalizable depth estimator to predict metric depth values, completely eliminating the reliance on geometric priors such as CAD models or depth images, significantly improving the algorithm's generalizability. This improvement not only simplifies the data preparation process (eliminating the need for time-consuming depth data acquisition or synthesis) but also enables the model to generalize directly to unknown scenes, addressing the performance degradation of previous methods when deployed across environments. Furthermore, while maintaining real-time performance, this solution uses differentiable rendering to co-align object appearance and semantic information, transforming the pose estimation problem into an optimization problem. This not only avoids the need for ground-truth pose annotations but also achieves more robust pose prediction, particularly in complex scenes with sparse textures or varying illumination.

[0031] The execution subject of this method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed three-dimensional relative posture estimation device, or it can be a three-dimensional relative posture estimation device set in the electronic device, which can be implemented by software, hardware, or a combination of both.

[0032] In the description of the embodiments of the present invention, it should be understood that “a plurality of” means two or more than two, unless otherwise clearly and specifically defined.

[0033] Figure 1 This is one of the flow charts of the three-dimensional relative posture estimation method provided by the present invention, such as Figure 1 As shown, the method includes the following steps: Step 110 : Obtain a single support image and a query image, perform metric depth estimation on the support image based on a metric depth estimation model pre-trained across datasets, and obtain a support depth map.

[0034] Specifically, the support image and the query image can be RGB images taken by a camera. The input image used as the benchmark usually contains known scene information, such as target objects and environment. Refers to the image for which 3D pose estimation is required. Both the support image and the query image contain the target object. Furthermore, the target object in the support image can also be obtained. and query image The segmentation masks in , which are used to focus the processing on the target object, can be pre-obtained by any existing segmentation algorithm (such as Segment Anything Model - SAM). The intrinsic parameter matrix of the camera Also as input, known parameters for subsequent geometric transformation and rendering.

[0035] Then, a metric depth estimation is performed on the support image to obtain a support depth map. Here, an advanced monocular depth estimation algorithm (e.g., Depth Anything v2) can be used to estimate the support image A corresponding depth map. To ensure the geometric shape of the reconstructed 2.5D model is accurate, what is needed is metric depth, i.e., the estimated depth values have consistent units with the spatial dimensions of the image (e.g., both in meters). This is different from models that only provide relative depth information, which can cause objects to be distorted (e.g., stretched or squashed) due to scale ambiguity.

[0036] One main way to obtain metric depth is to use a model pre-trained on a large and diverse external dataset (e.g., Hypersim) that is capable of directly outputting metric depth to enable cross-dataset depth estimation. Alternatively, one can also consider fine-tuning a depth estimation model on a specific dataset to obtain more accurate in-distribution metric depth, or scale-align the relative depth estimation results by using a small amount of known GT depth information. Preferably, the embodiments of the present application use the cross-dataset metric depth estimation method to maximize its generality and training-free nature. The estimated support depth map is denoted as .

[0037] At step 120, semantic features of the target object in the support image and the query image are extracted respectively to obtain a support semantic map and a query semantic map.

[0038] Specifically, the extraction of semantic features for the support semantic map of the support image can be achieved by a self-supervised learning model DINOv2. For the target object (e.g., different pose instances of the same animal) in the support image and the query image, DINOv2 automatically identifies and distinguishes the structured semantic parts (e.g., head, torso, limbs, tail, etc.) of the target object. These part features are distinguishable (e.g., the features of the head are significantly different from the features of the limbs in the vector space) and can reflect the local structure information of the target object.

[0039] The high-dimensional features containing semantic parts extracted by DINOv2 are mapped to a low-dimensional image space to generate corresponding semantic maps. Finally, two semantic maps are obtained, a support semantic map and a query semantic map. The support semantic map is a spatial distribution map of the target object in the support image containing its part-level semantic features. The query semantic map is a spatial distribution map of the target object in the query image also containing part-level semantic features.

[0040] In some embodiments, step 120 specifically includes: Step 121, based on the pre-trained visual model, the semantic features of the target objects in the support image and the query image are extracted respectively, obtaining high-dimensional support semantic features and high-dimensional query semantic features; Step 122, based on principal component analysis, the high-dimensional support semantic features and the high-dimensional query semantic features are respectively processed by dimension reduction transformation, obtaining support semantic graph and query semantic graph.

[0041] Specifically, first, a pre-trained visual feature extraction model is used to extract high-dimensional semantic features, i.e., high-dimensional support semantic features and high-dimensional query semantic features, from the support image and the query image respectively. These features can capture deep information related to object components or surface attributes in the image.

[0042] A powerful pre-trained visual model DINOv2 is used as a semantic feature extractor, denoted as . The support image processed by mask to retain only the object region is input into , obtaining high-dimensional support semantic features . Similarly, the query image processed by mask is input into , obtaining high-dimensional query semantic features . In an embodiment, the features tokens output by the last layer of DINOv2 ViT-L model can be used as high-dimensional semantic features.

[0043] On this basis, in order to facilitate subsequent texture mapping and image-based comparison, principal component analysis (PCA) is needed to convert high-dimensional semantic features and into semantic graphs with lower dimensions and spatial correspondence (similar to RGB images, such as three channels). In order to ensure the consistency of semantic features between the support image and the query image, the projection matrix in the PCA dimension reduction transformation is calculated and fixed according to the semantic features of the support image, and then this fixed transformation is applied to the semantic features of the query image.

[0044] Specifically, first, the projection matrix of PCA is calculated using the features of the support image , and then this fixed projection matrix is applied to and , obtaining the reduced support semantic graph and the query semantic graph . In this way, it is ensured that the semantic values in the support semantic graph and the query semantic graph have consistency in the same semantic space.

[0045] At step 130, a 2.5D support mesh model of the target object is constructed based on the support depth map, the support image, and the support semantic map.

[0046] Specifically, the support depth map is obtained by performing metric depth estimation on the support image, and each pixel value in the support depth map represents the physical distance (in meters or millimeters) from the corresponding scene point to the camera, having a metric scale significance. The support image, i.e., the input RGB image, contains color and texture information of the target object. The support semantic map identifies the local structure information of the target object.

[0047] Therefore, based on the physical distance information provided by the support depth map, the color and texture information provided by the support image, and the local structure information provided by the support semantic map, a 2.5D support mesh model of the target object can be constructed.

[0048] The 2.5D support mesh model is an intermediate representation form between a 2D image and a complete 3D model. It contains the three-dimensional geometric structure (vertex coordinates, face topology) of the target object, but may not completely cover the entire three-dimensional space (such as only reconstructing the visible surface of the object in the 2D image). It is usually stored in the form of a triangular mesh (Triangle Mesh) with a texture map extracted from the support image.

[0049] In some embodiments, step 130 specifically includes: At step 131, a 2.5D point cloud data of the target object is constructed based on the support depth map and the camera intrinsic parameters; At step 132, a 2.5D surface mesh is constructed by applying a triangulation algorithm based on the 2.5D point cloud data of the target object; At step 133, the support image and the support semantic map are used as texture information and mapped onto the 2.5D surface mesh by texture mapping technology to obtain the 2.5D support mesh model of the target object.

[0050] Specifically, for the construction of the 2.5D support mesh model, the surface mesh and texture are realized by first constructing the 2.5D point cloud data.

[0051] The estimated support depth map and the camera intrinsic parameters are used to transform the valid pixel points in the support depth map to the three-dimensional camera coordinate system by inverse projection, forming a 2.5D point cloud representing the visible surface of the target object This step directly uses single-view depth information to construct the front geometry of the object, avoiding the difficulty of complete and possibly inaccurate 3D shape reconstruction or hallucination generation for unseen objects.

[0052] According to the standard pinhole camera model, each valid pixel on the depth map is inversely projected and their corresponding depth values Convert the points in step 130 to the coordinate system of the three-dimensional camera These three-dimensional points collectively constitute the 2.5D point cloud of the visible front of the target object .

[0053] Based on the generated 2.5D point cloud , reconstruct its corresponding 2.5D surface mesh model Since the 2.5D point cloud usually retains the adjacency relationship of the original image pixels (i.e., regularly sampled in the image plane), the surface mesh can be efficiently reconstructed from these points Given that the point cloud usually has a regular topology in the image plane , the Delaunay triangulation algorithm is used to complete the efficient construction of the 2.5D mesh It can operate according to the projection of the point cloud on the image plane, quickly generating a 2.5D mesh composed of triangular facets.

[0054] In step 133, the original support image and the support semantic map are mapped as textures onto the 2.5D surface mesh constructed in step 132. This can be done through standard UV mapping techniques, where the UV coordinates of each mesh vertex can be directly obtained from its pixel position in the original support image. Finally, a 2.5D support model is obtained that simultaneously possesses geometric shape, color texture, and semantic texture }.

[0055] After constructing the 2.5D support mesh model of the target object through step 130, in order to realize a three-dimensional pose estimation scheme that does not require training and can achieve cross-scene generalization with only a single support image, through a differentiable rendering and comparison framework, the relative three-dimensional pose of the same object in the query image is iteratively optimized through gradient backpropagation.

[0056] In order to provide a reasonable initialization starting point for subsequent iterative optimization, first estimate the initial relative pose. The relative pose refers to the pose of the current camera relative to the support model coordinate system, represented by a rotation matrix R, which describes the direction of observing the model from different angles. First, a set of candidate relative poses is uniformly sampled through heuristic methods or based on prior knowledge. The Fibonacci lattice is used to uniformly sample multiple view directions on the unit sphere, and multiple different in-plane rotation angles are combined for each view direction to generate these candidate relative poses. Then, one of the multiple candidate relative poses is selected as the initial relative pose.

[0057] In some embodiments, Figure 2 is a flowchart of the initial relative pose determination method provided by the present application, as shown in Figure 2 The initial relative pose is determined based on the following steps: Step 210, a plurality of candidate relative poses are generated by uniform sampling; Step 220, a 2.5D support mesh model is rendered from each candidate relative pose to an image plane by a differentiable renderer, to obtain a rendered image and a rendered semantic map under each candidate relative pose; Step 230, based on the rendered image and the rendered semantic map under each candidate relative pose, and the query image and the query semantic map, an initial loss under each candidate relative pose is determined; Step 240, the candidate relative pose with the minimum initial loss is taken as the initial relative pose.

[0058] Specifically, in order to find a good initial solution in a wide pose space, thereby avoiding the optimization process from falling into a poor local minimum, step 210 first generates a series of widely distributed candidate relative poses One implementation is to use the Fibonacci lattice algorithm to uniformly sample different viewpoint directions (defined by azimuth and elevation angles) on a unit sphere, and for each sampled viewpoint direction, further uniformly sample different in-plane rotation angles, a total of candidate relative poses can be generated. For example, in the experiment, the number of candidate relative poses can be set to , , thereby obtaining 4000 candidate relative poses.

[0059] For each candidate relative pose , a differentiable renderer is used to render the constructed textured 2.5D support mesh model from the pose to the image plane, to obtain the corresponding rendered image and rendered semantic map ( ) under each candidate relative pose. During the rendering process, a backface culling operation needs to be performed, i.e., removing those mesh patches that are facing inside or back of the model from the current viewpoint, to ensure the correctness of the rendering result.

[0060] That is, the 2.5D support mesh model is rendered in real time, and the triangle patches in the mesh model that face away from the camera are removed. Specifically, for each triangle patch in the mesh model, the dot product of the surface normal vector of the triangle patch and the line-of-sight vector from the camera to the center of the triangle patch is calculated. In the camera coordinate system, the line-of-sight vector is usually along the optical axis direction, such as [0, 0, 1]. If the dot product is greater than or equal to 0, it indicates that the triangle patch faces away from the camera, and the triangle patch will be removed and will not participate in rendering.

[0061] After obtaining the rendered image and the rendered semantic map under each candidate relative pose, an initial loss under each candidate relative pose is determined based on the rendered image and the rendered semantic map under each candidate relative pose, and the query image and the query semantic map.

[0062] In step 230, for each rendered candidate view pair, the rendered image and the rendered semantic map are compared with the query image pair, the query image and the query semantic map , and a comprehensive loss function value is calculated.

[0063] The loss function value can be divided into two parts, an RGB loss and a semantic loss. For example, the RGB loss , and the semantic loss , where is a multi-scale structural similarity index, which is a commonly used image quality / similarity evaluation index. The initial loss can be set as the simple sum or the weighted sum of the two parts of the loss, and the initial loss is represented by the formula: , where the weight is set according to experience, such as .

[0064] All candidate relative poses are compared to obtain the initial loss , and the candidate relative pose that minimizes the loss value is selected as the initial relative pose for the subsequent iterative optimization stage.

[0065] After obtaining the initial relative pose, the initial relative pose is taken as the current relative pose, and the following steps are performed: In step 140, the 2.5D support mesh model is rendered from the current relative pose to the image plane by using the differentiable renderer to obtain the rendered image and the rendered semantic map under the current relative pose; In step 150, the current relative pose is optimized to obtain the next relative pose based on the query image, the rendered image, the query semantic map, and the rendered semantic map.

[0066] Specifically, the rendering process is similar to step 220. The same differentiable renderer as in step 220 is used , according to the current relative pose , a textured 2.5D support model and camera intrinsics , a rendered image under the current relative pose is obtained and a rendered semantic map . Mathematically, it is represented as: Note that this process also performs back-face culling.

[0067] Then step 150 is performed, which optimizes the current relative pose based on the query image, the rendered image, the query semantic map and the rendered semantic map to obtain the next relative pose. In some embodiments, this step specifically includes: Step 151, based on the query image, the rendered image, the query semantic map and the rendered semantic map, determines the optimization loss under the current relative pose; Step 152, based on the optimization loss under the current relative pose, determines the pose gradient by backpropagation through the rendering pipeline; Step 153, based on the pose gradient, optimizes the current relative pose to obtain the next relative pose.

[0068] Specifically, the optimization loss under the current relative pose can be determined based on the query image, the rendered image, the query semantic map and the rendered semantic map. The optimization loss can measure whether the rendering result under the current camera view is aligned with the query image and its semantic map. By comparing the differences between the four sets of data, a multi-modal joint loss function is calculated to measure the error of the current pose estimation.

[0069] In some embodiments, based on the difference between the query image and the rendered image, an image similarity loss is determined; based on the difference between the query semantic map and the rendered semantic map, a semantic similarity loss is determined; the image similarity loss and the semantic similarity loss are weighted and summed to obtain the optimization loss under the current relative pose. The optimization loss is represented by the formula:

[0070] wherein, is the image similarity loss, is the semantic similarity loss.

[0071] Then, based on the optimization loss under the current relative pose, the pose gradient is determined by backpropagation through the rendering pipeline. Since the renderer and the loss function are both differentiable, the gradient of the optimization loss with respect to the pose parameter (using quaternion representation of the rotation matrix) is the pose gradient which is calculated by backpropagation through the entire rendering and comparison chain.

[0072] Use the calculated pose gradient , update the current posture parameters through a gradient descent optimization algorithm (such as Adam optimizer) For example, if Represents the posture parameters, then the update rule can be expressed as: , learning rate You can use a fixed value (such as 0.01) or use a learning rate scheduling strategy (such as ReduceLROnPlateau ) for dynamic adjustment. This allows the current relative posture to be optimized to obtain the next relative posture.

[0073] Step 160 : cyclically perform rendering and optimization using the next relative pose as the current relative pose until an iteration termination condition is reached, and determine the last relative pose as the 3D relative pose of the target object in the query image.

[0074] Specifically, in this embodiment, relative pose estimation is an iterative optimization process, with each iteration performing both the rendering in step 140 and the pose optimization in step 150. The iteration termination condition can be a preset number of iterations, such as 30 iterations; or the optimization process terminates when the change in pose parameters or the improvement in the loss function falls below a certain threshold, thus meeting the iteration termination condition. The final relative pose P at this point is the final estimated 3D relative pose from the support object coordinate system to the query object coordinate system.

[0075] The method provided by the embodiment of the present invention does not require any form of supervised learning or pre-trained network for pose regression. Instead, it estimates the relative pose of unseen objects directly from a single support image and a single query image through an optimization process based on physical rendering and image matching.

[0076] Based on any of the above embodiments, the effectiveness of the present method is verified on the LineMOD benchmark dataset, where the LineMOD dataset contains 13 low-texture real objects, each of which has approximately 1,200 images under different lighting conditions.

[0077] During the experiment, the depth information of the support image is obtained through the cross-dataset metric depth estimation method of Depth Anything v2, which means that the method adopted by the present invention in the experiment only uses RGB images as support input, without the need for real-world annotation of the target dataset or fine-tuning of the depth model for a specific dataset.

[0078] In addition, the experiment uses the accuracy under different angle error thresholds (in Usually take 5, 10, 15, 30°) as a quantitative evaluation index. The angle error is calculated by the geodesic distance between the predicted pose and the real pose. In order to conduct comprehensive performance evaluation, the training-free and label-free method proposed in the present application is compared with a variety of current advanced supervised relative pose estimation algorithms, including ZSP, LoFTR, RelPose++, 3DAHV and DVMNet. For the comparison methods that need training, the experiments report their performance in the cross-dataset (i.e. only using the model trained on the external large-scale dataset such as Objaverse, without fine-tuning on the target dataset) setting. The method of the present application naturally has the generalization ability across objects and datasets due to its training-free characteristic. All experiments are carried out on a single NVIDIA 4090 GPU. The semantic feature extraction uses the last layer output of the DINOv2 ViT-L model, the differentiable renderer is nvdiffrast, the initial pose is selected from 4000 uniformly sampled candidates, the optimizer is Adam, the initial learning rate is set to 0.01, and the iterative optimization is 30 rounds.

[0079] The experimental results of the present application are clearly shown in Table 1. On the LineMOD dataset, the method provided by the present application significantly outperforms the supervised method DVMNet without any training and any annotation, especially in the Acc@30° and Acc@15° indicators, which exceeds 6.78% and 6.88% respectively.

[0080]

[0081] The experimental results shown in Table 1 strongly prove that the training-free generalizable pose estimation method based on a single support image proposed in the present application can achieve or even exceed the performance level of the supervised method requiring a large amount of annotation and training, especially having a significant advantage in cross-dataset generalization and processing unseen objects, using only limited information.

[0082] The three-dimensional relative pose estimation device provided by the present application is described below, and the three-dimensional relative pose estimation device described below can be mutually corresponding to the three-dimensional relative pose estimation method described above.

[0083] Based on the above embodiments, Figure 3 is a structural schematic diagram of the three-dimensional relative pose estimation device provided by the present application, as Figure 3 shown, the device comprises: An image acquisition unit 310 is configured to acquire a single support image and a query image, perform metric depth estimation on the support image based on a cross-dataset pre-trained metric depth estimation model, and obtain a support depth map. The semantic extraction unit 320 is configured to extract semantic features of the target object in the support image and the query image respectively, to obtain a support semantic graph and a query semantic graph. The mesh construction unit 330 is configured to construct a 2.5D support mesh model of the target object based on the support depth map, the support image and the support semantic graph. The image rendering unit 340 is configured to determine an initial relative pose from a plurality of uniformly sampled candidate relative poses, take the initial relative pose as a current relative pose, and render the 2.5D support mesh model from the current relative pose to an image plane by a differentiable renderer to obtain a rendered image and a rendered semantic graph under the current relative pose. The pose optimization unit 350 is configured to optimize the current relative pose based on the query image, the rendered image, the query semantic graph and the rendered semantic graph to obtain a next relative pose. The pose determination unit 360 is configured to cyclically execute rendering and optimization by taking the next relative pose as the current relative pose until an iteration termination condition is reached, and determine a last relative pose as a three-dimensional relative pose of the target object in the query image.

[0084] Based on the above embodiment, the pose optimization unit is specifically configured to: determine an optimization loss under the current relative pose based on the query image, the rendered image, the query semantic graph and the rendered semantic graph; determine a pose gradient by backward propagation through a rendering pipeline based on the optimization loss under the current relative pose; optimize the current relative pose based on the pose gradient to obtain a next relative pose.

[0085] Based on the above embodiment, the pose optimization unit is specifically configured to: determine an image similarity loss based on a difference between the query image and the rendered image; determine a semantic similarity loss based on a difference between the query semantic graph and the rendered semantic graph; weight and sum the image similarity loss and the semantic similarity loss to obtain the optimization loss under the current relative pose.

[0086] Based on the above embodiment, the device further includes an initial pose determination unit configured to: generate a plurality of candidate relative poses by uniform sampling; render the 2.5D support mesh model from each candidate relative pose to an image plane by a differentiable renderer to obtain a rendered image and a rendered semantic graph under each candidate relative pose; determine an initial loss of each candidate relative pose based on the rendered image and the rendered semantic map under each candidate relative pose, and the query image and the query semantic map; take the candidate relative pose with the minimum initial loss as the initial relative pose.

[0087] Based on the above embodiment, the semantic extraction unit is specifically configured to: extract semantic features of the target object in the support image and the query image based on the pre-trained visual model, to obtain high-dimensional support semantic features and high-dimensional query semantic features; perform dimension reduction transformation processing on the high-dimensional support semantic features and the high-dimensional query semantic features based on principal component analysis, to obtain the support semantic map and the query semantic map.

[0088] Based on the above embodiment, the grid construction unit is specifically configured to: construct 2.5D point cloud data of the target object based on the support depth map and the camera intrinsic parameter; apply a triangulation algorithm to construct a 2.5D surface grid based on the 2.5D point cloud data of the target object; draw the support image and the support semantic map onto the 2.5D surface grid as texture information by texture mapping technology, to obtain a 2.5D support grid model of the target object.

[0089] Based on the above embodiment, the image rendering unit is specifically configured to: discard triangular facets of the 2.5D support grid model that face away from the camera direction during rendering.

[0090] Figure 4 An example of a schematic diagram of the physical structure of an electronic device is shown in FIG. 1. Figure 4As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke a logic instruction in the memory 430 to execute a three-dimensional relative pose estimation method, which includes: acquiring a single support image and a query image, performing metric depth estimation on the support image based on a cross-dataset pre-trained metric depth estimation model to obtain a support depth map; extracting semantic features of a target object in the support image and the query image respectively to obtain a support semantic map and a query semantic map; constructing a 2.5D support grid model of the target object based on the support depth map, the support image, and the support semantic map; determining an initial relative pose from a plurality of uniformly sampled candidate relative poses, taking the initial relative pose as a current relative pose, rendering the 2.5D support grid model from the current relative pose to an image plane through a differentiable renderer to obtain a rendered image and a rendered semantic map under the current relative pose, and optimizing the current relative pose based on the query image, the rendered image, the query semantic map, and the rendered semantic map to obtain a next relative pose; taking the next relative pose as the current relative pose to cyclically execute rendering and optimization until an iteration termination condition is reached, and determining the last relative pose as a three-dimensional relative pose of the target object in the query image.

[0091] In addition, the logic instruction in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0092] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program, when executed by a processor, enables a computer to perform the three-dimensional relative pose estimation method provided by the above-mentioned methods, which comprises: obtaining a single support image and a query image, performing metric depth estimation on the support image based on a metric depth estimation model pre-trained across datasets to obtain a support depth map; extracting semantic features of a target object in the support image and the query image respectively to obtain a support semantic map and a query semantic map; constructing a 2.5D support mesh model of the target object based on the support depth map, the support image and the support semantic map; determining an initial relative pose from a plurality of uniformly sampled candidate relative poses, taking the initial relative pose as a current relative pose, rendering the 2.5D support mesh model from the current relative pose to an image plane through a differentiable renderer to obtain a rendered image and a rendered semantic map under the current relative pose, and optimizing the current relative pose based on the query image, the rendered image, the query semantic map and the rendered semantic map to obtain a next relative pose; taking the next relative pose as the current relative pose to cyclically perform rendering and optimization until an iteration termination condition is reached, and determining the last relative pose as the three-dimensional relative pose of the target object in the query image.

[0093] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements a three-dimensional relative pose estimation method provided by the above-mentioned methods, which comprises: obtaining a single support image and a query image, performing metric depth estimation on the support image based on a metric depth estimation model pre-trained across datasets to obtain a support depth map; extracting semantic features of a target object in the support image and the query image respectively to obtain a support semantic map and a query semantic map; constructing a 2.5D support mesh model of the target object based on the support depth map, the support image and the support semantic map; determining an initial relative pose from a plurality of uniformly sampled candidate relative poses, taking the initial relative pose as a current relative pose, rendering the 2.5D support mesh model from the current relative pose to an image plane through a differentiable renderer to obtain a rendered image and a rendered semantic map under the current relative pose, and optimizing the current relative pose based on the query image, the rendered image, the query semantic map and the rendered semantic map to obtain a next relative pose; taking the next relative pose as the current relative pose to cyclically perform rendering and optimization until an iteration termination condition is reached, and determining the last relative pose as the three-dimensional relative pose of the target object in the query image.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0095] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A three-dimensional relative pose estimation method, characterized in that: include: Obtain a single support image and a query image, perform metric depth estimation on the support image based on a metric depth estimation model pre-trained across datasets, and obtain a support depth map; Extracting semantic features of the target object in the support image and the query image respectively to obtain a support semantic graph and a query semantic graph; Constructing a 2.5D support mesh model of the target object based on the support depth map, the support image, and the support semantic map; Determining an initial relative pose from a plurality of uniformly sampled candidate relative poses, using the initial relative pose as a current relative pose, and rendering the 2.5D support mesh model from the current relative pose to an image plane using a differentiable renderer to obtain a rendered image and a rendered semantic map at the current relative pose; Based on the query image, the rendered image, the query semantic graph, and the rendered semantic graph, optimizing the current relative posture to obtain a next relative posture; The next relative posture is used as the current relative posture to perform rendering and optimization in a loop until an iteration termination condition is reached, and the last relative posture is determined as the three-dimensional relative posture of the target object in the query image.

2. The three-dimensional relative posture estimation method according to claim 1, characterized in that The optimizing the current relative posture to obtain a next relative posture based on the query image, the rendered image, the query semantic graph, and the rendered semantic graph includes: Determining an optimization loss at a current relative pose based on the query image, the rendered image, the query semantic graph, and the rendered semantic graph; Based on the optimization loss at the current relative pose, the pose gradient is determined through backpropagation through the rendering pipeline; Based on the posture gradient, the current relative posture is optimized to obtain a next relative posture.

3. The three-dimensional relative posture estimation method according to claim 2, wherein: The determining of the optimization loss at the current relative posture based on the query image, the rendered image, the query semantic graph, and the rendered semantic graph includes: determining an image similarity loss based on a difference between the query image and the rendered image; determining a semantic similarity loss based on a difference between the query semantic graph and the rendered semantic graph; The image similarity loss and the semantic similarity loss are weightedly summed to obtain the optimization loss under the current relative posture.

4. The three-dimensional relative posture estimation method according to any one of claims 1 to 3, characterized in that The initial relative posture is determined based on the following steps: Generate multiple candidate relative poses by uniform sampling; Rendering the 2.5D support mesh model from each candidate relative pose to an image plane using a differentiable renderer to obtain a rendered image and a rendered semantic map at each candidate relative pose; Determining an initial loss for each candidate relative pose based on the rendered image and the rendered semantic map for each candidate relative pose, and the query image and the query semantic map; The candidate relative pose with the smallest initial loss is determined as the initial relative pose.

5. The three-dimensional relative posture estimation method according to any one of claims 1 to 3, characterized in that: The extracting semantic features of the target object in the support image and the query image respectively to obtain a support semantic graph and a query semantic graph includes: Extracting semantic features of the target object in the support image and the query image based on a pre-trained visual model to obtain high-dimensional support semantic features and high-dimensional query semantic features; Based on the principal component analysis method, dimensionality reduction transformation processing is performed on the high-dimensional support semantic features and the high-dimensional query semantic features to obtain the support semantic graph and the query semantic graph.

6. The three-dimensional relative posture estimation method according to any one of claims 1 to 3, characterized in that: The constructing of a 2.5D support mesh model of the target object based on the support depth map, the support image and the support semantic map includes: Constructing 2.5D point cloud data of the target object based on the supported depth map and camera intrinsic parameters; Based on the 2.5D point cloud data of the target object, a 2.5D surface mesh is constructed using a triangulation algorithm; The supporting image and supporting semantic map are used as texture information and mapped onto the 2.5D surface mesh through texture mapping technology to obtain a 2.5D supporting mesh model of the target object.

7. The three-dimensional relative posture estimation method according to any one of claims 1 to 3, characterized in that: The step of rendering the 2.5D supporting mesh model from the current relative pose to the image plane by a differentiable renderer to obtain a rendered image and a rendered semantic map at the current relative pose includes: During rendering, triangles in the 2.5D support mesh model that face away from the camera are culled.

8. A three-dimensional relative posture estimation device, characterized in that: include: An image acquisition unit is used to acquire a single support image and a query image, and perform metric depth estimation on the support image based on a metric depth estimation model pre-trained across data sets to obtain a support depth map; A semantic extraction unit, configured to extract semantic features of target objects in the support image and the query image, respectively, to obtain a support semantic graph and a query semantic graph; A mesh construction unit, configured to construct a 2.5D support mesh model of the target object based on the support depth map, the support image, and the support semantic map; an image rendering unit, configured to determine an initial relative pose from a plurality of uniformly sampled candidate relative poses, use the initial relative pose as a current relative pose, render the 2.5D support mesh model from the current relative pose to an image plane using a differentiable renderer, and obtain a rendered image and a rendered semantic map at the current relative pose; a posture optimization unit, configured to optimize the current relative posture to obtain a next relative posture based on the query image, the rendered image, the query semantic graph, and the rendered semantic graph; The posture determination unit is used to perform rendering and optimization in a loop using the next relative posture as the current relative posture until an iteration termination condition is reached, and determine the last relative posture as the three-dimensional relative posture of the target object in the query image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the three-dimensional relative pose estimation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional relative pose estimation method according to any one of claims 1 to 7 is implemented.

Images