Object six-dimensional pose detection method, device and electronic equipment

Abstract

The application provides a kind of object six-dimensional pose detection method, device and electronic equipment, the method comprises: calling three-dimensional model with metric scale feature, wherein the three-dimensional model with metric scale feature is the three-dimensional model obtained after metric scale alignment processing to normalized three-dimensional model, the normalized three-dimensional model is the three-dimensional model generated based on single anchor point image, and the anchor point image includes target object;Obtain query image, wherein the query image includes target object;Based on the three-dimensional model with metric scale feature, obtain the first six-dimensional pose of target object in query image;Obtain the second six-dimensional pose of target object in anchor point image;Based on the first six-dimensional pose and the second six-dimensional pose, obtain the relative pose of the target object in the query image relative to the object pose in the anchor point image.It is realized that under the condition of no pre-knowledge 3D model and only relying on single image, high efficiency, accurate 6D pose estimation can still be realized.

Description

Technical Field

[0001] This invention relates to the field of pose detection technology, and in particular to a method, apparatus and electronic device for six-dimensional pose detection of an object. Background Technology

[0002] In the fields of computer vision and robotics, accurate estimation of the six-dimensional pose (also known as 6D pose) of a target object is a core technical requirement for realizing applications such as human-computer interaction, autonomous robot operation, and augmented reality (AR).

[0003] Traditional 6D pose estimation methods typically rely on matching a known 3D model of the target object to calculate its pose parameters in the camera coordinate system, or on using stereo vision to obtain spatial geometric relationships from multi-view images to aid pose calculation. However, in practical industrial inspection, consumer AR, or unstructured scenarios, these conditions are often not met, meaning that a 3D model cannot be known in advance and only a single image is required.

[0004] Therefore, how to achieve efficient and accurate 6D pose estimation without a known 3D model and relying only on a single image has become a technical challenge that urgently needs to be solved in the field of computer vision. Summary of the Invention

[0005] This invention provides a method, apparatus, and electronic device for six-dimensional object pose detection, which enables efficient and accurate 6D pose estimation even without a known 3D model and relying only on a single image.

[0006] This invention provides a method for six-dimensional object pose detection. The method includes: calling a three-dimensional model with metric scale features, wherein the three-dimensional model with metric scale features is a three-dimensional model obtained by metric scale alignment processing of a normalized three-dimensional model, and the normalized three-dimensional model is a three-dimensional model generated based on a single anchor point image, wherein the anchor point image includes the target object; acquiring a query image, wherein the query image includes the target object; obtaining a first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features; acquiring a second six-dimensional pose of the target object in the anchor point image; and obtaining a relative pose of the target object in the query image relative to the object pose in the anchor point image based on the first six-dimensional pose and the second six-dimensional pose.

[0007] According to a method for six-dimensional pose detection of an object provided by the present invention, the three-dimensional model with metric scale features is obtained in the following manner: Based on camera intrinsic parameters, the normalized three-dimensional model is rendered from multiple spherical viewpoints to obtain multiple rendered views of the normalized three-dimensional model; feature point matching is performed on each rendered view and the anchor point image to obtain matching feature two-dimensional point pairs; based on the matching feature two-dimensional point pairs, matching feature three-dimensional point pairs corresponding to the matching feature two-dimensional point pairs are obtained; the scaling factor is optimized based on a loss function to obtain an optimized scaling factor, so as to minimize the function value of the loss function, wherein the function value of the loss function is determined based on the three-dimensional feature points under the rendered views after scaling according to the optimized scaling factor and the three-dimensional feature points under the anchor point image; based on the optimized scaling factor and the normalized three-dimensional model, the three-dimensional model with metric scale features is obtained.

[0008] According to a method for six-dimensional pose detection of an object provided by the present invention, after obtaining a pair of three-dimensional matching features corresponding to the two-dimensional matching feature pair, the method further includes: estimating the initial six-dimensional pose of the target object in the normalized three-dimensional model relative to the anchor point image based on the two-dimensional matching feature pair and the three-dimensional matching feature pair; scaling the translation vector in the initial six-dimensional pose based on the optimized scaling factor to obtain a preliminary six-dimensional pose estimate of the target object in the anchor point image; inputting the rendered image of the preliminary six-dimensional pose estimate and the anchor point image into a basic pose estimation network to obtain the change in displacement vector and the change in rotation matrix output by the basic pose estimation network; and obtaining the updated pose of the target object based on the change in displacement vector and the change in rotation matrix. The normalized 3D model is obtained, and the normalized 3D model updated with the pose of the target object is used as the normalized 3D model. The steps of obtaining multiple rendering views of the normalized 3D model are repeated until the steps of obtaining the displacement vector change and rotation matrix change output by the basic pose estimation network are obtained, until the number of iterations repeated meets the preset requirements, so as to obtain the displacement vector change and rotation matrix change. Based on the displacement vector change and rotation matrix change, the optimized scaling factor is updated to obtain the re-optimized scaling factor. The step of obtaining the 3D model with metric scale characteristics based on the optimized scaling factor and the normalized 3D model specifically includes: obtaining the 3D model with metric scale characteristics based on the re-optimized scaling factor and the normalized 3D model.

[0009] According to the present invention, a method for detecting a six-dimensional pose of an object is provided to obtain the second six-dimensional pose of the target object in the anchor point image. This is achieved by scaling the translation vector in the initial six-dimensional pose based on a re-optimized scaling factor to obtain the second six-dimensional pose of the target object in the anchor point image.

[0010] According to a six-dimensional pose detection method for an object provided by the present invention, the normalized three-dimensional model is obtained by: segmenting and cropping the anchor point image to obtain a processed anchor point image after removing noise from the anchor point image; obtaining a normal map corresponding to the processed anchor point image based on the processed anchor point image; and inputting the processed anchor point image and the normal map into a three-dimensional generation model to obtain the normalized three-dimensional model output by the three-dimensional generation model.

[0011] According to the present invention, a method for detecting a six-dimensional pose of an object is provided. The method for obtaining the first six-dimensional pose of the target object in a query image based on the three-dimensional model with metric scale features includes: calling a pre-trained pose estimation model, wherein the pose estimation model is used to estimate the six-dimensional pose; inputting the three-dimensional model with metric scale features and the query image into the pose estimation model to obtain the first six-dimensional pose of the target object in the query image output by the pose estimation model.

[0012] According to the present invention, a six-dimensional pose detection method for an object is provided, wherein the pose estimation model is pre-trained in the following manner: acquiring text prompt information, wherein the text prompt information is prompt information for multi-texture processing of a normalized three-dimensional model; inputting the normalized three-dimensional model and the text prompt information into a generative large model to obtain multiple normalized three-dimensional models with different texture features output by the generative large model; rendering the multiple normalized three-dimensional models with different texture features respectively to obtain multiple rendered images; using the rendered images as a training dataset to pre-train the pose estimation model to obtain a trained pose estimation model.

[0013] The present invention also provides a six-dimensional pose detection device for an object, the device comprising: a calling module for calling a three-dimensional model with metric scale features, wherein the three-dimensional model with metric scale features is a three-dimensional model obtained by metric scale alignment processing of a normalized three-dimensional model, the normalized three-dimensional model is a three-dimensional model generated based on a single anchor point image, the anchor point image including the target object; a first acquisition module for acquiring a query image, wherein the query image includes the target object; a processing module for obtaining a first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features; a second acquisition module for acquiring a second six-dimensional pose of the target object in the anchor point image; and a generation module for obtaining a relative pose of the target object in the query image relative to the object pose in the anchor point image based on the first six-dimensional pose and the second six-dimensional pose.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the six-dimensional pose detection method for an object as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the six-dimensional pose detection method for an object as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the object six-dimensional pose detection method as described above.

[0017] This invention provides a method, apparatus, and electronic device for six-dimensional object pose detection. The method includes: calling a three-dimensional model with metric scale features, wherein the three-dimensional model with metric scale features is a three-dimensional model obtained by metric scale alignment processing of a normalized three-dimensional model, and the normalized three-dimensional model is a three-dimensional model generated based on a single anchor point image, wherein the anchor point image includes a target object; acquiring a query image, wherein the query image includes the target object; obtaining a first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features; acquiring a second six-dimensional pose of the target object in the anchor point image; and obtaining a relative pose of the target object in the query image relative to the object pose in the anchor point image based on the first six-dimensional pose and the second six-dimensional pose. This achieves efficient and accurate 6D pose estimation even without a known 3D model and relying only on a single image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the six-dimensional pose detection method for objects provided by the present invention.

[0020] Figure 2 This is one of the flowcharts provided by the present invention for obtaining a three-dimensional model with metric scale characteristics.

[0021] Figure 3 This is the second schematic diagram of the process for obtaining a three-dimensional model with metric scale characteristics provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the process of training the pose estimation model provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the structure of the six-dimensional pose detection device for objects provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] To address the challenge of accurately estimating the 6D pose of a target object when no prior 3D model is available and only a single image is available, this application employs a technique that combines single-view 3D generation, coarse-to-fine alignment, and domain randomized dataset generation to achieve efficient and high-precision 6D pose estimation of novel objects in real-world scenes.

[0027] Figure 1 This is a flowchart illustrating the six-dimensional pose detection method for objects provided by the present invention.

[0028] The following will combine Figure 1 The process of the six-dimensional pose detection method for objects provided by the present invention will be described.

[0029] In an exemplary embodiment of the present invention, combined with Figure 1 As can be seen, the six-dimensional pose detection method for objects can include steps 110 to 150, and each step will be described below.

[0030] In step 110, a 3D model with metric scale features is invoked. The 3D model with metric scale features is a 3D model obtained by metric scale alignment processing of a normalized 3D model. The normalized 3D model is a 3D model generated based on a single anchor point image, and the anchor point image includes the target object.

[0031] In one embodiment, a three-dimensional model O with metric scale characteristics can be invoked. M It should be noted that the 3D model with metric scale characteristics is the 3D model obtained by metric scale alignment processing of the normalized 3D model. This alignment processing can include coarse alignment and / or fine alignment processing. The processes of coarse alignment and / or fine alignment processing will be explained below, using the normalized 3D model O as an example. N Perform metric scale alignment to ensure that the resulting 3D model O has metric scale characteristics. M It can match real objects in the anchor point image I A The scale and posture in the middle.

[0032] In yet another embodiment, the normalized 3D model can be based on a single anchor point image I. A The generated 3D model (also known as a 3D model) includes the target object in its anchor point image. An anchor point image can be understood as a key image used in fields such as computer vision, augmented reality (AR), and robot localization to establish a stable spatial reference point or coordinate system in a real-world scene. Essentially, it is a predefined 2D image with rich visual features (such as corners, textures, and patterns). The system identifies the position and pose (i.e., 6D pose) of this image and uses it as an "anchor point" to bind virtual content or calculate the spatial position of other objects. In this embodiment, a technique combining single-view 3D generation and coarse-to-fine alignment is employed to achieve efficient and high-precision 6D pose estimation of novel objects (such as target objects in a query image) in a real-world scene.

[0033] In step 120, a query image is obtained, wherein the query image includes the target object.

[0034] In step 130, based on the three-dimensional model with metric scale features, the first six-dimensional pose of the target object in the query image is obtained.

[0035] In one embodiment, the query image I can be obtained. QThe query image may also include the target object. In this embodiment, the target object can be adjusted according to the actual situation, and no specific limitation is made to the target object in this embodiment. The query image can be considered as the target image that needs to be analyzed or matched, which is input by the user or captured by the system in real time in a computer vision and image retrieval system.

[0036] In another embodiment, it can be based on a three-dimensional model O with metric scale characteristics. M Query image I is obtained Q The first six-dimensional pose of the target object .

[0037] In step 140, the second six-dimensional pose of the target object in the anchor point image is obtained.

[0038] In step 150, based on the first six-dimensional pose and the second six-dimensional pose, the relative pose of the object in the relative anchor point image of the target object in the query image is obtained.

[0039] In another embodiment, the second six-dimensional pose of the target object in the anchor point image can be obtained. Furthermore, the relative transformation between the anchor view and the query view is calculated using the estimated absolute pose. That is, based on the first six-dimensional pose and the second six-dimensional pose, the relative pose of the target object in the query image relative to the object pose in the anchor image is obtained. The relative pose of the target object in the query image relative to the object pose in the anchor image can be expressed by the following formula:

[0040]

[0041] in, This represents the relative pose of the target object in the relative anchor point image of the query image. In this embodiment, a high-quality 6D pose and scale estimation generation strategy can be achieved using only a single image (corresponding to the anchor point image). This invention effectively integrates the generative model into an integrated process of training and inference, realizing 3D object modeling and pose estimation based on a single image. This overcomes the limitation of traditional methods requiring multiple images, demonstrating the powerful performance of single-view 3D generation technology in 6D pose estimation tasks. Based on this invention, for deformable objects, it is possible to perform real-time model optimization during the 6D pose detection process, thereby achieving more accurate object pose estimation.

[0042] This invention provides a method for six-dimensional object pose detection. The method includes: calling a three-dimensional model with metric scale features, wherein the three-dimensional model with metric scale features is a three-dimensional model obtained by metric scale alignment processing of a normalized three-dimensional model, and the normalized three-dimensional model is a three-dimensional model generated based on a single anchor point image, wherein the anchor point image includes the target object; acquiring a query image, wherein the query image includes the target object; obtaining a first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features; acquiring a second six-dimensional pose of the target object in the anchor point image; and obtaining a relative pose of the target object in the query image relative to the object pose in the anchor point image based on the first six-dimensional pose and the second six-dimensional pose. This method achieves efficient and accurate 6D pose estimation even without a known 3D model and relying only on a single image.

[0043] Figure 2 This is one of the flowcharts provided by the present invention for obtaining a three-dimensional model with metric scale characteristics.

[0044] The following will combine Figure 2 The process of obtaining a three-dimensional model with metric scale characteristics is explained.

[0045] In an exemplary embodiment of the present invention, combined with Figure 2 As can be seen, obtaining a 3D model with metric scale characteristics can include steps 210 to 250, and each step will be described below.

[0046] In step 210, based on camera intrinsic parameters, a normalized 3D model is rendered from multiple spherical perspectives to obtain multiple rendered views of the normalized 3D model.

[0047] In step 220, feature point matching is performed on each rendered view and the anchor point image to obtain matching feature two-dimensional point pairs.

[0048] In one embodiment, during the coarse alignment stage, a normalized 3D model O can be rendered from n spherical viewpoints using camera intrinsics K. N Feature points are extracted using SuperPoint and passed through SuperGlue in each rendered view. i with I A Matching is performed between the points, and the view with the most matches is selected to obtain the matching feature 2D point pairs. Among them, the matched 2D point pairs (also known as matching feature 2D point pairs) (p' i ∈ R 2 , p i ∈ R 2 (From the rendered view and I respectively) A ).

[0049] In step 230, based on the matching feature two-dimensional point pairs, the matching feature three-dimensional point pairs corresponding to the matching feature two-dimensional point pairs are obtained.

[0050] In one embodiment, the data from the rendered view and I can be utilized. A The depth information is used to obtain 3D points (P') represented in normalized coordinate system and camera coordinate system. i ∈ R 3 , P i ∈ R 3 That is, based on the two-dimensional point pairs of matching features, we obtain the three-dimensional point pairs of matching features corresponding to the two-dimensional point pairs of matching features.

[0051] In yet another embodiment, the matching feature three-dimensional point pair corresponding to the matching feature two-dimensional point pair can be represented by the following formula:

[0052]

[0053] Where N represents the number of 3D point pairs matching the feature; This indicates the depth information of the rendered view; Indicate I A Depth information; K represents camera intrinsic parameters; This indicates the coordinates of the feature 3D points in the rendered view within the matched feature 3D point pair; In the context of matching feature 3D point pairs, I A The coordinates of the three-dimensional points in the feature; This indicates the coordinates of the feature 2D point in the rendered view within the matched feature 2D point pair; In the context of matching feature pairs of two-dimensional points, I A The coordinates of the two-dimensional feature points in the middle.

[0054] In step 240, the scaling factor is optimized based on the loss function to obtain the optimized scaling factor and the result, so as to minimize the function value of the loss function. The function value of the loss function is determined based on the 3D feature points in the rendered view and the 3D feature points in the anchor point image after scaling according to the optimized scaling factor.

[0055] In step 250, a three-dimensional model with metric scale characteristics is obtained based on the optimized scaling factor and the normalized three-dimensional model.

[0056] In yet another embodiment, it can be done at P' i and in image o i The corresponding 2D projection p iThe Perspective-n-Point (PnP) algorithm is applied to estimate the initial 6-DOF pose (i.e., the initial six-dimensional pose of the target object relative to the anchor point image in the normalized 3D model below), but scale ambiguity exists.

[0057] In another embodiment, the scaling factor (corresponding to the scaling factor) s can be jointly optimized to minimize the following L2 loss (corresponding to the loss function):

[0058]

[0059] Where R represents the rotation matrix; This represents the coordinates of the three-dimensional feature points in the rendered view after scaling according to the optimized scaling factor. This represents the coordinates of the 3D feature points in the anchor image. Based on the aforementioned process, the reprojection result can be preserved. Based on this optimization, the object's pose can be obtained. (Corresponding to the preliminary six-dimensional pose estimation of the target object described in the anchor point image below) and a rough estimate of the scaling factor s. That is, the scaling factor can be optimized based on the loss function to obtain the optimized scaling factor s, so as to minimize the function value of the loss function.

[0060] Furthermore, it is possible to base this on the optimized scaling factor s and the normalized 3D model O. N The three-dimensional model O with the metric scale feature is obtained. M For example, O M = s · O N .

[0061] Figure 3 This is the second schematic diagram of the process for obtaining a three-dimensional model with metric scale characteristics provided by the present invention.

[0062] The following will combine Figure 3 Another process for obtaining a 3D model with metric scale characteristics is described.

[0063] In an exemplary embodiment of the present invention, combined with Figure 3 As can be seen, after obtaining the matching feature three-dimensional point pairs corresponding to the matching feature two-dimensional point pairs, obtaining the three-dimensional model with metric scale features may further include steps 310 to 360, which will be described in detail below:

[0064] In step 310, based on the matching feature 2D point pairs and matching feature 3D point pairs, the initial six-dimensional pose of the target object relative to the anchor point image in the normalized 3D model is estimated.

[0065] In step 320, the translation vector in the initial six-dimensional pose is scaled based on the optimized scaling factor to obtain a preliminary six-dimensional pose estimate of the target object in the anchor point image.

[0066] In one embodiment, it can be done at P' i and in image o i The corresponding 2D projection p i The Perspective-n-Point (PnP) algorithm is used to estimate the initial 6-DOF pose, but scale ambiguity exists. In other words, the PnP algorithm is used to estimate the initial 6-DOF pose based on these 3D-2D point pairs, that is, based on matching feature 2D point pairs and matching feature 3D point pairs, the initial six-dimensional pose of the target object relative to the anchor point image in the normalized 3D model is estimated.

[0067] Furthermore, based on the optimized scaling factor, the translation vector in the initial six-dimensional pose is scaled to obtain a preliminary six-dimensional pose estimate of the target object in the anchor point image. This yields a preliminary object pose estimate. .

[0068] In step 330, the rendering image and anchor point image of the preliminary six-dimensional pose estimation are input into the basic pose estimation network to obtain the change in displacement vector and the change in rotation matrix output by the basic pose estimation network.

[0069] In step 340, based on the changes in the displacement vector and the changes in the rotation matrix, a normalized 3D model of the target object after pose update is obtained. The normalized 3D model of the target object after pose update is used as the normalized 3D model. The steps of obtaining multiple rendering views of the normalized 3D model are repeated until the changes in the displacement vector and the changes in the rotation matrix output by the basic pose estimation network are obtained. The number of iterations repeated is until the preset requirement is met, so as to obtain the changes in the displacement vector and the changes in the rotation matrix.

[0070] In step 350, the optimized scaling factor is updated based on the change in displacement vector and the change in rotation matrix to obtain the re-optimized scaling factor.

[0071] In step 360, a three-dimensional model with metric scale characteristics is obtained based on the re-optimized scaling factor and the normalized three-dimensional model.

[0072] It should be noted that although a rough alignment result has been obtained, further refinement is still needed due to feature noise and model imperfections.

[0073] In one embodiment, the estimate can be based on a coarse alignment. Given the scale s, a network derived from FoundationPose is used to predict the changes in pose, updating ΔR ∈ SO(3) and Δt. In application, the rendered image of the initial six-dimensional pose estimation and the anchor point image can be input into the basic pose estimation network to obtain the changes in displacement vector Δt and rotation matrix ΔR output by the basic pose estimation network.

[0074] Furthermore, the attitude update formula can be expressed as follows:

[0075] t = t + Δt,

[0076]

[0077] In another embodiment, after each pose update, the coarse-stage feature matching and scale optimization can be rerun using the current pose estimate to jointly refine the scale *s* and translation *t*. This iterative process alternates between pose refinement and scale / translation optimization until convergence or the maximum number of iterations is reached. This produces a 3D model O with a metric scale. M =s · O N and its relative to the anchor point image I A Precise 6D pose of the camera frame That is, based on the changes in the displacement vector and the changes in the rotation matrix, a normalized 3D model of the target object after pose update can be obtained. The normalized 3D model of the target object after pose update is used as the normalized 3D model. The steps of obtaining multiple rendering views of the normalized 3D model are repeated until the changes in the displacement vector and the changes in the rotation matrix output by the basic pose estimation network are obtained. This process is repeated until the number of iterations meets the preset requirements to obtain the changes in the displacement vector and the changes in the rotation matrix.

[0078] Furthermore, based on the changes in the displacement vector and the rotation matrix, the optimized scaling factor is updated, thus obtaining a further optimized scaling factor. In application, the further optimized scaling factor and the normalized 3D model can be used to obtain a 3D model with metric scale characteristics, i.e., O... M = s · O N It should be noted that 's' here refers to the scaling factor after optimization.

[0079] In this embodiment, coarse alignment provides a reasonable initial pose and scale, but errors caused by feature noise and model defects require further refinement. We employ a strategy that combines rendering with comparative refinement and scale optimization based on coarse alignment to iteratively refine the pose and scale, thereby obtaining a more accurate 3D model O with metric scale characteristics. M This lays the foundation for obtaining the accurate six-dimensional pose of the target object in the query image.

[0080] In yet another exemplary embodiment of the present invention, the description continues with the previously described embodiments, wherein the second six-dimensional pose of the target object in the anchor point image is obtained. This can be achieved in the following ways:

[0081] Based on the re-optimized scaling factor, the translation vector in the initial six-dimensional pose is scaled to obtain the second six-dimensional pose of the target object in the anchor point image.

[0082] In this embodiment, since the re-optimized scaling factor s is the result of coarse and fine alignment iterations, the rationality and accuracy of the re-optimized scaling factor s can be effectively ensured, thereby enabling a more accurate second six-dimensional pose of the target object in the anchor point image. .

[0083] In an exemplary embodiment of the present invention, the normalized three-dimensional model can be obtained in the following manner:

[0084] The anchor point image is segmented and cropped to obtain a processed anchor point image after removing noise.

[0085] Based on the processed anchor point image, a normal map corresponding to the processed anchor point image is obtained;

[0086] The processed anchor point image and normal map are input into the 3D generation model to obtain the normalized 3D model output by the 3D generation model.

[0087] In one embodiment, segmentation techniques can be used to crop the input anchor image to minimize the impact of background noise on the model generation process, thereby obtaining a processed anchor image with noise removed from the anchor image.

[0088] In another embodiment, the processed image (corresponding to the processed anchor point image) I can be used. A cropped An image is input into the normal estimation module, which generates a normal map X, that is, the normal map corresponding to the processed anchor point image.

[0089] Furthermore, the normal map X and the original cropped image I are then compared. A cropped The input is fed into the 3D generative model (corresponding to the three-dimensional generative model), thereby obtaining the normalized three-dimensional model O output by the three-dimensional generative model. N This process ensures geometric consistency and visual detail. Through this embodiment, a standardized texture object model O is obtained. N The model is located in a coordinate system centered on the object. This model O NThis will serve as the basis for subsequent alignment and resizing steps to match real-world dimensions.

[0090] In yet another exemplary embodiment of the present invention, the first six-dimensional pose of the target object in the query image, based on a three-dimensional model with metric scale characteristics, can be obtained in the following manner:

[0091] Call the pre-trained pose estimation model, which is used to estimate the six-dimensional pose;

[0092] The three-dimensional model with metric scale features and the query image are input into the pose estimation model to obtain the first six-dimensional pose of the target object in the query image output by the pose estimation model.

[0093] In one embodiment, a pre-trained pose estimation model can be invoked, and further, a 3D model O with metric scale features can be used. M By inputting the query image into the pose estimation model, the first six-dimensional pose of the target object in the query image can be obtained from the output of the pose estimation model. .

[0094] For query image I Q In attitude estimation, the derived recovery scale model O is used. M This invention employs a rendering and comparison strategy similar to that used in fine alignment. To handle the symmetry of objects, a pose selection module is introduced, which selects the most reasonable pose from the available symmetrical poses. Relative to query image I Q The 6D pose obtained from the camera coordinate system is denoted as .

[0095] Figure 4 This is a schematic diagram of the process of training the pose estimation model provided by the present invention.

[0096] The following will combine Figure 4 The process of training the pose estimation model provided by this invention will be explained.

[0097] In an exemplary embodiment of the present invention, combined with Figure 4 As can be seen, training the pose estimation model can include steps 410 to 440, which will be described in detail below.

[0098] In step 410, text prompt information is obtained, wherein the text prompt information is a prompt information for multi-texture processing of the normalized 3D model.

[0099] In step 420, the normalized 3D model and text prompt information are input into the generative large model to obtain multiple normalized 3D models with different texture features output by the generative large model.

[0100] In one embodiment, to avoid generating a normalized 3D model O N It can capture shape and initial texture, but only represents a single instance. It lacks the appearance and geometric variations present in real-world objects, limiting its generalization ability under various environmental conditions (such as lighting and occlusion). It can be used to train the mechanical energy robustness of the pose estimation model.

[0101] However, training a robust pose estimator requires a large and diverse dataset, and manually collecting such datasets is costly. To bridge the gap between simulation and reality and achieve effective training, this invention proposes generating a diverse set of object texture variants.

[0102] During application, text prompts are acquired; and the normalized 3D model and text prompts are input into a generative large model, resulting in multiple normalized 3D models with different texture features output by the generative large model. In one example, the text-guided 3D generative model Trellis can be used to generate the normalized 3D model O. N With text prompts as input, the model can generate variants with diverse textures, styles, or slight geometric changes while keeping the core structure unchanged.

[0103] In step 430, multiple normalized 3D models with different texture features are rendered to obtain multiple rendered images.

[0104] In step 440, the rendered image is used as a training dataset to pre-train the pose estimation model, resulting in a trained pose estimation model.

[0105] In another embodiment, these models can be rendered using Blender under random viewpoints, lighting, background, and occlusion conditions to form a large-scale synthetic dataset for fine-tuning rendering and comparing refined networks. This significantly improves pose estimation performance. This strategy demonstrates the ability to quickly adapt to new objects using only a single reference image, proving the practical value of generative models in data-efficient 6D pose estimation. Specifically, multiple normalized 3D models with different texture features are rendered separately to obtain multiple rendered images. These rendered images are then used as a training dataset to pre-train the pose estimation model, resulting in a trained pose estimation model.

[0106] In this embodiment, a text-guided generative enhancement method is used to generate diverse 3D models. A large-scale, domain-randomized training dataset is constructed using Blender rendering, effectively reducing the domain difference between the training dataset and reality, enabling robust fine-tuning of the pose estimation model. By utilizing the generative model to randomize object surface textures and constructing a dataset containing multiple poses and occlusion relationships, the domain distance between the training dataset and real data is reduced, thereby achieving better fine-tuning results.

[0107] As described above, the object six-dimensional pose detection method provided by this invention can achieve high-quality 6D pose and scale estimation generation strategy with only a single image. This invention proposes an integrated process that effectively fuses the generative model into the training and inference stages, realizing 3D object modeling and pose estimation based on a single image. This overcomes the limitation of traditional methods requiring multiple images, and experiments have demonstrated the powerful performance of single-view 3D generation technology in 6D pose estimation tasks.

[0108] In addition, to compensate for the inherent gap between the scale of generative 3D models and real objects, a coarse-to-fine alignment module is proposed. It does not require prior information about the camera's pose relative to the object. It jointly estimates the scale and pose through 2D-3D feature matching and performs accurate alignment and pose estimation, which can restore the true size and pose of the object.

[0109] Furthermore, a text-guided generative enhancement method is employed to generate diverse 3D models, and a large-scale, domain-randomized training dataset is constructed using Blender rendering. This effectively reduces the domain difference between the training dataset and reality, enabling robust fine-tuning of the pose estimation model. By randomizing object surface textures using generative models and constructing a dataset containing various poses and occlusion relationships, the domain distance between the training dataset and real data is reduced, resulting in better fine-tuning performance.

[0110] The object six-dimensional pose detection device provided by the present invention is described below. The object six-dimensional pose detection device described below can be referred to in correspondence with the object six-dimensional pose detection method described above.

[0111] Figure 5 This is a schematic diagram of the structure of the six-dimensional pose detection device for objects provided by the present invention.

[0112] The following will combine Figure 5 The structure of the six-dimensional pose detection device for objects provided by the present invention will be described.

[0113] In an exemplary embodiment of the present invention, combined with Figure 5As can be seen, the object six-dimensional pose detection device may include a calling module 510, a first acquisition module 520, a processing module 530, a second acquisition module 540, and a generation module 550. Each module will be described in detail below.

[0114] The calling module 510 can be configured to call a three-dimensional model with metric scale features, wherein the three-dimensional model with metric scale features is a three-dimensional model obtained by metric scale alignment processing of a normalized three-dimensional model, and the normalized three-dimensional model is a three-dimensional model generated based on a single anchor point image, wherein the anchor point image includes the target object.

[0115] The first acquisition module 520 can be configured to acquire a query image, wherein the query image includes the target object;

[0116] The processing module 530 can be configured to obtain the first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features.

[0117] The second acquisition module 540 can be configured to acquire the second six-dimensional pose of the target object in the anchor point image;

[0118] The generation module 550 can be configured to obtain the relative pose of the object in the relative anchor point image of the target object in the query image based on the first six-dimensional pose and the second six-dimensional pose.

[0119] In an exemplary embodiment of the present invention, the calling module 510 can obtain a three-dimensional model with metric scale characteristics in the following manner:

[0120] Based on camera intrinsics, the normalized 3D model is rendered from multiple spherical perspectives to obtain multiple rendered views of the normalized 3D model.

[0121] Each rendered view is matched with the anchor point image to obtain a pair of matching feature points in two dimensions.

[0122] Based on the two-dimensional point pairs of matching features, a three-dimensional point pair of matching features corresponding to the two-dimensional point pairs of matching features is obtained;

[0123] The scaling factor is optimized based on the loss function to obtain an optimized scaling factor, so as to minimize the function value of the loss function. The function value of the loss function is determined based on the three-dimensional feature points in the rendered view and the three-dimensional feature points in the anchor point image after scaling according to the optimized scaling factor.

[0124] Based on the optimized scaling factor and the normalized 3D model, the 3D model with metric scale characteristics is obtained.

[0125] In an exemplary embodiment of the present invention, the calling module 510 may further be configured to:

[0126] Based on the matching feature two-dimensional point pairs and the matching feature three-dimensional point pairs, the initial six-dimensional pose of the target object relative to the anchor point image in the normalized three-dimensional model is estimated;

[0127] Based on the optimized scaling factor, the translation vector in the initial six-dimensional pose is scaled to obtain a preliminary six-dimensional pose estimate of the target object in the anchor point image;

[0128] The rendering of the preliminary six-dimensional pose estimation and the anchor point image are input into the basic pose estimation network to obtain the change in displacement vector and the change in rotation matrix output by the basic pose estimation network.

[0129] Based on the change in displacement vector and the change in rotation matrix, a normalized 3D model of the target object after pose update is obtained. The normalized 3D model of the target object after pose update is used as the normalized 3D model. The steps of obtaining multiple rendering views of the normalized 3D model are repeated until the change in displacement vector and the change in rotation matrix output by the basic pose estimation network are obtained. The number of iterations repeated is until the preset requirement is met, so as to obtain the change in displacement vector and the change in rotation matrix.

[0130] Based on the change in the displacement vector and the change in the rotation matrix, the optimized scaling factor is updated to obtain the further optimized scaling factor.

[0131] The calling module 510 can obtain the 3D model with metric scale characteristics based on the optimized scaling factor and the normalized 3D model in the following way:

[0132] Based on the re-optimized scaling factor and the normalized 3D model, the 3D model with metric scale characteristics is obtained.

[0133] In an exemplary embodiment of the present invention, the second acquisition module 540 may acquire the second six-dimensional pose of the target object in the anchor point image in the following manner:

[0134] Based on the re-optimized scaling factor, the translation vector in the initial six-dimensional pose is scaled to obtain the second six-dimensional pose of the target object in the anchor point image.

[0135] In an exemplary embodiment of the present invention, the calling module 510 can obtain the normalized three-dimensional model in the following manner:

[0136] The anchor point image is segmented and cropped to obtain a processed anchor point image after removing noise from the anchor point image;

[0137] Based on the processed anchor point image, a normal map corresponding to the processed anchor point image is obtained;

[0138] The processed anchor point image and the normal map are input into the 3D generation model to obtain the normalized 3D model output by the 3D generation model.

[0139] In an exemplary embodiment of the present invention, the processing module 530 may obtain the first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features in the following manner:

[0140] The pre-trained pose estimation model is invoked, wherein the pose estimation model is used to estimate the six-dimensional pose;

[0141] The three-dimensional model with metric scale features and the query image are input into the pose estimation model to obtain the first six-dimensional pose of the target object in the query image output by the pose estimation model.

[0142] In an exemplary embodiment of the present invention, the processing module 530 may obtain the attitude estimation model in the following manner:

[0143] Obtain text prompt information, wherein the text prompt information is a prompt information for multi-texture processing of the normalized 3D model;

[0144] The normalized 3D model and the text prompt information are input into the generative large model to obtain multiple normalized 3D models with different texture features output by the generative large model.

[0145] Multiple normalized 3D models with different texture features were rendered separately to obtain multiple rendered images.

[0146] The rendered image is used as a training dataset to pre-train the pose estimation model, resulting in a trained pose estimation model.

[0147] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communications bus 640. The processor 610 can call logical instructions in the memory 630 to execute a six-dimensional object pose detection method. This method includes: calling a three-dimensional model with metric scale features, wherein the three-dimensional model with metric scale features is a three-dimensional model obtained after metric scale alignment processing of a normalized three-dimensional model, and the normalized three-dimensional model is a three-dimensional model generated based on a single anchor point image, the anchor point image including the target object; acquiring a query image, wherein the query image includes the target object; obtaining a first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features; acquiring a second six-dimensional pose of the target object in the anchor point image; and obtaining a relative pose of the target object in the query image relative to the object pose in the anchor point image based on the first six-dimensional pose and the second six-dimensional pose.

[0148] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the object six-dimensional pose detection method provided by the above methods. The method includes: calling a three-dimensional model with metric scale features, wherein the three-dimensional model with metric scale features is a three-dimensional model obtained after metric scale alignment processing of a normalized three-dimensional model, and the normalized three-dimensional model is a three-dimensional model generated based on a single anchor point image, wherein the anchor point image includes a target object; obtaining a query image, wherein the query image includes the target object; obtaining a first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features; obtaining a second six-dimensional pose of the target object in the anchor point image; and obtaining a relative pose of the target object in the query image relative to the object pose in the anchor point image based on the first six-dimensional pose and the second six-dimensional pose.

[0150] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the object six-dimensional pose detection method provided by the above methods. The method includes: calling a three-dimensional model with metric scale features, wherein the three-dimensional model with metric scale features is a three-dimensional model obtained after metric scale alignment processing of a normalized three-dimensional model, the normalized three-dimensional model being a three-dimensional model generated based on a single anchor point image, the anchor point image including a target object; acquiring a query image, wherein the query image includes the target object; obtaining a first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features; acquiring a second six-dimensional pose of the target object in the anchor point image; and obtaining a relative pose of the target object in the query image relative to the object pose in the anchor point image based on the first six-dimensional pose and the second six-dimensional pose.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for six-dimensional pose detection of an object, characterized in that, The method includes: A 3D model with metric scale features is invoked, wherein the 3D model with metric scale features is a 3D model obtained by metric scale alignment processing of a normalized 3D model, and the normalized 3D model is a 3D model generated based on a single anchor point image, wherein the anchor point image includes the target object. Obtain a query image, wherein the query image includes the target object; Based on the three-dimensional model with metric scale features, the first six-dimensional pose of the target object in the query image is obtained; Obtain the second six-dimensional pose of the target object in the anchor point image; Based on the first six-dimensional pose and the second six-dimensional pose, the relative pose of the object in the relative anchor point image of the target object in the query image is obtained, wherein the three-dimensional model with metric scale features is obtained in the following way: Based on camera intrinsics, the normalized 3D model is rendered from multiple spherical perspectives to obtain multiple rendered views of the normalized 3D model. Each rendered view is matched with the anchor point image to obtain a pair of matching feature points in two dimensions. Based on the two-dimensional point pairs of matching features, a three-dimensional point pair of matching features corresponding to the two-dimensional point pairs of matching features is obtained; The scaling factor is optimized based on the loss function to obtain an optimized scaling factor, so as to minimize the function value of the loss function. The function value of the loss function is determined based on the three-dimensional feature points in the rendered view and the three-dimensional feature points in the anchor point image after scaling according to the optimized scaling factor. Based on the optimized scaling factor and the normalized 3D model, the 3D model with metric scale characteristics is obtained, wherein... After obtaining the matching feature three-dimensional point pair corresponding to the matching feature two-dimensional point pair, the method further includes: Based on the matching feature two-dimensional point pairs and the matching feature three-dimensional point pairs, the initial six-dimensional pose of the target object relative to the anchor point image in the normalized three-dimensional model is estimated; Based on the optimized scaling factor, the translation vector in the initial six-dimensional pose is scaled to obtain a preliminary six-dimensional pose estimate of the target object in the anchor point image; The rendering of the preliminary six-dimensional pose estimation and the anchor point image are input into the basic pose estimation network to obtain the change in displacement vector and the change in rotation matrix output by the basic pose estimation network. Based on the change in displacement vector and the change in rotation matrix, a normalized 3D model of the target object after pose update is obtained. The normalized 3D model of the target object after pose update is used as the normalized 3D model. The steps of obtaining multiple rendering views of the normalized 3D model are repeated until the change in displacement vector and the change in rotation matrix output by the basic pose estimation network are obtained. The number of iterations repeated is until the preset requirement is met, so as to obtain the change in displacement vector and the change in rotation matrix. Based on the change in the displacement vector and the change in the rotation matrix, the optimized scaling factor is updated to obtain the further optimized scaling factor. The process of obtaining the 3D model with metric scale characteristics based on the optimized scaling factor and the normalized 3D model specifically includes: Based on the re-optimized scaling factor and the normalized 3D model, the 3D model with metric scale characteristics is obtained.

2. The six-dimensional pose detection method for an object according to claim 1, characterized in that, The second six-dimensional pose of the target object in the anchor point image is obtained in the following way: Based on the re-optimized scaling factor, the translation vector in the initial six-dimensional pose is scaled to obtain the second six-dimensional pose of the target object in the anchor point image.

3. The object six-dimensional pose detection method according to any one of claims 1 to 2, characterized in that, The normalized 3D model was obtained in the following way: The anchor point image is segmented and cropped to obtain a processed anchor point image after removing noise from the anchor point image; Based on the processed anchor point image, a normal map corresponding to the processed anchor point image is obtained; The processed anchor point image and the normal map are input into the 3D generation model to obtain the normalized 3D model output by the 3D generation model.

4. The object six-dimensional pose detection method according to any one of claims 1 to 2, characterized in that, The process of obtaining the first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features specifically includes: The pre-trained pose estimation model is invoked, wherein the pose estimation model is used to estimate the six-dimensional pose; The three-dimensional model with metric scale features and the query image are input into the pose estimation model to obtain the first six-dimensional pose of the target object in the query image output by the pose estimation model.

5. The six-dimensional pose detection method for an object according to claim 4, characterized in that, The attitude estimation model is pre-trained using the following method: Obtain text prompt information, wherein the text prompt information is a prompt information for multi-texture processing of the normalized 3D model; The normalized 3D model and the text prompt information are input into the generative large model to obtain multiple normalized 3D models with different texture features output by the generative large model. Multiple normalized 3D models with different texture features were rendered separately to obtain multiple rendered images. The rendered image is used as a training dataset to pre-train the pose estimation model, resulting in a trained pose estimation model.

6. A six-dimensional pose detection device for an object, characterized in that, The apparatus is used to implement the six-dimensional pose detection method for an object as described in any one of claims 1 to 5, and the apparatus comprises: The calling module is used to call a 3D model with metric scale characteristics, wherein the 3D model with metric scale characteristics is a 3D model obtained by metric scale alignment processing of a normalized 3D model, and the normalized 3D model is a 3D model generated based on a single anchor point image, wherein the anchor point image includes the target object. The first acquisition module is used to acquire a query image, wherein the query image includes the target object; The processing module is used to obtain the first six-dimensional pose of the target object in the query image based on the three-dimensional model with metric scale features; The second acquisition module is used to acquire the second six-dimensional pose of the target object in the anchor point image; The generation module is used to obtain the relative pose of the object in the relative anchor point image of the target object in the query image based on the first six-dimensional pose and the second six-dimensional pose.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the six-dimensional pose detection method for an object as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the six-dimensional pose detection method for objects as described in any one of claims 1 to 5.

Images