Single-view unknown object 6D pose estimation method based on segmentation and new view angle synthesis

Abstract

The invention discloses a single-view unknown object 6D pose estimation method based on segmentation and new view angle synthesis, and relates to the field of object pose estimation. The method comprises the following steps: designing a segmentation module and a new view angle synthesis module of a pose estimation network, and obtaining a segmentation result of a target object by using the segmentation module; generating a target object multi-view-angle embedding template by using a new view angle synthesis module; matching the embedding representation zquery'of the segmentation result with a multi-view embedding template of the target object to find out the embedding representation most similar to zquery ', and determining a preliminary rotation attitude estimation result R of the target object; and estimating the relative translation vector t of the target object, and combining the R to obtain the relative pose (R, t) of the target object. According to the method, robust 6D pose estimation of the unknown object in a complex visual scene can be realized only by taking a single reference image as input, and the pose estimation precision and robustness of the unknown object in the complex scene are improved while the real-time requirement is met.

Description

Technical Field

[0001] The present invention relates to the field of object pose estimation, and in particular to a 6D pose estimation method for an unknown object in a single view based on segmentation and new perspective synthesis. Background Art

[0002] Object pose estimation technology aims to determine the 6D pose information of a target object in the camera coordinate system, namely its 3D translation vector and 3D rotation matrix. This technology is a key enabler for environmental perception and spatial understanding and is widely used in fields such as autonomous driving, robotics, and augmented reality. This technology typically uses images, point clouds, or multimodal data to accurately restore the pose of an object in 3D space. Existing pose estimation methods can be categorized into instance-level pose estimation, category-level pose estimation, and unknown object pose estimation, depending on the completeness of the object's prior information. Instance-level methods rely on detailed 3D models of the target object and are suitable for the precise localization of specific objects. Category-level methods target object categories with common structural properties and offer stronger generalization capabilities. However, in real-world applications, systems often encounter unknown objects for which a 3D model or a large number of labeled samples are unavailable. Therefore, the task of unknown object pose estimation has emerged.

[0003] Estimating the 6D pose of an unknown object from a single view is a challenging task, particularly important in resource-constrained applications or those with high real-time requirements. The lack of depth information in a single image makes estimating the spatial position of an object inherently uncertain. Furthermore, background interference, occlusions, and the similarity between objects in complex scenes further exacerbate the difficulty of pose estimation. Furthermore, traditional methods often lack generalization capabilities when faced with unknown objects, making them difficult to adapt to the needs of identifying new objects in open environments.

[0004] To address these challenges, researchers have proposed various approaches. For example, combining deep learning techniques with geometric constraints can effectively improve the accuracy of pose estimation. This approach enhances the model's understanding of complex scenes by learning the correspondence between image feature points and 3D models. Furthermore, training with synthetic data can significantly improve the model's adaptability to diverse objects and scenes, thereby enhancing its generalization capabilities. However, existing methods still have limitations. In particular, when dealing with complex scenes and unknown objects, estimation accuracy and computational efficiency still need to be further improved. Summary of the Invention

[0005] In response to the shortcomings of existing technologies in processing the 6D pose estimation task of unknown objects, the present invention provides a single-view 6D pose estimation method for unknown objects based on segmentation and new perspective synthesis, aiming to meet real-time requirements while improving the accuracy and robustness of pose estimation of unknown objects in complex scenes.

[0006] The technical solution of the present invention is:

[0007] A 6D pose estimation method for an unknown object in a single view based on segmentation and novel perspective synthesis comprises the following steps:

[0008] Step 1: Design a segmentation module for the pose estimation network and use it to obtain the segmentation result of the target object I query’ ;

[0009] Step 2: Design a new perspective synthesis module for the pose estimation network and use it to generate a multi-perspective embedding template for the target object.

[0010] Step 3: Use template matching method to segment the result I query’ The embedding representation z query’ Match with the target object multi-view embedding template to find the z query’ The most similar embedding representation determines the preliminary rotation pose estimation result R of the target object;

[0011] Step 4: Estimate the relative translation vector t of the target object, and combine it with the preliminary rotation pose estimation result R of the target object obtained in step 4 to obtain the relative pose (R, t) of the target object.

[0012] Furthermore, according to the single-view unknown object 6D pose estimation method, step 1 includes the following steps:

[0013] Step 1.1: Input reference image I ref and query image I query , using the suggestible segmentation model from the query image I query Extract and reference image I ref The target object is matched with several candidate masks that may be the target object’s area, and a candidate mask set M = {m 1 ,m 2 ,...,m j}; Each mask m j Represents a binary region in the query image where the target object may exist, where the image region corresponding to the pixel with value 1 is the mask region that matches the target object in the reference image;

[0014] Step 1.2: Select the top-S candidate masks from the candidate mask set M and preliminarily determine the S candidate regions in the query image where the target object corresponding to the reference image is located;

[0015] Step 1.3: For each image pair formed by combining the image area corresponding to each mask in the Top-S candidate mask with the reference image, use the Efficient LoFTR model to establish the local feature correspondence between the reference image and the image area, so that each image pair obtains n key point matching pairs consisting of the key points in the reference image and the key points matching the key points in the image area, calculate the confidence of each key point matching pair, and filter out the key point matching pairs with low confidence in each image pair, and then solve the average of the confidence of the remaining key point matching pairs as the value of the matching quality index of each image pair, and finally use the image area in the image pair corresponding to the largest matching quality index value as the segmentation result I of the target object. query '.

[0016] Furthermore, according to the single-view unknown object 6D pose estimation method, the method of selecting the Top-S candidate masks from the candidate mask set M in step 1.2 is: using the pre-trained DINO-v2 model to extract the reference image I ref The class token feature CLSP and each mask m in M j The category token feature CLST(j) of the reference image I ref The category token feature CLSP is respectively associated with each mask m in M j The cosine similarity of the category token feature CLST(j) is used to construct a 1×j-dimensional similarity matrix S(P,T,j)=<CLSP,CLST(j)> , select the masks corresponding to the S highest similarity values and get the Top-S candidate masks.

[0017] Furthermore, according to the single-view unknown object 6D pose estimation method, step 2 includes the following steps:

[0018] Step 2.1: Load the variational autoencoder VAE, conditional encoder, U-Net and MLP modules in the pre-trained latent diffusion model LDM to establish a new perspective synthesis module of the pose estimation network, and use the conditional encoder in the new perspective synthesis module to generate the conditional embedding representation of the relative rotation posture change ΔR Where C is consistent with the temporal embedding dimension of the attention module in U-Net; the relative rotational pose change ΔR is the change in the rotational pose of the target object in the query image view relative to its rotational pose in the reference image view;

[0019] Step 2.2: Initialize the multi-layer perceptron MLP in the new perspective synthesis module using Xavier uniform distribution: z pose Input into MLP and set MLP to "single layer" mode, that is, use only one linear layer to transform z pose Project to an embedding vector of the specified dimension;

[0020] Step 2.3: Input the reference image I in step 1.1 ref , the segmentation result I obtained in step 1.3 query’ and all possible N relative rotation attitude changes ΔR1, ΔR2, ..., ΔR N Input the new perspective synthesis module together;

[0021] Step 2.4: Input image data, i.e. reference image I ref And the segmentation result I query’ Normalize the image and then generate the reference image I through the variational autoencoder VAE ref The embedding representation z ref And the segmentation result I query’ The embedding representation z query’ ;

[0022] Step 2.5: Based on the input reference image I ref The embedding representation z ref With each relative rotation attitude change ΔR i (i=1,2,...,N), the new perspective synthesis module outputs the corresponding template embedding representation set (z1,z2,...,z N ) as the target object multi-view embedding template, where z i (i=1,2,...,N) represents the relative rotation change ΔR of the reference image i The embedding representation under the template embedding representation set (z1,z2,...,z N ) is generated as follows:

[0023] Based on the pose ambiguity, the possible solutions of ΔR are modeled as the reference images I ref and query image I query A Gaussian distribution determined jointly by:

[0024] p(ΔR)=argmaxN(z query’ |μ=z(z ref ,ΔR),∑=∑(z ref ,ΔR)) (4)

[0025] Where p() represents the probability of obtaining the correct solution of ΔR; N() represents the Gaussian distribution; the mean μ of the Gaussian distribution is the embedded representation z(z ref ,ΔR); the covariance ∑ of the Gaussian distribution is ∑(z ref ,ΔR); the segmentation result is embedded in z query’ is considered as an observation of this Gaussian distribution;

[0026] z(z ref ,ΔR)=∫ M z(ΔR,M)p(M|z ref )dM (5)

[0027] Where M is the 3D shape distribution of the target object; z(ΔR,M) represents the embedding representation generated based on ΔR and M; p(M|z ref ) is embedded in z for a given reference image ref After that, the posterior probability distribution of the target object shape;

[0028] Construct the training loss function of the pose estimation network shown in formula (6) to approximate the process expressed in formula (5):

[0029]

[0030] Where F is the new perspective synthesis module, z t1 is the embedded representation of the reference image during the training of the pose estimation network; z t2 It is the embedded representation of the target object under ΔR during the training process of the pose estimation network;

[0031] Finally, the parameters of the new perspective synthesis module are optimized by minimizing the loss function shown in formula (6), and then the new perspective synthesis module is used to ref and N relative rotation attitude changes ΔR1, ΔR2, ..., ΔR N Generate template embedding representation set (z1,z2,...,z N ).

[0032] Furthermore, according to the single-view unknown object 6D pose estimation method, the training loss function of the pose estimation network shown in formula (6) is L2 loss.

[0033] Furthermore, according to the single-view unknown object 6D pose estimation method, in step 2.3, all possible N relative rotation pose changes ΔR1, ΔR2, ..., ΔR are defined based on the icosahedron recursive subdivision strategy. N .

[0034] Furthermore, according to the single-view unknown object 6D pose estimation method, in step 3, in order to find the z-axis from the target object multi-view embedding template, query’ The most similar embedding representation, the pose estimation network uses the following method to retrieve the multi-view embedding template of the target object:

[0035] First, z query’ Expands to a template embedded set (z1,z2,...,z N ) where a single embedding represents a tensor with the same dimensions and computes the squared Euclidean distance:

[0036] d i =||z query’ -z i || 2 , i=1...N (7)

[0037] where d i represents z query’ Embedded with the i-th template z i degree of difference;

[0038] Then, by obtaining the template embedding set (z1,z2,...,z N ) in each embedding representation and z query’ The similarity score s i :

[0039] s i =-∑d i (h,w) (8)

[0040] Where h and w represent d i Dimensions;

[0041] Then, all similarity scores are sorted, and the A relative rotation posture changes corresponding to the top-A higher similarity scores are selected as the solutions of the initial rotation posture;

[0042] Finally, the template embedding corresponding to the highest similarity score is found from the solution of the initial rotation posture, and the corresponding relative rotation posture change is taken as the preliminary rotation posture estimation result R of the target object. Specifically, R is expressed as x i Relative to x i 'The rotation matrix, x i represents the key points in the reference image, x i 'Represents the segmentation result I query’ Zhong and x i The corresponding key points.

[0043] Furthermore, according to the single-view unknown object 6D pose estimation method, step 4 includes the following steps:

[0044] Step 4.1: Based on the reference image I ref and segmentation result I query’ The key point matching pairs in the ,use the five-point method to obtain the essential matrix;

[0045] Step 4.2: Recover the relative translation vector t of the target object from the essential matrix, and combine it with the preliminary rotation pose estimation result R obtained in step 3 to obtain the relative pose (R, t) of the target object.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) Significantly enhances the robustness of unknown object pose estimation in complex scenes. This paper combines the cue-guided segmentation model (SAM), coarse-grained and fine-grained feature matching mechanisms (DINO and EfficientLoFTR), and a pose solution strategy that decouples embedding retrieval and triangulation. It can achieve robust 6D pose estimation of unknown objects in complex visual scenes with occlusion, background interference, and a large number of objects. Compared with traditional methods, it has stronger adaptability and practical value in open environments.

[0048] (2) Complete 6D pose estimation is achieved using only a single reference image. Unlike traditional estimation methods that require multi-view training data or rely on precise 3D models, this method only requires a single reference image as input to complete rotational and translational pose estimation of the target object, greatly reducing the perception cost of actual deployment and making it suitable for dynamic interaction scenarios and rapid response tasks.

[0049] (3) Improving generation efficiency and reducing detail errors by introducing the Latent Diffusion Model (LDM). This paper introduces a pre-trained latent diffusion model in the new perspective synthesis and embedding generation stage, performing conditional image embedding generation only in the low-dimensional latent space. Compared with the high-overhead generation process of the traditional diffusion model in the original pixel space, it effectively avoids the loss of high-frequency details and the generation of noise, while significantly improving the inference efficiency and system real-time performance, providing high-quality semantic feature support for embedding retrieval and subsequent pose evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flowchart of the method for estimating 6D pose of an unknown object in a single view based on segmentation and new perspective synthesis in this embodiment;

[0051] Figure 2 This is an experimental result diagram of the single-view unknown object 6D pose estimation method based on segmentation and new perspective synthesis in this embodiment. DETAILED DESCRIPTION

[0052] To facilitate understanding of the present application, the present application will be described more comprehensively below with reference to the relevant drawings.

[0053] Figure 1 This is a flow chart of the 6D pose estimation method for a single-view unknown object based on segmentation and new perspective synthesis in this embodiment. Figure 1 Describe the method in detail, such as Figure 1 As shown, the method includes the following steps:

[0054] Step 1: Design a segmentation module for the pose estimation network and use it to obtain the segmentation result of the target object I query’ .

[0055] Step 1.1: Input reference image I ref and query image I query , using the suggestible segmentation model from the query image I query Extract and reference image I ref Several candidate masks that may be the target object's area are matched to form a candidate mask set, which provides prior information for subsequent matching and pose estimation.

[0056] Specifically, the input reference image I ref and query image I query , through the promptable segmentation model SAM (SegmentAnything Model) from the query image I query Extract and reference image I ref The matched candidate masks of several possible target object areas are composed of three main components: an image encoder for extracting image feature information and generating an image embedding representation, a prompt encoder for encoding prompt information and generating a prompt embedding vector, and a mask decoder for predicting the target object mask.

[0057] In actual operation, firstly the input reference image I ref and query image I query Perform preprocessing, including normalization and necessary size padding, to adapt to the input specifications required by the image encoder. The normalization process is as follows:

[0058]

[0059] Where x represents the pixel value in the original input image; x' represents the normalized pixel value; pixel_mean and pixel_std are the normalization parameters preset in the SAM segmentation model, where pixel_mean represents the mean of each channel (such as RGB) of the image; pixel_std represents the standard deviation of each channel of the image.

[0060] Then, the SAM model is used to query image I query Dense hint points are uniformly sampled on the pixel grid as foreground hints, and the hint encoder is called to embed and encode these hint points to generate a sparse hint embedding vector and a dense hint embedding vector. The mask decoder receives the image embedding representation from the image encoder and the hint embedding representation from the hint encoder, and outputs its predicted low-resolution mask. Finally, the low-resolution mask is restored to the original resolution of the image through interpolation and binarized according to the set mask threshold to obtain a candidate mask set of the following form: M = {m 1 ,m 2 ,...,m j}. Where j is the number of masks, each mask m j It represents a binary region in the query image where the target object may exist. The image region corresponding to the pixel with value 1 is the mask region that matches the target object in the reference image.

[0061] Step 1.2: Filter out the top-S candidate masks from the candidate mask set obtained in step 1.1, and preliminarily determine the S candidate regions in the query image where the target object corresponding to the reference image is located.

[0062] The specific method is: use the pre-trained DINO-v2 model to extract the reference image I provided by the user ref And each mask m in the candidate mask set j The category token features [CLS] are defined as CLS P and CLS T (j), and calculate the reference image I ref Class Token Features CLS P Respectively with each mask m in the mask set j Class Token Features CLS T (j), construct a 1×j-dimensional similarity matrix S(P,T,j)= <CLS P ,CLS T (j)>, select the S masks corresponding to the highest similarity values to obtain the Top-S candidate masks, and preliminarily determine the S candidate regions where the target objects corresponding to the reference image are located in the query image. In this embodiment, S=5.

[0063] Step 1.3: To further improve the accuracy of pose estimation, the image area corresponding to each mask in the Top-S candidate mask obtained in step 1.2 is combined with the reference image to form each image pair, and the EfficientLoFTR model is used for fine-grained verification and optimization to obtain the segmentation result I of the target object. query’ .

[0064] Specifically, for each image pair formed by combining the image area corresponding to each mask in the Top-S candidate masks obtained in step 1.2 with the reference image, the Efficient LoFTR (Local Feature TRansformer) matching framework based on the Transformer architecture is used to establish a local feature correspondence between the reference image and the query image area where each mask in the Top-S candidate masks is located, so that each image pair obtains n key point matching pairs consisting of the key point in the reference image and the key point matching the key point in the image area.

[0065] In this embodiment, the pose estimation network first performs grayscale conversion and normalization on the image area corresponding to each mask in the Top-S candidate masks and the reference image, and constructs S image pairs formed by combining the image area corresponding to each mask in the processed Top-S candidate masks with the reference image. The image pairs are input into the Efficient LoFTR model in sequence. After forward propagation through the Efficient LoFTR model, the key point set and the matching confidence vector c in the image area corresponding to the query image and the candidate mask in each image pair are obtained, and finally S key point sets are obtained. The matching confidence vector c = {c1, c2, ...} represents the reliability of the matching of each pair of key points in each key point set. By defining the confidence threshold σ, the key point matching pairs with low confidence are filtered, and only the key point matching pairs satisfying ci≥σ are retained. In this embodiment, σ = 0.8. The matching quality index Criteria is defined as:

[0066]

[0067] where c i is the confidence of the i-th key point matching pair, and II is the indicator function.

[0068] Finally, the area where the candidate mask corresponding to the maximum value of Criteria is located is selected as the reliable matching result to reduce the impact of mismatching on pose estimation. Finally, the area where the candidate mask corresponding to the maximum value of Criteria is located is used as the segmentation result of the target object I query '.

[0069] Step 2: Design a new perspective synthesis module for the pose estimation network and use it to generate a multi-perspective embedding template for the target object.

[0070] Step 2.1: Load the variational autoencoder VAE, conditional encoder, U-Net, and MLP modules in the pre-trained latent diffusion model (LDM) to establish the new perspective synthesis module of the pose estimation network in this embodiment, and use the conditional encoder in the new perspective synthesis module to generate an embedded representation of the relative rotation posture change ΔR.

[0071] The relative rotational posture change ΔR is a change in the rotational posture of the target object in the viewing angle of the query image relative to the rotational posture of the target object in the viewing angle of the reference image.

[0072] The parameters of the variational autoencoder VAE are frozen during the training of the entire pose estimation network and are only used to generate the reference image I ref And the segmentation result I query’ The embedding representation of avoids gradient updates during training, thereby maintaining the consistency of the embedding space.

[0073] Step 2.2: Use Xavier uniform distribution to initialize the multilayer perceptron (MLP) in the new perspective synthesis module.

[0074] The relative rotation posture change ΔR is encoded by the conditional encoder and expressed as a conditional embedding In the input MLP, in this embodiment, the weight parameters of MLP are initialized using Xavier uniform distribution; the role of MLP is to nonlinearly map the relative rotation posture change ΔR, so that it has stronger expression ability and structural adjustability in the high-dimensional embedding space, thereby obtaining a time embedding vector suitable for conditional control. Specifically, each downsampling and upsampling module in U-Net receives z pose As a modulation signal to guide the feature conversion process. In this embodiment, C = 192 * 4 = 768, which is consistent with the time embedding dimension of the attention module in U-Net. Then the MLP used for posture mapping in this embodiment is set to "single layer" mode, that is, only one linear layer is used to transform z pose Projected into an embedding vector of the specified dimension, as follows:

[0075] z pose =MLP(△R) =W·△R+b (3)

[0076] Where W is a C×d weight matrix used to map the input vector to a high-dimensional conditional embedding space, where d is the dimension of ΔR and C is the dimension of the output embedding representation; b is a bias vector that is consistent with the dimension C of the output embedding representation and is used to enhance expressiveness.

[0077] Step 2.3: Input the reference image I in step 1.1 ref , the segmentation result I obtained in step 1.3 query’ Together with all possible relative rotation posture changes ΔR, they are input into the new perspective synthesis module;

[0078] In this embodiment, N=342 relative rotation posture changes are defined based on the recursive subdivision strategy of the icosahedron, which is represented as a set P=(ΔR1, ΔR2, ..., ΔR N ), where ΔR1, ΔR2, ..., ΔR N All possible relative rotational posture changes defined in this embodiment. The recursive icosahedron subdivision strategy is to initialize a regular icosahedron on the sphere as the basic mesh structure, and recursively subdivide each triangular face of the regular icosahedron. Specifically, the midpoint of each edge of the triangular face is determined and these midpoints are projected onto the unit sphere. Each triangular face is split into multiple smaller triangles with the help of these midpoints, thereby achieving a higher density directional sampling effect and gradually approximating the sphere. This set evenly covers the three-dimensional sphere, ensuring that all potential perspectives of the object are included.

[0079] Step 2.4: Input image data, i.e. reference image I ref And the segmentation result I query’ Normalize the image and then generate the reference image I through the variational autoencoder VAE ref The embedding representation z ref And the segmentation result I query’ The embedding representation z query’ ;

[0080] After the image data is normalized, the pixel value range is limited to the interval [-1, 1] to improve the training stability of the new perspective synthesis module in the pose estimation network and the convergence speed of the pose estimation network.

[0081] Step 2.5: Generate a multi-view embedding template of the target object using the reference image and a predefined set of relative rotation pose changes through the new view synthesis module.

[0082] The reference image I ref With each relative rotation attitude change ΔR i The new perspective synthesis module is jointly input and the corresponding template embedding representation set (z1, z2, ..., z N) as the target object multi-view embedding template, where z i (i=1,2,...,N) represents the relative rotation change ΔR of the reference image i The embedding representation under the template embedding representation set (z1,z2,...,z N ) is generated as follows:

[0083] Based on the pose ambiguity, the possible solutions of the relative rotation pose change ΔR are modeled as the reference image I ref and query image I query The Gaussian distribution determined by the two methods is used to design the training loss function of the pose estimation network to generate the embedding representation of the reference image under the relative rotation pose change ΔR. The details are as follows:

[0084] Due to the limited information in a single query image perspective, the rotational posture of the target object in the query image perspective relative to its rotational posture in the reference image perspective, i.e., the relative rotational posture change ΔR, may have multiple possible solutions. Therefore, this embodiment models the possible solutions of the relative rotational posture change ΔR as the reference image I ref and query image I query A Gaussian distribution determined jointly by:

[0085] p(ΔR)=argmaxN(z query’ |μ=z(z ref ,ΔR),∑=∑(z ref ,ΔR)) (4)

[0086] Where p() represents the probability of obtaining the correct solution for the relative rotation posture change ΔR; N() represents the Gaussian distribution; the mean μ of the Gaussian distribution is the embedded representation z(z ref ,ΔR); the covariance ∑ of the Gaussian distribution is ∑(z ref ,ΔR); the segmentation result is embedded in z query’ are considered as observations of this Gaussian distribution.

[0087] This embodiment uses z(z ref ,ΔR) represents the embedding representation generated after the target object in the reference image undergoes a relative rotation posture change of ΔR:

[0088] z(z ref ,ΔR)=∫ M z(ΔR,M)p(M|z ref )dM (5)

[0089] Where M is the 3D shape distribution of the target object; z(ΔR,M) represents the embedding representation generated based on the relative rotation posture change ΔR and the 3D shape distribution M of the target object; p(M|z ref ) is the posterior probability distribution of the target object shape given the embedded representation of the reference image.

[0090] Since the calculation process of formula (5) is too complex, if an integral is calculated each time, the efficiency of the pose estimation network will be seriously reduced. Therefore, this embodiment approximates the process expressed by formula (5) by constructing the training loss function of the pose estimation network shown in formula (6) to approximate the integral process shown in formula (5).

[0091]

[0092] The loss is L2 loss, where F is the new perspective synthesis module, z t1 is the embedded representation of the reference image during the training of the pose estimation network; z t2 It is the embedded representation of the target object under ΔR during the training process of the pose estimation network.

[0093] Then, the parameters of the new perspective synthesis module are optimized by minimizing the loss function shown in formula (6), and then the new perspective synthesis module is used to ref and N relative rotation attitude changes ΔR1, ΔR2, ..., ΔR N Generate template embedding representation set (z1,z2,...,z N ).

[0094] Step 3: Use template matching method to segment the result I query’ The embedding representation z query’ Match the target object's multi-view embedded template to find the z query’ The most similar embedding representation is used to obtain the solution of the object's rotation posture, from which the preliminary rotation posture estimation result R of the target object is determined;

[0095] In order to find the target object from the multi-view embedding template query’ The most similar embedding representation, the pose estimation network uses the following method to retrieve the multi-view embedding template of the target object:

[0096] First, z query’ Expands to a template embedded set (z1,z2,...,z N ) where a single embedding represents a tensor with the same dimensions and computes the squared Euclidean distance:

[0097] d i =||z query’ -z i ||2 , i=1...N (7)

[0098] Where di represents the segmentation result and embedding z query’ and template embedding representation set (z1,z2,...,z N ) in which the i-th template embedding represents z i Then, by obtaining the template embedding representation set (z1,z2,...,z N ) and the segmentation result embedding representation z query’ Similarity score:

[0099] s i =-∑d i (h,w) (8)

[0100] where s i The segmentation result is embedded in z query’ and the template embedding representation set (z1,z2,...,z N ) where the i-th embedding represents z i Similarity score; h, w represent d i dimension.

[0101] Then, all similarity scores are sorted, and the A relative rotation pose changes corresponding to the template embedding representations of the top-A (for example, A=5) with higher similarity scores are selected as the initial rotation pose solution. The relative rotation pose change corresponding to the template embedding representation corresponding to the highest similarity score is found as the preliminary rotation pose estimation result R of the target object, which is used to guide the subsequent translation estimation and optional pose refinement operations. Specifically, R is expressed as x i Relative to x′ i The rotation matrix, x i represents the key point in the reference image, x′ i Represents the segmentation result I query’ Zhong and x i The corresponding key points.

[0102] Step 4: Estimate the relative translation vector t of the target object, and combine it with the preliminary rotation pose estimation result R of the target object obtained in step 4 to obtain the relative pose (R, t) of the target object;

[0103] Step 4.1: Based on the reference image I ref and segmentation result I query’ The five-point method is used to obtain the essential matrix for the key point matching pairs in .

[0104] Assume that the reference image I ref and segmentation result I query’ The key point matching pairs in (xi , x′ i ), x i represents the key point in the reference image, x′ i Represents the segmentation result I query’ Zhong and x i For the corresponding key points, the camera intrinsic parameter matrix K is known. According to the projection relationship, we have:

[0105] x′ i =K[R|t]X i , x i =K[I|0]X i (9)

[0106] Where R is x i Relative to x′ i The rotation matrix of [I|0]; the identity matrix; X i For key point matching pairs (x i , x i ') in three-dimensional space. According to geometric constraints, key point matching corresponds to the following basic relationship:

[0107]

[0108] Where E is the essential matrix, which can be solved by the Five-Point Algorithm, provided that at least five keypoint matching pairs are met. The Five-Point Algorithm effectively reduces the degrees of freedom by utilizing the special properties of the essential matrix (rank 2, two equal singular values), requiring only five keypoint matching pairs to solve the problem.

[0109] Step 4.2: Recover the relative translation vector t of the target object from the essential matrix, and combine it with the preliminary rotation pose estimation result R obtained in step 3 to obtain the relative pose (R, t) of the target object.

[0110] Since the essential matrix can be written as E = [t] × R, where [t] × is the antisymmetric matrix form of the translation vector t, which is defined as:

[0111]

[0112] where t x , t y , t z Represent the values of the translation vector t on the x, y, and z axes respectively.

[0113] Therefore, the essential matrix E is decomposed by SVD (Singular Value Decomposition) to obtain two sets of candidate solutions:

[0114] (R,t), (R,-t)

[0115] Finally, in order to filter out the true solution, a triangulation operation is performed on each set of candidate solutions, that is, the coordinates X of the key point matching pairs in three-dimensional space are triangulated using the two camera projection matrices P1 = K[I|0] and P2 = K[R|t]. i , and detect whether the point is in front of both cameras (i.e., in the positive depth region). Only a unique set of solutions that satisfy all keypoint matching pairs with positive depth can be considered a legal pose solution.

[0116] Figure 2 This is the effect diagram of using the method of the present invention on a specific image. Figure 2 (a) is the rendered image using the pose estimation result; Figure 2 (b) is the query image; Figure 2 (c) is the reference image; Figure 2 (d) is the minimum rectangular bounding box image of the mask. As can be seen from the figure, the method of the present invention can fully combine the complementary advantages of RGB images and image embedding representation to achieve good detection effect.

[0117] In addition, the present invention uses T-Less, a publicly available dataset for pose estimation, for evaluation and comparison. As shown in Table 1, the present invention significantly outperforms other existing models in accuracy on the test set of this real-world T-Less dataset. Furthermore, as shown in Table 2, the present invention also performs well on LineMOD Occlussion, a dataset for complex scenes with severe occlusion, demonstrating the robustness of the present method to occluded scenes.

[0118] Table 1 Performance comparison of the method in this embodiment, NOPE method and 3DiM method on the T-Less dataset

[0119]

[0120] Table 2 Performance of the present invention on the LineMOD Occulussion dataset

[0121]

[0122] Table 3 Performance comparison of the method of the present invention

[0123]

[0124] It should be understood that, inspired by the technical concept of the present invention, those skilled in the art may make various improvements or changes based on the above content without departing from the content of the present invention, which still fall within the scope of protection of the present invention.

Claims

1. A 6D pose estimation method for unknown objects in a single view based on segmentation and novel perspective synthesis, characterized by: The method comprises the following steps: Step 1: Design a segmentation module for the pose estimation network and use it to obtain the segmentation result of the target object I query’ ; Step 2: Design a new perspective synthesis module for the pose estimation network and use it to generate a multi-perspective embedding template for the target object. Step 3: Use template matching method to segment the result I query’ The embedding representation z query’ Match with the target object multi-view embedding template to find the z query’ The most similar embedding representation determines the preliminary rotation pose estimation result R of the target object; Step 4: Estimate the relative translation vector t of the target object, and combine it with the preliminary rotation pose estimation result R of the target object obtained in step 4 to obtain the relative pose (R, t) of the target object.

2. The single-view unknown object 6D pose estimation method according to claim 1, characterized in that The step 1 comprises the following steps: Step 1.1: Input reference image I ref and query image I query , using the suggestible segmentation model from the query image I query Extract and reference image I ref The target object is matched with several candidate masks that may be the target object’s area, and a candidate mask set M = {m 1 ,m 2 ,...,m j }; Each mask m j Represents a binary region in the query image where the target object may exist, where the image region corresponding to the pixel with value 1 is the mask region that matches the target object in the reference image; Step 1.2: Select the top-S candidate masks from the candidate mask set M and preliminarily determine the S candidate regions in the query image where the target object corresponding to the reference image is located; Step 1.3: For each image pair formed by combining the image area corresponding to each mask in the Top-S candidate mask with the reference image, use the Efficient LoFTR model to establish the local feature correspondence between the reference image and the image area, so that each image pair obtains n key point matching pairs consisting of the key points in the reference image and the key points matching the key points in the image area, calculate the confidence of each key point matching pair, and filter out the key point matching pairs with low confidence in each image pair, and then solve the average of the confidence of the remaining key point matching pairs as the value of the matching quality index of each image pair, and finally use the image area in the image pair corresponding to the largest matching quality index value as the segmentation result I of the target object. query '.

3. The single-view unknown object 6D pose estimation method according to claim 2, characterized in that The method for selecting the top-S candidate masks from the candidate mask set M described in step 1.2 is as follows: the reference image I is extracted using the pre-trained DINO-v2 model ref The class token feature CLSP and each mask m in M j The category token feature CLST(j) of the reference image I ref The category token feature CLSP is respectively associated with each mask m in M j The cosine similarity of the category token feature CLST(j) is used to construct a 1×j-dimensional similarity matrix S(P,T,j)=<CLSP,CLST(j)> , select the masks corresponding to the S highest similarity values and get the Top-S candidate masks.

4. The single-view unknown object 6D pose estimation method according to claim 2, characterized in that The step 2 comprises the following steps: Step 2.1: Load the variational autoencoder VAE, conditional encoder, U-Net and MLP modules in the pre-trained latent diffusion model LDM to establish a new perspective synthesis module of the pose estimation network, and use the conditional encoder in the new perspective synthesis module to generate the conditional embedding representation of the relative rotation posture change ΔR Where C is consistent with the temporal embedding dimension of the attention module in U-Net; the relative rotational pose change ΔR is the change in the rotational pose of the target object in the query image view relative to its rotational pose in the reference image view; Step 2.2: Initialize the multi-layer perceptron MLP in the new perspective synthesis module using Xavier uniform distribution: z pose Input into MLP and set MLP to "singlelayer" mode, that is, use only one linear layer to transform z pose Project to an embedding vector of the specified dimension; Step 2.3: Input the reference image I in step 1.1 ref , the segmentation result I obtained in step 1.3 query’ and all possible N relative rotation attitude changes ΔR1, ΔR2, ..., ΔR N Input the new perspective synthesis module together; Step 2.4: Input image data, i.e. reference image I ref And the segmentation result I query’ Normalize the image and then generate the reference image I through the variational autoencoder VAE ref The embedding representation z ref And the segmentation result I query’ The embedding representation z query’ ; Step 2.5: Based on the input reference image I ref The embedding representation z ref With each relative rotation attitude change ΔR i (i=1,2,...,N), the new perspective synthesis module outputs the corresponding template embedding representation set (z1,z2,...,z N ) as the target object multi-view embedding template, where z i (i=1,2,...,N) represents the relative rotation change ΔR of the reference image i The embedding representation under the template embedding representation set (z1,z2,...,z N ) is generated as follows: Based on the pose ambiguity, the possible solutions of ΔR are modeled as the reference images I ref and query image I query A Gaussian distribution determined jointly by: p(ΔR)=argmaxN(z query’ |μ=z(z ref ,ΔR),∑=∑(z ref ,ΔR)) (4) Where p() represents the probability of obtaining the correct solution of ΔR; N() represents the Gaussian distribution; the mean μ of the Gaussian distribution is the embedded representation z(z ref ,ΔR); the covariance ∑ of the Gaussian distribution is ∑(z ref ,ΔR); the segmentation result is embedded in z query’ is considered as an observation of this Gaussian distribution; z(z ref ,ΔR)=∫ M z(ΔR,M)p(M|z ref )dM (5) Where M is the 3D shape distribution of the target object; z(ΔR,M) represents the embedding representation generated based on ΔR and M; p(M|z ref ) is embedded in z for a given reference image ref After that, the posterior probability distribution of the target object shape; Construct the training loss function of the pose estimation network shown in formula (6) to approximate the process expressed in formula (5): ∑ (zt1,zt2,ΔR) ||F(z ref ,ΔR)-z t2 || 2 (6) Where F is the new perspective synthesis module, z t1 is the embedded representation of the reference image during the training of the pose estimation network; z t2 It is the embedded representation of the target object under ΔR during the training process of the pose estimation network; Finally, the parameters of the new perspective synthesis module are optimized by minimizing the loss function shown in formula (6), and then the new perspective synthesis module is used to ref and N relative rotation attitude changes ΔR1, ΔR2, ..., ΔR N Generate template embedding representation set (z1,z2,...,z N ).

5. The single-view unknown object 6D pose estimation method according to claim 4, characterized in that The training loss function of the pose estimation network shown in formula (6) is L2 loss.

6. The single-view unknown object 6D pose estimation method according to claim 4, characterized in that In step 2.3, all possible N relative rotation pose changes ΔR1, ΔR2, ..., ΔR are defined based on the icosahedron recursive subdivision strategy. N .

7. The single-view unknown object 6D pose estimation method according to claim 4, characterized in that In step 3, in order to find the target object with the same z query’ The most similar embedding representation, the pose estimation network uses the following method to retrieve the multi-view embedding template of the target object: First, z query’ Expands to a template embedded set (z1,z2,...,z N ) where a single embedding represents a tensor with the same dimensions and computes the squared Euclidean distance: d i =||z query’ -z i || 2 ,i=1...N (7) where d i represents z query’ Embedded with the i-th template z i degree of difference; Then, by obtaining the template embedding set (z1,z2,...,z N ) in each embedding representation and z query’ The similarity score s i : s i =-∑d i (h,w) (8) Where h and w represent d i Dimensions; Then, all similarity scores are sorted, and the A relative rotation posture changes corresponding to the top-A higher similarity scores are selected as the solutions of the initial rotation posture; Finally, the template embedding corresponding to the highest similarity score is found from the solution of the initial rotation posture, and the corresponding relative rotation posture change is taken as the preliminary rotation posture estimation result R of the target object. Specifically, R is expressed as x i Relative to x i 'The rotation matrix, x i represents the key points in the reference image, x i 'Represents the segmentation result I query’ Zhong and x i The corresponding key points.

8. The single-view unknown object 6D pose estimation method according to claim 4, characterized in that The step 4 comprises the following steps: Step 4.1: Based on the reference image I ref and segmentation result I query’ The key point matching pairs in the ,use the five-point method to obtain the essential matrix; Step 4.2: Recover the relative translation vector t of the target object from the essential matrix, and combine it with the preliminary rotation pose estimation result R obtained in step 3 to obtain the relative pose (R, t) of the target object.

Images