A generalizable 6D pose estimation method for untrained targets

Abstract

This invention discloses a generalizable 6D pose estimation method for untrained targets. The invention includes: pre-training a 6D pose estimation network using a training dataset composed of depth images rendered from multi-class 3D models; in an offline stage, generating a geometric prior knowledge base for the untrained target object using the pre-trained network and the target object's 3D model; and utilizing an observation camera C... o After observing an untrained target object, the original depth image is obtained. Then, combined with the geometric prior knowledge base Q, a pre-trained 6D pose estimation network is used to estimate the 6D pose of the original depth image, thus obtaining the 6D pose estimation result of the current untrained target object. This invention achieves generalizable 6D pose estimation for untrained targets, solving the problems caused by inaccurate viewpoint encoding and inconsistencies between prior information and observed information. It can achieve efficient and accurate 6D pose estimation for untrained targets without adjusting any pre-trained network parameters.

Description

Technical Field

[0001] This invention relates to a 6D pose estimation method in the field of computer vision technology, and particularly to a generalizable 6D pose estimation method for untrained targets. Background Technology

[0002] 6D pose estimation is a crucial step in applications such as robotics and augmented reality. In recent years, many deep learning-based 6D pose estimation methods have achieved excellent prediction results on publicly available datasets. However, these algorithms cannot directly estimate objects that the network did not encounter during training (hereinafter referred to as untrained targets or unseen objects). They require significant time and effort to build datasets for these untrained targets and to retrain the network, consuming substantial computational resources and time, to achieve 6D pose estimation for these objects. This severely hinders the application of deep learning algorithms in the real world.

[0003] Currently, some 6D pose estimation methods exist that can directly estimate untrained targets without changing the model parameters. Specifically, the method first renders a multi-view reference image using a known 3D model of the untrained target, and then uses a viewpoint encoder to generate a prior viewpoint feature set for the reference image. Next, observation viewpoint feature symbols are generated for the untrained target in the observation image. Template matching is then used to find the closest reference viewpoint for the observation viewpoint feature symbols, and the 6D pose of the untrained target in the observation image is obtained based on the pose information in the matched reference image.

[0004] However, this approach does not interact with viewpoint features and geometric prior knowledge. Since some novel appearances of untrained targets may not have been learned by the network, it is difficult for the network to accurately encode viewpoint information from a single observation image, thus affecting the correctness of subsequent template matching and the accuracy of 6D pose estimation. On the other hand, these methods do not consider the problem of non-uniform viewpoints: an object with the same rotational component relative to the camera will appear differently in a standard viewpoint (where the camera's optical axis passes through the object's center) and a non-standard viewpoint (where the camera's optical axis does not pass through the object's center). Because the camera's optical axis is often offset from the object's center during observation (i.e., a non-standard viewpoint), while the reference image is captured (rendered) from a standard viewpoint, the object in the observation image and the object in the matched reference image, although having the same appearance, have different 3D rotational components, leading to significant errors in the final 6D pose estimation. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention proposes a generalizable 6D pose estimation method for untrained targets. It solves the problems of inaccurate viewpoint encoding and misalignment between the observed viewpoint and the prior information viewpoint during the estimation process, and achieves efficient and accurate prediction of the 6D pose of untrained targets without changing any parameters in the network.

[0006] The specific technical solution of the present invention is as follows:

[0007] I. A generalizable 6D pose estimation method for untrained targets

[0008] S1: Construct a 6D pose estimation network. The 6D pose estimation network includes a feature extraction network, a view encoder, a view interaction attention module, and an in-plane rotation prediction head connected in sequence. The 6D pose estimation network is pre-trained using a training dataset composed of depth images rendered by multi-class 3D models.

[0009] S2: In the offline stage, multi-view depth reference images are rendered on the known 3D model of the untrained target object to obtain the corresponding multi-view depth reference images. The number of viewpoints is N. Then, the multi-view depth reference images are scaled and cropped before being fed into the pre-trained feature extraction network and viewpoint encoder to obtain the prior viewpoint feature set. The out-of-plane rotation of the depth reference image and its 3D model relative to the virtual camera. and in the prior viewpoint feature set The corresponding prior viewpoint feature symbol The geometric prior knowledge base Q is composed of;

[0010] S3: Inference phase, using the observation camera C with known internal parameters. o After observing the untrained target object, the original depth image is obtained. Then, combined with the geometric prior knowledge base Q, the 6D pose of the original depth image is estimated by a pre-trained 6D pose estimation network, and the 6D pose estimation result of the current untrained target object is obtained.

[0011] Specifically, S3 is:

[0012] S31: During the online phase, the observation camera C utilizes known intrinsic parameters. o After observing untrained target objects, an observation depth map is obtained. After target segmentation and image preprocessing of the observation depth map, a processed depth map is obtained.

[0013] S32: Input the processed depth map into the pre-trained feature extraction network and view encoder to obtain the observation feature map F. o and observation viewpoint feature v o ;

[0014] S33: The observation viewpoint feature symbol v o and prior viewpoint feature symbol set After being input together into the pre-trained viewpoint interaction attention module for feature exchange and optimization, the optimized observation viewpoint feature symbol v′ is obtained. o With the optimized prior viewpoint feature symbol set Update the geometric prior knowledge base and denote it as Q′;

[0015] S34: First, the template matching method is used to retrieve the optimized observation viewpoint feature v′ from the geometric prior knowledge base Q′. o The closest prior viewpoint feature is used as the matching object, and the depth reference image corresponding to the matching object is recorded as the matching reference image. Then match the reference image out-of-plane rotation As an initial out-of-plane rotation estimate for untrained target objects in the observed depth map Next, based on the observed feature map F o Matching reference image Calculate the in-plane rotation of untrained target objects in the observed depth map. and displacement components This yields the initial 6D pose estimation results for the untrained target object in the observed depth map;

[0016] S35: Align the initial 6D pose estimation results of the untrained target object in the observation depth map with a non-uniform viewpoint to obtain the final 6D pose estimation result.

[0017] In step S34, the template matching method is used to retrieve and optimize the observation viewpoint feature symbol v′ from the geometric prior knowledge base Q′. o The closest prior viewpoint features include:

[0018] The optimized observation viewpoint feature symbol v′ is calculated using the template matching method. o The feature symbol of each prior viewpoint in the geometric prior knowledge base Q′ The cosine similarity is used to determine the prior viewpoint feature with the highest cosine similarity. As the closest prior viewpoint feature.

[0019] In S34, based on the observed feature map F o Matching reference image Calculate the in-plane rotation of untrained target objects in the observed depth map, including:

[0020] Match reference image The feature map generated after feature extraction by the pre-trained feature extraction network is denoted as the matching feature map. From the observed feature map F o and matching feature maps A feature map is constructed and input into a pre-trained in-plane rotation prediction head. The in-plane rotation prediction head directly regresses the estimated in-plane rotation amount and uses it as the in-plane rotation amount of the untrained target object in the observed depth map.

[0021] In S34, based on the observed feature map F o Matching reference image Calculate the displacement components of untrained target objects in the observed depth map, including:

[0022] First, the observation camera C is calculated using the following formula. o Distance between the center of the untrained target object

[0023]

[0024] Among them, f o f vir For example, observation camera C o With virtual camera C vir focal length, t z,m To match the reference image Objects and virtual camera C vir The distance, l m l o Matching reference images The length of the diagonal of the 2D bounding box of an object in the depth map;

[0025] Then, based on the observation camera C o internal reference K o And the displacement components are solved using the back projection method, based on the center of the 2D bounding box of the untrained target object in the depth map. The formula is as follows:

[0026]

[0027] Among them, c x ,c y The x and y coordinates of the center of the 2D bounding box of the untrained target object in the observation depth map are given, and T is the transpose operation.

[0028] Specifically, S35 is:

[0029] S351: To the observation camera C o The vector γ from the optical center to the center of the observed untrained target object applies an in-plane rotation of the untrained target object in the depth measurement map. Get around C o Vector after optical axis rotation The formula is as follows:

[0030]

[0031] S352: Solve for the deviation matrix R using the following formula c :

[0032]

[0033] Where I is a 3×3 identity matrix. Let be the antisymmetric matrix of the vector; Ψ() is the Rodrigo rotation;

[0034] S353: Using the deviation matrix R c Eliminate initial out-of-plane rotation estimation of untrained target objects in the observed depth map Non-uniform viewpoint bias was analyzed to obtain the final 6D pose estimation result. The formula is as follows:

[0035]

[0036] In S1, the depth image obtained by rendering each 3D model specifically includes K APN samples. The K APN samples specifically include K depth images, K depth images with in-plane rotation applied, and K×M depth images with out-of-plane rotation applied.

[0037] In S1, the view encoder is pre-trained using contrastive learning. During the pre-training process, the feature map corresponding to the depth image is used as the benchmark, the feature map corresponding to the depth image with in-plane rotation is used as the positive example, and the feature map corresponding to the depth image with out-of-plane rotation is used as the negative example.

[0038] II. A computer device

[0039] The device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the generalizable 6D pose estimation method for untrained targets.

[0040] III. A computer-readable storage medium

[0041] The medium stores a computer program that, when executed by a processor, implements the steps of the generalizable 6D pose estimation method for untrained targets.

[0042] The beneficial effects of this invention are as follows:

[0043] (1) The method described in this invention can achieve 6D pose estimation of any rigid object, including untrained targets, without changing the parameters of the neural network. This solves the problem that deep learning algorithms need to spend a lot of manpower and computing power to construct untrained target datasets and retrain the network when estimating unseen objects.

[0044] (2) In the process of network pre-training, the present invention uses large-scale multi-class virtual data and adopts InfoNCE loss for comparative learning, which improves the neural network's ability to encode viewpoint information and its generalization.

[0045] (3) This invention solves the inaccuracy caused by encoding based solely on observed images by using the viewpoint interaction attention module. By introducing geometric prior knowledge, it promotes communication between viewpoints and achieves more accurate viewpoint encoding.

[0046] (4) The perspective alignment module proposed in this invention effectively eliminates the out-of-plane rotation estimation error caused by the inconsistency between the perspective of the observation information and the perspective of the prior information during the template matching process, and optimizes the 6D pose estimation accuracy.

[0047] In summary, this invention can avoid any changes to network parameters when estimating the 6D pose of an untrained target, generate more accurate viewpoint codes for the observed image, and eliminate estimation errors caused by inconsistent viewpoints, thus having good engineering practical value. Attached Figure Description

[0048] Figure 1 This is a flowchart of the 6D pose estimation method in this invention.

[0049] Figure 2 This is a network structure diagram constructed by the 6D pose estimation method in this invention.

[0050] Figure 3 This is a schematic diagram of multi-viewpoint depth reference image rendering in this invention.

[0051] Figure 4 This is a schematic diagram of the view alignment module aligning non-uniform viewpoints in this invention.

[0052] Figure 5 These are untrained, general-purpose industrial parts used in the embodiments of this invention.

[0053] Figure 6 This is a visualization result of 6D pose estimation of untrained general-purpose industrial parts in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the following description, in conjunction with the accompanying drawings and embodiments, applies the methods described in this invention to 6D pose estimation of untrained general-purpose industrial parts, providing a more detailed explanation of the invention. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] like Figure 1 As shown, this invention provides a generalizable 6D pose estimation method for untrained targets, comprising the following steps:

[0056] S1: Construct a 6D pose estimation network, such as Figure 2 As shown, the 6D pose estimation network includes a feature extraction network, a view encoder, a view interaction attention module, and an in-plane rotation prediction head connected in sequence. The 6D pose estimation network is pre-trained on a large scale using a training dataset composed of depth images rendered by multi-class 3D models. The pre-trained 6D pose estimation network can still achieve accurate 6D pose estimation without retraining even if the pre-trained model encounters an untrained target during the inference stage.

[0057] The feature extraction network is a VGG-11 network, used to extract image features, taking a 128×128 pixel depth image as input, and the feature map... For output;

[0058] The view encoder encodes viewpoint features and consists of a CBAM self-attention module, a 2D convolutional layer, a pooling layer, and a fully connected layer. It takes a feature map F as input and encodes the viewpoint features. For output;

[0059] The viewpoint interaction attention module is a scaled dot product self-attention model used to optimize viewpoint features, which is a tensor formed by concatenating viewpoint features from S different viewpoints. As input, the optimized tensor As output;

[0060] An in-plane rotation prediction head is used to estimate in-plane rotation. Its network structure consists of a two-dimensional convolutional layer and two consecutive fully connected layers. It takes a feature map F as input and estimates the in-plane rotation matrix. For output;

[0061] Random sampling was performed on the original ShapeNet dataset to create a pre-training dataset containing 10,000 models. These 10,000 models were then used to generate a large amount of virtual depth data as training data. This training data was then used to perform large-scale pre-training of the entire 6D pose estimation network.

[0062] The steps for generating each set of training data during network pre-training are as follows:

[0063] S1-1: Randomly select a 3D model from the ShapeNet dataset, and uniformly select 16 standard viewpoints in its hemispherical space (with the virtual camera's optical axis aligned with the center of the 3D object), rendering 16 depth images as the baseline A for contrastive learning. i (i = 1, 2, ... 16);

[0064] S1-2: For all benchmarks A i A random in-plane rotation (causing the virtual camera to rotate around the optical axis) is applied and rendered to obtain 16 depth images with the applied in-plane rotation, which serve as positive examples P for contrastive learning. i ;

[0065] S1-3: For each reference A i Five random out-of-plane rotations (changing the camera's viewpoint relative to the 3D model) are applied and rendered as a negative example of contrastive learning (NG). i,j (j = 1, 2, ..., 5) form 16 APN (Anchor-Positive-Negative Sample) samples. i ={A i ,P i ,NG i,j};

[0066] The viewpoint encoder is pre-trained using contrastive learning. For the k-th (1≤k≤16) APN sample in each iteration, its participation in encoder training is as follows:

[0067] S1-4: Perform target segmentation on each depth image in the k-th APN sample using the Mask R-CNN instance segmentation algorithm, and reduce the resolution of the segmented image to 128×128. Input the segmented image into the feature extraction network to obtain the feature map F. k A ,F k P ,F k,j NG (j = 1, 2, ..., 5);

[0068] S1-5: Transfer feature map F k A ,F k P ,F k,j NG Input the viewpoint encoder to obtain the viewpoint feature symbols v corresponding to the baseline, positive examples, and negative examples of the encoded viewpoint information. k A ,v k P ,v k,jNG ;

[0069] S1-6: Using InfoNCE Loss as the contrastive learning loss function L for the encoder InfoNCE :

[0070]

[0071] in, S cos () is used to calculate the cosine similarity between feature vectors x and y of the same dimension, where τ is the temperature coefficient, set to 0.1.

[0072] The training steps for the in-plane rotation prediction head are as follows:

[0073] S1-7: Calculate feature map F k A With F k P Correlation heatmap matrix between The correlation heatmap matrix H θ Input to the in-plane rotation prediction head, where This represents the set of real numbers with a channel count of 512×8×8;

[0074] S1-8: Direct Regression Benchmark A for In-Plane Rotation Prediction Head k Positive example P k In-plane rotation between

[0075] S1-9: Calculate the prediction loss L of the in-plane rotating prediction head using the following loss function. θ :

[0076] L θ = -ln((1.0+ζ) / 2.0),

[0077]

[0078] Where ζ is R, which is measured by cosine similarity. θ and The difference value, S cos () is the function for calculating cosine similarity. Represents the reference A k Apply in-plane rotation R θ 2D spatial rotation amount, R θ Reference A k Compared with positive example P k The true value of the in-plane rotation between them. This is a tensor flattening operation.

[0079] The entire 6D pose estimation network is pre-trained on a large scale using an end-to-end training method, and the total loss function is:

[0080] L=α(λ1L InfoNCE +λ2L θ )

[0081] Where λ1 and λ2 are two loss weighting factors, λ1 = 10 and λ2 = 1.5, and α is a fixed scaling factor with a value of 10.

[0082] The entire training process was performed in PyCharm, using the Adam adaptive moment estimator as the network parameter optimizer. The training epochs were set to 50, with an initial learning rate of 10. -5 The final learning rate is 10. -8 .

[0083] S2: In the offline phase, a multi-view depth reference image is obtained by rendering a known 3D model of the untrained target object using multi-view depth reference images. The number of viewpoints is N. Specifically, it is as follows: Figure 3 As shown, virtual camera C vir A multi-viewpoint depth reference image is achieved by uniformly selecting N standard viewpoints in the hemispherical space of a known 3D model of the target object. The rendering process is as follows: To reduce repetitive appearance features in Q, N=5000 is set for asymmetrical objects; for generally symmetrical objects, viewpoint sampling is performed uniformly on one side of the axis of symmetry, and N=2500 is set; for rotating objects, viewpoint sampling is performed uniformly on a generatrix in hemispherical space, and N=60 is set.

[0084] Next, the multi-view depth reference image is scaled and cropped before being fed into a pre-trained feature extraction network and view encoder to obtain a priori view feature symbol set. Specifically, using depth reference images As input, based on the object's maximum diameter and C vir Acquisition of intrinsic and extrinsic parameters The 2D bounding box of the object is extracted and cropped, and the resolution of the cropped image is adjusted to 128×128 pixels. Then, it passes through a feature extraction network and a viewpoint encoder to extract prior viewpoint features. For output. From the depth reference image. The out-of-plane rotation of the 3D model of the depth reference image relative to the virtual camera. and in the prior viewpoint feature set The corresponding prior viewpoint feature symbol The geometric prior knowledge base Q is composed of the following conditions:

[0085] S3: During the online phase, the observation camera C utilizes known intrinsic parameters.o After observing the untrained target object, the original depth image is obtained. Then, combined with the geometric prior knowledge base Q, the 6D pose of the original depth image is estimated by a pre-trained 6D pose estimation network, and the 6D pose estimation result of the current untrained target object is obtained.

[0086] S3 specifically refers to:

[0087] S31: Inference phase, using the observation camera C with known internal parameters. o After observing untrained target objects, an observation depth map is obtained. After target segmentation and image preprocessing of the observation depth map, a processed depth map is obtained. The image preprocessing includes: using the Mask R-CNN instance segmentation algorithm to segment the target objects in the observation depth map to obtain the corresponding target object 2D bounding boxes and segmentation masks; and using the 2D bounding boxes to crop the observation depth image, filtering out depth values ​​other than the target object depth information through the segmentation mask, and adjusting the resolution of the cropped image to 128×128 pixels.

[0088] S32: Input the processed depth map into the pre-trained feature extraction network and view encoder to obtain the observation feature map F. o and observation viewpoint feature v o Observation feature map F o The feature map is generated after feature extraction by a pre-trained feature extraction network from the processed depth map; among which, the depth camera C observes the general parts. o The intrinsic parameter matrix is ​​as follows:

[0089]

[0090] S33: The observation viewpoint feature symbol v o and prior viewpoint feature symbol set After being input together into the pre-trained viewpoint interaction attention module for feature exchange and optimization, the optimized observation viewpoint feature symbol v′ is obtained. o With the optimized prior viewpoint feature symbol set Update the geometric prior knowledge base and denote it as Q′, that is, replace the prior viewpoint feature symbol set in the geometric prior knowledge base Q.

[0091] Specifically:

[0092] First, the viewpoint feature symbol v encoded by the observed image is... o and The components are spliced ​​together to form a 64×S tensor Z, where S=N+1.

[0093] Then, Z is simultaneously used as a query, key, and value, and input into the viewpoint interaction attention module.

[0094] Then perform the scaled dot product self-attention operation:

[0095]

[0096] Where q, K, and V are respectively the values ​​after linear transformation (All are fully connected layers) queries, keys, and values, d k Z′ represents the dimension of the viewpoint feature symbol, with a value of 64, and Z′ is the optimized tensor. This operation facilitates communication between the observing viewpoint information and the prior viewpoint information, and outputs the optimized observing viewpoint feature symbol v′. o With prior viewpoint feature set And update the geometric prior knowledge base to

[0097] S34: Estimating the initial 6D pose using unwrapping: First, the template matching method is used to retrieve the optimized observation viewpoint feature v′ from the geometric prior knowledge base Q′. o The closest prior viewpoint feature is used as the matching object, and the depth reference image corresponding to the matching object is recorded as the matching reference image. Then match the reference image out-of-plane rotation As an initial out-of-plane rotation estimate for untrained target objects in the observed depth map Next, based on the observed feature map F o Matching reference image Calculate the in-plane rotation of untrained target objects in the observed depth map. and displacement components This yields the initial 6D pose estimation results for the untrained target object in the observed depth map;

[0098] In S34, the template matching method is used to retrieve and match the optimized observation viewpoint feature v′ in the geometric prior knowledge base Q′. o The closest prior viewpoint features include:

[0099] The optimized observation viewpoint feature symbol v′ is calculated using the template matching method. o The feature symbol of each prior viewpoint in the geometric prior knowledge base Q′ The cosine similarity is used to determine the prior viewpoint feature with the highest cosine similarity. As the closest prior viewpoint feature.

[0100] In S34, based on the observed feature map F o Matching reference image Calculate the in-plane rotation of untrained target objects in the observed depth map, including:

[0101] Match reference image The feature map generated after feature extraction by the pre-trained feature extraction network is denoted as the matching feature map. From the observed feature map F o and matching feature maps A feature map is constructed and input into a pre-trained in-plane rotation prediction head. The in-plane rotation prediction head directly regresses the estimated in-plane rotation amount and uses it as the in-plane rotation amount of the untrained target object in the observed depth map. The initial 3D rotation component estimate can be expressed as:

[0102] In S34, based on the observed feature map F o Matching reference image Calculate the displacement components of untrained target objects in the observed depth map, including:

[0103] First, the observation camera C is calculated using the following formula. o Distance between the center of the untrained target object

[0104]

[0105] Among them, f o f vir For example, observation camera C o With virtual camera C vir focal length, t z,m To match the reference image Objects and virtual camera C vir The distance, l m l o Matching reference images The length of the diagonal of the 2D bounding box of an object in the depth map;

[0106] Then, based on the observation camera C o internal reference K o And the displacement components are solved using the back projection method, based on the center of the 2D bounding box of the untrained target object in the depth map. The formula is as follows:

[0107]

[0108] Among them, c x ,c y The x and y coordinates of the center of the 2D bounding box of the untrained target object in the observation depth map are given, and T is the transpose operation.

[0109] Therefore, the initial 6D pose estimation result of the untrained target object in the observed depth map can be expressed as:

[0110] S35: The viewpoint alignment module aligns the initial 6D pose estimation results of the untrained target object in the observation depth map with a non-uniform viewpoint, eliminating the out-of-plane rotation error caused by this factor, and obtaining the final 6D pose estimation result. The viewpoint alignment module mainly solves the problem of viewpoint inconsistency between the observation image captured under a non-standard viewpoint (the optical axis of the real camera does not pass through the center of the observed target object) and the reference image rendered under a standard viewpoint. It eliminates the out-of-plane rotation estimation error caused by template matching under the premise of non-uniform viewpoints. A schematic diagram of the viewpoint alignment module eliminating the out-of-plane rotation estimation error caused by non-uniform viewpoints is shown below. Figure 4 As shown.

[0111] S35 specifically refers to:

[0112] S351: To the observation camera C o The vector γ from the optical center to the center of the observed untrained target object applies an in-plane rotation of the untrained target object in the depth measurement map. Get around C o Vector after optical axis rotation The formula is as follows:

[0113]

[0114] S352: Solve for the deviation matrix R using the following formula c , to align vectors With observation camera C o Optical axis [0,0,1] T :

[0115]

[0116] Where I is a 3×3 identity matrix. Ψ is the antisymmetric matrix of the vector; Ψ() is the Rodrigo rotation.

[0117] S353: Using the deviation matrix R c Eliminate initial out-of-plane rotation estimation of untrained target objects in the observed depth map Non-uniform viewpoint bias was analyzed to obtain the final 6D pose estimation result. The formula is as follows:

[0118]

[0119] in, To eliminate out-of-plane rotation due to non-uniform viewpoint bias, and to satisfy...

[0120] This invention addresses, without altering any network parameters, such as... Figure 5 The eight untrained, general-purpose industrial parts shown were subjected to 6D pose estimation. The pose estimation visualization results are displayed in [the image / database / etc.]. Figure 6 In the middle. For example Figure 5 and Figure 6 As shown, this invention achieves accurate 6D pose estimation for multiple untrained, off-center industrial common parts in three different industrial scenarios without changing any network parameters.

[0121] The present invention provides a generalizable 6D pose estimation device for untrained targets, comprising one or more processors, for implementing a generalizable 6D pose estimation method for untrained targets as described in the above embodiments.

[0122] An embodiment of the generalizable 6D pose estimation device for untrained targets of this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the device with data processing capabilities loading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, the following is a hardware structure diagram of any device with data processing capabilities containing the generalizable 6D pose estimation device for untrained targets of this invention. Besides the processor, memory, network interface, and non-volatile memory, the device in the embodiment may also include other hardware depending on its actual function, which will not be elaborated further.

[0123] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] This invention also provides a readable storage medium storing a program that, when executed by a processor, implements a generalizable 6D pose estimation method for an untrained target as described in the above embodiments.

[0126] The readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the readable storage medium can include both internal storage units of any data processing device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0127] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A generalizable 6D pose estimation method for untrained targets, characterized in that, Includes the following steps: S1: Construct a 6D pose estimation network. The 6D pose estimation network includes a feature extraction network, a view encoder, a view interaction attention module, and an in-plane rotation prediction head connected in sequence. The 6D pose estimation network is pre-trained using a training dataset composed of depth images rendered by multi-class 3D models. S2: In the offline stage, multi-view depth reference images are rendered on the known 3D model of the untrained target object to obtain the corresponding multi-view depth reference images. The number of viewpoints is N. Then, the multi-view depth reference images are scaled and cropped before being fed into the pre-trained feature extraction network and viewpoint encoder to obtain the prior viewpoint feature set. The out-of-plane rotation of the depth reference image and its 3D model relative to the virtual camera. and in the prior viewpoint feature set The corresponding prior viewpoint feature symbol The geometric prior knowledge base Q is composed of; S3: Inference phase, using the observation camera C with known internal parameters. o After observing the untrained target object, the original depth image is obtained. Then, combined with the geometric prior knowledge base Q, the 6D pose of the original depth image is estimated by the pre-trained 6D pose estimation network to obtain the 6D pose estimation result of the current untrained target object. Specifically, S3 is: S31: During the online phase, the observation camera C utilizes known intrinsic parameters. o After observing untrained target objects, an observation depth map is obtained. After target segmentation and image preprocessing of the observation depth map, a processed depth map is obtained. S32: Input the processed depth map into the pre-trained feature extraction network and view encoder to obtain the observation feature map F. o and observation viewpoint feature v o ; S33: The observation viewpoint feature symbol v o and prior viewpoint feature symbol set After being input together into the pre-trained viewpoint interaction attention module for feature exchange and optimization, the optimized observation viewpoint feature symbol v′ is obtained. o With the optimized prior viewpoint feature symbol set Update the geometric prior knowledge base and denote it as Q′; S34: First, the template matching method is used to retrieve the optimized observation viewpoint feature v′ from the geometric prior knowledge base Q′. o The closest prior viewpoint feature is used as the matching object, and the depth reference image corresponding to the matching object is recorded as the matching reference image. Then match the reference image out-of-plane rotation As an initial out-of-plane rotation estimate for untrained target objects in the observed depth map Next, based on the observed feature map F o Matching reference image Calculate the in-plane rotation of untrained target objects in the observed depth map. and displacement components This yields the initial 6D pose estimation results for the untrained target object in the observed depth map; S35: Align the initial 6D pose estimation results of the untrained target object in the observation depth map with a non-uniform viewpoint to obtain the final 6D pose estimation result.

2. The generalizable 6D pose estimation method for untrained targets according to claim 1, characterized in that, In step S34, the template matching method is used to retrieve and optimize the observation viewpoint feature symbol v′ from the geometric prior knowledge base Q′. o The closest prior viewpoint features include: The optimized observation viewpoint feature symbol v′ is calculated using the template matching method. o The feature symbol of each prior viewpoint in the geometric prior knowledge base Q′ The cosine similarity is used to determine the prior viewpoint feature with the highest cosine similarity. As the closest prior viewpoint feature.

3. The generalizable 6D pose estimation method for untrained targets according to claim 1, characterized in that, In S34, based on the observed feature map F o Matching reference image Calculate the in-plane rotation of untrained target objects in the observed depth map, including: Match reference image The feature map generated after feature extraction by the pre-trained feature extraction network is denoted as the matching feature map. From the observed feature map F o and matching feature maps A feature map is constructed and input into a pre-trained in-plane rotation prediction head. The in-plane rotation prediction head directly regresses the estimated in-plane rotation amount and uses it as the in-plane rotation amount of the untrained target object in the observed depth map.

4. The generalizable 6D pose estimation method for untrained targets according to claim 1, characterized in that, In S34, based on the observed feature map F o Matching reference image Calculate the displacement components of untrained target objects in the observed depth map, including: First, the observation camera C is calculated using the following formula. o Distance between the center of the untrained target object Among them, f o f vir For example, observation camera C o With virtual camera C vir focal length, t z,m To match the reference image Objects and virtual camera C vir The distance, l m l o Matching reference images The length of the diagonal of the 2D bounding box of an object in the depth map; Then, based on the observation camera C o internal reference K o And the displacement components are solved using the back projection method, based on the center of the 2D bounding box of the untrained target object in the depth map. The formula is as follows: Among them, c x ,c y The x and y coordinates of the center of the 2D bounding box of the untrained target object in the observation depth map are given, and T is the transpose operation.

5. The generalizable 6D pose estimation method for untrained targets according to claim 1, characterized in that, Specifically, S35 is: S351: To the observation camera C o The vector γ from the optical center to the center of the observed untrained target object applies an in-plane rotation of the untrained target object in the depth measurement map. Get around C o Vector after optical axis rotation The formula is as follows: S352: Solve for the deviation matrix R using the following formula c : Where I is a 3×3 identity matrix. Let be the antisymmetric matrix of the vector; Ψ() is the Rodrigo rotation; S353: Using the deviation matrix R c Eliminate initial out-of-plane rotation estimation of untrained target objects in the observed depth map Non-uniform viewpoint bias was analyzed to obtain the final 6D pose estimation result. The formula is as follows:

6. The generalizable 6D pose estimation method for untrained targets according to claim 1, characterized in that, In S1, the depth image obtained by rendering each 3D model specifically includes K APN samples. The K APN samples specifically include K depth images, K depth images with in-plane rotation applied, and K×M depth images with out-of-plane rotation applied.

7. The generalizable 6D pose estimation method for untrained targets according to claim 6, characterized in that, In S1, the view encoder is pre-trained using contrastive learning. During the pre-training process, the feature map corresponding to the depth image is used as the benchmark, the feature map corresponding to the depth image with in-plane rotation is used as the positive example, and the feature map corresponding to the depth image with out-of-plane rotation is used as the negative example.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the generalizable 6D pose estimation method for an untrained target as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the generalizable 6D pose estimation method for an untrained target as described in any one of claims 1 to 7.

Images