Generalized 6D pose estimation method for untrained target

Abstract

The invention discloses a generalization 6D pose estimation method for an untrained target. The method comprises the following steps: pre-training a 6D pose estimation network by using a training data set consisting of depth images obtained by rendering a multi-class 3D model; in the off-line stage, the pre-trained network and the target object 3D model are utilized to generate a geometric priori knowledge base for an untrained target object; an original depth image is obtained after an untrained target object is observed by using an observation camera Co, and a 6D pose estimation result of the current untrained target object is obtained after 6D pose estimation is performed on the original depth image through a pre-trained 6D pose estimation network in combination with a geometric priori knowledge base Q. According to the invention, generalization 6D pose estimation for the untrained target is realized, the problems caused by inaccurate viewpoint coding and non-uniform prior information view angle and observation information view angle are solved, and efficient and accurate 6D pose estimation for the untrained target can be realized without adjusting any pre-training network parameters.

Description

Technical Field

[0001] The present invention relates to a 6D pose estimation method in the field of computer vision technology, and in particular to a generalizable 6D pose estimation method for untrained targets. Background Art

[0002] 6D pose estimation is a very important part in applications such as robotics and augmented reality. In recent years, many 6D pose estimation methods based on deep learning have achieved excellent prediction results on public datasets. However, these algorithms cannot directly estimate objects that the network has not encountered during the training phase (hereinafter referred to as untrained targets, also known as unseen objects). They need to spend time and energy to build datasets for these untrained targets and consume a lot of computing power and time for retraining to achieve 6D pose estimation for these objects. This seriously hinders the application of deep learning algorithms in the real world.

[0003] At present, there are some 6D pose estimation methods that can directly estimate untrained targets without changing model parameters. The specific method is to first render a reference image under multiple viewpoints through a known untrained target 3D model, and use a viewpoint encoder to generate a priori viewpoint feature set for the reference image; then generate an observation viewpoint feature for the untrained target in the observation image; use template matching to find the closest reference viewpoint for the observation viewpoint feature, and obtain the 6D pose of the untrained target in the observation image based on the pose information in the matched reference image.

[0004] However, this method does not interact viewpoint features with geometric prior knowledge. Since the novel appearance of some untrained targets may not have been learned by the network, it is difficult for the network to accurately encode viewpoint information based on a single observation image, which affects the correctness of subsequent template matching and the accuracy of 6D pose estimation. On the other hand, these methods do not consider the problem of non-uniform perspectives: an object with the same rotation component relative to the camera has different appearances when photographed under a standard perspective (the camera optical axis passes through the center of the object) and a non-standard perspective (the camera optical axis does not pass through the center of the object). Since the camera optical axis is often deviated from the center of the object when observing (i.e., non-standard perspective), and the reference image is photographed (rendered) under a standard perspective, the object in the observation image and the object in the matched reference image have the same appearance, but have different 3D rotation components, which will lead to obvious errors in the final 6D pose estimation. Summary of the invention

[0005] In order to solve the problems existing in the background technology, the present invention proposes a generalizable 6D pose estimation method for untrained targets, which solves the problems of inaccurate viewpoint encoding and misalignment of observation information perspective with prior information perspective in the estimation process, and realizes efficient and accurate prediction of the 6D pose of untrained targets without changing any parameters in the network.

[0006] The specific technical solutions of the present invention are as follows:

[0007] 1. A generalizable 6D pose estimation method for untrained objects

[0008] S1: constructing a 6D pose estimation network, wherein the 6D pose estimation network includes a feature extraction network, a viewpoint encoder, a viewpoint interaction attention module, and an in-plane rotation prediction head connected in sequence, and pre-training the 6D pose estimation network using a training data set consisting of depth images obtained by rendering a multi-category 3D model to obtain a pre-trained 6D pose estimation network;

[0009] S2: In the offline stage, the known 3D model of the untrained target object is rendered with a multi-view depth reference image to obtain the corresponding multi-view depth reference image, with the number of viewpoints being N. The multi-view depth reference image is then scaled and cropped and sent to the pre-trained feature extraction network and viewpoint encoder to obtain the prior viewpoint feature set. The out-of-plane rotation of the 3D model of the depth reference image and the virtual camera And in the prior viewpoint feature set The corresponding prior viewpoint signature in Constitute a geometric prior knowledge base Q;

[0010] S3: Inference stage, using the observation camera C with known internal parameters o After observing the untrained target object, the original depth image is obtained. Then, combined with the geometric prior knowledge base Q, the 6D pose of the original depth image is estimated through the pre-trained 6D pose estimation network to obtain the 6D pose estimation result of the current untrained target object.

[0011] The S3 is specifically:

[0012] S31: In the online stage, the observation camera C with known internal parameters is used o After observing the untrained target object, an observed depth map is obtained, and after performing target segmentation and image preprocessing on the observed depth map, a processed depth map is obtained;

[0013] S32: Input the processed depth map into the pre-trained feature extraction network and viewpoint encoder to obtain the observation feature map F o and the observation point signature v o ;

[0014] S33: The observation viewpoint feature v o and the prior viewpoint signature set After being input into the pre-trained viewpoint interaction attention module for feature exchange and optimization, the optimized observation viewpoint feature symbol v′ is obtained. o and the optimized prior viewpoint feature set Update the geometric prior knowledge base and record it as Q′;

[0015] S34: First, the template matching method is used to retrieve the optimized observation viewpoint feature v′ in the geometric prior knowledge base Q′ o The closest prior viewpoint feature is used as the matching object and the depth reference image corresponding to the matching object is recorded as the matching reference image. Then match the reference image The out-of-plane rotation As the initial out-of-plane rotation estimate of the untrained target object in the observed depth map Then according to the observed characteristic graph F o Matching reference image Calculate the in-plane rotation of untrained target objects in the observed depth map and displacement components Thus, the initial 6D pose estimation result of the untrained target object in the observed depth map is obtained;

[0016] S35: Perform non-uniform perspective alignment on the initial 6D pose estimation result of the untrained target object in the observed depth map to obtain the final 6D pose estimation result.

[0017] In S34, the optimized observation viewpoint feature v′ is retrieved from the geometric prior knowledge base Q′ using a template matching method. o The closest prior viewpoint signatures include:

[0018] The optimized observation viewpoint signature v′ is calculated using the template matching method o Each prior viewpoint feature in the geometric prior knowledge base Q′ The cosine similarity of the prior viewpoint feature with the highest cosine similarity As the closest prior viewpoint feature.

[0019] In S34, according to the observed characteristic graph F o Matching reference image Calculate the in-plane rotation of the untrained target object in the observed depth map, including:

[0020] Will match the reference image The feature map generated after feature extraction by the pre-trained feature extraction network is recorded as the matching feature map From the observed characteristic graph F o and matching feature maps A feature map is formed and input into the pre-trained in-plane rotation prediction head, which directly regresses the estimated value of the in-plane rotation and uses it as the in-plane rotation of the untrained target object in the observed depth map.

[0021] In S34, according to the observed characteristic graph F o Matching reference image Calculate the displacement components of the untrained target objects in the observed depth map, including:

[0022] First, the observation camera C is calculated using the following formula o The distance from the center of the untrained target object

[0023]

[0024] Among them, f o 、f vir They are observation cameras C o With virtual camera C vir The focal length, t z,m To match the reference image The object and the virtual camera C vir The distance, l m , l o Match the reference image Observe the diagonal length of the 2D bounding box of the object in the depth map;

[0025] According to the observation camera C o The internal reference K o And observe the center of the 2D bounding box of the untrained target object in the depth map, and use the back projection method to solve the displacement component The formula is as follows:

[0026]

[0027] Among them, c x ,c y are the horizontal and vertical coordinate values ​​of the center of the 2D bounding box of the untrained target object in the observed depth map, and T is the transposition operation.

[0028] The S35 is specifically:

[0029] S351: Towards observation camera C o The vector γ from the optical center to the center of the observed untrained target object applies the in-plane rotation of the untrained target object in the depth map Get around C o The vector after the optical axis is rotated The formula is as follows:

[0030]

[0031] S352: Solve the deviation matrix R using the following formula c :

[0032]

[0033] Where I is the 3×3 identity matrix, is the antisymmetric matrix of the vector; Ψ() is the Rodrigo rotation;

[0034] S353: Using the deviation matrix R c Eliminate initial out-of-plane rotation estimates of untrained target objects in observed depth maps The non-uniform view deviation in the final 6D pose estimation result is obtained The formula is as follows:

[0035]

[0036] In S1, each depth image obtained by rendering the 3D model specifically includes K APN samples, and the K APN samples specifically include K depth images, K depth images with in-plane rotation applied, and K×M depth images with out-of-plane rotation applied.

[0037] In S1, the viewpoint encoder is pre-trained using contrastive learning. During the pre-training process, the feature map corresponding to the depth image is used as a reference, the feature map corresponding to the depth image with an in-plane rotation amount applied is used as a positive example, and the feature map corresponding to the depth image with an out-of-plane rotation amount applied is used as a negative example.

[0038] 2. A computer device

[0039] The device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the generalizable 6D pose estimation method for untrained targets when executing the computer program.

[0040] 3. A computer-readable storage medium

[0041] The medium stores a computer program, which, when executed by a processor, implements the steps of the generalizable 6D pose estimation method for untrained targets.

[0042] The beneficial effects of the present invention are:

[0043] (1) The method described in the present invention can realize 6D pose estimation of any rigid object including untrained targets without changing the neural network parameters, solving the problem that deep learning algorithms need to spend a lot of manpower and computing power to construct untrained target data sets and retrain the network when estimating unseen objects.

[0044] (2) The present invention uses large-scale multi-category virtual data in the network pre-training process and adopts InfoNCEloss for comparative learning, which improves the neural network's encoding ability and generalization of viewpoint information.

[0045] (3) The present invention solves the inaccuracy caused by encoding only using observed images through a viewpoint interaction attention module, promotes communication between viewpoints by introducing geometric prior knowledge, and achieves more accurate viewpoint encoding.

[0046] (4) The perspective alignment module proposed in the present invention effectively eliminates the out-of-plane rotation estimation error caused by the inconsistency between the observation information perspective and the prior information perspective during the template matching process, thereby optimizing the 6D pose estimation accuracy.

[0047] In summary, when estimating the 6D pose of an untrained target, the present invention can avoid any change of network parameters, generate more accurate viewpoint encoding for the observed image, and eliminate the estimation error caused by inconsistent viewing angles, thus having good engineering practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the 6D pose estimation method in the present invention.

[0049] Figure 2 It is a network structure diagram constructed by the 6D pose estimation method in the present invention.

[0050] Figure 3 It is a schematic diagram of multi-viewpoint depth reference image rendering in the present invention.

[0051] Figure 4 It is a schematic diagram of the perspective alignment module in the present invention aligning non-uniform perspectives.

[0052] Figure 5 It is an untrained industrial general purpose part used in the embodiments of the present invention.

[0053] Figure 6 It is a visualization result of 6D pose estimation of untrained industrial general parts in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments by applying the method described in the present invention to the 6D pose estimation of untrained industrial general parts. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] like Figure 1 As shown, the present invention provides a generalizable 6D pose estimation method for an untrained target, comprising the following steps:

[0056] S1: Construct a 6D pose estimation network, such as Figure 2 As shown, the 6D pose estimation network includes a feature extraction network, a viewpoint encoder, a viewpoint interaction attention module and an in-plane rotation prediction head connected in sequence. The 6D pose estimation network is pre-trained on a large scale using a training data set consisting of depth images obtained by rendering multi-category 3D models to obtain a pre-trained 6D pose estimation network. Even if the pre-trained model encounters an untrained target in the inference stage, it can still achieve accurate 6D pose estimation without retraining.

[0057] The feature extraction network is a VGG-11 network, which is used to extract image features. The 128×128 pixel depth image is used as input. The feature map is the output;

[0058] The viewpoint encoder is used to encode the viewpoint feature, which is composed of a CBAM self-attention module, a two-dimensional convolutional layer, a pooling layer and a fully connected layer. It takes the feature map F as input and the viewpoint feature is is the output;

[0059] The viewpoint interaction attention module is a scaled dot product self-attention model, which is used to optimize the viewpoint signature, a tensor concatenated from the viewpoint signatures under S different viewpoints. As input, take the optimized tensor As output;

[0060] The in-plane rotation prediction head is used to estimate the in-plane rotation amount. Its network structure is a two-dimensional convolutional layer and two consecutive fully connected layers. It takes the feature map F as input and estimates the in-plane rotation matrix. is the output;

[0061] Random sampling is performed from the original ShapeNet dataset to form a pre-training dataset with 10,000 models. A large amount of virtual depth data is generated from these 10,000 models as training data. These training data are used to perform large-scale pre-training of the entire 6D pose estimation network.

[0062] The steps for generating each set of training data during network pre-training are as follows:

[0063] S1-1: Randomly select a 3D model from the ShapeNet dataset, and evenly select 16 standard viewpoints in its hemisphere space (the virtual camera optical axis is aligned with the center of the 3D object), and render 16 depth images as the benchmark for comparative learning. i (i=1,2,...16);

[0064] S1-2: for all benchmarks A i Apply random in-plane rotation (rotate the virtual camera around the optical axis) and render to obtain 16 depth images with in-plane rotation applied as positive examples P for comparative learning i ;

[0065] S1-3: For each benchmark A i Apply 5 random out-of-plane rotations (change the camera's viewpoint relative to the 3D model) and render as a counterexample for comparative learning i,j , (j=1,2,...,5), forming 16 APN (Anchor-Positive-Negative Sample) samples, APN i ={A i ,P i ,NG i,j};

[0066] The viewpoint encoder is pre-trained using contrastive learning. For the kth (1≤k≤16) APN sample in each iteration, the steps involved in encoder training are as follows:

[0067] S1-4: Use the MaskRCNN instance segmentation algorithm to segment each depth image in the kth APN sample, and change the resolution of the segmented image to 128×128, input it into the feature extraction network, and obtain the feature map F k A ,F k P ,F k,j NG (j=1,2,...,5);

[0068] S1-5: The feature map F k A ,F k P ,F k,j NG Input the viewpoint encoder to obtain the viewpoint feature v corresponding to the benchmark, positive example and negative example of the encoded viewpoint information k A ,v k P ,v k,jNG ;

[0069] S1-6: Using InfoNCE Loss as the encoder’s contrastive learning loss function L InfoNCE :

[0070]

[0071] in, S cos () is used to calculate the cosine similarity between feature vectors x and y of the same dimension, τ is the temperature coefficient, set to 0.1.

[0072] The training steps of the in-plane rotation prediction head are as follows:

[0073] S1-7: Calculate feature map F k A With F k P Correlation heatmap matrix between The correlation heat map matrix H θ Input to the in-plane rotation prediction head, where represents a real number set with a channel number of 512×8×8;

[0074] S1-8: In-plane rotation prediction head direct regression benchmark A k 、Positive Example P k The in-plane rotation between

[0075] S1-9: Calculate the prediction loss L of the in-plane rotation prediction head using the following loss function θ :

[0076] L θ = -ln((1.0+ζ) / 2.0),

[0077]

[0078] Among them, ζ is the R measured by cosine similarity θ and The difference value, S cos () is the calculation function of cosine similarity, Represents the benchmark A k Apply in-plane rotation R θ The 2D space rotation, R θ A k With the positive example P k The true value of the in-plane rotation between Flatten operation for tensors.

[0079] The large-scale pre-training of the entire 6D pose estimation network adopts an end-to-end training method, and the overall loss function is:

[0080] L = α(λ1L InfoNCE +λ2L θ )

[0081] Among them, λ1 and λ2 are two loss weight factors, λ1=10, λ2=1.5, and α is a fixed proportional factor with a value of 10.

[0082] The entire training process was carried out in Pycharm. The network parameter optimizer used the adaptive moment estimation optimizer Adam. The training rounds were set to 50 times and the initial learning rate was 10 -5 , the final learning rate is 10 -8 .

[0083] S2: In the offline stage, a multi-view depth reference image is rendered for a known 3D model of an untrained target object to obtain a corresponding multi-view depth reference image. The number of viewpoints is N. Specifically, Figure 3 As shown, the virtual camera C vir A multi-view depth reference image is achieved by uniformly selecting N standard viewpoints in the hemisphere space of the known 3D model of the target object. In order to reduce the repeated appearance features in Q, N=5000 is set for asymmetric objects; for general symmetrical objects, the viewpoints are uniformly sampled on one side of the symmetry axis, and N=2500 is set; for rotational objects, the viewpoints are uniformly sampled on a generatrix in the hemispherical space, and N=60 is set.

[0084] Then, the multi-view depth reference image is scaled and cropped and then fed into the pre-trained feature extraction network and viewpoint encoder to obtain the prior viewpoint feature set. Specifically, the depth reference image As input, according to the maximum diameter of the object and C vir Internal and external parameter acquisition The 2D bounding box of the object is cropped and the cropped image resolution is adjusted to 128×128 pixels. Then, the feature extraction network and viewpoint encoder are used to extract the viewpoint feature. is the output. By the depth reference image and the out-of-plane rotation of the 3D model of the depth reference image relative to the virtual camera And in the prior viewpoint feature set The corresponding prior viewpoint signature in Constitute a geometric prior knowledge base Q, satisfying

[0085] S3: In the online stage, the observation camera C with known internal parameters is usedo After observing the untrained target object, the original depth image is obtained. Then, combined with the geometric prior knowledge base Q, the 6D pose of the original depth image is estimated through the pre-trained 6D pose estimation network to obtain the 6D pose estimation result of the current untrained target object.

[0086] S3 is specifically:

[0087] S31: Inference stage, using the observation camera C with known internal parameters o After observing the untrained target object, an observed depth map is obtained. After performing target segmentation and image preprocessing on the observed depth map, a processed depth map is obtained. The image preprocessing includes: using the MaskRCNN instance segmentation algorithm to segment the target object in the observed depth map to obtain the corresponding target object 2D bounding box and segmentation mask; and using the 2D bounding box to crop the observed depth image, and filtering the depth value except the depth information of the target object through the segmentation mask, and adjusting the cropped image resolution to 128×128 pixels.

[0088] S32: Input the processed depth map into the pre-trained feature extraction network and viewpoint encoder to obtain the observation feature map F o and the observation point signature v o , observation feature map F o is the feature map generated after the processed depth map is extracted by the pre-trained feature extraction network; among them, the depth camera C observing the general part o The internal parameter matrix is ​​as follows:

[0089]

[0090] S33: The observation viewpoint feature v o and the prior viewpoint signature set After being input into the pre-trained viewpoint interaction attention module for feature exchange and optimization, the optimized observation viewpoint feature symbol v′ is obtained. o and the optimized prior viewpoint feature set Update the geometric prior knowledge base and record it as Q′, that is, replace the prior viewpoint feature set in the geometric prior knowledge base Q

[0091] Specifically:

[0092] First, the viewpoint feature v encoded by the observed image o and Concatenate them to form a tensor Z of 64×S dimensions, where S=N+1.

[0093] Then Z is used as the query, key, and value at the same time and input into the viewpoint interaction attention module.

[0094] Then perform the scaled dot product self-attention operation:

[0095]

[0096] Among them, q, K, and V are respectively linearly transformed (both are fully connected layers) query, key and value, d k is the dimension of the viewpoint signature, the value is 64, and Z′ is the optimized tensor. This operation realizes the communication between the observation viewpoint information and the prior viewpoint information, and outputs the optimized observation viewpoint signature v′ o Prior viewpoint signature set And update the geometric prior knowledge base as

[0097] S34: Estimate the initial 6D pose using disentanglement: First, use the template matching method to retrieve the optimized observation viewpoint feature v′ in the geometric prior knowledge base Q′ o The closest prior viewpoint feature is used as the matching object and the depth reference image corresponding to the matching object is recorded as the matching reference image. Then match the reference image The out-of-plane rotation As the initial out-of-plane rotation estimate of the untrained target object in the observed depth map Then according to the observed characteristic graph F o Matching reference image Calculate the in-plane rotation of untrained target objects in the observed depth map and displacement components Thus, the initial 6D pose estimation result of the untrained target object in the observed depth map is obtained;

[0098] In S34, the template matching method is used to retrieve the optimized observation viewpoint feature v′ in the geometric prior knowledge base Q′. o The closest prior viewpoint signatures include:

[0099] The optimized observation viewpoint signature v′ is calculated using the template matching method o Each prior viewpoint feature in the geometric prior knowledge base Q′ The cosine similarity of the prior viewpoint feature with the highest cosine similarity As the closest prior viewpoint feature.

[0100] In S34, according to the observed characteristic graph F o Matching reference image Calculate the in-plane rotation of the untrained target object in the observed depth map, including:

[0101] Will match the reference image The feature map generated after feature extraction by the pre-trained feature extraction network is recorded as the matching feature map From the observed characteristic graph F o and matching feature maps A feature map is formed and input into the pre-trained in-plane rotation prediction head, which directly regresses the estimated value of the in-plane rotation and uses it as the in-plane rotation of the untrained target object in the observed depth map. The initial 3D rotation component estimate can be expressed as:

[0102] In S34, according to the observed characteristic graph F o Matching reference image Calculate the displacement components of the untrained target objects in the observed depth map, including:

[0103] First, the observation camera C is calculated using the following formula o The distance from the center of the untrained target object

[0104]

[0105] Among them, f o 、f vir They are observation cameras C o With virtual camera C vir The focal length, t z,m To match the reference image The object and the virtual camera C vir The distance, l m , l o Match the reference image Observe the diagonal length of the 2D bounding box of the object in the depth map;

[0106] According to the observation camera C o The internal reference K o And observe the center of the 2D bounding box of the untrained target object in the depth map, and use the back projection method to solve the displacement component The formula is as follows:

[0107]

[0108] Among them, c x ,c y are the horizontal and vertical coordinate values ​​of the center of the 2D bounding box of the untrained target object in the observed depth map, and T is the transposition operation.

[0109] Therefore, the initial 6D pose estimation result of the untrained target object in the observed depth map can be expressed as

[0110] S35: Use the perspective alignment module to perform non-uniform perspective alignment on the initial 6D pose estimation results of the untrained target object in the observed depth map, eliminate the out-of-plane rotation error caused by this factor, and obtain the final 6D pose estimation result. The perspective alignment module mainly solves the problem of inconsistent perspectives between the observation image taken under non-standard perspectives (the optical axis of the real camera does not pass through the center of the observed object) and the reference image rendered under standard perspectives, and eliminates the out-of-plane rotation estimation error caused by template matching under the premise of inconsistent perspectives. The schematic diagram of the perspective alignment module eliminating the out-of-plane rotation estimation error caused by non-uniform perspectives is shown in Figure 4 shown.

[0111] S35 is specifically:

[0112] S351: Towards observation camera C o The vector γ from the optical center to the center of the observed untrained target object applies the in-plane rotation of the untrained target object in the depth map Get around C o The vector after the optical axis is rotated The formula is as follows:

[0113]

[0114] S352: Solve the deviation matrix R using the following formula c , to align the vectors With observation camera C o The optical axis is [0,0,1] T :

[0115]

[0116] Where I is the 3×3 identity matrix, is the antisymmetric matrix of the vector; Ψ() is the Rodrigo rotation.

[0117] S353: Using the deviation matrix R c Eliminate initial out-of-plane rotation estimates of untrained target objects in observed depth maps The non-uniform view deviation in the final 6D pose estimation result is obtained The formula is as follows:

[0118]

[0119] in, To eliminate the out-of-plane rotation after non-uniform viewing angle deviation,

[0120] The present invention does not change any network parameters. Figure 5 The 8 untrained industrial common parts shown in the figure have been subjected to 6D pose estimation. The pose estimation visualization results are shown in Figure 6 In. Figure 5 and Figure 6 As shown, the present invention achieves accurate 6D pose estimation for multiple untrained industrial common parts that deviate from the camera center in three different industrial scenarios without changing any network parameters.

[0121] An embodiment of the present invention provides a generalizable 6D pose estimation device for an untrained target, comprising one or more processors, for implementing a generalizable 6D pose estimation method for an untrained target in the above embodiment.

[0122] An embodiment of a generalizable 6D pose estimation device for untrained targets of the present invention can be applied to any device with data processing capabilities, and the arbitrary device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located to read the corresponding computer program instructions in the non-volatile memory into the memory and run it. From the hardware level, it is a hardware structure diagram of any device with data processing capabilities in which a generalizable 6D pose estimation device for untrained targets of the present invention is located. In addition to the processor, memory, network interface, and non-volatile memory, the manufacturing of any device with data processing capabilities in which the device in the embodiment is located is usually based on the actual function of the arbitrary device with data processing capabilities. It can also include other hardware, which will not be repeated here.

[0123] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0124] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] An embodiment of the present invention further provides a readable storage medium having a program stored thereon. When the program is executed by a processor, a generalizable 6D pose estimation method for an untrained target in the above embodiment is implemented.

[0126] The readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0127] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A generalizable 6D pose estimation method for untrained targets, characterized in that: The following steps are involved: S1: constructing a 6D pose estimation network, wherein the 6D pose estimation network includes a feature extraction network, a viewpoint encoder, a viewpoint interaction attention module, and an in-plane rotation prediction head connected in sequence, and pre-training the 6D pose estimation network using a training data set consisting of depth images obtained by rendering a multi-category 3D model to obtain a pre-trained 6D pose estimation network; S2: In the offline stage, the known 3D model of the untrained target object is rendered with a multi-view depth reference image to obtain the corresponding multi-view depth reference image, with the number of viewpoints being N. The multi-view depth reference image is then scaled and cropped and sent to the pre-trained feature extraction network and viewpoint encoder to obtain the prior viewpoint feature set. The out-of-plane rotation of the 3D model of the depth reference image and the virtual camera And in the prior viewpoint feature set The corresponding prior viewpoint signature in Constitute a geometric prior knowledge base Q; S3: Inference stage, using the observation camera C with known internal parameters o After observing the untrained target object, the original depth image is obtained. Then, combined with the geometric prior knowledge base Q, the 6D pose of the original depth image is estimated through the pre-trained 6D pose estimation network to obtain the 6D pose estimation result of the current untrained target object.

2. A generalizable 6D pose estimation method for untrained targets according to claim 1, characterized in that: The S3 is specifically: S31: In the online stage, the observation camera C with known internal parameters is used o After observing the untrained target object, an observed depth map is obtained, and after performing target segmentation and image preprocessing on the observed depth map, a processed depth map is obtained; S32: Input the processed depth map into the pre-trained feature extraction network and viewpoint encoder to obtain the observation feature map F o and the observation viewpoint signature v o ; S33: The observation viewpoint feature v o and the prior viewpoint signature set After being input into the pre-trained viewpoint interaction attention module for feature exchange and optimization, the optimized observation viewpoint feature symbol v′ is obtained. o and the optimized prior viewpoint feature set Update the geometric prior knowledge base and record it as Q′; S34: First, the template matching method is used to retrieve the optimized observation viewpoint feature v′ in the geometric prior knowledge base Q′ o The closest prior viewpoint feature is used as the matching object and the depth reference image corresponding to the matching object is recorded as the matching reference image. Then match the reference image The out-of-plane rotation As the initial out-of-plane rotation estimate of the untrained target object in the observed depth map Then according to the observed feature map F o Matching reference image Calculate the in-plane rotation of untrained target objects in the observed depth map and displacement components Thus, the initial 6D pose estimation result of the untrained target object in the observed depth map is obtained; S35: Perform non-uniform perspective alignment on the initial 6D pose estimation result of the untrained target object in the observed depth map to obtain the final 6D pose estimation result.

3. A generalizable 6D pose estimation method for untrained targets according to claim 2, characterized in that: In S34, the optimized observation viewpoint feature v′ is retrieved from the geometric prior knowledge base Q′ using a template matching method. o The closest prior viewpoint signatures include: The optimized observation viewpoint signature v′ is calculated using the template matching method o Each prior viewpoint feature in the geometric prior knowledge base Q′ The cosine similarity of the prior viewpoint feature with the highest cosine similarity As the closest prior viewpoint feature.

4. The generalizable 6D pose estimation method for untrained targets according to claim 2, characterized in that: In S34, according to the observed characteristic graph F o Matching reference image Calculate the in-plane rotation of the untrained target object in the observed depth map, including: Will match the reference image The feature map generated after feature extraction by the pre-trained feature extraction network is recorded as the matching feature map From the observed characteristic graph F o and matching feature maps A feature map is formed and input into the pre-trained in-plane rotation prediction head, which directly regresses the estimated value of the in-plane rotation and uses it as the in-plane rotation of the untrained target object in the observed depth map.

5. The generalizable 6D pose estimation method for untrained targets according to claim 2, characterized in that: In S34, according to the observed characteristic graph F o Matching reference image Calculate the displacement components of the untrained target objects in the observed depth map, including: First, the observation camera C is calculated using the following formula o The distance from the center of the untrained target object Among them, f o 、f vir They are observation cameras C o With virtual camera C vir The focal length, t z,m To match the reference image The object and the virtual camera C vir The distance, l m , l o Match the reference image Observe the diagonal length of the 2D bounding box of the object in the depth map; According to the observation camera C o The internal reference K o And observe the center of the 2D bounding box of the untrained target object in the depth map, and use the back projection method to solve the displacement component The formula is as follows: Among them, c x ,c y are the horizontal and vertical coordinate values ​​of the center of the 2D bounding box of the untrained target object in the observed depth map, and T is the transposition operation.

6. A generalizable 6D pose estimation method for untrained targets according to claim 2, characterized in that: The S35 is specifically: S351: Towards observation camera C o The vector γ from the optical center to the center of the observed untrained target object applies the in-plane rotation of the untrained target object in the depth map Get around C o The vector after the optical axis is rotated The formula is as follows: S352: Solve the deviation matrix R using the following formula c : Where I is the 3×3 identity matrix, is the antisymmetric matrix of the vector; Ψ() is the Rodrigo rotation; S353: Using the deviation matrix R c Eliminate initial out-of-plane rotation estimates of untrained target objects in observed depth maps The non-uniform view deviation in the final 6D pose estimation result is obtained The formula is as follows:

7. The generalizable 6D pose estimation method for untrained targets according to claim 1, characterized in that: In S1, each depth image obtained by rendering the 3D model specifically includes K APN samples, and the K APN samples specifically include K depth images, K depth images with in-plane rotation applied, and K×M depth images with out-of-plane rotation applied.

8. A generalizable 6D pose estimation method for untrained targets according to claim 7, characterized in that: In S1, the viewpoint encoder is pre-trained using contrastive learning. During the pre-training process, the feature map corresponding to the depth image is used as a reference, the feature map corresponding to the depth image with an in-plane rotation amount applied is used as a positive example, and the feature map corresponding to the depth image with an out-of-plane rotation amount applied is used as a negative example.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the generalizable 6D pose estimation method for untrained targets described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the generalizable 6D pose estimation method for untrained targets described in any one of claims 1 to 8 are implemented.

Images