Transparent object six-degree-of-freedom pose estimation method

Abstract

The application provides a six-degree-of-freedom pose estimation method for a transparent object, and the implementation steps are as follows: obtaining a training sample set and a test sample set; constructing a six-degree-of-freedom pose estimation network model for the transparent object and iteratively training the model; and obtaining a six-degree-of-freedom pose estimation result for the transparent object. In the process of training the pose estimation network model and obtaining the six-degree-of-freedom pose estimation result for the transparent object, the coarse estimation module and the fine estimation module are used to directly regress the RGB image and the depth image of the transparent object to obtain the six-degree-of-freedom pose of the transparent object, and the calculation process is simple, the image information loss is small, and the fine estimation module is used to fuse the edge features of each depth image and the edge features of the rendered depth image of the corresponding CAD model of the transparent object, so that the fine estimation of the coarse estimation result is realized, the source of the depth information is enriched, the error of the depth information is reduced, and the estimation accuracy and efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of machine vision, and relates to a six-degree-of-freedom pose estimation method for transparent objects, in particular to a six-degree-of-freedom pose estimation method for transparent objects based on deep learning, which can be applied to the fields of robot navigation and automatic driving. BACKGROUND

[0002] In the field of robots and machine vision, six-degree-of-freedom pose estimation of a target object is a key problem, which provides accurate spatial position and direction information for the operation of the target object, including three translational degrees of freedom and three rotational degrees of freedom of the object relative to the camera; however, for the transparent objects commonly seen in daily life, due to the optical characteristics of weak texture and strong specular reflection, the depth image of the transparent object obtained by the sensor has errors, and it is difficult to accurately estimate the six-degree-of-freedom pose of the transparent object.

[0003] At present, the six-degree-of-freedom pose estimation methods for transparent objects mainly include two categories: key point matching-based estimation methods and deep learning-based estimation methods, wherein the estimation accuracy of the key point matching-based estimation methods is poor; the deep learning-based estimation methods use deep learning methods to estimate the six-degree-of-freedom pose of the target object from the input RGB image or depth image, which takes into account the calculation accuracy and calculation efficiency, and gradually becomes a research hotspot.

[0004] With the development of deep learning technology, more and more deep learning-based estimation methods are applied to the six-degree-of-freedom pose estimation of transparent objects, and the accuracy and computational efficiency of the six-degree-of-freedom pose estimation of transparent objects are continuously improved. For example, the patent application with the application publication number CN 113313810 A and the title of "a 6D pose parameter calculation method of a transparent object" discloses a 6D pose parameter calculation method of a transparent object based on deep learning. The invention first collects the original RGB image and the original depth image of the transparent object, then uses a fully convolutional network to extract low-level and high-level features in the original RGB image, and generates a transparent object segmentation map of the original RGB image based on the high-level features through an attention mechanism. At the same time, the original RGB image is processed through a Deeplabv3 network to obtain the surface normal map of the transparent object. Next, the obtained segmentation map and surface normal map are used to mask pre-process and globally optimize the original depth image respectively to obtain the corrected depth image. Finally, the pixel-level dense feature fusion model proposed by the method is used to estimate the six-degree-of-freedom pose parameters of the transparent object by taking the corrected depth image as the input. The invention has high estimation accuracy. However, the invention estimates the six-degree-of-freedom pose by optimizing the depth image of the transparent object through the segmentation map and the surface normal map, which increases the computational complexity and causes the loss of image information. Moreover, the invention corrects the error in the depth information by pre-processing and globally optimizing the original depth image, but these processes are still based on the original depth image with errors, which affects the further improvement of the estimation accuracy. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a six-degree-of-freedom pose estimation method for transparent objects to solve the technical problem of low six-degree-of-freedom pose estimation accuracy of transparent objects in the prior art.

[0006] To achieve the above purpose, the technical solution adopted by the present application includes the following steps:

[0007] (1) Obtain a training sample set and a test sample set:

[0008] Obtain a training sample set B train including RGB images and six-degree-of-freedom pose labels of P transparent objects, as well as depth images and CAD models, and a test sample set B test including RGB images, depth images and CAD models of D transparent objects, wherein P>D, the six-degree-of-freedom pose of the pth transparent object is T p =(S p ,R p ), S p and R p represent the translation pose and the rotation pose respectively;

[0009] (2) Construct a six-DOF pose estimation network model for transparent objects:

[0010] A transparent object six-DOF pose estimation network model O is constructed, which includes a cascaded coarse estimation module, an image rendering module, and a fine estimation module. The output of the coarse estimation module is also connected to the fine estimation module. The coarse estimation module is used to perform a coarse estimation of the six-DOF pose of each transparent object; the image rendering module is used to render the CAD model; and the fine estimation module is used to perform a fine estimation of the coarse estimation results.

[0011] (3) Iteratively train the six-DOF pose estimation network model for transparent objects:

[0012] Through training sample set B train The six-DOF pose estimation network model O for transparent objects is iteratively trained to obtain the trained six-DOF pose estimation network O for transparent objects. * ;

[0013] (4) Obtain the six-DOF pose estimation results of the transparent object:

[0014] Test sample set B test As a trained transparent object six-DOF pose estimation network O * The input is propagated forward to obtain B. test The six-degree-of-freedom pose estimation results for the corresponding transparent object.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] (1) In the process of training the pose estimation network model and obtaining the six-degree-of-freedom pose estimation results of transparent objects, the present invention directly regresses the RGB image and depth image of the transparent object through the coarse estimation module and the fine estimation module to obtain the six-degree-of-freedom pose of the transparent object. This avoids the defects of the prior art, which uses the depth image optimized by the segmentation map and the surface normal map to obtain the pose estimation results, resulting in complex calculation and large loss of image information. This effectively improves the estimation accuracy and efficiency.

[0017] (2) The refined estimation module of the present invention enriches the source of depth information by performing feature fusion on the edge features of each depth image and the edge features of the rendered depth image of the corresponding CAD model. This avoids the defects of the prior art, which only preprocesses and optimizes the original depth image, resulting in large depth information errors, and further improves the estimation accuracy. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0019] Figure 2 This is a schematic diagram of the structure of the transparent object six-DOF pose estimation network model constructed in this invention.

[0020] Figure 3 This is a schematic diagram of the coarse estimation module constructed according to the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the refined estimation module constructed in this invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0023] Reference Figure 1 The present invention includes the following steps:

[0024] Step 1) Obtain the training sample set and the test sample set:

[0025] (1a) Obtain RGB images, depth images and CAD models of M transparent objects, including N categories, each containing multiple transparent objects. Crop each RGB image and depth image, and then save the cropped RGB images and depth images of the same size in the same image storage format to obtain M preprocessed RGB images and depth images, where N≥10 and M≥500.

[0026] In this embodiment: 500 transparent objects, each containing multiple transparent objects in 10 categories, are sampled using an RGB-D camera to obtain RGB and depth images of each transparent object. Each RGB and depth image is cropped, and then the cropped RGB and depth images of the same size are saved as PNG format, resulting in 500 preprocessed RGB and depth images. Simultaneously, a corresponding CAD model is generated using Blender software based on the RGB and depth images of each object.

[0027] (1b) Label the three translational degrees of freedom and three rotational degrees of freedom in the RGB images of more than half of the P transparent objects in each category to obtain the six-degree-of-freedom pose labels of the P transparent objects, where the six-degree-of-freedom pose of the p-th transparent object is T. p =(S p ,R p ), S p R p These represent the translation pose and rotation pose, respectively:

[0028] S p =(t x ,t y ,t z )

[0029]

[0030] q p =e + fj1 + gj2 + hj3

[0031]

[0032] Among them, t x t y and t z These represent the translational pose of the center of the transparent object relative to the optical center of the camera along the x, y, and z axes of the camera coordinate system; R p This indicates that the quaternion q p The parameters describe the rotation matrix; e, f, g, and h represent the quaternions q. p The four components, j1, j2 and j3, represent the imaginary parts of f, g and h;

[0033] (1c) The training sample set B is composed of the preprocessed RGB images of P transparent objects and their six-DOF pose labels, as well as the depth image and CAD model. train The remaining D = MP transparent objects are combined into a test sample set B, consisting of their RGB images, depth images, and CAD models. test ;

[0034] This embodiment uses preprocessed RGB images of 400 transparent objects and their six-DOF pose labels, along with a training sample set B of depth images and CAD models. train The remaining 100 transparent objects' RGB images, depth images, and CAD model test sample set B test .

[0035] Step 2) Construct a six-DOF pose estimation network model for a transparent object:

[0036] A six-DOF pose estimation network model for a transparent object is constructed, and its structure is as follows: Figure 2 As shown:

[0037] Constructing cascaded such Figure 3 The coarse estimation module, image rendering module, and such are shown. Figure 4 The fine estimation module shown is a transparent object six-DOF pose estimation network model O, whose output is also connected to the fine estimation module.

[0038] (2a) Constructing the coarse estimation module:

[0039] The coarse estimation module includes a backbone network, a first branch network, and a second branch network. The backbone network includes a segmentation network consisting of A convolutional layers and a translational pose estimation module cascaded thereon. The inputs of the first and second branch networks are connected to the outputs of the A-1 and A-2 convolutional layers in the segmentation network, respectively, and the outputs of the first and second branch networks are connected to the output and input of the translational pose estimation module, respectively, where A≥5.

[0040] (2b) Constructing a refined estimation module:

[0041] The refined estimation module includes a cascaded feature extraction network and a pose update network. The feature extraction network includes a first ResNet18 sub-network and a second ResNet18 sub-network arranged in parallel, each consisting of C convolutional modules, and a global average pooling layer cascaded with the outputs of these two sub-networks. The pose update network includes three parallel regression branch networks, where C ≥ 5. The first and second ResNet18 sub-networks have the same structure, specifically: the first layer of the network is the input layer, receiving a 224×224 image as input, followed by convolutional module 1 with a kernel size of 7×7 and a stride of 2, outputting 64 feature maps with a size of 112x112; then a 3x3 max pooling layer with a stride of 2; the kernel size and number of convolutional layers from convolutional module 2 to convolutional module C-1 gradually increase, where C ≥ 5.

[0042] Step 3) Iteratively train the six-DOF pose estimation network model for the transparent object:

[0043] Through training sample set B train The six-DOF pose estimation network model O for transparent objects is iteratively trained to obtain the trained six-DOF pose estimation network O for transparent objects. * ;

[0044] (3a) The initial number of iterations is i, the maximum number of iterations is I, I≥300, and the weight parameters of the coarse estimation network and the fine estimation network in the i-th iteration are θ and θ, respectively. i ω i And let i = 1;

[0045] (3b) The coarse estimation module performs a coarse estimation of the six-DOF pose in the p-th RGB image; the image rendering module uses the results of the coarse estimation... Render the p-th CAD model; the refinement estimation module uses the p-th depth image and the rendered depth image of the p-th CAD model to... Perform refined estimation to obtain the refined six-DOF pose corresponding to the p-th training sample.

[0046] (3b1) The segmentation network extracts edge features from the p-th RGB image and then performs semantic segmentation. At the same time, the first and second branch networks pool the intermediate features output from the A-1 and A-2 convolutional layers respectively and then perform regression operations to obtain the translational pose components along the z-axis of the camera coordinate system. The segmentation network extracts the boundary features of transparent objects in the RGB image of the p-th training sample through C stacked convolutional layers. It then performs cropping and pooling on the obtained features to generate a feature map with a channel size of 512. Semantic segmentation is then used to obtain a segmentation mask map of the input RGB image. The first and second branches perform pooling and regression operations on the intermediate features output from the (A-1)-th and (A-2)-th convolutional layers, respectively. The second branch first sums the features extracted from different layers in the segmentation network, then reduces the spatial size and number of channels of the features through two 3×3 convolutional layers. The processed features are then fed into three fully connected layers to process the translation component t. z 1 The return;

[0047] (3b2) The translation pose estimation module uses the mask image output by the segmentation network and the regressed translation pose components output by the second branch network. For each transparent object and The translation pose is calculated; and the translation pose is calculated. and Translational pose of the p-th transparent object The specific process is as follows: The 2D bounding box of the transparent object in the RGB image is calculated using the segmentation mask; the target center of the transparent object in the RGB image is located; and the translation component of the regression from branch network 2 is used. Estimating translational pose by using the distance from a transparent object to the camera. The other two components;

[0048] (3b3) The translation pose estimation module obtains Quaternions regressing with the first branch network Indicated By stitching the images together, a rough estimate of the six-DOF pose is obtained. The first branch network adds the features extracted from different layers in the segmentation network, performs max pooling on the features using two RoI layers to extract the features of the region of interest, adds the features of the two regions of interest, and feeds them into three fully connected layers to regress the rotation quaternion of the transparent object. And represented as a rotation matrix

[0049] (3b4) The first and second ResNet18 subnetworks extract edge features from the p-th depth image and the rendered depth image of the p-th CAD model, respectively;

[0050] (3b5) The global average pooling layer performs global average pooling on the edge features of the p-th depth image and the difference between the edge features and the rendered depth image of the corresponding CAD model, to obtain the fused feature F. p ;

[0051] (3b6) In the pose update network, F is used through three regression branches respectively. p Coarse estimation results for the p-th RGB image In and as well as Regression is performed to obtain the refined six-DOF pose corresponding to the p-th training sample. In the pose update network, the first, second, and third regression branches are used sequentially to update F. p With coarse six degrees of freedom pose The results of the splicing were obtained by regression. and as well as Will and Adding them together gives the translation pose. quaternion q 2 Represented as rotational pose and The refined six-DOF pose of the p-th training sample is obtained by connecting the two sides.

[0052] (3c) Through each pose T p and its estimated value The calculated translational pose loss value L S and rotation pose loss value L R Calculate the loss value L of the network model O, and then use stochastic gradient descent to adjust the weight parameters θ using L. i and ω i The network model O is updated to obtain the updated model after this iteration. i ;

[0053] Translation pose loss value L S Rotation pose loss value L R And the loss value L of the network model O, calculated using the following formulas:

[0054]

[0055] L=(1-β)L S +βLR

[0056] in, These represent the translation pose S respectively. p and rotation pose R p The estimated value, β is the weight, |·| represents the absolute value operation, and ||·|| represents the 1 norm;

[0057] For the weight parameter θ i and ω i The update is performed using the following formulas:

[0058]

[0059] Where, ω i-1 θ i-1 Let denot represent the weight parameters of the coarse estimation network and the fine estimation network in the (i-1)th iteration, respectively, and let α represent the learning rate. This indicates the partial derivative operation;

[0060] (3d) Determine if i≥I holds true. If yes, obtain the trained transparent object six-DOF pose estimation network O. * , otherwise, let i=i+1, O i =O, and proceed to step (3b).

[0061] Step 4) Obtain the six-DOF pose estimation results of the transparent object:

[0062] Test sample set B test As a trained transparent object six-DOF pose estimation network O * The input is propagated forward to obtain B. test The six-degree-of-freedom pose estimation results for the corresponding transparent object.

Claims

1. A method for estimating the pose of a transparent object with six degrees of freedom, characterized in that, Includes the following steps: (1) Obtain the training sample set and the test sample set: Get including The training sample set includes RGB images of a transparent object and their six-DOF pose labels, as well as depth images and CAD models. and including Test sample set of RGB image, depth image and CAD model of a transparent object. ,in, , No. The six-DOF pose of a transparent object is , , These represent translational pose and rotational pose, respectively. (2) Construct a six-DOF pose estimation network model for transparent objects: A transparent object six-DOF pose estimation network model is constructed, comprising a cascaded coarse estimation module, an image rendering module, and a fine estimation module, with the output of the coarse estimation module also connected to the fine estimation module. The coarse estimation module is used to roughly estimate the six-degree-of-freedom pose of each transparent object; the image rendering module is used to render the CAD model; and the fine estimation module performs fine estimation on the coarse estimation results using each depth image and the corresponding rendered depth image of the CAD model. (3) Iteratively train the six-DOF pose estimation network model for transparent objects: Through training sample set Six-DOF pose estimation network model for transparent objects Through iterative training, a well-trained six-DOF pose estimation network for transparent objects is obtained. ; (4) Obtain the six-DOF pose estimation results of the transparent object: Test sample set As a trained transparent object six-DOF pose estimation network The input is propagated forward to obtain The corresponding six-DOF pose estimation results for the transparent object.

2. The method according to claim 1, characterized in that, The steps for obtaining the training sample set and the test sample set in step (1) are as follows: (1a) Obtaining including Each category contains multiple transparent objects. The system takes an RGB image, a depth image, and a CAD model of a transparent object, and crops each RGB and depth image. Then, it saves the cropped RGB and depth images of the same size in the same image storage format to obtain the preprocessed image. A series of RGB images and depth images, of which , ; (1b) For more than half of each category The three translational degrees of freedom and three rotational degrees of freedom of the RGB image of a transparent object are labeled to obtain... A six-DOF pose label for a transparent object, where the first... The six-DOF pose of a transparent object is , , These represent the translation pose and rotation pose, respectively: ； ； ； ； in, , and These represent the points along the camera coordinate system relative to the optical center of the transparent object. , and Translation pose along the three coordinate axes; Indicates the use of quaternions The rotation matrix described by the parameters; , , and Representing quaternions The four components, , and express , and The imaginary part; (1c) The pre-processed The training sample set consists of RGB images of a transparent object and their six-DOF pose labels, as well as depth images and CAD models. , the remaining The test sample set consists of RGB images, depth images, and CAD models of a transparent object. .

3. The method according to claim 1, characterized in that, The transparent object six-DOF pose estimation network model described in step (2) includes: The coarse estimation module includes a backbone network, a first branch network, and a second branch network; the backbone network consists of... A segmentation network consisting of stacked convolutional layers and a cascaded translation pose estimation module; the inputs of the first and second branch networks are respectively connected to the first convolutional layer in the segmentation network. , The outputs of each convolutional layer are connected, and the outputs of the first and second branch networks are connected to the output and input of the translation pose estimation module, respectively. ; The refined estimation module includes a cascaded feature extraction network and a pose update network; the feature extraction network consists of parallel networks, each composed of... The network consists of a first ResNet18 subnetwork and a second ResNet18 subnetwork composed of convolutional modules, and a global average pooling layer cascaded with the outputs of these two subnetworks; the pose update network includes three parallel regression branches, wherein... .

4. The method according to claim 3, characterized in that, The six-DOF pose estimation network model for transparent objects described in step (3) The iterative training process involves the following steps: (3a) Initialize the number of iterations to be The maximum number of iterations is , , No. The weight parameters of the coarse estimation network and the fine estimation network in the next iteration are respectively , and order ; (3b) Coarse estimation module for the first A rough estimate of the six-DOF pose in an RGB image is performed. The image rendering module uses the results of a rough estimate. For the The first CAD model is rendered; the fine estimation module is executed through the first... Image depth and the first Rendering depth image of a CAD model A refined estimate is performed to obtain the first... Refined six-DOF poses corresponding to the training samples ; (3c) Through each pose and its estimated value Calculated translation pose loss value and rotation pose loss value Computational network model loss value Then, the stochastic gradient descent method is used to... For weight parameters and The network model is updated to obtain the updated model for this iteration. ; (3d) judgment If true, then a well-trained six-DOF pose estimation network for transparent objects is obtained. Otherwise, let , Then proceed with step (3b).

5. The method according to claim 4, characterized in that, The coarse estimation module described in step (3b) is for the first... The steps for roughly estimating the six-DOF pose in an RGB image are as follows: (3b1) Segmentation network for the first After edge feature extraction of the RGB image, semantic segmentation is performed. Simultaneously, the first and second branch networks respectively process the edge features of the RGB image. , The intermediate features output by each convolutional layer are pooled and then subjected to regression to obtain the translational pose components along the z-axis of the camera coordinate system. ; (3b2) The translation pose estimation module uses the mask image output by the segmentation network and the regressed translation pose components output by the second branch network. Calculate the translational pose components of each transparent object. and and will , and Composition of the first Translational pose of a transparent object ; (3b3) The output of the translation pose estimation module Quaternions regressing with the first branch network Indicated By stitching the images together, a rough estimate of the six-DOF pose is obtained. .

6. The method according to claim 4, characterized in that, The refined estimation module described in step (3b) uses the first... Image depth and the first The steps for refining the coarse estimation result using the rendered depth image of a CAD model are as follows: (3b4) The first and second ResNet18 subnetworks are paired with the first... Image depth and the first Edge features were extracted from the rendered depth images of each CAD model; (3b5) Global average pooling layer for the first The edge features of the depth image and the difference between the edge features of the rendered depth image of the corresponding CAD model are subjected to global average pooling to obtain the fused features. ; (3b6) The three regression branches in the pose update network are passed through For the Rough estimation results of RGB images In , and as well as Perform regressions separately to obtain the first... The refined six-DOF pose corresponding to each training sample .

7. The method according to claim 4, characterized in that, The translation pose loss value mentioned in step (3c) Rotation pose loss value and network models loss value The calculation formulas are as follows: ； ； ； in, , They represent the translation pose respectively. and rotation pose The estimated value, Indicates the weighting coefficient. This indicates the absolute value operation. This indicates the operation of taking the 1-norm.

8. The method according to claim 4, characterized in that, The weight parameters described in step (3c) and The update is performed using the following formulas: ； ； in, , They represent the first The weight parameters of the network are estimated in the next iteration, and then refined in the next iteration. Indicates the learning rate. This indicates the partial derivative operation.

Images