Transparent object six-degree-of-freedom pose estimation method

Abstract

The invention provides a six-degree-of-freedom pose estimation method for a transparent object. The method comprises the following implementation steps: acquiring a training sample set and a test sample set; constructing a six-degree-of-freedom pose estimation network model of the transparent object and carrying out iterative training on the model; and obtaining a six-degree-of-freedom pose estimation result of the transparent object. In the process of training the pose estimation network model and obtaining the six-degree-of-freedom pose estimation result of the transparent object, the RGB image and the depth image of the transparent object are subjected to direct regression through the coarse estimation module and the fine estimation module so as to obtain the six-degree-of-freedom pose of the transparent object. The calculation process is simple, the image information loss is small, the refined estimation module carries out feature fusion on the edge feature of each depth image and the edge feature of the rendering depth image of the corresponding CAD model, refined estimation is carried out on the rough estimation result, the source of depth information is enriched, the error of the depth information is reduced, and the accuracy of the depth information is improved. And the estimation precision and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision technology and relates to a six-degree-of-freedom pose estimation method for a transparent object, and specifically to a six-degree-of-freedom pose estimation method for a transparent object based on deep learning, which can be applied to fields such as robot navigation and autonomous driving. Background Art

[0002] In the field of robotics and machine vision, six-degree-of-freedom pose estimation of target objects is a key issue. It provides accurate spatial position and orientation information for operating the target object, including three translational degrees of freedom and three rotational degrees of freedom of the object relative to the camera. However, for transparent objects commonly seen in daily life, due to their optical characteristics of weak texture and strong specular reflection, the depth images of transparent objects obtained by the sensor have errors, making it difficult to accurately estimate the six-degree-of-freedom pose of transparent objects.

[0003] At present, the six-degree-of-freedom pose estimation methods for transparent objects mainly include two categories: estimation methods based on key point matching and estimation methods based on deep learning. Among them, the estimation accuracy of the estimation method based on key point matching is poor; the estimation method based on deep learning uses deep learning methods to estimate the six-degree-of-freedom pose of the target object from the input RGB image or depth image, taking into account both computational accuracy and computational efficiency, and has gradually become a research hotspot.

[0004] With the development of deep learning technology, more and more estimation methods based on deep learning are applied to the six-degree-of-freedom pose estimation of transparent objects, and the accuracy and computational efficiency of the six-degree-of-freedom pose estimation of transparent objects are continuously improved. For example, the patent application with application publication number CN 113313810 A and titled "A method for calculating 6D pose parameters of a transparent object" discloses a method for calculating 6D pose parameters of a transparent object based on deep learning. The invention first collects the original RGB image and the original depth image of the transparent object, and then uses a fully convolutional network to extract low-level and high-level features in the original RGB image, and generates a transparent object segmentation map of the original RGB image through an attention mechanism based on the high-level features; at the same time, the original RGB image is processed through a Deeplabv3 network to obtain a surface normal map of the transparent object; next, the original depth image is masked and preprocessed and globally optimized using the obtained segmentation map and surface normal map to obtain a corrected depth image; finally, the corrected depth image is used as input, and the six-degree-of-freedom pose parameters of the transparent object are estimated through a pixel-level dense feature fusion model proposed by the method, and the invention has a high estimation accuracy. However, this invention estimates the six-degree-of-freedom pose through a depth image optimized by a segmentation map of a transparent object and a surface normal map, which increases the computational complexity and also causes loss of image information; and this invention corrects errors in the depth information through preprocessing and global optimization of the original depth image, but these processes are still based on the original depth image with errors, affecting further improvement of the estimation accuracy. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a method for estimating the six-degree-of-freedom posture of a transparent object to solve the technical problem of low accuracy of six-degree-of-freedom posture estimation of transparent objects in the prior art.

[0006] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0007] (1) Obtain training sample set and test sample set:

[0008] Obtain a training sample set B including P transparent objects’ RGB images and their six-degree-of-freedom pose labels, as well as depth images and CAD models train , and a test sample set B including RGB images, depth images and CAD models of D transparent objects test , where P>D, and the six-degree-of-freedom pose of the pth transparent object is T p =(S p ,R p ), S p , R p Represent the translation pose and rotation pose respectively;

[0009] (2) Constructing a six-degree-of-freedom pose estimation network model for transparent objects:

[0010] Constructing a transparent object six-degree-of-freedom pose estimation network model O including a cascaded rough estimation module, an image rendering module and a refined estimation module, wherein the output end of the rough estimation module is also connected to the refined estimation module; wherein the rough estimation module is used to roughly estimate the six-degree-of-freedom pose of each transparent object; the image rendering module is used to render the CAD model; and the refined estimation module is used to refine the rough estimation result;

[0011] (3) Iterative training of the transparent object six-degree-of-freedom pose estimation network model:

[0012] Through the training sample set B train Iteratively train the transparent object six-degree-of-freedom pose estimation network model O to obtain the trained transparent object six-degree-of-freedom pose estimation network O * ;

[0013] (4) Obtain the six-degree-of-freedom pose estimation result of the transparent object:

[0014] The test sample set B test As a trained transparent object six-DOF pose estimation network O * The input is forward propagated to obtain B test The corresponding six-degree-of-freedom pose estimation results of the transparent object.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] (1) In the process of training the pose estimation network model and obtaining the six-degree-of-freedom pose estimation results of the transparent object, the present invention directly regresses the RGB image and the depth image of the transparent object through a coarse estimation module and a refined estimation module to obtain the six-degree-of-freedom pose of the transparent object, thereby avoiding the defects of the prior art of obtaining pose estimation results using a depth image optimized by a segmentation map and a surface normal map, which results in complex calculations and large image information loss, and effectively improves the estimation accuracy and efficiency.

[0017] (2) The refined estimation module of the present invention enriches the source of depth information by fusing the edge features of each depth image with the edge features of the rendered depth image of the corresponding CAD model, avoiding the defect of the prior art that only preprocesses and globally optimizes the original depth image, resulting in large depth information errors, and further effectively improves the estimation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart for implementing the present invention.

[0019] Figure 2 A schematic diagram of the structure of a six-degree-of-freedom pose estimation network model for a transparent object constructed in the present invention.

[0020] Figure 3 A schematic diagram of the structure of the rough estimation module constructed in the present invention.

[0021] Figure 4 A schematic diagram of the structure of the refined estimation module constructed for the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0023] Reference Figure 1 , the present invention comprises the following steps:

[0024] Step 1) Obtain training sample set and test sample set:

[0025] (1a) obtaining RGB images, depth images, and CAD models of a total of M transparent objects, each of which includes multiple transparent objects in N categories, and cropping each RGB image and depth image, and then saving the cropped RGB images and depth images of the same size in the same image storage format to obtain preprocessed M RGB images and depth images, where N ≥ 10 and M ≥ 500;

[0026] In this embodiment, 500 transparent objects in 10 categories, each containing multiple transparent objects, are sampled using an RGB-D camera to obtain an RGB image and a depth image of each transparent object, each RGB image and depth image is cropped, and then the cropped RGB images and depth images of the same size are saved in PNG format to obtain 500 pre-processed RGB images and depth images; at the same time, a corresponding CAD model is generated using Blender software according to the RGB image and depth image of each object;

[0027] (1b) Annotate the three translational degrees of freedom and three rotational degrees of freedom in the RGB images of more than half of the P transparent objects in each category to obtain the six-degree-of-freedom pose labels of the P transparent objects, where the six-degree-of-freedom pose of the pth transparent object is T p =(S p ,R p ), S p , R p Represent the translation pose and rotation pose respectively:

[0028] S p =(t x ,t y ,t z )

[0029]

[0030] q p =e+fj1+gj2+hj3

[0031]

[0032] Among them, t x ,t y and t z Respectively represent the translational position of the transparent object center relative to the optical center of the camera along the three coordinate axes of the camera coordinate system x, y and z; R p Represented by quaternion q p The parameters describe the rotation matrix; e, f, g, and h represent the quaternion q p The four components of, j1, j2 and j3 represent the imaginary parts of f, g and h;

[0033] (1c) The preprocessed RGB images of P transparent objects and their six-degree-of-freedom pose labels, as well as the depth images and CAD models form a training sample set B train , the remaining D = MP transparent objects’ RGB images, depth images and CAD models form the test sample set B test ;

[0034] In this embodiment, the preprocessed RGB images of 400 transparent objects and their six-degree-of-freedom pose labels, as well as the training sample set B of the depth image and CAD model are used. train , the remaining 100 transparent object RGB images, depth images and CAD model test sample set B test .

[0035] Step 2) Construct a transparent object six-degree-of-freedom pose estimation network model:

[0036] Construct a transparent object six-degree-of-freedom pose estimation network model, whose structure is as follows Figure 2 As shown:

[0037] Constructing a cascade of Figure 3 The rough estimation module, image rendering module and Figure 4 The refined estimation module shown, and the output end of the coarse estimation module is also connected to the transparent object six-degree-of-freedom pose estimation network model O of the refined estimation module;

[0038] (2a) Construct a rough estimation module:

[0039] A coarse estimation module includes a backbone network, a first branch network and a second branch network; the backbone network includes a segmentation network formed by stacking A convolutional layers and a translation pose estimation module cascaded therewith; the input ends of the first and second branch networks are respectively connected to the output ends of the A-1 and A-2 convolutional layers in the segmentation network, and the output ends of the first and second branch networks are respectively connected to the output and input ends of the translation pose estimation module, wherein A≥5;

[0040] (2b) Construct a refined estimation module:

[0041] A refined estimation module includes a cascaded feature extraction network and a pose update network; the feature extraction network includes a first ResNet18 subnetwork and a second ResNet18 subnetwork arranged in parallel and each consisting of C convolutional modules, and a global average pooling layer cascaded with the output ends of the two subnetworks; the pose update network includes three regression branch networks arranged in parallel, where C≥5; wherein the first ResNet18 subnetwork and the second ResNet18 subnetwork have the same structure, specifically: the first layer of the network is the input layer, which receives a 224×224 image as input, followed by a convolution module 1 with a convolution kernel size of 7×7 and a step size of 2, outputting 64 feature maps with a size changed to 112x112; followed by a 3x3 maximum pooling layer with a step size of 2; the convolution kernel size and the number of convolution layers from convolution module 2 to convolution module C-1 gradually increase, where C≥5.

[0042] Step 3) Iteratively train the transparent object six-degree-of-freedom pose estimation network model:

[0043] Through the training sample set B train Iteratively train the transparent object six-degree-of-freedom pose estimation network model O to obtain the trained transparent object six-degree-of-freedom pose estimation network O * ;

[0044] (3a) The number of initial iterations is i, the maximum number of iterations is I, I ≥ 300, and the weight parameters of the rough estimation network and the refined estimation network in the i-th iteration are θ i ,ω i , and let i = 1;

[0045] (3b) The rough estimation module roughly estimates the six-degree-of-freedom pose in the pth RGB image; the image rendering module uses the rough estimation result Render the p-th CAD model; the refined estimation module compares the p-th depth image with the rendered depth image of the p-th CAD model. Perform a refined estimate to obtain the refined six-degree-of-freedom pose corresponding to the p-th training sample

[0046] (3b1) The segmentation network extracts edge features from the pth RGB image and then performs semantic segmentation. At the same time, the first and second branch networks pool the intermediate features output by the A-1th and A-2th convolutional layers respectively and then perform regression operations to obtain the translation pose component along the z-axis of the camera coordinate system. The segmentation network extracts the boundary features of transparent objects in the RGB image of the pth training sample through C stacked convolutional layers, and clips and pools the obtained features through the pooling layer to generate a feature map with a channel size of 512, and obtains the segmentation mask map of the input RGB image through semantic segmentation; the first and second branch networks perform pooling and regression operations on the intermediate features output by the A-1 and A-2 convolutional layers respectively; the second branch network first adds the features extracted by different layers in the segmentation network, and reduces the spatial size and number of channels of the features through two 3×3 convolutional layers, and sends the processed features to three fully connected layers for translation component t z 1 The return of

[0047] (3b2) The translation pose estimation module uses the mask image output by the segmentation network and the regressed translation pose component output by the second branch network. For each transparent object and The translation pose is calculated; and The translation pose of the pth transparent object The specific process is: the 2D bounding box of the transparent object in the RGB image is calculated by the segmentation mask, the target center of the transparent object in the RGB image is located, and the translation component regressed using branch network 2 Estimate the translation pose as the distance from the transparent object to the camera The other two components of

[0048] (3b3) The translation pose estimation module obtains Quaternion regressed from the first branch network Indicated Splice to get a rough estimate of the six-degree-of-freedom pose The first branch network adds the features extracted by different layers in the segmentation network, uses two RoI layers to perform maximum pooling operations on the features, extracts the features of the region of interest, adds the features of the two regions of interest, and sends them to three fully connected layers to regress the rotation quaternion of the transparent object. And represented as a rotation matrix

[0049] (3b4) the first and second ResNet18 sub-networks extract edge features from the p-th depth image and the rendered depth image of the p-th CAD model respectively;

[0050] (3b5) The global average pooling layer performs global average pooling on the edge features of the pth depth image and the difference between the edge features of the rendered depth image of the corresponding CAD model to obtain the fusion feature F p ;

[0051] (3b6) The pose update network uses three regression branch networks to respectively use F p Rough estimation result of the pth RGB image In and as well as Perform regression to obtain the refined six-degree-of-freedom pose corresponding to the p-th training sample In the pose update network, the first, second, and third regression branch networks are used to update F p and coarse 6DOF pose The splicing result is regressed to obtain and as well as Will and Add together to get the translation pose The quaternion q 2 Represented as a rotation pose and Connect them to get the refined six-degree-of-freedom pose of the pth training sample

[0052] (3c) Through each pose T p and its estimated value The calculated translation pose loss value L S And the rotation pose loss value L R Calculate the loss value L of the network model O, and then use the stochastic gradient descent method to adjust the weight parameter θ through L i and ω i Update and get the network model O after this iteration i ;

[0053] Translation pose loss value L S , rotation pose loss value L R , and the loss value L of the network model O, the calculation formulas are:

[0054]

[0055] L = (1-β) L S +βLR

[0056] in, They represent the translation pose S p and the rotation pose R p The estimated value of , β is the weight, |·| means taking the absolute value operation, ||·|| means taking the 1 norm;

[0057] For the weight parameter θ i and ω i To update, the update formulas are:

[0058]

[0059] Among them, ω i-1 ,θ i-1 They represent the weight parameters of the rough estimation network and the refined estimation network in the i-1th iteration, α represents the learning rate, represents the partial derivative operation;

[0060] (3d) Determine whether i≥I is true. If so, obtain the trained transparent object six-degree-of-freedom pose estimation network O * , otherwise, let i=i+1, O i =O, and execute step (3b).

[0061] Step 4) Obtain the six-degree-of-freedom pose estimation result of the transparent object:

[0062] The test sample set B test As a trained transparent object six-DOF pose estimation network O * The input is forward propagated to obtain B test The corresponding six-degree-of-freedom pose estimation results of the transparent object.

Claims

1. A method for estimating the six-degree-of-freedom pose of a transparent object, characterized in that: The steps include: (1) Obtain training sample set and test sample set: Obtain a training sample set B including P transparent objects’ RGB images and their six-degree-of-freedom pose labels, as well as depth images and CAD models train , and a test sample set B including RGB images, depth images and CAD models of D transparent objects test , where P>D, and the six-degree-of-freedom pose of the pth transparent object is T p =(S p ,R p ), S p , R p Represent the translation pose and rotation pose respectively; (2) Constructing a six-degree-of-freedom pose estimation network model for transparent objects: Constructing a transparent object six-degree-of-freedom pose estimation network model O including a cascaded rough estimation module, an image rendering module and a refined estimation module, wherein the output end of the rough estimation module is also connected to the refined estimation module; wherein the rough estimation module is used to roughly estimate the six-degree-of-freedom pose of each transparent object; the image rendering module is used to render the CAD model; and the refined estimation module is used to refine the rough estimation result; (3) Iterative training of the transparent object six-degree-of-freedom pose estimation network model: Through the training sample set B train Iteratively train the transparent object six-degree-of-freedom pose estimation network model O to obtain the trained transparent object six-degree-of-freedom pose estimation network O * ; (4) Obtain the six-degree-of-freedom pose estimation result of the transparent object: The test sample set B test As a trained transparent object six-DOF pose estimation network O * The input is forward propagated to obtain B test The corresponding six-degree-of-freedom pose estimation results of the transparent object.

2. The method according to claim 1, characterized in that: The steps for obtaining the training sample set and the test sample set in step (1) are as follows: (1a) obtaining RGB images, depth images, and CAD models of a total of M transparent objects, each of which includes multiple transparent objects in N categories, and cropping each RGB image and depth image, and then saving the cropped RGB images and depth images of the same size in the same image storage format to obtain preprocessed M RGB images and depth images, where N ≥ 10 and M ≥ 500; (1b) Annotate the three translational degrees of freedom and three rotational degrees of freedom in the RGB images of more than half of the P transparent objects in each category to obtain the six-degree-of-freedom pose labels of the P transparent objects, where the six-degree-of-freedom pose of the pth transparent object is T p =(S p ,R p ), S p , R p Represent the translation pose and rotation pose respectively: S p =(t x ,t y ,t z ) q p =e+fj1+gj2+hj3 Among them, t x ,t y and t z Respectively represent the translational position of the transparent object center relative to the optical center of the camera along the three coordinate axes of the camera coordinate system x, y and z; R p Represented by quaternion q p The parameters describe the rotation matrix; e, f, g, and h represent the quaternion q p The four components of, j1, j2 and j3 represent the imaginary parts of f, g and h; (1c) The preprocessed RGB images of P transparent objects and their six-degree-of-freedom pose labels, as well as the depth images and CAD models form a training sample set B train , the remaining D = MP transparent objects’ RGB images, depth images and CAD models form the test sample set B test .

3. The method according to claim 1, characterized in that The transparent object six-degree-of-freedom pose estimation network model described in step (2), wherein: A coarse estimation module includes a backbone network, a first branch network and a second branch network; the backbone network includes a segmentation network formed by stacking A convolutional layers and a translation pose estimation module cascaded therewith; the input ends of the first and second branch networks are respectively connected to the output ends of the A-1 and A-2 convolutional layers in the segmentation network, and the output ends of the first and second branch networks are respectively connected to the output and input ends of the translation pose estimation module, wherein A≥5; A refined estimation module includes a cascaded feature extraction network and a pose update network; the feature extraction network includes a first ResNet18 subnetwork and a second ResNet18 subnetwork arranged in parallel and each consisting of C convolutional modules, and a global average pooling layer cascaded with the output ends of the two subnetworks; the pose update network includes three regression branch networks arranged in parallel, where C ≥ 5.

4. The method according to claim 3, characterized in that The iterative training of the transparent object six-degree-of-freedom pose estimation network model O described in step (3) is implemented as follows: (3a) The number of initial iterations is i, the maximum number of iterations is I, I ≥ 300, and the weight parameters of the rough estimation network and the refined estimation network in the i-th iteration are θ i ,ω i , and let i = 1; (3b) The coarse estimation module roughly estimates the six-degree-of-freedom pose in the p-th RGB image; The image rendering module roughly estimates the result Render the p-th CAD model; the refined estimation module compares the p-th depth image with the rendered depth image of the p-th CAD model. Perform a refined estimate to obtain the refined six-degree-of-freedom pose corresponding to the pth training sample (3c) Through each pose T p and its estimated value The calculated translation pose loss value L S And the rotation pose loss value L R Calculate the loss value L of the network model O, and then use the stochastic gradient descent method to adjust the weight parameter θ through L i and ω i Update and get the network model O after this iteration i ; (3d) Determine whether i≥I is true. If so, obtain the trained transparent object six-degree-of-freedom pose estimation network O * , otherwise, let i=i+1, O i =O, and execute step (3b).

5. The method according to claim 4, characterized in that The rough estimation module described in step (3b) performs a rough estimation of the six-degree-of-freedom pose in the p-th RGB image, and the implementation steps are as follows: (3b1) The segmentation network extracts edge features from the pth RGB image and then performs semantic segmentation. At the same time, the first and second branch networks pool the intermediate features output by the A-1th and A-2th convolutional layers respectively and then perform regression operations to obtain the translation pose component along the z-axis of the camera coordinate system. (3b2) The translation pose estimation module uses the mask image output by the segmentation network and the regressed translation pose component output by the second branch network. Calculate the translation pose component of each transparent object and and will and The translation pose of the pth transparent object (3b3) The output of the translation pose estimation module Quaternion regressed from the first branch network Indicated Splice to get a rough estimate of the six-degree-of-freedom pose 6. The method according to claim 4, characterized in that The refined estimation module in step (3b) performs a refined estimation on the rough estimation result through the p-th depth image and the rendered depth image of the p-th CAD model, and the implementation steps are as follows: (3b4) the first and second ResNet18 sub-networks extract edge features from the p-th depth image and the rendered depth image of the p-th CAD model respectively; (3b5) The global average pooling layer performs global average pooling on the edge features of the pth depth image and the difference between the edge features of the rendered depth image of the corresponding CAD model to obtain the fusion feature F p ; (3b6) The three regression branches in the pose update network are connected by F p Rough estimation result of the pth RGB image In and as well as Regression is performed separately to obtain the refined six-degree-of-freedom pose corresponding to the p-th training sample 7. The method according to claim 4, characterized in that The translation pose loss value L described in step (3c) S , rotation pose loss value L R , and the loss value L of the network model O, the calculation formulas are: L=(1-β)L S +βL R in, They represent the translation pose S p and the rotation pose R p The estimated value of , β represents the weight coefficient, |·| represents the absolute value operation, and ||·|| represents the 1-norm operation.

8. The method according to claim 4, characterized in that The weight parameter θ described in step (3c) i and ω i To update, the update formulas are: Among them, ω i-1 ,θ i-1 They represent the weight parameters of the rough estimation network and the refined estimation network in the i-1th iteration, α represents the learning rate, Represents the partial derivative operation.

Images