An accurate monocular depth estimation method based on planar and disparity metrics

Abstract

The present invention discloses an accurate monocular depth estimation method based on plane and disparity metrics, including motion decomposition and plane definition: based on plane disparity decomposition, the motion in the image is decomposed into two parts: plane homography and residual disparity; source image distortion and motion cancellation: the source image is distorted according to the defined plane homography to offset the motion of the plane; residual disparity processing and depth prediction: after the source image is distorted, the remaining image motion is analyzed and processed; implementation and integration: the optimized algorithm is integrated into the visual system of the autonomous driving vehicle, and tested and verified in the actual environment. The present invention calculates the induced two-dimensional image motion of three-dimensional points by constructing a neural network, predicting depth and camera translation, and using it to reconstruct the target image in a self-supervised monocular method. We efficiently solve the scale ambiguity problem without the need for ground truth data from additional sensors.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an accurate monocular depth estimation method based on plane and disparity measurement. Background Art

[0002] Understanding the 3D structure of a scene is relatively easy for humans. We can easily infer the environment around us and break it down into different objects. Having this ability is crucial for self-driving vehicles to be able to navigate different environments. Training deep networks to estimate depth has proven successful in computer vision research.

[0003] However, many of these methods are supervised and require expensive ground truth depth to implement. Another approach uses a stereo setup that must be carefully calibrated. Both methods cannot leverage the large amounts of unlabeled videos that are readily available for training. On the other hand, self-supervised monocular depth estimation methods that do not rely on stereo supervision do not suffer from these limitations and have begun to approach the level of supervised or stereo methods in practice. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, an accurate monocular depth estimation method based on plane and disparity measurement is adopted to solve the problems raised in the above background technology.

[0005] An accurate monocular depth estimation method based on plane and disparity metrics comprises the following steps:

[0006] Step S1, motion decomposition and plane definition: Based on plane parallax decomposition, a plane is defined, and the motion in the image is decomposed into two parts: plane homography and residual parallax, wherein the homography is used to represent the plane motion from the source image to the target image;

[0007] Step S2, source image distortion and motion offset: distort the source image according to the defined plane homography to offset the motion of the plane;

[0008] Step S3, residual parallax processing and depth prediction: after the source image is distorted, the remaining image motion is analyzed and processed;

[0009] Step S4, implementation and integration: Integrate the optimized algorithm into the vision system of the autonomous driving vehicle and test and verify it in a real environment.

[0010] As a further solution of the present invention: the specific steps of plane parallax decomposition include:

[0011] Define Π to represent a 3D plane, where H represents aligning Π on the target image I t and source image I sHomography transformation between; set p and p' to represent the 3D point coordinate system x = [X, Y, Z] in I t and I s projection on;

[0012] Transform p′ through the homography transformation H to obtain the image point p w :

[0013] p w ~Hp′

[0014] By s Transform and get the aligned image I w , so that the plane Π matches between them, then p w The displacement between and p is calculated as:

[0015]

[0016] Where K is the intrinsic parameter matrix of the camera, t = [tx, ty, tz] is the target image I t and source image I s The translation vector between c is the distance from the source view camera to the plane Π; the structure is represented by γ = Zh, where h is the distance from point x to Π; where, when x is on the plane Π, that is, h = 0, then p w =p;

[0017] A monocular depth estimation network framework based on self-supervision is designed. The framework includes a deep network for estimating depth and a posture network for estimating translation between frames. The posture network receives the source image and the image I s ,I w , and outputs the translation vector t; the deep network receives the target image I t , and output I t Depth map

[0018] For each pixel p = [x, y], such that Represents its estimated depth, using the camera intrinsics and the estimated depth to backproject p to obtain the corresponding 3D point In the camera coordinate system, the formula is:

[0019]

[0020] The following relationship is obtained:

[0021]

[0022] Where N is the normal vector of the plane Π, Yes The estimated distance to plane Π, is an estimate of the structural variable γ; obtain the reconstructed target image I t All the parameters required are obtained by aligning the image I w Transform to get

[0023] For I t For each pixel p on the image, we calculate p based on the depth and translation predicted by the two networks. w , and then inverse transform I w and obtain To rebuild I t :

[0024]

[0025] By minimizing I t and to monitor the differences between

[0026] Perform preprocessing steps on the dataset to obtain aligned images;

[0027] By using the road as the plane Π, a homography is calculated for each pair of consecutive frames, and the frames are transformed according to the calculated homography to obtain a uniform distribution of the target image I t and source image I s Calculate a homography H, and then transform I according to H s Transform and obtain the transformed source image I w .

[0028] As a further solution of the present invention: the specific steps of the monocular depth estimation network framework based on self-supervision include:

[0029] Design a monocular depth estimation network framework based on the U-Net architecture, and use the ResNet pre-trained on ImageNet as the encoder of the depth estimation network;

[0030] Using the same architecture as the current monocular depth estimation network, the depth is estimated by multiplying the output of the last sigmoid layer by the predicted maximum depth value, where the depth network starts with a single target image I t Takes as input and outputs a depth estimate for each pixel;

[0031] The second network receives the transformed source image I w and the target image I t As input, and outputting only the translation vector between views, the network is similar to the pose network, where the output is a three-element vector representing the translation;

[0032] Define the self-supervised training loss of the network and define the photometric loss function as a linear combination of L1 distance and structural similarity to minimize the target image I t and reconstruct the image The difference between , the photometric loss is defined as follows:

[0033]

[0034] For each target image, two aligned images, one obtained by deforming the previous frame and the other obtained by deforming the next frame; introduce the per-pixel minimum reprojection error and calculate the minimum value of Lphoto for each pixel of the aligned images across the previous frame and the next frame;

[0035] Define Lsmooth as an edge-based smoothness loss to encourage depth prediction to be locally smooth;

[0036] The total loss function is then a combination of Lsmooth and Lphoto averaged over all N pixels, and the formula is:

[0037]

[0038] Among them, min means calculating the minimum value of the alignment frame between the previous frame and the next frame, and λ is a hyperparameter used to control the impact of the loss term.

[0039] Compared with the prior art, the present invention has the following technical effects:

[0040] The above techniques produce metrically accurate depth estimates using the known camera height. Unlike previous methods that rely on estimating the full rigid motion of the camera, our method only needs to estimate the camera translation. Using two neural networks, we predict depth and camera translation. Given the camera height, we can compute the induced 2D image motion of 3D points and use it to reconstruct the target image in a self-supervised monocular approach. We efficiently resolve the scale ambiguity problem without requiring ground truth data from additional sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0042] Figure 1 A schematic diagram of the steps of the monocular depth estimation method according to the embodiment disclosed in the present application;

[0043] Figure 2 It is a schematic diagram of plane parallax decomposition according to an embodiment disclosed in the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, a method for accurate monocular depth estimation based on plane and disparity measurement includes the following steps:

[0046] In this embodiment, a method for predicting a metric scaled depth map without using any ground truth depth supervision is described. The method is mainly divided into two key parts: 1. Planar disparity decomposition; 2. Scaled monocular depth estimation neural network;

[0047] Step S1, motion decomposition and plane definition: Based on plane parallax decomposition, a plane is defined, and the motion in the image is decomposed into two parts: plane homography and residual parallax, wherein the homography is used to represent the plane motion from the source image to the target image;

[0048] Step S2, source image distortion and motion offset: distort the source image according to the defined plane homography to offset the motion of the plane;

[0049] Step S3, residual parallax processing and depth prediction: after the source image is distorted, the remaining image motion is analyzed and processed;

[0050] Step S4, implementation and integration: Integrate the optimized algorithm into the vision system of the autonomous driving vehicle and test and verify it in a real environment.

[0051] In this embodiment, the specific steps of plane parallax decomposition include:

[0052] Define Π to represent a 3D plane, where H represents aligning Π on the target image I t and source image I s Homography transformation between; set p and p' to represent the 3D point coordinate system x = [X, Y, Z] in I t and I s projection on;

[0053] Transform p′ through the homography transformation H to obtain the image point p w :

[0054] p w ~Hp′

[0055] We omit the step of converting to homogeneous coordinates. Note that bys Transform and get the aligned image I w , so that the plane Π matches between them, then p w The displacement between and p is calculated as:

[0056]

[0057] Where K is the intrinsic parameter matrix of the camera, t = [tx, ty, tz] is the target image I t and source image I s The translation vector between c is the distance from the source view camera to the plane Π; the structure is represented by γ = Zh, where h is the distance from point x to Π; where, when x is on the plane Π, that is, h = 0, then p w =p;

[0058] Typically, we design a self-supervised monocular depth estimation network framework, which consists of a deep network for estimating depth and a pose network for estimating the translation between frames; it should be noted that, unlike other methods, we do not need to estimate the rotation between two views. Specifically, the pose network receives the source image and the image I s ,I w , and outputs the translation vector t; the deep network receives the target image I t , and output I t Depth map

[0059] For each pixel p = [x, y], such that Represents its estimated depth, using the camera intrinsics and the estimated depth to backproject p to obtain the corresponding 3D point In the camera coordinate system, the formula is:

[0060]

[0061] Therefore, if Figure 2 As shown on the right side, the following relationship is obtained:

[0062]

[0063] Where N is the normal vector of the plane Π, Yes The estimated distance to plane Π, and is an estimate of the structural variable γ; therefore, we obtain the reconstructed target image I t All the parameters required are obtained by aligning the image I w Transform to get

[0064] For It For each pixel p on the image, we calculate p based on the depth and translation predicted by the two networks. w , and then inverse transform I w and obtain To rebuild I t :

[0065]

[0066] By minimizing I t and to monitor the differences between

[0067] In order to obtain aligned images, preprocessing steps are performed on the dataset to obtain aligned images;

[0068] By using the road as the plane Π, a homography is calculated for each pair of consecutive frames, and the frames are transformed according to the calculated homography to obtain a uniform distribution of the target image I t and source image I s Calculate a homography H, and then transform I according to H s Transform and obtain the transformed source image I w .

[0069] In this embodiment, the specific steps of the monocular depth estimation network framework based on self-supervision include:

[0070] The specific steps of the monocular depth estimation neural network with scale are:

[0071] Design a monocular depth estimation network framework based on the U-Net architecture, and use the ResNet pre-trained on ImageNet as the encoder of the depth estimation network;

[0072] For the decoder, we use the same architecture as the current monocular depth estimation network, and estimate the depth directly by multiplying the output of the last sigmoid layer by 250, that is, the predicted maximum depth value, instead of estimating the disparity as in the general architecture. t takes as input, and outputs a depth estimate for each pixel (the output of our depth decoder is at a scale in meters);

[0073] The second network receives the transformed source image I w and the target image I t As input, and outputting only the translation vector between views, the network is similar to the pose network, except that in our case the output is a three-element vector representing the translation, and the scale is in meters;

[0074] The self-supervised training loss of the network is defined as:

[0075] Define the self-supervised training loss of the network and define the photometric loss function as a linear combination of L1 distance and structural similarity (SSIM) to minimize the target image I t and reconstruct the image The difference between , the photometric loss is defined as follows:

[0076]

[0077] Here, we set α = 0.85. Note that for each target image, one of the two aligned images is obtained by deforming the previous frame, and the other is obtained by deforming the next frame; the per-pixel minimum reprojection error is introduced, and the minimum value of Lphoto for each pixel of the aligned images across the previous frame and the next frame is calculated;

[0078] Define Lsmooth as an edge-based smoothness loss to encourage depth prediction to be locally smooth;

[0079] The total loss function is then a combination of Lsmooth and Lphoto averaged over all N pixels, and the formula is:

[0080]

[0081] Among them, min means calculating the minimum value of the alignment frame between the previous frame and the next frame, and λ is a hyperparameter used to control the impact of the loss term.

[0082] Beneficial effects:

[0083] Traditional self-supervised monocular depth estimation suffers from scale ambiguity, which affects accuracy;

[0084] The new method combines plane geometry and disparity metrics to improve the accuracy of depth estimation;

[0085] Aligning camera frames by a common ground plane eliminates the effect of rotational components;

[0086] Use independent neural networks to predict depth and camera translation respectively to improve prediction accuracy;

[0087] Based on the known camera height assumption, the target image is reconstructed to further improve the accuracy and stability of depth estimation.

[0088] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.

Claims

1. An accurate monocular depth estimation method based on plane and disparity metrics, characterized in that: The following steps are involved: Step S1, motion decomposition and plane definition: Based on plane parallax decomposition, a plane is defined, and the motion in the image is decomposed into two parts: plane homography and residual parallax, wherein the homography is used to represent the plane motion from the source image to the target image; The specific steps of plane parallax decomposition include: Define Π to represent a 3D plane, where H represents aligning Π on the target image and the source image Homography transformation between ; setting and Respectively represent the 3D point coordinate system x=[X,Y,Z] in and projection on Through homography transformation right Transform to get the image point : By the source image Transform to get aligned images , so that the plane π matches between them, then The displacement between and p is calculated as: Among them, K is the intrinsic parameter matrix of the camera, t=[tx,ty,tz] is the target image and the source image The translation vector between is the distance from the source view camera to the plane Π; the structure is represented by γ=Zh, where h is the distance from point x to Π; when x is on the plane Π, that is, h=0, then ; The monocular depth estimation network framework based on self-supervision is designed. The framework includes a deep network for estimating depth and a posture network for estimating the translation between frames. The posture network receives the source image and the image , , and outputs the translation vector t; the deep network receives the target image , and output Depth map ; For each pixel p=[x,y], such that Represents its estimated depth, using the camera intrinsics and the estimated depth to backproject p to obtain the corresponding 3D point In the camera coordinate system, the formula is: The following relationship is obtained: Where N is the normal vector of the plane Π and ĥ is the point The estimated distance to plane Π, is an estimate of the structural variable γ; obtain the reconstructed target image All the parameters needed to align the images Transform to get ; for For each pixel p on the image, the depth and translation predicted by the two networks are calculated , and then the reverse transformation and obtain To rebuild : By minimizing and to monitor the differences between Perform preprocessing steps on the dataset to obtain aligned images; By using the road as the plane Π, a homography is calculated for each pair of consecutive frames, and the frames are transformed according to the calculated homography to obtain a uniform distribution for each target image. and the source image Calculate a homography H for Transform and obtain the transformed source image ; Step S2, source image distortion and motion offset: distort the source image according to the defined plane homography to offset the motion of the plane; Step S3, residual parallax processing and depth prediction: after the source image is distorted, the remaining image motion is analyzed and processed; Step S4, implementation and integration: Integrate the optimized algorithm into the vision system of the autonomous driving vehicle and test and verify it in a real environment.

2. The method for accurate monocular depth estimation based on plane and disparity metrics according to claim 1, characterized in that: The specific steps of the monocular depth estimation network framework based on self-supervision include: Design a monocular depth estimation network framework based on the U-Net architecture, and use the ResNet pre-trained on ImageNet as the encoder of the depth estimation network; Using the same architecture as the current monocular depth estimation network, the depth is estimated by multiplying the output of the last sigmoid layer by the predicted maximum depth value, where the depth network is based on a single target image. Takes as input and outputs a depth estimate for each pixel; The second network receives the transformed source image and the target image As input, and outputting only the translation vector between views, the network is similar to the pose network, where the output is a three-element vector representing the translation; Define the self-supervised training loss of the network and define the photometric loss function as a linear combination of L1 distance and structural similarity to minimize the target image and reconstruct the image The difference between , the photometric loss is defined as follows: For each target image, two aligned images, one obtained by deforming the previous frame and the other obtained by deforming the next frame; introduce the per-pixel minimum reprojection error and calculate the minimum value of Lphoto for each pixel of the aligned images across the previous frame and the next frame; Define Lsmooth as an edge-based smoothness loss to encourage depth prediction to be locally smooth; The total loss function is then a combination of Lsmooth and Lphoto averaged over all N pixels, and the formula is: Among them, min means calculating the minimum value of the alignment frame between the previous frame and the next frame, and λ is a hyperparameter used to control the impact of the loss term.

Images