Geometric constraint based deep learning monocular depth estimation method

Abstract

The application provides a monocular depth estimation method based on geometric constraint and deep learning, which can efficiently estimate the depth information of an image scene while ensuring accuracy and integrity. The existing algorithms use deep learning methods for depth estimation, which only learn image data and ignore the spatial geometric prior information in the existing scene, resulting in poor depth estimation effect, slow learning speed, and large time and resource consumption. The application introduces a plane geometric constraint method to estimate the depth of the image scene in the deep learning process, which makes up for the aforementioned shortcomings of the existing methods.

Description

Technical Field

[0001] This invention is a deep learning monocular depth estimation method based on geometric constraints, belonging to the field of 3D stereo vision. Background Technology

[0002] Monocular depth estimation is an important research direction in computer vision. Its purpose is to estimate the scene depth information of a corresponding image from a single RGB scene image. Monocular depth estimation has a wide range of applications in autonomous driving, robotics, 3D modeling and other fields. The geometric constraint-based deep learning monocular depth estimation method adds planar geometric constraints during the deep learning process, obtains the corresponding feature maps by learning geometric parameters, and finally obtains the depth information of each pixel in the corresponding image by fusing these feature maps.

[0003] Although existing algorithms have achieved good depth estimation results, the depth estimation effect of existing methods is limited because the deep learning methods used in depth estimation only learn from image data and ignore the spatial geometric prior information in the existing scene. In addition, deep learning networks have problems such as difficulty in convergence, slow learning speed, and high time and resource consumption. Summary of the Invention

[0004] The purpose of this invention is to propose a geometrically constrained deep learning monocular depth estimation method that can efficiently estimate the depth information of an image scene while ensuring accuracy and completeness.

[0005] This invention estimates the depth of an image scene by introducing planar geometric constraints during deep learning. First, the original image is encoded by an encoder. Then, the encoded feature map is decoded by a decoder with planar geometric constraints. Finally, the decoded feature map is fed into a self-attention model to generate the corresponding depth map. By introducing planar geometric constraints into the decoder, this invention solves the problem of existing methods ignoring prior spatial geometric information, resulting in better performance in depth estimation.

[0006] The geometrically constrained deep learning monocular depth estimation method of the present invention includes the following: 1) Read the original image and input it into the encoder. The encoder encodes the original image. The encoder is composed of an EfficientNet B5 backbone network. The encoder generates feature maps by encoding the original image. There are 5 sizes of feature maps, which are 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image.

[0007] 2) The decoder reads feature maps of different sizes generated by the encoder and decodes them. During the decoding process, spatial planar geometric constraints are introduced to generate decoded feature maps. Upsampling is performed sequentially during the decoding process, and planar geometric constraints are introduced in the 1 / 4 to 1 / 2 feature map stage to generate a 1 / 2-sized feature map containing geometric information.

[0008] 3) The feature map obtained from the decoder is input into the self-attention model, which learns the depth distribution information in the scene, including adaptive depth block vectors and attention distribution features. Finally, the depth block vectors and attention distribution feature map are combined to generate the final depth map. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of a deep learning monocular depth estimation method based on geometric constraints.

[0010] Figure 2 This is a schematic diagram of a decoder structure based on geometric constraints. Detailed Implementation

[0011] To more clearly describe the technical content of the present invention, the invention will be further described below in conjunction with specific embodiments.

[0012] 1) Read the original image with a size of H×W×3, where H is the number of pixel rows in the image, W is the number of pixel columns in the image, and 3 represents an RGB three-channel image. H×W×3 means that the image as a whole is an RGB three-channel image with a length and width of H and W respectively. Input the original image into the encoder network Efficient Net B5. The encoder includes nine convolutional layers and generates feature maps of five sizes, which are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image.

[0013] 2) A 1 / 32-sized feature map is passed through a 3×3 convolutional layer and upsampled, then merged with a 1 / 16-sized feature map via residual connections. The merged feature map is then upsampled and merged with a 1 / 8-sized feature map via residual connections.

[0014] 3) The merged feature map is upsampled and then merged with a feature map of 1 / 4 size. The feature maps of different sizes are then merged with multiple depth geometric constraint layers of different granularities through residual connections, namely 1×1, 2×2, 4×4 and 8×8 granularities. The depth geometric constraint layer introduces spatial geometric prior information into the overall network, which improves the depth estimation effect.

[0015] 4) The calculation steps for the deep geometric constraint layer are as follows: A point P in the same plane in space satisfies the plane formula:

[0016] in, Let n be the plane parameter, n be the normal vector of the space plane, and a, b, c be the three components of the normal vector n. In the pinhole camera model, a coordinate system is established with the camera's optical center as the origin, the sensor plane as the XY plane, and the Z-axis is established in the direction of the sensor. A point (u, v) in the camera plane and a point P(X, Y, Z) in space satisfy the following mapping relationship:

[0017] Where u is the horizontal coordinate of point P on the image plane, v is the vertical coordinate of point P on the image plane, u0 is the horizontal offset of the principal point relative to the camera's optical center, and v0 is the vertical offset of the principal point relative to the camera's optical center. The principal point is also the origin of the image plane coordinate system. It is the distance from the image plane to the optical center, which is the theoretical basis of the finite camera model; Substituting the coordinates (X, Y, Z) of point P in space into the plane formula (1), we get:

[0018] After rearranging equation (3), the formula for calculating the depth Z of a point in space is:

[0019] By learning the calculation parameters of different granular plane depths from the feature maps of the previous stage, a depth geometric constraint layer is obtained, which contains depth feature maps of different granular planes.

[0020] 5) Merge the depth feature map in the depth geometry constraint layer with the feature map of size 1 / 4.

[0021] 6) After upsampling, merge it with the feature map of size 1 / 2.

[0022] 7) Finally, after a 3 A convolutional layer of 3 is used to obtain the corresponding decoded feature map.

[0023] 8) Input the decoded feature map into the self-attention model, convolve the decoded feature map through a 16×16 convolutional layer to generate a block map vector, and feed the block map vector into the self-attention model encoder to learn the adaptive depth block vector b for each pixel. i and depth probability distribution p i .

[0024] 9) Apply the following formula to the adaptive depth block vector b i Convert to depth block center vector c(b) i), c(b) in formula (5) i It's block b. i The center is mapped back to the actual depth value in the dataset:

[0025] Where, d min， d max These represent the upper and lower bounds of the distance in the monocular depth estimation model, respectively, and are hyperparameters that can be set according to the problem. 10) Finally, the depth block center vector c(b) is... i ) and depth distribution probability p i Combined to generate the final estimated depth:

[0026] Where b k Represents the adaptive depth block vector b i The k-th component, p k p represents the depth distribution probability i and b i The kth value corresponds to the kth component. N represents the depth block center vector, which is divided into N blocks and is a hyperparameter that can be set. 11) Calculate the loss based on the estimated depth and the actual depth, and update the network weights through backpropagation.

Claims

1. A deep learning monocular depth estimation method based on geometric constraints, characterized in that, It includes the following steps: 1) Read the original image with dimensions H×W×3, where H is the number of pixel rows, W is the number of pixel columns, and 3 represents an RGB three-channel image. W 3. The image as a whole is an RGB three-channel image with a length and width of H and W respectively. The original image is input into the encoder network Efficient Net B5. The encoder includes nine convolutional layers and generates feature maps of five sizes, which are 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image. 2) The 1 / 32 feature map is passed through a 3×3 convolutional layer and upsampled, and then merged with the 1 / 16 feature map through residual connections. The merged feature map is then upsampled and merged with the 1 / 8 feature map through residual connections. The merged feature map is then upsampled and merged with the 1 / 4 feature map. 3) Feature maps of different sizes are then merged with multiple depth geometric constraint layers of different granularities through residual connections, namely 1×1, 2×2, 4×4, and 8×8 granularities. 4) The calculation steps for the deep geometric constraint layer are as follows: A point P in the same plane in space satisfies the plane formula: in , For planar parameters, In the pinhole camera model, a point (u, v) in the camera plane and a point P(X, Y, Z) in space satisfy a mapping relationship: Substituting the coordinates (X, Y, Z) of point P in space into the plane formula (1), we get: After rearranging equation (3), the formula for calculating the depth Z of a point in space is: The calculation parameters for the depth of planes of different granularities are learned from the feature maps of the previous stage, and a depth geometric constraint layer is obtained, which contains depth feature maps of planes of different granularities. 5) Merge the depth feature map in the depth geometry constraint layer with the feature map of size 1 / 4; 6) After further upsampling, merge with the feature map of size 1 / 2; 7) Finally, after a 3 A convolutional layer of 3 is used to obtain the corresponding decoded feature map; 8) Input the decoded feature map into the self-attention model, and then process it through 16... The 16-layer convolutional layer convolves the decoded feature map and generates a patch image vector. This patch image vector is then fed into the self-attention model encoder to learn the adaptive normalized depth patch vector b for each pixel. i and depth probability distribution p i ; 9) Apply the following formula to the adaptive depth block vector b i Convert to depth block center vector c(b) i ), c(b) in formula (5) i It's block b. i The center is mapped back to the actual depth value in the dataset: 10) Finally, the depth block center vector c(b) of each pixel is divided. i ) and depth distribution probability p i Combined to generate the final estimated depth: 11) Calculate the loss based on the estimated depth and the actual depth, and update the network weights through backpropagation.

Images