A monocular absolute depth estimation method sensitive to high-resolution images

Abstract

This invention proposes a monocular absolute depth estimation method sensitive to high-resolution images. The method involves stitching binocular input images vertically using a horizontal data stitching module. Models trained on small datasets exhibit higher generalization ability and prediction accuracy. The image is then input into the model, and skip connections and a depth decoder module are used to enhance the sensitivity of depth estimation at high resolutions, resulting in a normalized relative depth map. This relative depth map is then processed by an absolute scale recovery module to obtain an absolute scale depth map with true depth. This method addresses the problems of scale ambiguity, limited accuracy improvement at high resolutions, and poor generalization on small datasets in monocular depth estimation. Furthermore, it demonstrates strong adaptability to application scenarios involving dynamic objects, occlusion, and stationary cameras.

Description

Technical Field

[0001] This invention relates to the field of image estimation, and in particular to a monocular absolute depth estimation method that is sensitive to high-resolution images. Background Technology

[0002] Depth estimation is widely used in environmental perception, scene understanding, and 3D reconstruction, playing a crucial role in these fields. Vision-based depth estimation is divided into monocular and multi-view depth estimation. Compared to multi-view depth estimation, monocular depth estimation has advantages such as lower cost, simpler installation, and easier data acquisition, making it a more challenging and promising research direction. Currently, mainstream monocular depth estimation methods are divided into supervised learning methods and self-supervised learning methods. Obtaining a large number of accurate ground truth depth values ​​using supervised learning methods is a significant challenge. Therefore, models based on self-supervised learning are closer to practical applications. However, the lack of data quantity and diversity is a thorny issue for monocular depth estimation methods, which reduces the model's generalization ability, and self-supervised monocular depth estimation inherently suffers from scale ambiguity. Furthermore, most methods that use multiple frames as input perform poorly in depth estimation for scenes containing dynamic objects, occlusion, or static scenes.

[0003] Currently, most methods for estimating monocular depth using deep learning have poor generalization ability for small datasets, do not significantly improve depth prediction results for high-resolution data, and most cannot recover the absolute depth. Summary of the Invention

[0004] The main objective of this invention is to overcome the aforementioned deficiencies in the prior art and propose a monocular absolute depth estimation method that is sensitive to high-resolution images. This method can improve the prediction accuracy of self-supervised monocular depth estimation under high-resolution cameras and predict the absolute depth. It overcomes the disadvantage of existing algorithms requiring large datasets for self-supervised training and has strong adaptability to application scenarios with dynamic objects, occlusion, and stationary cameras.

[0005] The present invention adopts the following technical solution:

[0006] A monocular absolute depth estimation method sensitive to high-resolution images includes:

[0007] The self-supervised monocular absolute depth estimation network is input with a monocular image sequence / binocular image pair. The self-supervised monocular absolute depth estimation network includes a data horizontal stitching module, a skip connection and depth decoder module, and an absolute scale recovery module.

[0008] The input image is augmented by a data horizontal stitching module, and a self-supervised training network is constructed using reprojection loss and multi-scale pixel smoothing loss.

[0009] High-resolution features are extracted using skip connections and a depth decoder module. Each frame of data is processed by skip connections and a depth decoder module to output a relative depth map of the corresponding frame.

[0010] The relative depth map is scaled back using the absolute scale recovery module to obtain an absolute depth map containing the absolute scale.

[0011] Specifically,

[0012] The data horizontal stitching module partially stitches together two pairs of binocular input images. The stitching method is determined by the stitching rate and the corresponding uniform probability. A new input image is reconstructed by vertically stitching a certain proportion of the region in another input image, and the two parts are randomly vertically flipped to obtain the stitched image and the vertically flipped image.

[0013] Specifically, the skip connection and deep decoder module, wherein the skip connection structure consists of three different types of nodes: a feature extraction node, a first feature fusion node, and a second feature fusion node, wherein the feature extraction node consists of residual blocks; the first feature fusion node has a 3×3 convolution operation; the second feature fusion node consists of a fully connected layer and a 1×1 convolutional layer; the disparity is decoded by the deep network decoder, which contains a 3×3 convolution and a sigmoid activation function; through dense skip connections, each node in the decoder obtains a final aggregated feature map, an intermediate aggregated feature map, and an original feature map from the encoder.

[0014] Specifically, the absolute scale recovery module includes:

[0015] Calculate the surface normals to determine a surface normal for each pixel in the input image; all pixels need to be projected into 3D space, as shown in the following formula:

[0016]

[0017] in, It represents the pixel in the i-th row and j-th column in a two-dimensional space with homogeneous coordinates. These are the corresponding points in three-dimensional space. This refers to the depth of the corresponding point. For camera internal parameters;

[0018] For each pixel, the 8-neighbor rule is used to determine the plane surrounding it. Its 8 neighbors are divided into 4 pairs, forming 4 surfaces, generating 4 surface normals, as shown in the following formula:

[0019] ;

[0020] in, It refers to The y-th element of the x-th pair The normal number of the surface; point The final normalized surface normal is obtained by normalizing and averaging four estimated normals:

[0021]

[0022] The calculated camera height is:

[0023]

[0024] in, It is the vector projection of the camera height;

[0025] Scale factors for absolute depth scale recovery The calculation formula is:

[0026]

[0027] in, The scale factor for recovering the true camera height and absolute depth scale. This is the scaling factor used to convert a relative depth map into an absolute depth map; thus, the scaling factor is obtained. Then, the conversion from relative scale depth map to absolute scale depth map is realized.

[0028] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0029] This invention provides a monocular absolute depth estimation method sensitive to high-resolution images. The method involves stitching binocular input images vertically using a horizontal data stitching module, resulting in a model trained on small datasets exhibiting higher generalization ability and prediction accuracy. The image is then input into the model, and skip connections and a depth decoder module are used to enhance the sensitivity of depth estimation at high resolutions, yielding a normalized relative depth map. Finally, the relative depth map is processed by an absolute scale recovery module to obtain an absolute scale depth map with true depth. This invention addresses the problems of scale ambiguity, limited accuracy improvement at high resolutions, and poor generalization on small datasets in monocular depth estimation. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the overall monocular self-supervised absolute depth estimation method provided in this embodiment of the invention.

[0031] Figure 2 This is a self-supervised network framework for obtaining relative depth maps provided in this embodiment of the invention;

[0032] Figure 3 This is a structural diagram of the data horizontal splicing module provided in an embodiment of the present invention;

[0033] Figure 4 This is a structural diagram of the absolute scale recovery module provided in an embodiment of the present invention;

[0034] Figure 5 This is a structural diagram of a novel deep network skip connection and deep decoder provided in an embodiment of the present invention; Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0036] Self-supervised monocular depth estimation methods are easier to train on without requiring ground truth depth data, and the cost is also lower. However, because there are no ground truth depth data for reference, self-supervised monocular depth estimation cannot obtain true scale information. Furthermore, most current methods offer only minor improvements in depth prediction accuracy for high-resolution inputs and exhibit poor generalization on small datasets. Therefore, addressing these issues remains a challenge.

[0037] The following detailed description will take a specific application of the present invention as an example.

[0038] like Figure 1 This is a schematic flowchart of the overall implementation of this invention. First, a monocular image sequence or a pair of binocular images is input. Further, the input images undergo data augmentation via a horizontal data stitching module to enhance the model's generalization ability. A self-supervised training network is constructed using reprojection loss and multi-scale pixel smoothing loss. Further, skip connections and a depth decoder module enable the network to acquire more semantic information and high-resolution features from the input images. Each frame of data is processed by the deep network to output a relative depth map for the corresponding frame. Further, the relative depth map undergoes scale recovery via an absolute scale recovery module to obtain an absolute depth map containing the absolute scale. This completes the entire monocular self-supervised absolute scale depth estimation method.

[0039] This invention provides a self-supervised monocular absolute depth estimation model sensitive to high resolution, such as... Figure 2 As shown, it specifically includes:

[0040] The network input consists of a monocular image sequence or a pair of stereo images, the data-augmented image, camera intrinsics, and the ground truth depth value of the corresponding input image. The input passes through a depth network and a pose network to obtain disparity maps (disp) at four scales and pose matrices (T) for adjacent frames. The disparity maps are transformed into depth maps (depth), which, combined with the pose matrix (T) and camera intrinsics, are converted into a corresponding 3D point cloud. This point cloud is then projected onto 2D pixel coordinates to reconstruct a 2D color image. The input image and the reconstructed 2D color image constitute the reprojection loss and multi-scale pixel smoothing loss constraints for self-supervised training.

[0041] This embodiment utilizes a data horizontal splicing module to improve the model's generalization ability when the amount of input data is small.

[0042] Combination Figure 3 As shown, the horizontal data stitching module is used to improve the prediction accuracy of small datasets when stereo image pairs are used as input to the network. This module partially stitches together two pairs of stereo input images, with the stitching method primarily determined by the stitching rate and the corresponding uniform probability. A new input image is reconstructed by vertically stitching a certain proportion of a region (probability p) from another input image, and these two parts are randomly vertically flipped to obtain the stitched image and the vertically flipped image, as shown. Figure 3 As shown. Furthermore, stitching applies not only to the target view but also to its corresponding depth cue and source view. However, each stitching operation can only be performed between images of the same category, for example, both being target views. The stitching hyperparameters for all input images in the same batch must be configured identically.

[0043] The network model in this embodiment of the invention acquires images for training in two ways: the input images can be a monocular continuous image sequence or a binocular image pair. A self-supervised deep network and pose network are constructed using image reconstruction loss, thereby causing the loss function to converge.

[0044] When using binocular image pairs as input to a network, a data horizontal stitching module can be used to improve the model's generalization ability when the amount of input data is small.

[0045] The module stitches two images with different semantics together vertically, dividing the images into small batches for processing.

[0046] Furthermore, the absolute scale recovery module provided in this embodiment of the invention performs scale recovery after predicting the relative depth map, thereby enabling the generation of an absolute depth map solely based on monocular images; it also provides a monocular absolute depth scale recovery method based on a fixed camera height, such as... Figure 4As shown, dense ground points are formed by identifying the ground. Each point on the ground is used to estimate the height of a camera, which can obtain a large number of camera heights. The surface normal vector method is used to calculate the camera height obtained from each ground point. The estimated average camera height is obtained by statistical methods. Height values ​​that differ too much from the average value are discarded. Finally, a camera height calculated from a large number of ground points is obtained. The module needs to be used when there are enough ground points in the monocular image and the camera height above the ground must be fixed.

[0047] Statistical methods are used to estimate the overall camera height, as outliers have almost no impact on the scale factor estimation results. The first step is to calculate the surface normal, determining a surface normal for each pixel in the input image. All pixels need to be projected into 3D space, as shown in the following formula:

[0048]

[0049] in, It represents the pixel in the i-th row and j-th column in a two-dimensional space with homogeneous coordinates. These are the corresponding points in three-dimensional space. This refers to the depth of the corresponding point. This refers to the camera's internal parameters.

[0050] For each pixel, the 8-neighbor rule is used to determine several planes surrounding it. Its 8 neighbors are divided into 4 pairs, forming 4 surfaces, thus generating 4 surface normals, as shown in the following formula:

[0051]

[0052] in, It refers to The y-th element of the x-th pair This is the index of the surface normal. Point The final normalized surface normal is obtained by normalizing and averaging four estimated normals:

[0053]

[0054] The calculated camera height is:

[0055]

[0056] in, It is a vector projection of the camera height.

[0057] Furthermore, the calculated camera height is divided by the actual camera height to obtain a scaling factor. This factor is the scaling factor for converting the relative depth map into an absolute depth map, thus completing the restoration of the absolute scale and obtaining the absolute depth map.

[0058] Scale factors for absolute depth scale recovery The calculation formula is:

[0059]

[0060] in, To obtain the scale factor for the actual camera height. After that, the conversion from relative scale depth map to absolute scale depth map can be realized.

[0061] The monocular self-supervised deep network employs a network architecture based on monodepth2, which utilizes skip connections. Skip connections are a core component of U-Net, aiming to recover information lost during downsampling. However, directly combining features from different network layers may yield poor results due to differences in semantic level and spatial resolution. Deep neural networks exhibit more semantic features as the layers deepen. Since low-level features can include more semantic information, such as relatively clearer semantic boundaries, fusion becomes easier, allowing for a clearer depth estimate to be obtained by decoding these features.

[0062] To reduce the semantic and spatial resolution gap between different layers, this invention also proposes a novel skip connection architecture and deep network decoder, significantly improving depth prediction accuracy for high-resolution input images. Its structure is as follows: Figure 5 As shown, the network mainly consists of three different types of nodes. Ei is a feature extraction node, mainly composed of residual blocks. Ci represents a feature fusion node with only 3×3 convolution operations. Di is a feature fusion node, mainly composed of fully connected layers and 1×1 convolutional layers. The disparity is decoded by the deep network decoder, which contains 3×3 convolutions and a sigmoid activation function. Through dense skip connections, each node in the decoder presents the final aggregated feature map, intermediate aggregated feature maps, and the original feature map from the encoder. The decoder can then use high-resolution features with richer semantic information to predict a sharper depth map.

[0063] The decoder compresses feature maps to represent channel information using global average pooling and employs two fully connected (FC) layers and a sigmoid function to measure the importance of each feature while reweighting them. Then, 1×1 convolutions are used to fuse the channels to obtain high-quality feature maps. In addition to the original encoder and decoder nodes of UNet, numerous intermediate nodes are added to aggregate image features, forming dense skip connections. Through this dense skip connection structure, each node in the decoder presents the final aggregated feature map, intermediate aggregated feature maps, and the original feature map from the encoder. The decoder can then leverage high-resolution features with richer semantic information to predict sharper depth maps.

[0064] This invention provides a self-supervised monocular absolute depth prediction method. Based on the Monodepth2 self-supervised network framework, it proposes a data horizontal stitching module, an absolute scale recovery module, a novel deep network skip connection, and a deep network decoder. By directly inputting consecutive monocular image sequences or binocular image pairs, it can output absolute scale depth maps, exhibiting high generalization ability on small datasets and high depth prediction accuracy at high resolutions. The embodiments of this invention can run in real-time on a single GPU and CPU; depth prediction on a CPU alone can achieve 39fps.

[0065] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A monocular absolute depth estimation method sensitive to high resolution images, characterized in that, Comprise: Input monocular image sequence / binocular image pair in self-supervised monocular absolute depth estimation network, the self-supervised monocular absolute depth estimation network includes data horizontal splicing module, skip connection and depth decoder module and absolute scale recovery module; Input image is enhanced by data horizontal splicing module, and is constituted into self-supervised training network by using re-projection loss and multi-scale pixel smoothing loss; High-resolution features are extracted by using skip connection and depth decoder module, and relative depth map of each frame data is output by skip connection and depth decoder module; Relative depth map is recovered by absolute scale recovery module, and absolute depth map containing absolute scale is obtained; The skip connection and depth decoder module, wherein the skip connection structure is composed of three different types of nodes;Respectively, feature extraction node, first feature fusion node, second feature fusion node, wherein the feature extraction node is composed of residual block;The first feature fusion node has 3x3 convolution operation;Second feature fusion node is composed of full connection layer and 1x1 convolution layer;Parallax is decoded by depth decoder module, and the depth decoder module includes 3x3 convolution and Sigmoid activation function;Through dense skip connection, each node in the decoder obtains final aggregated feature map, intermediate aggregated feature map and original feature map from the encoder; The absolute scale recovery module, specifically comprising: Calculate the surface normal, determine a surface normal for each pixel in the input image;All pixel points need to be projected into 3D space, the formula is as follows: ； wherein, is a pixel in the i-th row and j-th column of a two-dimensional space with one homogeneous coordinate, is the corresponding point in a three-dimensional space, refers to the depth of the corresponding point, is the camera intrinsic parameter; For each pixel point, use 8-neighbor rule to determine the plane around it, its 8 neighbors are divided into 4 pairs, to form 4 curved surfaces, generate 4 curved surface normals, the formula is as follows: ； wherein, refers to the yth element of the xth pair, is the index of the surface normal; point The final normalized surface normal is obtained by normalizing and averaging the four estimated normals: ； The calculated camera height is: ； wherein is the vector projection of the camera height; Scale factor for absolute depth scale restoration The formula for calculating the scale factor is: ； wherein, is the scale factor for absolute depth scale recovery for the real camera height is the scale factor for absolute depth scale recovery for the real camera height; and is the scale factor for absolute depth scale recovery for the real camera height. 2.The monocular absolute depth estimation method sensitive to high-resolution images according to claim 1, characterized in that, The data horizontal splicing module is to splice two pairs of binocular input images partially, and the splicing mode is determined by splicing rate and corresponding uniform probability;A new input image is reconstructed by vertically splicing a certain proportion of area in another input image, and the two parts are randomly vertically flipped to obtain spliced image and vertically flipped image after splicing.

Images