An unsupervised monocular depth estimation method based on depth interval estimation

Abstract

The application discloses an unsupervised monocular depth estimation method based on depth interval estimation, and specifically comprises the following steps: a depth estimation model is used to process a video frame as follows: 1: a feature image F of a current frame is extracted; 2: a spatial tensor of F is spatially reconstructed to obtain a vector x p ; 3: an encoding sequence is obtained according to x p ; 4: the encoding sequence is fed into a Transform module to obtain a depth interval distribution b and a range attention map P; and 5: a depth estimation value of the current frame is calculated according to P and b. The application can improve the quality and accuracy of depth estimation in various scenes.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology. Background Technology

[0002] Scene depth estimation is a classic problem in computer vision, playing a crucial role in many areas, including the understanding and reconstruction of 3D scenes. Dense depth maps from the real world can significantly advance many applications, such as autonomous driving, obstacle avoidance in drones, scene perception, augmented reality, and semantic segmentation of images.

[0003] Traditional methods for acquiring high-precision target depth information typically employ depth sensors, including LiDAR, structured light-based depth sensors, and stereo cameras. However, depth maps obtained through these devices are costly, highly susceptible to environmental changes, and are primarily used in the technology research and development and testing phases, not yet widely adopted. In contrast, image-based depth estimation methods are low-cost, effectively overcome the shortcomings of the aforementioned sensors, and have a wider range of applications.

[0004] Stereo vision methods are a common approach to recovering scene depth from ordinary 2D images. They involve using two cameras to observe the same scene, obtaining two images, and then using triangulation to derive depth information from the parallax between the two images. However, stereo vision methods require at least two cameras, and their relative positions must remain fixed, limiting their application. For scenes with sparse textures, it's difficult to find enough features for matching, leading to significantly increased depth estimation errors or even failure in binocular images. Therefore, many researchers are now focusing on depth estimation from monocular images.

[0005] Monocular depth estimation is a method for obtaining scene depth using a single image. Methods for estimating depth from a single RGB image are also called monocular depth estimation methods. This has been a hot research topic in computer vision in recent years. However, this method suffers from ill-posedness, a problem stemming from the fact that a single RGB image may correspond to an infinite number of real-world scenes, while the image lacks stable cues to constrain these possibilities. Because monocular images lack reliable depth cues such as motion and stereoscopic relationships, recovering depth, which is only possible in three-dimensional space, is inherently an ill-posed problem. Theoretically, the true depth of a point in a monocular image can have an infinite number of solutions. Therefore, researchers have proposed various methods to achieve depth estimation from monocular images.

[0006] Existing monocular depth estimation methods typically estimate the absolute depth of all pixels in an image. This method is susceptible to environmental interference, leading to large estimation errors and neglecting global pixel information. Furthermore, absolute depth estimation of all pixels is unnecessary in many scenarios. Additionally, the weights for each pixel in depth estimation are consistent across all regression networks, causing the training network to adjust parameters for less important pixels. This results in increased computational cost and reduced accuracy. Moreover, current techniques have weak generalization capabilities because the kernel of a convolutional network is a fitting function, leading to overfitting. A good fit for one scene may result in a weak fit for another. Summary of the Invention

[0007] Purpose of the invention: In order to solve the problems existing in the prior art, the present invention provides an unsupervised monocular depth estimation method based on depth interval estimation.

[0008] Technical Solution: This invention provides an unsupervised monocular depth estimation method based on depth interval estimation. A depth estimation model is constructed and trained. The trained depth estimation model then processes the video frame sequence as follows:

[0009] Step 1: Take the t-th frame in the video frame sequence as the current frame I. t The convolutional neural network extracts the feature image F of the current frame; F has a length of H, a width of W, and a number of channels of C.

[0010] Step 2: Input F into the Transform module. The Embedding convolutional layer outputs a spatial tensor corresponding to F. Then, spatial reconstruction is performed on this spatial tensor to obtain a one-dimensional vector x. p ;

[0011] Step 3: Randomly generate a vector x that is the same as a one-dimensional vector. p Vectors of equal length, and x p Add the resulting vector to obtain the encoded sequence;

[0012] Step 4: Feed the encoded sequence into the Transform module. The Transform module outputs the depth sub-interval distribution b and the range attention map P.

[0013] Step 5: Calculate the depth estimate of the current frame based on the range attention map P and the N-dimensional depth sub-interval distribution b.

[0014] Furthermore, step 5 specifically involves calculating the depth estimate of the x-th pixel in the current frame according to the following formula.

[0015]

[0016] Where P[x] represents the x-th vector of the range attention map P, S(b k The expression ) represents the statistical result of the depth sub-interval distribution b, as shown below:

[0017]

[0018] Where k = 1, ..., N, b k d is the k-th dimension value of the depth subinterval b in N-dimensional vector form. min This represents the preset minimum depth value, d. max This indicates the preset maximum depth value.

[0019] Furthermore, a pose estimation network and an encoding / decoding network are incorporated when training the depth estimation model; the pose estimation network is used to estimate the pose change estimation matrix between two adjacent frames. t' represents the frame adjacent to the current frame; the encoding / decoding network is used to construct the moving object masking module; specifically, the moving object masking module is constructed as follows:

[0020] The Cost-Volume structure for optical measurement error information is calculated using the following formula:

[0021]

[0022] Among them, C t′ [x,d] represents the photometric error of the x-th pixel between the current frame and the reconstructed frame at a random depth d; the value of d ranges from [d...]. min ,d max All integer depth values ​​between ]; and set C as the set of all pixel photometric errors between the current frame and the reconstructed frame obtained from each adjacent frame. t’ (d) Set all photometric errors C t’ (d) The summation yields the Cost-Volume structure of the optical measurement error information; in the above formula, pe t’ The expression for [x,d] is as follows:

[0023]

[0024] Where SSIM(.) represents the image structure similarity calculation function, and ||·||1 represents the L1 distance between image pixels. I represents the x-th pixel in the reconstructed frame of the current frame. t [x] represents the x-th pixel in the current frame. This indicates the reconstructed frame of the current frame. The expression is:

[0025]

[0026] Among them, I t’ This represents the frame adjacent to the current frame, where K is the camera intrinsic parameter. The function expression is:

[0027]

[0028] The Cost-Volume and the current frame are input into the encoding / decoding network to obtain the moving object mask module Mask.

[0029] Furthermore, when training the depth estimation model, the loss function L is:

[0030]

[0031] Among them, L rec L is the image reconstruction loss function. pixel Let L be the element center density loss function. bins For bidirectional chamfer loss function, For coefficients;

[0032] L rec The expression is:

[0033] L rec =∑ t ,pe(I t I t’→t )

[0034] Where I t′→t The expression is as follows:

[0035]

[0036] The expression for the function pe(.) is shown below:

[0037]

[0038] Among them, I a and I b These represent any two image frames;

[0039] L pixel The expression is as follows:

[0040]

[0041] Where T represents the number of pixels with effective depth values, and λ is a constant. d x This represents the actual depth value of the x-th pixel in the current frame;

[0042] L binsThe expression is:

[0043] L Bins =chamfer(X,S(b))+chamfer(S(b),X)

[0044] Where X represents all d * The statistical distribution values, chamfer(.) represents the chamfer distance calculation function; c(b) is the N c(b) values. k A set consisting of N S(b) groups; k = 1, ..., N. k ) Composition; d * By [d] min ,d max All of the above make C t' [x,d] is the smallest d composition.

[0045] Beneficial Effects: This invention combines the performance improvement of depth classification with the visual quality of depth mapping regression by using adaptive discretized depth intervals, resulting in more accurate depth estimation for monocular vision, better depth continuity, and better results under standard evaluation metrics. This invention divides the depth range into multiple units and uses the center value of each unit to adaptively estimate each image. This method improves the performance of self-supervised depth estimation while reducing the difficulty of network training and data acquisition. Furthermore, addressing the problems of low depth prediction accuracy and limited generalization ability of prediction models in many current learning-based monocular depth estimation studies, this invention uses a depth and self-motion network design based on depth distribution constraints to solve the problem of estimating high-quality dense depth maps from a single color image. Overall, this invention can improve the quality and accuracy of depth estimation in various scenarios. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method of the present invention;

[0047] Figure 2 This is an overview diagram of the overall framework of the model of the present invention;

[0048] Figure 3 Here is the flowchart for the deep computing module;

[0049] Figure 4 Here is a diagram of the pose estimation network structure;

[0050] Figure 5 This is a schematic diagram of visual synthesis based on reconstructed frames;

[0051] Figure 6 The encoding / decoding network framework diagram for the moving object masking module. Detailed Implementation

[0052] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0053] This invention provides an unsupervised monocular depth estimation method based on depth interval estimation, which completes image depth estimation according to the following steps S1-S6;

[0054] like Figure 1 and Figure 2 As shown, the feature extraction network and the depth calculation module are sequentially connected to form a depth estimation model. The feature extraction network outputs an incompletely decoded feature map of the encoded image to the depth calculation module in real time, and the depth calculation module outputs the depth estimation map of the image. Simultaneously, the pose estimation network takes consecutive frames as input to obtain the inter-frame camera transformation matrix (i.e., the pose change estimation matrix). This inter-frame camera transformation matrix is ​​input to the encoding / decoding network to obtain the image masking matrix (i.e., the moving object masking module). At the same time, a reconstructed frame of the current frame is constructed based on the depth estimation value of the current frame. The loss is calculated using the feature extraction network, the depth calculation module, the pose estimation network, and the encoding / decoding network for supervision, thus enabling training. The specific steps are as follows.

[0055] Step S1: Obtain a sequence of video frames arranged in chronological order, and construct multiple training samples. Each training sample includes the current frame and the frames before and after the current frame.

[0056] Step S2: Obtain the incompletely decoded feature image F after encoding the current frame image.

[0057] Step S21: Input the current frame into the convolutional neural network. The convolutional neural network outputs an incompletely decoded feature image F with dimensions H*W*C, where H is the length of the feature image, W is the width of the feature image, and C is the number of channels of F.

[0058] like Figure 3 As shown, step S3: Constructing the depth calculation module, specifically: using the feature image F obtained in step S2 as input, inputting it into the transformer module, outputting the N-dimensional unit width vector b of the image depth interval distribution and the range attention map R, and calculating the depth estimate of the current frame based on the range attention map P and the N-dimensional depth sub-interval distribution b. The specific steps of step S3 are as follows:

[0059] Step S31: Input F obtained in step S2 into the Transform module. The Transform module contains an Embedding convolutional layer with a kernel size of p*p, a stride of p, and an output channel count of E. The output dimension of the Embedding convolutional layer is... The spatial tensor is then used to reconstruct the spatial tensor, thereby... The spatial tensor deformation yields a one-dimensional vector x. p ∈R S×E S = HW / p 2 R represents a real number, and x p The effective sequence length fed into the Transformer.

[0060] Step S32: Randomly generate a vector x that is the same as a one-dimensional vector. p Vectors of equal length, and x p Adding this to the randomly generated vector yields the encoded sequence.

[0061] Step S33: Feed the encoded sequence into the Transform block. The Transform block outputs the depth sub-interval distribution b and the range attention map P.

[0062] Step S34: Calculate the current frame I according to the following formula t Each pixel depth value The image size is H*W, and H*W images are... The combination forms the depth estimate for the current frame.

[0063]

[0064] Where · represents the dot product of two vectors, P[x] represents the x-th vector of the range attention map, and S(b) represents the statistical result of the depth sub-interval b, the expression of which is as follows:

[0065]

[0066] Where b k d is the k-th dimension value of the depth subinterval b in N-dimensional vector form. min This represents the preset minimum depth value, d. max This represents the preset maximum depth value, consisting of N S(b) values. k ) form an N-dimensional vector S(b).

[0067] In training the depth estimation model composed of the feature extraction module and the depth calculation module, this embodiment introduces a pose estimation network and an encoding / decoding network.

[0068] like Figure 4 As shown, step S4: self-motion estimation based on spatial geometric constraints, for each training sample, including the current frame I t The current frame and its adjacent frames before and after it are used as input, with the inter-frame camera motion transformation matrix (i.e., pose change estimation matrix) as the input. As the output, a pose estimation network is constructed, and the reconstructed frame of the current frame is calculated using a formula. The specific steps are as follows:

[0069] Step S41: Transfer the current frame image I t Image I adjacent to the current frame t’ The input is fed into the pose estimation network, and the current frame image I is obtained through the convolutional layers and fully connected layers in the pose estimation network. t The scalar values ​​of translation and rotation between adjacent frames are used to generate a pose change estimation matrix. t′∈{…,t-2,t-1,t+1,t+2,…}.

[0070] Step S42: Obtain the depth estimate using step S3. By combining the pose change estimation matrix and the camera intrinsic parameter matrix K, the depth estimation map is converted into a 3D point cloud and the local coordinate system is transformed. It is then reprojected onto a 2D plane to obtain the reconstructed frame of the current frame. The calculation formula is as follows:

[0071]

[0072] Among them, p' e This represents the homogeneous form of any pixel in the source frame. P represents the depth estimate of the source frame. s Indicates the target frame with p' e The homogeneous form of the corresponding pixels. This represents the pose change estimation matrix from the source frame to the target frame. In this embodiment, the target frame is the reconstructed frame.

[0073] In the above formula, p' e Left multiplication p' e Mapping to 3D space, the origin of the homogeneous coordinate system is set to the camera position at the source frame, followed by left multiplication. The origin of the homogeneous coordinate system of the pixel is moved to the next frame, that is, the camera position at the target frame time, and finally multiplied by K on the left to map the point in three-dimensional space onto the two-dimensional pixel.

[0074] like Figure 5 and Figure 6 As shown, step S5: Based on the optical measurement error calculation formula, using the current frame and its reconstructed frame of the video sequence, camera intrinsic parameters K, and pose change estimation matrix... Based on the depth estimation map, the optical metric error information structure Cost-Volume is calculated. This Cost-Volume is then input into the encoding / decoding network to estimate the two-dimensional binary masking matrix. The moving object masking module is then constructed, and the specific steps are as follows:

[0075] Step S51: Calculate the Cost-Volume structure of the optical measurement error information according to the following formula:

[0076]

[0077] Among them, C t' [x,d] represents the photometric error of the x-th pixel between the current frame and the reconstructed frame at a random depth d; the value of d ranges from [d...]. min ,d max All integer depth values ​​between ]; and set C as the set of all pixel photometric errors between the current frame and the reconstructed frame obtained from any adjacent frames. t’ (d) Set all photometric errors C t’ (d) The summation yields the Cost-Volume structure of the optical measurement error information; in the above formula, pe t' The expression for [x,d] is as follows:

[0078]

[0079] Where SSIM(.) represents the image structure similarity calculation function, and ||·||1 represents the L1 distance between image pixels. I represents the x-th pixel in the reconstructed frame of the current frame. t [x] represents the x-th pixel in the current frame. This indicates the reconstructed frame of the current frame. The expression is:

[0080]

[0081] Among them, I t’ This represents the frame adjacent to the current frame, where K is the camera intrinsic parameter. The function expression is:

[0082]

[0083] Step S52: Input the Cost-Volume obtained in step S51, along with the current frame image, into the encoding / decoding network (the encoding / decoding network in this embodiment uses the ResNet encoding / decoding network) to estimate the moving object mask matrix Mask.

[0084] Step S6: Calculate the loss error using the error formula with the current frame, the reconstructed frame of the current frame, and the depth estimate of the current frame. In this embodiment, the loss function L during training consists of three parts: the image reconstruction loss function L... rec Cell center density loss function L pixel and bidirectional chamfer loss function L bins The formula for calculating the loss function L is as follows:

[0085]

[0086] L = Mask(L) rec +L pixel )L bins

[0087] L rec =∑ t’ pe(I t I t’→t )

[0088]

[0089] in,, It is L pixel The loss coefficient is set to 0.5 in this embodiment; the expression for the function pe(.) is as follows:

[0090]

[0091] Among them, I a and I b These represent any two image frames.

[0092] L Pixel The expression is as follows:

[0093]

[0094] Where T represents the number of pixels with effective face values, λ is a constant, and in this embodiment λ = 0.85, α = 10. d x This represents the actual depth of the x-th pixel in the current frame image.

[0095] L bins The expression is:

[0096] L Bins =chamfer(X,S(b))+chamfer(S(b),X)

[0097] Where X represents all d * The statistical distribution values, chamfer(.) represents the chamfer distance calculation function; c(b) is the N c(b) values. k A set consisting of N S(b) groups; k = 1, ..., N. k ) Composition; d * By [d] min ,d max All of the above make C t' [x,d] is the smallest d composition.

[0098] Make C t' [x,d] is the smallest d composition.

[0099] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

Claims

1. An unsupervised monocular depth estimation method based on depth interval estimation, characterized in that, The method involves constructing a depth estimation model and training it. The trained depth estimation model then processes the video frame sequence as follows: Step 1: Take the t-th frame in the video frame sequence as the current frame. A convolutional neural network extracts the feature image F of the current frame; F has a length of H, a width of W, and a number of channels. ; Step 2: Input F into the Transform module. The Embedding convolutional layer outputs a spatial tensor corresponding to F. Then, spatial reconstruction is performed on this spatial tensor to obtain a one-dimensional vector. ; Step 3: Randomly generate a one-dimensional vector. Vectors of equal length, and Add the resulting vector to obtain the encoded sequence; Step 4: Feed the encoded sequence into the Transform module. The Transform module outputs the depth sub-interval distribution. and range attention map ; Step 5: Based on the range attention map and N-dimensional depth sub-interval distribution The depth estimate of the current frame is calculated. ; When training the depth estimation model, a pose estimation network and an encoding / decoding network are used; the pose estimation network is used to estimate the pose change estimation matrix between two adjacent frames. ; This represents the frame adjacent to the current frame; the encoding / decoding network is used to construct the moving object masking module. ; When training the depth estimation model, the loss function L is: ； in, The image reconstruction loss function is... The unit center density loss function, For bidirectional chamfer loss function, For coefficients; Specifically, construct the moving object masking module. for: The Cost-Volume structure for optical measurement error information is calculated using the following formula: ； in, This represents the photometric error of the x-th pixel between the current frame and the reconstructed frame at a random depth d; the value of d ranges from... All integer depth values ​​between; the photometric error of all pixels between the current frame and the reconstructed frame obtained from each adjacent frame is set as a set. Set all photometric errors The summation yields the optical measurement error information structure Cost –Volume; in the above formula The expression is as follows: ； in, This represents the function for calculating image structural similarity. Represents image pixels distance, This represents the x-th pixel in the reconstructed frame of the current frame. This represents the x-th pixel in the current frame. This indicates the reconstructed frame of the current frame. The expression is: ； in, This represents the frame adjacent to the current frame, where K is the camera intrinsic parameter. The function expression is: ； The Cost-Volume and the current frame are input into the encoding / decoding network to obtain the moving object masking module. .

2. The unsupervised monocular depth estimation method based on depth interval estimation according to claim 1, characterized in that, Step 5 specifically involves calculating the depth estimate of the x-th pixel in the current frame using the following formula. : ； in, Representation range attention map The x-th vector, Represents the distribution of depth sub-intervals The statistical results are expressed as follows: ； in, , It is a depth sub-interval distribution in N-dimensional vector form. The Dimensional value, This represents the preset minimum depth value. This indicates the preset maximum depth value.

3. The unsupervised monocular depth estimation method based on depth interval estimation according to claim 2, characterized in that, The expression is: ； in The expression is as follows: ； function The expression is as follows: ； in, and These represent any two image frames; The expression is as follows: ； Where T represents the number of pixels with effective depth values, and λ is a constant. , This represents the actual depth value of the x-th pixel in the current frame; The expression is: ； Where X represents all The statistical distribution value, This represents the function for calculating chamfer distance; k = 1,…,N. From N composition; Depend on All of the above make The smallest d-component.

Images