Unsupervised vehicle-mounted monocular depth estimation method based on confidence mask

Abstract

The present invention discloses an unsupervised vehicle-mounted monocular depth estimation method based on confidence masking. The implementation steps are as follows: constructing an unsupervised vehicle-mounted monocular depth estimation network; a feature extraction subnetwork outputs feature maps of the source frame and the target frame; a posture estimation subnetwork outputs a projection matrix from the source frame to the target frame; a dynamic target perception subnetwork outputs a confidence mask; and a depth decoding subnetwork recovers the depth map of the target frame. By modeling the confidence mask output by the dynamic target perception subnetwork, the present invention can effectively reduce errors introduced by missing data or noise. In particular, under low-quality input, the robustness of depth reconstruction of moving targets in dynamic scenes is improved, enabling the present invention to obtain more detailed depth information and better visual effects in dynamic areas. The method can be effectively applied to unsupervised vehicle-mounted depth perception scenarios for autonomous driving.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and further relates to a method for unsupervised monocular image depth estimation based on confidence masks. This method can be used to perceive depth information of the surrounding environment through a monocular camera in driving scenarios. Background Art

[0002] In the field of monocular depth estimation, the goal is to estimate the depth information of a scene from a single image. Monocular depth estimation methods based on supervised learning assume that there is a nonlinear mapping relationship between the RGB image and the corresponding depth image. This mapping relationship is learned from a large amount of data, and the extracted depth features are used to reconstruct the depth information of the scene. Therefore, monocular depth estimation methods can be used for environmental depth perception in driving scenarios. Supervised learning methods rely on a large amount of high-quality data, and obtaining high-quality depth data requires a lot of manpower and material costs. Existing depth data is very limited and lacks diversity, resulting in poor reconstruction of scene depth information by current monocular depth estimation, poor generalization performance in different scenarios, and difficulty in practical application.

[0003] Jilin University disclosed a monocular depth estimation method in its patent application "An unsupervised monocular depth estimation method based on uncertainty analysis" (patent application number: 202111185472.5; application publication number: CN114549297A). This method proposes an unsupervised depth estimation network based on uncertainty, and proposes a customized likelihood function to represent uncertainty based on the observed depth, the depth of the model output, and the variance of the noise, and quantify the uncertainty of the output result. Finally, it is proposed to use Retinex illumination theory to deal with illumination changes in the scene, so as to construct a brightness loss function, and reconstruct the depth information of the scene based on the brightness loss, smoothness loss and uncertainty loss in unsupervised learning. The shortcoming of this method is that although the Retinex illumination theory is used to solve the interference of illumination changes on depth estimation, in complex dynamic environments, illumination changes will still affect the accuracy of depth estimation, resulting in a decrease in the reconstruction effect of depth information of dynamic scenes.

[0004] Zhou et al. proposed an unsupervised monocular depth estimation method based on a generative adversarial network in their paper “Unsupervised monocular depth estimation using adversarial networks” (Transactions on Image Processing. IEEE, 2022: 4305-4319). The method introduces an adversarial training mechanism between a generator and multiple discriminators. Specifically, the generator is tasked with generating a depth map from a monocular image, while multiple discriminators are used to evaluate the authenticity of the generated depth map, thereby promoting the generator to learn depth information. The core innovation of this method is to optimize the generator and discriminator through adversarial loss, thereby improving the accuracy and robustness of depth estimation. Although the depth map prediction effect of this method is significant, the method still has the disadvantage that, since the method relies on the texture information of the image to infer the depth, it may be difficult for the generator to accurately infer the depth in low-texture areas, resulting in large fluctuations in the error of the depth map estimation in low-texture areas and a decrease in accuracy. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the above-mentioned existing technologies and propose an unsupervised vehicle-mounted monocular depth estimation method based on confidence mask, aiming to solve the problems of poor robustness of existing monocular depth estimation technologies and difficulty in reconstructing depth information of moving targets in dynamic scenes.

[0006] The technical idea for achieving the purpose of the present invention is that the present invention can generate a confidence mask for dynamic targets by constructing a dynamic target perception subnetwork, which can clearly mark the confidence of the areas that need attention. During the network training phase, based on the confidence mask, a lower weight is applied to the occluded areas where the depth estimation is unreliable, thereby avoiding the erroneous influence of these areas, and a higher weight is given to the areas with high confidence, thereby improving the depth estimation accuracy of the area. This mechanism helps the network model automatically ignore unreliable depth predictions when facing complex scenes (such as dynamic objects, low-texture areas or highly reflective areas, etc.), and focus on optimizing reliable areas, thereby reducing errors and improving the accuracy of depth estimation, thereby solving the problem of difficulty in reconstructing moving targets in dynamic scenes in the prior art. The present invention forms a closed-loop system through the alternating mechanism of feature-level optimization of forward reasoning and image-level optimization of back propagation. Each round of optimization can use the results of the previous round to adjust the model parameters and gradually reduce the error in the depth map prediction. This mechanism enables the model to obtain more accurate depth maps in each round of iteration by continuously refining the prediction. Compared with existing methods, the closed-loop mechanism of the present invention can make full use of previous optimization results, avoid the local optimal problem that may be caused by a single optimization, and thus improve the accuracy and robustness of depth estimation.

[0007] According to the above ideas, the technical solution of the present invention includes the following steps:

[0008] Step 1: Construct a dynamic target perception subnetwork. The first layer of 3D convolution has a convolution kernel size of 3×3×3, a stride of 1, and a padding of 1. The second layer of 3D convolution has a convolution kernel size of 5×5×5, a stride of 2, and a padding of 2. The third layer is a batch normalization layer, and the fourth layer is a sigmoid activation layer.

[0009] Step 2: Construct an unsupervised vehicle-mounted monocular depth estimation network consisting of a feature extraction subnetwork, a posture estimation subnetwork, a dynamic target perception subnetwork, and a depth decoding subnetwork;

[0010] Step 3: From the unsupervised vehicle-mounted monocular depth estimation dataset, read two adjacent frames of the video sequence one by one as the source frame and the target frame; all the source frames and the target frames form a training set;

[0011] Step 4: Input the training set into the unsupervised vehicle-mounted monocular depth estimation network, and use the mini-batch gradient descent method to iteratively update the parameters of the unsupervised vehicle-mounted monocular depth estimation network until the image reconstruction loss function converges, thereby obtaining a trained unsupervised vehicle-mounted monocular depth estimation network. The training process is as follows:

[0012] Step 4.1: Output the feature maps corresponding to the source frame and the target frame through the feature extraction sub-network;

[0013] Step 4.2: The pose estimation subnetwork outputs the forward projection matrix from the source frame to the target frame, and the reverse projection matrix from the target frame to the source frame.

[0014] Step 4.3: Use the forward projection matrix to project the feature map of the source frame to the target frame for feature alignment;

[0015] In step 4.4, the aligned features are input into the dynamic target perception subnetwork to obtain the confidence mask of the dynamic target;

[0016] Step 4.5, using the confidence mask and the feature map corresponding to the target frame, the depth decoding sub-network recovers the depth map of the target frame;

[0017] Step 4.6, using the inverse projection matrix, project the depth map of the target frame to the source frame, and calculate the image reconstruction loss between the source frame and the reconstructed source frame;

[0018] In step 5, two temporally adjacent frames of image data generated during the unsupervised driving process are input into the trained unsupervised vehicle-mounted monocular depth estimation network, and the depth estimation result of the scene is output.

[0019] Furthermore, the feature extraction subnetwork adopts the network structure design and parameter setting of the ResNet34 backbone network.

[0020] Furthermore, the posture estimation subnetwork adopts the network structure design and parameter setting of the posture estimation network in SfmLearner.

[0021] Furthermore, the unsupervised vehicle-mounted monocular depth estimation dataset is a sequence of RGB images of continuous frames; each frame of the image is accompanied by time sequence information between adjacent frames; the dataset does not require real depth labels.

[0022] Furthermore, the image reconstruction loss is obtained by the following formula:

[0023]

[0024] Among them, L re represents the image reconstruction loss, L pe represents the minimum photometric reprojection loss, L dis represents the parallax balance loss, N represents the total number of pixels in the image, i and j represent the coordinates of the pixels in the image, SSIM represents the structural similarity index, α is set to 0.85, I s and represent the source frame and the reconstructed source frame respectively, and Respectively represent the pixel with source frame coordinates (i, j) and the pixel with reconstructed source frame coordinates (i, j), and Represent the partial derivatives of the depth map with respect to x and y respectively.

[0025] Furthermore, the forward projection matrix is ​​as follows:

[0026] M proj =K·[R s→t |T s→t ]·K -1

[0027]

[0028] Among them, M proj represents the forward projection matrix, K represents the camera intrinsic parameter matrix, R s→t represents the forward rotation matrix, T s→t Represents the forward translation matrix, the superscript -1 represents the inverse operation, t x ,t y ,t z Represents the displacement along the x, y, and z axes of the three-dimensional coordinate system, θ x ,θ y ,θ z Indicates the rotation angle around the x, y, and z axes of the three-dimensional coordinate system.

[0029] Furthermore, the inverse projection matrix is ​​as follows:

[0030] M reproj =K·[R t→s |T t→s ]·K -1

[0031]

[0032] Among them, M reproj Represents the inverse projection matrix, R t→s represents the inverse rotation matrix, T t→s Represents the inverse translation matrix.

[0033] Furthermore, the feature alignment of the source frame by projecting the feature map to the target frame is obtained by the following formula:

[0034]

[0035] Among them, u s→t ,v s→t Represents the aligned feature plane F s The pixel coordinates, u s ,v s Denotes the characteristic plane F s The pixel coordinates of , d represents the depth of the camera.

[0036] Furthermore, the step of outputting the confidence mask of the dynamic target corresponding to the alignment feature through the dynamic target perception sub-network is as follows:

[0037] The first step is to solve the L1 norm of the aligned features to obtain the residual feature map;

[0038] In the second step, the feature maps of different depth layers in the residual feature map are aggregated through two 3D convolutional layers and a batch normalization layer of the dynamic object perception subnetwork;

[0039] In the third step, the different channel features of the aggregated features are accumulated and then activated by the nonlinear sigmoid of the dynamic target perception sub-network to obtain the confidence mask of the dynamic target.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] First, the dynamic target perception subnetwork constructed by the present invention generates a confidence mask for dynamic targets, which is used to indicate the degree of credibility of the depth estimation in each area. During the network training phase, based on the confidence mask, a lower weight is applied to the occluded areas where the depth estimation is unreliable, thereby overcoming the defects of poor robustness of the existing monocular depth estimation technology and avoiding the instability caused by target motion or occlusion. This enables the present invention to effectively reduce errors introduced by data loss or noise, especially in the case of low-quality input (such as low-resolution or blurred images). The robustness of the output can still be maintained. For dynamic targets in complex scenes, the confidence mask can control the depth estimation at the boundary, thereby improving the accuracy of unsupervised vehicle-mounted monocular depth estimation.

[0042] Second, the present invention constructs a loop consistency optimization based on negative feedback, obtains a depth map output by feature-level optimization through forward propagation, and then reversely reprojects the depth map to the input to calculate the loss for a round of back propagation. This iteration improves the network's depth reconstruction of dynamic targets, overcoming the problem that the existing technology is difficult to reconstruct depth information of moving targets in dynamic scenes, so that the present invention can obtain more refined depth information and better visual effects in dynamic areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the present invention;

[0044] Figure 2 Schematic diagram of the unsupervised vehicle-mounted monocular depth estimation network of the present invention;

[0045] Figure 3 2 is a comparison chart of the results of the simulation experiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] Reference Figure 1 , further describing the implementation steps of the embodiment of the present invention.

[0048] Step 1: Read two adjacent frames of the video sequence as the source frame and the target frame.

[0049] The training data and test data are read from a monocular image depth estimation dataset. The monocular image depth estimation dataset used in the embodiment of the present invention is the KITTI dataset.

[0050] Read two adjacent source frames I in the time series of the KITTI data set in sequence s and target frame I t As image input, used to build unsupervised learning.

[0051] Reference Figure 2 , the working process of the unsupervised vehicle-mounted monocular depth estimation network constructed by the present invention is further described.

[0052] The unsupervised vehicle-mounted monocular depth estimation network of the present invention is composed of a feature extraction subnetwork, a posture estimation subnetwork, a dynamic target perception subnetwork, and a depth decoding subnetwork.

[0053] Step 2. The feature extraction sub-network extracts the deep features of the source frame and the target frame respectively.

[0054] The feature extraction sub-network adopts the network structure design and parameter setting of ResNet34, the backbone network in the existing deep feature extractor. The backbone network ResNet34 includes a four-stage residual connection module BasicBlock. Its network structure and extracted image features are shown in Table 1.

[0055] Table 1. Feature extraction network structure and extracted image features

[0056]

[0057] The h and w in Table 1 are the height and width of the input image respectively.

[0058] Extract source frame I using ResNet34 s and target frame I t The features of F are obtained respectively s and F t , which is used for subsequent feature-level optimization based on the forward projection matrix to construct the cost volume of feature alignment under different depth assumptions to model the confidence of dynamic targets and reconstruct the depth map of the target frame.

[0059] Step 3. The pose estimation subnetwork obtains the forward and reverse projection matrix from the source frame to the target frame.

[0060] The pose estimation subnetwork uses the input source frame and target frame to estimate the forward translation matrix T of the mobile camera from the source frame to the target frame. s→t , forward rotation matrix R s→t， And the inverse translation matrix T from the target frame to the source frame t→s and the inverse rotation matrix R t→s , forward T s→t and forward R s→t As shown below,

[0061]

[0062] Among them, θ x ,θ y ,θ z Indicates the rotation angle around the x, y, and z axes, t x ,t y ,t z Represents the displacement along the x, y, and z axes. The conversion between the forward rotation matrix and the inverse rotation matrix, and the forward translation matrix and the inverse translation matrix can be performed according to the following formula:

[0063]

[0064] Among them, T t→s represents the inverse translation matrix, R t→s represents the inverse rotation matrix.

[0065] Step 4. Feature-level optimization of forward inference based on the forward projection matrix.

[0066] Assume that the camera intrinsic parameter matrix is ​​K, the camera focal length is f, and the world coordinate is P w (x w ,y w ,z w ), the camera coordinate is P c (x c ,y c ,z c ), pixel coordinate is P pix (u, v), dx, dy are the actual lengths of pixels, and the source frame camera coordinate system is used as the reference coordinate system.

[0067] The process of feature-level optimization is as follows:

[0068] Step 4.1, P w Convert to P c 、P c Go to P pix , the camera intrinsic parameter matrix K can be expressed as follows:

[0069]

[0070] Step 4.2, let the characteristic plane F s The pixel coordinates are (u s ,v s ), with a depth of d, let the feature plane The pixel coordinates are (u s→t ,v s→t ), through the camera model F s Forward reprojection to The feature alignment for the plane is as follows:

[0071]

[0072] Step 4.3, so far, the cost volume C of inter-frame feature alignment is constructed based on different camera depths d (i.e., parallax) d , is a multidimensional tensor used to represent the matching cost or similarity between deep features under different depth hypotheses, as shown below:

[0073]

[0074] Step 4.4, by the cost body C d The depth features of different depth layers are fused through three-dimensional convolution, and the target frame I is output through the dynamic target perception module DOM and the deep decoding network Decoder. t Normalized target depth map D t , d max is the maximum depth value, as follows:

[0075] D t =d max ·Sigmoid(Decoder(DOM(Conv 3D (C d ))))

[0076] Among them, the Decoder of the deep decoding network is used to decode the output of the dynamic target perception module DOM, upsample layer by layer to restore the depth details. The network structure is shown in Table 2.

[0077] Table 2. Overview of deep decoding network structure

[0078]

[0079] In Table 2, h and w are the height and width of the input image respectively.

[0080] In the process of deep feature alignment, the feature difference of dynamic targets is constructed under different depth assumptions. dThe DOM is enhanced, and the confidence mask of the dynamic target is obtained through a Sigmoid activation layer, which indicates the confidence distribution of the dynamic target area. The depth decoder network upsamples the depth features including the confidence mask and restores the depth details. The last layer of Sigmoid outputs a normalized depth map, which is compared with the maximum depth value d max The product of the target frame I is obtained t The corresponding target depth map D t ,This process is feature-level optimization of forward reasoning based on forward projection.

[0081] Step 5. Image-level optimization based on back-propagation of the inverse projection matrix.

[0082] A round of feature-level optimization for forward reasoning requires a round of image-level optimization for backpropagation to iteratively improve the output D t The accuracy of the target depth map D is obtained by t and target frame I t Combined with the translation matrix T s→t and the rotation matrix R s→t Back-reprojection to the source frame to construct an image-level aligned reconstructed source frame And establish I s and Self-supervised training is performed using the image reconstruction loss.

[0083] The process of image-level optimization is as follows:

[0084] Step 5.1, let the target frame I t The pixel coordinates are (u t ,v t ), the corresponding depth is D t (u t ,v t ), reconstruct the source frame The pixel coordinates are (u t→s ,v t→s ), the target frame I is transformed into t Reverse reprojection to reconstruct the source frame The images are aligned as follows:

[0085]

[0086] Among them, u s→t ,v s→t Represents the aligned feature plane F s The pixel coordinates, u s ,v s Denotes the characteristic plane F s The pixel coordinates of , d represents the depth of the camera.

[0087] Step 5.2, establish I s and The image reconstruction loss L re , by the minimum photometric reprojection loss L pe and parallax balance loss L dis Composition, that is

[0088] Moving targets or occluded areas in dynamic scenes will lead to increased reprojection errors because the pixels in these areas do not correspond in different views. By introducing the minimum reprojection loss L pe , dynamically masking invalid pixel areas caused by occlusion or dynamic targets to improve the robustness of the loss function, as shown below:

[0089]

[0090] Among them, L re represents the image reconstruction loss, N represents the total number of pixels in the image, i and j represent the coordinates of the pixels in the image, SSIM represents the structural similarity index, α is set to 0.85, I s and represent the source frame and the reconstructed source frame respectively.

[0091] Minimum reprojection loss L pe By introducing a combination of L1 and single-scale SSIM terms as the reprojection loss, self-supervised training based on visual geometric constraints without the need for deep labels is achieved.

[0092] Parallax smoothing loss L dis By penalizing unreasonable drastic changes in the depth map, the predicted depth map is made more continuous in space. This smoothness helps reduce noise, especially in areas where the reprojection loss cannot fully constrain the model. It can reduce the overfitting of the model to a specific data distribution and make the predicted depth performance more stable, as shown below:

[0093]

[0094] Among them, L dis Represents the parallax balance loss.

[0095] Step 5.3: The entire training process consists of a round of forward inference feature-level optimization and a round of back-propagation image-level optimization, forming a negative feedback loop consistency mechanism to iteratively improve the target depth map D in dynamic scenes. t precision.

[0096] The effects of the present invention are further described below with reference to experiments.

[0097] 1. Simulation experimental conditions.

[0098] The computer processor used in the simulation experiment of the present invention is Intel (R) Xeon (R) Gold 6148 CPU @ 2.40GHz, running memory 128G, and the graphics card is an NVIDIA GeForce RTX 3090 GPU with a video memory of 24GB. The operating system is 64-bit Ubuntu 18.04 (LTS), and the deep learning framework is PyTorch version 1.10.1. The network training uses the Adam optimizer to update the network parameters, wherein the parameters of the Adam optimizer are β1 = 0.9, β2 = 0.999, and the weight decay term is 0.01. 100 epochs are set, the initial learning rate is 1e-5, and it is linearly reduced to 1e-6 during the training iteration. The image reconstruction loss L is used. re To train the network.

[0099] The dataset used in the simulation experiments of this paper is the KITTI dataset, which is one of the widely used benchmark datasets in the fields of autonomous driving and computer vision, especially in the monocular depth estimation task. This dataset contains images captured from real-world scenes and corresponding sensor data, including depth information from laser radar (LIDAR). For the monocular depth estimation task, the KITTI dataset provides monocular image sequences and real depth maps (obtained by LIDAR) to facilitate training and evaluating depth estimation algorithms.

[0100] 2. Experimental content and results analysis:

[0101] The simulation content of the present invention is to use the method of the present invention and four existing technologies (MonoFormer method, PlaneDepth method, DS-Depth method, DCPI-Depth method) under the above experimental conditions to perform monocular depth estimation on the KITTI test dataset and obtain the predicted depth map, such as Figure 3 shown.

[0102] The four existing methods used in the simulation experiment of the present invention are as follows:

[0103] The MonoFormer method of the prior art refers to a method proposed by Bae et al. in their paper “Deep Digging into the Generalization of Self-Supervised Monocular Depth Estimation” (In Proceedings of the AAAI Conference on Artificial Intelligence, 2023, pp. 187-196.) to improve the performance of self-supervised depth estimation models on different datasets.

[0104] The PlaneDepth method of the prior art refers to a self-supervised depth estimation method based on orthogonal plane modeling proposed by Wang et al. in their published paper “PlaneDepth: Self-Supervised Depth Estimation Via Orthogonal Planes” (In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2023, pp. 21425-21434.), which more effectively captures and utilizes the geometric structure of the scene by introducing orthogonal plane constraints.

[0105] The DS-Depth method of the prior art refers to the fusion cost volume of dynamic and static depth estimation proposed by Miao et al. in their published "DS-Depth: Dynamic and Static Depth Estimation via a Fusion Cost Volume" (In Proceedings of the IEEE Transactions On Circuits and Systems For Video Technology, 2024, pp. 2564-2576.), which improves the accuracy in dynamic scenes by collaboratively optimizing the depth estimation of dynamic and static objects.

[0106] The DCPI-Depth method of the prior art refers to the dense correspondence prior introduced by Zhang et al. in their publication “DCPI-Depth: Explicitly Infusing Dense Correspondence Prior to Unsupervised MonocularDepth Estimation” (In Proceedings of the Computing Research Repository, 2024, pp. abs / 2405.16960) to explicitly capture the depth information in the image and better understand the global and local structure of the image.

[0107] The following combination Figure 3 The simulation diagram of the present invention is further described.

[0108] Figure 3 in Figure 3 (a) is the input image of the simulation experiment of the present invention, Figure 3(b) is the predicted depth map of the existing method MonoFormer, Figure 3 (c) is the predicted depth map of the existing method PlaneDepth, Figure 3 (d) is the predicted depth map of the existing method DS-Depth, Figure 3 (e) is the predicted depth map of the existing method DCPI-Depth, Figure 3 (f) is the predicted depth map of the method of the present invention.

[0109] To verify the error of the present invention's simulation experiments, the present invention and four existing techniques were evaluated using objective error metrics on the KITTI test set. The evaluation metrics were: absolute relative error (AbsRel), mean squared relative error (SqRel), root mean square error (RMSE), and root mean square logarithmic error (RMSElog) (mean logarithmic error log10). Lower values ​​for these four metrics indicate lower depth estimation error and better performance.

[0110] The evaluation results of the objective indicators of error in quality evaluation are shown in Table 3.

[0111] Table 3. Error objective index evaluation table of the present invention and four traditional methods on the KITTI test dataset

[0112] Method of the present invention AbsRel↓ SqRel↓ RMSE↓ RMSElog↓ MonoFormer 0.108 0.806 4.594 0.184 PlaneDepth 0.084 0.549 3.981 0.169 DS-Depth 0.095 0.698 4.329 0.173 DCPI-Depth 0.090 0.655 4.113 0.167 Ours 0.090 0.518 3.588 0.163

[0113] As can be seen from Table 3, the error term Sq Rel and RMSE of the proposed method on the KITTI test set are significantly better than those of other existing comparison methods. Figure 3 (f) It can be seen that the subjective results also have finer texture details, which further proves that the method of the present invention can better present the depth information in the image and significantly improve the overall perceptual quality of the image.

[0114] In order to verify the effect of the simulation experiment of the present invention, the threshold value δ<1.25 is used. i , i = 1, 2, and 3 to evaluate the model and the quality of the depth estimation results of the present invention and four traditional methods. The higher the three evaluation indicators, the higher the depth estimation accuracy. The results are shown in Table 4.

[0115] Table 4. Objective index evaluation table of the accuracy of the present invention and the comparative method on the KITTI test dataset

[0116] Method of the present invention δ1(%)↑ δ2(%)↑ δ3(%)↑ MonoFormer 0.884 0.963 0.983 PlaneDepth 0.911 0.968 0.984 DS-Depth 0.905 0.966 0.984 DCPI-Depth 0.914 0.969 0.985 Ours 0.916 0.971 0.986

[0117] As can be seen from Table 4, the accuracy terms δ1, δ2, and δ3 of the present invention on the KITTI test set are slightly ahead of the best comparison method. It is significantly better than other existing methods in terms of accuracy in dynamic target areas and achieves the best visual effect. This shows that the present invention is more robust and practical, and achieves better depth estimation effect.

Claims

1. An unsupervised vehicle-mounted monocular depth estimation method based on confidence mask, characterized in that: A dynamic target perception subnetwork is constructed to generate a confidence mask for the dynamic target. The confidence mask is used to recover the depth map of the dynamic scene during forward reasoning, and the depth map is used to calculate the image reconstruction loss during backpropagation. The steps of the estimation method include the following: Step 1: Construct a dynamic target perception subnetwork. The first layer of 3D convolution has a convolution kernel size of 3×3×3, a stride of 1, and a padding of 1. The second layer of 3D convolution has a convolution kernel size of 5×5×5, a stride of 2, and a padding of 2. The third layer is a batch normalization layer, and the fourth layer is a sigmoid activation layer. Step 2: Construct an unsupervised vehicle-mounted monocular depth estimation network consisting of a feature extraction subnetwork, a posture estimation subnetwork, a dynamic target perception subnetwork, and a depth decoding subnetwork; Step 3: From the unsupervised vehicle-mounted monocular depth estimation dataset, two adjacent frames of the video sequence are read one by one as the source frame and the target frame; All source frames and target frames form a training set; Step 4: Input the training set into the unsupervised vehicle-mounted monocular depth estimation network, and use the mini-batch gradient descent method to iteratively update the parameters of the unsupervised vehicle-mounted monocular depth estimation network until the image reconstruction loss function converges, thereby obtaining a trained unsupervised vehicle-mounted monocular depth estimation network. The training process is as follows: Step 4.1: The feature extraction subnetwork outputs the feature maps corresponding to the source frame and the target frame; In step 4.2, the pose estimation subnetwork outputs the forward projection matrix from the source frame to the target frame, and the reverse projection matrix from the target frame to the source frame; Step 4.3: Use the forward projection matrix to project the feature map of the source frame to the target frame for feature alignment; Step 4.4: Output the confidence mask of the dynamic target through the dynamic target perception sub-network. The steps are as follows: The first step is to solve the L1 norm of the aligned features to obtain the residual feature map; In the second step, two 3D convolutional layers and a batch normalization layer of the dynamic object perception sub-network are used to aggregate the feature maps of different depth layers in the residual feature map to obtain the aggregated features; In the third step, the sigmoid activation function of the dynamic target perception sub-network is used to activate the accumulated features of different channels of the aggregated features and obtain the confidence mask of the dynamic target; Step 4.5, using the confidence mask and the feature map corresponding to the target frame, the depth decoding sub-network recovers the depth map of the target frame; Step 4.6, using the inverse projection matrix, project the depth map of the target frame to the source frame, and calculate the image reconstruction loss between the source frame and the reconstructed source frame; In step 5, two temporally adjacent frames of image data generated during the unsupervised driving process are input into the trained unsupervised vehicle-mounted monocular depth estimation network, and the depth estimation result of the scene is output.

2. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1 is characterized in that The feature extraction subnetwork described in step 2 adopts the ResNet34 network structure design and parameter setting of the backbone network in the deep feature extractor.

3. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that The posture estimation subnetwork described in step 2 adopts the network structure design and parameter setting of the posture estimation network in SfmLearner.

4. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that The unsupervised vehicle-mounted monocular depth estimation dataset described in step 3 is a sequence of RGB images of continuous frames; each frame of the image is accompanied by the time sequence information between adjacent frames, and the dataset does not require real depth labels.

5. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that The image reconstruction loss function in step 4 is obtained by the following formula: Among them, L re represents the image reconstruction loss, L pe represents the minimum photometric reprojection loss, L dis represents the parallax balance loss, N represents the total number of pixels in the image, i and j represent the coordinates of the pixels in the image, SSIM represents the structural similarity index, α is set to 0.85, I s 、 represent the source frame and the reconstructed source frame respectively, Respectively represent the pixel with source frame coordinates (i, j), and the pixel with reconstructed source frame coordinates corresponding to the source frame coordinates (i, j), Represent the partial derivatives of the depth map with respect to x and y respectively.

6. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that The forward projection matrix described in step 4.2 is as follows: M proj =K·[R s→t |T s→t ]·K -1 Among them, M proj represents the forward projection matrix, K represents the camera intrinsic parameter matrix, R s→t represents the forward rotation matrix, T s→t Represents the forward translation matrix, the superscript -1 represents the inverse operation, t x ,t y ,t z Represents the displacement along the x, y, and z axes of the three-dimensional coordinate system, θ x ,θ y ,θ z Indicates the rotation angle around the x, y, and z axes of the three-dimensional coordinate system.

7. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 5, characterized in that The inverse projection matrix described in step 4.2 is as follows: M reproj =K·[R t→s |T t→s ]·K -1 Among them, M reproj Represents the inverse projection matrix, R t→s represents the inverse rotation matrix, T t→s Represents the inverse translation matrix.

8. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 5, characterized in that: The steps for projecting the feature map of the source frame to the target frame for feature alignment described in step 4.3 are as follows: Among them, u s→t ,v s→t Represents the aligned feature plane F s The pixel coordinates, u s ,v s Denotes the characteristic plane F s The pixel coordinates of , d represents the depth of the camera.

Images