Unsupervised vehicle-mounted monocular depth estimation method based on confidence mask

Abstract

The invention discloses an unsupervised vehicle-mounted monocular depth estimation method based on a confidence mask. The unsupervised vehicle-mounted monocular depth estimation method comprises the following implementation steps: constructing an unsupervised vehicle-mounted monocular depth estimation network; the feature extraction sub-network outputs feature maps of the source frame and the target frame; the attitude estimation sub-network outputs a projection matrix from a source frame to a target frame; the dynamic target sensing sub-network outputs a confidence mask; and the depth decoding sub-network recovers the depth map of the target frame. Modeling is carried out through the confidence mask output by the dynamic target perception sub-network, errors caused by data missing or noise introduction can be effectively reduced, especially under low-quality input, the robustness of depth reconstruction of the moving target in a dynamic scene is improved, and the robustness of depth reconstruction of the moving target in the dynamic scene is improved. According to the method, finer depth information and a better visual effect in a dynamic area can be obtained, and the method can be efficiently applied to an unsupervised vehicle-mounted depth perception scene of automatic driving.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and further relates to an unsupervised monocular image depth estimation method based on confidence mask in the field of monocular image depth estimation technology. The present invention can be used to perceive the depth information of the surrounding environment through a monocular camera in a driving scene. Background Art

[0002] In the field of monocular depth estimation, the goal is to estimate the depth information of a scene from a single image. Among them, the monocular depth estimation method based on supervised learning assumes that there is a nonlinear mapping relationship between the RGB image and the corresponding depth image. This mapping relationship is learned from a large amount of data, and the extracted depth features are used to reconstruct the depth information of the scene. Therefore, the monocular depth estimation method can be used for environmental depth perception in driving scenes. Among them, the supervised learning method relies on a large amount of high-quality data, and obtaining high-quality depth data requires a lot of manpower and material costs. The existing depth data is very limited and lacks diversity, resulting in poor reconstruction of scene depth information for current monocular depth estimation, poor generalization performance in different scenes, and difficulty in practical application.

[0003] Jilin University disclosed a monocular depth estimation method in its patent application "An Unsupervised Monocular Depth Estimation Method Based on Uncertainty Analysis" (patent application number: 202111185472.5; application publication number: CN114549297A). This method proposes an unsupervised depth estimation network based on uncertainty, and proposes a custom likelihood function to represent uncertainty based on the observed depth, the depth of the model output, and the variance of the noise, and quantify the uncertainty of the output result. Finally, it is proposed to use the Retinex illumination theory to deal with the illumination changes in the scene, so as to construct a brightness loss function, and reconstruct the depth information of the scene based on the brightness loss, smoothness loss and uncertainty loss in unsupervised learning. The shortcoming of this method is that although the Retinex illumination theory is used to solve the interference of illumination changes on depth estimation, in complex dynamic environments, illumination changes will still affect the accuracy of depth estimation, resulting in a decrease in the reconstruction effect of depth information of dynamic scenes.

[0004] Zhou et al. proposed an unsupervised monocular depth estimation method based on generative adversarial network in their paper "Unsupervised monocular depth estimation using adversarial networks" (Transactions on Image Processing. IEEE, 2022: 4305-4319). The method introduces an adversarial training mechanism between a generator and multiple discriminators. Specifically, the task of the generator is to generate a depth map from a monocular image, and multiple discriminators are used to evaluate the authenticity of the generated depth map, so as to promote the generator to learn depth information. The core innovation of this method is to optimize the generator and discriminator through adversarial loss, thereby improving the accuracy and robustness of depth estimation. Although the prediction effect of the depth map of this method is significant, the disadvantage of this method is that since this method relies on the texture information of the image to infer the depth, for low-texture areas, it may be difficult for the generator to accurately infer the depth, resulting in large fluctuations in the error of the depth map estimation in low-texture areas and decreased accuracy. Summary of the invention

[0005] The purpose of the present invention is to address the shortcomings of the above-mentioned prior art and propose an unsupervised vehicle-mounted monocular depth estimation method based on confidence mask, aiming to solve the problems of poor robustness of existing monocular depth estimation technology and difficulty in reconstructing depth information of moving targets in dynamic scenes.

[0006] The technical idea for achieving the purpose of the present invention is that the present invention can generate a confidence mask of a dynamic target by constructing a dynamic target perception subnetwork, and the mask can clearly mark the confidence of the area that needs attention. In the network training stage, based on the confidence mask, a lower weight is applied to the occluded area where the depth estimation is unreliable, thereby avoiding the erroneous influence of these areas, and a higher weight is given to the area with high confidence, so as to improve the depth estimation accuracy of the area. This mechanism helps the network model to automatically ignore unreliable depth predictions when facing complex scenes (such as dynamic objects, low-texture areas or high-reflection areas, etc.), and concentrate on optimizing reliable areas, thereby reducing errors and improving the accuracy of depth estimation, thereby solving the problem that dynamic scene moving targets in the prior art are difficult to reconstruct. The present invention forms a closed-loop system through the alternating mechanism of the feature-level optimization of the forward reasoning and the image-level optimization of the back propagation. Each round of optimization can use the previous results to adjust the model parameters and gradually reduce the errors in the depth map prediction. This mechanism continuously refines the prediction so that the model can obtain a more accurate depth map in each round of iteration. Compared with the existing methods, the closed-loop mechanism of the present invention can make full use of the previous optimization results, avoiding the local optimal problem that may be caused by a single optimization, thereby improving the accuracy and robustness of depth estimation.

[0007] According to the above ideas, the technical solution of the present invention includes the following steps:

[0008] Step 1: Construct a dynamic target perception subnetwork. The convolution kernel size of the first layer of three-dimensional convolution is 3×3×3, the step size is 1, and the padding is 1; the convolution kernel size of the second layer of three-dimensional convolution is 5×5×5, the step size is 2, and the padding is 2. The third layer is a batch normalization layer, and the fourth layer is a sigmoid activation layer.

[0009] Step 2: construct an unsupervised vehicle-mounted monocular depth estimation network consisting of a feature extraction subnetwork, a posture estimation subnetwork, a dynamic target perception subnetwork, and a depth decoding subnetwork;

[0010] Step 3, from the unsupervised vehicle-mounted monocular depth estimation dataset, read two adjacent frames of the video sequence one by one as source frames and target frames; all source frames and target frames form a training set;

[0011] Step 4: Input the training set into the unsupervised vehicle-mounted monocular depth estimation network, and use the small batch gradient descent method to iteratively update the parameters of the unsupervised vehicle-mounted monocular depth estimation network until the image reconstruction loss function converges, thereby obtaining a trained unsupervised vehicle-mounted monocular depth estimation network; the training process is as follows:

[0012] Step 4.1, output the feature map corresponding to the source frame and the target frame through the feature extraction sub-network;

[0013] Step 4.2, the posture estimation subnetwork outputs the forward projection matrix from the source frame to the target frame, and the reverse projection matrix from the target frame to the source frame;

[0014] Step 4.3, using the forward projection matrix, project the feature map of the source frame to the target frame for feature alignment;

[0015] Step 4.4, input the aligned features into the dynamic target perception subnetwork to obtain the confidence mask of the dynamic target;

[0016] Step 4.5, using the confidence mask and the feature map corresponding to the target frame, the depth decoding sub-network restores the depth map of the target frame;

[0017] Step 4.6, using the inverse projection matrix, project the depth map of the target frame to the source frame, and calculate the image reconstruction loss between the source frame and the reconstructed source frame;

[0018] In step 5, two temporally adjacent frames of image data generated during the unsupervised driving process are input into a trained unsupervised vehicle-mounted monocular depth estimation network, and a depth estimation result of the scene is output.

[0019] Furthermore, the feature extraction subnetwork adopts the network structure design and parameter setting of the ResNet34 backbone network.

[0020] Furthermore, the posture estimation subnetwork adopts the network structure design and parameter setting of the posture estimation network in SfmLearner.

[0021] Furthermore, the unsupervised vehicle-mounted monocular depth estimation dataset is a sequence of RGB images of continuous frames; each frame of the image is accompanied by time sequence information between adjacent frames; the dataset does not require a real depth label.

[0022] Furthermore, the image reconstruction loss is obtained by the following formula:

[0023]

[0024] Among them, L re represents the image reconstruction loss, L pe represents the minimum photometric reprojection loss, L dis represents the parallax balance loss, N represents the total number of pixels in the image, i and j represent the coordinates of the pixels in the image, SSIM represents the structural similarity index, α is set to 0.85, and I s and represent the source frame and the reconstructed source frame respectively, and Respectively represent the pixel with source frame coordinates (i, j) and the pixel with reconstructed source frame coordinates (i, j), and They represent the partial derivatives of the depth map with respect to x and y respectively.

[0025] Furthermore, the forward projection matrix is ​​as follows:

[0026] M proj =K·[R s→t |T s→t ]·K -1

[0027]

[0028] Among them, M proj represents the forward projection matrix, K represents the camera intrinsic matrix, R s→t represents the forward rotation matrix, T s→t represents the forward translation matrix, the superscript -1 represents the inverse operation, t x ,t y ,t z represents the displacement along the x, y, and z axes of the three-dimensional coordinate system, θ x ,θ y ,θ z Represents the rotation angle around the x, y, and z axes of the three-dimensional coordinate system.

[0029] Furthermore, the inverse projection matrix is ​​as follows:

[0030] M reproj =K·[R t→s |T t→s ]·K -1

[0031]

[0032] Among them, M reproj represents the inverse projection matrix, R t→s represents the inverse rotation matrix, T t→s Represents the inverse translation matrix.

[0033] Furthermore, the feature map of the source frame is projected onto the target frame for feature alignment by the following formula:

[0034]

[0035] Among them, u s→t ,v s→t Represents the aligned feature plane F s The pixel coordinates, u s ,v s Denotes the characteristic plane F s The pixel coordinates of , d represents the depth of the camera.

[0036] Furthermore, the step of outputting the confidence mask of the dynamic target corresponding to the alignment feature through the dynamic target perception subnetwork is as follows:

[0037] The first step is to solve the L1 norm of the aligned features to obtain the residual feature map;

[0038] In the second step, the feature maps of different depth layers in the residual feature map are aggregated through two 3D convolutional layers and a batch normalization layer of the dynamic object perception subnetwork;

[0039] In the third step, the different channel features of the aggregated features are accumulated and then activated by the nonlinear sigmoid of the dynamic target perception subnetwork to obtain the confidence mask of the dynamic target.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] First, the dynamic target perception subnetwork constructed by the present invention generates a confidence mask for dynamic targets, which is used to indicate the degree of credibility of the depth estimation in each area. In the network training stage, based on the confidence mask, a lower weight is applied to the occluded area where the depth estimation is unreliable, which overcomes the defect of poor robustness of the prior art monocular depth estimation technology and avoids the instability caused by target motion or occlusion, so that the present invention can effectively reduce the errors introduced by data loss or noise, especially in low-quality input (such as low-resolution or blurred images). The robustness of the output can still be maintained. For dynamic targets in complex scenes, the confidence mask can control the depth estimation at the boundary, which improves the accuracy of unsupervised vehicle-mounted monocular depth estimation.

[0042] Second, the present invention constructs a loop consistency optimization based on negative feedback, obtains the depth map output by the feature-level optimization of one forward propagation, and then reversely reprojects the depth map to the input to calculate the loss for a round of back propagation. This iteration improves the network's depth reconstruction of dynamic targets, overcomes the problem that the prior art is difficult to reconstruct depth information of moving targets in dynamic scenes, and enables the present invention to obtain more refined depth information and better visual effects in dynamic areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the present invention;

[0044] Figure 2 is a schematic diagram of the unsupervised vehicle-mounted monocular depth estimation network of the present invention;

[0045] Figure 3 It is a comparison chart of the results of the simulation experiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0047] Reference Figure 1 , further describing the implementation steps of the embodiment of the present invention.

[0048] Step 1. Read two adjacent frames of the video sequence as a source frame and a target frame.

[0049] The training data and the test data are read from a monocular image depth estimation dataset. The monocular image depth estimation dataset used in the embodiment of the present invention is a KITTI dataset.

[0050] Read two adjacent source frames I in the KITTI data set in time series s and target frame I t As image input, used to build unsupervised learning.

[0051] Reference Figure 2 , the working process of the unsupervised vehicle-mounted monocular depth estimation network constructed by the present invention is further described.

[0052] The unsupervised vehicle-mounted monocular depth estimation network of the present invention is composed of a feature extraction subnetwork, a posture estimation subnetwork, a dynamic target perception subnetwork, and a depth decoding subnetwork.

[0053] Step 2. The feature extraction subnetwork extracts the deep features of the source frame and the target frame respectively.

[0054] The feature extraction sub-network adopts the network structure design and parameter setting of ResNet34, the backbone network in the prior art deep feature extractor. The backbone network ResNet34 includes a four-stage residual connection module BasicBlock, and its network structure and extracted image features are shown in Table 1.

[0055] Table 1. List of feature extraction network structures and extracted image features

[0056]

[0057] The h and w in Table 1 are the height and width of the input image respectively.

[0058] Extract source frame I using ResNet34 s and target frame I t The characteristics of F s and F t , which is used for subsequent feature-level optimization based on the forward projection matrix to construct the cost volume of feature alignment under different depth assumptions to model the confidence of dynamic targets and reconstruct the depth map of the target frame.

[0059] Step 3. The pose estimation subnetwork obtains the forward and reverse projection matrix from the source frame to the target frame.

[0060] The pose estimation subnetwork uses the input source frame and target frame to estimate the forward translation matrix T of the mobile camera from the source frame to the target frame. s→t , forward rotation matrix R s→t， And the inverse translation matrix T from the target frame to the source frame t→s and the inverse rotation matrix R t→s , positive T s→t and positive R s→t As shown below,

[0061]

[0062] Among them, θ x ,θ y ,θ z Indicates the rotation angle around the x, y, and z axes, t x ,t y ,t z Represents the displacement along the x, y, and z axes. The conversion between the forward rotation matrix and the inverse rotation matrix, and the forward translation matrix and the inverse translation matrix can be performed according to the following formula:

[0063]

[0064] Among them, T t→s represents the inverse translation matrix, R t→s Represents the inverse rotation matrix.

[0065] Step 4. Feature-level optimization of forward reasoning based on the forward projection matrix.

[0066] Assume that the camera intrinsic parameter matrix is ​​K, the camera focal length is f, and the world coordinate is P w (x w ,y w ,z w ), the camera coordinate is P c (x c ,y c ,z c ), the pixel coordinate is P pix (u,v), dx, dy are the actual lengths of pixels, and the source frame camera coordinate system is used as the reference coordinate system.

[0067] The process of feature-level optimization is as follows:

[0068] Step 4.1, P w Convert to P c , P c Go to P pix , the camera intrinsic parameter matrix K can be expressed as follows:

[0069]

[0070] Step 4.2, let the feature plane F s The pixel coordinates are (u s ,v s ), with a depth of d, let the feature plane The pixel coordinates are (u s→t ,v s→t ), and F s Forward reprojection to The feature alignment of the plane is as follows:

[0071]

[0072] Step 4.3, so far, the cost volume C of inter-frame feature alignment is constructed based on different camera depths d (i.e., parallax) d , is a multidimensional tensor used to represent the matching cost or similarity between deep features under different depth assumptions, as shown below:

[0073]

[0074] Step 4.4, from the cost body C d The deep features of different depth layers are fused through three-dimensional convolution, and the target frame I is output through the dynamic target perception module DOM and the deep decoding network Decoder. t The normalized target depth map D t , d max is the maximum depth value, as follows:

[0075] D t =d max ·Sigmoid(Decoder(DOM(Conv 3D (C d ))))

[0076] Among them, the Decoder of the deep decoding network is used to decode the output of the dynamic target perception module DOM, upsample layer by layer to restore the depth details. The network structure is shown in Table 2.

[0077] Table 2. List of deep decoding network structures

[0078]

[0079] In Table 2, h and w are the height and width of the input image respectively.

[0080] In the process of aligning deep features, the feature differences of dynamic targets are constructed under different depth assumptions. dThe DOM is enhanced in the image, and the confidence mask of the dynamic target is obtained through a Sigmoid activation layer, which indicates the confidence distribution of the dynamic target area. The deep decoder network upsamples the deep features including the confidence mask and restores the depth details. The last Sigmoid layer outputs a normalized depth map, which is compared with the maximum depth value d max The product of gets the target frame I t The corresponding target depth map D t , this process is a feature-level optimization of forward reasoning based on forward projection.

[0081] Step 5. Image-level optimization based on back-propagation of the inverse projection matrix.

[0082] A round of feature-level optimization for forward reasoning requires a round of image-level optimization for back-propagation to iteratively improve the output D t The accuracy of the target depth map D is obtained by t and target frame I t Combined with the translation matrix T s→t and the rotation matrix R s→t Inverse reprojection to source frame to construct image-level aligned reconstructed source frame And establish I s and The image reconstruction loss is used for self-supervised training.

[0083] The process of image-level optimization is as follows:

[0084] Step 5.1, let the target frame I t The pixel coordinates are (u t ,v t ), the corresponding depth is D t (u t ,v t ), reconstruct the source frame The pixel coordinates are (u t→s ,v t→s ), the target frame I is transformed into t Reverse reprojection to reconstruct source frame The images are aligned as follows:

[0085]

[0086] Among them, u s→t ,v s→t Represents the aligned feature plane F s The pixel coordinates, u s ,v s Denotes the characteristic plane F s The pixel coordinates of , d represents the depth of the camera.

[0087] Step 5.2, build I s and The image reconstruction loss L re , by the minimum photometric reprojection loss L pe and parallax balance loss L dis Composition, that is

[0088] Moving objects or occluded areas in dynamic scenes will cause the reprojection error to increase because the pixels in these areas do not correspond in different views. By introducing the minimum reprojection loss L pe , dynamically mask the invalid pixel areas caused by occlusion or dynamic targets to improve the robustness of the loss function, as shown below:

[0089]

[0090] Among them, L re represents the image reconstruction loss, N represents the total number of pixels in the image, i and j represent the coordinates of the pixels in the image, SSIM represents the structural similarity index, α is 0.85, I s and represent the source frame and the reconstructed source frame respectively.

[0091] Minimum reprojection loss L pe By introducing L 1 The combination of a single-scale SSIM term as the reprojection loss enables self-supervised training based on visual geometry constraints without the need for deep labels.

[0092] Parallax smoothing loss L dis By penalizing unreasonable and drastic changes in the depth map, the predicted depth map is made more continuous in space. This smoothness helps reduce noise, especially in areas where the reprojection loss cannot be fully constrained. It can reduce the overfitting of the model to a specific data distribution and make the predicted depth performance more stable, as shown below:

[0093]

[0094] Among them, L dis Represents the parallax balance loss.

[0095] Step 5.3: The entire training process consists of a round of forward reasoning feature-level optimization and a round of back-propagation image-level optimization, which constitutes a negative feedback loop consistency mechanism to iteratively improve the target depth map D in dynamic scenes. t accuracy.

[0096] The effects of the present invention are further described below in conjunction with experiments.

[0097] 1. Simulation experimental conditions.

[0098] The computer processor used in the simulation experiment of the present invention is an Intel (R) Xeon (R) Gold 6148 CPU @ 2.40 GHz, with a running memory of 128G, and a graphics card is an NVIDIA GeForce RTX 3090 GPU with a video memory of 24GB. The operating system is 64-bit Ubuntu 18.04 (LTS), and the deep learning framework is PyTorch version 1.10.1. The network training uses the Adam optimizer to update the network parameters, where the parameters of the Adam optimizer are β1 = 0.9, β2 = 0.999, and the weight decay term is 0.01. 100 epochs are set, and the initial learning rate is 1e-5, which is linearly reduced to 1e-6 during the training iteration. The image reconstruction loss L is used. re To train the network.

[0099] The dataset used in the simulation experiment of the present invention is the KITTI dataset, which is one of the widely used benchmark datasets in the field of autonomous driving and computer vision, especially in the monocular depth estimation task. The dataset contains images and corresponding sensor data captured from real-world scenes, including depth information from the laser radar (LIDAR). For the monocular depth estimation task, the KITTI dataset provides monocular image sequences and real depth maps (obtained by LIDAR) to train and evaluate depth estimation algorithms.

[0100] 2. Experimental content and results analysis:

[0101] The simulation content of the present invention is to use the method of the present invention and four existing technologies (MonoFormer method, PlaneDepth method, DS-Depth method, DCPI-Depth method) under the above experimental conditions to perform monocular depth estimation on the KITTI test data set to obtain the predicted depth map, such as Figure 3 shown.

[0102] Four methods of the prior art used in the simulation experiment of the present invention are as follows:

[0103] The MonoFormer method of the prior art refers to a method combining multi-task learning, adaptive loss and data enhancement in "Deep Digging into the Generalization of Self-Supervised Monocular Depth Estimation" (In Proceedings of the AAAI Conference on Artificial Intelligence, 2023, pp. 187-196.) proposed by Bae et al. to improve the performance of self-supervised depth estimation models on different data sets.

[0104] The PlaneDepth method of the prior art refers to a self-supervised depth estimation method based on orthogonal plane modeling proposed by Wang et al. in their published paper “PlaneDepth: Self-Supervised Depth Estimation Via Orthogonal Planes” (In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2023, pp. 21425-21434.), which more effectively captures and utilizes the geometric structure of the scene by introducing orthogonal plane constraints.

[0105] The DS-Depth method of the prior art refers to the fusion cost volume of dynamic and static depth estimation proposed by Miao et al. in their published paper “DS-Depth: Dynamic and Static Depth Estimation via a Fusion Cost Volume” (In Proceedings of the IEEE Transactions On Circuits and Systems For Video Technology, 2024, pp. 2564-2576). The fusion cost volume of dynamic and static depth estimation is proposed to improve the accuracy in dynamic scenes by collaboratively optimizing the depth estimation of dynamic and static objects.

[0106] The DCPI-Depth method of the prior art refers to the dense correspondence prior introduced by Zhang et al. in their publication “DCPI-Depth: Explicitly Infusing Dense Correspondence Prior to Unsupervised MonocularDepth Estimation” (In Proceedings of the Computing Research Repository, 2024, pp. abs / 2405.16960) to explicitly capture the depth information in the image and better understand the global and local structure of the image.

[0107] Combine the following Figure 3 The simulation diagram of the present invention is further described.

[0108] Figure 3 In Figure 3 (a) is the input image of the simulation experiment of the present invention, Figure 3 (b) is the predicted depth map of the existing method MonoFormer. Figure 3 (c) is the predicted depth map of the existing method PlaneDepth. Figure 3 (d) is the predicted depth map of the existing method DS-Depth. Figure 3 (e) is the predicted depth map of the existing method DCPI-Depth. Figure 3 (f) is the predicted depth map of the method of the present invention.

[0109] In order to verify the error of the simulation experiment of the present invention, the error objective index evaluation of the present invention and four existing technologies in the KITTI test set samples is evaluated. The evaluation indicators are: absolute relative error AbsRel, mean square relative error SqRel, root mean square error RMSE, root mean square logarithmic error RMSElog average logarithmic error log10. The lower the four evaluation indicators, the lower the error of depth estimation and the better the effect.

[0110] The evaluation results of objective indicators of error in quality evaluation are shown in Table 3.

[0111] Table 3. Error objective index evaluation table of the present invention and four traditional methods on the KITTI test data set

[0112] Method of the present invention AbsRel↓ SqRel↓ RMSE↓ RMSElog↓ MonoFormer 0.108 0.806 4.594 0.184 PlaneDepth 0.084 0.549 3.981 0.169 DS-Depth 0.095 0.698 4.329 0.173 DCPI-Depth 0.090 0.655 4.113 0.167 Ours 0.090 0.518 3.588 0.163

[0113] It can be seen from Table 3 that the error term Sq Rel and RMSE of the present invention on the KITTI test set are significantly better than other existing comparison methods. Figure 3(f) It can be seen that the subjective result also has finer texture details, which further proves that the method of the present invention can better present the depth information in the image and significantly improve the overall perceptual quality of the image.

[0114] In order to verify the effect of the simulation experiment of the present invention, the threshold value is δ<1.25. i , i=1, 2, 3 are used to evaluate the model, and the quality of the depth estimation results of the present invention and the four traditional methods is evaluated. The higher the three evaluation indicators are, the higher the accuracy of the depth estimation is. The results are shown in Table 4.

[0115] Table 4. Objective index evaluation table of the accuracy of the present invention and the comparative method on the KITTI test data set

[0116] Method of the present invention δ1(%)↑ δ2(%)↑ δ3(%)↑ MonoFormer 0.884 0.963 0.983 PlaneDepth 0.911 0.968 0.984 DS-Depth 0.905 0.966 0.984 DCPI-Depth 0.914 0.969 0.985 Ours 0.916 0.971 0.986

[0117] It can be seen from Table 4 that the accuracy term δ of the present invention on the KITTI test set is 1 ,δ 2 ,δ 3 It is slightly ahead of the best comparison method and significantly outperforms other existing methods in terms of accuracy in dynamic target areas, achieving the best visual effect, which shows that the present invention is more robust and practical and achieves better depth estimation effect.

Claims

1. An unsupervised vehicle-mounted monocular depth estimation method based on confidence mask, characterized in that: A dynamic target perception subnetwork is constructed to generate a confidence mask of the dynamic target; the confidence mask is used to restore the depth map of the dynamic scene during forward reasoning, and the depth map is used to calculate the image reconstruction loss during back propagation; the steps of the estimation method include the following: Step 1: Construct a dynamic target perception subnetwork. The convolution kernel size of the first layer of three-dimensional convolution is 3×3×3, the step size is 1, and the padding is 1; the convolution kernel size of the second layer of three-dimensional convolution is 5×5×5, the step size is 2, and the padding is 2. The third layer is a batch normalization layer, and the fourth layer is a sigmoid activation layer. Step 2: construct an unsupervised vehicle-mounted monocular depth estimation network consisting of a feature extraction subnetwork, a posture estimation subnetwork, a dynamic target perception subnetwork, and a depth decoding subnetwork; Step 3, from the unsupervised vehicle-mounted monocular depth estimation dataset, read two adjacent frames of the video sequence one by one as the source frame and the target frame; All source frames and target frames form a training set; Step 4: Input the training set into the unsupervised vehicle-mounted monocular depth estimation network, and use the small batch gradient descent method to iteratively update the parameters of the unsupervised vehicle-mounted monocular depth estimation network until the image reconstruction loss function converges, thereby obtaining a trained unsupervised vehicle-mounted monocular depth estimation network; the training process is as follows: Step 4.1, the feature extraction subnetwork outputs the feature maps corresponding to the source frame and the target frame; Step 4.2, the pose estimation subnetwork outputs the forward projection matrix from the source frame to the target frame, and the reverse projection matrix from the target frame to the source frame; Step 4.3, using the forward projection matrix, project the feature map of the source frame to the target frame for feature alignment; Step 4.4, the aligned features are fed into the dynamic target perception subnetwork to obtain the confidence mask of the dynamic target; Step 4.5, using the confidence mask and the feature map corresponding to the target frame, the depth decoding sub-network restores the depth map of the target frame; Step 4.6, using the inverse projection matrix, project the depth map of the target frame to the source frame, and calculate the image reconstruction loss between the source frame and the reconstructed source frame; In step 5, two temporally adjacent frames of image data generated during the unsupervised driving process are input into a trained unsupervised vehicle-mounted monocular depth estimation network, and a depth estimation result of the scene is output.

2. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that: The feature extraction subnetwork described in step 2 adopts the ResNet34 network structure design and parameter setting of the backbone network in the deep feature extractor.

3. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that: The posture estimation subnetwork described in step 2 adopts the network structure design and parameter setting of the posture estimation network in SfmLearner.

4. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that: The unsupervised vehicle-mounted monocular depth estimation dataset described in step 3 is a sequence of RGB images of continuous frames; each frame of the image is accompanied by the time sequence information between adjacent frames, and the dataset does not require a real depth label.

5. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that: The image reconstruction loss function in step 4 is obtained by the following formula: Among them, L re represents the image reconstruction loss, L pe represents the minimum photometric reprojection loss, L dis represents the parallax balance loss, N represents the total number of pixels in the image, i and j represent the coordinates of the pixels in the image, SSIM represents the structural similarity index, α is taken as 0.85, I s , represent the source frame and the reconstructed source frame respectively, They represent the pixel with source frame coordinates (i, j) and the pixel with reconstructed source frame coordinates corresponding to source frame coordinates (i, j). They represent the partial derivatives of the depth map with respect to x and y respectively.

6. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that: The forward projection matrix described in step 4.2 is as follows: M proj =K·[R s→t |T s→t ]·K -1 Among them, M proj represents the forward projection matrix, K represents the camera intrinsic matrix, R s→t represents the forward rotation matrix, T s→t represents the forward translation matrix, the superscript -1 represents the inverse operation, t x ,t y ,t z represents the displacement along the x, y, and z axes of the three-dimensional coordinate system, θ x ,θ y ,θ z Represents the rotation angle around the x, y, and z axes of the three-dimensional coordinate system.

7. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 5, characterized in that: The inverse projection matrix described in step 4.2 is as follows: M reproj =K·[R t→s |T t→s ]·K -1 Among them, M reproj represents the inverse projection matrix, R t→s represents the inverse rotation matrix, T t→s Represents the inverse translation matrix.

8. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 5, characterized in that: The steps for projecting the feature map of the source frame to the target frame for feature alignment described in step 4.3 are as follows: Among them, u s→t ,v s→t Represents the aligned feature plane F s The pixel coordinates, u s ,v s Denotes the characteristic plane F s The pixel coordinates of , d represents the depth of the camera.

9. The unsupervised vehicle-mounted monocular depth estimation method based on confidence mask according to claim 1, characterized in that: The steps of outputting the confidence mask of the dynamic target corresponding to the aligned features through the dynamic target perception subnetwork described in step 4.4 are as follows: The first step is to solve the L1 norm of the aligned features to obtain the residual feature map; In the second step, two 3D convolutional layers and a batch normalization layer of the dynamic object perception subnetwork are used to aggregate the feature maps of different depth layers in the residual feature map to obtain the aggregated features. In the third step, the sigmoid activation function of the dynamic target perception sub-network is used to activate the accumulated features of different channels of the aggregated features and obtain the confidence mask of the dynamic target.

Images