Self-supervised monocular depth estimation method based on uncertainty and edge enhancement

Abstract

The invention relates to a self-supervised monocular depth estimation method based on uncertainty and edge enhancement, and aims to solve the problems that self-supervised monocular depth estimation is susceptible to noise interference and the edge is fuzzy. According to the technical scheme, a self-supervised monocular depth estimation system composed of a data acquisition module, a data preprocessing module, a depth estimation teacher module, a depth estimation student module and a depth estimation refining module is constructed. The depth estimation teacher module is trained in a self-supervised monocular depth estimation mode, and the depth estimation student module is trained in an uncertainty perception self-distillation mode. And training a depth estimation refining module by adopting a self-supervised monocular depth estimation mode, an uncertainty perception self-distillation mode and an edge prediction multi-task joint training mode. And the trained self-supervised monocular depth estimation system carries out depth estimation on the video data to obtain a depth estimation result. Noise in self-supervised training is suppressed, the edge of the depth image is enhanced, and the precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a self-supervised monocular depth estimation method based on uncertainty and edge enhancement. Background Art

[0002] In the field of computer vision, depth estimation is a crucial task, which aims to recover the three-dimensional structure information of the scene from the two-dimensional image. Traditional depth estimation methods usually rely on special equipment such as binocular cameras or RGBD cameras to obtain depth information. Although these methods can provide relatively accurate depth maps, they are expensive, complex to install and calibrate, and are greatly limited in practical applications.

[0003] In recent years, with the rapid development of deep learning technology, self-supervised monocular depth estimation methods have gradually become a research hotspot. This method relies only on image sequences taken by a single camera, infers the depth information of the scene by mining visual clues and geometric constraints in the image, and does not require external supervisory signals or other types of sensors. Therefore, it has lower costs, higher flexibility and wider application prospects.

[0004] In the self-supervised monocular depth estimation method, researchers guide the training of the model by constructing various self-supervised loss functions. Among them, photometric error is a commonly used self-supervised signal, which uses the pixel correspondence between adjacent frames to estimate the depth and optimizes the model parameters by minimizing the photometric error. In addition, in order to further improve the accuracy and robustness of depth estimation, researchers have also proposed a variety of improvement strategies, such as auto-masking technology to ignore unreliable pixels, multi-scale loss functions to capture depth information of different scales, and smoothness constraints to encourage depth maps to remain continuous in space.

[0005] The "Unsupervised Learning of Depth and Ego-Motion from Video" proposed by ZHOU et al. (see ZHOU T, BROWN MA, SNAVELY N, et al. Unsupervised Learning of Depth and Ego-Motion from Video [J]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 6612-9) is a landmark research work. This method uses the pixel correspondence between consecutive video frames to simultaneously estimate the depth map and the camera's motion (i.e., self-motion), and achieves high-quality monocular depth estimation by constructing self-supervisory signals such as photometric loss and smoothness constraints. This method not only improves the accuracy of depth estimation, but also demonstrates the great potential of self-supervised learning in the field of depth estimation. In addition to the above work, there are some other self-supervised monocular depth estimation research worthy of attention. For example, "GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose" (see YIN Z, SHI J. GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose [J]. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2018: 1983-92.) This paper proposes an unsupervised deep learning framework for jointly learning monocular depth estimation, optical flow estimation, and camera pose estimation. The framework utilizes the geometric consistency of video sequences and optimizes network training through an adaptive geometric consistency loss."Digging Into Self-Supervised Monocular Depth Estimation, Research on Self-Supervised Monocular Depth Estimation" (see the literature GODARD C, AODHAO M, BROSTOW G J. Digging Into Self-Supervised Monocular Depth Estimation [J]. 2019 IEEE / CVF International Conference on Computer Vision (ICCV), 2018: 3827-37.) This paper deeply explores the key technologies and challenges in self-supervised monocular depth estimation, and proposes a series of improvement strategies, including minimal reprojection loss for handling occlusion, full-resolution multi-scale sampling methods to reduce visual artifacts, and automatic masking loss to ignore training pixels that violate the camera motion assumption. "R-MSFM: Recurrent Multi-Scale Feature Modulation for Monocular Depth Estimating, Recursive Multi-Scale Feature Modulation for Monocular Depth Estimating" (see the document ZHOU Z, FANX, SHIP, et al. R-MSFM: Recurrent Multi-Scale Feature Modulation for Monocular Depth Estimating [J]. 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 2021: 12757-66.) This paper introduces a new deep network architecture for monocular depth estimation. This architecture extracts per-pixel features, constructs a multi-scale feature modulation module, and iteratively updates the inverse depth using a fixed-resolution parameter-sharing decoder, achieving a richer representation while maintaining more accurate space.In addition, "Spatial-Aware Dynamic Lightweight Self-Supervised Monocular Depth Estimation" (see Song, Linna, Dian-xi Shi, Jianqiang Xia, Qianying Ouyang, Ziteng Qiao, Songchang Jin and Shaowu Yang. "Spatial-Aware Dynamic Lightweight Self-Supervised Monocular Depth Estimation." IEEE Robotics and Automation Letters 9 (2024): 883-890.) This paper focuses on achieving efficient and lightweight monocular depth estimation under the framework of self-supervised learning. This method effectively reduces the computational complexity and memory usage of the model by introducing a spatially aware dynamic network structure and an adaptive weight allocation strategy, while maintaining a high depth estimation accuracy.

[0006] However, despite the significant progress made in self-supervised monocular depth estimation methods, there are still some problems that need to be solved. Among them, noise interference and edge blur are the two main challenges. In real scenes, due to factors such as lighting changes, camera shake, object occlusion, etc., the acquired images often contain a lot of noise, which will seriously affect the accuracy of depth estimation. At the same time, since the edge areas of the image usually contain less texture and geometric information, the depth estimation accuracy in these areas is low and blur is prone to occur.

[0007] Therefore, how to solve the problem that self-supervised monocular depth estimation is susceptible to noise interference and edge blur has always been a technical issue of great concern to those skilled in the art. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a self-supervised monocular depth estimation method based on uncertainty and edge enhancement, which not only suppresses the noise in self-supervised training, but also enhances the edge of the depth image, improves the accuracy and reduces the blur.

[0009] The present invention comprises the following steps:

[0010] The first step is to build a self-supervised monocular depth estimation system. The self-supervised monocular depth estimation system consists of a data acquisition module, a data preprocessing module, a depth estimation teacher module, a depth estimation student module, and a depth estimation refinement module.

[0011] The data acquisition module is connected to the monocular camera and the data preprocessing module, acquires two-dimensional video data from the monocular camera, and sends the video data to the data preprocessing module;

[0012] The data preprocessing module is connected to the data acquisition module and the depth estimation teacher module, receives video data from the data acquisition module, preprocesses the video data, and obtains preprocessed image data. During training, the data preprocessing module sends the preprocessed image data to the depth estimation teacher module, the depth estimation student module, and the depth estimation refinement module; when performing depth estimation on the two-dimensional video data input by the user, the data preprocessing module sends the preprocessed image data to the depth estimation refinement module.

[0013] The function of the depth estimation teacher module is to assist in training the depth estimation refinement module. It only works during training and does not participate in the work when performing depth estimation on the two-dimensional video data input by the user. The depth estimation teacher module is connected to the data preprocessing module and the depth estimation student module, and is composed of a first depth estimation network, a first pose estimation network and a first reprojection module. The first depth estimation network is composed of a first depth encoder and a first depth decoder. The first depth estimation network performs depth estimation on the preprocessed image data received from the data preprocessing module, obtains a first depth map, and sends the first depth map to the first reprojection module; the first pose estimation network performs pose estimation on the preprocessed image data received from the data preprocessing module, obtains a first pose transformation matrix, and sends the first pose transformation matrix to the first reprojection module; the first reprojection module optimizes the first depth estimation network and the first pose estimation network to obtain an optimized first depth estimation network and a first pose estimation network; the optimized first depth estimation network performs depth estimation on the preprocessed image data received from the data preprocessing module, obtains an optimized first depth map, and sends the optimized first depth map to the depth estimation student module;

[0014] The function of the depth estimation student module is to assist in training the depth estimation refinement module. It only works during training and does not participate in the work when performing depth estimation on the two-dimensional video data input by the user. The depth estimation student module is connected to the data preprocessing module, the depth estimation teacher module, and the depth estimation refinement module, and is composed of a second depth estimation network and a first uncertainty-aware self-distillation module; the second depth estimation network is composed of a second depth encoder, a second depth decoder, and a first uncertainty decoder; the second depth estimation network uses the optimized first depth map received from the depth estimation teacher module as a pseudo-label, performs depth estimation and uncertainty estimation on the preprocessed image data received from the data preprocessing module, obtains a second depth map and a first uncertainty map, respectively, and sends the second depth map and the first uncertainty map to the first uncertainty-aware self-distillation module; the first uncertainty-aware self-distillation module optimizes the second depth estimation network to obtain an optimized second depth estimation network; the optimized second depth estimation network performs depth estimation and uncertainty estimation on the preprocessed image data received from the data preprocessing module, obtains an optimized second depth map and an optimized first uncertainty map, and sends the optimized second depth map and the optimized first uncertainty map to the depth estimation refinement module;

[0015] The depth estimation refinement module is connected to the data preprocessing module and the depth estimation student module, and is composed of a third depth estimation network, a second pose estimation network, a Canny operator module, a multi-scale reliability mask generation module, a second reprojection module, a second uncertainty-aware self-distillation module, and an edge enhancement module. During training, the multi-scale reliability mask generation module receives the optimized first uncertainty map from the depth estimation student module, binarizes the optimized first uncertainty map, obtains a multi-scale reliability mask, and sends the multi-scale reliability mask to the second reprojection module. The Canny operator module performs Canny edge calculation on the preprocessed image data received from the data preprocessing module (see the literature CANNYJ.A computational approach to edge detection[J].IEEE Transactions on patternanalysis and machine intelligence, 1986, (6): 679–98-–98.), obtain a Canny edge map, and send the Canny edge map to the edge enhancement module; the third depth estimation network is composed of a third depth encoder, a third depth decoder, a second uncertainty decoder and an edge decoder; the third depth estimation network performs depth estimation, uncertainty estimation and edge prediction on the preprocessed image data received from the data preprocessing module, and obtains a third depth map, a second uncertainty map and an edge prediction map respectively, and sends the third depth map to the second uncertainty-aware self-distillation module and the second reprojection module, sends the second uncertainty map to the second uncertainty-aware self-distillation module, and sends the edge prediction map to the edge enhancement module; the second pose The estimation network estimates the preprocessed image data received from the data preprocessing module to obtain the second pose transformation matrix, and sends the second pose transformation matrix to the second reprojection module; the edge enhancement module calculates the edge detection loss function according to the edge prediction map and the Canny edge map to optimize the third depth estimation network; the second reprojection module reprojects the third depth map, the second pose transformation matrix and the multi-scale reliability mask to optimize the third depth estimation network and the second pose estimation network; the second uncertainty-aware self-distillation module uses the optimized second depth map received from the depth estimation student module as a pseudo-label, distills the second uncertainty map and the third depth map to perform self-distillation optimization on the third depth estimation network, and obtains the optimized third depth estimation network. When performing depth estimation on the two-dimensional video data input by the user, the optimized third depth estimation network of the depth estimation refinement module performs depth estimation on the preprocessed image data received from the data preprocessing module to obtain the optimized third depth map, i.e., the depth estimation result.

[0016] The second step is to build an image dataset by:

[0017] 2.1 The data acquisition module acquires two-dimensional video data from the monocular camera. The number of image frames N in the video data is greater than 20,000, and sends the video data to the data preprocessing module;

[0018] 2.2 Initialize variable n = 1;

[0019] 2.3 The data preprocessing module uses a preprocessing method to preprocess the nth video data, the method is:

[0020] 2.3.1 scaling the image in the nth video data to a height of 192 pixels and a width of 640 pixels to obtain the scaled nth image;

[0021] 2.3.2 Normalize the pixels in the scaled nth image to between 0 and 1 to obtain a normalized nth image;

[0022] 2.3.3 Perform data augmentation on the normalized nth image to improve the robustness of training. The data augmentation methods include horizontal flipping, brightness adjustment (±0.2), saturation adjustment (±0.2), contrast adjustment (±0.2) and hue jitter (±0.1) to obtain the preprocessed nth image data.

[0023] 2.4 Let n=n+1; if n<N, go to 2.3; if n≥N, it means that the N video data have been preprocessed to obtain the preprocessed image data set, and go to the third step.

[0024] In the third step, the preprocessed image dataset is used to train the depth estimation teacher module, depth estimation student module and depth estimation refinement module in the self-supervised monocular depth estimation system. The training method of the depth estimation teacher module is the self-supervised monocular depth estimation method, the training method of the depth estimation student module is the uncertainty-aware self-distillation method, and the training method of the depth estimation refinement module is the self-supervised monocular depth estimation, uncertainty-aware self-distillation method and edge prediction multi-task joint training method.

[0025] 3.1 The depth estimation teacher module is trained using a self-supervised monocular depth estimation method, as follows:

[0026] 3.1.1 Set the total number of training rounds E, where E is a positive integer, preferably 30.

[0027] 3.1.2 Initialize the number of iterations e = 1;

[0028] 3.1.2.1 Initialize the number of trained samples n = 1;

[0029] 3.1.2.2 The data preprocessing module sends the nth image data (the nth original image in this case) in the preprocessed image data set to the first depth estimation network and the first pose estimation network;

[0030] 3.1.2.3 The first depth estimation network receives the nth image data in the preprocessed image data set, and performs depth estimation on the nth image data: the first depth encoder encodes the nth image data to obtain the nth first depth feature, and sends the nth first depth feature to a depth decoder; the first depth decoder decodes the nth first depth feature to obtain the nth first depth map, and sends the nth first depth map to the first reprojection module;

[0031] 3.1.2.4 The first pose estimation network receives the nth image data in the preprocessed image data set, performs pose estimation on the nth image data, obtains the nth first pose transformation matrix, and sends the nth first pose transformation matrix to the first reprojection module;

[0032] 3.1.2.5 The first reprojection module reprojects the nth first depth map and the nth first pose transformation matrix to obtain the reconstructed nth synthetic image, calculates the reprojection loss function between the nth synthetic image and the nth original image, and obtains the reprojection loss value L p :

[0033] L p =|I t -I s→t |,

[0034] Among them, I t is the nth original image, I s→t is the reconstructed nth synthetic image, |I t -I s→t | indicates request I t -I s→t The absolute value of

[0035] 3.1.2.6 The first reprojection module uses the Adam optimization algorithm (see the reference "Kingma DP, Ba J. Adam: A method for stochastic optimization [J]. arXiv preprint arXiv: 1412.6980, 2014", Kingma's paper: Adam, a stochastic optimization method) to minimize L p , to update the weight parameters of the first depth estimation network and the first pose estimation network to optimize the first depth estimation network and the first pose estimation network;

[0036] 3.1.2.7 Let n = n + 1. If n < N, go to 3.1.2.2 and use the next sample to continue training. If n ≥ N and e < E, go to 3.1.2.1 for the next round of training. If n ≥ N and e ≥ E, it means that all data have been trained and the number of training rounds has been reached, go to 3.1.3.

[0037] 3.1.3 The optimized first depth estimation network re-reads the preprocessed image data set of the data preprocessing module, performs depth estimation on the preprocessed image data set, obtains N optimized first depth maps (i.e., the 1st optimized first depth map, ..., the nth optimized first depth map, ..., the Nth optimized first depth map), and sends the N optimized first depth maps to the depth estimation student module.

[0038] 3.2 The depth estimation student module is trained using uncertainty-aware self-distillation as follows:

[0039] 3.2.1 Set the total number of iterations to be equal to E.

[0040] 3.2.2 Initialize the number of iterations e = 1;

[0041] 3.2.2.1 Initialize the number of trained samples n = 1;

[0042] 3.2.2.2 The data preprocessing module sends the nth image data (the nth original image in this case) in the preprocessed image data set to the second depth estimation network;

[0043] 3.2.2.3 The second depth estimation network receives the nth image data in the preprocessed data set, and performs depth estimation and uncertainty estimation on the nth image data: the second depth encoder encodes the nth image data to obtain the nth second depth feature, and sends the nth second depth feature to the second depth decoder and the first uncertainty decoder; the second depth decoder receives the nth second depth feature, decodes the nth second depth feature, obtains the nth second depth map, and sends the nth second depth map to the first uncertainty-aware self-distillation module; the first uncertainty decoder receives the nth second depth feature, performs uncertainty decoding on the nth second depth feature, obtains the nth first uncertainty map, and sends the nth first uncertainty map to the first uncertainty-aware self-distillation module;

[0044] 3.2.2.4 The first uncertainty-aware self-distillation module receives the nth optimized first depth map from the depth estimation teacher module;

[0045] 3.2.2.5 The first uncertainty-aware self-distillation module calculates the uncertainty-aware loss function between the nth second depth map, the nth first uncertainty map, and the nth optimized first depth map to obtain the first uncertainty-aware loss value L d :

[0046]

[0047] Among them, D t is the first depth map after the nth optimization, D s is the nth second depth map, ∑ s is the nth first uncertainty map;

[0048] 3.2.2.6 The first uncertainty-aware self-distillation module uses the Adam optimization algorithm to minimize L d , to update the weight parameters of the second depth estimation network to optimize the second depth estimation network;

[0049] 3.2.2.7 Let n=n+1. If n<N, go to 3.2.2.2 and use the next sample to continue training. If n≥N and e<E, go to 3.2.2.1 for the next round of training. If n≥N and e≥E, it means that all data have been trained and the number of training rounds has been reached, go to 3.2.3.

[0050] 3.2.3 The optimized second depth estimation network re-reads the preprocessed image data set from the data preprocessing module, performs depth estimation on the preprocessed image data set, and obtains N optimized second depth maps (i.e., the 1st optimized second depth map, …, the nth optimized second depth map, …, the Nth optimized second depth map) and N optimized first uncertainty maps (i.e., the 1st optimized first uncertainty map, …, the nth optimized first uncertainty map, …, the Nth optimized first uncertainty map), and sends the N optimized second depth maps and the N optimized first uncertainty maps to the depth estimation refinement module.

[0051] 3.3 The depth estimation refinement module is trained using self-supervised monocular depth estimation, uncertainty-aware self-distillation, and edge prediction multi-task joint training, as follows:

[0052] 3.3.1 Set the number of iterations to E.

[0053] 3.3.2 Initialize the number of iterations e = 1;

[0054] 3.3.2.1 Initialize the number of trained samples n = 1;

[0055] 3.3.2.2 The data preprocessing module sends the nth image data (the nth original image in this case) in the preprocessed image data set to the third depth estimation network, the second pose estimation network, and the Canny operator module;

[0056] 3.3.2.3 The multi-scale reliability mask generation module binarizes the nth optimized first uncertainty map received from the depth estimation student module to obtain a multi-scale reliability mask, and sends the multi-scale reliability mask to the second reprojection module;

[0057] 3.3.2.4 The Canny operator module performs Canny edge calculation on the nth image data received from the data preprocessing module to obtain the nth Canny edge map, and sends the nth Canny edge map to the edge enhancement module;

[0058] 3.3.2.5 The third depth estimation network performs depth estimation, uncertainty estimation and edge prediction on the nth image data received from the data preprocessing module: the third depth encoder receives the nth image data, encodes the nth image data, obtains the nth third depth feature, and sends the nth third depth feature to the third depth decoder, the second uncertainty decoder, and the edge decoder; the third depth decoder receives the nth third depth feature, decodes the nth third depth feature, obtains the nth third depth map, and sends the nth third depth map to the second reprojection module; the second uncertainty decoder receives the nth third depth feature, performs uncertainty decoding on the nth third depth feature, obtains the nth second uncertainty map, and sends the nth second uncertainty map to the second uncertainty-aware self-distillation module; the edge decoder receives the nth third depth feature, performs edge prediction on the nth third depth feature, obtains the nth edge prediction map, and sends the nth edge prediction map to the edge enhancement module;

[0059] 3.3.2.6 The second pose estimation network performs pose estimation on the nth image data received from the data preprocessing module to obtain a second pose transformation matrix, and sends the second pose transformation matrix to the second reprojection module;

[0060] 3.3.2.7 The second uncertainty-aware self-distillation module receives the nth optimized second depth map from the depth estimation student module;

[0061] 3.3.2.8 The second reprojection module reprojects the nth third depth map, the nth second pose transformation matrix and the nth multi-scale reliability mask, calculates the masked reprojection loss function, and obtains the masked reprojection loss value

[0062]

[0063] Where M is the multi-scale reliability mask, L p is the reprojection loss value;

[0064] 3.3.2.9 The edge enhancement module receives the nth edge prediction map from the third depth estimation network, receives the nth Canny edge map from the Canny operator module, calculates the edge detection loss function, and obtains the edge detection loss value L e :

[0065]

[0066] Among them, y is the nth Canny edge map, y i is the i-th pixel in y, p is the n-th edge prediction map, w i is the weight used to balance the number of positive and negative samples. If y i =1, then If y i =0, then

[0067] 3.3.2.10 The second uncertainty perception self-distillation module calculates the uncertainty perception loss function between the nth third depth map, the nth second uncertainty map and the nth optimized second depth map to obtain the second uncertainty perception loss value LL d :

[0068]

[0069] Among them, DD t is the second optimized depth map, DD s is the nth third depth map, ∑ r is the nth second uncertainty map;

[0070] 3.3.2.11 The second reprojection module uses the Adam optimization algorithm to minimize the reprojection loss value with mask to update the weight parameters of the third depth estimation network and the second pose estimation network to optimize the third depth estimation network and the second pose estimation network;

[0071] 3.3.2.12 The edge enhancement module uses the Adam optimization algorithm to minimize the edge detection loss value and updates the weight parameters of the third depth estimation network to optimize the third depth estimation network;

[0072] 3.3.2.13 The second uncertainty-aware self-distillation module uses the Adam optimization algorithm to minimize the uncertainty-aware loss value and updates the weight parameters of the third depth estimation network to optimize the third depth estimation network;

[0073] 3.3.2.14 Let n = n + 1. If n < N, go to 3.3.2.2 and use the next sample to continue training. If n ≥ N and e < E, go to 3.3.2.1 for the next round of training. If n ≥ N and e ≥ E, it means that all data have been trained and the number of training rounds has been reached, go to 3.3.3.

[0074] 3.3.3 The optimized third depth estimation network re-reads the preprocessed image data set from the data preprocessing module, performs depth estimation on the preprocessed image data set, obtains the optimized third depth map (i.e., the first optimized third depth map, ..., the nth optimized third depth map, ..., the Nth optimized third depth map), and obtains the trained self-supervised monocular depth estimation system, and then goes to the fourth step;

[0075] In the fourth step, the trained self-supervised monocular depth estimation system performs depth estimation on the two-dimensional video data input by the user to obtain an optimized third depth map (a three-dimensional depth map, i.e., a depth estimation result of the two-dimensional video data), the method is:

[0076] 4.1 The data acquisition module of the self-supervised monocular depth estimation system after training obtains the two-dimensional video data I that needs depth estimation taken by the user from the monocular camera, and sends I to the data preprocessing module;

[0077] 4.2 The data preprocessing module of the self-supervised monocular depth estimation system after training uses the preprocessing method described in step 2.3 to preprocess I to obtain the preprocessed image data I1, and sends I1 to the depth estimation refinement module;

[0078] 4.3 The third depth estimation network of the depth estimation refinement module of the trained self-supervised monocular depth estimation system performs depth estimation on I1: the third depth encoder receives I1, encodes I1, obtains a third depth feature, and sends the third depth feature to the third depth decoder; the third depth decoder receives the third depth feature, decodes the third depth feature, and obtains a third depth map I2. At this point, the self-supervised monocular depth estimation of the video data I is completed, and the depth estimation result, i.e., the three-dimensional third depth map I2, is obtained.

[0079] The following beneficial effects can be achieved by adopting the present invention:

[0080] (1) The present invention as a whole realizes a monocular depth estimation method with multi-scale uncertainty guidance and edge enhancement. The depth map and uncertainty map obtained by the teacher module and the student module assist the third step of joint training of self-supervised monocular depth estimation, uncertainty estimation and edge prediction, reduce noise interference, enhance the clarity of depth edges, and output more accurate depth information.

[0081] (2) The third step of the present invention is to generate a multi-scale reliability mask for depth maps of different scales. This strategy can make full use of uncertainty information at different scales, effectively shield pixel areas with high uncertainty, and effectively reduce the impact of noise on depth estimation results, thereby further improving the accuracy of depth estimation. This not only improves the stability of depth estimation, but also makes the model perform better in complex scenes.

[0082] (3) The present invention integrates uncertainty estimation and edge prediction into self-supervised monocular depth estimation. The uncertainty estimation of depth estimation in the third step helps to further reduce noise interference, so that the trained self-supervised monocular depth estimation system can better understand the complexity and uncertainty in the scene, and thus be more robust when dealing with difficulties such as occlusion and texture blur. The edge prediction in the third step helps the model capture the fine structural information of the image, further improving the accuracy of depth estimation. This multi-task collaborative working method enables the trained self-supervised monocular depth estimation system to understand the scene more comprehensively, thereby making more accurate depth judgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is the logical structure diagram of the self-supervised monocular depth estimation system constructed in the first step of the present invention.

[0084] Figure 2 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0085] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0086] like Figure 2 As shown, the overall technical solution of the present invention includes the following steps:

[0087] The first step is to build a self-supervised monocular depth estimation system. Figure 1 As shown, it consists of a data acquisition module, a data preprocessing module, a depth estimation teacher module, a depth estimation student module, and a depth estimation refinement module.

[0088] The data acquisition module is connected to the monocular camera and the data preprocessing module, acquires two-dimensional video data from the monocular camera, and sends the video data to the data preprocessing module;

[0089] The data preprocessing module is connected to the data acquisition module and the depth estimation teacher module, receives video data from the data acquisition module, preprocesses the video data, and obtains preprocessed image data. During training, the data preprocessing module sends the preprocessed image data to the depth estimation teacher module, the depth estimation student module, and the depth estimation refinement module; when performing depth estimation on the two-dimensional video data input by the user, the data preprocessing module sends the preprocessed image data to the depth estimation refinement module.

[0090] The function of the depth estimation teacher module is to assist in training the depth estimation refinement module. It only works during training and does not participate in the work when performing depth estimation on the two-dimensional video data input by the user. The depth estimation teacher module is connected to the data preprocessing module and the depth estimation student module, and is composed of a first depth estimation network, a first pose estimation network and a first reprojection module. The first depth estimation network is composed of a first depth encoder and a first depth decoder. The first depth estimation network performs depth estimation on the preprocessed image data received from the data preprocessing module, obtains a first depth map, and sends the first depth map to the first reprojection module; the first pose estimation network performs pose estimation on the preprocessed image data received from the data preprocessing module, obtains a first pose transformation matrix, and sends the first pose transformation matrix to the first reprojection module; the first reprojection module optimizes the first depth estimation network and the first pose estimation network to obtain an optimized first depth estimation network and a first pose estimation network; the optimized first depth estimation network performs depth estimation on the preprocessed image data received from the data preprocessing module, obtains an optimized first depth map, and sends the optimized first depth map to the depth estimation student module;

[0091] The function of the depth estimation student module is to assist in training the depth estimation refinement module. It only works during training and does not participate in the work when performing depth estimation on the two-dimensional video data input by the user. The depth estimation student module is connected to the data preprocessing module, the depth estimation teacher module, and the depth estimation refinement module, and is composed of a second depth estimation network and a first uncertainty-aware self-distillation module; the second depth estimation network is composed of a second depth encoder, a second depth decoder, and a first uncertainty decoder; the second depth estimation network uses the optimized first depth map received from the depth estimation teacher module as a pseudo-label, performs depth estimation and uncertainty estimation on the preprocessed image data received from the data preprocessing module, obtains a second depth map and a first uncertainty map, respectively, and sends the second depth map and the first uncertainty map to the first uncertainty-aware self-distillation module; the first uncertainty-aware self-distillation module optimizes the second depth estimation network to obtain an optimized second depth estimation network; the optimized second depth estimation network performs depth estimation and uncertainty estimation on the preprocessed image data received from the data preprocessing module, obtains an optimized second depth map and an optimized first uncertainty map, and sends the optimized second depth map and the optimized first uncertainty map to the depth estimation refinement module;

[0092] The depth estimation refinement module is connected to the data preprocessing module and the depth estimation student module, and is composed of a third depth estimation network, a second pose estimation network, a Canny operator module, a multi-scale reliability mask generation module, a second reprojection module, a second uncertainty-aware self-distillation module, and an edge enhancement module. During training, the multi-scale reliability mask generation module receives the optimized first uncertainty map from the depth estimation student module, binarizes the optimized first uncertainty map, obtains a multi-scale reliability mask, and sends the multi-scale reliability mask to the second reprojection module. The Canny operator module performs Canny edge calculation on the preprocessed image data received from the data preprocessing module (see the literature CANNYJ.A computational approach to edge detection[J].IEEE Transactions on patternanalysis and machine intelligence, 1986, (6): 679–98-–98.), obtain a Canny edge map, and send the Canny edge map to the edge enhancement module; the third depth estimation network is composed of a third depth encoder, a third depth decoder, a second uncertainty decoder and an edge decoder; the third depth estimation network performs depth estimation, uncertainty estimation and edge prediction on the preprocessed image data received from the data preprocessing module, and obtains a third depth map, a second uncertainty map and an edge prediction map respectively, and sends the third depth map to the second uncertainty-aware self-distillation module and the second reprojection module, sends the second uncertainty map to the second uncertainty-aware self-distillation module, and sends the edge prediction map to the edge enhancement module; the second pose The estimation network estimates the preprocessed image data received from the data preprocessing module to obtain the second pose transformation matrix, and sends the second pose transformation matrix to the second reprojection module; the edge enhancement module calculates the edge detection loss function according to the edge prediction map and the Canny edge map to optimize the third depth estimation network; the second reprojection module reprojects the third depth map, the second pose transformation matrix and the multi-scale reliability mask to optimize the third depth estimation network and the second pose estimation network; the second uncertainty-aware self-distillation module uses the optimized second depth map received from the depth estimation student module as a pseudo-label, distills the second uncertainty map and the third depth map to perform self-distillation optimization on the third depth estimation network, and obtains the optimized third depth estimation network. When performing depth estimation on the two-dimensional video data input by the user, the optimized third depth estimation network of the depth estimation refinement module performs depth estimation on the preprocessed image data received from the data preprocessing module to obtain the optimized third depth map, i.e., the depth estimation result.

[0093] The second step is to build an image dataset by:

[0094] 2.1 The data acquisition module acquires two-dimensional video data from the monocular camera. The number of image frames N in the video data is greater than 20,000, and sends the video data to the data preprocessing module;

[0095] 2.2 Initialize variable n = 1;

[0096] 2.3 The data preprocessing module uses a preprocessing method to preprocess the nth video data, the method is:

[0097] 2.3.1 scaling the image in the nth video data to a height of 192 pixels and a width of 640 pixels to obtain the scaled nth image;

[0098] 2.3.2 Normalize the pixels in the scaled nth image to between 0 and 1 to obtain a normalized nth image;

[0099] 2.3.3 Perform data augmentation on the normalized nth image to improve the robustness of training. The data augmentation methods include horizontal flipping, brightness adjustment (±0.2), saturation adjustment (±0.2), contrast adjustment (±0.2) and hue jitter (±0.1) to obtain the preprocessed nth image data.

[0100] 2.4 Let n=n+1; if n<N, go to 2.3; if n≥N, it means that the N video data have been preprocessed to obtain the preprocessed image data set, and go to the third step.

[0101] In the third step, the preprocessed image dataset is used to train the depth estimation teacher module, depth estimation student module and depth estimation refinement module in the self-supervised monocular depth estimation system. The training method of the depth estimation teacher module is the self-supervised monocular depth estimation method, the training method of the depth estimation student module is the uncertainty-aware self-distillation method, and the training method of the depth estimation refinement module is the self-supervised monocular depth estimation, uncertainty-aware self-distillation method and edge prediction multi-task joint training method.

[0102] 3.1 The depth estimation teacher module is trained using a self-supervised monocular depth estimation method, as follows:

[0103] 3.1.1 Set the total number of training rounds E, where E is a positive integer, preferably 30.

[0104] 3.1.2 Initialize the number of iterations e = 1;

[0105] 3.1.2.1 Initialize the number of trained samples n = 1;

[0106] 3.1.2.2 The data preprocessing module sends the nth image data (the nth original image in this case) in the preprocessed image data set to the first depth estimation network and the first pose estimation network;

[0107] 3.1.2.3 The first depth estimation network receives the nth image data in the preprocessed image data set, and performs depth estimation on the nth image data: the first depth encoder encodes the nth image data to obtain the nth first depth feature, and sends the nth first depth feature to a depth decoder; the first depth decoder decodes the nth first depth feature to obtain the nth first depth map, and sends the nth first depth map to the first reprojection module;

[0108] 3.1.2.4 The first pose estimation network receives the nth image data in the preprocessed image data set, performs pose estimation on the nth image data, obtains the nth first pose transformation matrix, and sends the nth first pose transformation matrix to the first reprojection module;

[0109] 3.1.2.5 The first reprojection module reprojects the nth first depth map and the nth first pose transformation matrix to obtain the reconstructed nth synthetic image, calculates the reprojection loss function between the nth synthetic image and the nth original image, and obtains the reprojection loss value L p :

[0110] L p =|I t -I s→t |,

[0111] Among them, I t is the nth original image, I s→t is the reconstructed nth synthetic image, |I t -I s→t | indicates request I t -I s→t The absolute value of

[0112] 3.1.2.6 The first reprojection module uses the Adam optimization algorithm (see the reference "Kingma DP, Ba J. Adam: A method for stochastic optimization [J]. arXiv preprint arXiv: 1412.6980, 2014", Kingma's paper: Adam, a stochastic optimization method) to minimize L p , to update the weight parameters of the first depth estimation network and the first pose estimation network to optimize the first depth estimation network and the first pose estimation network;

[0113] 3.1.2.7 Let n = n + 1. If n < N, go to 3.1.2.2 and use the next sample to continue training. If n ≥ N and e < E, go to 3.1.2.1 for the next round of training. If n ≥ N and e ≥ E, it means that all data have been trained and the number of training rounds has been reached, go to 3.1.3.

[0114] 3.1.3 The optimized first depth estimation network re-reads the preprocessed image data set of the data preprocessing module, performs depth estimation on the preprocessed image data set, obtains N optimized first depth maps (i.e., the 1st optimized first depth map, ..., the nth optimized first depth map, ..., the Nth optimized first depth map), and sends the N optimized first depth maps to the depth estimation student module.

[0115] 3.2 The depth estimation student module is trained using uncertainty-aware self-distillation as follows:

[0116] 3.2.1 Set the total number of iterations to be equal to E.

[0117] 3.2.2 Initialize the number of iterations e = 1;

[0118] 3.2.2.1 Initialize the number of trained samples n = 1;

[0119] 3.2.2.2 The data preprocessing module sends the nth image data (the nth original image in this case) in the preprocessed image data set to the second depth estimation network;

[0120] 3.2.2.3 The second depth estimation network receives the nth image data in the preprocessed data set, and performs depth estimation and uncertainty estimation on the nth image data: the second depth encoder encodes the nth image data to obtain the nth second depth feature, and sends the nth second depth feature to the second depth decoder and the first uncertainty decoder; the second depth decoder receives the nth second depth feature, decodes the nth second depth feature, obtains the nth second depth map, and sends the nth second depth map to the first uncertainty-aware self-distillation module; the first uncertainty decoder receives the nth second depth feature, performs uncertainty decoding on the nth second depth feature, obtains the nth first uncertainty map, and sends the nth first uncertainty map to the first uncertainty-aware self-distillation module;

[0121] 3.2.2.4 The first uncertainty-aware self-distillation module receives the nth optimized first depth map from the depth estimation teacher module;

[0122] 3.2.2.5 The first uncertainty-aware self-distillation module calculates the uncertainty-aware loss function between the nth second depth map, the nth first uncertainty map, and the nth optimized first depth map to obtain the first uncertainty-aware loss value L d :

[0123]

[0124] Among them, D t is the first depth map after the nth optimization, D s is the nth second depth map, ∑ s is the nth first uncertainty map;

[0125] 3.2.2.6 The first uncertainty-aware self-distillation module uses the Adam optimization algorithm to minimize L d , to update the weight parameters of the second depth estimation network to optimize the second depth estimation network;

[0126] 3.2.2.7 Let n=n+1. If n<N, go to 3.2.2.2 and use the next sample to continue training. If n≥N and e<E, go to 3.2.2.1 for the next round of training. If n≥N and e≥E, it means that all data have been trained and the number of training rounds has been reached, go to 3.2.3.

[0127] 3.2.3 The optimized second depth estimation network re-reads the preprocessed image data set from the data preprocessing module, performs depth estimation on the preprocessed image data set, and obtains N optimized second depth maps (i.e., the 1st optimized second depth map, …, the nth optimized second depth map, …, the Nth optimized second depth map) and N optimized first uncertainty maps (i.e., the 1st optimized first uncertainty map, …, the nth optimized first uncertainty map, …, the Nth optimized first uncertainty map), and sends the N optimized second depth maps and the N optimized first uncertainty maps to the depth estimation refinement module.

[0128] 3.3 The depth estimation refinement module is trained using self-supervised monocular depth estimation, uncertainty-aware self-distillation, and edge prediction multi-task joint training, as follows:

[0129] 3.3.1 Set the number of iterations to E.

[0130] 3.3.2 Initialize the number of iterations e = 1;

[0131] 3.3.2.1 Initialize the number of trained samples n = 1;

[0132] 3.3.2.2 The data preprocessing module sends the nth image data (the nth original image in this case) in the preprocessed image data set to the third depth estimation network, the second pose estimation network, and the Canny operator module;

[0133] 3.3.2.3 The multi-scale reliability mask generation module binarizes the nth optimized first uncertainty map received from the depth estimation student module to obtain a multi-scale reliability mask, and sends the multi-scale reliability mask to the second reprojection module;

[0134] 3.3.2.4 The Canny operator module performs Canny edge calculation on the nth image data received from the data preprocessing module to obtain the nth Canny edge map, and sends the nth Canny edge map to the edge enhancement module;

[0135] 3.3.2.5 The third depth estimation network performs depth estimation, uncertainty estimation and edge prediction on the nth image data received from the data preprocessing module: the third depth encoder receives the nth image data, encodes the nth image data, obtains the nth third depth feature, and sends the nth third depth feature to the third depth decoder, the second uncertainty decoder, and the edge decoder; the third depth decoder receives the nth third depth feature, decodes the nth third depth feature, obtains the nth third depth map, and sends the nth third depth map to the second reprojection module; the second uncertainty decoder receives the nth third depth feature, performs uncertainty decoding on the nth third depth feature, obtains the nth second uncertainty map, and sends the nth second uncertainty map to the second uncertainty-aware self-distillation module; the edge decoder receives the nth third depth feature, performs edge prediction on the nth third depth feature, obtains the nth edge prediction map, and sends the nth edge prediction map to the edge enhancement module;

[0136] 3.3.2.6 The second pose estimation network performs pose estimation on the nth image data received from the data preprocessing module to obtain a second pose transformation matrix, and sends the second pose transformation matrix to the second reprojection module;

[0137] 3.3.2.7 The second uncertainty-aware self-distillation module receives the nth optimized second depth map from the depth estimation student module;

[0138] 3.3.2.8 The second reprojection module reprojects the nth third depth map, the nth second pose transformation matrix and the nth multi-scale reliability mask, calculates the masked reprojection loss function, and obtains the masked reprojection loss value

[0139]

[0140] Where M is the multi-scale reliability mask, L p is the reprojection loss value;

[0141] 3.3.2.9 The edge enhancement module receives the nth edge prediction map from the third depth estimation network, receives the nth Canny edge map from the Canny operator module, calculates the edge detection loss function, and obtains the edge detection loss value L e :

[0142]

[0143] Among them, y is the nth Canny edge map, y i is the i-th pixel in y, p is the n-th edge prediction map, w i is the weight used to balance the number of positive and negative samples. If y i =1, then If y i =0, then

[0144] 3.3.2.10 The second uncertainty perception self-distillation module calculates the uncertainty perception loss function between the nth third depth map, the nth second uncertainty map and the nth optimized second depth map to obtain the second uncertainty perception loss value LL d :

[0145]

[0146] Among them, DD t is the second optimized depth map, DD s is the nth third depth map, ∑ r is the nth second uncertainty map;

[0147] 3.3.2.11 The second reprojection module uses the Adam optimization algorithm to minimize the reprojection loss value with mask to update the weight parameters of the third depth estimation network and the second pose estimation network to optimize the third depth estimation network and the second pose estimation network;

[0148] 3.3.2.12 The edge enhancement module uses the Adam optimization algorithm to minimize the edge detection loss value and updates the weight parameters of the third depth estimation network to optimize the third depth estimation network;

[0149] 3.3.2.13 The second uncertainty-aware self-distillation module uses the Adam optimization algorithm to minimize the uncertainty-aware loss value and updates the weight parameters of the third depth estimation network to optimize the third depth estimation network;

[0150] 3.3.2.14 Let n = n + 1. If n < N, go to 3.3.2.2 and use the next sample to continue training. If n ≥ N and e < E, go to 3.3.2.1 for the next round of training. If n ≥ N and e ≥ E, it means that all data have been trained and the number of training rounds has been reached, go to 3.3.3.

[0151] 3.3.3 The optimized third depth estimation network re-reads the preprocessed image data set from the data preprocessing module, performs depth estimation on the preprocessed image data set, obtains the optimized third depth map (i.e., the first optimized third depth map, ..., the nth optimized third depth map, ..., the Nth optimized third depth map), and obtains the trained self-supervised monocular depth estimation system, and then goes to the fourth step;

[0152] In the fourth step, the trained self-supervised monocular depth estimation system performs depth estimation on the two-dimensional video data input by the user to obtain an optimized third depth map (a three-dimensional depth map, i.e., a depth estimation result of the two-dimensional video data), the method is:

[0153] 4.1 The data acquisition module of the self-supervised monocular depth estimation system after training obtains the two-dimensional video data I that needs depth estimation taken by the user from the monocular camera, and sends I to the data preprocessing module;

[0154] 4.2 The data preprocessing module of the self-supervised monocular depth estimation system after training uses the preprocessing method described in step 2.3 to preprocess I to obtain the preprocessed image data I1, and sends I1 to the depth estimation refinement module;

[0155] 4.3 The third depth estimation network of the depth estimation refinement module of the trained self-supervised monocular depth estimation system performs depth estimation on I1: the third depth encoder receives I1, encodes I1, obtains a third depth feature, and sends the third depth feature to the third depth decoder; the third depth decoder receives the third depth feature, decodes the third depth feature, and obtains a third depth map I2. At this point, the self-supervised monocular depth estimation of the video data I is completed, and the depth estimation result, i.e., the three-dimensional third depth map I2, is obtained.

[0156] In order to verify the versatility and scalability of the present invention, the following experiments were conducted:

[0157] The system constructed in the first step of the present invention is trained and tested using the depth estimation dataset KITTI. The training set of the KITTI dataset contains 22,600 training image pairs, the validation set contains 888 image pairs, and the test set contains 697 images. When inputting data, the image is first scaled to 640×192 size and then input into the module for inference.

[0158] The evaluation index of depth estimation is an important criterion for measuring the performance of depth estimation algorithms. These indicators mainly focus on the error between the predicted depth and the actual depth, including Absolute Relative Error (AbsRel), Squared Relative Error (Sq Rel), Root Mean Squared Error (RMSE), and Root Mean Squared Logarithmic Error (RMSElog). Acc.1, Acc.2, and Acc.3 represent the accuracy of the depth estimation results, the larger the better. As shown in the following formula:

[0159]

[0160] Among them, d i is the depth value predicted by the depth estimation, is the actual depth map, and M is the number of pixels in the depth map.

[0161] The experiment uses SAD-Depth as the first, second, and third depth estimation networks (see the literature Song, Linna, Dian-xi Shi, Jianqiang Xia, Qianying Ouyang, Ziteng Qiao, Songchang Jin and Shaowu Yang. "Spatial-Aware Dynamic Lightweight Self-Supervised MonocularDepth Estimation." IEEE Robotics and Automation Letters 9(2024):883-890.), and ResNet18 as the pose estimation network (see the literature HE K, ZHANG X, REN S, et al. Deep Residual Learning for Image Recognition[J]. 2016IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2015:770-8.), and experiments are carried out on the KITTI dataset. The results are shown in Table 1.

[0162] It can be seen from Table 1 that after the self-supervised monocular depth estimation system of the present invention is trained with the KITTI data set, its third depth estimation network is significantly better than the SAD-Depth model in terms of depth error and accuracy, and surpasses a series of representative self-supervised monocular depth estimation methods. Specifically, the indicators Abs Rel, Sq Rel, RMSE and RMSE log balance the error between the predicted depth value and the true depth value, and the values ​​of these indicators of this system are low, showing extremely high prediction accuracy, which means that the predicted depth value is very close to the true value. At the same time, the indicators Acc.1, Acc.2 and Acc.3 measure the proportion of the predicted depth value within a certain threshold of the true depth value. The values ​​of these indicators of this system are high, which further verifies that the model can give predictions close to the true depth value on most pixels, fully proving the accuracy of its prediction. These results fully prove that the system can effectively integrate the information of the first two stages in the refinement stage, combine multi-scale uncertainty and edge information, and realize the precise refinement of the depth map and the fine capture of edge information, thereby significantly improving the accuracy and performance of depth estimation.

[0163] Table 1

[0164]

[0165] In Table 1, ZHOU, see the literature ZHOU T, BROWN MA, SNAVELY N, et al. Unsupervised Learning of Depth and Ego-Motion from Video[J]. 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2017: 6612-9;

[0166] GeoNet, see the literature YIN Z, SHI J. GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose [J]. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2018: 1983-92.

[0167] Monodepth2, see the literature GODARD C, AODHAO M, BROSTOW G J. Digging Into Self-Supervised Monocular Depth Estimation[J]. 2019IEEE / CVF International Conference on Computer Vision (ICCV), 2018:3827-37.

[0168] R-MSFM, see the literature ZHOU Z, FAN

[0169] SAD-Depth, see Song, Linna, Dian-xi Shi, Jianqiang Xia, Qianying Ouyang, Ziteng Qiao, Songchang Jin and Shaowu Yang. "Spatial-Aware Dynamic LightweightSelf-Supervised Monocular Depth Estimation." IEEE Robotics and Automation Letters 9(2024):883-890.

[0170] The above experimental examples have demonstrated the effectiveness of the self-supervised monocular depth estimation method described in the present invention in improving the depth estimation performance.

[0171] Finally, it should be pointed out that the above content is only used to illustrate the technical solution of the present invention and does not constitute a limitation. Those skilled in the art should understand that the specific implementation methods can be modified or replaced by equivalents without departing from the spirit and scope of the present invention, and these modifications or replacements should be included in the scope of protection of the claims of the present invention.

Claims

1. A self-supervised monocular depth estimation method based on uncertainty and edge enhancement, Features The following steps are involved: The first step is to build a self-supervised monocular depth estimation system. The self-supervised monocular depth estimation system consists of a data acquisition module, a data preprocessing module, a depth estimation teacher module, a depth estimation student module, and a depth estimation refinement module. The data acquisition module is connected to the monocular camera and the data preprocessing module, acquires two-dimensional video data from the monocular camera, and sends the video data to the data preprocessing module; The data preprocessing module is connected to the data acquisition module and the depth estimation teacher module, receives video data from the data acquisition module, preprocesses the video data, and obtains preprocessed image data; during training, the data preprocessing module sends the preprocessed image data to the depth estimation teacher module, the depth estimation student module, and the depth estimation refinement module; when performing depth estimation on the two-dimensional video data input by the user, the data preprocessing module sends the preprocessed image data to the depth estimation refinement module; The function of the depth estimation teacher module is to assist in training the depth estimation refinement module. The depth estimation teacher module is connected to the data preprocessing module and the depth estimation student module, and is composed of a first depth estimation network, a first pose estimation network and a first reprojection module; the first depth estimation network is composed of a first depth encoder and a first depth decoder; the first depth estimation network performs depth estimation on the preprocessed image data received from the data preprocessing module to obtain a first depth map, and sends the first depth map to the first reprojection module; The first pose estimation network performs pose estimation on the preprocessed image data received from the data preprocessing module to obtain a first pose transformation matrix, and sends the first pose transformation matrix to the first reprojection module; The first reprojection module optimizes the first depth estimation network and the first pose estimation network to obtain optimized first depth estimation network and first pose estimation network; The optimized first depth estimation network performs depth estimation on the preprocessed image data received from the data preprocessing module to obtain an optimized first depth map, and sends the optimized first depth map to the depth estimation student module; The function of the depth estimation student module is to assist in training the depth estimation refinement module. The depth estimation student module is connected to the data preprocessing module, the depth estimation teacher module, and the depth estimation refinement module, and is composed of a second depth estimation network and a first uncertainty-aware self-distillation module; the second depth estimation network is composed of a second depth encoder, a second depth decoder, and a first uncertainty decoder; the second depth estimation network receives the optimized first depth map from the depth estimation teacher module, performs depth estimation and uncertainty estimation on the preprocessed image data received from the data preprocessing module, obtains the second depth map and the first uncertainty map respectively, and sends the second depth map and the first uncertainty map to the first uncertainty-aware self-distillation module; the first uncertainty-aware self-distillation module optimizes the second depth estimation network to obtain an optimized second depth estimation network; the optimized second depth estimation network performs depth estimation and uncertainty estimation on the preprocessed image data received from the data preprocessing module, obtains an optimized second depth map and an optimized first uncertainty map, and sends the optimized second depth map and the optimized first uncertainty map to the depth estimation refinement module; The depth estimation refinement module is connected to the data preprocessing module and the depth estimation student module, and is composed of a third depth estimation network, a second pose estimation network, a Canny operator module, a multi-scale reliability mask generation module, a second reprojection module, a second uncertainty-aware self-distillation module, and an edge enhancement module; during training, the multi-scale reliability mask generation module receives the optimized first uncertainty map from the depth estimation student module, performs a binarization operation on the optimized first uncertainty map, obtains a multi-scale reliability mask, and sends the multi-scale reliability mask to the second reprojection module; the Canny operator module performs Canny edge calculation on the preprocessed image data received from the data preprocessing module to obtain a Canny edge map, and sends the Canny edge map to the edge enhancement module; The third depth estimation network is composed of a third depth encoder, a third depth decoder, a second uncertainty decoder and an edge decoder; the third depth estimation network performs depth estimation, uncertainty estimation and edge prediction on the preprocessed image data received from the data preprocessing module, obtains a third depth map, a second uncertainty map and an edge prediction map respectively, sends the third depth map to the second uncertainty-aware self-distillation module and the second reprojection module, sends the second uncertainty map to the second uncertainty-aware self-distillation module, and sends the edge prediction map to the edge enhancement module; the second pose estimation network estimates the preprocessed image data received from the data preprocessing module, obtains a second pose transformation matrix, and sends the second pose transformation matrix to the second reprojection module; The edge enhancement module calculates the edge detection loss function according to the edge prediction map and the Canny edge map to optimize the third depth estimation network; the second reprojection module reprojects the third depth map, the second pose transformation matrix and the multi-scale reliability mask to optimize the third depth estimation network and the second pose estimation network; the second uncertainty-aware self-distillation module receives the optimized second depth map from the depth estimation student module, distills the second uncertainty map and the third depth map to perform self-distillation optimization on the third depth estimation network, and obtains the optimized third depth estimation network; When performing depth estimation on the two-dimensional video data input by the user, the optimized third depth estimation network of the depth estimation refinement module performs depth estimation on the preprocessed image data received from the data preprocessing module to obtain an optimized third depth map, i.e., a depth estimation result; The second step is to build an image dataset by: 2.1 The data acquisition module acquires two-dimensional video data from the monocular camera. The number of image frames N in the video data is a positive integer. The video data is sent to the data preprocessing module. 2.2 Initialize variable n = 1; 2.3 The data preprocessing module uses a preprocessing method to preprocess the nth video data, the method is: 2.3.1 scaling the image in the nth video data to a height of 192 pixels and a width of 640 pixels to obtain the scaled nth image; 2.3.2 Normalize the pixels in the scaled nth image to between 0 and 1 to obtain a normalized nth image; 2.3.3 Perform data enhancement on the normalized nth image to obtain the preprocessed nth image data; 2.4 Let n = n + 1; If n<N, go to 2.3; If n≥N, it means that all N video data have been preprocessed, and the preprocessed image data set is obtained, and then go to the third step; In the third step, the preprocessed image dataset is used to train the depth estimation teacher module, depth estimation student module and depth estimation refinement module in the self-supervised monocular depth estimation system. The method is: 3.1 The depth estimation teacher module is trained using a self-supervised monocular depth estimation method, as follows: 3.1.1 Set the total number of training rounds E to a positive integer; 3.1.2 Initialize the number of iterations e = 1; 3.1.2.1 Initialize the number of trained samples n = 1; 3.1.2.2 The data preprocessing module sends the nth image data in the preprocessed image data set, i.e., the nth original image, to the first depth estimation network and the first pose estimation network; 3.1.2.3 The first depth estimation network receives the nth image data in the preprocessed image data set, and performs depth estimation on the nth image data: the first depth encoder encodes the nth image data to obtain the nth first depth feature, and sends the nth first depth feature to a depth decoder; the first depth decoder decodes the nth first depth feature to obtain the nth first depth map, and sends the nth first depth map to the first reprojection module; 3.1.2.4 The first pose estimation network receives the n-th image data in the preprocessed image dataset, performs pose estimation on the n-th image data, obtains the n-th first pose transformation matrix, and sends the n-th first pose transformation matrix to the first reprojection module; 3.1.2.5 The first reprojection module reprojects the nth first depth map and the nth first pose transformation matrix to obtain the reconstructed nth synthetic image, calculates the reprojection loss function between the nth synthetic image and the nth original image, and obtains the reprojection loss value L p : L p =|I t -I s→t |, Among them, I t is the nth original image, I s→t is the reconstructed nth synthetic image, |I t -I s→t | indicates request I t -I s→t The absolute value of 3.1.2.6 The first reprojection module uses the Adam optimization algorithm to minimize Lp, updates the weight parameters of the first depth estimation network and the first pose estimation network, so as to optimize the first depth estimation network and the first pose estimation network; 3.1.2.7 Let n = n + 1. If n < N, go to 3.1.2.2 to continue training with the next sample; if n ≥ N and e < E, go to 3.1.2.1 for the next round of training; if n ≥ N and e ≥ E, it means that all data has been trained and the number of training rounds has been reached, go to 3.1.3; 3.1.3 The optimized first depth estimation network rereads the preprocessed image dataset of the data preprocessing module, performs depth estimation on the preprocessed image dataset, obtains N optimized first depth maps, namely the 1st optimized first depth map,..., the n-th optimized first depth map,..., the N-th optimized first depth map, and sends the N optimized first depth maps to the depth estimation student module; 3.2 The depth estimation student module is trained using the uncertainty-aware self-distillation method, and the method is as follows: 3.2.1 Set the total number of iteration rounds to be equal to E; 3.2.2 Initialize the iteration number e = 1; 3.2.2.1 Initialize the number of trained samples n = 1; 3.2.2.2 The data preprocessing module sends the n-th image data in the preprocessed image dataset, that is, the n-th original image, to the second depth estimation network; 3.2.2.3 The second depth estimation network receives the n-th image data in the preprocessed dataset, and performs depth estimation and uncertainty estimation on the n-th image data: the second depth encoder encodes the n-th image data to obtain the n-th second depth feature, and sends the n-th second depth feature to the second depth decoder and the first uncertainty decoder; the second depth decoder receives the n-th second depth feature, decodes the n-th second depth feature to obtain the n-th second depth map, and sends the n-th second depth map to the first uncertainty-aware self-distillation module; the first uncertainty decoder receives the n-th second depth feature, decodes the uncertainty of the n-th second depth feature to obtain the n-th first uncertainty map, and sends the n-th first uncertainty map to the first uncertainty-aware self-distillation module; 3.2.2.4 The first uncertainty-aware self-distillation module receives the n-th optimized first depth map from the depth estimation teacher module; 3.2.2.5 The first uncertainty-aware self-distillation module calculates the uncertainty-aware loss function between the nth second depth map, the nth first uncertainty map, and the nth optimized first depth map to obtain the first uncertainty-aware loss value L d : Among them, D t is the first depth map after the nth optimization, D s is the nth second depth map, ∑ s is the nth first uncertainty map; 3.2.2.6 The first uncertainty-aware self-distillation module uses the Adam optimization algorithm to minimize L d , updating the weight parameters of the second depth estimation network to optimize the second depth estimation network; 3.2.2.7 Let n = n + 1. If n < N, go to 3.2.2.2 to continue training with the next sample; if n ≥ N and e < E, go to 3.2.2.1 for the next round of training; if n ≥ N and e ≥ E, it means that all data has been trained and the number of training rounds has been reached, go to 3.2.3; 3.2.3 The optimized second depth estimation network re-reads the preprocessed image data set from the data preprocessing module, performs depth estimation on the preprocessed image data set, and obtains N optimized second depth maps and N optimized first uncertainty maps, the N optimized second depth maps are the first optimized second depth map, ..., the nth optimized second depth map, ..., the Nth optimized second depth map, the N optimized first uncertainty maps are the first optimized first uncertainty map, ..., the nth optimized first uncertainty map, ..., the Nth optimized first uncertainty map, and the N optimized second depth maps and the N optimized first uncertainty maps are sent to the depth estimation refinement module; 3.3 The depth estimation refinement module is trained using self-supervised monocular depth estimation, uncertainty-aware self-distillation, and edge prediction multi-task joint training, as follows: 3.3.1 Set the number of iterations to E; 3.3.2 Initialize the number of iterations e = 1; 3.3.2.1 Initialize the number of trained samples n = 1; 3.3.2.2 The data preprocessing module sends the nth image data in the preprocessed image data set, that is, the nth original image, to the third depth estimation network, the second pose estimation network, and the Canny operator module; 3.3.2.3 The multi-scale reliability mask generation module binarizes the nth optimized first uncertainty map received from the depth estimation student module to obtain a multi-scale reliability mask, and sends the multi-scale reliability mask to the second reprojection module; 3.3.2.4 The Canny operator module performs Canny edge calculation on the nth image data received from the data preprocessing module to obtain the nth Canny edge map, and sends the nth Canny edge map to the edge enhancement module; 3.3.2.5 The third depth estimation network performs depth estimation, uncertainty estimation and edge prediction on the nth image data received from the data preprocessing module: the third depth encoder receives the nth image data, encodes the nth image data, obtains the nth third depth feature, and sends the nth third depth feature to the third depth decoder, the second uncertainty decoder, and the edge decoder; the third depth decoder receives the nth third depth feature, decodes the nth third depth feature, obtains the nth third depth map, and sends the nth third depth map to the second reprojection module; the second uncertainty decoder receives the nth third depth feature, performs uncertainty decoding on the nth third depth feature, obtains the nth second uncertainty map, and sends the nth second uncertainty map to the second uncertainty-aware self-distillation module; the edge decoder receives the nth third depth feature, performs edge prediction on the nth third depth feature, obtains the nth edge prediction map, and sends the nth edge prediction map to the edge enhancement module; 3.3.2.6 The second pose estimation network performs pose estimation on the nth image data received from the data preprocessing module to obtain a second pose transformation matrix, and sends the second pose transformation matrix to the second reprojection module; 3.3.2.7 The second uncertainty-aware self-distillation module receives the nth optimized second depth map from the depth estimation student module; 3.3.2.8 The second reprojection module reprojects the nth third depth map, the nth second pose transformation matrix and the nth multi-scale reliability mask, calculates the masked reprojection loss function, and obtains the masked reprojection loss value Where M is the multi-scale reliability mask, L p is the reprojection loss value; 3.3.2.9 The edge enhancement module receives the nth edge prediction map from the third depth estimation network, receives the nth Canny edge map from the Canny operator module, calculates the edge detection loss function, and obtains the edge detection loss value L e : Among them, y is the nth Canny edge map, y i is the i-th pixel in y, p is the n-th edge prediction map, w i is the weight used to balance the number of positive and negative samples. If y i =1, then If y i =0, then 3.3.2.10 The second uncertainty perception self-distillation module calculates the uncertainty perception loss function between the nth third depth map, the nth second uncertainty map and the nth optimized second depth map to obtain the second uncertainty perception loss value LL d : Among them, DD t is the second optimized depth map, DD s is the nth third depth map, ∑ r is the nth second uncertainty map; 3.3.2.11 The second reprojection module uses the Adam optimization algorithm to minimize the reprojection loss value with mask to update the weight parameters of the third depth estimation network and the second pose estimation network to optimize the third depth estimation network and the second pose estimation network; 3.3.2.12 The edge enhancement module uses the Adam optimization algorithm to minimize the edge detection loss value and updates the weight parameters of the third depth estimation network to optimize the third depth estimation network; 3.3.2.13 The second uncertainty-aware self-distillation module uses the Adam optimization algorithm to minimize the uncertainty-aware loss value and updates the weight parameters of the third depth estimation network to optimize the third depth estimation network; 3.3.2.14 Let n = n + 1. If n < N, go to 3.3.2.2 and use the next sample to continue training. If n ≥ N and e < E, go to 3.3.2.1 for the next round of training. If n ≥ N and e ≥ E, it means that all data have been trained and the number of training rounds has been reached, go to 3.3.

3. 3.3.3 The optimized third depth estimation network re-reads the preprocessed image data set from the data preprocessing module, performs depth estimation on the preprocessed image data set, obtains the optimized third depth map, i.e., the first optimized third depth map, ..., the nth optimized third depth map, ..., the Nth optimized third depth map, and obtains the trained self-supervised monocular depth estimation system, and then goes to the fourth step; In the fourth step, the trained self-supervised monocular depth estimation system performs depth estimation on the two-dimensional video data input by the user to obtain a self-supervised monocular depth estimation result for the two-dimensional video data, and the method is: 4.1 The data acquisition module of the trained self-supervised monocular depth estimation system obtains the two-dimensional video data I that needs depth estimation taken by the user from the monocular camera, and sends I to the data preprocessing module; 4.2 The data preprocessing module of the self-supervised monocular depth estimation system after training uses the preprocessing method described in step 2.3 to preprocess I to obtain the preprocessed image data I1, and sends I1 to the depth estimation refinement module; 4.3 The third depth estimation network of the depth estimation refinement module of the trained self-supervised monocular depth estimation system performs depth estimation on I1: the third depth encoder receives I1, encodes I1, obtains a third depth feature, and sends the third depth feature to the third depth decoder; the third depth decoder receives the third depth feature, decodes the third depth feature, and obtains the self-supervised monocular depth estimation result, that is, the three-dimensional third depth map I2.

2. A self-supervised monocular depth estimation method based on uncertainty and edge enhancement as claimed in claim 1, Features The number of image frames N in the video data in step 2.1 is greater than 20,000.

3. A self-supervised monocular depth estimation method based on uncertainty and edge enhancement as claimed in claim 1, Features The method for data enhancement of the normalized nth image described in step 2.3.3 includes horizontal flipping, brightness adjustment ±0.2, saturation adjustment ±0.2, contrast adjustment ±0.2 and hue jitter ±0.

1.

4. A self-supervised monocular depth estimation method based on uncertainty and edge enhancement as claimed in claim 1, Features The total number of training rounds E is 30.

Images