A dense reconstruction method based on self-supervised monocular depth estimation

Abstract

The application relates to the technical field of computer vision, and discloses a dense reconstruction method based on self-supervised monocular depth estimation, wherein the existing self-supervised method does not consider the scale problem, and the output depth estimation result has the problem of inconsistent scale between frames. The application is designed aiming at the problem, so as to realize a depth estimation method with consistent scale. The application innovatively proposes a mapping framework, which can obtain a current real-time dense reconstruction result and a global point cloud map under the condition of only one RGB camera input. In the prior art, the depth estimation method is not used for real-time dense reconstruction and establishment of a global point cloud map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a dense reconstruction method based on self-supervised monocular depth estimation. BACKGROUND

[0002] Monocular depth estimation from monocular camera is becoming an important task in computer vision and robotics. Popular deep learning-based methods can be divided into supervised-based methods and self-supervised methods. However, for supervised-based methods, the cost of obtaining ground truth is high. Therefore, self-supervised methods that do not require depth ground truth have become a promising alternative for depth estimation. Most current monocular self-supervised depth estimation methods are based on continuous frame image input, and the final prediction is the depth estimation result of the current frame. In contrast, it is often difficult to estimate the depth of the entire video sequence in the monocular depth estimation task, because the camera motion in the video sequence is unknown. The predicted depth map has a scale ambiguity problem in monocular depth estimation. If we use the previous single-frame depth estimation model to predict the video input, an unavoidable problem is that the scale between frames is often inconsistent, which is not necessary to consider in single-frame depth prediction. Moreover, it is also difficult to reconstruct in large scenes without scale constraints.

[0003] Current monocular depth estimation methods include Monodepth2 and SfMLearner. The Monodepth2 method is based on a U-Net network structure, which can output an estimated depth map from an input RGB image. However, there is no difference in error indicators and accuracy indicators between low-resolution and high-resolution input depth estimation results. The SfMLearner method is based on continuous frame image input, and the final prediction is the depth estimation result of the current frame. However, the predicted depth map has a scale ambiguity problem in monocular depth estimation, and the scale between frames is often inconsistent. SUMMARY

[0004] To solve the above problems, the present application provides a dense reconstruction method based on self-supervised monocular depth estimation. Existing self-supervised methods do not consider the scale problem, and the output depth estimation result has a scale inconsistency problem between frames. The present application is designed to solve this problem to achieve a scale-consistent depth estimation method.

[0005] To achieve the above purpose, the present application provides a dense reconstruction method based on self-supervised monocular depth estimation, comprising the following steps:

[0006] S1, preparing an RGB image and depth data, and the RGB image and the depth data correspond one-to-one;

[0007] S2, input the RGB image into a depth estimation network for training and calculating a loss function;

[0008] S3, network optimization, the loss calculated in step S2 is added and used for network optimization by back propagation until the error on the test data set no longer decreases;

[0009] S4, input the RGB image into the model optimized in step S3 to output the depth map of the current frame;

[0010] S5, input the RGB image and the depth map into a SLAM system to output a six-degree-of-freedom optimized pose;

[0011] S6, convert the RGB image and the depth map into a point cloud in the camera coordinate system by a point cloud projection formula;

[0012] S7, convert the six-degree-of-freedom optimized pose into the world coordinate system by a coordinate transformation formula, and add the point cloud of each frame to obtain a globally consistent point cloud map.

[0013] Preferably, the specific steps of the depth estimation network training stage in step S2 are as follows:

[0014] S21, the RGB image first enters a convolution layer and a BN layer for processing;

[0015] S22, then enters a ReLU activation function and a max pooling layer, the max pooling layer compresses the extracted features and simplifies the network complexity;

[0016] S23, then enters Layer1, the Layer1 is composed of two residual blocks, and the activation function used in the residual block is an ELU activation function;

[0017] S24, then enters Layer2, Layer3, and Layer4, the Layer2, Layer3, and Layer4 have the same structure as Layer1, and the convolution kernel with an increased step size is used for convolution operation instead of the down-sampling process, the size of the feature map is reduced by a factor of two, and the encoder network completes the task;

[0018] S25, then the image features output by the encoder in step S24 are sent into a decoder for integration and analysis, first enter an Upconv structure, fuse the feature maps of the same size, and then perform convolution operation and up-sampling operation;

[0019] S26, repeat the Upconv process of step S25 three times, and the input of each Upconv is the output of the previous layer network and the feature map of the same scale in the encoder network;

[0020] S27, the output end of each Upconv uses a Sigmoid function to convert the output result δ of the last Sigmoid function into a depth D using the following formula: D = a * sigmoid(δ) + b, and a and b are selected to constrain D between 0.1 and 100 units.

[0021]

[0022] Preferably, the specific step of calculating the loss function in step S2 is as follows:

[0023] S28, the photometric error loss function and the edge smoothness regular loss function are calculated using the depth D obtained in step S27;

[0024] S29, the photometric error loss function is calculated: the reconstructed image I is obtained using the inverse warping transformation Then, the accuracy of the visual quantity estimated by the network is measured by calculating the similarity between the reconstructed image I and the target image I t :

[0025]

[0026] Wherein α is a hyperparameter obtained by cross-validation method, set to 0.85, and SSIM is a structural similarity measurement index calculation algorithm, and the calculation formula is

[0027]

[0028] S210, the edge smoothness regular loss function is calculated: using only the photometric error loss function will produce ambiguity in some special areas, such as textureless areas or areas with repeated texture, which will have a large number of repeated pixels and repeated features, therefore, in order to reduce the influence of these areas on the final result, an edge smoothness regular loss function is introduced in the unsupervised depth estimation:

[0029]

[0030] Wherein is the partial derivative in x, y direction respectively, d t is the depth of the target frame, the regular term makes the image gradient not obvious, and the predicted depth value should be consistent in x and y directions;

[0031] S211, the minimum inter-frame loss is calculated: in unsupervised monocular depth estimation, the problem of the same object being occluded in different frames may occur, if occlusion occurs, the pixels will be matched incorrectly, resulting in a high photometric error, therefore, a multi-frame optimization depth estimation network is needed, assuming that three consecutive frames of images are used to optimize the framework, and the target frame is the middle time image, and the reference frame is the front and rear frames I ss∈{t-1,t+1}, minimizing photometric error to solve the occlusion problem, wherein represent images reconstructed from {I t-1 ,I t+1}, respectively,

[0032]

[0033] S212, calculate the automatic mask: in the automatic driving scene, the vehicle sometimes stops moving, for example, waiting for a red light or avoiding pedestrians, or sometimes, the front vehicle and the vehicle itself keep driving at the same speed, reaching a relatively static state, both of the above two cases cannot effectively build the matching relationship between images, therefore, the automatic mask needs to be introduced to filter these objects, the automatic mask calculation formula is:

[0034]

[0035] S213, calculate the scale-consistent loss function: scale consistency is explicitly performed by minimizing the geometric loss of the target frame depth and the projected frame depth, first, the depth map D t of the target frame is obtained, t′ and then they are projected onto the target coordinates by the inverse warping method, next, the depth D t′ of the projected image I x is estimated using the depth network, and the automatic mask M in step S212 is used, the scale consistency loss calculation formula is:

[0036]

[0037] S214, calculate the total loss function of unsupervised monocular depth estimation optimization:

[0038]

[0039] Preferably, the point cloud projection formula in step S6 is:

[0040]

[0041] Where x, y, z are the point cloud coordinate system, x', y' are the image coordinate system, D is the depth value, f x ,f y are the camera intrinsic parameters.

[0042] Preferably, the coordinate transformation formula in step S7 is:

[0043] P=(KT) -1 Dp

[0044] In the formula, K is a camera intrinsic (which is a known parameter of the camera), T is a transformation matrix of the camera to the world coordinate system, D is D in the RGBD camera (i.e. the depth estimation system in the framework outputs a depth map), p is the 6-DOF optimized pose obtained by the SLAM system in the framework, and P is the point cloud coordinates of the mapping part in the framework. After calculating P of each frame, the global point cloud map required finally can be obtained by directly accumulating.

[0045] Compared with the prior art, the present application has the beneficial effects that:

[0046] 1. The present application does not consider the scale problem in most existing self-supervised based methods, and the output depth estimation result has inconsistent scale between frames. The present application is designed to solve this problem to realize a scale consistent depth estimation method.

[0047] 2. The present application innovatively proposes a mapping framework, which can obtain the current real-time dense reconstruction result and the global point cloud map under the condition of only one RGB camera input. In the prior art, the depth estimation method is not used for real-time dense reconstruction and establishment of the global point cloud map. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A dense reconstruction method based on self-supervised monocular depth estimation is provided for the present application.

[0049] Figure 2 A depth estimation network structure diagram is provided for the present application.

[0050] Figure 3 A trajectory result diagram on KITTI odometry seq.9 during pose evaluation is provided for the present application.

[0051] Figure 4 A trajectory result diagram on KITTI odometry seq.10 during pose evaluation is provided for the present application.

[0052] Figure 5 An evaluation experimental result diagram of different distance segments in the depth estimation network of the present application is provided.

[0053] Figure 6 A point cloud diagram established by the pseudo RGBD on sequence 09 is provided for the present application.

[0054] Figure 7 A globally consistent dense reconstruction diagram of the pseudo RGBD camera of the present application is provided. DETAILED DESCRIPTION

[0055] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0056] The present application provides a dense reconstruction method based on self-supervised monocular depth estimation, comprising the following steps:

[0057] S1, preparing an RGB image and depth data, and the RGB image and the depth data correspond to each other; that is, each RGB image corresponds to the depth data at the current time, and the depth data is collected by a LiDAR sensor;

[0058] S2, inputting the RGB image into a depth estimation network for training and calculating a loss function;

[0059] S3, network optimization, adding the loss calculated in step S2 and optimizing the network by back propagation until the error on the test data set no longer decreases;

[0060] S4, inputting the RGB image into the model optimized in step S3 to output the depth map of the current frame;

[0061] S5, inputting the RGB image and the depth map into a SLAM system to output a six-degree-of-freedom optimized pose;

[0062] S6, converting the RGB image and the depth map into a point cloud in the camera coordinate system through a point cloud projection formula;

[0063] S7, converting the six-degree-of-freedom optimized pose into a world coordinate system through a coordinate transformation formula, and adding the point cloud of each frame to obtain a globally consistent point cloud map.

[0064] The network of the present application mainly has three parts: a depth estimation system, a SLAM system, and a mapping system, and the overall structure block diagram is as shown in Figure 1 .

[0065] In the depth estimation system, the input RGB image is input into the depth estimation network, a scale-consistent constraint is introduced, the depth map of the current frame is obtained, and the depth map is scale-consistent between frames. Then, the scale-consistent depth map and the RGB image are used to construct a pseudo RGBD camera. The depth estimation network used in this link is the currently public HR-Depth, and subsequent experiments are all carried out on this depth estimation network. However, in the framework of the present application, the depth estimation network can be replaced by any self-supervised monocular depth estimation network.

[0066] In the SLAM system, the pseudo RGBD camera is input into the SLAM system to obtain the optimized pose of 6 degrees of freedom. In the experiment of the present application, the SLAM system used is the RGBD version in ORB-SLAM3, and subsequent experiments are all carried out on this SLAM system. However, in the framework of the present application, the SLAM system can be replaced by any RGBD-SLAM system.

[0067] In the mapping system, the pseudo RGBD camera constructed in the previous step is combined with the optimized pose of 6 degrees of freedom generated by the SLAM system to establish the point cloud map of the current frame and the globally consistent point cloud map. Since the pseudo RGBD camera constructed in the previous step is a pseudo RGBD camera with consistent scale between frames, the point cloud map established in the following step is also a point cloud map with consistent scale.

[0068] Specifically, the specific steps of the depth estimation network training phase in step S2 are as follows:

[0069] S21, the RGB image first enters the convolution layer and the BN layer for processing; the BN layer can normalize the input data to prevent the occurrence of gradient disappearance or explosion phenomenon, and can speed up the training speed;

[0070] S22, then enter the ReLU activation function and the maximum pooling layer, the maximum pooling layer compresses the extracted features and simplifies the network complexity;

[0071] S23, then enter Layer1, the Layer1 is composed of two residual blocks, and the activation function used in the residual block is ELU activation function;

[0072] S24, then enter Layer2, Layer3, Layer4, the Layer2, Layer3, Layer4 and Layer1 are the same structure, across different levels, using a convolution kernel with larger step to replace the downsampling process, the size of the feature map is reduced by a factor of, and the encoder network completes the task;

[0073] S25, then the image features output by the encoder in step S24 are sent into the decoder for integration and analysis, first enter an Upconv structure, fuse the feature maps of the same size, and then perform convolution operation and upsampling operation;

[0074] S26, repeat the Upconv process of step S25 for 3 times, and the input of each Upconv is the output of the previous layer network and the feature map of the same scale in the encoder network;

[0075] S27, the output end of each Upconv uses a Sigmoid function, and the output result δ of the last Sigmoid function is converted into a depth D by using the following formula, and a and b are selected to constrain D between 0.1 and 100 units:

[0076]

[0077] The entire process of the depth estimation network is completed, and the depth estimation network structure is as shown in Figure 2 .

[0078] Specifically, the specific steps of calculating the loss function in the step S2 are as follows:

[0079] S28, the depth D obtained in the step S27 is used to calculate the photometric error loss function and the edge smoothing regular loss function;

[0080] S29, the photometric error loss function is calculated: the reconstructed image I is obtained by using the inverse warping transformation , and the similarity between the reconstructed image I and the target image I t is calculated to measure the accuracy of the visual quantity estimated by the network:

[0081]

[0082] Wherein, α is a hyperparameter obtained by cross-validation method, and is set to 0.85, and SSIM is a structural similarity measurement index calculation algorithm, and the calculation formula is

[0083]

[0084] S210, the edge smoothing regular loss function is calculated: using only the photometric error loss function will produce ambiguity in some special areas, such as textureless areas or areas with repeated textures, and these areas will have a large number of repeated pixels and repeated features, therefore, in order to reduce the influence of these areas on the final result, the edge smoothing regular loss function is introduced in the unsupervised depth estimation:

[0085]

[0086] Wherein is the partial derivative in x and y directions respectively, d t is the depth of the target frame, and the regular term makes the image gradient not obvious, so that the predicted depth value should be consistent in x and y directions as much as possible;

[0087] S211, calculate the minimum frame loss before and after: in unsupervised monocular depth estimation, the problem of the same object being occluded in different frames may occur, and if the pixels are matched incorrectly once occlusion occurs, it will bring higher photometric error, so it is necessary to optimize the depth estimation network with multiple frames, assuming that three consecutive frame images are used to optimize the framework, and the target frame is the middle time image, and the reference frame is the front and rear frames of the image s , s e {t-1, t+1}, minimize the photometric error to solve the occlusion problem, where represents the images reconstructed from {I t-1 ,I t+1}, respectively,

[0088]

[0089] S212, calculate the automatic mask: in the automatic driving scene, the vehicle sometimes stops moving, such as waiting for a red light or avoiding pedestrians, or sometimes the front vehicle and the vehicle keep the same speed to the same speed, reaching a kind of relative static state, the above two cases cannot effectively build the matching relationship between images, therefore, it is necessary to introduce an automatic mask to filter these objects, the automatic mask calculation formula is:

[0090]

[0091] S213, calculate the scale consistent loss function: by minimizing the geometric loss of the target frame depth and the projected frame depth, the scale consistency is explicitly performed, first get the depth map D t of the target frame, and project them onto the target coordinates by the inverse warping method, next, use the depth network to estimate the depth D t′ of the projected image I t′ , and use the automatic mask M in step S212, the scale consistency loss calculation formula is:

[0092] L sc = M log(1+|D t’ -D t |);

[0093] S214, calculate the total loss function of unsupervised monocular depth estimation optimization:

[0094]

[0095] Specifically, the point cloud projection formula in step S6 is:

[0096]

[0097] Where x, y, z are the point cloud coordinate system, x', y' are the image coordinate system, D is the depth value, f x ,fy is the camera intrinsic.

[0098] Specifically, the coordinate transformation formula in the step S7 is:

[0099] P=(KT) -1 Dp

[0100] In the formula, K is the camera intrinsic (is the known parameter of the camera), T is the transformation matrix of the camera to the world coordinate system, D is the D in the RGBD camera (that is, the depth estimation system in the frame comes out the depth map), p is the 6 degree of freedom optimized pose obtained by the SLAM system in the frame, and P is the point cloud coordinate of the mapping part in the frame. After calculating P of each frame, the global point cloud map required finally can be obtained by directly accumulating.

[0101] Experimental results:

[0102] The network of the depth estimation system part of the application is mainly trained on the KITTI dataset, which is often used as an evaluation benchmark in the field of depth estimation. The original image size in KITTI is 1242x375 pixels, however, in the experiment of the application, three different resolutions of 640x192 pixels, 1024x320 pixels and 1280x384 pixels are set to prove the effectiveness of the proposed method. In addition, the data division of Eigen et al. is used in the application, and the static frames are removed. Finally, 39810 pictures are used for training, 4424 for verification, and 697 for evaluation. In addition, similar to most experimental settings of depth estimation, in the setting of the application, the intrinsic of all images is the same, the principal point of the camera is set to the center of the image, and the focal length is the average of all focal lengths in KITTI. For stereo training, the transformation between the two stereo frames is set to a fixed length of pure horizontal translation. And for the true value evaluation, the respective projection depth map is generated by the original LiDAR scan. For pose estimation, the model of the application is trained on the KITTI odometry dataset sequences 00-08, and tested on sequences 09-10.

[0103] Experiment 1, pose evaluation

[0104] The experiment of this section aims to evaluate the pose results generated by different methods (directly generate pose / depth estimation combined with SLAM system to generate pose).

[0105] For the pose evaluation metrics, the relative translation error (Rel tr), the relative rotation error (Rel rot), and the root mean square error (RMSE) of the trajectory are adopted. To align the trajectories obtained using different algorithms with the ground truth trajectory, the EVO toolbox is used to recover the scale. Then, the official development package of the KITTI odometry dataset is used to calculate the relative rotation and translation errors for each trajectory with a length of (100,..., 800) meters. The measurement unit of the error is percentage (for translation) and degrees per meter (for rotation).

[0106] Quantitative results: From the results in the following table, it can be seen that the method of the present application (HR-Depth, the results of the method are represented by HR-Depth in the experimental table part) is superior to most of the other methods mentioned in the table in terms of RMSE and Rel tr of scale consistency loss. Although dvso, d3vo and zou are slightly better than our method, the present application only uses monocular images for training, dvso, d3vo uses stereo information, and zou uses a two-layer convolutional LSTM module, which requires more frames for training than the present application. The results of sequence 09 show that the scale-consistent HR-Depth is much better than other methods (the last row), and improves the performance of the initial version of HR-Depth (the second last row). However, the performance of the method of the present application in Rel rot is weak, which may be due to the large uncertainty of depth estimation at a long distance, and the depth estimation result is not as accurate as that at a short distance, thereby affecting the final result of the method. (The results in the following table are all the smaller the better, and the experimental results (Pseudo-RGBD SLAM) in the following part represent the pose results obtained by combining the depth estimation method similar to the present application with a SLAM system; the experimental results in the upper part represent the results of methods different from the method proposed in the present application).

[0107]

[0108] Qualitative results: In addition to the quantitative results, the trajectories under different methods are also plotted. As Figure 3 and Figure 4 the pseudo-RGBD method of the present application can best align with the real trajectory. In addition, only the method (HR-Depth) of the present application can successfully perform loop closure, while other methods cannot perform loop closure. Among them, groundtruth refers to the trajectory true value.

[0109] Experiment two, evaluation of different distance segments

[0110] The experiments in this section aim to evaluate the depth estimation results of the depth estimation network in the depth estimation part of the framework.

[0111] The present application compares the indicators of the same depth estimation method (all experiments in this section are carried out on the published self-supervised monocular depth estimation method HR-Depth, and this method is the depth estimation method used in all experiments in the present application, and is also the depth estimation method used in the depth estimation system in the framework proposed in the present application) under different maximum depth settings (from 30 meters to 80 meters), such as Figure 5 The results show that as the maximum depth increases, the depth estimation becomes worse, indicating that the depth estimation results of nearby points are better than those of more distant points.

[0112] There are five evaluation indicators mainly used for depth estimation tasks, which are: average absolute error (AbsRel), mean square error (Sq Rel), root mean square error (RMSE), root mean square logarithmic error (RMSE log) and relative accuracy (δi, i∈1.25,1.25 2 ,1.25 3 ). The calculation methods are as follows:

[0113]

[0114] Experiment three, point cloud mapping

[0115] In order to prove that the pseudo RGBD constructed by the present application is feasible and has good effect in the automatic driving scene, the depth estimation result of the present application and the 6 degree of freedom optimized pose are used to establish a real-time point cloud map and a globally consistent point cloud map (corresponding to the mapping part in the framework). On the other hand, it can also verify the accuracy of our method, that is, if the depth estimation result is unreliable, it is impossible to generate a reasonable point cloud map.

[0116] As shown in Figure 6 , it is a point cloud map established by the pseudo RGBD of the present application on sequence 09. The image in the lower left corner is an RGB image, and the depth estimation visualization result of the current frame is in the lower right corner. The lighter the color, the smaller the depth value, and the darker the color, the larger the depth value.

[0117] As shown in Figure 7 , it is a globally consistent dense reconstruction on KITTI Odometry dataset seq 07 by the pseudo RGBD camera of the present application. The figure shows that the pseudo RGBD camera of the present application can perform dense reconstruction on large outdoor scenes.

[0118] Compared to monocular RGB cameras, RGBD cameras are typically more expensive and generally unsuitable for outdoor scenes, with a limited maximum depth measurement capability. However, depth estimation techniques can transform an RGB camera into a pseudo-RGBD camera, allowing for depth estimation applications in outdoor scenes with a maximum depth of 80 meters, far exceeding the capabilities of RGBD cameras. On the other hand, monocular cameras cannot rely on a single image to obtain the relative distance between objects and the camera, meaning depth cannot be determined from a single image. To estimate depth, monocular SLAM systems rely on in-motion triangulation, requiring at least two frames to resolve camera motion and estimate pixel positions for initialization. Generally, the homography matrix H and the essential matrix F are calculated using the following formulas:

[0119]

[0120] p2 and p1 are the matches between two frames.

[0121] Furthermore, monocular SLAM initialization requires camera translation; otherwise, initialization will fail. However, by introducing depth information, it is unnecessary to calculate the depth of map points using two adjacent frames, nor is camera translation required, making the initialization process much easier. The introduction of depth information facilitates motion estimation for monocular RGB cameras. Therefore, this invention transforms an RGB camera into a pseudo-RGBD camera using a depth estimation method, where D comes from the depth estimation result of the RGB image. This pseudo-RGBD camera is used as input to the RGBD-SLAM system to obtain camera pose and dense monocular reconstruction. However, even with depth information, neither the pseudo-RGBD SLAM nor monocular SLAM of this invention can recover absolute scale.

[0122] On the other hand, scale ambiguity is one of the most significant problems in monocular depth estimation. Without any constraints on scale, it is difficult to construct point cloud maps in large scenes. To extend single-frame depth estimation methods to multi-frame video sequences, this invention proposes a scale-consistent loss, through which scale-consistent depth estimation results can be obtained from the video sequence. Scale consistency is explicitly performed by minimizing the geometric loss between the target frame depth and the projected frame depth in the depth estimation system. First, the depth map D of the target frame is obtained. t They are then projected onto the target coordinates using an anti-warping method. Next, a deep network can be used to estimate the projected image I. t′ Depth D t′ and using scale consistency loss

[0123] L sc =Mlog(1+|D t′ -D t |)

[0124] to constrain the scale of adjacent frames. Similar to Monodepth2 mining, the present application also uses an automatic mask M to filter static pixels. In this way, the scale of adjacent frames will remain consistent, and this feature can be maintained throughout the sequence.

[0125] In summary, the final training loss with scale consistency constraint is:

[0126] L final =L depth +L sc

[0127] where L depth is the loss function of self-supervised monocular depth estimation. By introducing this loss function, the depth map generated by the depth estimation system part in the present application is a depth map containing inter-frame scale consistency.

[0128] The SLAM system used in the present application is the public ORB-SLAM3, which uses the RGBD mode of the system, which can output the optimized pose of 6 degrees of freedom under the input of the RGBD camera.

[0129] Although the present application is described herein with reference to particular embodiments, it should be understood that these examples are merely illustrative of the principles and applications of the present application. It should therefore be understood that numerous modifications can be made to the illustrative embodiments, and that other arrangements can be devised without departing from the spirit and scope of the present application as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways other than the original claims describe. It should also be understood that features described in connection with individual embodiments can be used in other described embodiments.

Claims

1. A method of dense reconstruction based on self-supervised monocular depth estimation, characterized in that, The method comprises the following steps: S1, preparing an RGB image and depth data, and the RGB image and the depth data correspond to each other; S2, inputting the RGB image into a depth estimation network to perform training and calculating a loss function; S3, network optimization, adding the loss calculated in step S2 and using back propagation to optimize the network until the error on a test data set no longer decreases; S4, inputting the RGB image into the model optimized in step S3 to output a depth map of the current frame; S5, inputting the RGB image and the depth map into a SLAM system to output an optimized pose of six degrees of freedom; S6, converting the RGB image and the depth map into point clouds in a camera coordinate system through a point cloud projection formula; S7, converting the optimized pose of six degrees of freedom into a world coordinate system through a coordinate transformation formula, and adding point clouds of each frame to obtain a globally consistent point cloud map; The specific steps for calculating the loss function in step S2 are as follows: S28, calculating a photometric error loss function and an edge smoothing regular loss function using the depth D obtained in step S27; S29, compute photometric error loss function: in the reconstructed image I Afterwards, the accuracy of the visual quantity estimated by the network is measured by computing the similarity between the reconstructed image I and the target image I t S30, compute photometric error loss function: in the reconstructed image I Wherein, alpha is a hyperparameter obtained by cross-validation method, set to 0.85, and SSIM is a structural similarity measurement index calculation algorithm, and the calculation formula is S210, calculating an edge smoothing regular loss function: using only the photometric error loss function will produce ambiguity in some special areas, such as textureless areas or areas with repeated textures, which will have a large number of repeated pixels and repeated features, therefore, in order to reduce the influence of these areas on the final result, an edge smoothing regular loss function is introduced in unsupervised depth estimation: wherein are partial differentials in the x, y directions, respectively, d t is the depth of the target frame, the regularization term makes the predicted depth value as consistent as possible in the x and y directions when the image gradient is not obvious; S211、Computing the minimized front and back frame loss: In unsupervised monocular depth estimation, the problem of the same object being occluded in different frames may occur. If the pixels are incorrectly matched once occlusion occurs, a higher photometric error will be caused. Therefore, a multi-frame optimization depth estimation network is needed. Assuming that a three-frame image optimization framework is used, let the target frame be the middle time image, and the reference frame be the front and back frames of the image, I s , s e {t-1, t+1}, the photometric error is minimized to solve the occlusion problem, where represents the images reconstructed from {I t-1 ,I t+1}, respectively, S212, calculating an automatic mask, and the automatic mask calculation formula is: S213、Calculate scale-consistent loss function: scale consistency is explicitly enforced by minimizing a geometric loss between the depth of the target frame and the depth of the projected frame, first obtaining the depth map D of the target frame t and projecting them onto the target coordinates by the inverse warping method, next, using a depth network to estimate the depth D t′ of the projected image I t′ , and using the automatic mask M in step S212, the scale-consistent loss calculation formula is: L sc = M log(l + |D t ,-D t |); S214, calculating the total loss function of unsupervised monocular depth estimation optimization:

2. The method of claim 1, wherein, The specific steps of the depth estimation network training stage in step S2 are as follows: S21, the RGB image first enters a convolution layer and a BN layer for processing; S22, then entering a ReLU activation function and a maximum pooling layer, the maximum pooling layer compresses the extracted features and simplifies the network complexity; S23, then entering Layer1, the Layer1 is composed of two residual blocks, and the activation function used in the residual block is an ELU activation function; S24, then entering Layer2, Layer3 and Layer4, the Layer2, Layer3 and Layer4 have the same structure as Layer1, and the convolution kernel with an increased step is used for convolution operation to replace the downsampling process, the size of the feature map is reduced by a factor of two, and the encoder network completes the task; S25, then sending the image features output by the encoder in step S24 into a decoder for integration and analysis, first entering an Upconv structure, fusing the feature maps of the same size, and then performing convolution operation and upsampling operation; S26, repeating the Upconv process in step S25 for 3 times, and the input of each Upconv is the output of the previous layer network and the feature map of the same scale in the encoder network; S27, the output end of each Upconv uses a Sigmoid function, and the output result δ of the last Sigmoid function is converted into a depth D using the following formula, and a and b are selected to constrain D between 0.1 and 100 units:

3. The method of claim 2, wherein, The point cloud projection formula in the step S6 is: where x, y, z are the point cloud coordinate system, x', y' are the image coordinate system, D is the depth value, f x ,f y is the camera intrinsic parameter.

4. The method of claim 3, wherein, The coordinate transformation formula in the step S7 is: P = (KT) -1 Dp In the formula, K is a camera intrinsic parameter, T is a transformation matrix of a camera to a world coordinate system, D is a D in an RGBD camera, p is an optimized pose of 6 degrees of freedom obtained by a SLAM system in a frame, P is a point cloud coordinate of a mapping part in the frame, and P of each frame is calculated, and then the global point cloud map required finally can be obtained by directly accumulating.

Images