A depth completion method based on images and extremely sparse 3D points

Abstract

The present invention discloses a depth completion method based on images and extremely sparse three-dimensional points. The present invention uses deep learning and traditional optimization methods, respectively using different attributes of the input three-dimensional points to obtain correct depth estimates when the number of input sparse points is small. The present invention utilizes a set of depth basis to decouple the estimation of relative depth and absolute depth. The method is improved in terms of the robustness to the noise of the input three-dimensional points, and relatively good depth can be obtained in a system with high noise, so as to be applied to applications such as virtual reality and augmented reality.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and computer graphics, and in particular, to a dense localization and mapping method based on voxel neural implicit surfaces. Background Art

[0002] Dense depth is a kind of information for three-dimensional space measurement and is widely used in many fields such as robotics, autonomous driving, security monitoring, architectural planning, environmental monitoring, etc. For example, it can help robotics better identify objects in the environment, thereby helping robots to perform tasks such as grasping and obstacle avoidance; it can help autonomous driving systems better locate vehicles, thereby improving driving safety; it can help security monitoring systems better identify the surrounding environment, thereby improving security. Integrating dense depth with traditional vision can easily solve problems that are difficult to solve by traditional vision, such as efficient dense three-dimensional reconstruction, and occlusion and collision detection in AR and VR.

[0003] With the development of technology, many depth sensors have emerged that can directly obtain dense depth information, such as devices like Kinect and RealSense. In addition to these sensors that can directly obtain dense depth information, there are also sensors such as lidar. The depth information directly obtained by these sensors is usually semi-dense or sparse depth information. At this time, depth completion algorithms can be used to complete the semi-dense or sparse depth into dense depth. Most of the existing depth completion algorithms also focus on dealing with this application scenario. The number of semi-dense / sparse depths obtained by sensors is often greater than 1% of the total number of image pixels, and there are characteristics such as high depth accuracy and distribution regularity. When the number of points is small, especially when there is noise in sparse points, the accuracy is actually lower than the prediction result accuracy without inputting sparse points. This results in relatively high requirements for sensors and computing resources, thus limiting the application scenarios.

[0004] There is also some depth completion work based on sparse point clouds generated by SLAM or SFM. This method can completely get rid of the dependence on depth sensors and is thus an important direction of technological iteration. Compared with the semi-dense / sparse depth obtained by using sensors, the sparse points generated by SLAM or SFM are usually sparser. For example, there are only 100 - 200 points, far less than 1% of the total number of image pixels, and there are problems such as low accuracy and high noise. Existing depth completion systems are not robust to noise and are easily interfered by noise, resulting in poor final depth completion accuracy and even difficult convergence in network training. In fact, the sparse point accuracy of the SLAM system cannot be compared with that of sensors, and there will inevitably be a lot of noise. The present invention uses a multi-base depth prediction network to solve this problem. Summary of the Invention

[0005] To solve the problems in the prior art, a depth completion method based on images and extremely sparse 3D points. The method of the present invention includes two modules: network prediction and a depth solver with reliability verification. The system accepts RGB images and sparse depth points. A neural network is used to predict a set of depth bases and a confidence map. A traditional optimization method is used to solve a set of weights by using the input sparse points to combine the depth bases to obtain the final predicted depth. And the effectiveness of the predicted depth is judged.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention first provides a depth completion method based on images and extremely sparse 3D points, which includes the following steps:

[0008] Step 1: Input the RGB image and the coordinates and depths of the corresponding sparse points; use a depth completion neural network based on multi-basis fitting to predict a set of depth bases with the same resolution as the input RGB image and a confidence map. The final dense depth is obtained by weighted summation of each depth base, and the confidence map represents the confidence of the final depth;

[0009] Step 2: For each sparse point j input, the image coordinates p j corresponding to the sparse point and the observed depth value S j. are known; a set of depth values at the corresponding coordinate positions of each depth base are extracted using the coordinates of the sparse point; the depth values from different bases are weighted and summed with the weights of the corresponding bases to obtain the predicted depth value D j corresponding to the sparse point; by optimizing the weights of each base to minimize the difference between the predicted depth values and the observed depth values of all sparse points, this constitutes an optimization problem;

[0010] Step 3: Judge whether the optimization problem solved in Step 2 is an ill-posed problem; if so, directly output that the result is unreliable; otherwise, compare the obtained weights with the prior weights and judge. If the deviation from the prior weights exceeds the set threshold, output that the result is unreliable; if the deviation relative to the prior weights is within the allowable range, use the obtained weights to linearly combine the depth base set of the entire image to obtain the final depth result.

[0011] As a preferred solution of the present invention, in Step 1, a depth completion neural network is used to predict a set of depth bases with the same resolution as the input RGB image and a confidence map, specifically:

[0012] The depth completion neural network based on multi-basis fitting uses U-Net for feature encoding and decoding; the encoding network F eObtain a high-dimensional feature representation of 1 / 32 of the input resolution, add sinusoidal position encoding, and then use the decoding network F d Output the depth basis set and the confidence map with the same resolution as the input in two branches; different from the traditional U-Net network that only outputs the output of the last layer network as the final depth prediction result, the depth completion neural network based on multi-basis fitting outputs the outputs of the last several layers of the network as the depth prediction result.

[0013] As a preferred solution of the present invention, in step 2, the difference between the predicted depth values of all sparse points and the observed depth values is minimized by optimizing the weights of each basis, which constitutes an optimization problem, specifically:

[0014] For each sparse point j, its image coordinate is located at p j , the observed depth is S j , and its predicted values in different depth bases are B i (p j ), where B i is the predicted i-th depth basis; the predicted depth of this sparse point is obtained by linearly summing the weights of each basis where w i is the weight of the i-th basis, then the observation error of this sparse point is Then the observation errors of k sparse points can be statistically calculated as It can be regarded as an optimization problem where W is the vector composed of (w1, w2,..., w n ), and the weights of each basis are obtained by solving this optimization problem to minimize the total observation error.

[0015] Furthermore, when constructing the optimization problem, the determination of depth confidence can be added. If the confidence of the confidence map at point p j is less than a certain threshold, the observation of this point is not added to the construction of the optimization problem; otherwise, it is added to the construction of the optimization problem. The step of using confidence to judge is optional.

[0016] As a preferred solution of the present invention, in step 3, it is judged whether the optimization problem solved in step 2 is an ill-posed problem, specifically:

[0017] The optimization problem in step 2 can be written in matrix form, and by setting the sum of the observation errors to 0, the analytical solution of this problem can be obtained, which is represented by the matrix as W = (B T B) -1 B T S, where W is the n-dimensional vector composed of the weights of each basis to be solved, B is the k*n-dimensional matrix composed of B i (p j ), and S is composed of S jConstruct a k-dimensional vector and calculate the condition number of the optimization problem where λ max and λ min are the maximum eigenvalue and the minimum eigenvalue of the matrix (B T B), respectively. If the condition number cond is greater than a certain threshold (for example, the threshold is set to 50), then the problem is considered an ill-posed problem; otherwise, the problem is not an ill-posed problem.

[0018] Compared with the prior art, the advantages of the present invention are as follows:

[0019] 1) The present invention adds the prediction of confidence on the basis of the depth completion neural network based on multi-basis fitting, so that the network can better handle the situation of boundary depth modulation and obtain higher-precision depth. In addition, when adding the optimization of depth basis weights to problems such as multi-frame depth consistency optimization, adding confidence information can improve the stability of optimization and improve the optimization accuracy.

[0020] 2) In the problem of solving multi-basis weights, the present invention adds a judgment on whether the problem to be solved is well-posed, which can improve the robustness of the network to sparse point noise. Adding the judgment of whether it is well-posed can improve the depth prediction accuracy of the network in the case of large sparse point depth noise, and can also solve the problem that it is difficult for the network to converge when training on a high-noise data set.

[0021] 3) In the problem of solving multi-basis weights, adding a judgment on whether the problem to be solved is well-posed can improve the balance of each depth basis. Description of the Drawings

[0022] Figure 1 is a schematic diagram of the method of the present invention;

[0023] Figure 2 is a display diagram of the depth prediction effect of the present invention. Detailed Embodiments

[0024] The present invention will be described in detail below with reference to the accompanying drawings of the specification. The technical features of each embodiment of the present invention can be combined accordingly without conflict.

[0025] Referring to Figure 1 , the present invention includes two modules: a neural network prediction and a depth solver with reliability verification. The system receives RGB images and sparse depth points. A neural network is used to predict a set of depth bases and a confidence map. A set of weights is solved by using the input sparse points through a traditional optimization method to combine the depth bases to obtain the final predicted depth. And the effectiveness of the predicted depth is judged. The following is a detailed description of the present invention. It includes the following steps:

[0026] Step 1: Input the RGB image, as well as the coordinates and depths of the corresponding sparse points; use a depth completion neural network based on deep basis fitting (Qu C, Nguyen T, Taylor C. Depth completion via deep basis fitting[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2020:71-80.) to predict a set of depth bases with the same resolution as the input RGB image and a confidence map. The final dense depth is obtained by weighted summation of each depth basis, and the confidence map represents the confidence of the final depth; in this embodiment, the depth completion neural network based on deep basis fitting uses U-Net for feature encoding and decoding. Use the encoding network F e to obtain a high-dimensional feature representation with 1 / 32 of the input resolution, add sine position encoding, and then use the decoding network F d to output a depth basis set and a confidence map with the same resolution as the input in two branches. The loss function used for network training is as follows:

[0027] L = w d L dep + w c L conf + w r L reg

[0028] where L is the total loss term, L dep is the depth consistency term, L conf is the confidence loss term, L reg is the conditional book regularization loss term, w d 、w c 、w r are the weights of the corresponding loss terms, which are preset according to empirical values;

[0029] The specific definition of L dep is as follows:

[0030] L dep = ||D gt - D pred ||1 + ||ΔD gt - ΔD pred ||1

[0031] where D gt is the true value of the depth, D pred is the predicted final depth, ΔD gt is the gradient map of the true depth value, ΔD pred is the gradient map of the predicted depth, ||·||1 is the L1 loss;

[0032] L conf is specifically defined as follows:

[0033]

[0034] where C is the predicted confidence map;

[0035] L reg is specifically defined as follows:

[0036] L reg = log(λ max (B T B)) - log(λ min (B T B))

[0037] B is a k*n-dimensional matrix composed of B i (p j ), λ max and λ min are respectively the maximum eigenvalue and the minimum eigenvalue of the matrix (B T B).

[0038] Step 2: For each sparse point j in the input, given the image coordinates p j corresponding to the sparse point and the observed depth value S j .; Extract a set of depth values at the coordinate positions corresponding to each depth basis using the coordinates of the sparse point; Weighted sum the depth values from different bases with the weights of the corresponding bases to obtain the predicted depth value D j corresponding to the sparse point; Minimize the difference between the predicted depth values and the observed depth values of all sparse points by optimizing the weights of each basis, which constitutes an optimization problem.

[0039] For each sparse point j, its image coordinates are located at p j , the observed depth is S j , and its predicted values in different depth bases are B i (p j ), where B i is the predicted i-th depth basis; Combine the linear sum of the weights of each basis to obtain the predicted depth of the sparse point where w i is the weight of the i-th basis, then the observation error of the sparse point is Then the observation errors of k sparse points can be statistically calculated as It can be regarded as an optimization problem where W is a vector composed of (w1, w2,..., w n ), and the weights of each basis are obtained by solving this optimization problem to minimize the total observation error.

[0040] Further, when constructing the optimization problem, the determination of the depth confidence can be added. If the confidence of the confidence map at point p j is less than a certain threshold, the observation of this point will not be added to the construction of the optimization problem; otherwise, it will be added to the construction of the optimization problem. Using confidence to make this judgment is optional.

[0041] Step 3: Determine whether the optimization problem solved in Step 2 is an ill-posed problem; if so, directly output that the result is unreliable; otherwise, compare the obtained weights with the prior weights. If the deviation from the prior weights exceeds the set threshold, output that the result is unreliable; if the deviation relative to the prior weights is within the allowable range, use the obtained weights to perform a linear combination on the depth basis set of the entire map to obtain the final depth result.

[0042] First, the optimization problem in Step 2 can be written in matrix form. Let the sum of the observation errors be 0, then the analytical solution of this problem can be obtained, which is represented by the matrix as W = (B T B) -1 B T S, where W is the n-dimensional vector composed of the weights of each basis to be solved, B is the k*n-dimensional matrix composed of B i (p j ) and S is the k j dimensional vector composed of S where λ max and λ min are respectively the maximum eigenvalue and the minimum eigenvalue of the matrix (B T B). If the condition number cond is greater than 50, it is considered that this problem is an ill-posed problem; otherwise, this problem is not an ill-posed problem.

[0043] Secondly, determine whether the obtained weights are compared with the prior weights set for each depth basis during the network training process. If the deviation from the prior weights exceeds the set threshold, output that the result is unreliable. For reliable results, use the weights obtained in the previous step to perform a linear combination on the depth basis set of the entire map to obtain the final depth result.

[0044] Embodiment

[0045] Figure 1This is the network design diagram of the present invention, which includes a U-Net network for predicting multiple depth bases and a conditional number aware depth solver for solving the condition number and the weight of each basis. The present invention has been experimentally compared on public datasets such as ScanNet and Euroc. The present method has shown higher accuracy and robustness. Compared with the previous method (Qu C, Nguyen T, Taylor C. Depth completion via deep basis

[0046] fitting[C] / / Proceedings of the IEEE / CVF Winter Conference onApplications of Computer Vision.2020:71-80.), the network obtained by the present invention is more suitable for SLAM systems, which are extremely sparse points and have large sparse point noise. The network obtained by the previous method is prone to deep optimization divergence when applied to the SLAM system. Figure 2 In the figure, (a) is the input image, (b) is the final dense depth predicted by this method, (c), (d), (e), and (f) are the four depth bases obtained by the present invention, and (g), (h), (i), and (j) are the depth bases obtained by the previous method. It can be clearly seen that the information contained in the depth bases obtained by the present invention is more balanced, while the information of the depth bases obtained by the previous method is basically only contained in the depth base (g), which can easily lead to unstable optimization.

[0047] The present invention can be used in applications that combine various types of sparse depths to complete sparse depths into dense depths. For example, by integrating the present invention with a SLAM system, the sparse or semi-dense depth generated by the SLAM system can be completed into a dense depth, thereby further generating a dense model and completing real-time three-dimensional reconstruction. Similarly, the present invention can also restore the sparse point cloud generated by the SFM method into a dense model. In addition, dense depth can also be directly used in various augmented reality applications, such as directly using depth information to realize the display of occlusion effects of AR / VR applications, as well as collision detection. The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there can be many variations. All variations that can be directly derived or associated with the contents disclosed by a person of ordinary skill in the art should be considered to be within the scope of protection of the present invention.

Claims

1. A depth completion method based on images and extremely sparse 3D points, characterized in that, It includes the following steps: Step 1: Input an RGB image, as well as the coordinates and depths of the corresponding sparse points; use a depth completion neural network based on multi-basis fitting to predict a set of depth bases with the same resolution as the input RGB image and a confidence map. The final dense depth is obtained by weighted summation of each depth base, and the confidence map represents the confidence of the final depth; The depth completion neural network based on multi-basis fitting uses U-Net for feature encoding and decoding; the encoding network is used to obtain a high-dimensional feature representation with an input resolution of 1 / 32, add sine position encoding, and then use the decoding network to output a depth basis set and a confidence map with the same resolution as the input in two paths; the depth completion neural network based on multi-basis fitting outputs the output of the last several layers of the network as the depth prediction result; Step 2: For each sparse point j in the input, the image coordinates p corresponding to the sparse point are known j and the observed depth value S j. ; Extract a set of depth values at the coordinate positions corresponding to each depth basis using the coordinates of the sparse point; Weighted sum the depth values from different bases with the weights of the corresponding bases to obtain the predicted depth value D corresponding to the sparse point j ; Minimize the difference between the predicted depth values and the observed depth values of all sparse points by optimizing the weights of each basis, which constitutes an optimization problem In the said Step 2, the difference between the predicted depth values and the observed depth values of all sparse points is minimized by optimizing the weights of each basis, which constitutes an optimization problem, specifically: For each sparse point j, its image coordinates are located at p j , and the observed depth is S j , and its predicted values in different depth bases are , where is the i-th predicted depth base; the predicted depth of this sparse point is obtained by linearly summing the weights of each base , where is the weight of the i-th base, then the observation error of this sparse point is ; then the observation errors of k sparse points can be statistically calculated as ; it can be regarded as an optimization problem , where W is the vector formed by, and the weights of each base are obtained by solving this optimization problem to minimize the total observation error; Step 3: Determine whether the optimization problem solved in Step 2 is an ill-posed problem; if so, directly output that the result is unreliable; otherwise, compare the obtained weights with the prior weights. If the deviation from the prior weights exceeds the set threshold, output that the result is unreliable; if the deviation relative to the prior weights is within the allowable range, use the obtained weights to perform a linear combination of the depth base set of the entire image to obtain the final depth result.

2. The depth completion method based on an image and extremely sparse 3D points according to claim 1, wherein In the said Step 1, a depth completion neural network is used to predict a set of depth bases with the same resolution as the input RGB image and a confidence map, specifically: The loss function used for network training is as follows: ； where \(L\) is the total loss term, is the depth consistency term, is the confidence loss term, is the condition number regularization loss term, and and are the weights of the corresponding loss terms, which are preset according to empirical values; The specific definition is as follows: ； where is the ground truth of depth, is the final predicted depth, is the gradient map of the ground truth of depth, is the gradient map of the predicted depth, is the L1 loss; The specific definition is as follows: ； where C is the predicted confidence map.

3. The depth completion method based on images and extremely sparse three-dimensional points according to claim 1, wherein, When constructing the optimization problem, add the determination of the depth confidence. If the confidence of the confidence map at point p j is less than a certain threshold, the observation of this point is not added to the construction of the optimization problem; otherwise, it is added to the construction of the optimization problem.

4. The depth completion method based on an image and extremely sparse three-dimensional points according to claim 1, wherein In the said Step 3, determine whether the optimization problem solved in Step 2 is an ill-posed problem, specifically: The optimization problem in step 2 can be written in matrix form. By setting the sum of the observation errors to 0, the analytical solution of this problem can be obtained, which is represented in matrix form as , where W is an n-dimensional vector composed of the basis weights to be solved, B is a k*n-dimensional matrix composed of , S is a k-dimensional vector composed of S j . Calculate the condition number of the optimization problem , where are the maximum eigenvalue and the minimum eigenvalue of the matrix respectively. If the condition number is greater than a certain threshold, then this problem is considered an ill-posed problem; otherwise, this problem is not an ill-posed problem.

Images