A monocular view depth estimation method based on neural radiance field and semantic segmentation
By combining neural radiation fields and semantic segmentation, the fitting and adaptive capabilities of monocular view depth estimation are improved, the depth error caused by low-texture regions and occlusion is solved, and more accurate depth prediction is achieved.
Patent Information
- Application Number
- CN202210854710.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-18
- Publication Date
- 2025-12-09
- Estimated Expiration
- 2042-07-18
AI Technical Summary
Existing deep learning-based monocular depth estimation methods are insufficient in low-texture regions due to lack of features and depth abrupt changes caused by occlusion, and are difficult to handle depth ambiguity and depth errors caused by occlusion in indoor scenes.
We employ a method based on neural radiation fields and semantic segmentation, leveraging enhanced semantic information to improve the network's fitting and adaptive capabilities. Through depth prior prediction, depth optimization, and neural radiation field training, we generate more accurate depth maps, addressing the depth prediction problem in low-texture and edge regions.
It effectively addresses depth ambiguity and depth errors caused by occlusion in indoor scenes, improves depth prediction performance in low-texture and edge regions, simplifies the method, and enhances the accuracy of depth estimation.
Smart Images
Figure CN115393410B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of monocular depth estimation, and particularly relates to a monocular view depth estimation method based on neural radiance field and semantic segmentation information. BACKGROUND
[0002] Monocular depth estimation is a very important task in the field of three-dimensional reconstruction, and is a prerequisite for various downstream tasks of three-dimensional reconstruction. The traditional monocular depth is basically obtained by shooting with a depth camera, and the estimation problem is often calculated by using the motion structure method and the multi-view stereo method. Since such methods depend on the quality of the picture and the texture features, there are inherent limitations, and thus the development is slow. With the development of the field of deep learning, various ambiguity problems in the field of three-dimensional reconstruction have introduced new solutions. The three-dimensional reconstruction and depth estimation technology based on deep learning has developed rapidly in recent years. Monocular depth estimation is the most important part of three-dimensional reconstruction depth estimation. Because it has low hardware requirements, convenient operation and strong landing, it is an important link in the whole three-dimensional reconstruction process, and also provides strong support for subsequent virtual augmented reality technology, robot technology and automatic driving technology.
[0003] The existing monocular depth estimation method based on deep learning lacks a solution to the problem of insufficient features in low-texture areas, and cannot handle the depth ambiguity problem when it is difficult to have global information in indoor scenes. In addition, due to the complex mutual occlusion relationship between indoor objects, the conventional depth estimation method and the neural radiance field algorithm are difficult to cope with the depth mutation problem caused by occlusion, so that the depth estimation at some gaps is greatly affected by the foreground, and the error is great. SUMMARY
[0004] The application aims at providing a monocular view depth estimation method based on neural radiation field and semantic segmentation to solve the problems of the prior art. The method uses semantic segmentation or component segmentation based on neural radiation field, uses enhanced semantic information to improve the fitting and adaptive ability of the network, realizes depth estimation of multiple views in a single scene, effectively solves the depth ambiguity and depth mutation problems caused by occlusion in an indoor scene when it is difficult to have global information, makes the depth estimation at the partial gap not affected by the foreground, has small error, and greatly improves the monocular view depth, especially the prediction effect of the depth of low texture area and edge area. The method uses a semantic segmentation network and a depth prediction network to construct a three-dimensional scene or object point cloud for a semantic segmentation or component segmentation system, extracts point cloud geometric features using a learned geometric convolution kernel, predicts the edge of the point cloud using the network, and better extracts local features using edge information, so that the point cloud, especially the edge area, has better segmentation effect. The semantic information of the scene is used to improve the object differentiation of the picture, the problem of insufficient features in the low texture area is solved, and the neural radiation field is used to strengthen the learning of global information. The color map and the semantic segmentation map guide the density probability information obtained by the network to obtain more reasonable and real depth information, effectively solving the problems of insufficient features in the low texture area and the gap depth error caused by self-occlusion in the conventional monocular depth estimation, the method is simple, has good use effect, has advantages, and has wide application prospect.
[0005] The specific technical scheme for realizing the purpose of the application is: a monocular view depth estimation method based on neural radiation field and semantic segmentation, characterized in that a semantic segmentation or component segmentation method based on neural radiation field is used, enhanced semantic information is used to improve the fitting and adaptive ability of the network, low texture and self-occlusion problems are solved, and depth estimation of multiple views in a single scene is realized, specifically including the following steps:
[0006] Step 1: depth prior prediction
[0007] 1-1: input each scene picture into a three-dimensional reconstruction algorithm to obtain a sparse point cloud of the scene, and project the point cloud to each view to obtain a sparse depth map and a corresponding mask for the corresponding view.
[0008] 1-2: input the scene picture into a pre-trained depth prediction network, use the corresponding sparse depth map and scale-invariant loss function to guide the network to obtain a depth prior map corresponding to each picture, so that the network is over-fitted to the scene, that is, all pictures in the same scene share network parameters.
[0009] Step 2: depth prior optimization
[0010] 2-1: input each scene picture into the pre-trained semantic segmentation network separately, obtain the corresponding semantic segmentation grayscale picture result, splice the semantic segmentation grayscale picture and the depth prior picture obtained in step 1-2, and input them into the optimization network.
[0011] 2-2: the optimization network has an encoder-decoder structure as a whole, extracts picture information of different scales, and outputs a depth prediction picture of each picture in the scene; a scale-invariant loss function is calculated for the depth prediction picture and the sparse depth picture obtained in step 1-1, and the loss function of the depth prediction network in step 1-2 is weighted and summed to obtain a total loss function for guidance, which strengthens the semantic understanding of the scene and optimizes the depth prior picture obtained in step 1-2.
[0012] Step 3: neural radiance field training
[0013] 3-1: use the camera pose parameters of each picture to sample on the ray corresponding to the camera view angle of each pixel, and the sampling range is guided by the depth prior picture and the depth confidence picture calculated according to the depth prior picture, that is, the color picture information is converted into the information of a large number of spatial points containing the position and observation view angle of the points.
[0014] 3-2: input the coordinate information of all three-dimensional space points into a fully connected network, output the density probability value and segmentation information value corresponding to each point after a multi-layer perceptron and a residual structure, input the view angle information of the space points into the network, and output the color information value of each three-dimensional space point after an additional fully connected layer.
[0015] Step 4: body rendering output depth map
[0016] 4-1: use the body rendering formula and the output density probability information, segmentation information and color information to calculate the predicted segmentation probability distribution and predicted color value of each pixel, compare the generated segmentation information with the semantic segmentation picture information corresponding to the original picture, calculate the cross-entropy loss, compare the generated color information with the color information of the original picture, calculate the mean square error loss, and add the two functions of the cross-entropy loss and the mean square error loss by weighting, and perform back propagation to promote network training.
[0017] 4-2: use the density probability information obtained after the completion of the training and the depth information of each point when sampling to perform body rendering, obtain the final depth value of each pixel, and thus complete the generation of the depth map.
[0018] 4-3: compare the color picture obtained by body rendering with the original picture input into the neural radiance field, calculate the error as a filter, and filter and optimize the obtained depth map.
[0019] The step 1 depth prior prediction uses a scene picture and a conventional algorithm to obtain a more accurate sparse depth map as a guide, over-fits a conventional pre-trained depth network to the current scene, and all pictures share network parameters; the scale-invariant loss function refers to removing unusable data in the sparse depth map using a mask, then using a logarithmic function to scale the numerical range, then calculating the average error between pictures, adding the average error to each pixel of the prediction result, and then calculating the difference with the real reference value and summing.
[0020] The step 2 depth optimization network inputs the semantic information and depth information of the scene picture, and uses the class and edge information in the semantic information to guide the depth change in the low-texture area and edge area of the depth map; the depth optimization network uses a U-shaped encoding-decoding structure, fully utilizes the different scale information of the depth prediction map and the semantic segmentation map, and adds the original convolutional layer information at different scales when downsampling, and emphasizes the distinction between different modules in the picture.
[0021] The step 3 guidance of neural radiance field sampling using depth prior includes:
[0022] A1: First, project each depth map back to three-dimensional space to form a point, and then project it to all other views to obtain
[0023] the depth map projected by other views at each view, and take the K smallest error maps as reference to calculate
[0024] the average error, the depth value with small error is relatively accurate, and the depth value with large error is the opposite;
[0025] A2: Then sample the space points around the depth prior on the corresponding ray of each pixel, and scale the sampling range depending on the depth error map of the pixel, wherein, since the pixel depth error can only block the background, the sampling range is mainly offset to the rear.
[0026] The step 3 neural radiance field network additionally outputs semantic information unrelated to the observation view in addition to the conventional color output and density probability output, and each spatial point outputs a probability distribution containing 255 classes of semantics.
[0027] The step 4 neural radiance field network uses not only the color map for self-supervised guidance, but also the semantic segmentation information obtained by volume rendering and the previous semantic segmentation map for cross-entropy calculation, to further strengthen the influence of semantic information; the filtering refers to designing a planar bilateral filter according to the error between the generated color map and the original map, to further optimize the depth map to obtain the final result.
[0028] Compared with the prior art, the application solves the problems of insufficient features in low-texture areas and gap depth errors caused by self-occlusion in conventional monocular depth estimation, improves the fitting and adaptive ability of the network, enhances semantic information, and thus improves the prediction effect of depth, especially the depth of low-texture areas and edge areas, the method is simple, has good use effect, has advantages, and has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS
[0029] Figure 1 is a flowchart of the application;
[0030] Figure 2 is a flowchart of depth prior prediction;
[0031] Figure 3 is a flowchart of neural radiance field. DETAILED DESCRIPTION
[0032] Referring to Figure 1 , the application adopts a method of semantic segmentation or component segmentation based on neural radiance field, uses enhanced semantic information to improve the fitting and adaptive ability of the network, realizes depth estimation of multiple views in a single scene, and specifically includes the following steps:
[0033] S120-130: depth prior prediction;
[0034] S140-150: depth prior optimization;
[0035] S200-220: neural radiance field training;
[0036] S300-S320: volume rendering output depth map.
[0037] The application uses an existing pre-trained monocular depth estimation network to obtain the depth prior of a scene picture, uses a traditional three-dimensional reconstruction algorithm to obtain a scene sparse point cloud to perform overfitting of the network, inputs the scene picture into an existing pre-trained semantic segmentation network to obtain a semantic segmentation result, and inputs the semantic segmentation result and the depth prior into an advanced network to strengthen the semantic information of the depth prior, then inputs the picture information into a fully connected network to learn a neural radiance field, uses the depth prior to guide the sampling range, finally uses the semantic information output by the neural radiance field to further strengthen the semantic understanding of the picture, thereby optimizing the depth learning, using a volume rendering algorithm to output a depth map, and finally using a color map to filter and optimize the depth map.
[0038] The application will be further described in detail below in combination with the drawings and specific embodiments.
[0039] Embodiment 1
[0040] Step 1: depth prior prediction
[0041] Referring toFigure 2 The depth prior prediction specifically comprises the following steps:
[0042] S100: input the multi-view RGB image in the scene into a traditional three-dimensional reconstruction algorithm or software (the colmap software is used in this embodiment), to obtain a three-dimensional point cloud of the scene.
[0043] S110: project the obtained three-dimensional point cloud to the corresponding view angle of each picture, to obtain a corresponding sparse depth map, and output a corresponding mask to remove the depth area and the depth unreliable area.
[0044] S120: input the pictures in the scene into a pre-trained backbone monocular depth prediction network respectively, to output a corresponding depth prior map.
[0045] S130: calculate a scale-invariant loss function by using the sparse depth map obtained in S110 and the depth prior map obtained in S120, and perform back propagation to make the network overfit to the current scene.
[0046] In step 1, the pictures in the scene are input into a pre-trained depth prediction network, and the corresponding sparse depth map and scale-invariant loss function are used to guide the network, so that the network is overfitted to the scene (all pictures in the same scene share network parameters).
[0047] Step 2: depth prior optimization
[0048] Referring to Figure 2 The depth prior optimization specifically comprises the following steps:
[0049] S140-150: input the pictures in the scene into a semantic segmentation network respectively and directly perform inference prediction, without training the network, to obtain a semantic segmentation map corresponding to each picture in the scene, which includes an RGB image with specific semantics and a grayscale image that only distinguishes different categories, and only the grayscale image is taken here.
[0050] S160: the depth optimization network has a Unet structure, and the up-sampling and down-sampling are each four times. The network input is the depth prior map and the semantic segmentation map obtained in S130 (dimension splicing), and finally outputs a grayscale image, which is the optimized depth map.
[0051] S170: the scale-invariant loss function is calculated by using the sparse depth map obtained in S110 and the depth map output by the optimization network, and the loss function in S130 is weighted together to guide and perform back propagation on the depth prediction network and the optimization network.
[0052] The step 2 in the application is to input each scene picture into the available pre-trained semantic segmentation network separately to obtain the corresponding semantic segmentation grayscale picture result. The semantic segmentation grayscale picture and the depth prior picture obtained in S120 are spliced and input into the optimization network.
[0053] The optimization network in the step 2 has an encoder-decoder structure as a whole, thereby extracting picture information of different scales and outputting the depth prediction picture of each picture of the scene again. The scale-invariant loss function is calculated with the sparse depth picture, and the total loss function is obtained by weighting and summing the loss function of S130, thereby strengthening the semantic understanding of the scene and optimizing the obtained depth prior.
[0054] Step 3: neural radiance field training
[0055] Referring to Figure 3 The neural radiance field training specifically includes the following steps:
[0056] S200: Project all the depth priors of the same scene into the three-dimensional space to obtain corresponding three-dimensional space points, and project the three-dimensional space points of each picture into all other views, thereby obtaining the depth prior of itself and the depth prior projected from other views into the current view at each view. The K pictures closest to the depth prior of itself are taken, and the average of the errors of the K pictures and the depth prior of itself is taken as the confidence map of the depth at the current view.
[0057] S210: Make M rays from the center of the camera aperture of any picture in the scene, randomly select M pixels on the picture, and pass each ray through the respective pixel. The base point for sampling on the ray is set according to the depth prior in S130, and the sampling range depends on the depth confidence map corresponding to the current picture. If the pixel value (i.e., the error) of the confidence map is small, a smaller sampling range is selected around the sampling base point, and vice versa. The sampling range is set with upper and lower thresholds, and the sampling threshold is smaller within the base point and larger outside the base point, thereby realizing that most of the sampling occurs outside the base point. The same processing is performed on all pictures in the scene to obtain a large amount of spatial point information, including the position of the point and the viewing direction of the point.
[0058] S220: The network structure of the neural radiance field is a multilayer perceptron as a whole. The initial input of the network is the coordinate information of the spatial points obtained in S210, and a residual structure is additionally set at some fixed layers of the multilayer perceptron. The network input is spliced into the features again.
[0059] S230-250: After training the multiple hidden layers of the neural radiance field multi-layer perceptron, output the density probability value and semantic segmentation probability of each point. The semantic segmentation probability refers to the probability distribution of the point belonging to 255 semantic categories, and then the observation angle information of the point is spliced and input into the network, and the color value (RGB three-dimensional) of the point is output after the multi-layer perceptron.
[0060] The input data of the neural radiance field in step 3 is obtained by sampling on the ray corresponding to the camera view angle of each pixel using the camera pose and other parameters of each picture. The sampling range is guided by the aforementioned depth prior and the depth confidence map calculated according to the depth prior, that is, the color picture information is converted into a large amount of spatial point information, including the position and observation angle of the point.
[0061] The neural radiance field structure in step 3 is to input the coordinate information of all three-dimensional space points into a fully connected network, and output the density probability value and segmentation information value corresponding to each point after the multi-layer perceptron and residual structure. At the same time, the view angle information of the spatial point is input into the network, and the color information value of each three-dimensional space point is output after an additional fully connected layer.
[0062] S220: The network structure of the neural radiance field is a multi-layer perceptron as a whole. The initial input of the network is the coordinate information of the spatial point obtained in S210, and a residual structure is added to some fixed layers of the multi-layer perceptron. The network input is spliced into the features again.
[0063] Step 4: Body rendering output depth map
[0064] Referring to Figure 1 , the body rendering output depth map specifically includes the following steps:
[0065] S300-S310: For each point, the color value and semantic segmentation probability output in S240 and S250 are used to perform body rendering on the sampling points on the same ray using the density probability value obtained in S230 and the body rendering formula. The color value and semantic segmentation prediction of the picture pixel corresponding to the ray are obtained, and then the mean square error loss is calculated between the color value and the pixel RGB value of the original picture, and the cross-entropy loss function is calculated between the semantic segmentation prediction and the semantic segmentation map output in S140-150. The two loss functions of the cross-entropy loss and the mean square error loss are weighted and summed to obtain the total loss function, which guides the network learning to obtain a more reasonable density probability distribution through back propagation.
[0066] S320: error calculation is performed using the input neural radiance field sketch and the output RGB image. Unlike the loss value of the entire image calculated in S310, a confidence map is produced according to the pixel-by-pixel error of the image, and a planar bilateral filter is produced using the confidence map to further filter the predicted depth map (obtained by density probability value and sampling depth calculation) obtained by volume rendering, and the final depth map result is optimized.
[0067] The filtering operation in step 4 is to compare the color map obtained by volume rendering with the input neural radiance field sketch, calculate the error as the final filtering optimization of the depth map obtained by filtering.
[0068] The above is only a further description of the present application, and is not intended to limit the patent. Any equivalent implementation of the present application shall be included in the scope of the claims of the present patent.
Claims
1. A monocular view depth estimation method based on neural radiance fields and semantic segmentation, characterized in that The method of semantic segmentation or component segmentation based on neural radiance field is used to improve the fitting and adaptive ability of the network by using enhanced semantic information, and to realize depth estimation of multiple views in a single scene, which specifically includes the following steps: Step 1: depth prior prediction 1-1: input each scene picture into a three-dimensional reconstruction algorithm to obtain a sparse point cloud of the scene, and project the point cloud to each view to obtain a sparse depth map and a corresponding mask for each view; 1-2: input the scene picture into a pre-trained depth prediction network, and use the corresponding sparse depth map and scale-invariant loss function to guide the network to obtain a depth prior map corresponding to each picture, so that the network is over-fitted to the scene, that is, all pictures of the same scene share network parameters; Step 2: depth prior optimization 2-1: input each scene picture into a pre-trained semantic segmentation network to obtain a corresponding semantic segmentation grayscale image, and input the semantic segmentation grayscale image and the depth prior map obtained in step 1-2 into an optimization network; 2-2: the optimization network has an encoder-decoder structure, which extracts information of pictures at different scales and outputs a depth prediction map for each picture of the scene, calculates a scale-invariant loss function for the depth prediction map and the sparse depth map obtained in step 1-1, and adds the loss function of the depth prediction network in step 1-2 to obtain a total loss function for guidance, which strengthens the semantic understanding of the scene and optimizes the depth prior map obtained in step 1-2; Step 3: neural radiance field training 3-1: sample on the ray corresponding to the camera view of each pixel using the camera pose parameters of each picture, and the sampling range is guided by the depth prior map and the depth confidence map calculated according to the depth prior map, that is, the color picture information is converted into information of a large number of spatial points containing the position and observation angle of the points; 3-2: input the coordinate information of all three-dimensional space points into a fully connected network, and output the density probability value and segmentation information value corresponding to each point after a multi-layer perceptron and a residual structure, and input the view information of the space points into the network and output the color information value of each three-dimensional space point after an additional fully connected layer; Step 4: output depth map by volume rendering 4-1: calculate the predicted segmentation probability distribution and predicted color value of each pixel using the volume rendering formula and the output density probability information, segmentation information and color information, compare the generated segmentation information with the semantic segmentation information of the original picture, calculate the cross-entropy loss, compare the generated color information with the color information of the original picture, calculate the mean square error loss, and add the two functions of the cross-entropy loss and the mean square error loss by weighting, and perform back propagation to promote network training; 4-2: perform volume rendering using the trained density probability information and the depth information of each point when sampling to obtain the final depth value of each pixel, thereby completing the generation of the depth map; 4-3: compare the color map obtained by volume rendering with the original picture input into the neural radiance field to calculate the error as a filter to filter and optimize the obtained depth map.
2. The monocular view depth estimation method based on neural radiance fields and semantic segmentation according to claim 1, characterized in that The sparse depth map obtained in the step 1-1 is used as guidance, that is, a more accurate sparse depth map is obtained in advance by using a scene picture and a three-dimensional reconstruction algorithm as guidance, and a pre-trained depth network is over-fitted to the current scene, and all pictures share network parameters.
3. The monocular view depth estimation method based on neural radiance fields and semantic segmentation of claim 1, wherein The scale-invariant loss function in the step 1-2 is used for back propagation, the scale-invariant loss function is used to remove unavailable data in the sparse depth map by using a mask, then the numerical range is scaled according to a logarithmic function, the average error between pictures is calculated, and each pixel of the predicted result is added to the average error, and then the difference between the result and the true reference value is calculated and summed.
4. The monocular view depth estimation method based on neural radiance fields and semantic segmentation of claim 1, wherein In the step 2-1, the semantic segmentation grayscale image and the depth prior obtained in the step 1 are spliced, the semantic information and depth information of the scene picture are spliced, the class and edge information in the semantic information are used to guide the depth change in the low-texture area and the edge area of the depth map.
5. The monocular view depth estimation method based on neural radiance fields and semantic segmentation of claim 1, wherein In the step 2-2, the optimization network uses a U-shaped encoding-decoding structure, uses different scale information of the depth prediction map and the semantic segmentation map, and adds original convolution layer information at different scales when downsampling, and emphasizes the distinction between different modules in the picture.
6. The monocular view depth estimation method based on neural radiance fields and semantic segmentation of claim 1, wherein In the step 3-1, the depth confidence map calculated according to the depth prior is used for guidance, the depth prior is used for guidance of neural radiance field sampling, which specifically includes: A1: Project each depth map back to three-dimensional space to form a point, then project it to all other views to obtain a depth map projected from other views at each view, and take K error minimum maps as reference to calculate the average error; A2: Sample space points around the depth prior on the corresponding ray of each pixel, and scale the sampling range according to the depth error map of the pixel.
7. The monocular view depth estimation method based on neural radiance fields and semantic segmentation of claim 1, wherein In the step 3-2, the density probability value and the segmentation information value corresponding to each point are output, in addition to the conventional color output and density probability output, additional semantic information independent of the observation view is output, and each spatial point outputs a probability distribution containing 255 classes of semantic information.
8. The monocular view depth estimation method based on neural radiance fields and semantic segmentation of claim 1, wherein In the step 4-1, the generated segmentation information and the semantic segmentation information corresponding to the original picture are compared, in addition to using the color map for self-supervised guidance, the semantic segmentation information obtained by volume rendering and the previous semantic segmentation map are used for cross-entropy calculation, and the influence of the semantic information is further strengthened.
9. The monocular view depth estimation method based on neural radiance fields and semantic segmentation of claim 1, wherein In the step 4-3, the error is calculated as filtering, that is, a planar bilateral filter is designed according to the error between the generated color map and the original picture, and the depth map is further optimized to obtain the final result.