A high-precision three-dimensional reconstruction method based on sparse depth

Abstract

The application discloses a high-precision three-dimensional reconstruction method based on sparse depth, an RGB-D image sequence of sparse depth is input, the missing depth information in each frame of sparse depth map is effectively filled in by a depth completion network trained on a public data set, and a pre-trained visual large model Dinov2 is used to infer the semantic information of each frame; for each frame of RGB image, a joint sampling method based on feature point sampling and uniform sampling is used to extract the sampling points of light, and spatial three-dimensional points are generated through importance sampling and random sampling strategy; the position coding is carried out on the three-dimensional space sampling points to code the low-frequency features, the high-frequency features are processed by using hash coding, and the two are fused and input into a neural network to predict the color value, SDF value and semantic value; according to the current scene representation information, the pose of the current frame is rendered and optimized, the predicted value is compared with the real color, depth and semantic information of the current frame through the global key frame as a sliding window, the implicit scene representation is optimized, and a high-fidelity three-dimensional model is reconstructed from the sparse depth RGB-D image sequence. The application effectively solves the problem of insufficient reconstruction precision of the traditional three-dimensional reconstruction technology under the condition of sparse depth data, and has a wide application prospect, and is especially suitable for the field of three-dimensional reconstruction based on RGB-D data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision three-dimensional reconstruction, and in particular to a high-precision three-dimensional reconstruction method based on sparse depth. BACKGROUND

[0002] SLAM (Simultaneous Localization and Mapping) technology is crucial in the field of automation and robotics, supporting machines to navigate and map in unknown environments. However, traditional SLAM methods have limitations in capturing object contours and textures, often relying on low-level visual features to estimate geometry. Dense visual three-dimensional scene reconstruction technology, through high-fidelity three-dimensional reconstruction and real-time pose tracking, is widely used in autonomous navigation, environment perception, and augmented reality, etc., improving the performance and perception ability of robots and virtual reality systems.

[0003] Traditional dense visual three-dimensional scene reconstruction methods usually represent the scene through point clouds, meshes, or voxel grids, and usually use Structure from Motion (SFM) to estimate the pose of the image and construct a dense scene through Multi-View Stereo Matching (MVS) method. However, this process has problems such as low tracking accuracy, poor reconstruction quality, and inability to achieve end-to-end scene reconstruction. The recent introduction of the Neural Radiance Field (NeRF) reconstruction method has shown the potential for reconstructing high-fidelity scenes, but its use in real scene reconstruction applications is limited due to the need for consistent exposure of images, the absence of noise, missing, and blurring, and the need for accurate camera calibration data as input. For example, the existing invention "Unmanned driving environment perception method based on three-dimensional reconstruction" (CN202311792415.2) has the disadvantage of poor three-dimensional reconstruction quality using sparse depth information obtained by laser radar; "Indoor scene three-dimensional reconstruction method based on adaptive normal prior" (CN202410438152.3) has the disadvantage of long reconstruction time by using colmap tool to obtain the pose of each image.

[0004] To solve the above problems, the present application discloses a high-precision three-dimensional reconstruction method based on sparse depth, which can be widely used in high-quality reconstruction of scenes. This method does not require image pose input and can end-to-end reconstruct high-precision three-dimensional scene models from sparse depth. SUMMARY

[0005] The present application discloses a high-precision three-dimensional reconstruction method based on sparse depth, which can realize end-to-end reconstruction of high-fidelity three-dimensional models from sparse depth RGB-D image sequences. To achieve the above purpose, the present application provides a high-precision three-dimensional reconstruction method based on sparse depth, including the following steps:

[0006] A depth completion network model is used to fill in the missing depth for the input RGB-D image sequence;

[0007] An RGB image sequence is semantically segmented using a pre-trained visual large model Dinov2 to obtain semantic information;

[0008] A joint sampling method is used to extract sampling points as selected rays for each input image frame;

[0009] Depth-based importance sampling and random sampling are used to obtain three-dimensional space sampling points for all selected rays;

[0010] Position encoding is performed on the sampling points to encode low-frequency features, and hash encoding is used to process high-frequency features;

[0011] The features are cross-fused and input into a neural network for training;

[0012] The camera pose of the rendered optimized image frame is determined according to the current scene representation;

[0013] The sampling rays of the global keyframe sliding window are used to render the optimized implicit scene representation.

[0014] The depth completion network uses a pre-trained model obtained by training the depth completion network using the public dataset Synthetic RGB-D to complete the depth missing area of the current image.

[0015] The semantic segmentation uses a pre-trained visual large model Dinov2 feature extractor and ADE20K linear multi-scale segmentation head to perform semantic prediction on the RGB image.

[0016] The joint sampling extracts feature points from the input image as a preliminary sampling subset, and for the remaining pixels, a uniform grid is constructed on the image and uniform sampling is performed to supplement the selection of a fixed number of rays.

[0017] The three-dimensional space sampling points are obtained by sampling the selected rays in space, with 11 sampling points obtained within the neighborhood range of the pixel point as the center, and 32 sampling points uniformly sampled along the ray as the three-dimensional space sampling points of the ray.

[0018] The encoding uses one-blob position encoding on the three-dimensional space sampling points to encode their low-frequency feature information, and hash encoding on the three-dimensional space sampling points to encode their high-frequency feature information.

[0019] The cross fusion is that the position coded feature and the hash coded feature are connected and input into an SDF decoder for training, the latent feature output by the SDF decoder is connected with the position coded feature and input into a semantic decoder for training, and the latent feature decoded by the SDF decoder is connected with the hash coded feature and input into a color decoder for training.

[0020] The optimized pose is characterized in that the corresponding SDF value, semantic value and color value are rendered from the optimized scene representation by the current pose, and the current camera pose is loss optimized according to the true value.

[0021] The optimized implicit scene representation is characterized in that a global key frame set is established by taking the fixed-length image frame as a key frame, the light rays in the global key frame set are sampled, the light rays of the current frame are sampled according to the current optimized camera pose, the sampling points on the light rays are position coded and hash coded, the coded feature information is input into a neural network for training, the corresponding SDF value, semantic value and color value are rendered, different losses are made according to the true value, and finally the implicit scene representation is iteratively optimized. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description, and the content of the drawings can further understand the present application. The components shown in the drawings are not necessarily drawn according to the scale, but the emphasis is placed on showing the principles of the embodiments.

[0023] Figure 1 is a step schematic diagram of a high-precision three-dimensional reconstruction method based on sparse depth provided by the present application.

[0024] Figure 2 is a deep completion network structure diagram provided by an embodiment of the present application.

[0025] Figure 3 is a system diagram of reconstructing a high-precision three-dimensional model from sparse depth provided by an embodiment of the present application.

[0026] Figure 4 is a reconstruction effect comparison diagram of using the high-precision three-dimensional reconstruction method based on sparse depth and other methods on part of the scene of the Synthetic RGB-D data set provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] The application is described below in detail taking an embodiment of three-dimensional reconstruction based on a sparse depth RGB-D image sequence as an example, and examples of the embodiment are shown in the drawings, in which the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the application, and cannot be understood as a limitation of the application.

[0028] Referring to Figure 1 , Figure 2 and Figure 3 , the application provides a high-precision three-dimensional reconstruction method based on sparse depth, comprising the following steps:

[0029] S101, depth completion is performed on the sparse depth RGB-D image sequence.

[0030] Specifically, the depth image i with missing depth is input into a depth completion network, as shown in Figure 2 , the irregular sparse data is processed by 4 layers of sparse convolution to extract depth features at different levels. Then, the image features of the last layer are input into a channel attention mechanism (SE) and a spatial attention mechanism with multi-head attention to capture local and global structure information. At the same time, a U-Net network architecture is used to fuse feature information at different scales through a skip connection, and a two-dimensional convolution layer is used to decode the fused features each time, and then an upsampling operation is performed to restore high-resolution details. The depth information is further decoded by 4 layers of decoder, and the weights of the network are obtained by pre-training on a public dataset, so as to realize the depth completion of the sparse depth image.

[0031] S102, performing semantic segmentation on the RGB image sequence according to a visual large model.

[0032] Specifically, the semantic information of the image is obtained by using a pre-trained visual large model Dinov2 feature extractor and an ADE20K linear multi-scale segmentation head to perform semantic prediction on the RGB image.

[0033] S103, using a joint sampling method to extract sampling points of light rays.

[0034] Specifically, a feature point is extracted from each frame of RGB image i using an improved feature extraction algorithm as a preliminary sampling subset, and for the remaining pixels, a uniform grid is constructed on the image and uniform sampling is performed to supplement, so as to realize sampling of a specified number of light rays.

[0035] S104, obtaining three-dimensional points of light rays through importance and random sampling.

[0036] Specifically, the selected light rays are sampled in space, and 11 sampling points are obtained in the neighborhood range centered on the pixel depth. At the same time, 32 sampling points are uniformly sampled along the light ray as the three-dimensional spatial sampling points of the light ray.

[0037] S105. Encode feature information based on position encoding and hash encoding.

[0038] Specifically, the three-dimensional spatial sampling points of the light rays are positionally encoded using one-blob encoding to encode them into a 48-dimensional low-frequency feature vector, while hash encoding is used to encode them into a 64-dimensional high-frequency feature vector.

[0039] S106. Use cross- and fused features to train the neural network.

[0040] Specifically, the position-encoded features are concatenated with the hash-encoded features and then input into the SDF decoder to train and predict the corresponding SDF values ​​s. i The latent features output by the SDF decoder are concatenated with the position-encoded features and input into the semantic decoder for training and prediction of the corresponding semantic value m. i The latent features decoded by the SDF decoder are concatenated with the hash-encoded features and fed into the color decoder to train and predict the corresponding color value c. i .

[0041] S107. Optimize the pose of the image frame based on the current scene representation rendering.

[0042] Specifically, for each frame, the pose T of the current frame is first initialized using a uniform motion model. t Specifically:

[0043] N in the current frame is selected using a joint sampling strategy. t Each pixel is used to transform a sampled ray into a sampled ray based on the current pose. The SDF value of the current ray is then rendered based on the scene representation optimized from historical keyframes. i semantic value Color value and depth value The predicted SDF values ​​are obtained using a simple bell model. i Convert to weight value w i :

[0044]

[0045] Where tr is the cutoff distance of the symbolic distance field. The corresponding rendering process can then be represented as follows:

[0046] The total loss is optimized as follows:

[0047] L total = λ1L c + λ2L d + λ3L free-space + λ4L sdf + λ5L smooth + λ6L sem (4) wherein, L c , L d , L sdf , L sem are color loss, depth loss, SDF loss and semantic loss respectively, L free-space , L smooth are free space loss and smooth loss respectively, which are used to constrain the SDF value of the points outside the truncated region to be 1 and prevent noise reconstruction caused by hash collision in the unobserved free space region.

[0048] Based on the above loss function, the current camera pose is iterated continuously.

[0049] S108, render the sampling rays of the global key frame sliding window to optimize the implicit scene representation.

[0050] Specifically, a global key frame set is established by taking fixed length image frames as key frames, rays in the global key frame set are sampled, and the rays of the current frame are sampled by using the current optimized camera pose, the sampling points on all rays are position encoded and hash encoded, and the encoded feature information is cross-fused and input into a neural network for training to render the corresponding SDF value s i , semantic value color value and true value to calculate the above loss, and the implicit scene representation is optimized by loop iteration. In order to extract the triangular mesh model, the SDF value and color value corresponding to each mesh vertex are predicted by the network, and then the Marching Cube algorithm is used to extract the three-dimensional model of the scene surface, so as to realize the reconstruction of high-precision three-dimensional surface model from sparse depth, and accurately restore the geometric structure and texture information of the environment.

[0051] Finally, the experimental verification as the last stage of an embodiment process of the present application verifies the feasibility and effect of the method proposed by the present application through actual experiments.

[0052] In the experimental verification stage, the application selects other dense reconstruction methods to carry out quantitative experiments and qualitative experiments. In the quantitative experiment part, the estimated camera tracking trajectory accuracy comparison results are as shown in Table 1, the calculated index is ATE RMSE, the unit is centimeter, the smaller the value represents better pose tracking, the model of the application achieves the best trajectory estimation accuracy index in all scenes, and high-precision camera trajectory accuracy estimation is realized; the scene reconstruction index is as shown in Table 2, the depth L1 (DepthL1) (unit: centimeter), accuracy (Accuracy) (unit: centimeter), completion (Completion) (unit: centimeter) and completion ratio (Completion ratio) [<5cm%] are used to evaluate the reconstruction quality, and the results show that the application realizes high-quality scene reconstruction; in the qualitative experiment, as shown in Figure 4 The reconstruction results of the method of the application and other methods in the Synthetic RGB-D data set part scene show that the method of the application can realize the reconstruction of more rich and complex details, and proves the advantage of the method in complex scenes.

[0053] Table 1 Camera tracking trajectory accuracy comparison results of the method of the application and other methods in the Replica data set

[0054]

[0055] Table 2 Reconstruction results of the method of the application and other methods in the Synthetic RGB-D data set

[0056]

[0057] The above only discloses one preferred embodiment of the application, and of course cannot limit the scope of the rights of the application, and those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments are realized, and equivalent changes made according to the claims of the application still belong to the scope covered by the application.

Claims

1. A high-precision three-dimensional reconstruction method based on sparse depth, characterized in that, The method comprises the following steps: S1, using a depth completion network model to fill in the missing depth for the input RGB-D image sequence; S2, performing semantic segmentation on the RGB image sequence according to a pre-trained visual large model Dinov2 to obtain semantic information; S3, using a joint sampling method to extract sampling points as selected rays for each input image frame; S4, using depth-based importance sampling and random sampling to obtain three-dimensional space sampling points of the rays; S5, performing position encoding on the sampling points to encode low-frequency features, and simultaneously using hash encoding to process high-frequency features; S6, inputting the features into a neural network for training after cross-fusion, connecting the position-encoded features and the hash-encoded features, inputting the connected features into an SDF decoder for training, connecting the latent features output by the SDF decoder and the position-encoded features, inputting the connected features into a semantic decoder for training, and connecting the latent features decoded by the SDF decoder and the hash-encoded features, inputting the connected features into a color decoder for training; S7, rendering an optimized image frame according to the camera pose of the current scene representation, and optimizing the current camera pose by rendering the corresponding SDF value, semantic value and color value from the optimized scene representation according to the current pose and comparing the values with the true values; S8, using the sampling rays of the global key frame sliding window to render and optimize the implicit scene representation, establishing a global key frame set by taking fixed-length image frames as key frames, sampling the rays in the global key frame set and using the currently optimized camera pose to sample the rays of the current frame, performing position encoding and hash encoding on the sampling points on the rays, inputting the encoded feature information into a neural network for training, rendering the corresponding SDF value, semantic value and color value, and comparing the values with the true values to form different losses, and finally iteratively optimizing the implicit scene representation.

2. The high-precision three-dimensional reconstruction method based on sparse depth according to claim 1, wherein the depth completion network in S1 is characterized by, A pre-trained model is obtained by training a depth completion network using a public dataset Synthetic RGB-D, and the pre-trained model is used to complete the depth missing area of the current image. 3.The high-precision three-dimensional reconstruction method based on sparse depth according to claim 1, wherein the semantic segmentation in S2 is characterized by, The RGB image is semantically predicted by using a pre-trained visual large model Dinov2 feature extractor and an ADE20K linear multi-scale segmentation head.

4. The high-precision three-dimensional reconstruction method based on sparse depth of claim 1, wherein the joint sampling in S3 is characterized by, Feature points are extracted from the input image as a preliminary sampling set, and for the remaining pixels, a uniform grid is constructed on the image and uniform sampling is performed to supplement the selection of a fixed number of rays.

5. The high-precision three-dimensional reconstruction method based on sparse depth according to claim 1, wherein the three-dimensional space sampling points in S4 are characterized by, The selected rays are sampled in space to obtain 11 sampling points in the neighborhood range centered on the pixel depth, and 32 sampling points are uniformly sampled along the rays as three-dimensional space sampling points of the rays.

6. The high-precision three-dimensional reconstruction method based on sparse depth according to claim 1, wherein the encoding in S5 is characterized by, The three-dimensional space sampling points are one-blob position encoded to encode their low-frequency feature information, and the three-dimensional space sampling points are hash-encoded to encode their high-frequency feature information.

Images