High-precision three-dimensional reconstruction method based on sparse depth

Abstract

The invention discloses a high-precision three-dimensional reconstruction method based on sparse depth, and the method comprises the steps: inputting an RGB-D image sequence of sparse depth, effectively filling the missing depth information in each frame of sparse depth image through a depth completion network trained on a public data set, and reasoning the semantic information of each frame through a pre-trained visual large model Dinov2; for each frame of RGB image, a combined sampling method based on feature point sampling and uniform sampling is adopted to extract sampling points of light, and spatial three-dimensional points are generated through importance sampling and random sampling strategies; performing position coding on the three-dimensional space sampling points to code low-frequency features, processing high-frequency features by using Hash coding, fusing the low-frequency features and the high-frequency features, inputting the fused high-frequency features and the fused high-frequency features into a neural network, and predicting a color value, an SDF value and a semantic value; and rendering and optimizing the pose of the current frame according to the current scene representation information, comparing a predicted value with the real color, depth and semantic information of the current frame by taking a global key frame as a sliding window, and optimizing implicit scene representation, thereby realizing reconstruction of a high-fidelity three-dimensional model from the sparse depth RGB-D image sequence. The method effectively solves the problem that a traditional three-dimensional reconstruction technology is insufficient in reconstruction precision under the sparse depth data condition, has a wide application prospect, and is particularly suitable for the three-dimensional reconstruction related field based on RGB-D data.

Description

Technical Field

[0001] The present invention relates to the field of computer vision three-dimensional reconstruction, and in particular to a high-precision three-dimensional reconstruction method based on sparse depth. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) technology is crucial in the field of automation and robotics, supporting machines to navigate and build maps in unknown environments. However, traditional SLAM methods have limitations in capturing object contours and textures, and usually rely on low-level visual features to estimate geometric structures. Dense visual 3D scene reconstruction technology is widely used in autonomous navigation, environmental perception, augmented reality and other fields through high-fidelity 3D reconstruction and real-time pose tracking, improving the performance and perception capabilities of robots and virtual reality systems.

[0003] Traditional dense visual 3D scene reconstruction methods usually represent scenes through point clouds, grids or voxel grids, usually use motion recovery structure (SFM) to estimate the pose of the image, and construct dense scenes through multi-view stereo matching (MVS) methods. However, this process has problems such as low tracking accuracy, poor reconstruction quality, and inability to achieve end-to-end scene reconstruction. The neural radiance field (NeRF) reconstruction method proposed in recent years has shown the potential to reconstruct high-fidelity scenes. However, since it requires the image exposure to be consistent, the image must be noise-free, without missing or blurry, and requires images with accurate camera calibration data as input, it limits its widespread use in real scene reconstruction applications. For example: The existing invention "A method for unmanned driving environment perception based on three-dimensional reconstruction" (CN202311792415.2) has the disadvantage of poor quality of three-dimensional reconstruction by directly using sparse depth information obtained by lidar; "A method for three-dimensional reconstruction of indoor scenes based on adaptive normal prior" (CN202410438152.3) has the disadvantage of requiring the use of the colmap tool to obtain the pose of each image, resulting in a long reconstruction time.

[0004] In view of the above problems, the present invention discloses a high-precision 3D reconstruction method based on sparse depth, which can be widely used in high-quality reconstruction of scenes. The method does not require image pose input and reconstructs the properties of a high-precision scene 3D model end-to-end from sparse depth. Summary of the invention

[0005] The present invention discloses a high-precision 3D reconstruction method based on sparse depth, which can realize end-to-end reconstruction of a high-fidelity 3D model from a sparse depth RGB-D image sequence. To achieve the above purpose, the present invention provides a high-precision 3D reconstruction method based on sparse depth, comprising the following steps:

[0006] Use the depth completion network model to fill the missing depth of the input RGB-D image sequence;

[0007] Semantically segment the RGB image sequence based on the pre-trained visual model Dinov2 to obtain semantic information;

[0008] For each frame of input image, a joint sampling method is used to extract sampling points as selected rays;

[0009] For all selected rays, use depth-based importance sampling and random sampling to obtain the 3D space sampling points of the rays;

[0010] The sampling points are positionally encoded to encode low-frequency features, while hash coding is used to process high-frequency features;

[0011] Cross-fuse the features and input them into neural network training;

[0012] Rendering the optimized camera pose of the image frame based on the current scene representation;

[0013] Rendering optimizes implicit scene representation using sampled rays from a global keyframe sliding window.

[0014] The depth completion network obtains a pre-trained model by training the depth completion network using the public data set Synthetic RGB-D, and uses the pre-trained model to complete the depth missing area of ​​the current image.

[0015] The semantic segmentation performs semantic prediction on RGB images by using the pre-trained visual large model Dinov2 feature extractor and ADE20K linear multi-scale segmentation head.

[0016] The joint sampling extracts feature points from the input image as a preliminary sampling subset, and for the remaining pixels, a uniform grid is constructed on the image and uniform sampling is performed to supplement the sampling, so as to determine the selection of a fixed number of rays.

[0017] The three-dimensional spatial sampling points are obtained by spatially sampling the selected light rays, taking the pixel point as the center to obtain 11 sampling points in the neighborhood, and uniformly sampling 32 sampling points along the light rays as the three-dimensional spatial sampling points of the light rays.

[0018] The encoding performs one-blob position encoding on the three-dimensional space sampling points to encode their low-frequency feature information, and uses hash encoding on the three-dimensional space sampling points to encode their high-frequency feature information.

[0019] The cross fusion is to connect the position coded features with the hash coded features and input them into the SDF decoder training, connect the potential features output by the SDF decoder with the position coded features and input them into the semantic decoder training, and connect the potential features decoded by the SDF decoder with the hash coded features and input them into the color decoder training.

[0020] The optimized posture is characterized in that the corresponding SDF value, semantic value, color value and true value are rendered from the optimized scene representation through the current posture to optimize the current camera posture with loss.

[0021] The optimized implicit scene representation is characterized in that a global key frame set is established by using image frames of fixed length as key frames, light in the global key frame set is sampled and light of the current frame is sampled using the currently optimized camera pose, the sampling points on the light are position-encoded and hash-encoded, the encoded feature information is input into neural network training, corresponding SDF values, semantic values, color values ​​and true values ​​are rendered with different losses, and finally the implicit scene representation is optimized in a loop iteratively. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for the description of the embodiments or the prior art, and the present invention can be further understood in conjunction with the contents of the drawings. The components shown in the drawings are not necessarily drawn to scale, but the focus is on illustrating the principles of the embodiments.

[0023] Figure 1 It is a schematic diagram of the steps of a high-precision three-dimensional reconstruction method based on sparse depth provided by the present invention.

[0024] Figure 2 This is a depth completion network structure diagram provided by an embodiment of the present invention.

[0025] Figure 3 A system diagram for reconstructing a high-precision three-dimensional model from sparse depth provided by an embodiment of the present invention.

[0026] Figure 4 This is a comparison chart of reconstruction effects on some scenes of the Synthetic RGB-D dataset using a high-precision 3D reconstruction method based on sparse depth and other methods provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following describes in detail an embodiment of the present invention taking a sparse depth RGB-D image sequence for three-dimensional reconstruction as an example, and examples of the embodiment are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0028] See also Figure 1 , Figure 2 and Figure 3 The present invention provides a high-precision three-dimensional reconstruction method based on sparse depth, comprising the following steps:

[0029] S101, performing depth completion on a sparse depth RGB-D image sequence.

[0030] Specifically, the depth image i with missing depth is input into the depth completion network as follows Figure 2 As shown in the figure, irregular sparse data is processed through 4 layers of sparse convolution to extract deep features at different levels. Then, the image features of the last layer are input into the channel attention mechanism (SE) and the spatial attention mechanism with multi-head attention to capture local and global structural information. At the same time, the U-Net network architecture is adopted to fuse feature information of different scales through jump connections, and the fused features are decoded using a two-dimensional convolution layer each time, and then up-sampled to restore high-resolution details. The depth information is further decoded through a 4-layer decoder, and the weights of the network are obtained by pre-training on a public dataset, thereby realizing the depth completion of sparse depth images.

[0031] S102, performing semantic segmentation on the RGB image sequence according to the visual macro model.

[0032] Specifically, the semantic information of the image is obtained by using the pre-trained visual large model Dinov2 feature extractor and ADE20K linear multi-scale segmentation head to perform semantic prediction on the RGB image.

[0033] S103: Extract sampling points of the light rays using a joint sampling method.

[0034] Specifically, an improved feature extraction algorithm is used to extract feature points from each frame of RGB image i as a preliminary sampling subset. For the remaining pixels, a uniform grid is constructed on the image and uniform sampling is performed to supplement the sampling of a specified number of rays.

[0035] S104. Obtain the three-dimensional point of the light through importance and random sampling.

[0036] Specifically, the selected light is sampled in space, 11 sampling points are obtained within the neighborhood range with the pixel depth as the center, and 32 sampling points are uniformly sampled along the light as the three-dimensional spatial sampling points of the light.

[0037] S105. Encode feature information according to position coding and hash coding.

[0038] Specifically, the three-dimensional spatial sampling points of the light are position-coded and encoded into a 48-dimensional low-frequency feature vector using one-blob coding, and are encoded into a 64-dimensional high-frequency feature vector using hash coding.

[0039] S106. Use the crossed and fused features to input neural network training.

[0040] Specifically, the position-encoded features are concatenated with the hash-encoded features and then input into the SDF decoder for training and predicting the corresponding SDF value s. i , connect the potential features output by the SDF decoder with the position encoding features and input them into the semantic decoder to train and predict the corresponding semantic value m i , connect the latent features decoded by the SDF decoder with the hash coding features and send them to the color decoder for training to predict the corresponding color value c i .

[0041] S107 , rendering and optimizing the pose of the image frame according to the current scene representation.

[0042] Specifically, each frame image first uses the uniform motion model to initialize the pose T of the current frame t , specifically:

[0043]

[0044] Select N in the current frame through a joint sampling strategy t Pixels, use the current pose to convert the sampled pixels into sampled rays, and render the SDF value s of the current ray based on the scene representation optimized by the historical keyframes i , semantic value Color Value And the depth value Using a simple bell-shaped model, the predicted SDF value s i Convert to weight value w i :

[0045]

[0046] Among them, tr is the cutoff distance of the signed distance field. The corresponding rendering process can be expressed as follows:

[0047]

[0048] The total loss is optimized using:

[0049] L total =λ 1 L c +λ 2 L d +λ 3 L free-space +λ 4 L sdf +λ 5 L smooth +λ 6 L sem (4)

[0050] Among them, L c ,L d ,L sdf ,L sem They are color loss, depth loss, SDF loss and semantic loss, L free-space ,L smooth They are free space loss and smoothness loss, which are used to constrain the SDF value of points outside the truncation region to 1 and prevent noise reconstruction caused by hash conflicts in unobserved free space regions.

[0051] Based on the above loss function, the current camera pose is continuously iterated.

[0052] S108, rendering the sampling rays of the global key frame sliding window to optimize the implicit scene representation.

[0053] Specifically, a global keyframe set is established by taking a fixed-length image frame as a keyframe, sampling the light in the global keyframe set, and using the currently optimized camera pose to sample the light of the current frame. Position encoding and hash encoding are performed on the sampling points on all the light rays, and the encoded feature information is cross-fused and input into the neural network training to render the corresponding SDF value s i , semantic value Color Value The above loss is calculated with the true value, and the implicit scene representation is optimized through loop iteration. In order to extract the triangular mesh model, the SDF value and color value corresponding to each mesh vertex are predicted through the network, and then the Marching Cube algorithm is used to extract the three-dimensional model of the scene surface, thereby reconstructing a high-precision three-dimensional surface model from sparse depth and accurately restoring the geometric structure and texture information of the environment.

[0054] Finally, as the final stage of the process of an embodiment of the present invention, experimental verification verifies the feasibility and effect of the method proposed in the present invention through actual experiments.

[0055] In the experimental verification stage, the present invention selected other dense reconstruction methods for quantitative experiments and qualitative experiments. In the quantitative experiment part, the estimated camera tracking trajectory accuracy comparison results are shown in Table 1. The calculated index is ATE RMSE, in centimeters. The smaller the value, the better the posture tracking. The model of the present invention has achieved the best trajectory estimation accuracy indicators in all scenes, and achieved high-precision camera trajectory accuracy estimation; the scene reconstruction indicators are shown in Table 2, using depth L1 (DepthL1) (in centimeters), accuracy (Accuracy) (in centimeters), completion (Completion) (in centimeters) and completion ratio (Completion ratio) [<5cm%] to evaluate the reconstruction quality. The results show that the present invention achieves high-quality scene reconstruction; in the qualitative experiment, such as Figure 4 As shown, the reconstruction results of the method of the present invention and other methods on some scenes of the Synthetic RGB-D dataset show that the method of the present invention can reconstruct richer and more complex details, proving its advantages in complex scenes.

[0056] Table 1 Comparison results of camera tracking trajectory accuracy of the proposed method and other methods on the Replica dataset

[0057]

[0058] Table 2 Comparison of reconstruction results of the proposed method and other methods on the Synthetic RGB-D dataset

[0059]

[0060] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A high-precision 3D reconstruction method based on sparse depth, characterized in that: The following steps are involved: Use the depth completion network model to fill the missing depth of the input RGB-D image sequence; Semantically segment the RGB image sequence based on the pre-trained visual model Dinov2 to obtain semantic information; For each frame of input image, a joint sampling method is used to extract sampling points as selected rays; For all selected rays, use depth-based importance sampling and random sampling to obtain the 3D space sampling points of the rays; The sampling points are positionally encoded to encode low-frequency features, while hash coding is used to process high-frequency features; Cross-fuse the features and input them into neural network training; Rendering the optimized camera pose of the image frame based on the current scene representation; Rendering optimizes implicit scene representation using sampled rays from a global keyframe sliding window.

2. The depth completion network according to claim 1, characterized in that The pre-trained model is obtained by training the depth completion network using the public dataset SyntheticRGB-D, and the pre-trained model is used to complete the depth missing area of ​​the current image.

3. The semantic segmentation according to claim 1, characterized in that Semantic prediction is performed on RGB images by using the pre-trained vision model Dinov2 feature extractor and ADE20K linear multi-scale segmentation head.

4. The joint sampling according to claim 1, characterized in that: Feature points are extracted from the input image as a preliminary sampling subset. For the remaining pixels, a uniform grid is constructed on the image and uniform sampling is performed to supplement it to determine the selection of a fixed number of rays.

5. The three-dimensional space sampling point according to claim 1, characterized in that: The selected light is sampled spatially, and 11 sampling points are obtained within the neighborhood range with the pixel depth as the center. At the same time, 32 sampling points are uniformly sampled along the light as the three-dimensional spatial sampling points of the light.

6. The encoding according to claim 1, characterized in that One-blob position encoding is performed on the three-dimensional space sampling points to encode their low-frequency feature information. At the same time, hash coding is used for the three-dimensional space sampling points to encode their high-frequency feature information.

7. The cross-fusion according to claim 1, characterized in that: The position-encoded features are concatenated with the hash-encoded features and then fed into the SDF decoder for training. The latent features output by the SDF decoder are concatenated with the position-encoded features and then fed into the semantic decoder for training. The latent features decoded by the SDF decoder are concatenated with the hash-encoded features and then fed into the color decoder for training.

8. The optimized posture according to claim 1, characterized in that: The current camera pose is optimized by rendering the corresponding SDF value, semantic value, color value and true value from the optimized scene representation through the current pose.

9. The optimized implicit scene representation according to claim 1, characterized in that: A global keyframe set is established by taking image frames of fixed length as keyframes, sampling the light in the global keyframe set and sampling the light of the current frame using the currently optimized camera pose, performing position encoding and hash encoding on the sampling points on the light, inputting the encoded feature information into neural network training, rendering the corresponding SDF values, semantic values, color values ​​and true values ​​with different losses, and finally optimizing the implicit scene representation through iterative cycles.

Images