Three-dimensional reconstruction and rendering method based on outdoor unconstrained image set

Abstract

The invention discloses a three-dimensional reconstruction and rendering method based on an outdoor unconstrained image set, and the method comprises the steps: inputting a photo set randomly collected by the Internet, simplifying the sampling through random grid sampling, obtaining the appearance features based on a convolutional neural network, guiding the sampling through an occupied grid, and obtaining the features of a sampling point through Hash coding interpolation. And obtaining the light visibility based on the implicit function and the segmentation model. Experiments on a data set in the real world show that appearance rendering can be achieved, dynamic objects can be removed, meanwhile, the training speed and the rendering speed are increased by dozens of times, and the evaluation index is better. The method solves the problems of strict NeRF data requirements and slow speed, and can be widely applied to related applications of three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of computer vision three-dimensional reconstruction, and in particular to a three-dimensional reconstruction and rendering method based on an outdoor unconstrained image set. Background Art

[0002] In recent years, with the rapid development of computer graphics and computer vision technology, various 3D technologies have become a research hotspot in today's era. Through these technologies, people can immerse themselves in a digital multi-dimensional environment. The advent of Neural Radiance Fields (NeRF) has rapidly promoted the development of new views of synthetic realistic scenes. Perhaps in the near future, people can visit scenic spots around the world at home without leaving their homes, and witness the scenery of various scenic spots at different times and seasons.

[0003] Currently, NeRF has two serious problems. The first is that NeRF requires a set of pictures taken at almost the same time and capturing the target object. In other words, NeRF strictly controls the brightness of the scene, and the brightness needs to remain basically unchanged. However, this is impossible to achieve for the images of tourist attractions collected online. The images collected online are interfered by various factors such as different mobile phones, different times, different seasons, and objects being blocked. This has a great impact on NeRF's synthesis of realistic scenes. However, some people have previously proposed to introduce GLO to solve this problem. The second is that NeRF is very slow. The original NeRF training time is as long as several days. This is mainly caused by its sampling method and deep neural network, but there are also related technologies to solve this problem. For example, some people have used octrees and hash coding to solve this problem.

[0004] In view of the above problems, the present invention discloses a 3D reconstruction and rendering method based on an outdoor unconstrained image set, which realizes fast training and rendering by using accurate visible graphs and efficient sampling methods, removes dynamic objects, and realizes appearance rendering at the same time. The method does not require a specific static scene as input, but can realize reconstruction and rendering by finding any multi-view images with variable illumination and post-processing including dynamic objects from the Internet. Summary of the invention

[0005] The present invention discloses a 3D reconstruction and rendering method based on an outdoor unconstrained image set, which can be quickly trained and rendered, and remove dynamic objects while achieving appearance rendering. To achieve the above purpose, the present invention provides a 3D reconstruction and rendering method based on an outdoor unconstrained image set, comprising the following steps:

[0006] The input image is sampled through a random grid, and the appearance features are obtained through the appearance module;

[0007] The appearance features, direction and position are decoded into RGB through the RGB module;

[0008] Pass the input image through the visible map module to obtain the corresponding visible map;

[0009] Pass the RGB and visible images through the rendering module to obtain the rendered image.

[0010] The random grid sampling is used to simplify the light sampling. First, the corresponding image is obtained, and then a continuous area is randomly selected in the image for sampling and the image is passed through a convolutional neural network to obtain appearance features. The number of sampled rays is 1024, and finally a 32x32 low-resolution image is formed.

[0011] The RGB module first needs to build an occupancy grid, guide sampling, and use multi-resolution hash coding to obtain three-dimensional point features;

[0012] In order to skip sampling of blank areas, a k-scale occupancy grid cascade is maintained. Since the scene is outdoors, k=128 is set, and the resolution of each grid is 128. 3 , spanning a geometric growth domain centered at (0, 0, 0) [-2 k ,2 k ] 3 . During training, uniform sampling M = K·128 3 cells, without duplication, each voxel stores occupancy in the form of a single bit. The cells are arranged in a Morton curve, and it is worth noting that all cells are traversed at each step. When the light is moving forward, if the grid cell of the sample is high, the sample will be placed, and if the grid cell of the sample is low, the grid will be skipped;

[0013] Given the coordinates of a 3D point in space p = (x, y, z), find the voxel corresponding to each resolution where the point is located, obtain the coordinates of the eight vertices of the voxel, use the coordinates to get the index through the hash function, store the features in the corresponding index, and then perform trilinear interpolation to get the features of the point. The features interpolated from each resolution are spliced ​​together to get the final features of the point.

[0014] The sampling interval increases exponentially, and the sampling is terminated early according to the maximum number of samples and the threshold of the sampling point;

[0015] Generally, outdoor scenes are relatively large, and we ultimately hope that the number of sampling points in the near distance is greater than the number of sampling points in the far distance, so that we can ensure the quality of sampling in the near distance and sample objects in the far distance. Assuming t is the distance that the light travels, the step size is set to Δ t = t / 256, this step size selection grows exponentially in t. The sampling interval is set to Where s is the size of the largest axis of the scene boundary, and the maximum number of samples per ray is 1024;

[0016] At the same time, it is also calculated that when the transmittance of the light is lower than a certain threshold, the light is stopped from moving forward and the remaining contribution is set to 0, where the threshold ε is set to 1e-4.

[0017] The visible graph module obtains the corresponding visible graph by passing the input image through a classic image semantic segmentation pre-training model, and obtains the corresponding visible graph by using a classic text prompting method, and sends the pixel coordinates of the random grid sampling and their corresponding transient embedding into the implicit function to obtain their corresponding visible graph, and splices them into the final visible graph.

[0018] The rendering module uses volume rendering to supervise RGB and the final visible image and render the final image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for the description of the embodiments or the prior art, and the present invention can be further understood in conjunction with the contents of the drawings. The components shown in the drawings are not necessarily drawn to scale, but the focus is on illustrating the principles of the embodiments.

[0020] Figure 1 It is a schematic diagram of the steps of a three-dimensional reconstruction and rendering method based on an outdoor unconstrained image set provided by the present invention.

[0021] Figure 2 This is a quantitative experimental result diagram provided by an embodiment of the present invention.

[0022] Figure 3 , Figure 4 , Figure 5 , Figure 6 This is an appearance rendering provided by an embodiment of the present invention.

[0023] Figure 7 It is a comparison diagram of a visible image and a de-occlusion result provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following describes in detail an embodiment of the present invention taking 3D reconstruction of an RGB image set as an example, and examples of the embodiment are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0025] See also Figure 1The present invention provides a three-dimensional reconstruction and rendering method based on an outdoor unconstrained image set, comprising the following steps:

[0026] S101, using random grid sampling to simplify light sampling and obtain image appearance features through an appearance module.

[0027] Specifically, random grid sampling is used, and a custom cuda kernel is used to further improve the speed. In each training process, the corresponding image is first obtained, and then a continuous area is randomly selected in the image for sampling. The image is passed through a 6-layer convolutional neural network to obtain the image appearance features.

[0028] S102, RGB module first constructs an occupancy grid, guides sampling, and uses multi-resolution hash coding to obtain 3D point features.

[0029] Specifically, in order to skip sampling in blank areas, a k-scale occupancy grid cascade is maintained. Since the scene is outdoors, k=128 is set, and the resolution of each grid is 128. 3 , spanning a geometric growth domain centered at (0, 0, 0) [-2 k ,2 k ] 3 . During training, uniform sampling M = K·128 3 cells, without duplication, and each voxel stores occupancy in the form of a single bit. The cells are arranged in a Morton curve. It is worth noting that all cells are traversed at each step. As the light moves forward, if the grid cell of the sample is high, the sample will be placed. If the grid cell of the sample is low, the grid will be skipped. Based on this method, the entire space is divided into voxels of different resolutions, and each resolution is independent. The size of the hash table is T, the feature dimension of each entry is F, and the change in resolution depends on the growth factor r, which is calculated as follows:

[0030]

[0031] The hyperparameter N max Indicates the highest resolution, N min Indicates the lowest resolution, L indicates the number of levels. l is the corresponding level, then the resolution N l The size is calculated as:

[0032]

[0033] Given the coordinates p = (x, y, z) of a three-dimensional point in space, find the voxel corresponding to each resolution where the point is located, obtain the coordinates of the eight vertices of the voxel, and use the coordinates to get the index through the hash function. The hash function is defined as follows. The corresponding index stores the features, and then trilinear interpolation is performed to obtain the features of the point. The features interpolated from each resolution are spliced ​​together to obtain the final features of the point.

[0034]

[0035] Among them, p i is the coordinate of the three-dimensional point, π i is the only large prime number, ⊕ represents the bitwise XOR operation. In order to achieve independence, D-1 in the D dimension must be arranged, so they are π i Take π1=1, π2=2 654 435 761, and π3=805 459 861 respectively.

[0036] The sampling interval increases exponentially, and the sampling is terminated early according to the maximum number of samples and the threshold of the sampling point;

[0037] Specifically, considering that the scene is relatively large, it is hoped that the number of sampling points in the near distance is greater than the number of sampling points in the far distance, so that the quality of the near sampling can be guaranteed while the sampling of the distant objects can be guaranteed. Assuming t is the distance that the light travels, the step size is set to Δ t = t / 256, this step size selection grows exponentially in t. The sampling interval is set to Where s is the size of the largest axis of the scene boundary, and the maximum number of samples for each ray is 1024. It also calculates that when the transmittance of the ray is lower than a certain threshold, the ray will stop moving forward and the remaining contribution will be set to 0. We set the threshold ε = 1e-4. This setting can reduce the amount of calculation without affecting the final quality.

[0038] The RGB module decodes the final RGB of each sampling point.

[0039] Specifically, NeRF uses a multi-layer fully connected neural network to model 3D scenes. For any 3D point x = (x, y, z) in space and the viewing direction c=(r,g,b) and density σ. NeRF will construct a ray to render the color of each pixel: starting from the camera's optical center o, a ray r(t)=o+td is emitted along the direction d, so that the ray passes through the pixel and reaches the specified near plane t n To the far plane f Sampling is performed between the two pixels to obtain the position of the three-dimensional point, and then the position encoding γ(x) is used to transform each sampling position. The position encoding is as follows:

[0040]

[0041] Where L is a hyperparameter, which is mainly used to map the position to a higher-dimensional space, which makes the image rendered by NeRF richer and more realistic in contour details. Similarly, the position encoding of direction d is also transformed. NeRF uses volume rendering technology to predict the pixel color of the point

[0042]

[0043] where c i and σ i is γ(t i ) point color and density, δ i =t i+1 -t i is the distance between two points, α i =1-exp(-σ i δ i ), the α value is synthesized as the ray ending at γ(t i ) opacity, T i It is understood as the final cumulative transmittance of the rays.

[0044] NeRF uses the following form to represent density σ(t) and color c(t):

[0045]

[0046] Among them, θ = (θ1, θ2) are the parameters of MLP, γ x (·) and γ d (·) is a transformation that encodes the position x and direction d respectively. Considering the appearance of the object, a convolutional neural network is used to obtain features, and it can ensure that the convolutional neural network encoder is used under the same viewing direction. in Represents the feature parameters of the appearance encoding input to the network. For each image I i Encoded as appearance feature vector Therefore, equation (6) can be written as:

[0047]

[0048] Appearance coding Parameters It is learned together with the parameters θ of the neural radiance field and uses the L1 loss function:

[0049]

[0050] The image In the image I r Rendered in the direction of .

[0051] S103, input RGB image data and obtain the visible graph of the corresponding image through the segmentation model and implicit function.

[0052] Specifically, the DeepLabV3+ semantic segmentation pre-trained model is used to obtain the semantic segmentation visible map of the training image, and this visible map obtains the main reconstructed object area. Secondly, with the text prompt of the segment-anything method, static objects other than the main reconstructed object, such as the sky, the ground, etc., are input to obtain the corresponding visible map. Then, the two visible maps are spliced ​​accordingly. The pixel coordinates of the random grid sampling and their corresponding transient embedding are fed into the implicit function together to obtain their corresponding visible map.

[0053] Specifically, use MLP to construct the implicit function F ω to map the image pixel coordinates p = (u, v) and the image-related transient embedding τ i To a visible graph M m :

[0054] M m =F ω (p ij ,τ i ) (9)

[0055] The trained visible map represents the static and transient visibility, and decomposes the static and transient phenomena of the image in an unsupervised manner. The corresponding positions of the visible map predicted by the implicit function and the visible map obtained by the segmentation model are removed to the maximum value, and then the visible map is processed by morphological operations to obtain the final visible map.

[0056] S104: Render an image through volume rendering.

[0057] Specifically, use formula (7) as the basis for calculation of each sampling point, and use formula (5) to calculate the final pixel color.

[0058] The loss function of pixel color and final visible image M is as follows:

[0059]

[0060] Therefore, the final loss function is as follows:

[0061] L=λ0∑ i L a +∑ ij L m (12)

[0062] In the above two formulas, λ m and λ0 are both hyperparameters.

[0063] Finally, as the final stage of the process of an embodiment of the present invention, experimental verification verifies the feasibility and effect of the method proposed in the present invention through actual experiments.

[0064] In the experimental verification stage, the present invention selected other reconstruction methods for quantitative and qualitative experiments. In the quantitative experiment part, the model of the present invention achieved the best in all indicators. The quantitative experimental indicators are shown in Table 1. The calculated indicators are PSNR, SSIM, and LPIPS. The higher the PSNR and SSIM indicators, the better, and the lower the LPIPS indicator, the better; the training speed and rendering speed are shown in Table 2 and Table 3 respectively.

[0065] Table 1 Comparison results of the indicators of the method of the present invention and other methods on the Phototourism dataset.

[0066]

[0067] Table 2 Comparison results of training speed between the method of the present invention and other methods on the Phototourism dataset.

[0068]

[0069] Table 3 Rendering speed results of the method of the present invention and other methods on the Phototourism dataset.

[0070]

[0071] What is disclosed above is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the present invention. A person skilled in the art can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A three-dimensional reconstruction and rendering method based on an outdoor unconstrained image set, comprising the following steps: The input image is sampled through a random grid, and the appearance features are obtained through the appearance module; The appearance features, direction and position are decoded into RGB through the RGB module; Pass the input image through the visible map module to obtain the corresponding visible map; Pass the RGB and visible images through the rendering module to obtain the rendered image.

2. As described in claim 1, using random grid sampling to simplify light sampling, first, obtain the corresponding image, then randomly select a continuous area in the image for sampling and pass the image through a convolutional neural network to obtain appearance features, the number of sampled rays is 1024, and finally a 32x32 low-resolution image is formed.

3. The RGB module as claimed in claim 1, first constructing an occupancy grid, guiding sampling, and using multi-resolution hash coding to obtain three-dimensional point features; In order to skip sampling of blank areas, a k-scale occupancy grid cascade is maintained. Since the scene is outdoors, k=128 is set, and the resolution of each grid is 128. 3 , spanning a geometric growth domain centered at (0, 0, 0) [-2 k , 2 k ] 3 . During training, uniform sampling M = K·128 3 cells, without duplication, each voxel stores occupancy in the form of a single bit. The cells are arranged in a Morton curve, and it is worth noting that all cells are traversed at each step. As the ray moves forward, if the grid cell of the sample is high, the sample is placed, and if the grid cell of the sample is low, the grid is skipped. Given the coordinates of a 3D point in space p = (x, y, z), find the voxel corresponding to each resolution where the point is located, obtain the coordinates of the eight vertices of the voxel, use the coordinates to get the index through the hash function, store the features in the corresponding index, and then perform trilinear interpolation to get the features of the point. The features interpolated from each resolution are spliced ​​together to get the final features of the point. The sampling interval increases exponentially, and the sampling is terminated early according to the maximum number of samples and the threshold of the sampling point; Generally, outdoor scenes are relatively large, and we ultimately hope that the number of sampling points in the near distance is greater than the number of sampling points in the far distance, so that we can ensure the quality of sampling in the near distance and sample objects in the far distance. Assuming t is the distance that the light travels, the step size is set to Δ t = t / 256, this step size selection grows exponentially in t. The sampling interval is set to Where s is the size of the largest axis of the scene boundary, and the maximum number of samples per ray is 1024; At the same time, it is also calculated that when the transmittance of the light is lower than a certain threshold, the light is stopped from moving forward and the remaining contribution is set to 0, where the threshold ε is set to 1e-4.

4. The visible graph module as described in claim 1 obtains the corresponding visible graph by passing the input image through a classic image semantic segmentation pre-training model, and obtains the corresponding visible graph by using a classic text prompting method, and sends the pixel coordinates of the random grid sampling and its corresponding transient embedding into the implicit function to obtain its corresponding visible graph, and splices them into the final visible graph.

5. The rendering module as described in claim 1 uses volume rendering to supervise RGB and the final visible map to render the final image.

Images