An outdoor scene three-dimensional reconstruction method, device, electronic equipment and storage medium

Abstract

The application relates to an outdoor scene three-dimensional reconstruction method, device, electronic equipment and storage medium. The outdoor scene three-dimensional reconstruction method comprises the following steps: acquiring an outdoor scene image, and calculating a camera pose corresponding to the outdoor scene image; generating a visual light line for each pixel point in the outdoor scene image as an input of a neural radiation field; after encoding each visual light line, adding an appearance embedding vector and a transient embedding vector, and inputting the visual light line into a trained neural radiation field for neural rendering; based on an output result of the neural radiation field, for each visual light line, modeling is performed in a volume rendering mode to obtain an image of the outdoor scene image in a new observation direction. The outdoor scene three-dimensional reconstruction method can quickly and high-qualityly perform three-dimensional reconstruction on an outdoor landmark building by using multiple pictures taken around the landmark building.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a method, apparatus, electronic device, and storage medium for three-dimensional reconstruction of outdoor scenes. Background Technology

[0002] In recent years, more and more tourism businesses have begun to seek online development, with "virtual tours of the Forbidden City" being a relatively successful example. However, most of the currently popular online tourism adopts the form of web pages, which lacks interactivity.

[0003] With the rapid development of virtual reality technology, products combining virtual reality and tourism are gradually entering the public eye. Firstly, VR online tourism provides an immersive experience without leaving home. Secondly, VR online tourism can alleviate crowding to some extent, avoiding congestion during peak seasons like Golden Week. Finally, in terms of the travel experience, VR online tourism can simulate scenes not found in offline tourism, without the limitations of time and space; for example, visitors can freely choose scenes from different seasons or even ancient times.

[0004] The key technologies for VR online tourism are 3D reconstruction and 3D rendering. How to reconstruct a scene indistinguishable from the real-world environment is a crucial issue for VR online tourism. Technically speaking, this is a problem of generating new perspectives within a scene. It involves constructing a scene model using a series of images surrounding the scene, and then using this model to predict perspectives not present in the captured images. Currently, the mainstream approach is explicit 3D reconstruction, which involves first modeling the scene using point clouds and voxels, and then reconstructing the scene using rendering techniques. Many commercial products and open-source software utilize this technology, such as COLMAP. However, this method has several unresolved issues, such as the need for substantial storage space for point cloud information and the inability to handle reflective and transmissive planes effectively. While ray tracing technology can address these issues better, it incurs significant hardware overhead. Summary of the Invention

[0005] Based on this, the purpose of the present invention is to provide a method, apparatus, electronic device and storage medium for three-dimensional reconstruction of outdoor scenes, which uses multiple images taken around a landmark building to quickly and with high quality reconstruct the outdoor landmark building in three dimensions.

[0006] In a first aspect, the present invention provides a method for three-dimensional reconstruction of an outdoor scene, comprising the following steps:

[0007] Acquire outdoor scene images and calculate the camera pose corresponding to the outdoor scene images;

[0008] Each pixel in the outdoor scene image is used to generate a line of sight, which is then used as the input to the neural radiation field.

[0009] After encoding each line of sight ray, an appearance embedding vector and a transient embedding vector are added, and the vectors are input into a trained neural radiation field for neural rendering to obtain the color, volume density and uncertainty of each coordinate point on the line of sight ray.

[0010] Based on the output of the neural radiation field, for each line of sight ray, a volume rendering method is used to model it and obtain the color corresponding to each line of sight ray;

[0011] Based on the color corresponding to each line of sight, an image of the outdoor scene is obtained from the new viewing direction.

[0012] Furthermore, the input to the neural radiation field is the scene voxel coordinates, the light source direction, and the viewing direction;

[0013] The scene voxel coordinates and the viewing direction are represented by the sensing ray, and the light source direction is represented by the illumination ray;

[0014] The sensed light and the illumination light are calculated from the line-of-sight light using the camera pose.

[0015] Furthermore, after encoding each of the aforementioned gaze rays, an appearance embedding vector and a transient embedding vector are added, and then input into the trained neural radiation field for neural rendering, including the following steps:

[0016] After concatenating the appearance embedding vector with the encoded gaze ray, the concatenation is input into the static neural radiation field for neural rendering to obtain the static volume density and static color corresponding to the gaze ray.

[0017] The transient embedding vector is concatenated with the encoded line-of-sight ray and then input into the transient neural radiation field for neural rotation to obtain the transient volume density, transient color, and uncertainty corresponding to the line-of-sight ray.

[0018] Furthermore, encoding each of the aforementioned line-of-sight rays includes the following steps:

[0019] Multi-resolution hash encoding is used to sample the input line of sight at different sampling rates. The sampling rate N is controlled by three hyperparameters: Nmin, Nmax, and L. Nmax is the maximum value of N, Nmin is the minimum value of N, and N is L values ​​between Nmin and Nmax.

[0020] Before training, random feature values ​​are set at different sampling rates;

[0021] During training, a hash table is used to store feature values, and feature values ​​at different sampling rates are concatenated to obtain the result of multi-resolution hash encoding.

[0022] Furthermore, using the following formula, based on the output of the neural radiation field, for each of the line-of-sight rays, a volumetric rendering method is used to model the image and obtain the color corresponding to each line-of-sight ray:

[0023]

[0024]

[0025] For each line of sight ray r(t), the final rendered color can be represented as C(r), where T(t) represents the cumulative transparency of the ray from tn to t, σ is the static volume density calculated by neural rendering, c is the static color calculated by neural rendering, σ(τ) is the transient volume density calculated by neural rendering, and c(τ) is the transient color calculated by neural rendering.

[0026] Furthermore, the loss function of the neural radiation field is:

[0027]

[0028] in, It is the mean square error that incorporates uncertainty. It excludes the uncertainty from reaching a minimum at infinity. This is to prevent the model from using transient density to explain static phenomena; β is the uncertainty calculated by neural rendering. σ(τ) is the sum of transient and static colors, C is the color of the input image, σ(τ) is the transient volume density calculated by neural rendering, and λ is a hyperparameter.

[0029] Secondly, the present invention also provides an outdoor scene three-dimensional reconstruction device, comprising:

[0030] An outdoor scene image acquisition module is used to acquire outdoor scene images and calculate the camera pose corresponding to the outdoor scene images;

[0031] The light generation module is used to generate a line of sight ray for each pixel in the outdoor scene image, as the input of the neural radiation field;

[0032] The neural rendering module is used to encode each of the aforementioned line of sight, add appearance embedding vector and transient embedding vector, and input them into a trained neural radiation field for neural rendering to obtain the color, volume density and uncertainty of each coordinate point on the line of sight.

[0033] The volume rendering module is used to model each line of the line of sight using volume rendering based on the output of the neural radiation field, and to obtain the color corresponding to each line of sight.

[0034] The result generation module is used to obtain the image of the outdoor scene from the new viewing direction based on the color corresponding to each line of sight ray.

[0035] Thirdly, the present invention also provides an electronic device, comprising:

[0036] At least one memory and at least one processor;

[0037] The memory is used to store one or more programs;

[0038] When the one or more programs are executed by the at least one processor, the at least one processor performs the steps of an outdoor scene three-dimensional reconstruction method as described in any of the first aspects of the present invention.

[0039] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of an outdoor scene three-dimensional reconstruction method as described in any of the first aspects of the present invention.

[0040] This invention provides a method, apparatus, electronic device, and storage medium for 3D reconstruction of outdoor scenes. It employs multi-resolution hashing, a highly efficient encoding method, to achieve rapid modeling. Uncontrollable factors in the outdoor environment are mainly twofold: first, uncontrollable factors such as illumination color difference, which is addressed by appearance embedding; and second, the problem of temporary occlusion by pedestrians and vehicles, which is addressed by transient embedding. The overall training approach involves inputting generated light rays into a neural radiation field through multi-resolution hashing, rendering the output to obtain a generated image, and then calculating the loss function between the generated image and the original image. Through continuous training, the loss function is minimized. Experiments show that in the presence of occlusion, the reconstruction result of the original neural radiation field will have ghosting, while the reconstruction result using the method of this application effectively eliminates ghosting. Furthermore, compared with existing models, the 3D reconstruction method for outdoor scenes provided in this application significantly reduces reconstruction time while improving reconstruction quality.

[0041] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0042] Figure 1 This is a schematic diagram illustrating the steps of a three-dimensional reconstruction method for an outdoor scene provided by the present invention;

[0043] Figure 2 This is a schematic diagram of the model structure used in a preferred embodiment;

[0044] Figure 3 This is a schematic diagram of the process of training and optimizing a model in one embodiment;

[0045] Figure 4 This is an experimental flowchart illustrating the experimentation and adjustment of a 3D reconstruction method for an outdoor scene in one embodiment.

[0046] Figure 5 The image shows a comparison of the results of three-dimensional reconstruction of an outdoor scene in one embodiment. The left image is the result of reconstruction of the original neural radiation field, and the right image is the result of reconstruction by the method provided in this application.

[0047] Figure 6 This is a line graph showing the change in PSNR of the reconstruction results obtained by the three models over training time in a comparative experiment.

[0048] Figure 7 This is a schematic diagram of the structure of an outdoor scene three-dimensional reconstruction device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0050] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0051] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0052] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0053] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0054] To address the problems in the background art, embodiments of this application provide a method for three-dimensional reconstruction of outdoor scenes, such as... Figure 1 and Figure 2 As shown, the method includes the following steps:

[0055] S01: Acquire an outdoor scene image and calculate the camera pose corresponding to the outdoor scene image.

[0056] Pose, also known as position or orientation, refers to an object's location and orientation in space. Camera pose, in this context, refers to the camera's location and orientation in space. A camera's pose can be viewed as a transformation from its original position to its current position, involving a translation transformation and a rotation transformation (rotation first, then translation). This transformation can be described separately or together. Describing it separately involves treating rotation and translation as two separate processes; describing it together involves treating rotation and translation as a single, complete process.

[0057] Estimating the camera pose from an input image is a camera pose estimation problem. This involves using several feature points with known coordinates and their images within the camera image to determine the camera's coordinates and rotation angles within a coordinate system. This algorithm is implemented in the COLMAP open-source software and can be directly called, outputting a transformation matrix from natural coordinates to camera coordinates. The accuracy of this transformation matrix calculation is a key factor affecting the final output quality.

[0058] S02: Generate a line of sight ray for each pixel in the outdoor scene image, and use it as the input of the neural radiation field.

[0059] A ray can typically be represented as a beam. In computer graphics, a ray can be represented as o+td, where o represents the coordinates of the light source, d represents the direction vector of the ray's propagation, and t represents time. The input to the neural radiation field requires coordinates, the direction of the light source, and the viewing direction; these three inputs are represented by rays. Each pixel in the image generates a horizontal ray, and the perceived ray and the illumination ray are obtained through a transformation matrix from natural coordinates to camera coordinates. This transformation matrix is ​​obtained by the pose estimation algorithm in COLMAP during the data preprocessing stage.

[0060] S03: After encoding each line of sight ray, add appearance embedding vector and transient embedding vector, input them into the trained neural radiation field for neural rendering, and obtain the color, volume density and uncertainty of each coordinate point on the line of sight ray.

[0061] Regarding the encoding of light rays, this application compared frequency encoding and multi-resolution hash encoding in preliminary experiments. In a preferred embodiment, multi-resolution hash encoding, which has a shorter testing time, was selected.

[0062] Specifically, multi-resolution hashing first samples the input light rays at different sampling rates. The sampling rate N is controlled by three hyperparameters: Nmin, Nmax, and L. Nmax is the maximum value of N, Nmin is the minimum value of N, and N is any of the L values ​​between Nmin and Nmax. Before training, random feature values ​​are set at different sampling rates. These feature values ​​are trainable parameters that will continuously fit the scene during training. The feature values ​​are stored using a hash table. By concatenating the feature values ​​from different sampling rates, the multi-resolution hashing result is obtained. Using multi-resolution hashing can reduce the size of the network model without compromising the reconstruction results, thus accelerating the training speed.

[0063] Neural rendering involves encoding the generated light rays and inputting them into a neural radiation field for calculation. The neural radiation field consists of a multi-layered perceptron. The inputs are the scene voxel coordinates (x, y, z) (obtained by uniformly sampling points on the light rays), the light source direction (θ), and the viewing direction (φ). The input coordinates and viewing direction are represented by perceptual rays, while the light source direction is represented by illumination rays. The outputs include the colors R, G, and B of the coordinates on the light rays, the volume density σ, and the uncertainty β.

[0064] To address the issues of uncertain lighting conditions and transient occlusion by pedestrians and vehicles in outdoor scenes, this application introduces appearance embedding vectors and transient embedding vectors in neural rendering to solve the problem of reduced scene reconstruction quality of neural radiation fields caused by lighting changes and transient occlusion.

[0065] Appearance embedding and transient embedding represent an image as a vector, with calculations similar to word embedding. The specific structure involves inputting the encoded perceived light and appearance embedding into a multilayer perceptron (MLP) for calculating volume density. The MLP contains untrained parameters, which are continuously optimized during training to reduce the loss function value to the training objective. The output of the MLP is set as volume density and a feature vector, where the feature vector is the result of the MLP's calculation. The feature vector, illumination light, and transient embedding are then input into a multilayer perceptron with trained parameters for calculating color. The untrained parameters are continuously optimized during training to reduce the loss function value to the training objective. The output of the multilayer perceptron is set as the color and uncertainty of the coordinate points.

[0066] Preferably, the loss function is as follows:

[0067]

[0068] in, It is the mean square error that incorporates uncertainty. It excludes the uncertainty from reaching a minimum at infinity. This is to prevent the model from using transient density to explain static phenomena; β is the uncertainty calculated by neural rendering. σ(τ) is the sum of transient and static colors, C is the color of the input image, σ(τ) is the transient volume density calculated by neural rendering, and λ is a hyperparameter.

[0069] S04: Based on the output of the neural radiation field, for each line of sight ray, a volume rendering method is used to model it and obtain the color corresponding to each line of sight ray.

[0070] Volume rendering utilizes classic stereoscopic rendering principles, rendering along each ray in the viewing direction. For the rendering of a single ray, volume rendering multiplies and sums the color and opacity of points along that ray, as shown in the formula.

[0071]

[0072] in,

[0073] For each camera ray r(t), the final rendered color can be represented as C(r), where T(t) represents the cumulative transparency of the ray from tn to t, which can be understood as the probability that the ray can pass through from tn to t. σ is the volume density calculated by neural rendering, and c is the color calculated by neural rendering.

[0074] In the method of this application, since the output of neural rendering adds transient color and transient volume density, the volume rendering used in this application adopts the form used in the NeRF-W model, and the color corresponding to each line of sight is obtained using the following formula:

[0075]

[0076] in,

[0077] For each camera ray r(t), the final rendered color can be represented as C(r), where T(t) represents the cumulative transparency of the ray from tn to t, σ is the static volume density calculated by neural rendering, c is the static color calculated by neural rendering, σ(τ) is the transient volume density calculated by neural rendering, and c(τ) is the transient color calculated by neural rendering.

[0078] S05: Based on the color corresponding to each line of sight ray, obtain the image of the outdoor scene in the new viewing direction.

[0079] like Figure 3 and Figure 4 As shown, the 3D reconstruction model used in this application is mainly divided into a training phase and a testing phase, followed by a series of comparative and ablation experiments on the model. The main problems to be solved are twofold: firstly, to achieve speed; and secondly, to eliminate the impact of uncontrollable factors in the outdoor environment on the reconstruction quality. The plan is to use multi-resolution hash encoding, an efficient encoding method, to achieve rapid modeling. Furthermore, the uncontrollable factors in the outdoor environment are mainly manifested in two aspects: firstly, the uncontrollable factor of illumination color difference, which is solved by appearance embedding; and secondly, the problem of temporary occlusion by pedestrians and vehicles, which is solved by transient embedding. The overall training approach is to input the generated light rays into the neural radiation field through multi-resolution hash encoding, and then render the output to obtain the generated image. The loss function is then calculated between the generated image and the original image, and through continuous training, the loss function is minimized.

[0080] Specifically, the dataset is first divided into training and testing sets. During training, light rays are generated using the transformation matrix produced by COLMAP. These rays are then encoded and input into the neural radiation field model, which calculates information such as the ray's color, volume density, and uncertainty. The output is then volume-rendered to obtain the output image. During testing, the generated light rays are input into the neural radiation field model with pre-trained parameters to predict test images, and the quality of the generated test images is assessed.

[0081] After building the model, a series of comparative and ablation experiments were conducted. The purpose of these experiments was to adjust the network parameters to achieve optimal performance and to verify that the built model improved upon previous models, demonstrating enhanced performance across various metrics. In these experiments, the model's performance was evaluated across two dimensions: training and testing time, and reconstruction quality. The training and testing time was measured by time itself, while reconstruction quality was measured by both quantitative and qualitative metrics. Quantitative metrics included PSNR, SSIM, and PSINS, while qualitative metrics included the reconstructed image and its depth map information.

[0082] The first experiment is a comparative study of encoding methods, selecting different encoding techniques such as frequency encoding and multi-resolution hashing. Since encoding primarily affects training and testing time, time will be used as the performance metric. The second experiment is an ablation study to optimize for uncontrollable factors. The optimization methods for uncontrollable factors mainly employ appearance embedding and transient embedding, with reconstruction quality as the primary performance metric. The third experiment is a comparative study of network structure optimization, modifying the depth and width of the multilayer perceptron and evaluating both time and reconstruction quality. The fourth experiment is a comparative study with previous models, verifying the effectiveness of the proposed network in improving the model through a comparison of reconstruction quality and time.

[0083] Because this application incorporates appearance embedding and transient embedding on top of the neural radiation field, it decouples the illumination of each image from its 3D features and reduces the impact of severely occluded images on the final result. In a specific reconstruction result, such as... Figure 5 As shown, the left image is the result of reconstructing the original neural radiation field. In the presence of occlusion, afterimages of the occluder will appear. The right image is the result of reconstruction using the method of this patent, which effectively eliminates afterimages.

[0084] Regarding the reconstruction time, the NeRF-W neural radiation field variant for outdoor scenes requires up to 10 hours for reconstruction. The reconstruction time and quality of this patented model, based on a dataset of self-captured images, are shown in the table below. This table compares the reconstruction time and quality of NeRF, Instant-ngp, and our model.

[0085] PSNR (dB) Time (s) NeRF 16.40 7200 Instant-ngp 20.78 240 Ours 24.10 240

[0086] PSNR, or Peak Signal-to-Noise Ratio, is an engineering term that represents the ratio of the maximum possible power of a signal to the power of destructive noise that affects its representation accuracy. Because many signals have very wide dynamic ranges, PSNR is often expressed in logarithmic decibels (dB).

[0087] To calculate PSNR, you first need to know how to calculate MSE (mean squared error). For two m×n monochrome images I and K, if one is a noisy approximation of the other, then their mean squared error is defined as:

[0088]

[0089] PSNR is derived from MSE, using the following formula:

[0090]

[0091] Here, MAXI represents the maximum value of the color of an image point. If each sample point is represented by 8 bits, then it is 255.

[0092] Therefore, the smaller the MSE, the larger the PSNR; and the larger the PSNR, the better the image quality.

[0093] like Figure 6 As shown, the PSNR value is taken from the value when it tends to be stationary, and the time is the value when the PSNR tends to be stationary. The experimental data for NeRF and Instant-ngp are self-constructed datasets implemented on NeRF and Instant-ngp on JNeRF.

[0094] This application also provides an outdoor scene 3D reconstruction device, such as... Figure 5 As shown, the outdoor scene 3D reconstruction device 400 includes:

[0095] The outdoor scene image acquisition module 401 is used to acquire outdoor scene images and calculate the camera pose corresponding to the outdoor scene images.

[0096] The light generation module 402 is used to generate a line of sight ray for each pixel in the outdoor scene image, as the input of the neural radiation field;

[0097] The neural rendering module 403 is used to encode each of the aforementioned line of sight, add appearance embedding vector and transient embedding vector, and input them into the trained neural radiation field for neural rendering to obtain the color, volume density and uncertainty of each coordinate point on the line of sight.

[0098] The volume rendering module 404 is used to model each line of the line of sight using volume rendering based on the output of the neural radiation field, and to obtain the color corresponding to each line of sight.

[0099] The result generation module 405 is used to obtain the image of the outdoor scene image in the new viewing direction based on the color corresponding to each line of sight ray.

[0100] Preferably, the input to the neural radiation field is the scene voxel coordinates, the light source direction, and the viewing direction;

[0101] The scene voxel coordinates and the viewing direction are represented by the sensing ray, and the light source direction is represented by the illumination ray;

[0102] The sensed light and the illumination light are calculated from the line-of-sight light using the camera pose.

[0103] Preferably, the neural rendering module includes:

[0104] An appearance embedding unit is used to concatenate the appearance embedding vector with the encoded gaze ray and input it into a static neural radiation field for neural rendering to obtain the static volume density and static color corresponding to the gaze ray.

[0105] The transient embedding unit is used to concatenate the transient embedding vector with the encoded line-of-sight ray and input it into the transient neural radiation field for neural rotation to obtain the transient volume density, transient color and uncertainty corresponding to the line-of-sight ray.

[0106] Preferably, the neural rendering module includes:

[0107] The sampling unit is used to sample the input line-of-sight ray at different sampling rates using multi-resolution hash encoding. The sampling rate N is controlled by three hyperparameters: Nmin, Nmax, and L. Nmax is the maximum value of N, Nmin is the minimum value of N, and N is L values ​​between Nmin and Nmax.

[0108] The feature value setting unit is used to set random feature values ​​at different sampling rates before training.

[0109] The encoding result generation unit is used during the training process to store feature values ​​using a hash table, concatenate feature values ​​at different sampling rates, and obtain multi-resolution hash encoding results.

[0110] Preferably, based on the output of the neural radiation field, the following formula is used to model each line of the line of sight using volume rendering to obtain the color corresponding to each line of the line of sight:

[0111]

[0112]

[0113] For each line of sight ray r(t), the final rendered color can be represented as C(r), where T(t) represents the cumulative transparency of the ray from tn to t, σ is the static volume density calculated by neural rendering, c is the static color calculated by neural rendering, σ(τ) is the transient volume density calculated by neural rendering, and c(τ) is the transient color calculated by neural rendering.

[0114] Preferably, the loss function of the neural radiation field is:

[0115]

[0116] in, It is the mean square error that incorporates uncertainty. It excludes the uncertainty from reaching a minimum at infinity. This is to prevent the model from using transient density to explain static phenomena; β is the uncertainty calculated by neural rendering. σ(τ) is the sum of transient and static colors, C is the color of the input image, σ(τ) is the transient volume density calculated by neural rendering, and λ is a hyperparameter.

[0117] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0118] This application also provides an electronic device, including:

[0119] At least one memory and at least one processor;

[0120] The memory is used to store one or more programs;

[0121] When the one or more programs are executed by the at least one processor, the at least one processor performs the steps of the outdoor scene 3D reconstruction method as described above.

[0122] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0123] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the outdoor scene 3D reconstruction method described above.

[0124] Computer storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0125] This invention provides a method, apparatus, electronic device, and storage medium for 3D reconstruction of outdoor scenes. It employs multi-resolution hashing, a highly efficient encoding method, to achieve rapid modeling. Uncontrollable factors in the outdoor environment are mainly twofold: first, uncontrollable factors such as illumination color difference, which is addressed by appearance embedding; and second, the problem of temporary occlusion by pedestrians and vehicles, which is addressed by transient embedding. The overall training approach involves inputting generated light rays into a neural radiation field through multi-resolution hashing, rendering the output to obtain a generated image, and then calculating the loss function between the generated image and the original image. Through continuous training, the loss function is minimized. Experiments show that in the presence of occlusion, the reconstruction result of the original neural radiation field will have ghosting, while the reconstruction result using the method of this application effectively eliminates ghosting. Furthermore, compared with existing models, the 3D reconstruction method for outdoor scenes provided in this application significantly reduces reconstruction time while improving reconstruction quality.

[0126] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for three-dimensional reconstruction of an outdoor scene, characterized in that, Includes the following steps: Acquire outdoor scene images and calculate the camera pose corresponding to the outdoor scene images; Each pixel in the outdoor scene image is used to generate a line of sight, which is then used as the input to the neural radiation field. After encoding each line of sight ray, an appearance embedding vector and a transient embedding vector are added, and the vectors are input into a trained neural radiation field for neural rendering to obtain the color, volume density and uncertainty of each coordinate point on the line of sight ray. Based on the output of the neural radiation field, for each line of sight ray, a volume rendering method is used to model it and obtain the color corresponding to each line of sight ray; Based on the color corresponding to each line of sight, the image of the outdoor scene is obtained in the new viewing direction; After encoding each of the aforementioned line-of-sight rays, an appearance embedding vector and a transient embedding vector are added, and then input into a trained neural radiation field for neural rendering, including the following steps: After concatenating the appearance embedding vector with the encoded gaze ray, the concatenation is input into the static neural radiation field for neural rendering to obtain the static volume density and static color corresponding to the gaze ray. After concatenating the transient embedding vector with the encoded line-of-sight ray, the vector is input into the transient neural radiation field for neural rotation to obtain the transient volume density, transient color, and uncertainty corresponding to the line-of-sight ray. Encoding each of the aforementioned line-of-sight rays includes the following steps: Multi-resolution hash encoding is used to sample the input line of sight at different sampling rates. The sampling rate N is controlled by three hyperparameters: Nmin, Nmax, and L. Nmax is the maximum value of N, Nmin is the minimum value of N, and N is L values ​​between Nmin and Nmax. Before training, random feature values ​​are set at different sampling rates; During training, a hash table is used to store feature values, and feature values ​​at different sampling rates are concatenated to obtain the result of multi-resolution hash encoding.

2. The method for three-dimensional reconstruction of an outdoor scene according to claim 1, characterized in that: The inputs to the neural radiation field are scene voxel coordinates, light source direction, and viewing direction; The scene voxel coordinates and the viewing direction are represented by the sensing ray, and the light source direction is represented by the illumination ray; The sensed light and the illumination light are calculated from the line-of-sight light using the camera pose.

3. The method for three-dimensional reconstruction of an outdoor scene according to claim 1, characterized in that, Using the following formula, based on the output of the neural radiation field, for each of the visual rays, a volumetric rendering method is used to model and obtain the color corresponding to each visual ray: For each line of sight ray r(t), the final rendered color can be represented as C(r), where T(t) represents the cumulative transparency of the ray from tn to t, σ is the static volume density calculated by neural rendering, c is the static color calculated by neural rendering, σ(τ) is the transient volume density calculated by neural rendering, and c(τ) is the transient color calculated by neural rendering.

4. The method for three-dimensional reconstruction of an outdoor scene according to claim 1, characterized in that, The loss function of the neural radiation field is: in, It is the mean square error that incorporates uncertainty. It excludes the uncertainty from reaching a minimum at infinity. This is to prevent the model from using transient density to explain static phenomena; β is the uncertainty calculated by neural rendering. σ(τ) is the sum of transient and static colors, C is the color of the input image, σ(τ) is the transient volume density calculated by neural rendering, and λ is a hyperparameter.

5. An outdoor scene three-dimensional reconstruction apparatus for performing the outdoor scene three-dimensional reconstruction method as described in claim 1, characterized in that, include: An outdoor scene image acquisition module is used to acquire outdoor scene images and calculate the camera pose corresponding to the outdoor scene images; The light generation module is used to generate a line of sight ray for each pixel in the outdoor scene image, as the input of the neural radiation field; The neural rendering module is used to encode each of the aforementioned line of sight, add appearance embedding vector and transient embedding vector, and input them into a trained neural radiation field for neural rendering to obtain the color, volume density and uncertainty of each coordinate point on the line of sight. The volume rendering module is used to model each line of the line of sight using volume rendering based on the output of the neural radiation field, and to obtain the color corresponding to each line of sight. The result generation module is used to obtain the image of the outdoor scene from the new viewing direction based on the color corresponding to each line of sight ray.

6. An electronic device, characterized in that, include: At least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the outdoor scene three-dimensional reconstruction method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the outdoor scene three-dimensional reconstruction method as described in any one of claims 1-4.

Images