Uncertainty-Aware Fusion for Large-Scale NeRF

Abstract

A method for achieving high-fidelity novel viewpoint synthesis and 3D reconstruction of large-scale scenes is disclosed. The method includes acquiring images from video streams received from multiple video image capture devices, grouping the images into separate image clusters representing the large-scale 3D scene, and training a neural radiance field (NeRF) and an uncertainty multilayer perceptron (MLP). For each of the image clusters, generate multiple NeRFs and multiple uncertainty MLPs for the large-scale 3D scene, apply rendering loss and entropy loss to the multiple NeRFs, and perform uncertainty-based fusion on the multiple NeRFs. Define a fused NeRF, jointly fine-tune the multiple NeRFs and the multiple uncertainty MLPs, and apply the fused NeRF to novel viewpoint synthesis of the large-scale 3D scene during inference.

Description

[Technical field]

[0001] The present invention relates to 3D scene reconstruction, and more particularly to uncertainty-aware fusion for large-scale Neural Radiance Fields (NeRFs). [Background technology]

[0002] 2. Description of Related Art Neural implicit functions have proven to be an effective way to achieve 3D scene representation. Neural Radiance Field (NeRF) has emerged as a powerful tool to represent the shape and radiance of 3D space as continuous functions. This allows for an unprecedented level of fidelity in the synthesis of novel viewpoints. At its core, NeRF utilizes a multi-layer perceptron (MLP) to encode the color and density of each 3D point viewed from different directions. Despite its effectiveness, NeRF has been limited to small scenes so far due to the limited capacity of MLP. In other words, NeRF must trade off scale and resolution at a constant capacity. Summary of the Invention

[0003] A method is presented for realizing high-fidelity novel view synthesis and 3D reconstruction for large-scale scenes, the method includes acquiring images from video streams received from multiple video image capture devices, grouping the images into separate image clusters representing a large-scale 3D scene, training a neural radiance field (NeRF) and an uncertainty multilayer perceptron (MLP) for each of the image clusters to generate multiple NeRFs and multiple uncertainty MLPs for the large-scale 3D scene, applying a rendering loss and an entropy loss to the multiple NeRFs, performing an uncertainty-based fusion on the multiple NeRFs to define a fused NeRF, jointly fine-tuning the multiple NeRFs and the multiple uncertainty MLPs, and applying the fused NeRF to novel view synthesis of the large-scale 3D scene during inference.

[0004] A non-transitory computer-readable storage medium is presented that includes a computer-readable program for realizing high-fidelity novel viewpoint synthesis and 3D reconstruction for large-scale scenes, which, when executed on a computer, causes the computer to perform the steps of acquiring images from video streams received from multiple video image capture devices, grouping the images into separate image clusters representing a large-scale 3D scene, training a neural radiance field (NeRF) and an uncertainty multilayer perceptron (MLP) for each of the image clusters to generate multiple NeRFs and multiple uncertainty MLPs for the large-scale 3D scene, applying a rendering loss and an entropy loss to the multiple NeRFs, performing an uncertainty-based fusion on the multiple NeRFs to define a fused NeRF, jointly fine-tuning the multiple NeRFs and the multiple uncertainty MLPs, and applying the fused NeRF to a novel viewpoint synthesis of the large-scale 3D scene during inference.

[0005] A system for achieving high-fidelity novel view synthesis and 3D reconstruction for large-scale scenes is presented, the system includes a memory and one or more processors in communication with the memory, configured to acquire images from video streams received from multiple video image capture devices, group the images into separate image clusters representing a large-scale 3D scene, train a neural radiance field (NeRF) and an uncertainty multilayer perceptron (MLP) for each of the image clusters to generate multiple NeRFs and multiple uncertainty MLPs for the large-scale 3D scene, apply a rendering loss and an entropy loss to the multiple NeRFs, perform an uncertainty-based fusion on the multiple NeRFs to define a fused NeRF, jointly fine-tune the multiple NeRFs and the multiple uncertainty MLPs, and apply the fused NeRF to novel view synthesis of the large-scale 3D scene during inference.

[0006] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0007] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0008] [Figure 1] FIG. 1 is a block / flow diagram of an exemplary uncertainty-based fusion of NeRF for large-scale 3D scene representations, in accordance with an embodiment of the present invention.

[0009] [Diagram 2] FIG. 2 is a block / flow diagram of an exemplary large scale 3D scene representation according to an embodiment of the present invention.

[0010] [Diagram 3]FIG. 1 is a block / flow diagram of a practical application for achieving high-fidelity novel view synthesis and 3D reconstruction for large-scale scenes according to an embodiment of the present invention.

[0011] [Figure 4] FIG. 1 illustrates an exemplary processing system for achieving high-fidelity novel view synthesis and 3D reconstruction for large scenes, in accordance with an embodiment of the present invention.

[0012] [Diagram 5] FIG. 2 is a block / flow diagram of an exemplary method for achieving high-fidelity novel view synthesis and 3D reconstruction for large scenes, in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Implicit neural representations of 3D scenes have made great advances in recent years, including novel viewpoint synthesis (viewpoint-free rendering). In particular, Neural Radiance Field (NeRF) stands out as a powerful representation. Building on classical volumetric rendering, NeRF uses multi-layer perceptrons (MLPs) to regress volumetric density and color within a 3D volume, achieving unprecedented levels of fidelity in rendering complex scenes with viewpoint-dependent effects.

[0014] Although impressive, NeRFs represented as a single MLP also have drawbacks. First, NeRFs suffer from an inherent scale-resolution tradeoff due to the limited capacity of a single MLP and are therefore limited to small-scale scenes. Second, representing the entire scene as a single MLP does not lend itself to flexible allocation of different capabilities to different parts of the scene depending on their appearance complexity, or to varying the granularity of the representation across the space depending on practical needs. The exemplary embodiments show that these issues can be mitigated if a robust method for fusing multiple NeRFs is available, and we propose to regress an implicit uncertainty field towards this goal.

[0015] The problem of scalability has been relatively little studied in NeRFs. What exists are volumetric (or spatial) decomposition strategies. These methods achieve faster learning speeds by partitioning the space in a uniform way and representing each constituent region with a separate NeRF. This strategy has been applied to scale up 3D shape fitting from point clouds, but it is not as convenient when learning implicit representations from images alone. That is, as soon as rendering is done along the camera ray, it is tightly bound to a point in 3D space. Instead of spatial clustering or decomposition, the exemplary embodiment approaches the scalability problem from an alternative perspective: a synthetic NeRF based on image clustering. This is inspired by CMVS, a classical large-scale multi-view stereo system, which establishes locality by image clustering. Here, we assume that images are intentionally taken such that they form clusters, each focused on a part of the space, with overlap between each part. Image-level decomposition has the advantage that individual NeRFs can be trained separately in parallel. We observe that this separation results in higher fidelity as each NeRF is fully centered on the region covered by an image cluster.

[0016] Such synthetic NeRFs based on image clustering are also a convenient way to scale the representation capacity and granularity of the entire space: separate NeRFs can be trained for different image clusters of interest at different levels of resolution.

[0017] Despite its advantages in quality and flexibility, separate training requires an additional step of merging all the individual NeRFs to enable viewpoint-free rendering across domains of different NeRFs as if they were one NeRF. To this end, the exemplary embodiment uses MLP to regress the uncertainty field as a 6D continuous function. The uncertainty network takes as input 6D coordinates including spatial location, gaze direction, and distance, and outputs the uncertainty of the individual NeRFs. The resulting uncertainty quantification is employed for NeRF fusion. Moreover, the exemplary embodiment introduces an effective strategy for NeRF stitching without access to training images. It is also worth mentioning that this exemplary method opens the door to applications of fusing and augmenting crowdsourced NeRFs.

[0018] At the very least, our synthetic NeRF based on image clustering contributes to increasing the scalability and granularity flexibility of NeRF.The exemplary embodiment presents a NeRF fusion scheme by regressing a 6D uncertainty field using MLP and demonstrates state-of-the-art quality for viewpoint-free rendering in real-world room-level indoor scenes.

[0019] NeRF represents the scene as an implicit neural radiance field that contains volumetric density o and directional radiance c. The radiance field, realized by an MLP, is optimized by volumetric rendering followed by an image reconstruction loss. Specifically, given a pixel photographed from the camera center o along a viewing direction d, the volumetric rendering color C is given by

number

[0020] Here, r(t)=o+td, and [t n ,t f ] and

number

[0021] Regarding synthetic NeRF based on image clustering, the exemplary embodiment studies a problem setting in which images are intentionally captured to contain multiple image clusters that each focus on a region of the scene and overlap each other. In particular, we describe the following two configurations:

[0022] Local-local NeRF is a simple scenario where all of the individual NeRFs are on an equal footing: each image cluster is equally focused on a subspace of the scene at the same desired resolution.

[0023] In global-local NeRFs, it is not desirable to reconstruct the entire scene at the same level of granularity, which is a preference. For example, in an indoor environment, it may be desirable for the NeRF to capture only the overall structure of the entire room, while providing the ability to zoom in to inspect more detailed parts of the area of ​​interest. In an exemplary embodiment, a global NeRF and a local NeRF are separately utilized to capture the scene at each granularity.

[0024] This framework is motivated by an important observation: for pixels that are well covered by individual image clusters, the corresponding individual NeRFs have higher fidelity than a single NeRF trained on all images, due to the former's higher density. However, artifacts quickly appear as soon as the camera viewpoint crosses the regions of different individual NeRFs, a scenario that is unavoidable in viewpoint-free rendering. Thus, it remains an important question how to seamlessly stitch together individual NeRFs for viewpoint-free rendering. This exemplary framework is also closely related to recent spatial clustering-based composite NeRFs, e.g., Decomposed Radiance Fields (DeRF).

[0025] Regarding learning NeRF stitching, the first heuristic way to stitch together each NeRF is a one-shot selection based on visibility. That is, for each point along the ray being rendered, we can choose to query only the NeRF with the maximum number of training images that display that point. Furthermore, we argue that the selection of NeRFs in such a pipeline should also consider the gaze direction and distance. Intuitively, visibility counts should downgrade images observing points far from the viewpoint being rendered, and vice versa. Furthermore, we observe that while this approach works to some extent, it is far from satisfactory, mainly due to the lack of principled learning and optimization.

[0026] For the implicit uncertainty field, the exemplary method proposes to learn NeRF stitching by regressing the implicit uncertainty field as a 6D continuous function. Implemented as an MLP, this uncertainty network takes as input the spatial location (x,y,z) of a point, the gaze direction d, and the distance t.

number

[0027] The individual NeRFs are then stitched together and volumetrically rendered as follows:

number

number

[0028] The network is trained with the loss of image reconstruction.

number

[0029] Note that individual NeRFs are pre-trained in parallel for each image cluster, and only the uncertainty MLP is optimized. Furthermore, position encoding and quadrature are applied.

[0030] Even with respect to the image-less implicit uncertainty field, the exemplary method learns the uncertainty field without access to the training images of each NeRF for two reasons. First, this is illuminating in understanding the underlying behavior of the previous framework with images, as detailed later. Second, the lightweight nature of NeRFs makes them suitable for scene compression purposes, so images may not be stored after training if storage is a concern. We employ a simple and effective viewpoint voting strategy that uses only the camera poses of the training images. At each training iteration, the exemplary embodiment randomly selects an image and a pixel therein to input points along rays into the uncertainty MLP with associated 6D coordinates.

[0031] Assuming that the selected image belongs to image cluster k, the network output is directly trained with the loss

number

[0032] Here, w vote Is W k = l, which is a one-hot vector. In essence, each iteration casts one vote from the image cluster for a spatial point seen from a particular direction and distance. As training progresses, the uncertainty network must balance the competition between each source of power and find a smooth function that best describes the viewpoint distribution.

[0033] With regard to its relationship with the spatial clustering-based approach DeRF, it is argued that the proposed image clustering-based decomposition has several advantages and is also amenable to distributed computing. Indeed, the sequential nature of the exemplary method makes it, in principle, suboptimal compared to joint training on all images. However, it has been empirically observed that the benefits of the high fidelity of individual NeRFs outweigh the potential deficiencies arising from the sequential procedure and imperfect stitching.

[0034] Despite this discrepancy, we note that our goal is not to exclude spatial clustering-based methods such as DeRF. Rather, the strengths of DeRF are in fact compatible with our exemplary method in that they can also be applied inside individual NeRFs.

[0035] FIG. 1 is a block / flow diagram of an exemplary uncertainty-based fusion of NeRF for large-scale 3D scene representations, in accordance with an embodiment of the present invention.

[0036] FIG. 1 shows the proposed NeRF uncertainty-based fusion pipeline 100 (or image clustering-based composite NeRF scheme 100) for large-scale 3D scene representation. A camera 110 captures a set of images 120 that completely cover a scene (e.g., an apartment). The images 120 are grouped into respective clusters 130, each focusing on a small part of the scene. For each image cluster 130, the exemplary method trains individual NeRFs in parallel, similar to the standard NeRF framework. Then, with the trained individual NeRFs fixed, an uncertainty MLP 140 of all NeRFs is trained together by fusing NeRFs based on their uncertainty 150.

[0037] FIG. 2 is a block / flow diagram of an exemplary large scale 3D scene representation according to an embodiment of the present invention.

[0038] For large scenes 200 that are beyond what a single NeRF can model, the exemplary embodiment clusters the image collection into four groups, and the exemplary embodiment trains a separate NeRF for each cluster with uncertainty. For an exemplary 3D point, the exemplary embodiment fuses NeRF1 and NeRF2 based on their respective uncertainties at the point to obtain a fused density and color prediction. The exemplary embodiment trains each NeRF separately in parallel, followed by training an uncertainty prediction network. During training, the exemplary embodiment synthesizes viewpoints 170 and applies the rendering loss and the entropy loss 160 to optimize the uncertainty network. After training each sub-NeRF, the exemplary embodiment trains all the uncertainty networks jointly. During inference, the trained NeRFs and the uncertainties are applied to render novel viewpoints 170.

[0039] 1 and 2, the exemplary embodiments aim to develop a NeRF that achieves high-fidelity novel viewpoint synthesis and 3D reconstruction in unprecedented large-scale scenes. To this end, novel techniques are proposed, such as a divide-and-conquer strategy to cluster images into smaller distinct groups and train a NeRF for each group separately, quantification of the uncertainty of the NeRFs and an uncertainty-aware fusion of the individual NeRFs to obtain a composite NeRF that works on the whole scene, and a novel entropy loss that leads to a more accurate 3D shape of the NeRFs.

[0040] First, due to the limited capacity of MLPs, the exemplary embodiment trains multiple MLPs, each focused on a relatively small portion of the scene, so that each MLP can achieve high-quality encoding of the scene's shape and radiance. The advantage of this strategy is that it facilitates parallel training of multiple MLPs using distributed computing resources, thereby efficiently processing large scenes. Second, the exemplary embodiment requires that all individual NeRFs be combined to encode the entire scene in a global manner for large-scale novel view synthesis and 3D reconstruction. Thus, the exemplary embodiment fuses the individual NeRFs and handles stitching of overlapping regions in a principled probabilistic manner.

[0041] In particular, we derive the geometric uncertainty of density predictions for each NeRF for each 3D point based on the Jacobian of the rendering loss function, and find an approximate Gaussian distribution for each density prediction. This distribution is leveraged to fuse the respective NeRFs according to Bayes' rule. Since the geometric uncertainty is computationally expensive, we train a lightweight MLP to predict the uncertainty. Inference for novel view synthesis, all NeRFs where a 3D point is visible are queried and fused based on the uncertainty. Finally, the original NeRFs only optimize the rendering loss without regularizing for 3D shape, which often produces low-quality 3D reconstructions. To address this issue, the exemplary embodiment introduces an entropy loss that forces the density to be zero or very large, clearly distinguishing between empty and non-empty space. This has been observed to significantly improve the shape estimation of the NeRFs.

[0042] FIG. 3 is a block / flow diagram of a practical application 300 for achieving high-fidelity novel view synthesis and 3D reconstruction for large-scale scenes, according to an embodiment of the present invention.

[0043] A large 3D scene 310 needs to be reconstructed. NeRF1 is applied to a small section of the large 3D scene 310 to extract an image viewpoint 320. Similarly, NeRF2 is applied to another small section of the large 3D scene 310 to extract an image viewpoint 330. NeRF3, NeRF4, NeRF5 etc. can be applied to other sections of the large 3D scene 310 until all parts or sections or regions of the large 3D scene 310 are covered. For each NeRF an uncertainty MLP 340 is trained. All the NeRFs are fused based on the uncertainty and the fused NeRF is applied to a novel viewpoint synthesis 350.

[0044] Thus, according to Figs. 1-3, a divide-and-conquer strategy is employed to cluster images into smaller distinct groups and train NeRFs for each group separately, a quantification of the NeRFs' uncertainty and a fusion that takes into account the individual NeRFs' uncertainty are employed to obtain a composite NeRF that works on the whole scene, and a novel entropy loss is utilized to lead to more accurate 3D shapes in the NeRFs. The divide-and-conquer strategy allows each NeRF with limited capacity to encode only a small part of the scene, thereby preserving high fidelity of the scene representation. It is then important to fuse all NeRFs together, allowing a novel viewpoint to be rendered as if it were obtained from a single NeRF. As the principle method for this purpose, uncertainty-based fusion is utilized. The exemplary method derives Jacobian-based geometric uncertainty and applies an additional uncertainty MLP to regress such values ​​to significantly reduce the computational cost. Finally, the exemplary method improves the geometry learning of the NeRFs by proposing a novel entropy loss.

[0045] In conclusion, NeRFs achieve unprecedented levels of fidelity in novel view synthesis, but are limited to small-scale scenes, as the limited capacity of a single MLP leads to a trade-off between resolution and scale. To mitigate this issue, the exemplary embodiment proposes to exploit locality in a novel image clustering-based synthetic NeRF scheme. In this scheme, we group images into clusters, each focused on a subset of the scene, and learn a separate NeRF for each cluster. Such image-level scene decomposition is shown to have advantages over and above common volume decomposition strategies. A NeRF stitching framework is introduced that merges all NeRFs to render the entire scene. Specifically, the exemplary embodiment uses an MLP to regress an implicit uncertainty field as a 6D continuous function. The uncertainty network takes a 3D point, its spatial location, gaze direction, and distance as input, and outputs a quantification of its uncertainty for each individual NeRF. This is further utilized to fuse the volumetric density and color queried from the individual NeRFs. In addition to scaling up NeRFs, the exemplary framework also enables flexible configuration of NeRFs at each granularity. NeRF stitching demonstrates state-of-the-art rendering quality in indoor scenes.

[0046] FIG. 4 is an exemplary processing system for achieving high-fidelity novel view synthesis and 3D reconstruction for large scenes, in accordance with an embodiment of the present invention.

[0047] The processing system includes at least one processor (CPU) 904 operatively coupled to other components via a system bus 902. A graphical processing unit (GPU) 905, a cache 906, a read-only memory (ROM) 908, a random access memory (RAM) 910, an input / output (I / O) adapter 920, a network adapter 930, a user interface adapter 940, and a display adapter 950 are operatively coupled to the system bus 902. Additionally, a composite NeRF method based on image clustering 100 is electrically connected to the bus 902.

[0048] The storage devices 922 are operably coupled to the system bus 902 by the I / O adapter 920. The storage devices 922 may be any type of disk storage device (e.g., magnetic or optical disk storage device), solid-state magnetic device, or the like.

[0049] The transceiver 932 is operably coupled to the system bus 902 by the network adapter 930 .

[0050] User input device(s) 942 are operatively coupled to the system bus 902 by a user interface adapter 940. User input device 942 may be any of a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, a device incorporating the functionality of at least two of the aforementioned devices, and the like. Of course, other types of input devices may be used while maintaining the spirit of the invention. User input device(s) 942 may be the same type of user input device or different types of user input devices. User input device(s) 942 are used to input and output information to and from the processing system.

[0051] A display device 952 is operatively coupled to the system bus 902 by a display adapter 950 .

[0052] Of course, the processing system may include other elements (not shown) or omit certain elements, as would be readily contemplated by one of ordinary skill in the art. For example, various other input and / or output devices may be included in the system, depending on the particular implementation, as would be readily understood by one of ordinary skill in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, various configurations of additional processors, controllers, memory, etc. may also be utilized, as would be readily understood by one of ordinary skill in the art. These and other variations of the processing system are readily contemplated by one of ordinary skill in the art given the teachings of the invention provided herein.

[0053] FIG. 5 is a block / flow diagram of an exemplary method for achieving high-fidelity novel view synthesis and 3D reconstruction for large scenes, in accordance with an embodiment of the present invention.

[0054] In block 1001, images are acquired from video streams received from multiple video image capture devices.

[0055] In block 1003, the images are grouped into separate image clusters that represent a large 3D scene.

[0056] In block 1005, a neural radiance field (NeRF) and an uncertain multilayer perceptron (MLP) are trained for each of the image clusters to generate multiple NeRFs and multiple uncertain MLPs for the large 3D scene.

[0057] In block 1007, the rendering loss and the entropy loss are applied to multiple NeRFs.

[0058] In block 1009, uncertainty-based fusion is performed on the multiple NeRFs to define a fused NeRF, and the multiple NeRFs and the multiple uncertainty MLPs are jointly fine-tuned.

[0059] In block 1011, we apply fused NeRF for novel viewpoint synthesis of large-scale 3D scenes during inference.

[0060] As a result, according to an exemplary embodiment, data acquisition is performed to select a scene for scene representation learning, e.g., an apartment or building, and then images of the scene are captured. The images cover all parts of the scene required to perform de novo view synthesis. Then, a structure-from-motion algorithm is performed to find the camera pose for each image. Images are classified into separate clusters by viewpoint, and NeRFs and uncertainty MLPs for each image cluster are learned using rendering loss and entropy loss. Uncertainty-based NeRF fusion is performed to jointly fine-tune all NeRFs and uncertainty MLPs. Finally, during inference, the fused NeRFs are leveraged for de novo view synthesis of large-scale scenes.

[0061] As used herein, "data," "content," "information," and similar terms may be used interchangeably to refer to data that may be captured, transmitted, received, displayed, and / or stored in accordance with various exemplary embodiments. Thus, use of such terms should not be construed as limiting the spirit and scope of the present disclosure. Additionally, where a computing device is described herein as receiving data from another computing device, the data may be received directly from the other computing device or may be received indirectly via one or more intermediary computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and / or the like.

[0062] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," "computer," "apparatus," or "system." Additionally, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0063] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable CD-ROM, an optical data storage device, a magnetic data storage device, or any suitable combination of the foregoing. In this specification, a computer readable storage medium is any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0064] A computer-readable signal medium may include a propagated data signal having computer-readable program code embodied therein, for example as part of a baseband or carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic waves, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium, but may be any computer-readable medium that can communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0065] The program code embodied in the computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0066] Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language. The program code may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0067] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram blocks or modules.

[0068] These computer program instructions may also be stored on a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other device to function in a particular manner to produce an article of manufacture, where the instructions stored on the computer-readable medium include instructions that implement the functions / operations specified in the blocks or modules of the flowcharts and / or block diagrams.

[0069] Computer program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps, creating a computer-implemented process, such that the instructions executing on the computer or other programmable apparatus provide a process for implementing the functions / operations specified in the flowchart and / or block diagram blocks or modules.

[0070] It should be understood that the term "processor" as used herein is intended to include any processing device, such as, for example, one that includes a CPU and / or other processing circuitry, and that the term "processor" may refer to more than one processing device, and that various elements associated with a processing device may be shared by other processing devices.

[0071] The term "memory" as used herein is intended to include memory associated with a processor or CPU, such as, for example, RAM, ROM, fixed memory devices (e.g., hard drives), removable memory devices (e.g., diskettes), flash memory, etc. Such memory is considered a computer-readable storage medium.

[0072] Additionally, the phrase "input / output device" or "I / O device" as used herein is intended to include, for example, one or more input devices (e.g., a keyboard, mouse, scanner, etc.) for inputting data to a processing unit, and / or one or more output devices (e.g., speakers, displays, printers, etc.) for presenting results related to a processing unit.

[0073] The foregoing is understood in all respects to be illustrative and illustrative, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations can be implemented by those skilled in the art without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is set forth in the appended claims.

Claims

1. A method for achieving high-fidelity novel view synthesis and 3D reconstruction for large-scale scenes, comprising: acquiring images from the video streams received from a plurality of video image capture devices; Grouping the images into separate image clusters representing a large 3D scene; training a neural radiance field (NeRF) and an uncertainty multi-layer perceptron (MLP) for each of the image clusters to generate multiple NeRFs and multiple uncertainty MLPs for the large 3D scene; applying a rendering loss and an entropy loss to the plurality of NeRFs; performing uncertainty-based fusion on the plurality of NeRFs to define a fused NeRF, and jointly fine-tuning the plurality of NeRFs and the plurality of uncertainty MLPs; and applying the fused NeRF to novel view synthesis of the large scale 3D scene during inference.

2. The method of claim 1 , wherein the multiple uncertainty MLPs are trained jointly and the multiple NeRFs are trained in parallel.

3. The method of claim 1 , wherein a geometric uncertainty of a density prediction for each of the multiple NeRFs at each 3D point is derived based on a Jacobian of the rendering loss, resulting in an approximate Gaussian distribution for each density prediction.

4. The method of claim 1 , where the NeRF stitching is learned by regressing the uncertainty network's implicit uncertainty field as a 6D continuous function.

5. 5. The method of claim 4, wherein the uncertainty network takes as input, for a 3D point in the large 3D scene, the spatial location, viewing direction, and distance of the 3D point, and outputs, for each NeRF of the plurality of NeRFs, a quantification of uncertainty.

6. The method of claim 5 , wherein 6D coordinates associated with the 3D point are applied to query one uncertainty MLP of the plurality of uncertainty MLPs.

7. The method of claim 1 , wherein the uncertainty field is learned by employing a voting strategy that uses only the camera poses of training images, without access to training images of each of the multiple NeRFs.

8. 1. A non-transitory computer readable storage medium including a computer readable program for achieving novel high fidelity view synthesis and 3D reconstruction for large scenes, the computer readable program, when executed on a computer, causing the computer to: acquiring images from a video stream received from a plurality of video image capture devices; grouping the images into separate image clusters representing a large 3D scene; training a neural radiance field (NeRF) and an uncertainty multi-layer perceptron (MLP) for each of the image clusters to generate multiple NeRFs and multiple uncertainty MLPs for the large 3D scene; applying a rendering loss and an entropy loss to the plurality of NeRFs; performing uncertainty-based fusion on the plurality of NeRFs to define a fused NeRF, and jointly fine-tuning the plurality of NeRFs and the plurality of uncertainty MLPs; and applying the fused NeRF to a novel view synthesis of the large scale 3D scene during inference.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the multiple uncertainty MLPs are trained jointly and the multiple NeRFs are trained in parallel.

10. 9. The non-transitory computer-readable storage medium of claim 8, wherein a geometric uncertainty of a density prediction for each of the plurality of NeRFs at each 3D point is derived based on a Jacobian of the rendering loss, resulting in an approximate Gaussian distribution for each density prediction.

11. 10. The non-transitory computer-readable storage medium of claim 8, wherein the NeRF stitching is learned by regressing an implicit uncertainty field of the uncertainty network as a 6D continuous function.

12. 12. The non-transitory computer-readable storage medium of claim 11, wherein the uncertainty network takes as input, for a 3D point in the large 3D scene, the spatial location, viewing direction, and distance of the 3D point, and outputs, for each NeRF of the plurality of NeRFs, a quantification of uncertainty.

13. 13. The non-transitory computer-readable storage medium of claim 12, wherein 6D coordinates associated with the 3D point are applied to query one uncertainty MLP of the plurality of uncertainty MLPs.

14. 10. The non-transitory computer-readable storage medium of claim 8, wherein the uncertainty field is learned by employing a voting strategy that uses only camera poses of training images without access to training images of each of the plurality of NeRFs.

15. A system for achieving high-fidelity novel view synthesis and 3D reconstruction for large-scale scenes, comprising: Memory, and one or more processors in communication with the memory, the processors comprising: Acquiring images from the video stream received from a plurality of video image capture devices; Grouping the images into separate image clusters representing large 3D scenes; training a neural radiance field (NeRF) and an uncertainty multi-layer perceptron (MLP) for each of the image clusters to generate multiple NeRFs and multiple uncertainty MLPs for the large 3D scene; applying a rendering loss and an entropy loss to the plurality of NeRFs; performing uncertainty-based fusion on the plurality of NeRFs to define a fused NeRF; and jointly fine-tuning the plurality of NeRFs and the plurality of uncertainty MLPs; The system is configured to apply the fused NeRF to novel view synthesis of the large-scale 3D scene during inference.

16. The system of claim 15 , wherein the multiple uncertainty MLPs are trained jointly and the multiple NeRFs are trained in parallel.

17. 16. The system of claim 15, wherein a geometric uncertainty of a density prediction for each of the multiple NeRFs at each 3D point is derived based on a Jacobian of the rendering loss, resulting in an approximate Gaussian distribution for each density prediction.

18. The system of claim 15 , wherein the NeRF stitching is learned by regressing the uncertainty network's implicit uncertainty field as a 6D continuous function.

19. 20. The system of claim 18, wherein the uncertainty network takes as input, for a 3D point in the large 3D scene, the spatial location, viewing direction, and distance of the 3D point, and outputs, for each NeRF of the plurality of NeRFs, a quantification of uncertainty.

20. 20. The system of claim 19, wherein 6D coordinates associated with the 3D point are applied to query an uncertainty MLP of the plurality of uncertainty MLPs.

Images