High dynamic range viewpoint synthesis from noisy raw images

Abstract

A system and method for training a neural radiance field model for noisy scenes can leverage noisy raw images in a linear high dynamic range color space to train a neural radiance field model to generate viewpoint synthesis of low-light and / or high-contrast scenes. The trained model can then be utilized to accurately complete viewpoint rendering tasks without the pre-processing used to generate low dynamic range images. In some implementations, training on raw data of low-light scenes can enable training a neural radiance field model to generate high-quality viewpoint renderings of low-light scenes.

Description

[Technical field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 279,363, filed November 15, 2021, which is incorporated by reference in its entirety herein.

[0002] The present disclosure relates generally to training neural radiance field models on noisy raw images. More particularly, the present disclosure relates to training neural radiance field models for generating perspective renderings for low-light scenes by training the neural radiance field models on high dynamic range (HDR) images. [Background technology]

[0003] Neural Radiance Field (NeRF) can be used for novel view synthesis from a collection of input images and their camera poses. Like several other view synthesis methods, NeRF can use low dynamic range (LDR) images as input. These images may have been passed through a lossy camera pipeline that smooths details, clips highlights, and distorts the simple noise distribution of the raw sensor data. Summary of the Invention [Means for solving the problem]

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0005] One exemplary aspect of the present disclosure is directed to a computing system. The system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining a training dataset. The training dataset may include a plurality of three-dimensional locations, a plurality of two-dimensional gaze directions, and a plurality of noisy raw images. In some implementations, the plurality of noisy raw images may include a plurality of high dynamic range images including a plurality of raw bits structured in a raw format. The operations may include processing a first three-dimensional location of the plurality of three-dimensional locations and a first two-dimensional gaze direction of the plurality of two-dimensional gaze directions with a neural radiance field model to generate a viewpoint rendering. The neural radiance field model may include one or more multi-layer perceptrons. In some implementations, the viewpoint rendering may describe one or more predicted color values ​​and one or more predicted volume density values. The operations may include evaluating a loss function that evaluates a difference between the viewpoint rendering and a first image of the plurality of noisy raw images. The first image can be associated with at least one of a first three-dimensional position or a first two-dimensional gaze direction. The operations can include adjusting one or more parameters of the neural radiance field model based at least in part on the loss function.

[0006] In some implementations, the operations can include processing the viewpoint rendering with the color correction model to generate a color corrected rendering. The loss function can include a reweighted L2 loss. In some implementations, evaluating a loss function that evaluates a difference between the viewpoint rendering and a first image of the noisy plurality of raw images can include mosaic masking. Evaluating a loss function that evaluates a difference between the viewpoint rendering and a first image of the noisy plurality of raw images can include exposure adjustment. In some implementations, the operations can include obtaining an input gaze direction and an input position, processing the input gaze direction and the input position with a neural radiance field model to generate predicted quad-bayer filter data, and processing the predicted quad-bayer filter data to generate a novel viewpoint rendering. The loss function can include gradient stopping. Gradient stopping can mitigate against the neural radiance field model generalizing to low confidence values. In some implementations, the first image can include real-world photon signal data generated by a camera. The viewpoint rendering can include predicted photon signal data. In some implementations, the multiple noisy raw images can be associated with multiple Red-Green-Green-Blue data sets.

[0007] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for novel viewpoint rendering. The method can include acquiring, by a computing system including one or more processors, an input 2D gaze direction and an input 3D position associated with an environment. The method can include acquiring, by the computing system, a neural radiance field model. The neural radiance field model can be pre-trained on a training data set. In some implementations, the training data set can include a plurality of noisy input data sets associated with the environment. The training data set can include a plurality of training gaze directions and a plurality of training positions. The method can include processing, by the computing system, the input 2D gaze direction and the input 3D position with the neural radiance field model to generate predicted data. The predicted data can include one or more predicted density values ​​and one or more predicted color values. The method can include processing, by the computing system, the predicted data with an image augmentation block to generate a predicted viewpoint rendering. The predicted viewpoint rendering can describe a predicted scene rendering of the environment.

[0008] In some implementations, the image enhancement block can adjust a focus of the predicted data. The image enhancement block can adjust an exposure level of the predicted data. In some implementations, the image enhancement block can adjust a tone mapping of the predicted data. Each noisy input data set of the plurality of noisy input data sets can include photon signal data. In some implementations, each noisy input data set of the plurality of noisy input data sets can include signal data related to at least one of red values, green values, or blue values. Each noisy input data set of the plurality of noisy input data sets can include one or more noisy mosaic-type linear raw images.

[0009] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations can include obtaining a training data set. The training data set can include a plurality of raw input data sets. In some implementations, the training data set can include a plurality of respective gaze directions and a plurality of respective locations. The operations can include processing the first gaze direction and the first location with a neural radiance field model to generate first predicted data. The first predicted data can describe one or more first predicted color values ​​and one or more first predicted density values. The operations can include evaluating a loss function that evaluates a difference between the first predicted data and a first raw input data set of the plurality of raw input data sets. In some implementations, the first raw input data set can be associated with at least one of the first location or the first gaze direction. The operations can include adjusting one or more parameters of the neural radiance field model based at least in part on the loss function.

[0010] In some implementations, the one or more parameters can be associated with a learned three-dimensional representation associated with the environment. The loss function can include a tone mapping loss associated with processing at least one of the first predicted data or the first raw input data set. In some implementations, the operations can include processing the second gaze direction and the second location with the neural radiance field model to generate the second predicted data. The second predicted data can describe one or more second predicted color values ​​and one or more second predicted density values. The operations can include scaling the one or more second predicted color values ​​based on a shutter speed to generate the scaled second predicted data. The operations can include evaluating a loss function that evaluates a difference between the scaled second predicted data and a second raw input data set of the plurality of raw input data sets. In some implementations, the second raw input data set can be associated with at least one of the second location or the second gaze direction. The operations can include adjusting one or more additional parameters of the neural radiance field model based at least in part on the loss function.

[0011] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0012] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, explain associated principles.

[0013] Detailed descriptions of embodiments directed to persons skilled in the art are set forth herein, which refer to the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1A] FIG. 2 is a block diagram of an exemplary computing system for performing viewpoint rendering generation according to an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 2 is a block diagram of an example computing device for performing viewpoint rendering generation according to an example embodiment of the present disclosure. [Figure 1C] FIG. 2 is a block diagram of an example computing device for performing viewpoint rendering generation according to an example embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram of an exemplary neural radiance field model according to an exemplary embodiment of the present disclosure. [Diagram 3] FIG. 1 is a block diagram of an exemplary neural radiance field model according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 2 is a diagram of an exemplary viewpoint rendering pipeline according to an exemplary embodiment of the present disclosure. [Diagram 5] FIG. 1 is a block diagram of an example neural radiance field model training according to an example embodiment of the present disclosure. [Figure 6] FIG. 1 is a flow chart diagram of an example method for performing neural radiance field model training according to an example embodiment of the present disclosure. [Figure 7] 1 is a flowchart diagram of an exemplary method for performing novel viewpoint rendering according to an exemplary embodiment of the present disclosure. [Figure 8] FIG. 1 is a flow chart diagram of an example method for performing neural radiance field model training according to an example embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Reference numbers that are repeated among the drawings may identify like features in various implementations.

[0016] overview In general, the present disclosure is directed to a system and method for training a neural radiance field model on a noisy raw image in a linear high dynamic range (HDR) color space. For example, the system and method can utilize a noisy raw image in a linear HDR color space as an input for training one or more neural radiance field models. Thus, the system and method can avoid lossy post-processing that digital cameras apply to smooth out noisy images to produce visually appealing JPEG files. In some implementations, the system and method can assume a static scene and take the camera pose as a given input.

[0017] The systems and methods disclosed herein can include obtaining a training dataset. The training dataset can include a plurality of three-dimensional locations, a plurality of two-dimensional gaze directions, and a plurality of noisy raw images. In some implementations, the plurality of noisy raw images can include a plurality of high dynamic range images comprising a plurality of unprocessed bits structured in a raw format. The systems and methods can include processing a first three-dimensional location of the plurality of three-dimensional locations and a first two-dimensional gaze direction of the plurality of two-dimensional gaze directions with a neural radiance field model to generate a viewpoint rendering. In some implementations, the neural radiance field model can include one or more multi-layer perceptrons, and the viewpoint rendering can describe one or more predicted color values ​​and one or more predicted volume density values. Additionally and / or alternatively, the systems and methods can include evaluating a loss function that evaluates a difference between the viewpoint rendering and a first image of the plurality of noisy raw images. The first image can be associated with at least one of the first three-dimensional location or the first two-dimensional gaze direction. One or more parameters of the neural radiance field model can be adjusted based at least in part on the loss function.

[0018] In some implementations, the viewpoint renderings may be processed with a color correction model to produce color-corrected renderings. The loss function may include a reweighted L2 loss.

[0019] Additionally and / or alternatively, evaluating a loss function that evaluates the difference between the viewpoint rendering and the first image of the plurality of noisy raw images may include mosaic masking and / or exposure adjustment.

[0020] In some implementations, the systems and methods disclosed herein can optimize the neural radiance field model by optimizing the neural volumetric scene representation to match multiple images using gradient descent based at least in part on volumetric rendering loss. Additionally and / or alternatively, the systems and methods can utilize optimization techniques to harmonize content from multiple noisy raw images to jointly reconstruct and denoise the scene.

[0021] In some implementations, the raw data (e.g., raw image) may include raw bits saved by the camera in a raw format. Additionally and / or alternatively, the HDR data (e.g., HDR image) may include one or more images that use more than the standard 8 bits to represent color intensity.

[0022] In some implementations, sRGB can represent the opposite of the raw data (eg, a fully post-processed image that exists in a tone-mapped LDR color space).

[0023] The Neural Radiance Field (NeRF) model can include a multilayer perceptron (MLP)-based scene representation optimized to reproduce the appearance of a set of input images with known camera poses. The resulting reconstruction can be used to render novel viewpoints from previously unobserved poses. NeRF's MLP network can take 3D position and 2D viewing direction as inputs, and output volumetric density and color. To render each pixel in the output image, the NeRF model can use volume rendering to combine color and density from many points sampled along the corresponding 3D ray.

[0024] A standard NeRF model can take as input a clean low dynamic range (LDR) sRGB color space image with values ​​in the range [0;1]. Converting a raw HDR image to an LDR image can have two consequences: (1) details in bright areas may be lost when values ​​are clipped upward at 1 or heavily compressed and quantized to 8 bits by the tone mapping curve, and (2) after passing through a nonlinear tone mapping curve and clipping values ​​below 0, the noise distribution per pixel may no longer be 0-mean.

[0025] Systems and methods disclosed herein (e.g., RawNeRF) can include modifying NeRF to use noisy raw images in a linear HDR color space as input. The modification can enable the avoidance of lossy post-processing that digital cameras apply to smooth out noisy images to produce visually acceptable JPEG files. Training directly on raw data can effectively place RawNeRF among the multi-image denoisers capable of reconstructing scenes captured in near-darkness. Unlike common video or burst image denoising methods, RawNeRF can assume a static scene and predict the camera pose as a given input. Given these additional constraints, RawNeRF may be able to effectively exploit 3D multi-view consistency to average information across all of the input frames at once. Since each captured scene can contain 30-100 input images, this in turn can mean that RawNeRF can be more effective than feed-forward burst / video denoisers that typically only utilize 3-8 input images per output.

[0026] Additionally, because RawNeRF can preserve the full dynamic range of the input image, the system and method can enable HDR viewpoint synthesis applications, such as varying exposure settings and focus blur over the course of a newly rendered camera path, that would not be possible with an LDR representation.

[0027] In some implementations, the system and method can modify the NeRF to train directly on linear raw images instead, preserving the full dynamic range of the scene. The system and method can enable the system to perform a novel high dynamic range (HDR) viewpoint synthesis task that renders raw output from the reconstructed NeRF and manipulates focus, exposure, and tone mapping after the fact, in addition to varying the camera viewpoint. Although the raw data may appear significantly noisier than post-processed images, the NeRF of the system and method disclosed herein can be extremely robust to zero-mean distribution of raw noise, producing clean scene reconstructions that compare favorably with dedicated single- and multi-image deep denoising methods. This can enable the system and method (e.g., RawNeRF implementations) to reconstruct scenes from extremely noisy images captured in near-darkness.

[0028] HDR+ can perfect HDR for handheld raw image bursts with very little motion, RawNeRF can handle very wide baseline motion and can also create a 3D reconstruction of the scene (although static scenes may be used).

[0029] Neural Radiance Field (NeRF) can be utilized for high quality novel viewpoint synthesis from a collection of input images and their camera poses. In some implementations, NeRF can utilize 8-bit JPEG as input. Images may pass through a lossy camera pipeline that smooths details, clips highlights, and distorts the simple noise distribution of the raw sensor data. The systems and methods disclosed herein can instead modify NeRF to train directly on linear raw images, preserving the full dynamic range of the scene. In addition to varying camera viewpoints, the systems and methods can perform novel high dynamic range (HDR) viewpoint synthesis tasks that render raw output from the reconstructed NeRF and manipulate focus, exposure, and tone mapping after the fact. While the raw data can appear significantly noisier than post-processed images, the systems and methods can show that NeRF is extremely robust to zero-mean distributions of raw noise, producing clean scene reconstructions that compare favorably with dedicated single and multi-image deep denoising methods. The systems and methods can reconstruct scenes from extremely noisy images captured in near darkness.

[0030] The system and method can include obtaining a training dataset. The training dataset can include a plurality of three-dimensional locations, a plurality of two-dimensional gaze directions, and a plurality of noisy raw images. In some implementations, the plurality of noisy raw images can include a plurality of high dynamic range images comprising a plurality of unprocessed bits structured in a raw format. The system and method can include processing a first three-dimensional location of the plurality of three-dimensional locations and a first two-dimensional gaze direction of the plurality of two-dimensional gaze directions with a neural radiance field model to generate a viewpoint rendering. The neural radiance field model can include one or more multi-layer perceptrons. In some implementations, the viewpoint rendering can describe one or more predicted color values ​​and one or more predicted volume density values. The system and method can include evaluating a loss function that evaluates a difference between the viewpoint rendering and a first image of the plurality of noisy raw images. The first image can be associated with at least one of the first three-dimensional location or the first two-dimensional gaze direction. The system and method can include adjusting one or more parameters of the neural radiance field model based at least in part on the loss function.

[0031] The system and method may obtain a training dataset. The training dataset may include a plurality of three-dimensional positions, a plurality of two-dimensional gaze directions, and a plurality of noisy raw images. The plurality of noisy raw images may include a plurality of high dynamic range images including a plurality of raw bits structured in a raw format. In some implementations, the plurality of noisy raw images may be associated with a plurality of red-green-green-blue datasets. In some implementations, the plurality of noisy raw images may include a Bayer filter dataset generated based on raw signal data from one or more image sensors. The noisy raw image dataset may include data before exposure correction, color correction, and / or focus correction. The plurality of two-dimensional gaze directions and the plurality of three-dimensional positions may be associated with gaze directions and positions in an environment. The environment may include low illumination, and the plurality of noisy raw images may include low illumination.

[0032] A first three-dimensional location of the plurality of three-dimensional locations and a first two-dimensional gaze direction of the plurality of two-dimensional gaze directions may be processed using a neural radiance field model to generate a viewpoint rendering. The neural radiance field model may include one or more multi-layer perceptrons. In some implementations, the viewpoint rendering may describe one or more predicted color values ​​and one or more predicted volume density values. The neural radiance field model may be configured to process the gaze directions and positions to generate one or more predicted color values ​​and one or more predicted density values. The one or more predicted color values ​​and the one or more predicted density values ​​may be utilized to generate a viewpoint rendering. The viewpoint rendering may be a raw viewpoint rendering associated with one or more Bayer filter images associated with one or more red, blue, or green filters. The raw viewpoint rendering may be processed using one or more image enhancement blocks to generate an augmented image having one or more corrected colors, one or more corrected focus, one or more corrected exposure, and / or one or more corrected artifacts.

[0033] A loss function evaluating a difference between the viewpoint rendering and a first image of the noisy plurality of raw images may then be evaluated. The first image may be associated with at least one of a first three-dimensional location or a first two-dimensional gaze direction. In some implementations, the loss function may include a reweighted L2 loss. Evaluating the loss function evaluating a difference between the viewpoint rendering and a first image of the noisy plurality of raw images may include mosaic masking. Alternatively and / or additionally, evaluating the loss function evaluating a difference between the viewpoint rendering and a first image of the noisy plurality of raw images may include exposure adjustment. In some implementations, the first image may include real-world photon signal data generated by a camera. The viewpoint rendering may include predicted photon signal data.

[0034] The system and method can adjust one or more parameters of the neural radiance field model based at least in part on a loss function. The loss function can include gradient stopping. In some implementations, the gradient stopping can mitigate the neural radiance field model from generalizing to low confidence values.

[0035] In some implementations, the systems and methods can process the viewpoint renderings with a color correction model to generate color corrected renderings. Alternatively and / or additionally, the viewpoint renderings can be processed with an exposure correction model to generate exposure corrected renderings. The color correction model and / or the exposure correction model can be part of an image enhancement block. One or more image correction models can be configured to process the raw signal data and / or the predicted raw signal data. The one or more image correction models can be part of an image enhancement model and can be trained on the Bayer filtered signal data.

[0036] Additionally and / or alternatively, the systems and methods may include obtaining an input gaze direction and an input position, processing the input gaze direction and input position using a neural radiance field model to generate predicted quad-Bayer filter data, and processing the predicted quad-Bayer filter data to generate a novel viewpoint rendering.

[0037] The trained neural radiance field model may then be utilized for novel viewpoint synthesis. For example, the system and method may include obtaining an input 2D gaze direction and an input 3D position associated with the environment. The system and method may include obtaining a neural radiance field model. The neural radiance field model may be trained on a training data set. The training data set may include a plurality of noisy input data sets associated with the environment. In some implementations, the training data set may include a plurality of training gaze directions and a plurality of training positions. The system and method may include processing the input 2D gaze direction and the input 3D position with the neural radiance field model to generate predicted data. The predicted data may include one or more predicted density values ​​and one or more predicted color values. The system and method may include processing the predicted data with an image augmentation block to generate a predicted viewpoint rendering. The predicted viewpoint rendering may describe a predicted scene rendering of the environment.

[0038] The system and method can obtain an input 2D gaze direction and an input 3D position associated with an environment. The environment may include one or more objects. In some implementations, the environment may include low lighting. The input gaze direction and input 3D position can be associated with a request for a novel viewpoint rendering that indicates a predicted viewpoint of the environment associated with the position and gaze direction.

[0039] A neural radiance field model may then be obtained. The neural radiance field model may be pre-trained on a training dataset. The training dataset may include a plurality of noisy input datasets related to the environment. In some implementations, the training dataset may include a plurality of training gaze directions and a plurality of training positions. Each noisy input dataset of the plurality of noisy input datasets may include photon signal data. Additionally and / or alternatively, each noisy input dataset of the plurality of noisy input datasets may include signal data associated with at least one of red values, green values, or blue values. In some implementations, each noisy input dataset of the plurality of noisy input datasets may include one or more noisy mosaic-type linear raw images.

[0040] The input 2D gaze direction and the input 3D position may be processed using a neural radiance field model to generate predicted data. The predicted data may include one or more predicted density values ​​and one or more predicted color values. The predicted data may be utilized to generate predicted Bayer filter data, which may include predicted red filter data, predicted blue filter data, predicted first green filter data, and / or predicted second green filter data. The predicted data may relate to predicted raw image data, which may be processed to generate improved image data.

[0041] The predicted data may then be processed with an image enhancement block to generate a predicted viewpoint rendering. The predicted viewpoint rendering may describe a predicted scene rendering of the environment. In some implementations, the image enhancement block may adjust the focus of the predicted data. Additionally and / or alternatively, the image enhancement block may adjust the exposure level of the predicted data. The image enhancement block may adjust the tone mapping of the predicted data.

[0042] Alternatively and / or additionally, the system and method may include obtaining a training data set. The training data set may include a plurality of raw input data sets. In some implementations, the training data set may include a plurality of respective gaze directions and a plurality of respective locations. The system and method may include processing the first gaze direction and the first location with a neural radiance field model to generate first predicted data. The first predicted data may describe one or more first predicted color values ​​and one or more first predicted density values. The system and method may include evaluating a loss function that evaluates a difference between the first predicted data and a first raw input data set of the plurality of raw input data sets. In some implementations, the first raw input data set may be associated with at least one of the first location or the first gaze direction. The system and method may include adjusting one or more parameters of the neural radiance field model based at least in part on the loss function.

[0043] A training dataset may be obtained. The training dataset may include a plurality of raw input datasets. In some implementations, the training dataset may include a plurality of respective gaze directions and a plurality of respective positions. The plurality of respective gaze directions may include a plurality of two-dimensional gaze directions. The plurality of respective positions may include a plurality of three-dimensional positions. The raw input dataset may include one or more high dynamic range images.

[0044] The first gaze direction and the first position may be processed using a neural radiance field model to generate first predicted data. The first predicted data may describe one or more first predicted color values ​​and one or more first predicted density values. The first predicted data may be related to the predicted raw photon signal data.

[0045] A loss function that evaluates the difference between the first predicted data and a first raw input data set of the plurality of raw input data sets may then be evaluated. The first raw input data set may be associated with at least one of a first position or a first gaze direction. In some implementations, the loss function may include a tone mapping loss associated with processing at least one of the first predicted data or the first raw input data set. The loss function may penalize errors more in dark regions than in bright regions to match the way human perception compresses dynamic range. Penalizing may occur after both the first predicted data and the first raw input data set are passed through a tone mapping curve prior to loss function evaluation. In some implementations, the loss function may include a weighted loss function. The loss may be applied to active color channels of the mosaic-type raw input data and / or the first predicted data. Additionally and / or alternatively, camera intrinsics may be utilized to take into account radial distortion when generating light rays.

[0046] One or more parameters of the neural radiance field model may then be adjusted based at least in part on the loss function. The one or more parameters may be related to a learned three-dimensional representation associated with the environment. In some implementations, the one or more parameters may be adjusted to learn the environment.

[0047] In some implementations, the computing system can train a neural radiance field model for the environment using image data generated using different shutter speeds. For example, the computing system can process a second gaze direction and a second location with the neural radiance field model to generate second predicted data. The second predicted data can describe one or more second predicted color values ​​and one or more second predicted density values. The computing system can scale the one or more second predicted color values ​​based on the shutter speed to generate scaled second predicted data. A loss function that evaluates a difference between the scaled second predicted data and a second raw input data set of the plurality of raw input data sets can then be evaluated. The second raw input data set can be associated with at least one of the second location or the second gaze direction. One or more additional parameters of the neural radiance field model can be adjusted based at least in part on the loss function.

[0048] The systems and methods of the present disclosure provide several technical effects and advantages. As an example, the systems and methods can train a neural radiance field model on a noisy raw image. More specifically, the systems and methods can utilize the raw image to train the neural radiance field model. For example, in some implementations, the systems and methods can include training a neural radiance field model on a plurality of noisy raw images in a linear HDR color space. The neural radiance field model can then be utilized to generate a perspective rendering of the scene.

[0049] Another technical advantage of the systems and methods of this disclosure is the ability to generate perspective renderings for low-light scenes. For example, a neural radiance field model may be trained on data from low-light scenes, and the resulting trained model may then be utilized for novel perspective renderings of low-light scenes.

[0050] Another exemplary technical effect and advantage relates to reducing computational costs and computational times. The systems and methods disclosed herein can eliminate pre-processing steps for training neural radiance field models. The use of HDR images rather than LDR images can eliminate processing steps for correcting raw images.

[0051] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the drawings.

[0052] Exemplary Devices and Systems 1A illustrates a block diagram of an example computing system 100 for performing perspective rendering (e.g., perspective rendering of low-light and / or high-contrast scenes) according to an example embodiment of the present disclosure. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, communicatively coupled via a network 180.

[0053] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or game controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0054] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0055] In some implementations, the user computing device 102 can store or include one or more neural radiance field models 120. For example, the neural radiance field models 120 can be or include various machine learning type models, such as neural networks (e.g., deep neural networks) or other types of machine learning type models, including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. An exemplary neural radiance field model 120 is described with reference to FIGS. 2-5.

[0056] In some implementations, one or more neural radiance field models 120 can be received from a server computing system 130 over a network 180, stored in a user computing device memory 114, and then used or implemented by one or more processors 112. In some implementations, a user computing device 102 can implement multiple parallel instances of a single neural radiance field model 120 (e.g., for performing parallel viewpoint rendering across multiple instances of a low-light scene).

[0057] More specifically, the systems and methods may include training a neural radiance field model on a plurality of noisy raw images (e.g., a plurality of raw images) of low-light and / or high-contrast scenes. The trained neural radiance field model may then be utilized to generate perspective renderings for the low-light and / or high-contrast scenes.

[0058] Additionally or alternatively, one or more neural radiance field models 140 may be included in or stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the neural radiance field model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a viewpoint rendering service). Thus, one or more models 120 may be stored and implemented in the user computing device 102 and / or one or more models 140 may be stored and implemented in the server computing system 130.

[0059] The user computing device 102 may also include one or more user input components 122 for receiving user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a conventional keyboard, or other means by which a user may provide user input.

[0060] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0061] In some implementations, the server computing system 130 includes or is implemented by one or more server computing devices. In instances in which the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0062] As described above, the server computing system 130 may store or include one or more machine learning based neural radiance field models 140. For example, the models 140 may be or include various machine learning based models. Exemplary machine learning based models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are described with reference to FIGS. 2-5.

[0063] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 through interaction with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0064] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is implemented by one or more server computing devices.

[0065] The training computing system 150 may include a model trainer 160 that trains the machine learning based models 120 and / or 140 stored in the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as a mean squared error function, a likelihood loss function, a cross-entropy loss function, a hinge loss function, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters over several training iterations.

[0066] In some implementations, performing the backpropagation may include performing a truncated backpropagation through time. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0067] In particular, model trainer 160 may train neural radiance field model 120 and / or 140 based on a set of training data 162. Training data 162 may include, for example, a plurality of three-dimensional positions, a plurality of two-dimensional gaze directions, and a plurality of noisy raw images. Each of the plurality of noisy raw images may be associated with at least one position and at least one gaze direction.

[0068] In some implementations, if the user has given consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 against user-specific data received from the user computing device 102. In some instances, this process may be referred to as individualizing the model.

[0069] Model trainer 160 includes computer logic utilized to provide desired functionality. Model trainer 160 may be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as a RAM, a hard disk, or an optical or magnetic medium.

[0070] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may be carried over any type of wired and / or wireless connections using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0071] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0072] In some implementations, the input to the machine learning based model of the present disclosure may be image data. The machine learning based model may process the image data to generate an output. As an example, the machine learning based model may process the image data to generate an image recognition output (e.g., recognition of the image data, latent embedding of the image data, encoded representation of the image data, hash of the image data, etc.). As another example, the machine learning based model may process the image data to generate an image segmentation output. As another example, the machine learning based model may process the image data to generate an image classification output. As another example, the machine learning based model may process the image data to generate an image data modification output (e.g., alteration of the image data, etc.). As another example, the machine learning based model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning based model may process the image data to generate an upscaled image data output. As another example, the machine learning based model may process the image data to generate a prediction output.

[0073] In some implementations, the input to the machine learning model of the present disclosure may be text data or natural language data. The machine learning model may process the text data or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text data or natural language data to generate a latent text embedding output. As another example, the machine learning model may process the text data or natural language data to generate a translation output. As another example, the machine learning model may process the text data or natural language data to generate a classification output. As another example, the machine learning model may process the text data or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text data or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text data or natural language data to generate an upscaled text output or natural language output (e.g., text data or natural language data that is of higher quality than the input text or input natural language). As another example, the machine learning model may process the text data or natural language data to generate a prediction output.

[0074] In some implementations, the input to the machine learning based model of the present disclosure may be latent coding data (e.g., a latent space representation of the input, etc.). The machine learning based model may process the latent coding data to generate an output. As an example, the machine learning based model may process the latent coding data to generate a recognition output. As another example, the machine learning based model may process the latent coding data to generate a reconstruction output. As another example, the machine learning based model may process the latent coding data to generate an exploration output. As another example, the machine learning based model may process the latent coding data to generate a reclustering output. As another example, the machine learning based model may process the latent coding data to generate a prediction output.

[0075] In some implementations, the input to the machine learning based model of the present disclosure may be statistical data. The machine learning based model may process the statistical data to generate an output. As an example, the machine learning based model may process the statistical data to generate a recognition output. As another example, the machine learning based model may process the statistical data to generate a prediction output. As another example, the machine learning based model may process the statistical data to generate a classification output. As another example, the machine learning based model may process the statistical data to generate a segmentation output. As another example, the machine learning based model may process the statistical data to generate a segmentation output. As another example, the machine learning based model may process the statistical data to generate a visualization output. As another example, the machine learning based model may process the statistical data to generate a diagnostic output.

[0076] In some implementations, the input to the machine learning based model of the present disclosure may be sensor data. The machine learning based model may process the sensor data to generate an output. As an example, the machine learning based model may process the sensor data to generate a recognition output. As another example, the machine learning based model may process the sensor data to generate a prediction output. As another example, the machine learning based model may process the sensor data to generate a classification output. As another example, the machine learning based model may process the sensor data to generate a segmentation output. As another example, the machine learning based model may process the sensor data to generate a segmentation output. As another example, the machine learning based model may process the sensor data to generate a visualization output. As another example, the machine learning based model may process the sensor data to generate a diagnosis output. As another example, the machine learning based model may process the sensor data to generate a detection output.

[0077] In some cases, the machine learning based model may be configured to perform a task that includes encoding (and / or corresponding decoding) input data for reliable and / or efficient transmission or storage. For example, the task may be an audio compression task. The input may include audio data and the output may comprise compressed audio data. In another example, the input may include visual data (e.g., one or more images or videos) and the output may comprise compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding for the input data (e.g., input audio data or input visual data).

[0078] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task can be image classification, where the output is a set of scores, each score corresponding to a different object class, and represents a likelihood that one or more images show an object belonging to that object class. The image processing task can be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, a likelihood that the region shows an object of interest. As another example, the image processing task can be image segmentation, where the image processing output specifies, for each pixel in one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task can be depth estimation, where the image processing output specifies, for each pixel in one or more images, a respective depth value. As another example, the image processing task may be motion estimation, where the network input includes multiple images and the image processing output defines, for each pixel of one of the input images, the scene motion indicated at the pixel between images in the network input.

[0079] 1A illustrates one exemplary computing system that may be used to implement the present disclosure. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 may be both trained and used locally on the user computing device 102. In some of such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0080] 1B illustrates a block diagram of an exemplary computing device 10 that functions according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0081] Computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine learning model. For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0082] 1B, each application may communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0083] 1C illustrates a block diagram of an exemplary computing device 50 that functions according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0084] Computing device 50 includes several applications (e.g., applications 1-N). Each application is in communication with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0085] The central intelligence layer includes several machine learning models. For example, as shown in FIG. 1C, each machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) to all of the applications. In some implementations, the central intelligence layer is included within or implemented by the operating system of the computing device 50.

[0086] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As shown in FIG. 1C, the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0087] Example Model Configuration FIG. 2 illustrates a block diagram of an exemplary neural radiance field model 200 (e.g., RawNeRF) according to an exemplary embodiment of the present disclosure. In some implementations, the neural radiance field model 200 is trained to receive a set of input data 204 describing noisy raw image data (e.g., raw images and their respective positions and gaze directions) and, as a result of receiving the input data 204, provide output data 206 that can be evaluated against the training data to determine a gradient descent to back-propagate to the neural radiance field model for training. Thus, in some implementations, the neural radiance field model 200 can include a RawNeRF model 202 that, once trained, is operable to generate a viewpoint rendering for a low-light scene. The output data 206 can include a rendered rectilinear line of sight. The output data 206 can be processed with a color correction block 208 to generate an improved viewpoint rendering 210. The improved viewpoint rendering 210 can include a high dynamic range to low dynamic range conversion. Additionally and / or alternatively, the improved viewpoint rendering 210 may be generated based on exposure changes and / or tone mapping decisions.

[0088] The systems and methods of the present disclosure may differ from the low dynamic range neural radiance field pipeline 212. In particular, the low dynamic range neural radiance field pipeline 212 may include pre-processing the image data before training the neural radiance field model, which may train the neural radiance field model to output a viewpoint rendering that generalizes to the bias of the processed data. Alternatively and / or additionally, the systems and methods of the present disclosure may train the neural radiance field model 202 on input data 204 that includes noisy raw image data.

[0089] 3 illustrates a block diagram of an exemplary neural radiance field model 300 according to an exemplary embodiment of the present disclosure. The neural radiance field model 300 is similar to the neural radiance field model 200 of FIG. 2, except that the neural radiance field model 300 further includes an optional LDR pipeline in series with the HDR pipeline, rather than displayed in parallel as shown in FIG.

[0090] In particular, the training data 302 may include a number of noisy raw images, a number of 2D gaze directions, and a number of 3D positions. The 2D gaze directions and the 3D positions may be processed using a neural radiance field model 304 to generate predicted data 306. The predicted data 306 may include one or more predicted density values ​​and / or one or more predicted color values. The predicted data 306 may then be compared against ground truth data to evaluate a loss function 308.

[0091] For an LDR pipeline, the ground truth data may include processed image data. For example, noisy raw images associated with gaze direction and position may be processed using an image processing pipeline 310 to generate training data 312 having processed images. The predicted data 306 and the processed images may be utilized to evaluate the loss function 308. Gradient descent may then be backpropagated to the neural radiance field model 304 to adjust one or more parameters of the neural radiance field model 304.

[0092] Alternatively and / or additionally, for an HDR pipeline, the ground truth data may include noisy raw images. For example, the prediction data 306 and the noisy raw (unprocessed) images may be utilized to evaluate the loss function 308 to generate a gradient descent, which may be back-propagated to the neural radiance field model 304 to adjust one or more parameters of the neural radiance field model 304.

[0093] Both the LDR and HDR pipelines may include generating predicted data 306 that may be utilized to evaluate a loss function 308. However, the ground truth data and / or the loss function 308 may differ. In particular, the LDR pipeline may include processed image data as ground truth, which may train the neural radiance field model 304 to output low dynamic range data. Alternatively and / or additionally, the HDR pipeline may include unprocessed image data as ground truth, which may train the neural radiance field model 304 to output high dynamic range data.

[0094] 4 illustrates a diagram of an exemplary viewpoint rendering pipeline 400 according to an exemplary embodiment of the present disclosure. For the low dynamic range neural radiance field model pipeline 406, input data 402 (e.g., noisy mosaic-type linear raw images (e.g., RGGB Bayer filter image dataset)) may be processed (414) to demosaic the images and perform color correction and / or exposure correction. The resulting processed image data may then be utilized to train a neural radiance field model (416). The trained model may then be utilized to render a low dynamic range viewpoint of the environment for which the neural radiance field model was trained (418).

[0095] For the high dynamic range neural radiance field model pipeline 404, the input data 402 may be utilized to directly train a raw neural radiance field model (408). The trained model may be trained to render a high dynamic range viewpoint of the environment in which the raw neural radiance field model is trained (410). The high dynamic range viewpoint rendered at 410 may then be post-processed (412) to modify the exposure and tone mapping of the viewpoint rendering to produce an improved viewpoint rendering.

[0096] FIG. 5 illustrates a block diagram of an exemplary neural radiance field model training 500 according to an exemplary embodiment of the present disclosure. The neural radiance field model training 500 may include obtaining a training dataset. The training dataset may include one or more positions 502, one or more gaze directions, and / or one or more raw image datasets 514. For training, the three-dimensional positions 502 and the two-dimensional gaze directions 504 may be processed with a neural radiance field model 506 to generate predicted data 508. The predicted data 508 may include one or more predicted color values ​​and / or one or more predicted density values. The predicted data 508 and the raw image dataset 514 from the training dataset may be utilized to evaluate a loss function 516. The loss function 516 may then be utilized to adjust one or more parameters of the neural radiance field model 506.

[0097] Once the neural radiance field model 506 is trained, a new set of positions and gaze directions can be processed with the neural radiance field model 506 to generate predicted data 508, which may then be processed with the image augmentation model 510 to generate a new viewpoint rendering 512. The new viewpoint rendering 512 can be related to the processed image data.

[0098] Exemplary Methods 6 shows a flow chart diagram of an exemplary method for functioning in accordance with an exemplary embodiment of the present disclosure. Although FIG. 6 shows steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the particular order or configuration shown. Various steps of method 600 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0099] At 602, the computing system can obtain a training dataset. The training dataset can include a plurality of three-dimensional positions, a plurality of two-dimensional gaze directions, and a plurality of noisy raw images. In some implementations, the plurality of noisy raw images can include a plurality of high dynamic range images including a plurality of raw bits structured in a raw format. In some implementations, the plurality of noisy raw images can be associated with a plurality of red-green-green-blue datasets. In some implementations, the plurality of noisy raw images can include a Bayer filter dataset generated based on raw signal data from one or more image sensors. The noisy raw image dataset can include data before exposure correction, color correction, and / or focus correction. The plurality of two-dimensional gaze directions and the plurality of three-dimensional positions can be associated with gaze directions and positions in an environment. The environment can include low illumination, and the plurality of noisy raw images can include low illumination.

[0100] At 604, the computing system may process a first three-dimensional location of the plurality of three-dimensional locations and a first two-dimensional gaze direction of the plurality of two-dimensional gaze directions using a neural radiance field model to generate a viewpoint rendering. In some implementations, the neural radiance field model may include one or more multi-layer perceptrons. The viewpoint rendering may describe one or more predicted color values ​​and one or more predicted volume density values. The neural radiance field model may be configured to process the gaze directions and positions to generate one or more predicted color values ​​and one or more predicted density values. The one or more predicted color values ​​and the one or more predicted density values ​​may be utilized to generate the viewpoint rendering. The viewpoint rendering may be a raw viewpoint rendering associated with one or more Bayer filter images associated with one or more red, blue, or green filters. The raw viewpoint rendering may be processed with one or more image enhancement blocks to generate an augmented image having one or more corrected colors, one or more corrected focus, one or more corrected exposure, and / or one or more corrected artifacts.

[0101] At 606, the computing system can evaluate a loss function that evaluates a difference between the viewpoint rendering and a first image of the plurality of noisy raw images. The first image can be associated with at least one of a first three-dimensional location or a first two-dimensional gaze direction. In some implementations, the loss function can include a reweighted L2 loss. Evaluating the loss function that evaluates a difference between the viewpoint rendering and the first image of the plurality of noisy raw images can include mosaic masking. Alternatively and / or additionally, evaluating the loss function that evaluates a difference between the viewpoint rendering and the first image of the plurality of noisy raw images can include exposure adjustment. In some implementations, the first image can include real-world photon signal data generated by a camera. The viewpoint rendering can include predicted photon signal data.

[0102] At 608, the computing system can adjust one or more parameters of the neural radiance field model based at least in part on the loss function. The loss function can include gradient stopping. In some implementations, the gradient stopping can mitigate the neural radiance field model from generalizing to low confidence values.

[0103] In some implementations, the computing system can process the viewpoint renderings with a color correction model to generate a color corrected rendering. Alternatively and / or additionally, the viewpoint renderings can be processed with an exposure correction model to generate an exposure corrected rendering. The color correction model and / or the exposure correction model can be part of an image enhancement block. One or more image correction models can be configured to process the raw signal data and / or the predicted raw signal data. The one or more image correction models can be part of an image enhancement model and can be trained on the Bayer filtered signal data.

[0104] Additionally and / or alternatively, the computing system may obtain an input gaze direction and an input position, process the input gaze direction and input position using the neural radiance field model to generate predicted quad-Bayer filter data, and process the predicted quad-Bayer filter data to generate a novel viewpoint rendering.

[0105] 7 shows a flow chart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. Although FIG. 7 shows steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the particular order or configuration shown. Various steps of method 700 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0106] At 702, a computing system can obtain an input two-dimensional gaze direction and an input three-dimensional position associated with an environment. The environment may include one or more objects. In some implementations, the environment may include low lighting. The input gaze direction and input three-dimensional position can be associated with a request for a novel viewpoint rendering indicating a predicted viewpoint of the environment associated with the position and gaze direction.

[0107] At 704, the computing system may obtain a neural radiance field model. The neural radiance field model may be pre-trained on a training dataset. The training dataset may include a plurality of noisy input datasets associated with the environment. In some implementations, the training dataset may include a plurality of training gaze directions and a plurality of training positions. Each noisy input dataset of the plurality of noisy input datasets may include photon signal data. Additionally and / or alternatively, each noisy input dataset of the plurality of noisy input datasets may include signal data associated with at least one of red values, green values, or blue values. In some implementations, each noisy input dataset of the plurality of noisy input datasets may include one or more noisy mosaic-type linear raw images.

[0108] At 706, the computing system can process the input two-dimensional gaze direction and the input three-dimensional position with the neural radiance field model to generate predicted data. The predicted data can include one or more predicted density values ​​and one or more predicted color values. The predicted data can be utilized to generate predicted Bayer filter data, which can include predicted red filter data, predicted blue filter data, predicted first green filter data, and / or predicted second green filter data. The predicted data can relate to predicted raw image data, which can be processed to generate improved image data.

[0109] At 708, the computing system can process the predicted data using an image augmentation block to generate a predicted viewpoint rendering. The predicted viewpoint rendering can describe a predicted scene rendering of the environment. In some implementations, the image augmentation block can adjust the focus of the predicted data. Additionally and / or alternatively, the image augmentation block can adjust the exposure level of the predicted data. The image augmentation block can adjust the tone mapping of the predicted data.

[0110] 8 shows a flow chart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. Although FIG. 8 shows steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the specifically illustrated order or configuration. Various steps of method 800 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0111] At 802, the computing system may obtain a training dataset. The training dataset may include a plurality of raw input datasets. In some implementations, the training dataset may include a plurality of respective gaze directions and a plurality of respective positions. The plurality of respective gaze directions may include a plurality of two-dimensional gaze directions. The plurality of respective positions may include a plurality of three-dimensional positions. The raw input dataset may include one or more high dynamic range images.

[0112] At 804, the computing system can process the first gaze direction and the first position with the neural radiance field model to generate first predicted data. The first predicted data can describe one or more first predicted color values ​​and one or more first predicted density values. The first predicted data can be related to predicted raw photon signal data.

[0113] At 806, the computing system may evaluate a loss function that evaluates a difference between the first predicted data and a first raw input data set of the plurality of raw input data sets. The first raw input data set may be associated with at least one of a first position or a first gaze direction. In some implementations, the loss function may include a tone mapping loss associated with processing at least one of the first predicted data or the first raw input data set. The loss function may penalize errors more in dark regions than in bright regions to match the way human perception compresses dynamic range. Penalizing may occur after both the first predicted data and the first raw input data set are passed through a tone mapping curve prior to the loss function evaluation. In some implementations, the loss function may include a weighted loss function. The loss may be applied to active color channels of the mosaic-type raw input data and / or the first predicted data. Additionally and / or alternatively, camera intrinsic parameters may be utilized to account for radial distortion when generating light rays.

[0114] At 808, the computing system can adjust one or more parameters of the neural radiance field model based at least in part on the loss function. The one or more parameters can be associated with a learned three-dimensional representation associated with an environment. In some implementations, the one or more parameters can be adjusted to learn the environment.

[0115] In some implementations, the system and method may include training a neural radiance field model for the environment using image data generated using different shutter speeds. For example, the system and method may include processing a second gaze direction and a second location with the neural radiance field model to generate second predicted data. The second predicted data may describe one or more second predicted color values ​​and one or more second predicted density values. The system and method may include scaling the one or more second predicted color values ​​based on the shutter speed to generate scaled second predicted data. A loss function may then be evaluated that evaluates a difference between the scaled second predicted data and a second raw input data set of the plurality of raw input data sets. The second raw input data set may be associated with at least one of the second location or the second gaze direction. One or more additional parameters of the neural radiance field model may be adjusted based at least in part on the loss function.

[0116] Exemplary Implementations Neural Radiance Field (NeRF) can be utilized for high quality novel viewpoint synthesis from a collection of posed input images. NeRF can use tone-mapped low dynamic range (LDR) as input. The images may have been processed by a lossy camera pipeline that smooths details, clips highlights, and distorts the simple noise distribution of the raw sensor data. The systems and methods disclosed herein can include a modified NeRF to train directly on linear raw images, preserving the full dynamic range of the scene. By rendering raw output images from the resulting NeRF, the systems and methods can perform novel high dynamic range (HDR) viewpoint synthesis tasks. In addition to changing the camera viewpoint, the systems and methods can manipulate focus, exposure, and tone mapping after the fact. Although a single raw image appears significantly noisier than a post-processed one, the systems and methods can show that NeRF is extremely robust to zero-mean distribution of raw noise. When optimized over many noisy raw inputs (e.g., 25-200), NeRF can produce accurate scene representations that render novel viewpoints that outperform dedicated single- and multi-image deep raw denoisers operating on the same input images with a wide baseline. In some implementations, the systems and methods can reconstruct scenes from extremely noisy images captured in near-darkness.

[0117] Viewpoint synthesis methods (e.g., Neural Radiance Field (NeRF)) can utilize tone-mapped low dynamic range (LDR) images as input and directly reconstruct and render new viewpoints of the scene in the LDR space. Inputs for scenes that are well-lit and do not contain large luminance variations can be captured with minimal noise using a single fixed camera exposure setting. However, images taken at night or anywhere other than the brightest indoor spaces may have poor signal-to-noise ratios, and scenes with both daylight and shadow areas may have extreme contrast ratios that may rely on high dynamic range (HDR) to be accurately represented.

[0118] Systems and methods (e.g., systems and methods including RawNeRF) can modify NeRF to reconstruct scenes in a linear HDR color space by directly supervising noisy raw input images. The modifications can avoid lossy post-processing that cameras apply to compress dynamic range and smooth noise to produce visually pleasing 8-bit JPEGs. By preserving the full dynamic range of the raw inputs, systems and methods (e.g., systems and methods including RawNeRF) can enable a variety of novel HDR viewpoint synthesis tasks. Systems and methods can modify exposure levels and tone mapping algorithms applied to the rendered output to create synthetically refocused images with accurately rendered bokeh effects around out-of-focus light sources.

[0119] Beyond viewpoint synthesis applications, the system and method can show that training directly on raw data can effectively put RawNeRF into a multi-image denoiser capable of reconstructing scenes captured in near-darkness. Camera post-processing pipelines (e.g., HDR+) can introduce significant biases that corrupt the simple noise distribution of raw data and reduce variance and produce acceptable output images. Feeding images into NeRF can thus produce biased reconstructions with inaccurate colors, especially in the darkest regions of the scene. The system and method can take advantage of NeRF's ability to reduce variance by aggregating information across frames, demonstrating that RawNeRF may be capable of processing to produce clean reconstructions from many noisy raw inputs.

[0120] The systems and methods disclosed herein can assume a static scene and predict the camera pose as input. Given extra constraints, the systems and methods may be able to exploit 3D multi-view consistency to average information across nearly all of the input frames at once. In some implementations, the captured scenes can each contain 25-200 input images, which can mean that the systems and methods can remove more noise than feed-forward single or multi-image denoising networks that utilize 1-5 input images per output.

[0121] The system and method can include directly training a neural radiance field model on raw images capable of handling high dynamic range scenes, as well as noisy inputs captured in the dark. The system and method can outperform NeRF on noisy real and synthetic datasets, and can be a competitive multi-image denoiser for wide-baseline static scenes. The system and method can perform novel viewpoint synthesis applications by utilizing a linear HDR scene representation (e.g., a representation that can include data describing various exposures, tone mappings, and focus).

[0122] Systems and methods (e.g., systems and methods including RawNeRF) can include NeRF as a baseline for high-quality viewpoint synthesis, can utilize low-level image processing to optimize NeRF directly on noisy raw data, and can utilize HDR in computer graphics and computational photography to demonstrate new applications enabled by HDR scene reconstruction.

[0123] Novel viewpoint synthesis can use a set of input images and their camera poses to reconstruct a scene representation capable of rendering novel viewpoints. When the input images are densely sampled, the system and method can use direct interpolation in pixel space for viewpoint synthesis.

[0124] In some implementations, the view synthesis may include learning a volumetric representation rather than a mesh-based scene representation. The NeRF system may directly optimize the neural volumetric scene representation to align all input images using gradient descent against rendering losses. Various extensions may be utilized to improve the robustness of NeRF to changing lighting conditions, and / or supervised learning may be added using depth, time-of-flight data, and / or semantic segmentation labels. In some implementations, to account for inconsistent lighting or calibration deviations between cameras, the view synthesis method may be trained jointly using LDR data to evaluate scaling factors for each image. In some implementations, the system and method may include supervised learning with LDR images, and exposure may be evaluated through a differentiable tone mapping step to restore HDR approximately, but may not care about robustness to noise or supervised learning with raw data. The system and method may include denoising synthetically corrupted sRGB images with additive white Gaussian noise.

[0125] The systems and methods disclosed herein can take advantage of dynamic range preservation, which can enable maximum post-processing flexibility, allowing users to modify exposure, white balance, and tone mapping after the fact.

[0126] When capturing an image, the number of photons that hit a pixel on the camera sensor can be converted to an electric charge, and the charge can be recorded as a digital signal with a large bit depth (e.g., 10-14 bits). The value can be offset by a "black level" to allow for negative actual values ​​due to noise. After black level subtraction, the signal is a quantity x proportional to the expected number of photons that will arrive while the shutter is open. i The noisy measured value y iThe noise comes from both the physical fact that photon arrival can be a Poisson process ("shot" noise) and from noise in the readout circuitry that converts the analog electrical signal to a digital value ("read" noise). The combined shot and read noise distribution can be well modeled as a Gaussian whose variance is an affine function of its mean, which means that the error y i -x i It can be implied that the distribution of has zero mean.

[0127] A color camera may include a Bayer color filter array in front of the image sensor such that the spectral response curve of each pixel measures either red, green, or blue light. The pixel color values ​​may typically be arranged in a 2×2 square (e.g., a Bayer pattern) containing two green pixels, one red pixel, and one blue pixel, resulting in “mosaic” data. To generate a full resolution color image, the missing color channels may be interpolated using a demosaicing algorithm. The interpolation may spatially correlate the noise, and the checkerboard pattern of the mosaic may lead to different noise levels in alternating pixels.

[0128] The spectral response curve for each color filter element may vary between different cameras, and a color correction matrix may be used to convert the image from this camera-specific color space to a standardized color space. Additionally and / or alternatively, because human perception may be robust to color tints imparted by different light sources, the camera may attempt to take color tint into account by scaling each color channel by an estimated white balance factor (e.g., making a white surface appear RGB-neutral white). The two steps can generally be combined into a single linear 3×3 matrix transformation that may further correlate noise between color channels.

[0129] Humans may be able to distinguish smaller relative differences in dark areas compared to light areas of an image. That fact can be exploited by sRGB gamma compression, which can optimize the final image encoding by clipping values ​​outside [0,1] and apply a non-linear curve to the signal that dedicates more bits to dark areas at the expense of compressing bright highlights. In addition to gamma compression, tone mapping algorithms can be used to better preserve contrast in high dynamic range scenes (where bright areas are several orders of magnitude brighter than the darkest ones) when images are quantized to 8 bits.

[0130] Tone mapping can include the process of mapping linear HDR values ​​to a non-linear LDR space for visualization. The signal before tone mapping may be called high dynamic range (HDR) and the signal after may be called low dynamic range (LDR). Of all post-processing operations, tone mapping can affect the noise distribution such that clipping completely discards information in the brightest and darkest regions, and after the non-linear tone mapping curve, noise is no longer guaranteed to be Gaussian, or even 0-mean.

[0131] The Neural Radiance Field (NeRF) model can include a neural network-based scene representation that is optimized to reproduce the look of a set of input images with known camera poses. The resulting reconstruction can then be used to render novel viewpoints from previously unobserved poses. NeRF's multi-layer perceptron (MLP) network can take 3D position and 2D viewing direction as input, and can output volumetric density and color. To render each pixel in the output image, NeRF can use volume rendering to combine color and density from many points sampled along the corresponding 3D ray.

[0132] Standard NeRF can take as input a clean low dynamic range (LDR) sRGB color space image with values ​​in the range [0;1]. Converting a raw HDR image to an LDR image can have two consequences: (1) details in bright areas may be lost when values ​​are clipped from above at 1 and the overall image details are compressed by the tone mapping curve and subsequent quantization to 8 bits, and (2) after passing through a nonlinear tone mapping curve and clipping from below at 0, the noise distribution per pixel may become biased (no longer 0-averaged).

[0133] The systems and methods disclosed herein can optimize the NeRF directly for linear raw input data in HDR color space. The systems and methods can show that reconstructing the NeRF in raw space can be much more robust to noisy inputs, enabling novel HDR view synthesis applications.

[0134] Because color distribution in an HDR image can span many orders of magnitude, a standard L2 loss applied in HDR space can be completely dominated by errors in bright areas and, when tone mapped, can produce an image with dim dark regions with low contrast. The system and method can apply a loss that penalizes errors more strongly in dark regions to match the way human perception compresses dynamic range. One way to achieve such a result is to multiply the rendered estimate by 1 / 2 before the loss is applied.

[0135]

number

[0136] and the noisy observed intensity y can be passed through a tone mapping curve ψ, i.e.

[0137]

number

[0138] In some implementations, in low-light raw images, the observed signal y may be heavily corrupted by zero-mean noise, and the nonlinear tone map may introduce biases that change the prediction of the noisy signal (E[ψ(y)] ≠ ψ(E[y])). In order for the network to converge to an unbiased result, the system and method:

[0139]

number

[0140] One may use a weighted L2 loss of the form:

[0141] The system and method

[0142]

number

[0143] We can approximate a tone mapped loss of this form (1) by using a linearization of the tone curve ψ around

[0144]

number

[0145] In the above equation, sg(·) may denote gradient stopping, which treats the argument as a constant with zero derivative, thereby preventing it from affecting the loss gradient during backpropagation.

[0146] ε=10 -3A “gradient-supervised learning” tone curve ψ(z)=log(y+ε) with

[0147]

number

[0148] can be implied.

[0149] The results can closely correspond to the relative MSE loss used to achieve unbiased results when training on noisy HDR path tracing data in Noise2Noise. The curve ψ can be proportional to the μ-law function used for range compression in audio processing, and has sometimes been applied as a tone mapping function when supervised training a network to map from bursts of LDR images to HDR outputs.

[0150] In some implementations, the system and method can include variable exposure training. In scenes with extremely high dynamic range (e.g., 10-14 bit raw images), it may not be enough to capture both bright and dark areas in a single exposure. The system and method can address this potential problem with a "bracketing" mode included in many digital cameras, where multiple images with varying shutter speeds are captured in bursts and then merged to take advantage of bright highlights that are preserved in the shorter exposures and darker areas that are captured in more detail in the faster exposures.

[0151] The system and method can take advantage of variable exposure in RawNeRF. i Image I i(and all other capture parameters held constant), the system and method i By scaling the image I i The RawNeRF linear spatial color output can be "exposed" to match the luminance in the image. Due to sensor calibration drift, the varying exposure may not be precisely matched using shutter speed alone. For each unique shutter speed present in the set of captured images, the system and method may add a learned per-color channel scaling factor, which can be jointly optimized together with the NeRF network. The output color from the network

[0152]

number

[0153] The final RawNeRF "exposure" given is then:

[0154]

number

[0155] where c indexes the color channel,

[0156]

number

[0157] is the shutter speed t i and the learned scaling factor for channel c (for the longest exposure

[0158]

number

[0159] To account for the fact that pixels saturate in overexposed regions, the system and method may clip from above at 1. The scaled and clipped values ​​can be passed to the loss (Equation 4) previously described.

[0160] The systems and methods disclosed herein may utilize a mip-NeRF code base that can improve upon the positional coding used in the original NeRF method. See the paper on MLP scene representation and volumetric rendering algorithms for further details. The network architecture may include a modification that modifies the activation function for the output color of the MLP from a sigmoidal function to a logarithmic function to better parameterize linear radiance values. The systems and methods are run on a batch of 16k random rays sampled across all training images, and 10 over 500k steps of optimization. -3 From 10 -5 We can use an Adam optimizer with a learning rate that decays to

[0161] Extremely noisy scenes may benefit from a regularization loss on volume density to prevent partially transparent "floater" artifacts. For example, the systems and methods may apply a loss to the variance of the weight distribution used to accumulate color values ​​along rays during volume rendering.

[0162] When the raw input data is mosaic, the raw input data may include one color value per pixel. The system and method may apply a loss to the active color channel for each pixel such that optimizing NeRF effectively demosaices the input image. Because the resampling step may affect the raw noise distribution, the system and method may not undistort or downsample the input, but instead train using a full-resolution mosaic image (e.g., 12MP for the scene). In some implementations, the system and method may utilize camera intrinsic parameters to take radial distortion into account when generating rays. The system and method may utilize full-resolution post-processed JPEG images to calculate the camera pose.

[0163] The systems and methods disclosed herein may be robust to high levels of noise when applied to wide-baseline images of static scenes, to the extent that the system can act as a fair multi-image denoiser. Additionally and / or alternatively, the systems and methods may take advantage of HDR viewpoint synthesis applications, made possible by restoring the scene representation to preserve high dynamic range color values.

[0164] Deep learning methods for directly denoising images in the raw linear domain can include multi-image denoisers that can be applied to burst images or video frames. These multi-image denoisers can assume that there is a relatively small amount of motion between frames, but that there may be a large amount of object motion within the scene. When nearby frames can be well aligned, the method can merge information from similar image patches (e.g., across 2-8 adjacent images) to outperform single-image denoisers.

[0165] NeRF can be optimized for single-scene reconstruction that is consistent with the input image. By specializing in static scenes with a wide baseline and exploiting 3D multi-view information, RawNeRF can aggregate observations from input images that are much more widely spaced than common multi-image denoising methods.

[0166] To test the system, the system and method can acquire a real-world denoising dataset with three different scenes, each containing 101 noisy images and a clean reference image merged from a stable long exposure. The first 100 images can be taken handheld over a wide baseline (e.g., standard forward-looking NeRF capture) using a fast shutter speed to accentuate the noise. The system and method can then capture a stable burst of 50-100 longer exposures on a tripod and robustly merge them using HDR+ to create a clean ground truth frame. One additional tripod image taken with the original fast shutter speed can serve as the noisy input "base frame" for the deep denoising method. All images can be taken with a mobile device at 12MP resolution using a wide-angle lens and saved as 12-bit raw DNG files.

[0167] In some implementations, the systems and methods disclosed herein may utilize only the camera pose, while other techniques may rely on a denoiser that receives a noisy test image.

[0168] Given a complete 3D model of a scene, a physically based renderer can accurately simulate the camera lens defocus blur effect by tracing the refracted rays through each lens element, a process that can be very computationally expensive. In some implementations, the system and method can apply various blur kernels to different depth layers of the scene and composite them together. The system and method can apply a composite defocus blur rendering model to a set of pre-computed RGBA depth layers from a trained RawNeRF model (similar to a multi-planar image). Restoring linear HDR color can be important to achieve the characteristic oversaturated "bokeh ball" around defocused bright light sources.

[0169] Training the neural radiance field model may include a gradient weighted loss. For example, the system and method may reduce the impact of training while converging to an unbiased result by using the following loss:

[0170]

number

[0171] The result is a locally valid linear approximation to the error term, i.e.

[0172]

number

[0173] This can be achieved by using:

[0174] Noisy observation y i ,

[0175]

number

[0176] over the course of training, the true signal value x i =E[y i ], the system and method

[0177]

number

[0178] One can choose to linearize around

[0179] If a weighted L2 loss is used, then when the system is trained, the network

[0180]

number

[0181] (where x i is the true signal value). Therefore, that term is summed in the gradient weighted loss, i.e.

[0182]

number

[0183] is possible, and it is possible over the course of training

[0184]

number

[0185] Additionally and / or alternatively, the gradient of the reweighted loss 7 may be a linear approximation of the gradient of the tone-mapped loss 5, i.e.

[0186]

number

[0187] Additionally and / or alternatively, training may include using a weight variance normalizer, which may be a function of the combination weights used to calculate the final color for each ray. i For each ray segment [t i-1 ,t i ) for MLP output c i , σ i (see [3]), the weights are w i =(1-exp(-Δ i σ i ))exp(-Σ j<i Δ j σ j ) (13) It could be.

[0188] We use the weights to estimate a piecewise constant probability distribution p w If is defined, the variance normalizer is

[0189]

number

[0190] can be equal to

[0191] Calculate the average (predicted depth), i.e.

[0192]

number

[0193] Its value is

[0194]

number

[0195] Compute the normalizer, i.e.,

[0196]

number

[0197] In some implementations, the systems and methods may reduce rendering loss by up to 1×10 (for example, using higher weights in noisier or darker scenes that are more susceptible to “floater” artifacts). -2 and 1×10 -1 The weight between w Applying a regularizer with large weights may result in a minor loss of sharpness, but this can be improved by annealing the weights from 0 to 1 over the course of training.

[0198] The systems and methods may include scaling the loss by a derivative of the desired tone curve, i.e.

[0199]

number

[0200] The system and method include: -3 and p=1 was found to yield the best qualitative results.

[0201]

number

[0202] One can perform a hyperparameter sweep over loss weights of the form

[0203] In some implementations, the systems and methods may utilize a reweighted L1 loss or negative log-likelihood function of the actual camera noise model (using shot / read noise parameters from EXIF ​​data). Alternatively and / or additionally, RawNeRF models supervised trained with unweighted standard L2 or L1 losses may tend to diverge early during training, especially in extremely noisy scenes.

[0204] The systems and methods may utilize an unclipped sRGB gamma curve (extended as a linear function below 0 and an exponential function above 1) in the loss. Applying a logarithmic tone curve directly before the L2 loss (rather than reweighting by its gradient) may cause training to diverge.

[0205] color correction matrix C ccm is the corresponding RGB to XYZ matrix, i.e.

[0206]

number

[0207] can be used to convert from XYZ to camera RGB under D65 light source.

[0208] The system and method includes a single color transformation C directly from camera RGB to a standard linear RGB space. all These may be used to create the mapping. C all =rownorm((C rgb-xyz C ccm ) -1 ) (twenty four) In the above formula, rownorm normalizes each one so that the sum is 1.

[0209] The system and method can use the standard sRGB gamma curve as a base tone map for linear RGB spatial data, i.e.

[0210]

number

[0211] To minimize the effects of image noise, the system and method calculates the average color value for each Bayer filter channel (R, G1, G2, B) across the entire 12MP sensor.

[0212]

number

[0213] For example, the system and method may determine a maximum shutter speed t max Normalized luminance at speed t i is the normalized luminance ratio in

[0214]

number

[0215] In the case of perfect calibration, the plot may be equal to 1 everywhere since apportioning by shutter speed should perfectly normalize the luminance values. However, the quantity may decay for faster shutter speeds, and the quantity may decay at different rates for each color channel. In some implementations, a DSLR or mirrorless camera with a better sensor may be utilized.

[0216] The systems and methods can evaluate affine color matching between each output and the ground truth screen image. For all methods except SID and LDR NeRF, the method can be performed directly in raw Bayer space for each RGB plane separately. For SID and LDR NeRF (which output images in tone-mapped sRGB space), the method can be performed for each RGB plane on the tone-mapped sRGB clean image. If the ground truth channel is x and the channel to be matched is y, the systems and methods can be used to obtain a least-squares fit of the affine transformation ax+b≈y.

[0217]

number

[0218] can be calculated (where:

[0219]

number

[0220] (where denotes the average over all elements of z). The systems and methods can then apply an inverse transform as (yb) / a to align the estimated y to x. In some implementations where the alignment occurs in the raw domain, the systems and methods can post-process (yb) / a through our standard pipeline before computing the sRGB spatial metrics.

[0221] To render defocused images, the system and method can utilize a specific synthetic defocus rendering model for the specific task. To avoid prohibitively expensive rendering speed, the system and method can first pre-compute a multi-planar image representation from a trained neural radiance field model. The MPI can include a set of forward-parallel RGBA planes (with colors also in linear HDR space) linearly sampled in the disparity within the camera frustum at the center camera pose.

[0222] Additional Disclosures The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate serially or in parallel.

[0223] Although the present subject matter has been described in detail with respect to various specific exemplary embodiments thereof, each example is provided for the purpose of illustration and not limitation of the present disclosure. Those skilled in the art, upon understanding the above, can easily create modifications, variations, and equivalents of such embodiments. Thus, the present disclosure does not exclude modifications, variations, and / or additional inclusions of the present subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may also be used with another embodiment to create yet a further embodiment. Thus, it is intended that the present disclosure covers such modifications, variations, and equivalents. [Explanation of symbols]

[0224] 10. Computing Devices 50 Computing Devices 100 Computing Systems 102 User Computing Devices 112 processors 114 Memory 116 Data 118 Command 120 Neural Radiance Field Model 122 User Input Components 130 Server Computing System 132 processors 134 Memory 136 Data 138 Command 140 Neural Radiance Field Model 150 Training Computing System 152 processors 154 Memory 156 Data 158 Command 160 Model Trainer 162 Training Data 180 Network 200 Neural Radiance Field Model 202 RawNeRF Models 204 Input Data 206 Output Data 208 Color Correction Block 210 Improved Perspective Rendering 212 Low Dynamic Range Neural Radiance Field Pipeline 300 Neural Radiance Field Model 302 Training Data 304 Neural Radiance Field Model 306 Forecast Data 308 Loss Function 310 Image Processing Pipeline 400 Viewpoint Rendering Pipeline 402 Input Data 404 High Dynamic Range Neural Radiance Field Model Pipeline 406 Low Dynamic Range Neural Radiance Field Model Pipeline 410 High Dynamic Range Viewpoints 418 Low Dynamic Range Viewpoint 500 Neural Radiance Field Model Training 502 3D position 504 2D viewing direction 506 Neural Radiance Field Model 508 Forecast Data 510 Image Enhancement Model 512 New Viewpoint Rendering 514 raw image dataset 516 Loss Function

Claims

1. 1. A computing system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations including: acquiring a training dataset, the training dataset comprising a plurality of 3D positions, a plurality of 2D gaze directions, and a plurality of noisy raw images, the plurality of noisy raw images comprising a plurality of high dynamic range images comprising a plurality of raw bits structured in a raw format; processing a first one of the plurality of three-dimensional locations and a first one of the plurality of two-dimensional gaze directions with a neural radiance field model to generate a viewpoint rendering, the neural radiance field model comprising one or more multi-layer perceptrons, the viewpoint rendering describing one or more predicted color values ​​and one or more predicted volume density values; evaluating a loss function that evaluates a difference between the viewpoint rendering and a first image of the plurality of noisy raw images, the first image being associated with at least one of the first three-dimensional location or the first two-dimensional viewing direction; adjusting one or more parameters of the neural radiance field model based at least in part on the loss function; and A computing system comprising:

2. The operation, The computing system of claim 1 , further comprising processing the viewpoint rendering with a color correction model to generate a color corrected rendering.

3. 3. The computing system of claim 1 or 2, wherein the loss function comprises a reweighted L2 loss.

4. 4. The computing system of claim 1, wherein evaluating the loss function that evaluates the difference between the viewpoint rendering and the first image of the noisy plurality of raw images comprises mosaic masking.

5. 5. The computing system of claim 1, wherein evaluating the loss function that evaluates the difference between the viewpoint rendering and the first image of the noisy plurality of raw images comprises exposure adjustment.

6. The operation, Obtaining an input gaze direction and an input position; processing the input gaze direction and the input position with the neural radiance field model to generate predicted Quad Bayer filtered data; processing the predicted Quad Bayer filtered data to generate a new viewpoint rendering; The computing system of claim 1 , further comprising:

7. 7. The computing system of claim 1, wherein the loss function comprises gradient stopping, the gradient stopping mitigating the neural radiance field model from generalizing to low confidence values.

8. 8. The computing system of claim 1, wherein the first image comprises real-world photon signal data generated by a camera and the viewpoint rendering comprises predicted photon signal data.

9. 9. The computing system of claim 1, wherein the plurality of noisy raw images is associated with a plurality of Red-Green-Green-Blue color data sets.

10. 1. A computer-implemented method for novel viewpoint rendering, comprising: obtaining, by a computing system comprising one or more processors, an input 2D gaze direction and an input 3D position relative to an environment; obtaining, by the computing system, a neural radiance field model, the neural radiance field model having been trained on a training data set, the training data set comprising a plurality of noisy input data sets related to the environment, the training data set comprising a plurality of training gaze directions and a plurality of training positions; processing, by the computing system, the input 2D gaze direction and the input 3D position using the neural radiance field model to generate predicted data, the predicted data comprising one or more predicted density values ​​and one or more predicted color values; processing the predicted data by the computing system using an image enhancement block to generate a predicted viewpoint rendering, the predicted viewpoint rendering describing a predicted scene rendering of the environment; A method implemented by a computer.

11. The method of claim 10 , wherein the image enhancement block adjusts the focus of the predicted data.

12. The method according to claim 10 or 11, wherein the image enhancement block adjusts an exposure level of the predicted data.

13. The method according to claim 10 , wherein the image enhancement block adjusts a tone mapping of the predicted data.

14. 14. The method of claim 10, wherein each noisy input data set of the plurality of noisy input data sets comprises photon signal data.

15. 15. The method of claim 10, wherein each noisy input dataset of the plurality of noisy input datasets comprises signal data relating to at least one of a red value, a green value, or a blue value.

16. 16. The method of claim 10, wherein each noisy input dataset of the plurality of noisy input datasets comprises one or more noisy mosaic-type linear raw images.

17. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations including: acquiring a training dataset, the training dataset comprising a plurality of raw input datasets, the training dataset comprising a plurality of respective gaze directions and a plurality of respective positions; processing the first gaze direction and the first position with a neural radiance field model to generate first predicted data, the first predicted data describing one or more first predicted color values ​​and one or more first predicted density values; evaluating a loss function that evaluates a difference between the first prediction data and a first raw input dataset of the plurality of raw input datasets, the first raw input dataset being associated with at least one of the first location or the first gaze direction; adjusting one or more parameters of the neural radiance field model based at least in part on the loss function; and One or more non-transitory computer-readable media comprising:

18. 20. The one or more non-transitory computer-readable media of claim 17, wherein the one or more parameters relate to a learned three-dimensional representation associated with an environment.

19. 19. The one or more non-transitory computer-readable media of claim 17 or 18, wherein the loss function comprises a tone mapping loss associated with processing at least one of the first prediction data or the first raw input data set.

20. The operation, processing the second gaze direction and the second position with the neural radiance field model to generate second predicted data, the second predicted data describing one or more second predicted color values ​​and one or more second predicted density values; scaling the one or more second predicted color values ​​based on a shutter speed to generate scaled second predicted data; evaluating the loss function that evaluates a difference between the scaled second prediction data and a second raw input data set of the plurality of raw input data sets, the second raw input data set being associated with at least one of the second location or the second gaze direction; adjusting one or more additional parameters of the neural radiance field model based at least in part on the loss function; and 20. The one or more non-transitory computer-readable media of claim 17, further comprising:

Images