Efficient Neural Luminance Field Rendering

Abstract

We present a new neural luminance field (NeRF) representation based on textured polygons that can efficiently synthesize novel images using standard rendering pipelines. NeRF is represented as a collection of polygons with textures representing binary opacity and feature vectors. Traditional rendering of polygons using a mesh rasterizer (e.g., involving the use of a z-buffer) results in an image with features at every pixel, which are interpreted by a small view-dependent MLP running in the fragment shader to produce the final pixel color. This approach allows NeRF to be rendered using traditional polygon rasterization pipelines, resulting in massive pixel-level parallelism, achieving interactive frame rates on a wide range of computing platforms, including mobile phones.

Description

[Technical Field]

[0001] Priority claims This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 392,060, filed July 25, 2022, which is incorporated herein by reference.

[0002] The present disclosure relates generally to machine learning and, more particularly, to systems and methods that leverage polygon rasterization pipelines for efficient neural field rendering, for example, on mobile architectures. [Background technology]

[0003] Neural radiance fields (NeRF) have become a popular representation for novel view synthesis of 3D scenes. The NeRF model represents a scene using a multilayer perceptron (MLP) that computes a 5-dimensional implicit function that estimates density and radiance emanating from any position in any direction. It can be used in a volume rendering framework to generate novel images. See Mildenhall et al., Nerf: Representing scenes as neural radiance fields for view synthesis (ECCV, 2020). NeRF representations, optimized to minimize multi-view color consistency loss for a set of posed photographs, have demonstrated excellent ability to reproduce fine image details in novel views.

[0004] However, one of the main obstacles to widespread adoption of NeRF is that it requires specialized rendering algorithms that are not compatible with commonly available hardware. In particular, traditional NeRF implementations use volume rendering algorithms that evaluate large MLPs at hundreds of sample locations along rays per pixel to estimate and integrate density and radiance. This rendering process is far too slow for interactive visualization. Summary of the Invention

[0005] Aspects and advantages of embodiments of the present disclosure are set forth in part in the detailed description that follows, or may be learned from the detailed description, or may be learned by practice of the embodiments.

[0006] One exemplary aspect is directed to a computing device for efficient image rendering. The computing device includes one or more processors. The computing device includes one or more non-transitory computer-readable media that collectively store a mesh model of a scene and one or more texture images including learned feature and opacity data for the mesh model. The computing device includes a rendering pipeline configured to generate a rendered image of the scene from a specific camera pose. The rendering pipeline includes a mesh rasterizer configured to generate a feature image including a plurality of fragments, each fragment corresponding to at least a portion of the mesh model as seen from the specific camera pose, and for each fragment, the feature image includes respective learned feature and opacity data provided by one or more texture images for the corresponding portion of the mesh model. The rendering pipeline includes a machine-learned neural fragment shader configured to process the specific camera pose and the respective learned feature and opacity data for each fragment in the feature image to output a respective output color for each fragment for use in the rendered image.

[0007] Another exemplary aspect is directed to a computer-implemented method for generating a rendered image of a scene from a specific camera pose. The method includes, by a computing system including one or more computing devices, acquiring a mesh model of the scene and one or more texture images including learned feature and opacity data for the mesh model. The method includes generating, by the computing system, a feature image including a plurality of fragments, each fragment corresponding to at least a portion of the mesh model as seen from the specific camera pose, and for each fragment, the feature image including respective learned feature and opacity data provided by the one or more texture images for the corresponding portion of the mesh model. The method includes processing, by the computing system using a machine-learned neural fragment shader, the specific camera pose and the respective learned feature and opacity data for each fragment in the feature image to output a respective output color for each fragment. The method includes providing, by the computing system, the rendered image of the scene as an output, the rendered image of the scene having the respective output colors provided by the machine-learned neural fragment shader.

[0008] Another exemplary aspect is directed to one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations of one or more training iterations. The operations include acquiring training images depicting a scene from a camera pose. The operations include processing a plurality of positions associated with a learnable mesh model using a feature field model to generate feature data and using an opacity field model to generate opacity data. The operations include processing the feature data and the camera pose using a neural fragment shader to generate color data. The operations include alpha compositing the color data and the opacity data based on the camera pose to generate one or more output colors for a rendered image. The operations include determining a loss function that compares the rendered image to the training images to determine a loss. The operations include modifying one or more parameter values ​​for one or more of the learnable mesh model, the feature field model, the opacity field model, and the neural fragment shader based on the loss.

[0009] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following detailed description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the detailed description, serve to explain associated principles.

[0011] Detailed descriptions of embodiments directed to those skilled in the art are set forth herein with reference to the accompanying drawings. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an exemplary rendering pipeline for efficient neural luminance field rendering, according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram of an exemplary training approach for efficient neural luminance field rendering, according to an exemplary embodiment of the present disclosure. [Figure 3A] FIG. 1 is a block diagram of an exemplary computing system according to an exemplary embodiment of the present disclosure. [Figure 3B] FIG. 2 is a block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure. [Figure 3C] FIG. 2 is a block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Reference numbers repeated among the figures are intended to identify like features in the various embodiments.

[0014] overview This disclosure is generally directed to a novel neural luminance field (NeRF) representation based on textured polygons that can efficiently synthesize novel images in a standard rendering pipeline. In particular, NeRF can be represented as a collection of polygons with textures representing binary opacity and feature vectors. Traditional rendering of polygons using a mesh rasterizer (e.g., involving the use of a z-buffer) results in a feature image with features for every fragment (e.g., pixel). Features in the feature image can be interpreted by a neural fragment shader (e.g., a small view-dependent MLP running in the fragment shader) to generate the final pixel color. This approach enables NeRF to be rendered in a traditional polygon rasterization pipeline, which provides massive pixel-level parallelism, achieving interactive frame rates on a wide range of computing platforms, including mobile phones and other mobile or embedded architectures.

[0015] More specifically, NeRF has demonstrated an impressive ability to synthesize images of 3D scenes from novel views. However, they rely on specialized volume rendering algorithms based on ray marching that do not match the capabilities of widely deployed graphics hardware. Therefore, regular NeRF cannot be used for practical rendering on ubiquitous mobile architectures such as smartphones.

[0016] While approaches that precompute NeRF and "bake" it onto a sparse 3D voxel grid may speed up NeRF rendering, these approaches still rely on ray marching through the sparse voxel grid to generate features for each pixel, and therefore do not fully utilize the parallelism available in commodity graphics processing units (GPUs). Furthermore, these approaches require a significant amount of GPU memory to store volume textures, prohibiting them from running on typical mobile devices.

[0017] In light of these issues, this disclosure introduces MobileNeRF, a NeRF that can run on a variety of popular mobile devices in real time. A NeRF can be represented by a mesh model and one or more texture image(s). For example, the mesh model can include a set of polygons with associated textures, where the polygons roughly follow the surface of the scene, and the texture image(s) store the opacity and feature vectors learned for the polygons during training.

[0018] To render an image, an exemplary implementation may utilize a polygon rasterization pipeline framework. For example, the rendering pipeline may include a mesh rasterizer and a neural fragment shader. The mesh rasterizer may perform z-buffering and generate a feature vector for each fragment (e.g., pixel) of the image to be rendered. The feature vector for each fragment may then be processed by a neural fragment shader (e.g., a lightweight MLP running in a GLSL fragment shader) to generate an output color.

[0019] Some example implementations of this rendering pipeline do not sample rays or sort polygons by depth, and therefore can only model binary opacity. However, the rendering pipeline is able to fully exploit the parallelism offered by z-buffers and fragment shaders in modern graphics hardware, making it 10 times faster than current state-of-the-art approaches with the same output quality on a standard test scene.

[0020] Furthermore, the MobileNeRF approach described herein requires only a standard polygon rendering pipeline framework and can be implemented and accelerated on virtually any computing platform, allowing the MobileNeRF approach to work on mobile phones and other devices that previously could not support NeRF visualization at interactive rates.

[0021] Thus, the present disclosure provides several technical effects and advantages: As an example, the proposed model and associated rendering approach is 10 times faster than current techniques at the same output quality.

[0022] Among other exemplary technical effects and advantages, the proposed technique consumes less memory by storing surface textures instead of volume textures, allowing the proposed method to run on integrated GPUs with significantly less memory and power.

[0023] As yet another exemplary technical effect and advantage, the proposed technology can be implemented to run in a web browser and can be scalable to a large number of different devices, for example, the viewer can be written in HTML using triple.js and run on a large number of devices worldwide.

[0024] As yet another exemplary technical effect and advantage, the proposed technique allows for real-time manipulation of the reconstructed objects / scenes, which are simple triangular meshes.

[0025] Referring now to the drawings, exemplary embodiments of the present disclosure will be described in more detail.

[0026] An example rendering pipeline Figure 1 illustrates a block diagram of an exemplary rendering pipeline for efficient neural luminance field rendering, according to an exemplary embodiment of the present disclosure. Specifically, Figure 1 illustrates a rendering pipeline 18 configured to generate a rendered image 26 of a scene when provided with the following rendering assets / information: a mesh model 12 of the scene, one or more texture images 14 containing learned feature and opacity data for the mesh model 12, and a camera pose 16 at which the rendered image 26 is to be displayed / rendered.

[0027] The rendering pipeline 18 includes a mesh rasterizer 20 and a neural fragment shader 24. The mesh rasterizer 20 is configured to generate a feature image 22 that includes multiple fragments (e.g., pixels). Each fragment in the feature image 22 may correspond to at least a portion (e.g., a polygon) of the mesh model 12 that is visible from a particular camera pose 16. Furthermore, for each fragment, the feature image 22 includes respective learned feature and opacity data provided by one or more texture images for the corresponding portion (e.g., polygon) of the mesh model.

[0028] The neural fragment shader 24 is configured to process the particular camera pose 16 and the respective learned feature and opacity data for each fragment of the feature image 22 to output a respective output color for each fragment for use in the rendered image 26. That is, for each fragment (e.g., pixel) of the feature image 22, the neural fragment shader 24 can provide an output color, which can be used to generate the rendered image 26.

[0029] In some implementations, the learned feature and opacity data stored in texture image(s) 14 includes data output by one or more machine-learned neural luminance field models trained on training images depicting a scene. In some implementations, machine-learned neural fragment shader 24 is co-trained with the one or more machine-learned neural luminance field models. In some implementations, the one or more machine-learned neural luminance field models include a feature field model configured to process locations to generate feature data for the locations and an opacity field model configured to process locations to generate opacity data for the locations. In some implementations, mesh model 12 includes only polygons visible from one or more training images of the scene. Restricting mesh model 12 to polygons visible from one or more training images can improve the quality of mesh model 12 (e.g., by reducing erroneous or illusory scene structures that do not match the semantics of the scene).

[0030] 1 , in some implementations, feature image 22 may be generated at a first resolution, and rendering pipeline 18 may be configured to downsample feature image 22 to a second, smaller resolution before running machine-learned neural fragment shader 24 on feature image 22. For example, the second resolution may be half the first resolution. Supersampling feature image 22 and then subsampling before applying shader 24 may result in averaging feature and opacity information contained in supersampled feature image 22. Averaging feature and opacity information may provide benefits such as improved anti-aliasing of rendered image 26.

[0031] According to one aspect of the present disclosure, rendering pipeline 18 can be very similar in framework and execution to existing mesh model rendering pipelines. Thus, rendering pipeline 18 can run on many common computing devices, such as smartphones, mobile devices (e.g., tablets, smartwatches, home assistant devices, etc.), or embedded devices (e.g., in-vehicle computing devices, home appliances, Internet of Things devices, etc.). For example, in some implementations, the mesh rasterizer and machine-learned neural fragment shader can be executed using one or more graphics processing units and / or other high-speed hardware optimized for image rendering.

[0032] In some implementations, to achieve compatibility with these existing rendering frameworks, the rendering pipeline 18 and rendering assets, such as the mesh model 12 and texture image(s) 14, may be defined in, stored as, or otherwise executed using various common file formats. As an example, in some implementations, the mesh model 12 may be stored as an OBJ file, one or more texture images 14 may be stored as one or more Portable Network Graphics (PNG) files, and / or the weights of the machine-learned neural fragment shader 24 may be stored in a JavaScript Object Notation (JSON) file. As another example, in some implementations, the rendering pipeline 18 may be defined in, stored as, or otherwise executed using Hypertext Markup Language (HTML). As another example, in some implementations, the neural fragment shader 24 may be defined in, stored as, or otherwise executed using OpenGL Shading Language (GLSL).

[0033] Exemplary Training Approaches 2 shows a block diagram of an exemplary training approach for efficient neural luminance field rendering, according to an exemplary embodiment of the present disclosure. The process shown in FIG. 2 can be performed for each of several training images depicting a scene. In some implementations, training can be performed in multiple stages, each stage including a different sequence of the operations shown in FIG. 2 (and / or other operations).

[0034] Specifically, a computing system may acquire training images 206 depicting a scene from a camera pose 202. The computing system may process a plurality of positions associated with the learnable mesh model 204 with a feature field model 208 to generate feature data 212 and with an opacity field model 210 to generate opacity data 214. By way of example, feature field model 208 and opacity field model 210 may be neural luminance field models (e.g., implemented using a relatively large MLP).

[0035] The computing system may process the feature data 212 and the camera pose 202 with a neural fragment shader (e.g., implemented using a relatively small MLP) to generate color data 220. The computing system may perform alpha compositing 222 of the color data 220 and the opacity data 214 based on the camera pose 202 to generate one or more output colors for a rendered image 224.

[0036] The computing system may determine a loss function 226 that compares the rendered image 224 to the training images 206 to determine the loss. For example, the loss function 226 may be a mean squared error for pixel color.

[0037] Based on the loss, the computing system can modify one or more parameter values ​​for one or more of the trainable mesh model 204, the feature field model 208, the opacity field model 210, and the neural fragment shader 216. For example, the loss and loss function 226 can be backpropagated 228 through the model as shown by the dashed line in FIG.

[0038] In some implementations, for at least one of the one or more training iterations, the computing system may binarize 218 the opacity data before performing alpha compositing 222. For example, in some implementations, training may be performed without binarization 218 in the first stage (i.e., continuous opacity), while training may be performed with binarization 218 in the second stage. In some implementations, the second stage may include training with both binarization 218 and continuous opacity. In some implementations, the loss may be backpropagated through opacity binarization 218 using straight-through estimation.

[0039] In some implementations, alpha compositing 222 may occur on the feature data 212 and opacity data 214 before applying the neural fragment shader 216 (not shown). The neural fragment shader 216 may then be applied to the output of the alpha compositing 222 on the feature data 212.

[0040] In some implementations, for at least one of the one or more training iterations, the computing system may subsample the feature data 212 before processing the feature data 212 and the camera pose 202 in the neural fragment shader 216 to generate the color data 220.

[0041] In some implementations, after training iterations, the computing system may bake the feature data 212 and opacity data 214 output by feature field model 208 and opacity field model 210, respectively, into one or more texture images (not shown). For example, for every position (e.g., polygon) of mesh model 204, feature field model 208 and opacity field model 210 may be determined, and their respective outputs may be stored in one or more texture images for later use in rendering.

[0042] As another example, in some implementations, after one or more training iterations, any portions (e.g., polygons) of the mesh model that are not visible in any of multiple training images of the scene may be pruned. This may improve the quality of the mesh model 204 and reduce the memory requirements of the mesh model 204. In some implementations, the trainable mesh model 204 may be or include a grid mesh. Modifying one or more parameter values ​​of the trainable mesh model 204 may include updating the positions of the vertices of the grid mesh while holding the topology of the grid mesh fixed.

[0043] Exemplary Implementations This section describes an exemplary implementation of the generic framework described herein.

[0044] Given a collection of (calibrated) images, some exemplary implementations attempt to optimize the representation for efficient novel view synthesis. One example of a proposed representation includes a polygon mesh where texture maps store features and opacity. At rendering time, given the camera pose, some exemplary implementations employ a two-stage deferred rendering process.

[0045] Rendering Stage 1 - In some exemplary implementations, the mesh is rasterized and screened spatially to construct a feature image. For example, in some exemplary implementations, a deferred rendering buffer is created in GPU memory.

[0046] Rendering Stage 2 - In some example implementations, these features are converted into a color image via a fragment shader, e.g., a (neural) deferred renderer running on a small MLP, which receives the feature vector and view direction and outputs the pixel color.

[0047] An example of the proposed representation is constructed in three training stages, gradually moving from a classical NeRF-like continuous representation to a discrete representation.

[0048] Training Stage 1 (Stage 1)—Some exemplary implementations train a NeRF-like model with continuous opacity, where volume rendering orthogonal points are derived from a polygon mesh.

[0049] Training Stage 2 (Stage 2) In some exemplary implementations, opacity is binarized because classical rasterization can easily discard fragments but cannot properly deal with semi-transparent fragments.

[0050] Training Stage 3 (Stage 3) - In some exemplary implementations, extract a sparse polygon mesh, bake opacity and features into texture maps, and store neural deferred shader weights.

[0051] As an example, meshes can be stored as OBJ files, texture maps in PNG, and deferred shader weights in a (small) JSON file. In some exemplary implementations, we utilize the standard GPU rasterization pipeline, so one example of a proposed real-time renderer is simply an HTML web page.

[0052] Since representing a continuous signal in a discrete representation can introduce aliasing, some exemplary implementations also include a simple yet computationally efficient anti-aliasing solution based on supersampling (anti-aliasing).

[0053] Continuous training technique example (Training Stage 1) One exemplary proposed training setup consists of a polygonal mesh M = (T, V) and three learnable models, e.g., MLPs. The mesh topology T is fixed, but the vertex positions V and the MLPs are optimized in an auto-decoding manner by minimizing the mean squared error between the predicted and ground truth colors of pixels in the training images, similar to NeRF.

number

number

number

number

[0054] The small network H is an example of the proposed deferred neural shader, which outputs the color of each fragment given the fragment's features and the viewing direction. Finally, note that (2) does not composite by volume density, but by opacity [see, e.g., Eq. 8].

[0055] Example polygon mesh Without loss of generality, some exemplary implementations operate on polygon meshes used in synthetic 360° scenes, providing construction of forward-bounded 360° scenes. In some exemplary implementations, first, a regular grid g of size P×P×P is defined in a unit cube centered at the origin. In some exemplary implementations, V is instantiated by creating one vertex per voxel, and T is instantiated by creating one quadrangle (two triangles) per grid edge connecting the vertices of four adjacent voxels. In some exemplary implementations, V∈[-.5,5,+.5] is defined by locally parameterizing the vertex positions with respect to the voxel centers (and sizes). P×P×P×3 Free variables are obtained. During optimization, some exemplary embodiments initialize the vertex positions to V=0, which corresponds to a regular Euclidean lattice, and some exemplary embodiments regularize the vertex positions to prevent vertices from leaving their voxels and encourage them to return to their neutral positions whenever the optimization problem is under-constrained.

number

number

[0056] Example Query Execution Because solving for the MLP of some exemplary representations is computationally expensive, some exemplary implementations rely on an acceleration grid to limit the cardinality |K| of the orthogonal points. First, orthogonal points are generated only for the set of voxels intersected by the ray. Second, some exemplary implementations utilize an acceleration grid g to prune voxels that are unlikely to contain geometry. Finally, some exemplary implementations calculate the intersection points between rays incident on the vertices of voxels and the faces of M to obtain the final set of orthogonal points. Some exemplary implementations use barycentric interpolation to backpropagate gradients from the intersection points to the three vertices of the intersected triangle. In summary, some exemplary implementations calculate, for each input ray r,

number

number

[0057] This can be interpreted as a way to compute the so-called "surface field" during NeRF training, as opposed to after training. Some exemplary implementations further regularize the grid content by promoting its pointwise sparsity (e.g., lasso) and its spatial smoothness.

number

[0058] Exemplary Binarization Training (Training Stage 2) Rendering pipelines implemented in typical hardware do not natively support translucent meshes. Rendering translucent meshes requires complex (per-frame) sorting, so rendering is performed in back-to-front order to ensure correct alpha compositing. In some exemplary implementations, smooth opacity αK∈[0,1] is converted from opacity (3) to discrete / categorical opacity

number

number

[0059] α K Values ​​and Results

number

number

number

number

number

number

[0060] Once MSE_plus_binary(14) has converged, in some example implementations, while holding other weights fixed,

number

[0061] Exemplary discretization (training stage 3) After binarization and refinement, some exemplary implementations convert the representation to an explicit polygon mesh (e.g., OBJ format). Some exemplary implementations store quads only if they are at least partially visible in the training camera pose (i.e., non-visible quads are discarded). Some exemplary implementations then create a texture image whose size is proportional to the number of visible quads, and some exemplary implementations allocate K x K patches to the texture for each quad. Some exemplary implementations use K = 17, so that quads have a 16 x 16 texture with half-pixel border padding. Some exemplary implementations then iterate over the pixels of the texture, converting pixel coordinates to 3D coordinates and baking discrete opacity (e.g., Opacity(3) and dOpacity(12)) and feature (e.g., Feature(4)) values ​​into a texture map. Some exemplary implementations quantize the [0,1] range to 8-bit integers and store the texture in a (e.g., losslessly compressed) PNG image. An exemplary experiment shows that quantizing the [0,1] range with 8-bit precision, which is not applicable during backpropagation, does not significantly affect the rendering quality.

[0062] Example Anti-Aliasing In classical rasterization pipelines, aliasing is a problem that must be considered to obtain high-quality rendering. Classical NeRF creates the illusion of smooth edges through translucent volumes, but as described above, translucency requires polygon sorting every frame. Some exemplary implementations overcome this problem by utilizing anti-aliasing through supersampling. While it is possible to simply run (5) four times per pixel and average the resulting colors, the execution of the deferred neural shader H is the computational bottleneck of one exemplary proposed technique. Some exemplary implementations can overcome this problem by simply averaging the features, i.e., by averaging the inputs of the deferred neural shader rather than averaging its outputs. Some exemplary implementations first rasterize the features (at 2x resolution).

number

number

[0063] Note that with this modification, in some exemplary implementations, query H is performed only once per output pixel. Finally, this process is performed using discrete occupancies.

number

[0064] Example Rendering In some implementations, the result of the optimization process is a textured polygon mesh (the texture map stores features rather than colors) and a small MLP (which translates view directions and features into colors). Rendering this representation is done in two passes using a deferred rendering pipeline.

[0065] 1. In some example implementations, all faces of the textured mesh are z-buffered and rasterized to generate a 12-channel 2M x 2N feature image per pixel, including 8 channels of learned features, binary opacity, and 3D view direction.

[0066] 2. In some example implementations, an MxN output RGB image is synthesized by rendering a textured rectangle that uses the feature image as a texture, and linear filtering averages the features for anti-aliasing. In some example implementations, a small MLP is applied to pixels with non-zero alpha to convert the features to RGB color. The small MLP can be implemented as a GLSL fragment shader.

[0067] These rendering steps can be implemented within a classical rasterization pipeline. Because z-buffering with binary transparency is order-independent, polygons do not need to be depth-sorted for each new view, and polygons can therefore be loaded into the GPU's buffer at the start of execution. The MLP for converting features to color is very small, so it can be implemented in a GLSL fragment shader running in parallel for every pixel. Because these classical rendering steps are highly optimized for the GPU, one exemplary proposed rendering system can run at interactive frame rates on a wide variety of devices. It is also simple to implement, requiring only standard polygon rendering using a fragment shader. One exemplary interactive viewer is a JavaScript-based HTML web page rendered by WebGL via the threejs library.

[0068] Exemplary Devices and Systems 3A illustrates a block diagram of an exemplary computing system 100 according to an exemplary embodiment of the present disclosure. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0069] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0070] The computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operatively connected processors. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0071] In some implementations, the user computing device 102 can store or include one or more machine-learned models 120. For example, the machine-learned models 120 can be or otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks), or other types of machine-learned models, including nonlinear and / or linear models. The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine-learned models can leverage attention mechanisms, such as self-attention. For example, some exemplary machine-learned models can include multi-head self-attention models (e.g., Transformer models).

[0072] In some implementations, one or more machine-learned models 120 may be received from server computing system 130 over network 180, may be stored in user computing device memory 114, and may then be used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instances of a single machine-learned model 120 (e.g., to perform parallel rendering across multiple instances of a pixel).

[0073] Additionally or alternatively, one or more machine-learned models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned models 140 may be implemented by the server computing system 130 as part of a web service (e.g., an image rendering service). Thus, one or more models 120 may be stored and implemented at the user computing device 102 and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0074] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0075] The user computing device 102 may also include a rendering pipeline 124. For example, the rendering pipeline 124 may be the pipeline shown or described with reference to FIG. 1. The rendering pipeline 124 includes computer logic utilized to provide desired functionality. The rendering pipeline 124 may be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, the rendering pipeline 124 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, the rendering pipeline 124 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or optical or magnetic media.

[0076] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operatively connected processors. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to operate.

[0077] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. When server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0078] As described above, the server computing system 130 may store or otherwise include one or more machine-learned models 140. For example, the models 140 may be or otherwise include various machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine-learned models may utilize attention mechanisms such as self-attention. For example, some exemplary machine-learned models may include multi-head self-attention models (e.g., Transformer models).

[0079] Server computing system 130 may also include a rendering pipeline 142. For example, rendering pipeline 142 may be the pipeline shown or described with reference to FIG. 1. Rendering pipeline 142 includes computer logic utilized to provide desired functionality. Rendering pipeline 142 may be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, rendering pipeline 142 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, rendering pipeline 142 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or optical or magnetic media.

[0080] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 by interacting with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 can be separate from the server computing system 130 or can be part of the server computing system 130.

[0081] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operably connected processors. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0082] The training computing system 150 may include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored on the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation. For example, a loss function may be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent may be used to iteratively update parameters over several training iterations.

[0083] In some implementations, performing backpropagation may include performing truncated backpropagation over time. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0084] In particular, model trainer 160 may train machine-learned models 120 and / or 140 based on a set of training data 162. Training data 162 may include, for example, multiple images depicting a scene from known or estimated camera poses.

[0085] In some implementations, if the user provides consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 against user-specific data received from the user computing device 102. In some implementations, this process may be referred to as personalizing the model.

[0086] Model trainer 160 includes computer logic utilized to provide desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some embodiments, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or optical or magnetic media.

[0087] Network 180 can be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications over network 180 can occur over any type of wired or wireless link, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0088] 3A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, user computing device 102 can include model trainer 160 and training dataset 162. In such implementations, model 120 can be both trained and used locally on user computing device 102. In some such implementations, user computing device 102 can implement model trainer 160 that personalizes model 120 based on user-specific data.

[0089] 3B illustrates a block diagram of an exemplary computing device 10 operating in accordance with an exemplary embodiment of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0090] Computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine-learned model(s). For example, each application may include a machine-learned model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0091] 3B , each application may communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0092] 3C depicts a block diagram of an exemplary computing device 50 operating in accordance with an exemplary embodiment of the present disclosure. Computing device 50 can be a user computing device or a server computing device.

[0093] Computing device 50 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a common API across all applications).

[0094] The central intelligence layer includes multiple machine-learned models. For example, as illustrated in FIG. 3C , each machine-learned model can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine-learned model. For example, in some embodiments, the central intelligence layer can provide a single model to all applications. In some embodiments, the central intelligence layer is included within or otherwise implemented by the operating system of computing device 50.

[0095] The central intelligence layer can communicate with a central device data layer, which can be a centralized repository of data for computing device 50. As shown in FIG. 3C , the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0096] Additional Disclosures The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken on and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality among components. For example, the processes described herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0097] While the subject matter of the present disclosure has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided for purposes of illustration and not limitation of the present disclosure. Those skilled in the art, once they arrive at the foregoing understanding, will be able to readily produce modifications, variations, and equivalents to such embodiments. Accordingly, the disclosure of the subject matter does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment may be used with other embodiments to yield still other embodiments. Accordingly, the present disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. 1. A computing device for efficient image rendering, comprising: one or more processors; one or more non-transitory computer-readable media that collectively store a mesh model of a scene and one or more texture images that include learned feature and opacity data for the mesh model; a rendering pipeline configured to generate a rendered image of the scene from a particular camera pose, the rendering pipeline comprising: a mesh rasterizer configured to generate a feature image comprising a plurality of fragments, each fragment corresponding to at least a portion of the mesh model visible from the particular camera pose, and for each fragment, the feature image comprising respective learned features and opacity data provided by the one or more texture images for the corresponding portion of the mesh model; a machine-learned neural fragment shader configured to process the particular camera pose and the respective learned feature and opacity data for each fragment of the feature image to output a respective output color for each fragment for use in the rendered image.

2. 10. The computing device of claim 1, wherein the learned feature and opacity data comprises data output by one or more machine-learned neural luminance field models trained on training images depicting the scene.

3. The computing device of claim 2 , wherein the machine-learned neural fragment shader is co-trained with the one or more machine-learned neural luminance field models.

4. the one or more machine-learned neural intensity field models; a feature field model configured to process the locations to generate feature data for the locations; an opacity field model configured to process the locations to generate opacity data for the locations; The computing device of claim 2 , comprising:

5. the feature image comprises a first resolution; 2. The computing device of claim 1, wherein the rendering pipeline is configured to downsample the feature image to a second, smaller resolution before executing the machine-learned neural fragment shader on the feature image.

6. The computing device of claim 1 , wherein the mesh model includes only polygons that are visible in one or more training images of the scene.

7. The computing device of claim 1 , wherein the computing device comprises a smartphone, a mobile device, or an embedded device.

8. the one or more processors include one or more graphics processing units; The computing device of claim 1 , wherein the mesh rasterizer and the machine-learned neural fragment shader are executed using the one or more graphics processing units.

9. The mesh model is stored as an OBJ file, the one or more texture images are stored as one or more Portable Network Graphics (PNG) files; The computing device of claim 1 , wherein the machine-learned neural fragment shader weights are stored in a JavaScript Object Notation (JSON) file.

10. The computing device of claim 1 , wherein the rendering pipeline is defined in Hypertext Markup Language (HTML).

11. The computing device of any preceding claim, wherein the neural fragment shader is defined in the OpenGL Shader Language (GLSL).

12. 1. A computer-implemented method for generating a rendered image of a scene from a particular camera pose, comprising: obtaining, by a computing system including one or more computing devices, a mesh model of the scene and one or more texture images including learned feature and opacity data for the mesh model; generating, by the computing system, a feature image comprising a plurality of fragments, each fragment corresponding to at least a portion of the mesh model visible from the particular camera pose, and for each fragment, the feature image including respective learned features and opacity data provided by the one or more texture images for the corresponding portion of the mesh model; processing, by the computing system using a machine-learned neural fragment shader, the particular camera pose and the respective learned feature and opacity data for each fragment of the feature image to output a respective output color for each fragment; providing, by the computing system, as an output, the rendered image of the scene, the rendered image of the scene having the respective output colors provided by the machine-learned neural fragment shader; A computer-implemented method comprising:

13. the learned feature and opacity data comprises data output by one or more machine-learned neural luminance field models trained on training images depicting the scene; The computer-implemented method of claim 12 , wherein the machine-learned neural fragment shader is co-trained with the one or more machine-learned neural luminance field models.

14. the feature image comprises a first resolution; 14. The computer-implemented method of claim 12 or 13, further comprising downsampling the feature image to a second, smaller resolution by the computing system before processing the specific camera pose and the respective learned feature and opacity data for each fragment of the feature image to output the respective output color for each fragment by the computing system using the machine-learned neural fragment shader.

15. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, The operations include, for each of one or more training iterations: acquiring training images depicting a scene from a camera pose; processing a plurality of locations associated with the trainable mesh model using the feature field model to generate feature data and using the opacity field model to generate opacity data; processing the feature data and the camera pose with a neural fragment shader to generate color data; alpha blending the color data and the opacity data based on the camera pose to generate one or more output colors for a rendered image; determining a loss function that compares the rendered image with the training images to determine a loss; modifying one or more parameter values ​​for one or more of the trainable mesh model, the feature field model, the opacity field model, and the neural fragment shader based on the loss; 1. One or more non-transitory computer-readable media, including:

16. In at least one of the one or more training iterations, The one or more non-transitory computer-readable media of claim 15 , further comprising binarizing the opacity data before performing the alpha compositing.

17. In at least one of the one or more training iterations, 16. The one or more non-transitory computer-readable media of claim 15, further comprising subsampling the feature data before processing the feature data and the camera pose with the neural fragment shader to generate the color data.

18. After the one or more training iterations, 16. The one or more non-transitory computer-readable media of claim 15, further comprising baking the feature data and the opacity data output by the feature field model and the opacity field model, respectively, into one or more texture images.

19. After the one or more training iterations, The one or more non-transitory computer-readable media of claim 15 , further comprising pruning portions of a mesh model that are not visible in any of a plurality of training images of the scene.

20. the trainable mesh model comprises a grid mesh; 20. The one or more non-transitory computer-readable media of any one of claims 15-19, wherein modifying one or more parameter values ​​of the trainable mesh model comprises updating vertex positions of the grid mesh while holding the topology of the grid mesh fixed.

Images