Efficient neural radiation field rendering

Abstract

A new neural radiation field (NeRF) representation based on textured polygons is provided that can efficiently synthesize new images with standard rendering pipelines. NeRF is represented as a set of polygons with textures representing binary opacity and feature vectors. Conventional rendering of polygons with a grid rasterizer (e.g., including the use of z-buffers) results in images having features at each pixel that are interpreted by a small view-related MLP running in a fragment shader to produce final pixel colors. This way enables NeRF to be rendered with conventional polygon rasterization pipelines, which provides large scale pixel-level parallelism, achieving interactive frame rates over a wide range of computing platforms, including mobile phones.

Description

[0001] Priority claim

[0002] This application is based on and claims priority from U.S. Provisional Application No. 63 / 392,060, filed on July 25, 2022, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to machine learning and more particularly to systems and methods for utilizing a polygon rasterization pipeline for efficient neural field rendering, for example on mobile architectures. Background Art

[0004] Neural radiance fields (NeRF) have become a popular representation for novel view synthesis of 3D scenes. The NeRF model represents the scene using a multi-layer perceptron (MLP) that evaluates a 5D implicit function that estimates the density and radiance emanating from any location in any direction, which can be used in a volume rendering framework to produce novel images. See Mildenhall et al., Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. The NeRF representation, optimized to minimize the multi-view color consistency loss for a collection of posed photos, has shown remarkable ability to reproduce fine image details for novel views.

[0005] However, one of the main obstacles to the widespread adoption of NeRF is that it requires specialized rendering algorithms that are not well matched to commonly available hardware. Specifically, traditional NeRF implementations use volume rendering algorithms that evaluate large MLPs for each pixel at hundreds of sample locations along a ray to estimate and integrate density and radiation. This rendering process is too slow for interactive visualization. Summary of the invention

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] An example aspect relates to a computing device for performing efficient image rendering. The computing device includes one or more processors. The computing device includes one or more non-transitory computer-readable media that collectively store a mesh model of a scene and one or more texture images including learned features and opacity data of the mesh model. The computing device includes a rendering pipeline that is configured to generate a rendered image of the scene from a specified camera pose. The rendering pipeline includes a mesh rasterizer that is configured to generate a feature image including a plurality of fragments, wherein each fragment corresponds to at least a portion of the mesh model visible from the specified camera pose, and wherein for each fragment, the feature image includes corresponding learned features and opacity data provided by one or more texture images for the corresponding portion of the mesh model. The rendering pipeline includes a machine learning neural fragment shader that is configured to process the specified camera pose and the corresponding learned features and opacity data of each fragment of the feature image to output a corresponding output color for each fragment in the rendered image.

[0008] Another example aspect relates to a computer-implemented method for generating a rendered image of a scene from a specified camera pose. The method includes obtaining, by a computing system including one or more computing devices, a mesh model of the scene and one or more texture images including learned features and opacity data for the mesh model. The method includes generating, by the computing system, a feature image including a plurality of fragments, wherein each fragment corresponds to at least a portion of the mesh model visible from the specified camera pose, and wherein for each fragment, the feature image includes corresponding learned features and opacity data provided by the one or more texture images for a corresponding portion of the mesh model. The method includes processing, by the computing system, the specified camera pose and the corresponding learned features and opacity data for each fragment of the feature image using a machine learning neural fragment shader to output a corresponding output color for each fragment. The method includes providing, by the computing system, as an output, a rendered image of the scene, the rendered image of the scene having corresponding output colors provided by the machine learning neural fragment shader.

[0009] Another example aspect relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause the one or more processors to perform operations with respect to each of one or more training iterations. The operations include obtaining a training image depicting a scene from a camera pose. The operations include processing a plurality of positions associated with a learnable mesh model with a feature field model to generate feature data, and processing the plurality of positions with an opacity field model to generate opacity data. The operations include processing the feature data and the camera pose with a neural fragment shader to generate color data. The operations include performing alpha-compositing on the color data and the opacity data based on the camera pose to generate one or more output colors for a rendered image. The operations include evaluating a loss function that compares the rendered image to the training image to determine a loss. The operations include modifying one or more parameter values ​​of one or more of the following based on the loss: a learnable mesh model, a feature field model, an opacity field model, and a neural fragment shader.

[0010] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0013] Figure 1 Depicted is a block diagram of an example rendering pipeline for efficient neural radiance field rendering according to an example embodiment of the present disclosure.

[0014] Figure 2 Depicted is a block diagram of an example training approach for efficient neural radiance field rendering according to an example embodiment of the present disclosure.

[0015] Figure 3A A block diagram of an example computing system is depicted according to an example embodiment of the present disclosure.

[0016] Figure 3B A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0017] Figure 3C A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0018] Repeated reference numerals among the various drawings are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0019] Overview

[0020] In general, the present disclosure relates to a new neural radiance field (NeRF) representation based on textured polygons that can efficiently synthesize new types of images using a standard rendering pipeline. Specifically, NeRF can be represented as a collection of polygons with a texture that represents a binary opacity and a feature vector. Traditional rendering of polygons with a mesh rasterizer (e.g., including the use of a z-buffer) results in a feature image with features at each fragment (e.g., pixel). The features of the feature image can be interpreted by a neural fragment shader (e.g., a small view-dependent MLP running in the fragment shader) to produce the final pixel color. This approach enables NeRF to be rendered with a traditional polygon rasterization pipeline, which provides massive pixel-level parallelism, thereby achieving interactive frame rates on a wide range of computing platforms including mobile phones or other mobile or embedded architectures.

[0021] More specifically, NeRF has demonstrated the amazing ability to synthesize 3D scene images from novel views. However, they rely on specialized volume rendering algorithms based on ray marching, which do not match the capabilities of widely deployed graphics hardware. Therefore, typical NeRF cannot be realistically rendered on ubiquitous mobile architectures such as smartphones.

[0022] While pre-calculating and baking NeRF into a sparse 3D voxel grid can improve NeRF rendering speed, these approaches still rely on ray marching through the sparse voxel grid to generate per-pixel features, and therefore do not fully exploit the parallelism available in commodity graphics processing units (GPUs). In addition, these approaches require a large amount of GPU memory to store volume textures, which also inhibits the operation of these approaches on common mobile devices.

[0023] In view of these problems, the present disclosure introduces MobileNeRF, a NeRF that can run in real time on a variety of common mobile devices. NeRF can be represented by a mesh model and one or more texture images. For example, a mesh model can include a set of polygons with associated textures, where the polygons roughly follow the surface of the scene, and the texture image stores the opacity and feature vectors learned for the polygons during training.

[0024] To render an image, an example implementation may utilize a polygon rasterization pipeline framework. For example, a rendering pipeline may include a mesh rasterizer and a neural fragment shader. The mesh rasterizer may perform z-buffering to generate a feature vector for each fragment (e.g., pixel) of an image to be rendered. The feature vector for each fragment may then be processed by a neural fragment shader (e.g., a lightweight MLP running in a GLSL fragment shader) to generate an output color.

[0025] Some example implementations of this rendering pipeline do not sample rays or sort polygons in depth order, and may therefore model only binary opacity. However, the rendering pipeline can fully exploit the parallelism provided by the z-buffer and fragment shaders in modern graphics hardware, and may therefore be 10x faster than the current state-of-the-art at the same output quality for a standard test scene.

[0026] Furthermore, the MobileNeRF approach described herein requires only a standard polygon rendering pipeline framework that can be implemented and accelerated on nearly every computing platform, and thus it runs on mobile phones and other devices that were previously unable to support NeRF visualizations at interactive rates.

[0027] Therefore, the present disclosure provides a variety of technical effects and benefits. As an example, the proposed model and associated rendering method are 10 times faster than the current state-of-the-art model and associated rendering method at the same output quality.

[0028] As another example technical effect and benefit, the proposed technique consumes less memory by storing surface textures instead of volume textures, thereby enabling the proposed method to run on an integrated GPU with significantly smaller memory and power;

[0029] As another example technical effect and benefit, the proposed technology can be implemented to run on a web browser and is compatible with a large number and variety of devices. For example, a viewer can be written in HTML with three.js, which can be executed by a large number of devices around the world.

[0030] As yet another example technical effect and benefit, the proposed technique allows real-time manipulation of reconstructed objects / scenes since they are simple triangle meshes.

[0031] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

[0032] Example Rendering Pipeline

[0033] Figure 1A block diagram of an example rendering pipeline for efficient neural radiance field rendering according to an example embodiment of the present disclosure is depicted. Specifically, Figure 1 A rendering pipeline 18 is shown that is configured to generate a rendered image 26 of a scene when provided with the following rendering assets / information: a mesh model 12 of the scene, one or more texture images 14 including learned features and opacity data of the mesh model 12, and a camera pose 16 from which the rendered image 26 is to be viewed / rendered.

[0034] The rendering pipeline 18 includes a mesh rasterizer 20 and a neural fragment shader 24. The mesh rasterizer 20 is configured to generate a feature image 22 including a plurality of fragments (e.g., pixels). Each fragment in the feature image 22 may correspond to at least a portion (e.g., a polygon) of the mesh model 12 that is visible from the specified camera pose 16. In addition, for each fragment, the feature image 22 includes corresponding learned features and opacity data for the corresponding portion (e.g., polygon) of the mesh model provided by one or more texture images.

[0035] The neural fragment shader 24 is configured to process the specified camera pose 16 and the corresponding learned features and opacity data for each fragment of the feature image 22 to output a corresponding output color for each fragment in the rendered image 26. That is, for each fragment (e.g., pixel) of the feature image 22, the neural fragment shader 24 can provide an output color, and the rendered image 26 can be generated using the output color.

[0036] In some implementations, the learned features and opacity data stored in the texture image 14 include data output by one or more machine learning neural radiance field models trained on training images depicting a scene. In some implementations, the machine learning neural fragment shader 24 is jointly trained with the one or more machine learning neural radiance field models. In some implementations, the one or more machine learning neural radiance field models include: a feature field model that is configured to process a location to generate feature data for the location; and an opacity field model that is configured to process a location to generate opacity data for the location. In some implementations, the mesh model 12 includes only polygons that are visible from one or more training images of the scene. Limiting the mesh model 12 to polygons that are visible from the one or more training images can improve the quality of the mesh model 12 (e.g., by reducing errors or hallucinated scene structures that fail to match the semantics of the scene).

[0037] Still refer to Figure 1In some implementations, the feature image 22 is generated at a first resolution, and the rendering pipeline 18 can be configured to downsample the feature image 22 to a smaller second resolution before executing the machine learning neural fragment shader 24 on the feature image 22. For example, the second resolution can be half of the first resolution. Supersampling the feature image 22 first and then subsampling before applying the shader 24 can result in averaging the feature and opacity information contained in the supersampled feature image 22. The averaging of the feature and opacity information can provide benefits such as improved anti-aliasing of the rendered image 26.

[0038] According to one aspect of the present disclosure, the rendering pipeline 18 can be highly similar to the existing mesh model rendering pipeline in terms of framework and execution. Therefore, the rendering pipeline 18 can be executed on a large number of common computer devices such as smartphones, mobile devices (e.g., tablets, smart watches, home assistant devices, etc.) or embedded devices (e.g., vehicle computing devices, home appliances, Internet of Things devices, etc.). For example, in some implementations, one or more graphics processing units and / or other acceleration hardware optimized for image rendering can be used to execute a mesh rasterizer and a machine learning neural fragment shader.

[0039] In some implementations, to achieve compatibility with these existing rendering frameworks, the rendering pipeline 18 and rendering assets such as the mesh model 12 and the texture image 14 can be defined in various common file types, stored as various common file types, or otherwise executed using various common file types. As an example, in some implementations, the mesh model 12 can be stored as an OBJ file; one or more texture images 14 can be stored as one or more portable network graphics (PNG) files; and / or the weights of the machine learning neural fragment shader 24 can be stored in a JavaScript Object Notation (JSON) file. As another example, in some implementations, the rendering pipeline 18 can be defined in Hypertext Markup Language (HTML), stored as HTML, or otherwise executed using HTML. As another example, in some implementations, the neural fragment shader 24 can be defined in OpenGL Shading Language (GLSL), stored as GLSL, or otherwise executed using GLSL.

[0040] Example training method

[0041] Figure 2 Depicted is a block diagram of an example training approach for efficient neural radiance field rendering according to an example embodiment of the present disclosure. Figure 2The process shown can be performed for each of a plurality of training images depicting a scene. In some implementations, the training can be performed in multiple stages, each of which contains Figure 2 Different arrangements of the illustrated operations (and / or other operations).

[0042] Specifically, the computing system may obtain a training image 206 depicting a scene from a camera pose 202. The computing system may process a plurality of locations associated with the learnable mesh model 204 with a feature field model 208 to generate feature data 212, and may process the plurality of locations with an opacity field model 210 to generate opacity data 214. As an example, the feature field model 208 and the opacity field model 210 may be neural radiance field models (e.g., implemented using a relatively large MLP).

[0043] The computing system can process the feature data 212 and the camera pose 202 with a neural fragment shader (e.g., implemented using a relatively small MLP) to generate color data 220. The computing system can perform alpha blending 222 on the color data 220 and the opacity data 214 based on the camera pose 202 to generate one or more output colors for a rendered image 224.

[0044] The computing system may evaluate a loss function 226 that compares the rendered image 224 to the training image 206 to determine a loss. For example, the loss function 226 may be a mean squared error over pixel colors.

[0045] The computing system may modify one or more parameter values ​​of one or more of the following based on the loss: the learnable mesh model 204, the feature field model 208, the opacity field model 210, and the neural fragment shader 216. For example, the loss and the loss function 226 may be as follows: Figure 2 The dashed line shows back propagation 228 through these models.

[0046] In some implementations, for at least one of the one or more training iterations, the computing system may binarize 218 the opacity data before performing alpha fusion 222. For example, in some implementations, training may be performed without binarization 218 in the first stage (i.e., continuous opacity), while training may be performed with binarization 218 in the second stage. In some implementations, the second stage may include training with both binarization 218 and also continuous opacity. In some implementations, the loss may be back-propagated through the opacity binarization 218 using straight-through estimation.

[0047] In some implementations, the feature data 212 and the opacity data 214 can be alpha-blended 222 prior to applying the neural fragment shader 216 (not shown). The neural fragment shader 216 can then be applied to the output of the alpha-blending 222 implemented on the feature data 212.

[0048] In some implementations, for at least one of the one or more training iterations, the computing system may subsample the feature data 212 before processing the feature data 212 and the camera pose 202 with the neural fragment shader 216 to generate the color data 220 .

[0049] In some implementations, after a training iteration, the computing system may bake the feature data 212 and opacity data 214 output by the feature field model 208 and the opacity field model 210, respectively, into one or more texture images (not shown). For example, for all locations (e.g., polygons) of the mesh model 204, the feature field model 208 and the opacity field model 210 may be evaluated, and their respective outputs may be stored in one or more texture images for use during later rendering.

[0050] As another example, in some implementations, after one or more training iterations, any portions (e.g., polygons) of the mesh model that are not visible in any of the plurality of training images of the scene can be pruned. This can improve the quality of the mesh model 204 and reduce the memory requirements of the mesh model. In some implementations, the learnable mesh model 204 can be or include a grid mesh. Modifying one or more parameter values ​​of the learnable mesh model 204 can include updating vertex positions of the grid mesh while keeping the topology of the grid mesh fixed.

[0051] Example Implementation

[0052] This section describes an example implementation of the general framework described herein.

[0053] Given a series of (calibrated) images, some example implementations work on optimizing the representation to achieve efficient novel view synthesis. One example representation proposed includes a polygon mesh whose texture map stores features and opacity. At rendering time, given the camera pose, some example implementations adopt a two-stage deferred rendering process:

[0054] Rendering stage 1 - some example implementations rasterize the mesh into screen space and build a feature image, e.g., some example implementations create a deferred render buffer in GPU memory;

[0055] Rendering stage 2 - some example implementations convert these features into a color image via a (neural) deferred renderer, such as a small MLP, running in the fragment shader, which receives a feature vector and a view direction and outputs a pixel color.

[0056] The proposed representation of an example is built in three training stages that gradually move from a classical NeRF-like continuous representation to a discrete representation:

[0057] Training Phase 1 (Phase 1) - some example implementations train a NeRF-like model with continuous opacity, where volume rendering quadrature points are derived from a polygonal mesh;

[0058] Training Phase 2 (Phase 2) - Some example implementations binarize opacity because while classical rasterization can easily discard fragments, they cannot handle semi-transparent fragments properly.

[0059] Training Phase 3 (Phase 3) - Some example implementations extract a sparse polygon mesh, bake opacity and features into a texture map, and store the weights of the neural deferred shader.

[0060] As an example, meshes may be stored as OBJ files, texture maps as PNG, and deferred shader weights in (small) JSON files.Since some example implementations employ a standard GPU rasterization pipeline, one example real-time renderer proposed is simply an HTML web page.

[0061] Since representing a continuous signal with a discrete representation may introduce aliasing, some example implementations also include a simple yet computationally efficient anti-aliasing solution based on supersampling (anti-aliasing).

[0062] Example Techniques for Continuous Training (Training Phase 1)

[0063] An example training setup proposed consists of a polygonal mesh and three learnable models such as MLP. is fixed, but the vertex position And MLP is optimized similarly to NeRF in an automatic decoding manner by minimizing the mean squared error between the predicted color and the true value color of the pixels in the training image:

[0064]

[0065] The predicted color C(.) is obtained by taking the product of the (depth-sorted) points The radiation c along the ray r(t)=o+td kPerform alpha blending to obtain:

[0066]

[0067] The opacity α k and view-dependent radiation c k By at position p k = r(t k ) to give:

[0068]

[0069] Small Network is an example proposed deferred neural shader that outputs the color of each fragment taking into account fragment characteristics and viewing direction. Finally, it should be noted that (2) is not fused with volume density, but with opacity [see, for example, Equation 8].

[0070] Example polygon mesh

[0071] Without loss of generality, some example implementations operate with respect to the polygonal mesh used in synthesizing a 360° scene, and provide configurations for front-facing scenes and unbounded 360° scenes. Some example implementations first define a regular grid of size P×P×P in a unit cube centered at the origin. Some example implementations instantiate by creating one vertex per voxel And instantiate it by creating a quad (two triangles) per grid edge The grid edge connects the vertices of four adjacent voxels. Some example implementations parameterize the vertex positions locally relative to the voxel center (and size), resulting in Free variables. During optimization, some example implementations initialize the vertex positions to This corresponds to a regular Euclidean lattice, and some example implementations regularize these vertex positions to prevent vertices from leaving their voxels and force vertices to return to their neutral positions whenever the optimization problem is underconstrained:

[0072]

[0073] Where whenever v is outside its corresponding voxel, the indicator function

[0074] Example quadrature

[0075] Since evaluating some example representations for MLP is computationally expensive, some example implementations rely on an acceleration grid to limit the cardinality of the product points. First, quadrature points are generated only for the set of voxels that intersect the ray; then, some example implementations use an acceleration grid to prune voxels that are unlikely to contain geometry. Finally, some example implementations compute the ray and the incident voxel vertex The intersection points between the faces of to obtain the final set of quadrature points. Some example implementations use barycentric interpolation to backpropagate the gradient from the intersection point to the three vertices in the intersecting triangle. In summary, in some implementations, for each input ray r:

[0076]

[0077] where (a∩b) = true if a intersects b, and the acceleration grid is supervised to fused alpha visibility T across viewpoints during training. k α k Set an upper limit.

[0078]

[0079] in / is a stop gradient operator that prevents the acceleration grid from (adversely) affecting the image reconstruction quality.

[0080] This can be interpreted as a way of computing the so-called "surface field" during NeRF training rather than after training. Some example implementations additionally regularize the content of the grid by promoting its point-wise sparsity (e.g., lasso) as well as its spatial smoothness:

[0081]

[0082] Example Binarization Training (Training Phase 2)

[0083] The rendering pipeline implemented in typical hardware does not natively support semi-transparent meshes. Rendering semi-transparent meshes requires tedious (per-frame) sorting so that rendering is performed in back-to-front order to ensure correct alpha blending. Some example implementations do this by adding a smooth opacity alpha k ∈[0, 1] Convert from opacity (3) to discrete / categorical opacity To overcome this problem, some example implementations use a straight-through estimator to optimize discrete opacity via photometric supervision:

[0084]

[0085] It should be noted that regardless of α k The value and the obtained In any case, the gradient will be discretized (i.e. ) are passed transparently. To stabilize training, some example implementations then train the continuous model and the discrete model together:

[0086]

[0087] in is the discrete opacity model The corresponding output radiation:

[0088]

[0089] Once MSE_plus_binary(14) has converged, some example implementations minimize Come to and The weights in are applied with a fine-tuning step, while the weights of the others are fixed.

[0090] Example discretization (training phase 3)

[0091] After binarization and fine-tuning, some example implementations convert the representation into an explicit polygon mesh (e.g., in OBJ format). Some example implementations only store quads if they are at least partially visible in the training camera pose (i.e., invisible quads are discarded). Some example implementations then create a texture image whose size is proportional to the number of visible quads, and for each quad, some example implementations allocate a K×K patch in the texture. Some example implementations use K=17, so that the quads have a 16×16 texture with half-pixel border padding. Some example implementations then iterate over the pixels of the texture, convert the pixel coordinates to 3D coordinates, and bake the values ​​of discrete opacity (e.g., opacity(3) and dOpacity(12)) and features (e.g., features(4)) into the texture map. Some example implementations quantize the [0, 1] range to 8-bit integers and store the texture into a (e.g., losslessly compressed) PNG image. Example experiments show that quantizing the [0, 1] range with 8-bit precision (which is not taken into account during backpropagation) does not significantly affect the rendering quality.

[0092] Example Anti-Aliasing

[0093] In the classic rasterization pipeline, aliasing is an issue that must be considered in order to achieve high quality rendering. As discussed earlier, translucency would require polygon sorting per frame, and classic NeRF would create the illusion of smooth edges via translucent volumes. Some example implementations overcome this problem by using anti-aliasing via supersampling. While one could simply perform (5) four times per pixel and average the resulting colors, delaying the neural shader would require more work. The execution of is the computational bottleneck of one example technique proposed. Some example implementations may overcome this problem by simply averaging the features, that is, averaging the inputs to the deferred neural shader instead of averaging its outputs. Some example implementations first rasterize the features (at 2× resolution):

[0094]

[0095] The sub-pixel features are then averaged to produce an anti-aliased representation, which some example implementations feed into one of the example neural deferred shaders proposed:

[0096]

[0097] in Calculate the average value between sub-pixels (eg, four in the example implementation), and d δ is the ray r δ direction.

[0098] However, it should be noted that with this change, some example implementations only query Finally, for the discrete occupancy This process can be applied similarly to (15). These changes to anti-aliasing can be applied to MSE_plus_binary (14) in training stage 2.

[0099] Example Rendering

[0100] In some implementations, the result of the optimization process is a textured polygon mesh (where the texture map stores features instead of colors) and a small MLP (which converts view direction and features to colors). Rendering this representation is done in two steps using the deferred rendering pipeline:

[0101] 1. Some example implementations rasterize all faces of a textured mesh with a z-buffer to produce a 2M×2N feature image with 12 channels per pixel (including 8 learned feature channels), binary opacity, and 3D view direction;

[0102] 2. Some example implementations synthesize an M×N output RGB image by rendering a textured rectangle that uses the feature image as its texture while linear filtering to average the features for anti-aliasing. Some example implementations apply a small MLP to pixels with non-zero alpha to convert the features into RGB colors. The small MLP can be implemented as a GLSL fragment shader.

[0103] These rendering steps can be implemented in a classic rasterization pipeline. Since the z-buffer with binary transparency is order-independent, polygons do not need to be sorted into depth order for each new view, and therefore the polygons can be loaded into a buffer in the GPU at once at the beginning of execution. Since the MLP for converting features to colors is very small, it can be implemented in a GLSL fragment shader that runs in parallel for all pixels. These classic rendering steps are highly optimized on the GPU, and therefore a proposed example rendering system can run at interactive frame rates on a wide variety of devices. The rendering system is also easy to implement because it only requires standard polygon rendering with a fragment shader. An example interactive viewer is an HTML web page with Javascript that is rendered with WebGL via the threejs library.

[0104] Example Apparatus and Systems

[0105] Figure 3A A block diagram of an example computing system 100 is depicted, in accordance with an example embodiment of the present disclosure. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0106] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smart phone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0107] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 executed by the processor 112 to cause the user computing device 102 to perform operations.

[0108] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine-learned models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model).

[0109] In some implementations, one or more machine learning models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel rendering across multiple instances of pixels).

[0110] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 130 as part of a web service (e.g., an image rendering service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0111] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0112] The user computing device 102 may also include a rendering pipeline 124. For example, the rendering pipeline 124 may be a reference Figure 1 The rendering pipeline 124 includes computer logic for providing the desired functionality. The rendering pipeline 124 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, the rendering pipeline 124 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the rendering pipeline 124 includes one or more sets of computer executable instructions stored in a tangible computer-readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0113] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 executed by the processor 132 to cause the server computing system 130 to perform operations.

[0114] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0115] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include a variety of machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model).

[0116] The server computing system 130 may also include a rendering pipeline 142. For example, the rendering pipeline 142 may be a reference Figure 1 The rendering pipeline 142 includes computer logic for providing the desired functionality. The rendering pipeline 142 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, the rendering pipeline 142 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the rendering pipeline 142 includes one or more sets of computer executable instructions stored in a tangible computer-readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0117] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0118] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0119] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques (e.g., such as error back propagation). For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters in multiple training iterations.

[0120] In some implementations, performing error backpropagation may include performing truncated backpropagation through time. The model trainer 160 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0121] Specifically, model trainer 160 can train machine learning models 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, a plurality of images depicting a scene from known or estimated camera poses.

[0122] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0123] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0124] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0125] Figure 3A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 may be both trained and used locally at the user computing device 102. In some such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0126] Figure 3B Depicted is a block diagram of an example computing device 10 performing in accordance with an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0127] Computing device 10 includes multiple applications (e.g., application 1 to application N). Each application contains its own machine learning library and machine learning model. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0128] like Figure 3B As shown, each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0129] Figure 3C Depicted is a block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0130] The computing device 50 includes a plurality of applications (e.g., Application 1 to Application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0131] The central intelligence layer includes multiple machine learning models. For example, Figure 3C As shown, a corresponding machine learning model can be provided for each application, and the corresponding machine learning model can be managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system.

[0132] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for the computing device 50. Figure 3C As shown, the central device data layer can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0133] Additional disclosure

[0134] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0135] Although the present subject matter has been described in detail with respect to various specific example embodiments of the present subject matter, each example is provided by way of explanation rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, modifications, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, variations, and / or additions to the present subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of an embodiment may be used together with another embodiment to obtain yet further embodiments. Therefore, the present disclosure is intended to encompass such changes, variations, and equivalents.

Claims

1. A computing device for performing efficient image rendering, the computing device comprising: one or more processors; one or more non-transitory computer-readable media collectively storing a mesh model of a scene and one or more texture images including learned features and opacity data for the mesh model; as well as A rendering pipeline, the rendering pipeline being configured to generate a rendered image of the scene from a specified camera pose, the rendering pipeline comprising: a mesh rasterizer configured to generate a feature image comprising a plurality of fragments, wherein each fragment corresponds to at least a portion of the mesh model visible from the specified camera pose, and wherein for each fragment, the feature image comprises a respective learned feature and opacity data provided by the one or more texture images for a corresponding portion of the mesh model; A machine learning neural fragment shader is configured to process the specified camera pose and the corresponding learned features and opacity data for each fragment of the feature image to output a corresponding output color for each fragment in the rendered image.

2. A computing device as described in any preceding claim, wherein the learned features and the opacity data include data output by one or more machine learning neural radiance field models trained on training images depicting the scene.

3. A computing device as described in claim 2, wherein the machine learning neural fragment shader is jointly trained with the one or more machine learning neural radiation field models.

4. The computing device of claim 2 or 3, wherein the one or more machine learning neural radiation field models include: a feature field model configured to process a location to generate feature data for the location; as well as An opacity field model is configured to process the location to generate opacity data for the location.

5. A computing device as described in any preceding claim, wherein the feature image comprises a first resolution, and wherein the rendering pipeline is configured to downsample the feature image to a smaller second resolution before executing the machine learning neural fragment shader on the feature image.

6. Computer apparatus as claimed in any preceding claim, wherein the mesh model comprises only polygons visible from one or more training images of the scene.

7. A computing device as claimed in any preceding claim, wherein the computing device comprises a smartphone, a mobile device or an embedded device.

8. A computing device as described in any preceding claim, wherein the one or more processors include one or more graphics processing units, and wherein the mesh rasterizer and the machine learning neural fragment shader are executed using the one or more graphics processing units.

9. A computing device as claimed in any preceding claim, wherein: The mesh model is stored as an OBJ file; The one or more texture images are stored as one or more portable network graphics (PNG) files; and The weights of the machine learning neural fragment shader are stored in a JavaScript Object Notation (JSON) file.

10. A computing device as claimed in any preceding claim, wherein the rendering pipeline is defined in Hypertext Markup Language (HTML).

11. A computing device as described in any preceding claim, wherein the neural fragment shader is defined in OpenGL Shading Language (GLSL).

12. A computer-implemented method for generating a rendered image of a scene from a specified camera pose, the method comprising: Obtaining, by a computing system including one or more computing devices, a mesh model of the scene and one or more texture images including learned features of the mesh model and opacity data; generating, by the computing system, a feature image comprising a plurality of segments, wherein each segment corresponds to at least a portion of the mesh model visible from the specified camera pose, and wherein, for each segment, the feature image comprises a respective learned feature and opacity data for a corresponding portion of the mesh model provided by the one or more texture images; processing, by the computing system, the specified camera pose and the corresponding learned features and opacity data for each fragment of the feature image using a machine learning neural fragment shader to output a corresponding output color for each fragment; and The rendered image of the scene is provided as output by the computing system, the rendered image of the scene having the corresponding output color provided by the machine learning neural fragment shader.

13. The computer-implemented method of claim 12, wherein: The learned features and the opacity data comprise data output by one or more machine-learned neural radiance field models trained on training images depicting the scene; and The machine learning neural fragment shader is jointly trained with the one or more machine learning neural radiance field models.

14. The computer-implemented method of claim 12 or 13, wherein: The feature image comprises a first resolution; and The method further includes downsampling, by the computing system, the feature image to a smaller second resolution before processing, by the computing system, the specified camera pose and the corresponding learned features and opacity data for each fragment of the feature image using the machine learning neural fragment shader to output the corresponding output color for each fragment.

15. One or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising, for each of one or more training iterations: Obtain training images depicting the scene from camera poses; processing a plurality of locations associated with the learnable grid model with a feature field model to generate feature data, and processing the plurality of locations with an opacity field model to generate opacity data; Processing the feature data and the camera pose with a neural fragment shader to generate color data; performing alpha blending on the color data and the opacity data based on the camera pose to generate one or more output colors for a rendered image; evaluating a loss function that compares the rendered image to the training image to determine a loss; as well as One or more parameter values ​​of one or more of: the learnable mesh model, the feature field model, the opacity field model, and the neural fragment shader are modified based on the loss.

16. The one or more non-transitory computer-readable media of claim 15, further comprising, for at least one training iteration of the one or more training iterations: The opacity data is binarized before performing the alpha blending.

17. The one or more non-transitory computer-readable media of claim 15 or 16, further comprising, for at least one training iteration of the one or more training iterations: The feature data is sub-sampled before processing the feature data and the camera pose with the neural fragment shader to generate the color data.

18. The one or more non-transitory computer-readable media of claim 15, 16, or 17, further comprising, after the one or more training iterations: The feature data and the opacity data respectively output by the feature field model and the opacity field model are baked into one or more texture images.

19. The one or more non-transitory computer-readable media of any one of claims 15 to 18, further comprising, after the one or more training iterations: Portions of the mesh model that are not visible in any training image of a plurality of training images of the scene are pruned.

20. One or more non-transitory computer-readable media as described in any one of claims 15 to 19, wherein the learnable mesh model includes a grid mesh, and wherein modifying one or more parameter values ​​of the learnable mesh model includes updating vertex positions of the grid mesh while keeping the topology of the grid mesh fixed.

Images