Ray tracing for rendering meshes produced by machine learning techniques

Abstract

Various aspects of this disclosure relate to three-dimensional (3D) computer graphics processing. According to one aspect, the disclosure includes: a memory configured to store learned triangle meshes and learned feature textures; a graphics processing unit (GPU) coupled to the memory, the GPU configured to render an inferred three-dimensional (3D) scene using ray tracing based on the learned triangle meshes and learned feature textures; and a display unit coupled to the GPU, the display unit configured to display the inferred 3D scene.

Description

[0001] Cross-references to related applications

[0002] This patent application claims priority to pending U.S. nonprovisional application No. 18 / 420,268, filed January 23, 2024, which has been assigned to the assignee of this application and is expressly incorporated herein by reference, as fully set forth below and for all applicable purposes. Technical Field

[0003] This disclosure relates generally to the field of information processing, and more particularly to three-dimensional (3D) computer graphics processing using machine learning (ML). Background Technology

[0004] In three-dimensional (3D) computer graphics, a requirement is to synthesize 3D scenes from multiple two-dimensional (2D) images. Various graphics processing techniques have emerged for rendering graphical meshes as part of 3D scene synthesis. Many of these techniques are computationally demanding, leading to significant interest in more efficient 3D computer graphics processing using machine learning (ML). Summary of the Invention

[0005] To provide a basic understanding of one or more aspects of this disclosure, an overview of those aspects is given below. This overview is not a comprehensive summary of all intended features of this disclosure, nor is it intended to identify key or essential elements of all aspects of this disclosure, nor to depict the scope of any or all aspects of this disclosure. Its sole purpose is to present some concepts of one or more aspects of this disclosure in a simplified form as a prelude to the more detailed description that follows.

[0006] In one aspect, this disclosure provides three-dimensional (3D) computer graphics processing. Accordingly, this disclosure provides an apparatus comprising: a memory configured to store learned triangle meshes and learned feature textures; a graphics processing unit (GPU) coupled to the memory, the GPU configured to render an inferred three-dimensional (3D) scene based on the learned triangle meshes and learned feature textures using ray tracing; and a display unit coupled to the GPU, the display unit configured to display the inferred 3D scene.

[0007] In one example, the GPU includes a shader processor configured to process learned triangle meshes and learned feature textures. In another example, the GPU also includes a ray traversal unit configured to perform ray tracing. In one example, ray tracing includes the determination of primary visibility. In yet another example, ray tracing includes a bounding volume hierarchy (BVH) technique.

[0008] In one example, the shader processor is further configured to infer the inferred 3D scene to output view-dependent colors. In one example, the shader processor is further configured to utilize ray tracing and a simplified neural network to synthesize the inferred 3D scene. In one example, the shader processor is further configured to backpropagate multiple 2D images to a simplified neural network, an initial feature field neural network, and an initial opacity field neural network to generate an updated learned triangle mesh, an updated learned feature texture, and an updated simplified neural network. In one example, the shader processor is further configured to use forward propagation and ray tracing to infer the updated simplified neural network. In one example, the shader processor is further configured to utilize the updated learned triangle mesh and the updated learned feature texture, and ray tracing, to synthesize the inferred 3D scene using an updated simplified neural network.

[0009] Another aspect of this disclosure provides a method comprising: using an initial mesh and an initial feature texture generated from multiple two-dimensional (2D) images by an initial feature field neural network and an initial opacity field neural network; and using ray tracing to synthesize an initial three-dimensional (3D) scene using an initial simplified neural network, the initial mesh, and the initial feature texture.

[0010] In one example, the initial simplified neural network is a multilayer perceptron (MLP) neural network. In one example, the initial mesh is a collection of three-dimensional (3D) spatial samples representing geometric objects. In one example, ray tracing includes the determination of primary visibility. In one example, ray tracing includes the bounding volume hierarchy (BVH) technique.

[0011] In one example, the method also includes using forward propagation to synthesize an initial three-dimensional (3D) scene. In another example, the method also includes backpropagating the initial 3D scene to an initial simplified neural network, an initial feature field neural network, and an initial opacity field neural network to create a trained simplified neural network using forward propagation and ray tracing.

[0012] In one example, the method further includes using an updated learned mesh and updated learned feature textures, and leveraging the ray tracing, to synthesize an inferred three-dimensional (3D) scene using an updated simplified neural network. In one example, ray tracing includes the determination of primary visibility. In one example, ray tracing includes a bounding volume hierarchy (BVH) technique.

[0013] In one example, the method also includes outputting one or more view-dependent 3D scenes from an updated mesh and updated feature textures. In one example, the initial simplified neural network has a lower dimensionality than the initial feature field neural network and the initial opacity field neural network.

[0014] In one example, the method also includes establishing an initial feature field neural network and an initial opacity field neural network. In another example, the method also includes ingesting multiple two-dimensional (2D) images for machine learning (ML) training.

[0015] Another aspect of this disclosure provides an apparatus comprising: a component for generating an initial mesh and an initial feature texture using multiple two-dimensional (2D) images by means of an initial feature field neural network and an initial opacity field neural network; and a component for synthesizing an initial three-dimensional (3D) scene using an initial simplified neural network, the initial mesh, and the initial feature texture by means of ray tracing.

[0016] In one example, the apparatus further includes: components for backpropagating an initial 3D scene to an initial simplified neural network, an initial feature field neural network, and an initial opacity field neural network to create a trained simplified neural network using forward propagation and ray tracing; and components for synthesizing an inferred 3D scene using an updated simplified neural network with an updated learned mesh and updated learned feature textures and ray tracing.

[0017] In one example, the device also includes: components for establishing an initial feature field neural network and an initial opacity field neural network; and components for ingesting multiple two-dimensional (2D) images for machine learning (ML) training.

[0018] Another aspect of this disclosure provides a non-transitory computer-readable medium storing computer-executable code operable on a device including at least one processor and at least one memory coupled to the at least one processor, wherein the at least one processor is configured to implement three-dimensional (3D) scene synthesis using ray tracing, the computer-executable code including: instructions for causing a computer to use an initial mesh and an initial feature texture generated by an initial feature field neural network and an initial opacity field neural network using a plurality of two-dimensional (2D) images; and instructions for causing a computer to use ray tracing to synthesize an initial three-dimensional (3D) scene using an initial simplified neural network, an initial mesh, and an initial feature texture.

[0019] In one example, the non-transitory computer-readable medium also includes: instructions for instructing a computer to backpropagate an initial 3D scene to an initial simplified neural network, an initial feature field neural network, and an initial opacity field neural network to create a trained simplified neural network using forward propagation and ray tracing; and instructions for instructing a computer to synthesize an inferred three-dimensional (3D) scene using an updated learned mesh and an updated learned feature texture and ray tracing, using an updated simplified neural network.

[0020] These and other aspects of this disclosure will be more fully understood upon reading the following detailed description. Other aspects, features, and embodiments of this disclosure will become apparent to those skilled in the art after reading the following description of specific exemplary embodiments of the invention in conjunction with the accompanying drawings. Although features of the invention may be discussed below with respect to certain embodiments and drawings, all embodiments of the invention may include one or more of the advantageous features discussed herein. In other words, while one or more embodiments may be discussed as having certain advantageous features, one or more such features may also be used according to the various embodiments of the invention discussed herein. Similarly, although exemplary embodiments may be discussed below as embodiments of an apparatus, system, or method, it should be understood that such exemplary embodiments may be implemented in various apparatuses, systems, and methods. Attached Figure Description

[0021] Figure 1 An example information processing system is shown.

[0022] Figure 2 An example overview of NeRF technology is provided.

[0023] Figure 3 An example of bulk density is shown.

[0024] Figure 4 An example of volume rendering is shown.

[0025] Figure 5 An example of training a multilayer perceptron (MLP) is shown.

[0026] Figure 6 An example machine learning (ML) inference algorithm flow is illustrated.

[0027] Figure 7 An example overview of ray tracing is shown.

[0028] Figure 8 An example apparatus for three-dimensional (3D) scene synthesis is illustrated.

[0029] Figure 9An example flowchart illustrating the synthesis of a 3D scene using ray tracing and machine learning (ML) is shown. Detailed Implementation

[0030] The detailed description below, taken in conjunction with the accompanying drawings, is intended as a description of various configurations and not as representing only the configurations in which the concepts described herein can be practiced. To provide a thorough understanding of the various concepts, the detailed description includes specific details. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring these concepts.

[0031] While for the purpose of simplification, a method is shown and described as a series of actions, it should be understood and recognized that the method is not restricted by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions, depending on one or more aspects. For example, those skilled in the art will understand and recognize that a method may alternatively be represented as a series of related states or events, such as in a state diagram. Furthermore, not all exemplified actions are required to implement a method according to one or more aspects.

[0032] Information processing systems (e.g., computing systems or system-on-a-chip (SoCs) with multiple slices (e.g., processing engines) can be used to composite 3D scenes using multiple 2D images. The compositing or rendering of a 3D scene can be performed using multiple 2D images as the basis for 3D scene rendering. In one example, 3D scene rendering may be computationally demanding, making execution on a given computing platform potentially not real-time. That is, the computational processing rate of 3D scene rendering may exceed the capability of a given computing platform to complete execution within the desired timeline (e.g., at a real-time display rate).

[0033] Figure 1An example information processing system 100 is illustrated. In one example, the information processing system 100 includes multiple processing engines, such as a central processing unit (CPU) 120, a digital signal processor (DSP) 130, a graphics processing unit (GPU) 140, a display processing unit (DPU) 180, etc. In one example, various other functions may be included in the information processing system 100, such as a support system 110, a modem 150, memory 160, cache memory 170, and a video display 190. For example, the multiple processing engines and various other functions may be interconnected via an interconnect data bus 105 to transfer data and control information. For example, memory 160 and / or cache memory 170 may be shared among the CPU 120, GPU 140, and other processing engines. In one example, the CPU 120 may include a first internal memory that is not shared with other processing engines. In one example, the GPU 140 may include a second internal memory that is not shared with other processing engines. In one example, any of the multiple processing engines may have internal memory that is not shared with other processing engines.

[0034] In one example, the generation of 3D graphics assets for 3D scene compositing (e.g., the generation of meshes for triangles or textures) has evolved from simple rough sketches to fully cinematic representations of both natural and man-made objects. In another example, mesh and texture creation tools have been developed to run on graphics processing units (GPUs). For instance, most GPUs have post-vertex transform caches optimized for meshes representing tessellations (e.g., tiling) of connections that exhibit appropriate locality similar to space-fill curves. In yet another example, co-optimization techniques can be used to set the average triangle size in pixels based on GPU layers (e.g., in the range of 30 to 100). In one example, a typical GPU has an appropriate ratio of pixel-fill throughput to triangle-processing throughput. For example, GPU processing of triangle meshes with closely spaced repeating vertices or very small triangles can lead to significant performance degradation, as GPUs can be optimized for common offline mesh optimization tools.

[0035] In one example, the problem with fixed-function triangle processing is the hard upper limit on the number of triangles per unit time. For example, the number of triangles per unit time may not easily scale up (unlike the number of ALUs in a GPU), making it possible for content creators to optimize mesh sizes to fit this hard upper limit for high quality until real-time performance constraints (e.g., 30 to 90 frames per second) are met.

[0036] In one example, various compositing techniques allow rendering more triangles per unit time than the GPU's fixed-function hardware design capability. For instance, computer shader tools running in an asynchronous computation pipeline can be used to cull (i.e., edit) geometry. This approach can, for example, reduce the number of triangles encountered by fixed-function triangle processing hardware. In another example, virtualized geometry systems can be used in conjunction with various procedural culling and Level of Detail (LoD) techniques to overcome the hard upper limit on the number of triangles per unit time. In yet another example, compositing techniques that combine advanced culling, geometric LoD, and computation shader-based rasterization can increase the number of triangles that can be processed in real time.

[0037] In one example, many compositing techniques involve high complexity in the creation of graphical assets. That is, the creation of graphical assets may require complex, time-consuming offline processing to achieve results that could be consumed in real time.

[0038] In one example, different methods can be used to construct 3D scene synthesis. For instance, one synthesis method employs a neural network. For example, the neural network is used to implement a mapping function from input coordinates to output coordinates. For example, a learning or training process can be used to iteratively construct the neural network. In one example, the learning process is based on machine learning (ML).

[0039] In one example, a deep neural network (DNN) is used to create graphical assets in real-time using rendering techniques to support interactive 3D graphics. One method for 3D scene synthesis is the Neural Radiation Field (NeRF), which uses a trained multilayer perceptron (MLP) to synthesize novel geometric views from arbitrary angles and depths of focus. For example, an MLP can be trained using a small number of 2D images (e.g., mobile phone camera images). In one example, the creation of NeRF graphical assets can be performed by training a DNN. For example, content creation becomes simplified and ubiquitous because DNN training tools are widely available, unlike other complex and expensive training tools. For example, an MLP is a fully connected (i.e., non-convolutional) neural network.

[0040] While content creation using NeRF can be straightforward, rendering it at real-time speeds can be challenging. In one example, NeRF rendering is analogous to volumetric ray traversal, which requires extensive sampling (e.g., hundreds of rays per pixel) and results in hundreds of millions of MLP inference runs per frame.

[0041] In one example, the modification to NeRF rendering could employ a latent representation of the radiation field involving triangular meshes and textures. In another example, the latent representation of an entity has a lower dimension than the entity itself (e.g., a simpler representation). That is, the latent representation can use simpler elements (e.g., triangular meshes, textures, etc.) to represent the entity. For example, textures could include per-pixel neural features instead of regular colors or normals. In one example, the latent representation allows for more efficient use of existing fixed-function processor hardware for both triangles and textures with better performance (e.g., throughput) than volumetric NeRF techniques.

[0042] In one example, existing fixed-function processor hardware for triangle processing has evolved alongside mesh optimization software tools. For instance, existing hardware is designed for mesh creation without neural rendering techniques. For example, ML training follows different processing conventions than non-neural rendering techniques (e.g., space-filled meshes, large pixel density per triangle, etc.), which can lead to performance degradation. In one example, the ML technique generates disconnected triangles (e.g., no opportunity for post-transformation vertex reuse) and uses relatively small triangle sizes (e.g., 15 pixels per triangle, compared to 50 pixels per triangle for a typical GPU).

[0043] In one example, the existing triangle rasterization pipeline has linear complexity relative to the number of triangles. In one example, each triangle may need to be scanned and transformed to identify which pixel is covered, and then the covered pixels may need to be depth-tested.

[0044] In one example, ray tracing techniques using bounding volume hierarchies (BVHs) have logarithmic complexity relative to the number of triangles. For example, logarithmic complexity can be measured by an amount equal to the logarithmic base of the BVH tree (e.g., 2, 4, 6, or 8). In one example, BVH is a data structure technique used to aggregate entities (e.g., triangles) into bounding boxes in a hierarchical manner. In one example, ray tracing is a synthetic transport model used to simulate the propagation of light from a source to a destination.

[0045] In one example, if ray tracing technology needs to determine the triangles that cover a given screen pixel, the ray tracing algorithm can synthetically emit rays from the camera onto the display to find the nearest triangle that intersects with the rays. For example, using BVH in ray tracing technology allows for the rejection of bounding boxes containing hundreds or thousands of triangles (e.g., axis-aligned bounding boxes AABB) without having to examine each triangle individually.

[0046] In one example, a modified ray tracing technique that processes a triangular mesh generated by an ML tool can perform ray tracing to determine primary visibility. In another example, BVH construction and triangular mesh creation (e.g., via DNN training) can be seamlessly integrated to obtain the rendering benefits of BVH without additional computational cost. In yet another example, primary visibility implies the existence of a direct optical path from the source (e.g., the observer) to the destination (e.g., the scene).

[0047] In one example, the ray tracing technique used to determine primary visibility can be applied to different ML techniques for generating geometry, such as MobileNeRF, Sparse Neural Radiation Mesh SNeRG, and signed distance field (SDF) generation processes with traveling cubes or tetrahedrons. In one example, ray tracing for large meshes (e.g., with millions of triangles) can achieve superior throughput performance compared to other rasterization techniques.

[0048] In one example, ray tracing can be broken down into multiple incremental steps. For example, an accelerated structure can be constructed for all triangles in a frame (e.g., to minimize testing each ray for a large number of triangles). For example, ray generation can be performed by the application based on a desired rendering algorithm. For example, N rays can be generated per screen pixel, where for real-time applications, N can be between ¼ and 4. For example, the determination of the intersection between rays and triangles for a given pixel can use an accelerated structure. For example, if an intersection is found, the application can specify the contribution of the ray to the color of the given pixel. For example, if multiple intersections are found, additional rays can be generated. For example, if a low ratio of rays to pixels exists, denoising can be performed to remove high-frequency noise. For example, the application can specify a denoising (i.e., noise removal) algorithm that matches the desired rendering algorithm.

[0049] In one example, Neural Radiation Field (NeRF) is a technique for creating new 3D scenes and generating 3D volumetric representations using multiple 2D images taken from different viewpoints. For example, NeRF can employ neuron-inspired layered structures to represent datasets. For example, radiation in physical space is the distribution of light energy across a given region in a specific direction over time intervals, measured in watts per square meter per steradian. For example, a field is a continuous energy distribution over a defined spatial volume. For example, simulating radiation in space numerically simulates radiation in physical space.

[0050] Figure 2An example overview of NeRF technique 200 is illustrated. In one example, NeRF technique 200 begins with multiple 2D images 210 as input. In another example, the multiple 2D images 210 are used to generate a neural network 220 (e.g., optimizing NeRF_after optimization). In yet another example, the neural network is used to render a new 3D scene or view 230. Multiple 2D images are used as the basis.

[0051] Figure 3 A volume density example 300 is illustrated. In one example, a first ray 310 and a second ray 320 are traced from a source 330 to a 3D scene 350 via a pixel reference 340. In one example, the first ray 310 terminates at a first endpoint 351 having a first volume density of 1. In one example, the second ray 320 terminates at a second endpoint 352 having a second volume density of 1.

[0052] Figure 4 A volumetric rendering example 400 is illustrated. In one example, source 410 emits ray 420 through pixel reference 430 to produce rendered pixel 440 in 2D image 450. In one example, rendered pixel 440 is produced by accumulating transmittance along ray 420 (per sample / step) and accumulating radiance along ray 420 to obtain pixel color using the accumulated transmittance.

[0053] In one example, transmittance represents the probability of no particle collision along ray 420. In one example, the estimated color function C(r) at a distance r can be expressed as a cumulative sum over layers of a table.

[0054] C(r) = Σ i T i [1-exp( i i )] c i (where i = 1 to N)

[0055] in, i =Volume density of layer i i =Incremental distance of layer i c i =Color of layer i T i =Up to the incremental transmittance of layer i = exp[-Σ j j j(where j = 1 to i-1) Figure 5 A training example 500 for a multilayer perceptron (MLP) is illustrated. In one example, multiple 2D images 510 are used as input. In one example, the multiple 2D images are indexed by a five-dimensional input vector. In one example, the five-dimensional input vector includes 3D spatial locations. r and 2D angle orientation k For example, 3D spatial location r It can be represented using three Cartesian coordinates (x, y, z). For example, 2D angular orientation. k Two angular coordinates can be used ( , ) is used to represent this.

[0056] In one example, a neural network 520 provides a functional mapping between input scenario 510 and output scenario 530. For example, a neural network. 520 is a multilayer perceptron (MLP).

[0057] In one example, a neural network 520 provides output scenarios, while 530 offers multiple colors and volume densities. In one example, multiple colors are decomposed into red, green, and blue (RGB) components. For example, output scene 530 includes a first ray 531 associated with a first image 533 and a second ray 532 associated with a second image 534.

[0058] In one example, volumetric rendering example 540 shows the distance between a first volumetric density profile 541 and a first ray 531, and the distance between a second volumetric density profile 542 and a second ray 532. For example, the first volumetric density profile 541 is used to calculate a first estimated color function using the first ray 531, and the second volumetric density profile 542 is used to calculate a second estimated color function using the second ray 532.

[0059] In one example, rendering loss example 550 illustrates the calculation of a first rendering loss 551 for a first estimated color function and a second rendering loss 552 for a second estimated color function. For example, the first rendering loss 551 and the second rendering loss 552 are calculated as the squared difference with the ground reality (gt) reference function.

[0060] In one example, volumetric rendering can be computationally intensive. For instance, with 256 neural network (e.g., MLP) queries per ray and 762,000 rays per scene, volumetric rendering computation could require approximately 200 million neural network queries per rendered image. For example, the sample GPU throughput resulted in approximately 30 seconds per rendered image (i.e., significantly slower than real-time).

[0061] In one example, using NeRF techniques to render images may not be efficient on some GPU architectures. In this example, the GPU architecture could be optimized for parallel processing of triangular geometry, rasterization, and pixel operations (e.g., texture mapping). For instance, NeRF requires sequential ray sampling and accumulation, and ray beaming access to different segments of the scene volume.

[0062] In one example, the image rendering process using NeRF can transform a 3D scene volume representation into a surface geometry representation. In this example, the surface geometry representation is better matched to the GPU architecture.

[0063] Figure 6 An example machine learning (ML) inference algorithm flow 600 is illustrated. In one example, multiple inference inputs 610 include camera orientation 611, learned mesh 612 (e.g., learned triangular mesh), and learned feature texture 613. For example, camera orientation 611 can be determined by two angular coordinates ( , (This is specified.) For example, camera orientation 611 can be specified by the unit direction vector. k The unit direction vector is specified to have Cartesian coordinates (sin...). cos , sin sin cos For example, the learned mesh 612 and learned feature texture 613 are generated by an ML learning algorithm. In one example, multiple inference inputs 610 are propagated via ray tracing 615 or rasterization 616 to generate multiple rendered feature images 620. In one example, the multiple rendered feature images 620, along with per-pixel feature values ​​621, are ingested by a neural network 630 (e.g., an MLP) to generate a final rendered image 640 along with a final per-pixel color 631.

[0064] Figure 7An example overview of ray tracing 700 is illustrated. In one example, multiple ray tracing inputs 710 include a learned mesh 711 (e.g., a learned triangle mesh), a view orientation 712, and a learned feature texture 713. In one example, the multiple ray tracing inputs 710 are inputs to a ray tracing module 720. In one example, the ray tracing module 720 includes an offline BVH build 721, a BVH block 722, a real-time ray query compute shader dispatch 723, a shader 724, a feature texture lookup block 725 (when a ray hits), and a ray traversal block 726. In one example, ray beams from the ray tracing module 720 are used to generate multiple ray tracing feature images 730.

[0065] Figure 8 An example apparatus 800 for compositing three-dimensional (3D) scenes is illustrated. In one example, apparatus 800 includes an application 810, a graphics processing unit (GPU) 820, and a display unit 830. In one example, GPU 820 includes a shader processor (SP) 821, a ray traversal unit (RTU) 822, and a memory 823.

[0066] In one example, the ray query compute shader runs on SP 821 and calls RTU 822 to perform ray traversal and intersection operations to the BVH. In another example, the intersection determination is returned to SP 821. In yet another example, the BVH data is stored in memory 823 and retrieved by RTU 822 as needed.

[0067] Figure 9 Example flowchart 900 illustrates 3D scene synthesis using ray tracing and machine learning (ML). In box 910, multiple 2D images are ingested for machine learning (ML) training. In one example, multiple 2D images are ingested for machine learning (ML) training. In one example, the multiple 2D images are indexed by a five-dimensional input vector. In one example, the five-dimensional input vector includes 3D spatial locations. r and 2D angle orientation k For example, 3D spatial location r It can be represented using three Cartesian coordinates (x, y, z). For example, 2D angular orientation. k Two angular coordinates can be used ( , ) is used to represent this.

[0068] In box 920, an initial feature field neural network and an initial opacity field neural network are established. In one example, an initial feature field neural network and an initial opacity field neural network are established. In one example, the initial feature field neural network is a feature field multilayer perceptron (MLP) neural network. In one example, the initial opacity field neural network is an opacity field multilayer perceptron (MLP) neural network. In one example, the initial feature field neural network specifies the color distribution in 3D space. In one example, the initial opacity field neural network specifies the volume density distribution in 3D space.

[0069] In box 930, an initial mesh and initial feature texture are generated from multiple two-dimensional (2D) images using an initial feature field neural network and an initial opacity field neural network. In one example, an initial mesh and initial feature texture are generated from multiple two-dimensional (2D) images using an initial feature field neural network and an initial opacity field neural network.

[0070] In one example, the initial mesh is a collection of 3D spatial samples representing a geometric object. In one example, the initial feature texture is a 2D image of the texture features of the geometric object. In one example, the initial mesh is a learnable mesh. In one example, the initial feature texture is a learnable feature texture. In one example, iteration can be used to update the initial mesh to generate an updated mesh. In one example, iteration can be used to update the initial feature texture to generate an updated feature texture.

[0071] In box 940, ray tracing is used to synthesize an initial 3D scene using an initial simplified neural network, an initial mesh, and initial feature textures. In one example, ray tracing is used to synthesize an initial 3D scene using an initial simplified neural network, an initial mesh, and initial feature textures. In one example, ray tracing includes a bounding volume hierarchy (BVH) technique. In one example, ray tracing includes the determination of primary visibility. In one example, the initial simplified neural network has a lower dimensionality than the initial feature field neural network and the initial opacity field neural network.

[0072] In box 950, the initial 3D scene is backpropagated to the initial simplified neural network, the initial feature field neural network, and the initial opacity field neural network to create a trained simplified neural network using forward propagation and ray tracing. In one example, the initial 3D scene is backpropagated to the initial simplified neural network, the initial feature field neural network, and the initial opacity field neural network to create a trained simplified neural network using forward propagation and ray tracing. In one example, ray tracing includes the bounding volume hierarchy (BVH) technique. In one example, ray tracing includes the determination of primary visibility. In one example, forward propagation is used to synthesize the initial 3D scene. In one example, forward propagation means processing the input data in the direction toward the output.

[0073] In box 960, an inferred 3D scene is synthesized using an updated simplified neural network, leveraging the updated learned mesh and updated learned feature textures, and employing ray tracing. In one example, an inferred 3D scene is synthesized using an updated simplified neural network, leveraging the updated learned mesh and updated learned feature textures, and employing ray tracing.

[0074] In one example, iteration can be used to update a learned mesh (e.g., a learned triangle mesh) to generate an updated learned mesh (e.g., an updated learned triangle mesh). In one example, iteration can be used to update a learned feature texture to generate an updated learned feature texture. In one example, iteration can be used to update a trained simplified neural network to generate an updated simplified neural network.

[0075] In one example, backpropagation and forward propagation can be performed iteratively. In another example, the iterative backpropagation and forward propagation can be terminated when a stopping rule is reached. In another example, the stopping rule is a prior condition imposed on the initial simplified neural network. In yet another example, backpropagation means processing the output data in the direction toward the input.

[0076] In one example, the inferred 3D scene includes view-dependent colors. In another example, the inferred 3D scene is based on neural network optimization. For example, the neural network optimization uses a trained simplified neural network. In one example, the neural network optimization uses forward propagation.

[0077] In one respect, Figure 9 One or more steps in the process of providing 3D scene synthesis using ray tracing and machine learning (ML) may be performed by one or more processors, which may include hardware, software, firmware, etc. For example, one or more processors may be used to execute software or firmware that is executing... Figure 9The steps required in the flowchart. Software should be interpreted broadly to mean instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, processes, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description languages, or others.

[0078] Software may reside on a computer-readable medium. The computer-readable medium may be a non-transitory computer-readable medium. Examples of non-transitory computer-readable media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical disks (e.g., compact optical disks (CDs) or digital versatile optical disks (DVDs)), smart cards, flash memory devices (e.g., card, stick, or key drives), random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, removable disks, and any other suitable medium for storing software and / or instructions accessible and readable by a computer. By way of example, the computer-readable medium may also include a carrier wave, a transmit line, and any other suitable medium for transmitting software and / or instructions accessible and readable by a computer. The computer-readable medium may reside in a processing system, outside the processing system, or distributed across multiple entities including the processing system. The computer-readable medium may be implemented in a computer program product. For example, a computer program product may include a computer-readable medium in packaging material. The computer-readable medium may include software or firmware. Those skilled in the art will recognize that the optimal implementation of the functions described throughout this disclosure depends on the specific application and the overall design constraints imposed on the system as a whole.

[0079] Any circuitry included in the processor is provided merely as an example, and other components for performing the described functions may be included in various aspects of this disclosure, including, but not limited to, instructions stored in a computer-readable medium, or any other suitable means or components described herein and utilizing, for example, the processes and / or algorithms described herein with respect to the example flowcharts.

[0080] Within this disclosure, the term "exemplary" is used to mean "serving as an example, instance, or illustration." Any specific implementation or aspect described herein as "exemplary" is not necessarily to be construed as superior to or better than other aspects of this disclosure. Similarly, the term "aspect" does not require that all aspects of this disclosure include the features, advantages, or modes of operation discussed. The term "coupled" is used herein to refer to direct or indirect coupling between two objects. For example, if object A physically contacts object B, and object B contacts object C, objects A and C can still be considered coupled to each other, even if they are not in direct physical contact. The term "circuit" is used broadly, and it is intended to include both hardware implementations of electronic devices and conductors (where these electronic devices and conductors, when connected and configured, perform the functions described in this disclosure, without limitation on the type of electronic circuit) and software implementations of information and instructions (where these information and instructions, when executed by a processor, perform the functions described in this disclosure).

[0081] One or more of the components, steps, features, and / or functions illustrated in the accompanying drawings may be rearranged and / or combined into a single component, step, feature, or function, or embodied in several components, steps, or functions. Additional elements, components, steps, and / or functions may be added without departing from the novel features disclosed herein. The apparatus, devices, and / or components illustrated in the accompanying drawings may be configured to perform one or more of the methods, features, or steps described herein. The novel algorithms described herein can also be efficiently implemented in software and / or embedded in hardware.

[0082] It should be understood that the specific order or hierarchy of steps in the disclosed methods is an example of an exemplary process. It should be understood that the specific order or hierarchy of steps in these methods may be rearranged based on design preferences. The appended method claims present the elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented, unless specifically stated herein.

[0083] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are to be consistent with the full scope of the claims, wherein references to elements in the singular form are not intended to mean “one and only one”, but rather “one or more”, unless specifically stated otherwise. Unless specifically stated otherwise, the term “some” means one or more. The phrase “at least one of” referring to the list of items means any combination of those items, including individual members. As an example, “at least one of a, b, or c” is intended to cover: a; b; c; a and b; a and c; b and c; and a, b, and c. All structural and functional equivalents of the elements throughout the various aspects described herein that are known to or will later be known to a person skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly recited in the claims. No element of a claim shall be construed under the provisions of 35 USC §112, paragraph 6, unless the element is explicitly stated using the phrase “for a component of” or, in the case of a method claim, the element is stated using the phrase “for a step of”.

[0084] Those skilled in the art will understand that various features of different implementations can be combined or modified and still remain within the spirit and scope of this disclosure.

Claims

1. An apparatus, the apparatus comprising: A memory configured to store learned triangular meshes and learned feature textures; A graphics processing unit (GPU) coupled to the memory, the GPU being configured to render an inferred three-dimensional (3D) scene using ray tracing based on the learned triangular mesh and the learned feature texture; and A display unit coupled to the GPU is configured to display the inferred 3D scene.

2. The apparatus of claim 1, wherein the GPU includes a shader processor configured to process the learned triangular mesh and the learned feature texture.

3. The apparatus of claim 2, wherein the GPU further comprises a ray tracing unit configured to perform the ray tracing.

4. The apparatus of claim 3, wherein the ray tracing includes the determination of primary visibility.

5. The apparatus of claim 3, wherein the ray tracing comprises enclosing volume hierarchy (BVH) technology.

6. The apparatus of claim 3, wherein the shader processor is further configured to infer the inferred three-dimensional (3D) scene to output view-related colors.

7. The apparatus of claim 6, wherein the shader processor is further configured to utilize the ray tracing to synthesize the inferred three-dimensional (3D) scene using a simplified neural network.

8. The apparatus of claim 7, wherein the shader processor is further configured to backpropagate a plurality of two-dimensional (2D) images to the simplified neural network, the initial feature field neural network, and the initial opacity field neural network to generate an updated learned triangle mesh and an updated learned feature texture and an updated simplified neural network.

9. The apparatus of claim 8, wherein the shader processor is further configured to use forward propagation and leverage the ray tracing to infer the updated simplified neural network.

10. The apparatus of claim 9, wherein the shader processor is further configured to synthesize the inferred three-dimensional (3D) scene using the updated learned triangular mesh and the updated learned feature texture and the ray tracing, employing the updated simplified neural network.

11. A method, the method comprising: The initial mesh and initial feature texture are generated from multiple two-dimensional (2D) images using an initial feature field neural network and an initial opacity field neural network; as well as Using ray tracing, an initial simplified neural network, the initial mesh, and the initial feature texture are used to synthesize an initial three-dimensional (3D) scene.

12. The method of claim 11, wherein the initial simplified neural network is a multilayer perceptron (MLP) neural network.

13. The method of claim 11, wherein the initial mesh is a collection of three-dimensional (3D) spatial samples representing geometric objects.

14. The method of claim 11, wherein the ray tracing includes the determination of primary visibility.

15. The method of claim 11, wherein the ray tracing comprises bounding volume hierarchy (BVH) technology.

16. The method of claim 11, further comprising using forward propagation to synthesize the initial three-dimensional (3D) scene.

17. The method of claim 11, further comprising backpropagating the initial 3D scene to the initial simplified neural network, the initial feature field neural network, and the initial opacity field neural network to create a trained simplified neural network using forward propagation and leveraging the ray tracing.

18. The method of claim 17, further comprising using an updated learned mesh and an updated learned feature texture and utilizing the ray tracing to synthesize an inferred three-dimensional (3D) scene using an updated simplified neural network.

19. The method of claim 18, wherein the ray tracing includes the determination of primary visibility.

20. The method of claim 18, wherein the ray tracing comprises enclosing volume hierarchy (BVH) technology.

21. The method of claim 18, further comprising outputting one or more view-dependent 3D scenes from an updated mesh and updated feature textures.

22. The method of claim 11, wherein the initial simplified neural network has a lower dimension than the initial feature field neural network and the initial opacity field neural network.

23. The method of claim 21, further comprising establishing the initial feature field neural network and the initial opacity field neural network.

24. The method of claim 23, further comprising ingesting the plurality of two-dimensional (2D) images for machine learning (ML) training.

25. An apparatus comprising: Used for the following components: initial meshes and initial feature textures generated from multiple two-dimensional (2D) images using an initial feature field neural network and an initial opacity field neural network; and The component is used to synthesize an initial three-dimensional (3D) scene using ray tracing, an initial simplified neural network, the initial mesh, and the initial feature texture.

26. The apparatus of claim 25, further comprising: The component is used to backpropagate the initial 3D scene to the initial simplified neural network, the initial feature field neural network, and the initial opacity field neural network to create a trained simplified neural network using forward propagation and ray tracing. and The component is used to synthesize an inferred three-dimensional (3D) scene using an updated learned mesh and updated learned feature textures and by utilizing the ray tracing.

27. The apparatus of claim 26, further comprising: Components used to establish the initial feature field neural network and the initial opacity field neural network; and A component used to ingest the multiple two-dimensional (2D) images for machine learning (ML) training.

28. A non-transitory computer-readable medium storing computer-executable code operable on a device, the device including at least one processor and at least one memory coupled to the at least one processor, wherein the at least one processor is configured to perform three-dimensional (3D) scene compositing using ray tracing, the computer-executable code comprising: Instructions for enabling a computer to use an initial mesh and initial feature texture generated from multiple two-dimensional (2D) images by an initial feature field neural network and an initial opacity field neural network; and Instructions for enabling the computer to synthesize an initial three-dimensional (3D) scene using the ray tracing, an initial simplified neural network, the initial mesh, and the initial feature texture.

29. The non-transitory computer-readable medium according to claim 28, further comprising: Instructions for instructing the computer to backpropagate the initial 3D scene to the initial simplified neural network, the initial feature field neural network, and the initial opacity field neural network, in order to use forward propagation and leverage the ray tracing to create a trained simplified neural network; and Instructions for enabling the computer to synthesize an inferred three-dimensional (3D) scene using an updated learned mesh and updated learned feature textures and by utilizing ray tracing, and using an updated simplified neural network.

30. The non-transitory computer-readable medium according to claim 29, wherein the non-transitory computer-readable medium further comprises: Instructions for enabling the computer to establish the initial feature field neural network and the initial opacity field neural network; and Instructions for the computer to ingest the plurality of two-dimensional (2D) images for machine learning (ML) training.

Images