Method and system for real-time neural network radiance caching for path tracing

Abstract

A real-time neural network radiance cache for path tracing is disclosed. A real-time neural radiance cache technique for path-traced global illumination is implemented using a neural network for caching the diffuse radiation component of global illumination. The neural (network) radiance cache handles fully dynamic scenes and makes no assumptions about cameras, lighting, geometry, and materials. Compared to conventional caching, the data-driven approach circumvents many of the difficulties of caching algorithms, such as positioning, interpolating, and updating cache points. The neural radiance cache is trained via online learning during rendering. The advantages of the neural radiance cache are denoising and real-time performance. Importantly, the runtime overhead and memory footprint of the neural radiance cache are stable and independent of scene complexity.

Description

[0001] Priority declaration

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 155,830 (Attorney Docket No. 513416), entitled “Real-Time Neural Radiance Caching for Path Tracing,” filed on March 3, 2021, the entire contents of which are incorporated herein by reference. Background Art

[0003] Photorealistic rendering using path tracing relies on both direct and indirect lighting to produce realistic images. Path tracing can be used to simulate global illumination, but it remains a challenge for real-time rendering. Indirect lighting involves multiple bounces, including contributions from sources such as windows or open doors. Even in offline rendering, performing global illumination is still cumbersome, especially when considering high-order indirect lighting. Long paths are traced from the camera and intersecting objects in the scene to accumulate radiance for each pixel in the image. Typically, due to sampling, accumulated radiance is noisy, and longer paths tend to accumulate more noise. Conventionally, irradiance probes are used to reduce the noise of indirect lighting. However, irradiance probes need to be precomputed and only include the diffuse component of indirect lighting. A particularly attractive approach to reducing the time required to trace long paths is to cache radiance samples for later reuse. The contents of the cache are also precomputed. Therefore, neither irradiance probes nor caches are suitable solutions for real-time rendering of dynamic scenes. There is a need to address these and / or other issues associated with existing techniques. Summary of the Invention

[0004] A real-time neural radiance cache technique for path-traced global illumination is implemented using a neural network for caching the diffuse radiation component of global illumination. The neural (network) radiance cache handles fully dynamic scenes and makes no assumptions about cameras, lighting, geometry, and materials. Compared to conventional caching, the data-driven approach circumvents many of the difficulties of conventional caching algorithms, such as locating, interpolating, and updating cache points. The neural radiance cache is trained to approximate the scene's indirect lighting via online learning during rendering, rather than storing pre-computed values for the scene.

[0005] A path is traced from the camera to an intersection location in the scene. The intersection location and view direction are input to the Neural Radiance Cache to generate an estimate of the radiance contribution at that intersection location. The traced path is terminated in the Neural Radiance Cache to shorten the path and reduce noise. The Neural Radiance Cache learns quickly for real-time rendering, provides accurate estimates of sharp edges of objects and shadows, and produces accurate overall color. The advantages of the Neural Radiance Cache are noise reduction and real-time performance. Importantly, the runtime overhead and memory footprint of the Neural Radiance Cache are stable and independent of scene complexity.

[0006] Embodiments of the present disclosure relate to a neural network radiation cache. Systems and methods are disclosed for processing a three-dimensional (3D) position associated with a light transport path through a scene by a neural network radiation cache model to generate a radiation (reflected light) prediction at the 3D position. An image of the scene corresponding to a camera connected to the light transport path is generated using the radiation prediction, and the rendered (light transport) path is extended to generate an extended path terminating at a terminal 3D position. The neural network radiation cache model processes the terminal 3D position to generate a second radiation prediction at the terminal 3D position, and updates weights of the neural network radiation cache model based on the second radiation prediction and the second radiation prediction propagated to the 3D position. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present system and method for neural network radiance caching is described in detail below with reference to the accompanying drawings, in which:

[0008] Figure 1A Shown is a path traced through a scene according to the prior art.

[0009] Figure 1B Paths terminating in a neural network radiance cache are shown, according to an embodiment.

[0010] Figure 1C is a conceptual diagram illustrating first and second optical transmission paths, wherein the second optical transmission path is extended to produce an extended optical transmission path suitable for implementing some embodiments of the present disclosure.

[0011] Figure 1D is a block diagram of a neural network radiance caching system suitable for implementing some embodiments of the present disclosure.

[0012] Figure 1E A reference image and an image generated using a neural network radiance cache are shown according to an embodiment.

[0013] Figure 2A A flowchart is shown of a method for generating images using a neural network radiance cache and training a neural network radiance cache suitable for implementing some embodiments of the present disclosure.

[0014] Figure 2B Portions of an image generated as the number of training cycles increases according to an embodiment are shown.

[0015] Figure 2C Path scattering interactions of different materials according to an embodiment are shown.

[0016] Figure 2D Path termination according to scattering interactions according to an embodiment is shown.

[0017] Figure 2E A flowchart of a method for terminating a rendering or training path into a neural network radiance cache suitable for implementing some embodiments of the present disclosure is shown.

[0018] Figure 3A is a conceptual diagram illustrating batch neural network evaluation suitable for implementing some embodiments of the present disclosure.

[0019] Figure 3B The distribution of batches across thread blocks suitable for implementing some embodiments of the present disclosure is shown.

[0020] Figure 3C A per-thread-block matrix multiplication suitable for implementing some embodiments of the present disclosure is shown.

[0021] Figure 3D A flowchart of a method for batch neural network evaluation suitable for implementing some embodiments of the present disclosure is shown.

[0022] Figure 3E An example parallel processing unit suitable for implementing some embodiments of the present disclosure is shown.

[0023] Figure 4A Shown in Figure 3E An example general-purpose processing cluster within a parallel processing unit suitable for implementing some embodiments of the present disclosure.

[0024] Figure 4B Some embodiments suitable for implementing the present disclosure are shown. Figure 3E An example memory partitioning unit of a parallel processing unit.

[0025] Figure 4C Shown Figure 4A Examples of streaming multiprocessors suitable for implementing some embodiments of the present disclosure.

[0026] Figure 5A is a conceptual diagram of a processing system implemented using the PPU of FIG. 4 suitable for implementing some embodiments of the present disclosure.

[0027] Figure 5B An exemplary system is shown that can implement the various architecture and / or functionality of various previous embodiments.

[0028] Figure 5C Components of an exemplary system that can be used to train and utilize machine learning in at least one embodiment are shown.

[0029] Figure 6 An exemplary streaming system suitable for implementing some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0030] A system and method involving a real-time neural network radiance cache for path tracing is disclosed. During rendering, short paths that terminate in a neural (network) radiance cache are traced. Input to the neural radiance cache includes query vertex x k , which can be located on a surface or within a volume in the scene. In response to the input, the neural radiance cache produces the scattered radiance L at the vertex s A prediction or approximation of reflected light (radiance) is made based on the image. The approximation aggregates many samples from spatially and temporally nearby locations in the scene. A subset of the rendered light transport paths is extended by a training suffix (one or more additional segments / vertices) to provide trained (extended) light transport paths that are used to update parameters (e.g., weights) of the neural radiance cache.

[0031] In an embodiment, a single (short) light transport path is traced for each pixel to provide a rendering path. In an embodiment, the rendering path is reused and extended to generate a training path. In an embodiment, sparse sampling is used to generate a small number of training paths from the rendering path (e.g., 3% of the rendering path may be extended). In an embodiment, sparse sampling is used to select the rendering path, and the number of sparse samples remains constant as the image resolution increases. In an embodiment, a radiance approximation is collected at each intersection (vertex) along the training path to update the neural radiance cache for each vertex along the training path (and rendering path).

[0032] Figure 1AA path traced through a scene according to the prior art is shown. A path is traced from a camera 101 through a pixel 102 of an image and into the scene. Direct illumination can be determined at the first intersection (bounce) of the path with a surface or light source. Indirect illumination includes contributions beyond the first intersection. Conventionally, a path terminates at a terminal vertex when it leaves the scene (e.g., leaves a window 104) or reaches a light source 103. Each path contains multiple segments or rays defined by vertices at which the path intersects a surface in the scene. At each vertex, the direction of the next path segment is determined. When a path terminates, the radiance contribution is collected and continuously transferred backward along the path from the terminal vertex to any intermediate vertices, terminating at the first vertex in the path to provide a radiance contribution for calculating the color of the pixel. Conventionally, tracing more paths improves image quality (reduces noise), but also increases the cost in terms of rendering time. Tracing long paths cannot usually be performed in real time.

[0033] Figure 1B Paths terminating in a neural network radiance cache are shown, according to an embodiment. In contrast to the long paths traced to perform conventional global illumination, Figure 1B The paths in are terminated early, and the terminal vertices 105, 110, 112, and 114 are processed by the neural radiance cache to generate an approximation of indirect brightness, reflected light, or radiance at each terminal vertex. Terminating paths into the neural radiance cache saves computation and, more importantly, replaces single-sample estimates with approximations that aggregate many samples from spatially and temporally close locations. As a result, noise is significantly reduced, however, the feasibility of the cache for real-time applications still depends on how efficiently and quickly the neural radiance cache can be queried and updated.

[0034] After rendering an image using the Neural Radiance Cache to approximate indirect luminance, the Neural Radiance Cache is trained before rendering the next image. In an embodiment, rendering a single frame includes calculating pixel color and updating the Neural Radiance Cache. Alternating between rendering and training allows the Neural Radiance Cache to adapt to changes in the scene, camera position, etc.

[0035] Figure 1C is a conceptual diagram illustrating a first light transport path and a second light transport path suitable for implementing some embodiments of the present disclosure, wherein the second light transport path is expanded to generate a training path. A scene includes a light source 116 and several surfaces, including surfaces 122 and 118. In one embodiment, a rendering light transport path is traced for each pixel. The first and second light transport paths x and y are traced through the scene starting at camera 115 at vertices x0 and y0.

[0036] like Figure 1CAs shown, the first light transport path is a short rendering path that terminates in the neural radiance cache at the terminal vertex x2 located on the surface 118. In the short rendering path including two fragments, the intermediate vertex x1 that hits the surface 122 is included between the vertex x2 and the camera 115. The neural radiance cache is queried at the vertex x2 to obtain the estimated radiance 124 at the vertex or In an embodiment, the rendering path is terminated once the estimate provided by the neural radiance cache is deemed sufficiently accurate.

[0037] When the estimate is deemed sufficiently accurate, vertex x2 is identified as the terminal vertex. A heuristic may be evaluated at each vertex to determine whether the estimate generated by the Neural Radiance Cache is sufficiently accurate or whether to trace another fragment. In an embodiment, the heuristic indicates whether the extension of the rendering path is large enough to obscure small inaccuracies in the Neural Radiance Cache. Note that the estimate provided by the Neural Radiance Cache at vertex x1 is not deemed sufficiently accurate, so the rendering path does not terminate at vertex x1. Multiple importance sampling may be applied to determine the direction in which to trace fragments of a short rendering path. In an embodiment, the 3D position of a vertex and the direction ω into that vertex are considered to be sufficiently accurate. k is fed into the neural radiation cache to generate

[0038] The radiance at each vertex is collected and continuously transmitted to intermediate vertices along a path starting from the terminal vertex and ending at camera 115 to produce a radiance approximation for the pixel. In an embodiment, a next event estimate can be used to integrate the light from the emitter at each intermediate vertex.

[0039] The second light transport path is a short rendering path that terminates in the neural radiance cache at the terminal vertex y2 on surface 118. The intermediate vertex y1 that hits surface 122 is included in the short rendering path between vertex y2 and camera 115. The neural radiance cache is queried at vertex y2 to obtain an estimated radiance 125 at the vertex. The estimated radiance 125 provided by the neural radiance cache at the terminal vertex y2 is collected and transported along the short rendering path to the intermediate vertex y1 to produce a radiance approximation for the pixel. Once all rendering paths for an image are traced, the neural radiance cache can be optimized through training.

[0040] To optimize the neural radiation cache, a small number (typically below 3%) of short rendering paths are extended by a few vertices called "training suffixes". In an embodiment, the training suffix includes at least one additional vertex compared to the rendering path. A second light transport path y extends from the terminal vertex y2 to generate a training suffix 120 including two additional vertices y3 and y4. The combined rendering path and training suffix 120 produce a training path that is longer than the rendering path. The training path can be terminated based on the same heuristics as the rendering path by using the terminal vertex of the rendering path as the primary vertex instead of the vertex located at the camera 115. For example, to terminate the training path at vertex y4, the terminal vertex of the rendering path (y2) is used as the primary vertex.

[0041] The neural radiance cache itself is used to generate target values for training. More specifically, the neural radiance cache is evaluated at the terminal vertex of a longer training path to generate a target value for training. A radiance estimate 127 at the terminal vertex of the training path is generated by the neural radiance cache and then transferred to other vertices in the training suffix 120 to generate a radiance approximation 128 at the intermediate vertex y3. Radiance approximation 128 can be used as a reference value for training the neural radiance cache. Furthermore, in a second light transmission path, radiance approximation 128 is transferred to vertices y2 and y1 to generate additional radiance approximations 129 and 126 at vertices y2 and y1, respectively. The additional radiance approximations can also be used as additional reference values for training the neural radiance cache. For example, the additional radiance approximation 129 at y2 can be used as a reference value compared to the radiance estimate 125 provided by the neural radiance cache at the terminal vertex y2. Online training gradually increases the number of simulated light bounces in conjunction with the termination of the training path into the neural radiance cache.

[0042] As in any data-driven approach, the quality of the estimates or predictions produced by the neural radiance cache depends on the reference or target value L used to train the neural radiance cache. s The unique challenge of rendering dynamic scenes in real time requires continuous adaptation of the neural radiance cache according to the changing radiance field (e.g., due to moving lights or geometry). Therefore, it is not possible to precompute the exact target value L for optimization. s .

[0043] Updating the neural radiation cache with its own values is similar to the concept of Q-learning as described by Ken Dahm and Alexander Keller in their talk “Learning Light Transport the Reinforced Way” presented at the ACM SIGGRAPH 2017 (SIGGRAPH17) in New York, USA, page 2, item 73. The Q-learning approach has two distinct advantages over path tracing estimation: when estimating L s It trades off unwanted noise against (potential) bias when training. It also allows capturing global illumination as long as the training process is repeated: the radiance learned by one training path is transferred using multiple other training paths in the next iteration. Thus, each iteration increases the number of simulated light bounces.

[0044] However, Q-learning on a neural radiance cache also has two caveats: First, the terminal vertex of a training path can reach scene positions for which the neural radiance cache has not been trained, which may cause larger approximation errors. The second disadvantage is that the iterative optimization can only simulate a subset of multi-bounce lighting rather than all light transmissions. Specifically, only transmissions from emitters that can be reached by the training path will bounce back and forth further, and only then, if the terminal vertex of the training path in a subsequent frame lands near the current optimization point (i.e., in Figure 1C In

[15] , y4 needs to be close to y2 or y3. Both of these caveats can be mitigated almost for free by making a small fraction u of the training path truly unbiased, meaning tracing long paths and thus injecting the correct source values for propagation by the Q-learning mechanism. In an embodiment, u = 1 / 16, i.e., except for Russian roulette, every 16th training suffix will never terminate.

[0045] In an embodiment, the performance of the neural radiance cache is improved by providing additional attribute inputs along with the query vertex to improve rendering quality. The additional inputs may include view direction, material properties, surface normal vectors (if the query vertex is not in a volume), etc. For example, each of the different surfaces 118 and 122 may be a different type of material with different light scattering properties. In an embodiment, the query vertex and additional inputs are pre-processed before being input into the neural radiance cache.

[0046] More illustrative information will now be provided regarding various optional architectures and features that can implement the aforementioned framework, depending on the user's desires. It should be noted that the following information is provided for illustrative purposes and should not be construed as limiting in any way. Any of the following features may be optionally incorporated with or without excluding the other features described.

[0047] Figure 1D is a block diagram of a neural network radiance caching system 150 suitable for implementing some embodiments of the present disclosure. The neural network radiance caching system 150 includes a path tracer 130, a neural radiance cache 135, and a loss function 140. It should be understood that this and other arrangements described herein are presented by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, orders, functional groupings, etc.) may be used in addition to or in place of the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in a memory. Moreover, those skilled in the art will understand that any system that performs the operations of the neural network radiance caching system 150 is within the scope and spirit of the embodiments of the present disclosure.

[0048] During rendering, the path tracer 130 receives scene and camera data and traces short rendering paths, inputting intersection data into the neural (network) radiance cache 135 to generate a bounce light approximation. The intersection data may include one or more of the vertex's 3D position, direction of incidence (field of view direction), and attributes such as material properties, surface normal vector (if the query vertex is not in the volume), etc. In an embodiment, the 3D position is the terminal vertex of the first segment that defines the path of the ray. In an embodiment, the 3D position is the terminal position of subsequent segments that define the path. The bounce light approximation is used to calculate the color of a pixel of the image.

[0049] The Neural Radiance Cache 135 estimates or predicts diffuse radiance, which is the most computationally expensive component of the rendering equation.

[0050]

[0051] Indicates that after scattering at x, in the direction ω I The radiation energy leaving point x. For a given incident direction ω I , the integrand is the bidirectional scattering distribution function (BSDF) f s (x,ω,ω I ), incident radiation rate Li (x,ω I ), and the shortening term |cosθ i |, where θ I Yes I The angle between L and the surface normal at x. The neural radiation cache 135 converts L s Estimated as cached (approximate) radiance The estimated radiance corresponding to the terminal vertex of the rendering path is back-propagated to the intermediate vertices in each path to reconstruct the image. Similarly, in an embodiment, the estimated radiance corresponding to the terminal vertex of the training path is propagated back along each training path to produce an updated reflected light approximation for each intermediate vertex between the terminal vertex and the camera.

[0052] For training or optimization, the path tracer 130 extends a subset of short paths traced by at least one fragment during rendering to produce training paths that end at new terminal vertices. In an embodiment, the path tracer 130 stores rendering path data for at least a portion of the rendering path in a memory for reuse in generating training paths. In an embodiment, pixels are organized into tiles, and a single rendering path in each tile is selected for extension to produce a training path. In an embodiment, a single rendering path is selected based on a random or quasi-random offset to produce a uniformly sparse set of training paths in screen space. In an embodiment, the tile size is dynamically adjusted at each frame based on the number of training paths generated during image reconstruction. Therefore, the processing cost of training can be decoupled from the image resolution.

[0053] Compared to generating the original training path, when the training path is generated by extending the existing rendering path, the processing overhead for training is reduced because the computation is shared between the two paths. In contrast, conventional caching techniques based on probe volumes use a separate set of rays to update the cache, and this separate set of rays does not contribute to the image itself. Furthermore, training the neural radiance cache 135 is equivalent to regressing many samples from spatially and temporally close locations, i.e., a form of path-space denoising. Therefore, by replacing the sample radiance estimates with the estimates provided by the neural radiance cache 135, the variance is significantly reduced.

[0054] In an embodiment, the estimated radiance generated by the neural radiance cache 135 for each terminal vertex of the rendering path and each terminal vertex of the training path is stored together with the terminal vertex. In an embodiment, the estimated radiance generated by the neural radiance cache 135 for each terminal vertex of the training path is stored together with the terminal vertex, and the approximate radiance calculated for each intermediate vertex is also stored for use by the loss function 140 as a target for optimizing the neural radiance cache 135.

[0055] The loss function 140 receives the radiance approximation of the rendering path and the updated radiance approximation of the training path and calculates the updated weights W of the neural network radiance cache 135 t , where t is the gradient descent step. To facilitate learning from noisy training data, when the training signal - the reflected radiation L s (x,ω) can only estimate the relative Loss. In an embodiment, the normalization of the loss is driven by the neural prediction:

[0056]

[0057] where ∈ = 0.01 and sg(·) indicates that its independent variable is considered constant in the optimization, i.e. no gradient is back-propagated with respect to it. In an embodiment, for L s For the spectral values of (x,ω), the loss of each color channel is normalized by the squared brightness across the spectrum.

[0058] In an embodiment, the estimated radiance generated by the neural radiance cache 135 at the terminal vertex is multiplied by at least one of the diffuse reflectance, the specular reflectance, the normal vector, the roughness, the albedo, the BRDF, and the phase function of the material properties associated with the terminal vertex. In an embodiment, the estimated radiance is multiplied by the sum of the diffuse reflectance and the specular reflectance of the material associated with the terminal vertex, α(x,ω)+β(x,ω), the diffuse reflectance α and the specular reflectance β.

[0059] When rendering dynamic content (e.g., changing camera positions or animated geometry), the neural radiance cache 135 needs to be continuously adapted. In an embodiment, a high learning rate is used when optimizing the neural radiance cache 135 by gradient descent. In addition, multiple (e.g., 4) gradient descent steps can be performed per frame, resulting in even faster adaptation. In an embodiment, each step uses a disjoint random subset of the training data collected while rendering the frame to prevent the same data from being seen twice.

[0060] However, a side effect of such aggressive optimization scheduling can also be temporal artifacts like flickering and oscillation across rendered frames - even when the scene and camera are static, due to the presence of noise in the approximate radiance target. In an embodiment, updates to the neural radiance cache 135 parameters (weights) are filtered for rendering to reduce temporal artifacts (e.g., flickering and oscillation). In an embodiment, the filtering is not fed back into training and is applied only to the version of the parameters used for rendering.

[0061] In an embodiment, oscillations are dampened by averaging the updated weights produced by the loss function 140 during optimization. More specifically, in an embodiment, the network weight W produced by the t-th gradient descent step is calculated as t The exponential moving average (EMA) of , which creates the weights used by the neural radiance cache 135 to evaluate 3D positions during rendering The second set or version of the exponential moving average readings.

[0062]

[0063] The bias of the average is corrected for small t, and α∈[0,1] controls the strength of the exponential average. In an embodiment, for the weights A good compromise between fast adaptation and stable evolution in time, α = 0.99. In an embodiment, α = 0. In an embodiment, α = 0.90. As previously described, the averaging process is not fed back into the training loop; the second set of weights Depends on W t , but not the other way around.

[0064] Figure 1E A reference image 155 and an image 145 generated using a neural network radiance cache are shown in accordance with an embodiment. Many details of the reference image 155 are accurately represented in the image 145. For example, the shape and edge details of the awning 158 (including shading and color) are very accurate in the image 145. However, the shadows 156 of the thin strings between the ceiling lights are missing in the image 145.

[0065] The neural network radiance cache system 150 can achieve real-time frame rates on current hardware and handle a wide range of materials and lighting configurations. Figure 1E The scene shown in

[15] can be rendered with dynamic geometry, materials, and / or lighting. The Neural Radiance Cache 135 follows several principles, including but not limited to achieving rendering of dynamic content, robustness, and predictable performance and resource consumption. The Neural Radiance Cache 135 adapts to dynamic geometry, materials, and lighting by alternating between rendering and training. The Neural Radiance Cache 135 is also agnostic to materials and scene geometry to provide a robust solution.

[0066] Conventional neural network caches suggest training a set of local neural network caches, depending on the location of a single point light source. While lighting can change dynamically and area lighting can be approximated using a set of point lights at the expense of multiple cache queries, the geometry and materials must remain static due to the cost of the training process. In contrast, all lighting in the scene can be considered in a single query of the neural radiance cache 135 for each terminal vertex. Conventional pre-trained neural network models must generalize to novel configurations and, more challenging, content that may have never been observed before. In contrast, the neural radiance cache 135 is able to quickly adapt to changes and new content, rather than relying on pre-training and generalization, and does not require pre-computation and / or pre-training.

[0067] Fluctuations in rendering workload and memory usage lead to unstable and unpredictable frame rates, jeopardizing real-time performance. The Neural Radiance Cache 135 provides predictable performance and resource consumption, with stable runtime overhead and memory usage, both independent of scene complexity. In the worst case, rendering cost scales linearly with the number of pixels being rendered. To achieve generalization via real-time adaptation, multi-bounce lighting is efficiently estimated at the expense of tracing a single ray (or a very short path).

[0068] Figure 2A A flow chart is shown of a method 200 for generating images using a neural network radiance cache and training a neural network radiance cache suitable for implementing some embodiments of the present disclosure. Each block of the method 200 described herein comprises a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in a memory. The method may also be embodied as computer usable instructions stored on a computer storage medium. To name a few, the method may be provided by a standalone application, a service, or a hosted service (standalone or in combination with other hosted services), or a plug-in to another product. Furthermore, by way of example, with respect to Figure 1D Method 200 is described with reference to the neural network radiation cache system 150 of FIG. However, this method may additionally or alternatively be performed by any system or any combination of systems, including but not limited to the systems described herein. Furthermore, one of ordinary skill in the art will understand that any system that performs method 200 is within the scope and spirit of the embodiments of the present disclosure.

[0069] In step 205, a neural network radiance cache model processes 3D positions associated with light transport paths through the scene to generate radiance predictions (estimated radiance or reflected light) at the 3D positions. In embodiments, the light transport path begins at a camera. In embodiments, the light transport path begins at a location between the left eye and the right eye. In embodiments, the light transport path begins at an intermediate vertex in the rendering path. In embodiments, the neural network radiance cache is neural radiance cache 135. In embodiments, the 3D positions are vertices. In embodiments, a second 3D position associated with the rendered path and at which a second radiance prediction is calculated is importance sampled based on the 3D position. The 3D position and the second 3D position define a segment of the rendering path, and a direction originating at the 3D position toward the next 3D position (e.g., the second 3D position) is selected with a probability proportional to the predicted reflected light at the second 3D position. In embodiments, the radiance prediction includes only ambient occlusion. In embodiments, the neural network radiance cache model processes additional 3D positions associated with additional rendering paths from additional cameras. In embodiments, the additional 3D positions are received via the network. In one embodiment, the 3D position is generated by at least one of a rasterizer, a ray tracer, a ray stepper, a sphere tracer, and a cone tracer.

[0070] In one embodiment, the neural network radiance cache model also processes additional inputs to generate radiance predictions, the additional inputs being at least one of a material property, a view direction, or a normal vector associated with the 3D position. In an embodiment, the 3D position and the additional inputs are encoded into a higher dimensional space before processing them. In an embodiment, the 3D position is parameterized by 2D surface coordinates.

[0071] In an embodiment, the input to the neural radiance cache 135 is augmented by additional parameters related to scattered radiance: surface normal n, surface roughness r, diffuse reflectance α, and specular reflectance β. Being able to exploit such correlations, the estimated radiance provided by the neural radiance cache 135 becomes more accurate. It is easier for the neural radiance cache 135 to identify correlations between additional parameters when they are (almost) linear. This is already the case for diffuse and specular reflectance. However, the quantities x, ω, n, and r have highly nonlinear relationships with scattered radiance. For these quantities, a well-chosen encoding to a higher dimensional space can make the relationships more linear and, thereby, the estimated radiance significantly more accurate.

[0072] Extra dimensions do not come for free because they increase the required memory traffic and the cost of the first layer of the neural radiation cache 135. Therefore, in an embodiment, the quantities x, ω, n, and r are encoded using as few extra dimensions as possible while still benefiting from linearization. To this end, single-blob encoding works well when the scale of the nonlinearity is of approximately the same order of magnitude as the size of the blob. Single-blob encoding is described by Thomas Müller, Brian McWilliams, Fabric Rousselle, Markus Gross, and Jan Novák in "Neural Importance Sampling," ACM Trans. Graph. 38, 5, article 145 (October 2019), page 19. The encoding is a good fit for ω, n, and r because small changes in these parameters do not typically change the scattered radiation much. In an embodiment, ω, n, and r are encoded using a very small number (e.g., k=4) of evenly spaced blobs.

[0073] However, small changes in position x can give rise to large changes in the scattered radiation, for example, along shadows and geometric boundaries or in outdoor environments that are much larger than the viewing cone. Therefore, single spot coding is not suitable for robustly encoding position in several dimensions. Instead, frequency coding can be used, which uses a geometric hierarchy of periodic functions to represent a high dynamic range of values in several encoded dimensions. Frequency coding is described by Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng in "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" at ECCV 2020. In an embodiment, frequency coding is used to represent 3D position. In an embodiment, 12 sinusoidal functions are used, each with a frequency of 2 d , d∈{0,…,11}. To save dimensionality, the cosine term of the original method can be omitted without compromising the approximation quality of the neural radiance cache 135. In summary, the input to the neural radiance cache 135 can include the concatenation of the frequency encoding position x, the single spot encoding parameters ω, n, and r, and the diffuse and specular reflectances α and β as is. In an embodiment, the resulting total number of input dimensions to the neural radiance cache 135 is 62.

[0074] At step 210, an image of the scene corresponding to at least one camera is generated using the radiosity prediction. In an embodiment, the radiosity prediction is multiplied by at least one of diffuse reflectance, specular reflectance, normal vector, roughness, albedo, BRDF, and phase function of material properties associated with the 3D location. In an embodiment, the generated image is used to train, test, or certify a neural network employed in a machine, robot, or autonomous vehicle.

[0075] In an embodiment, instead of or in addition to generating an image of a scene, a radiation prediction is used to generate measurements. In an embodiment, the measurements are calculated by evaluating a rendering equation using the radiation prediction. In an embodiment, the measurement value can be calculated as a function, such as the average value of the space occupied by pixels when creating an image. In an embodiment, the measurement is the result of a point query (e.g., how much light from direction y reaches position x). The measurement can be used for non-visible wavelengths of light, to create light probes (such as spherical images), to simulate light sensors in robots, and so on. The measurement can also be used to generate images.

[0076] In step 215, an extended optical transmission path terminating at the terminal 3D position is generated. In an embodiment, the extended optical transmission path is generated by extending the optical transmission path from the 3D position. In an embodiment, the terminal 3D position is not included in the optical transmission path. In an embodiment, the optical transmission path is extended by at least one additional segment to generate the extended optical transmission path, and the extended optical transmission path includes at least one additional segment compared to the optical transmission path. In an embodiment, the extended optical transmission path is a separate optical transmission path that does not include the 3D position. In an embodiment, the updated weight is also based on a second radiation prediction propagating to the 3D position. In an embodiment, the extended optical transmission path does not extend from the optical transmission path and does not include the 3D position.

[0077] In one embodiment, the light transport path and the additional light transport path are traced to generate a light transport path for each pixel of the image, and a subset of rendering paths is selected from the rendering path and the additional rendering path, wherein the rendering path is extended. In one embodiment, the subset of rendering paths is selected according to at least one of a uniform distribution and a non-uniform distribution.

[0078] At step 220, the neural network radiance cache model processes the terminal 3D position to generate a second radiance prediction at the terminal 3D position. In an embodiment, the neural radiance cache model 135 processes the terminal 3D position to generate the second radiance prediction. In an embodiment, the light source is evaluated at the terminal 3D position to generate the second radiance prediction. At step 225, the weights of the neural network radiance cache model are updated based on the second radiance prediction. In an embodiment, the weights are also updated based on the second radiance prediction propagated to the 3D position. In an embodiment, a filtered version of the weights is used to generate the radiance prediction at the 3D position. In an embodiment, the filter is a moving average over the sequence of weight updates.

[0079] In an embodiment, the neural network radiance cache model processes a second 3D position associated with a second rendering pass through the scene according to the updated weights to produce a second radiance prediction at the second 3D position, and generates a second image of the scene using the radiance prediction. In an embodiment, at least one of a camera, lighting, geometry, and materials is changed for the second rendering pass compared to the rendered pass.

[0080] In an embodiment, the neural network radiance cache model is trained on a server or in a data center, and the image is streamed to the user device. In one embodiment, one or more of steps 205, 210, 215, 220, and 225 are performed within a cloud computing environment. In one embodiment, one or more of steps 205, 210, 215, 220, and 225 are performed on a server or in a data center, and the image is streamed to the user device. In one embodiment, one or more of steps 205, 210, 215, 220, and 225 are performed on a virtual machine that includes a portion of a GPU.

[0081] Figure 2BPortions of images 230 and 235 generated as the number of training cycles increases according to an embodiment are shown. Both images 230 and 235 are rendered by training the neural radiance cache 135 from scratch without performing any pre-training. The number of rendered frames and training cycles increases from 1 to 1024. To illustrate the training behavior, the neural radiance cache 135 is directly visualized at the first non-specular vertex of each pixel in image 230. Image 235 is rendered using the neural radiance cache 135 at the end of a short path, the estimation error is filtered, and the cached values converge to generate a satisfactory quality portion of image 235 in as few as 8 frames (approximately 70 ms). After the first 64 frames (approximately 0.5 seconds), the overall color is correct and only subtle high-frequency artifacts remain. When the neural radiance cache 135 is used to estimate the radiance at the terminal vertex determined based on heuristics, the high-frequency artifacts are hidden behind the path detours, and the neural radiance cache 135 can be used with intervening training after only 8 frames have been rendered. Adaptation of the neural radiance cache 135 via online training is fast enough to be used for online adaptation of animation content.

[0082] As previously described, rendering paths can be terminated based on a heuristic based on the area spread of the path vertices. A path can be terminated when the spread or scattering of the approximate reflected light at the vertex becomes large enough to blur out the small-scale inaccuracies of the neural radiance cache 135. Different surface materials correspond to different scattering interactions. Therefore, changing the material of an object in the scene can result in a shorter or longer rendering path. As done for rendering paths, longer training paths can be truncated once the area spread of the suffix is large enough. However, for training paths, the terminal vertex of the rendering path is treated as the primary vertex.

[0083] Figure 2C Path scattering interactions of different materials according to an embodiment are shown. Specular material 240, such as a mirror, produces a reflected light contribution with low scattering interaction, shown as a cone with a small radius r 242 for a particular length d 244 of the segment or ray. Glossy material 245, such as a smooth plastic ball, produces a reflected light contribution with a medium scattering interaction with a larger radius 246. Diffuse material 247, such as wood particles, produces a reflected light contribution with a high scattering interaction with a large radius 248.

[0084] Figure 2DPath termination according to scattering interactions, according to an embodiment, is illustrated. The camera vertex is indexed as y0 and the primary vertex is indexed as y1. Once the spread or scattering interaction (shown as the radius of the cone base at the primary vertex) becomes large enough, the path can terminate. For example, the size of the footprint of the path a(y1y2) at the terminal vertex y2 of the rendered path can be compared to the size of the directly visible surface in the image plane a0. Longer training paths can be terminated by the same heuristic applied to the vertices of the suffix, i.e., comparing a(y2y3y4) to a0.

[0085] As described in the technical report “A customized designed Density Estimation Method for Light Transport,” by Philippe Bekaert et al., 2003.

[0086] where p is the BSDF sampling PDF, and θ I Yes i with x i The angle between the surface normals at .

[0087] To terminate the path, expand the subpath a(x1…x n ) is compared with the expansion at that principal vertex as observed from that camera (assuming a spherical image plane and ignoring constant factors):

[0088]

[0089] That is, if a(x1…x n )>c·a0, the path is terminated, where c is a hyperparameter that trades variance (longer paths) for bias and speed (shorter paths). In the embodiment, c=0.01 produces satisfactory results. Finally, if the rendering path is selected to become the training path, the heuristic method will be used again, and when a(x n …x m )>c·a0, this time the training suffix is terminated. Figure 2D The heuristic is illustrated in , where the rendering path ends at vertex n=2 and the training suffix ends at n=4.

[0090] Figure 2EA flow chart of a method 270 is shown for terminating a rendering or training path into a neural radiation cache suitable for implementing some embodiments of the present disclosure. Each block of the method 270 described herein includes a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in a memory. The method may also be embodied as computer-usable instructions stored on a computer storage medium. The method may be provided by a standalone application, a service, or a hosted service (standalone or in combination with other hosted services), or a plug-in to another product, to name a few. Furthermore, by way of example, with respect to Figure 1D Method 270 is described with reference to the neural network radiation cache system 150 of FIG. However, this method may additionally or alternatively be performed by any system or any combination of systems, including but not limited to the systems described herein. Furthermore, one of ordinary skill in the art will understand that any system that performs method 270 is within the scope and spirit of the embodiments of the present disclosure.

[0091] In step 275, a first segment of a rendering (light transport) path or a training suffix of a training (light transport) path is generated. The first segment of the rendering path may be defined by a vertex (3D position) at the camera and a first vertex. The first segment of the training suffix may be defined by a terminal vertex and a first vertex of the rendering path. In step 285, a determination is made as to whether to terminate the (rendering or training) path. In an embodiment, the path may be terminated based on a heuristic based on area expansion of the path vertices. In an embodiment, the path terminates at the 3D position based on a first size of a first footprint of the path at the 3D position compared to a second size of a second footprint of the path at the visible 3D position. In an embodiment, the extended path terminates at the terminal vertex 3D position based on a third size of a third footprint of the extended path at the terminal vertex 3D position compared to the second size.

[0092] If the path is terminated at step 285, the first vertex is the terminal 3D position and the method 270 proceeds to step 290. Otherwise, at step 280, the path is extended another segment before returning to step 285. At step 290, the neural radiance cache 135 processes the terminal 3D position to generate a terminal radiance prediction (estimated radiance or reflected light) at the terminal 3D position. At step 295, the terminal radiance prediction is propagated back to each 3D position that defines a segment of the path.

[0093] In an embodiment, for real-time performance, a rendering budget of 16.6 milliseconds is used to achieve a frame rate of 60 frames per second. The rendering budget includes tracing of the path, shading at each vertex, and querying and updating the neural radiance cache 135. In practice, the rendering budget only allows a few milliseconds to handle the neural radiance cache 135 overhead. Therefore, the neural radiance cache 135 is trained by extending a subset of the rendering paths to generate training paths. Reusing rendering paths reduces training time, and relying on the neural radiance cache 135 to estimate the radiance of the training paths leads to rapid convergence of weights and also achieves optimization. In contrast to conventional neural network techniques, no pre-training of the neural radiance cache is required.

[0094] Shorter paths improve performance, and shorter paths also reduce noise in the rendered image because the predicted radiance has been aggregated (averaged) at the termination point. Even though shorter paths increase the systematic error, the inaccuracies of the Neural Radiance Cache are hidden behind fewer correctly simulated path segments. The Neural Network Radiance Cache System 150 handles fully dynamic scenes without assumptions about cameras, lighting, geometry, and materials. The streamlined Neural Network Radiance Cache System 150 architecture can be designed to maximize the quality-cost tradeoff when rendering fully dynamic scenes. To take full advantage of these opportunities, a fully fused neural network implementation of the Neural Radiance Cache 135 is customized for modern GPUs.

[0095] Fully fused neural network

[0096] The neural network cache 135 can be implemented as a fully connected neural network in a GPU programming language to fully utilize the GPU's memory hierarchy. In the context of the following description, a fully connected neural network is referred to as a "fully fused" neural network, which is configured for execution by a processor by limiting slow global memory accesses to reads and writes to inputs to and outputs from the fully connected neural network. Although the neural network cache 135 is described in the context of a GPU, the neural network cache 135 can be implemented using other processors. Performance can be significantly improved by tailoring the implementation to the specific characteristics of the processor (e.g., memory bandwidth, register storage capacity, parallel processing width, etc.).

[0097] The computational cost of a fully connected neural network scales quadratically with its width, while its memory traffic scales linearly. Modern GPUs have much greater computational throughput than they have memory bandwidth, but this means that for narrow neural networks (such as the Neural Radiant Cache 135), linear memory traffic is the bottleneck. Therefore, the key to improving performance is to minimize the amount of communication to slow "global" memory (VRAM and high-level cache) and make full use of fast on-chip memory (low-level cache, "shared" memory, and registers).

[0098] A fully fused approach can be used to implement an entire neural network as a single GPU kernel, designed so that only slow global memory accesses are reading and writing inputs and outputs to / from the neural network. Intermediate values generated by each layer of the neural network are stored in shared memory or registers. Furthermore, as opposed to building a kernel from outside an existing framework, implementing a kernel from scratch allows the implementation to be specifically tailored to the neural network architecture and the GPU on which it is executed.

[0099] In an embodiment, the fully fused neural network architecture includes seven fully connected layers and five hidden layers, each hidden layer including 64 neurons with a rectified linear unit (ReLU) activation function. The output layer reduces the 64 dimensions to three RGB values. In an embodiment, none of these layers has a bias vector because the bias does not result in any significant quality benefit and omitting the bias makes the fully fused implementation simpler and more efficient. Note that the neural network is shallow enough so that vanishing gradients are not a problem. Therefore, there is no need to use residual layers with skip links to assist in training.

[0100] Figure 3A is a conceptual diagram 301 illustrating batch neural network evaluation suitable for implementing some embodiments of the present disclosure. The evaluation is performed for a large number of inputs (e.g., N≈2 for a 1920×1080 pixel frame) by performing weight matrix multiplication and element-wise application of the activation function at each layer, except that there may be no activation function in the output layer. 21 ) evaluates a multilayer perceptron (MLP) of L fully connected layers of M neurons. In an embodiment, the batch input includes multiple terminal vertices of the rendering or training path. The intermediate results (neural activations) of each layer are then processed by ReLU. The processing workload of each batch is parallelized into blocks, each of which is processed by a thread block. In an embodiment, the batch is split into blocks of 128 elements wide. If the MLP is narrow (e.g., M hidden =M in = 64 neurons wide), the weight matrix fits in the on-chip registers, and the middle 64×128 neuron activations fit in the on-chip shared memory. For a constant batch size, the computational cost scales with the square of the number of neurons (e.g., O(M 2)), and the memory traffic scales linearly with the number of neurons (e.g., O(M)).

[0101] Figure 3B The distribution of batches across thread blocks 303 suitable for implementing some embodiments of the present disclosure is shown. Using Compute Unified Device Architecture (CUDA) terminology: a given batch of input vectors is split into block-column segments, each processed by a single thread block. The thread blocks independently alternate between weight matrix multiplication and element-wise application of the activation function. By making the thread blocks small enough so that all interneuron activations fit into on-chip shared memory, traffic to global (off-chip) memory is minimized. This is a key advantage of the fully converged neural network approach.

[0102] Figure 3C 306 is shown, which is suitable for implementing some embodiments of the present disclosure. The matrix multiplication performed by each thread block converts the i-th layer H i Transformed into the pre-activated next layer H' i+1 . Matrix multiplication is cut into blocks of 16×16 elements to match the size of the hardware accelerated half-precision matrix multiplier. A reduction in global memory traffic can be achieved by minimizing the number of thread blocks by making each thread block fit as much as possible into shared memory. In an embodiment, using a 64-neuron wide neural network, the sweet spot is met when each thread block processes 128 columns of the batch. Each thread block thus computes the matrix product of a 64×64 weight matrix with a 64×128 block of data.

[0103] Each warp of a thread block computes a single block row (e.g. Figure 3C In an embodiment, by first converting the corresponding 16×64 stripe weights from W i load them into registers and then multiply them by H for all 64×16 block columns i , each warp of the thread block computes H' for a 16×128 block row i+1 Because the weights fit in the warp's registers, they can be reused for H' computed by the warp. i+1 Thus, in an embodiment, each thread block loads a different block row of the weight matrix from global memory exactly once, and only in H memory located in fast shared memory. i Perform multiple passes on the same process.

[0104] For training, the forward and backward passes allow for the same matrix multiplication structure as during inference (for the short rendering path). However, the training pass may require additional global memory traffic because intermediate activations and associated gradients must be written out for backpropagation. In addition, additional matrix multiplications are necessary to convert the results of backpropagation into gradients of the weight matrix.

[0105] Figure 3D A flow chart of a method 340 for batch neural network evaluation suitable for implementing some embodiments of the present disclosure is shown. Each block of the method 340 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, different functions can be implemented by a processor executing instructions stored in a memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. To name a few, the method can be provided by a standalone application, a service, or a hosted service (standalone or in combination with other hosted services), or a plug-in to another product. Furthermore, by way of example, Figure 1D Method 340 is described with reference to the neural network radiation cache system 150 of FIG. However, this method may additionally or alternatively be performed by any system or any combination of systems, including but not limited to the systems described herein. Furthermore, one of ordinary skill in the art will understand that any system that performs method 340 is within the scope and spirit of the embodiments of the present disclosure.

[0106] At step 345, the input to the neural network model is divided into a plurality of portions, each of which is sized for execution in a processing core by a single thread block. In an embodiment, the neural network model is a neural radiance cache 135. In an embodiment, the processing core is Figure 4C Processing core 442 is shown in FIG.

[0107] At step 355, the weights of the neural network model are loaded into a register file within the processor to process the input. In an embodiment, the register file is Figure 4C 455. At step 360, a set of processing cores within the processor independently processes the parts in parallel, wherein the weights of the first layer of the neural network model are applied to each of the parts to generate an intermediate result (neural activation) for each part.

[0108] At step 365, the intermediate results of each portion are stored in a memory shared among the set of processing cores. In an embodiment, the memory is Figure 4C The shared memory / L1 cache 465 shown in FIG. In step 375, each partial intermediate result is processed by a subsequent layer of the neural network model to produce a subsequent intermediate result until the last layer of the neural network model generates an output.

[0109] Method 340 can be used to implement a real-time neural radiance cache technique for path-traced global illumination. The technique is capable of robustly handling dynamic content while providing predictable performance and memory and computational resource consumption. The neural radiance cache 135 achieves high rendering quality and can also generalize via online adaptation. The neural radiance cache 135 can be implemented as a fully fused neural network using fixed-function hardware (GPU tensor cores). In an embodiment, the neural radiance cache 135 trades more computation for expensive memory accesses.

[0110] Parallel processing architecture

[0111] Figure 3E A parallel processing unit (PPU) 400 is shown, according to an embodiment. The PPU 300 may be used to implement the neural network radiance cache system 150. In an embodiment, a processor (such as the PPU 300) may be configured to implement a neural network model, such as the neural radiance cache 135. In an embodiment, the method 340 may be implemented by one or more processing cores included in the PPU 300. The neural network model may be implemented as software instructions executed by the processor, or in other embodiments, the processor may include a matrix of hardware elements configured to process a set of inputs (e.g., electrical signals representing values) to generate a set of outputs, which may represent activations of the neural network model. In still other embodiments, the neural network model may be implemented as a combination of software instructions and processing performed by the matrix of hardware elements. Implementing the neural network model may include determining a set of parameters for the neural network model by, for example, supervised or unsupervised training of the neural network model and (or in the alternative) performing inference using the parameter set to process a novel input set.

[0112] In an embodiment, the PPU 300 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 300 is a latency-hiding architecture designed for processing many threads in parallel. A thread (i.e., an execution thread) is an instance of an instruction set configured to be executed by the PPU 300. In one embodiment, the PPU 300 is a graphics processing unit (GPU) that is configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 300 can be used to perform general-purpose computations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that the processor is described for illustrative purposes only and any processor may be used in addition to and / or in place of the processor.

[0113] One or more PPUs 300 can be configured to accelerate thousands of high-performance computing (HPC), data center, cloud computing, and machine learning applications. The PPU 300 can be configured to accelerate numerous deep learning systems and applications for autonomous vehicles, simulations, computational graphics such as ray or path tracing, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0114] As shown in Figure 3, the PPU 300 includes an input / output (I / O) unit 305, a front-end unit 315, a scheduler unit 320, a work distribution unit 325, a hub 330, a crossbar switch (Xbar) 370, one or more general processing clusters (GPCs) 350, and one or more memory partitioning units 380. The PPU 300 can be connected to a host processor or other PPUs 300 via one or more high-speed NVLink 310 interconnects. The PPU 300 can be connected to a host processor or other peripheral devices via interconnect 302. The PPU 300 can also be connected to a local memory 304 including multiple memory devices. In one embodiment, the local memory can include multiple dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem, in which multiple DRAM dies are stacked within each device.

[0115] The NVLink 310 interconnect enables the system to scale and include one or more PPUs 300 in conjunction with one or more CPUs, supporting cache coherency between the PPU 300 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 310 through the hub 330 to or from other units of the PPU 300, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 5A NVLink 310 is described in more detail.

[0116] I / O unit 305 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 302. I / O unit 305 can communicate with the host processor directly via interconnect 302, or through one or more intermediary devices (such as a memory bridge). In one embodiment, I / O unit 305 can communicate with one or more other processors (e.g., one or more PPUs 300) via interconnect 302. In one embodiment, I / O unit 305 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating over a PCIe bus, and interconnect 302 is a PCIe bus. In alternative embodiments, I / O unit 305 can implement other types of known interfaces for communicating with external devices.

[0117] I / O unit 305 decodes data packets received via interconnect 302. In one embodiment, the data packets represent commands configured to cause PPU 300 to perform various operations. I / O unit 305 sends the decoded commands to various other units of PPU 300 as specified by the commands. For example, some commands may be sent to front-end unit 315. Other commands may be sent to hub 330 or other units of PPU 300, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 305 is configured to route communications between and among the various logical units of PPU 300.

[0118] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 300 for processing. The workload may include many instructions and data to be processed by those instructions. A buffer is an area of memory that is accessible (e.g., read / write) by both the host processor and the PPU 300. For example, the I / O unit 305 may be configured to access a buffer in system memory connected to the interconnect 302 via a memory request transmitted over the interconnect 302. In one embodiment, the host processor writes a command stream into the buffer and then sends a pointer to the start of the command stream to the PPU 300. The front end unit 315 receives pointers to one or more command streams. The front end unit 315 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 300.

[0119] Front-end unit 315 is coupled to scheduler unit 320, which configures various GPCs 350 to process tasks defined by one or more streams. Scheduler unit 320 is configured to track state information associated with the various tasks managed by scheduler unit 320. The state may indicate which GPC 350 a task is assigned to, whether the task is active or inactive, the priority associated with the task, and the like. Scheduler unit 320 manages the execution of multiple tasks on one or more GPCs 350.

[0120] Scheduler unit 320 is coupled to work distribution unit 325, which is configured to dispatch tasks for execution on GPCs 350. Work distribution unit 325 can track a number of scheduled tasks received from scheduler unit 320. In one embodiment, work distribution unit 325 manages a pending task pool and an active task pool for each GPC 350. The pending task pool can include a number of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular GPC 350. The active task pool can include a number of time slots (e.g., 4 time slots) for tasks being actively processed by GPC 350. When a GPC 350 completes execution of a task, the task is evicted from the active task pool of GPC 350, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 350. If an active task on GPC 350 has become idle, such as while waiting for a data dependency to be resolved, the active task may be evicted from GPC 350 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 350 .

[0121] Work distribution unit 325 communicates with one or more GPCs 350 via XBar (crossbar) 370. XBar 370 is an interconnect network that couples many units of PPU 300 to other units of PPU 300. For example, XBar 370 can be configured to couple work distribution unit 325 to a specific GPC 350. Although not explicitly shown, one or more other units of PPU 300 can also be connected to XBar 370 via hub 330.

[0122] Tasks are managed by the scheduler unit 320 and dispatched to the GPCs 350 by the work distribution unit 325. The GPCs 350 are configured to process tasks and generate results. The results can be consumed by other tasks within the GPC 350, routed to a different GPC 350 via the XBar 370, or stored in the memory 304. The results can be written to the memory 304 via the memory partition unit 380, which implements a memory interface for reading data from and writing data to the memory 304. The results can be sent to another PPU 300 or CPU via the NVLink 310. In one embodiment, the PPU 300 includes a number U of memory partition units 380, which is equal to the number of independent and distinct memory 304 memory devices coupled to the PPU 300. Figure 4B The memory partition unit 380 is described in more detail.

[0123] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 300. In one embodiment, multiple computing applications are executed simultaneously by the PPU 300, and the PPU 300 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 300. The driver kernel outputs the tasks to one or more streams being processed by the PPU 300. Each task can include one or more related groups of threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can execute in parallel. Collaborating threads can refer to multiple threads that include instructions to perform tasks and that can exchange data through shared memory. In combination Figure 4C Describes threads and cooperative threads in more detail.

[0124] Figure 4A According to one embodiment, Figure 3E PPU 300 GPC 350. Figure 4A As shown, each GPC 350 includes multiple hardware units for processing tasks. In one embodiment, each GPC 350 includes a pipeline manager 410, a pre-raster operation unit (PROP) 415, a raster engine 425, a work distribution crossbar (WDX) 480, a memory management unit (MMU) 490, and one or more data processing clusters (DPCs) 420. It should be understood that Figure 4A The GPC350 may include instead Figure 4A Other hardware units or other than the units shown in Figure 4A Other hardware units besides those shown in .

[0125] In one embodiment, the operation of GPC 350 is controlled by pipeline manager 410. Pipeline manager 410 manages the configuration of one or more DPCs 420 for processing tasks assigned to GPC 350. In one embodiment, pipeline manager 410 can configure at least one of one or more DPCs 420 to implement at least a portion of a graphics rendering pipeline. For example, DPC 420 can be configured to execute vertex shading programs on programmable streaming multiprocessors (SMs) 440. Pipeline manager 410 can also be configured to route packets received from work distribution unit 325 to appropriate logic units within GPC 350. For example, some packets can be routed to fixed-function hardware units within PROP 415 and / or raster engine 425, while other packets can be routed to DPC 420 for processing by primitive engine 435 or SM 440. In one embodiment, pipeline manager 410 can configure at least one of one or more DPCs 420 to implement a neural network model and / or a compute pipeline.

[0126] PROP unit 415 is configured to route data generated by raster engine 425 and DPC 420 to the raster operations (ROP) unit, in conjunction with Figure 4B The PROP unit 415 may also be configured to perform optimizations for color blending, organize pixel data, perform address translation, and the like.

[0127] The raster engine 425 includes several fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster engine 425 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile aggregation engine. The setup engine receives the transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are sent to the coarse raster engine to generate coverage information for the primitives (e.g., the x, y coverage mask of the tile). The output of the coarse raster engine is sent to the culling engine, where fragments associated with primitives that fail the z-test are culled, and to the clipping engine, where fragments outside the viewing frustum are clipped. Those fragments remaining after clipping and culling can be passed to the fine raster engine to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. The output of the raster engine 425 includes, for example, fragments to be processed by the fragment shader implemented in the DPC 420.

[0128] Each DPC 420 included in the GPC 350 includes an M pipeline controller (MPC) 430, a primitive engine 435, and one or more SMs 440. The MPC 430 controls the operation of the DPC 420 and routes packets received from the pipeline manager 410 to appropriate units in the DPC 420. For example, packets associated with vertices may be routed to the primitive engine 435, which is configured to fetch vertex attributes associated with the vertices from the memory 304. Conversely, packets associated with shading programs may be sent to the SM 440.

[0129] SM 440 includes a programmable streaming processor configured to process tasks represented by multiple threads. Each SM 440 is multi-threaded and configured to simultaneously execute multiple threads (e.g., 32 threads) from a particular thread group. In one embodiment, SM 440 implements a SIMD (single instruction, multiple data) architecture, in which each thread in a thread group (e.g., a warp) is configured to process a different data set based on the same instruction set. All threads in a thread group execute the same instructions. In another embodiment, SM 440 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group is configured to process a different data set based on the same instruction set, but in which individual threads in a thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within warps when threads within the warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby achieving equal concurrency between all threads within and between warps. When execution state is maintained for each individual thread, threads executing the same instruction can be converged and executed in parallel for maximum efficiency. Figure 4C Describe SM440 in more detail.

[0130] MMU 490 provides an interface between GPC 350 and memory partition unit 380. MMU 490 can provide virtual to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, MMU 490 provides one or more translation lookaside buffers (TLBs) for performing translations from virtual addresses to physical addresses in memory 304.

[0131] Figure 4B According to one embodiment, Figure 3E The memory partition unit 380 of the PPU 300. Figure 4BAs shown, the memory partition unit 380 includes a raster operations (ROP) unit 450, a level 2 (L2) cache 460, and a memory interface 470. The memory interface 470 is coupled to the memory 304. The memory interface 470 can implement a 32-, 64-, 128-, or 1024-bit data bus for high-speed data transfer. In one embodiment, the PPU 300 incorporates U memory interfaces 470, one for each pair of memory partition units 380, where each pair of memory partition units 380 is connected to a corresponding memory device of the memory 304. For example, the PPU 300 can be connected to up to Y memory devices, such as a high-bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0132] In one embodiment, memory interface 470 implements an HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 300, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, where the HBM2 stack includes two 128-bit channels per die, for a total of 8 channels and a data bus width of 1024 bits.

[0133] In one embodiment, memory 304 supports single-error correction, double-error detection (SECDED) error correction code (ECC) to protect data. ECC provides increased reliability for computing applications sensitive to data corruption. Reliability is particularly important in large cluster computing environments where PPU 300 processes very large data sets and / or long-running applications.

[0134] In one embodiment, the PPU 300 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 380 supports unified memory to provide a single, unified virtual address space for the CPU and PPU 300 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 300 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 300 where the pages are accessed more frequently. In one embodiment, NVLink 310 supports address translation services that allow the PPU 300 to directly access the CPU's page tables and provide full access to the CPU's memory by the PPU 300.

[0135] In one embodiment, the copy engine transfers data between multiple PPUs 300 or between a PPU 300 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 380 can then service the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, multiple copy engines operate on fixed memory (e.g., non-pageable) between multiple processors, which significantly reduces the available memory. Due to hardware page faults, addresses can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0136] Data from memory 304 or other system memory can be retrieved by memory partition unit 380 and stored in L2 cache 460, which is located on-chip and shared between various GPCs 350. As shown, each memory partition unit 380 includes a portion of L2 cache 460 associated with the corresponding memory 304. Lower-level caches can then be implemented in multiple units within a GPC 350. For example, each SM 440 can implement a level 1 (L1) cache. The L1 cache is a dedicated memory dedicated to a particular SM 440. Data from the L2 cache 460 can be retrieved and stored in each L1 cache for processing in the functional units of the SM 440. The L2 cache 460 is coupled to the memory interface 470 and the XBar 370.

[0137] The ROP unit 450 performs graphics raster operations related to pixel color such as color compression, pixel blending, etc. The ROP unit 450 also implements depth testing in conjunction with the raster engine 425, receiving the depth of the sample position associated with the pixel fragment from the culling engine of the raster engine 425. The depth of the sample position associated with the fragment is tested against the corresponding depth in the depth buffer. If the fragment passes the depth test for the sample position, the ROP unit 450 updates the depth buffer and sends the result of the depth test to the raster engine 425. It will be understood that the number of memory partition units 380 can be different than the number of GPCs 350, and therefore each ROP unit 450 can be coupled to each GPC 350. The ROP unit 450 tracks packets received from different GPCs 350 and determines to which GPC 350 the results generated by the ROP unit 450 are routed via the Xbar 370. Although in Figure 4B In the embodiment shown, ROP unit 450 is included within memory partition unit 380, but in other embodiments, ROP unit 450 may be external to memory partition unit 380. For example, ROP unit 450 may reside in GPC 350 or another unit.

[0138] Figure 4C According to one embodiment, Figure 4A Streaming multiprocessor 440. Figure 4C As shown, SM 440 includes an instruction cache 405, one or more (K) scheduler units 445, a register file 455, one or more processing cores 442, one or more special function units (SFUs) 452, one or more load / store units (LSUs) 454, an interconnection network 458, and a shared memory / L1 cache 465.

[0139] As described above, the work distribution unit 325 schedules tasks for execution on the GPCs 350 of the PPU 300. Tasks are assigned to specific DPCs 420 within the GPC 350 and, if the task is associated with a shader program, may be assigned to an SM 440. The scheduler unit 445 receives tasks from the work distribution unit 325 and manages the scheduling of instructions for one or more thread blocks assigned to the SM 440. The scheduler unit 445 schedules thread blocks for execution as warps of parallel threads, where each thread block is assigned at least one warp. In one embodiment, each warp executes 32 threads. The scheduler unit 445 can manage multiple different thread blocks, assign warps to different thread blocks, and then dispatch instructions from multiple different cooperative groups to various functional units (i.e., cores 442, SFUs 452, and LSUs 454) during each clock cycle.

[0140] Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling the expression of richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. Conventional programming models provide a single, simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than the thread block granularity and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0141] Cooperative Groups enable programmers to explicitly define thread groups at sub-block (e.g., as small as a single thread) and multi-block granularity and perform collective operations, such as synchronization, on threads in a cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative Group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks.

[0142] The dispatch unit 448 is configured to deliver instructions to one or more functional units. In this embodiment, the scheduler unit 445 includes two dispatch units 448, which enable scheduling two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 445 may include a single dispatch unit 448 or additional dispatch units 448.

[0143] Each SM 440 includes a register file 455 that provides a set of registers for the functional units of SM 440. In one embodiment, register file 455 is divided between each functional unit so that each functional unit is allocated a dedicated portion of register file 455. In another embodiment, register file 455 is divided between the different warps executed by SM 440. Register file 455 provides temporary storage for operands connected to the data paths of the functional units.

[0144] Each SM 440 includes L processing cores 442. In one embodiment, SM 440 includes a large number (e.g., 128, etc.) of different processing cores 442. Each core 442 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (FLU) and an integer arithmetic logic unit (ALU). In one embodiment, the FLU implements the IEEE 754-2008 standard for floating-point operations. In one embodiment, cores 442 include 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0145] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 442. Specifically, the tensor cores are configured to perform deep learning matrix operations, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0146] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be 16-bit floating point or 32-bit floating point matrices. The tensor cores operate on 16-bit floating point input data as well as 32-bit floating point accumulations. The 16-bit floating point multiplication requires 64 operations to produce a full-precision product, which is then accumulated using 32-bit floating point additions with other intermediate products of the 4×4×4 matrix multiplication. In practice, tensor cores are used to perform larger two-dimensional or higher-dimensional matrix operations built from these smaller elements. APIs (such as the CUDA 9 C++ API) expose specialized matrix load, matrix multiplication and accumulation, and matrix store operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that the 16×16 size matrix spans all 32 threads of the warp.

[0147] Each SM 440 also includes M SFUs 452 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 452 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 452 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texels) from memory 304 and sample the texture map to generate sampled texture values for use in shader programs executed by the SM 440. In one embodiment, the texture map is stored in shared memory / L1 cache 465. The texture unit implements texture operations, such as filtering operations using mip maps (i.e., texture maps at different levels of detail). In one embodiment, each SM 440 includes two texture units.

[0148] Each SM 440 also includes N LSUs 454, which implement load and store operations between the shared memory / L1 cache 465 and the register file 455. Each SM 440 includes an interconnect network 458 that connects each functional unit to the register file 455 and the shared memory / L1 cache 465. In one embodiment, the interconnect network 458 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 455 and a memory location in the shared memory / L1 cache 465.

[0149] Shared memory / L1 cache 465 is an on-chip memory array that allows data storage and communication between SM 440 and primitive engine 435, as well as between threads in SM 440. In one embodiment, shared memory / L1 cache 465 includes 128KB of storage capacity and is in the path from SM 440 to memory partition unit 380. Shared memory / L1 cache 465 can be used to cache reads and writes. One or more of shared memory / L1 cache 465, L2 cache 460, and memory 304 are backing stores.

[0150] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used by programs as a cache that does not utilize the shared memory. For example, if the shared memory is configured to use half of its capacity, texture and load / store operations can use the remaining capacity. Integration within shared memory / L1 cache 465 enables shared memory / L1 cache 465 to function as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0151] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 3E The fixed-function graphics processing unit is shown bypassed, creating a simpler programming model. In a general-purpose parallel computing configuration, work distribution unit 325 assigns and distributes thread blocks directly to DPC 420. The threads in the block execute the same program, using unique thread IDs in computations to ensure each thread produces unique results, using SM 440 to execute the program and perform computations, using shared memory / L1 cache 465 to communicate between threads, and using LSU 454 to read and write global memory through shared memory / L1 cache 465 and memory partition unit 380. When configured for general-purpose parallel computing, SM 440 can also write commands that scheduler unit 320 can use to start new work on DPC 420.

[0152] The PPU 300 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 300 is included on a single semiconductor substrate. In another embodiment, the PPU 300 is included on a system-on-chip (SoC) along with one or more other devices (such as an additional PPU 300, a memory 304, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0153] In one embodiment, PPU 300 may be included on a graphics card that includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on a desktop computer's motherboard. In another embodiment, PPU 300 may be an integrated graphics processing unit (iGPU) or parallel processor included in a chipset on the motherboard.

[0154] Exemplary Computing System

[0155] Systems with multiple GPUs and CPUs are being used across various industries as developers expose and exploit greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0156] Figure 5A is a conceptual diagram of a processing system 500 implemented using the PPU 400 of FIG. 4 according to one embodiment. The exemplary system 565 may be configured to implement Figure 2A 、 2E 3D and 3D, respectively. The processing system 500 includes a CPU 530, a switch 510, and each of the plurality of PPUs 400 and a corresponding memory 404.

[0157] NVLink 410 provides a high-speed communication link between each PPU 400. Figure 5B4 shows a specific number of NVLink 410 and interconnect 402 connections, but the number of connections connected to each PPU 400 and CPU 530 can vary. Switch 510 interfaces between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLink 410 can be located on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols that interface between various different connections and / or links.

[0158] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and the CPU 530, and the switch 510 interfaces between the interconnect 402 and each PPU 400. The PPUs 400, memory 404, and interconnect 402 may be located on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), the interconnect 402 provides one or more communication links between each PPU 400 and the CPU 530, and the switch 510 interfaces between each PPU 400 using NVLink 410 to provide one or more high-speed communication links between the PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between the PPUs 400 and the CPU 530 through the switch 510. In yet another embodiment (not shown), the interconnect 402 provides one or more communication links directly between each PPU 400. One or more NVLink 410 high-speed communication links may be implemented as a physical NVLink interconnect or as an on-chip or on-die interconnect using the same protocol as NVLink 410 .

[0159] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that emulates on-chip operation and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 525 may be implemented as a circuit board substrate, and each of the PPU 400 and / or memory 404 may be a packaged device. In one embodiment, the CPU 530, switch 510, and parallel processing module 525 are located on a single semiconductor platform.

[0160] In one embodiment, the signaling rate of each NVLink 410 is 20 to 25 Gbit / s, and each PPU 400 includes six NVLink 410 interfaces (e.g., Figure 5A As shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 Gbit / s in each direction, with six links providing 400 Gbit / s. When the CPU 530 also includes one or more NVLink 410 interfaces, the NVLink 410 can be used exclusively for Figure 5A PPU to PPU communication shown, or some combination of PPU to PPU and PPU to CPU.

[0161] In one embodiment, NVLink 410 allows direct load / store / atomic access from the CPU 530 to the memory 404 of each PPU 400. In one embodiment, NVLink 410 supports coherency operations, allowing data read from memory 404 to be stored in the cache hierarchy of the CPU 530, reducing cache access latency for the CPU 530. In one embodiment, NVLink 410 includes support for Address Translation Services (ATS), allowing the PPU 400 to directly access page tables within the CPU 530. One or more NVLinks 410 can also be configured to operate in a low-power mode.

[0162] Figure 5B An exemplary system 565 is shown in which various architectures and / or functionalities of various aforementioned embodiments may be implemented. The exemplary system 565 may be configured to implement Figure 2A 、 2E and methods 200, 270, and / or 350 shown in 3D.

[0163] As shown, a system 565 is provided that includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 can directly or indirectly couple one or more of the following devices: main memory 540, network interface 535, CPU 530, display device 545, input device 560, switch 510, and parallel processing system 525. The communication bus 575 can be implemented using any suitable protocol and can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The communication bus 575 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, HyperTransport, and / or another type of bus or link. In some embodiments, there is a direct connection between the components. For example, the CPU 530 can be directly connected to the main memory 540. Further, the CPU 530 can be directly connected to the parallel processing system 525. Where there is a direct connection or a point-to-point connection between components, the communication bus 575 may include a PCIe link for performing the connection. In these examples, a PCI bus need not be included in the system 565.

[0164] although Figure 5B The various blocks of are shown as being connected via communication bus 575, but this is not intended to be limiting and is provided for clarity only. For example, in some embodiments, presentation components (such as display device(s) 545) may be considered I / O components, such as input device(s) 560 (e.g., if the display is a touch screen). As another example, CPU 530 and / or parallel processing system 525 may include memory (e.g., main memory 540 may represent a storage device in addition to parallel processing system 525, CPU 530, and / or other components). In other words, Figure 5B The computing devices are illustrative only. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are contemplated. Figure 5B within the range of computing devices.

[0165] System 565 also includes a main memory 540. Control logic (software) and data are stored in main memory 540, which can take the form of various computer-readable media. Computer-readable media can be any available media that can be accessed by system 565. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0166] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storage of information (e.g., computer-readable instructions, data structures, program modules, and / or other data types). For example, main memory 540 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by system 565. As used herein, computer storage media does not include the signals themselves.

[0167] Computer storage media can implement computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may refer to a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example, and not limitation, computer storage media may include wired media (such as a wired network or a direct wired connection) and wireless media (such as acoustic, RF, infrared, and other wireless media). Combinations of any of the above should also be included within the scope of computer-readable media.

[0168] The computer program, when executed, enables the system 565 to perform various functions. The CPU 530 may be configured to execute at least some of the computer-readable instructions to control one or more components of the system 565 to perform one or more of the methods and / or processes described herein. The CPUs 530 may each include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of processing multiple software threads simultaneously. The CPUs 530 may include any type of processor and may include different types of processors depending on the type of system 565 implemented (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of system 565, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The system 565 may also include one or more CPUs 530 in addition to one or more microprocessors or supplementary coprocessors such as math coprocessors.

[0169] In addition to or in lieu of the CPU 530, the parallel processing module 525 can be configured to execute at least some of the computer-readable instructions to control one or more components of the system 565 to perform one or more of the methods and / or processes described herein. The parallel processing module 525 can be used by the system 565 to render graphics (e.g., 3D graphics) or perform general-purpose computations. For example, the parallel processing module 525 can be used for general-purpose computations on a GPU (GPGPU). In embodiments, the CPU(s) 530 and / or the parallel processing module 525 can perform any combination of methods, processes, and / or portions thereof, either separately or in combination.

[0170] System 565 also includes (one or more) input devices 560, parallel processing system 525, and (one or more) display devices 545. Display device 545 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. Display device 545 can receive data from other components (e.g., parallel processing system 525, CPU 530, etc.) and output data (e.g., as images, video, sound, etc.).

[0171] The network interface 535 can enable the system 565 to be logically coupled to other devices including input devices 560, display device(s) 545, and / or other components, some of which may be built into (e.g., integrated into) the system 565. Illustrative input devices 560 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. The input devices 560 can provide a natural user interface (NUI) that processes in-air gestures, voice, or other physiological input generated by the user. In some instances, the input can be transmitted to an appropriate network element for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, in-air gestures, head and eye tracking, and touch recognition associated with the display of the system 565 (as described in more detail below). The system 565 may include a depth camera for gesture detection and recognition, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof. Additionally, system 565 may include an accelerometer or gyroscope that enables detection of motion (e.g., as part of an inertial measurement unit (IMU)). In some examples, system 565 may use the output of the accelerometer or gyroscope to render immersive augmented or virtual reality.

[0172] Furthermore, the system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) (such as the Internet), a peer-to-peer network, a cable network, etc.) for communication purposes via the network interface 535. The system 565 can be included in a distributed network and / or cloud computing environment.

[0173] The network interface 535 may include one or more receivers, transmitters, and / or transceivers that enable the system 565 to communicate with other computing devices via an electronic communications network, including wired and / or wireless communications. The network interface 535 may include components and functionality that enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., via Ethernet or InfiniBand communications), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0174] The system 565 may also include an auxiliary storage device (not shown). The auxiliary storage device 610 includes, for example, a hard drive and / or a removable storage drive, which represents a floppy disk drive, a tape drive, a compact disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a known manner. The system 565 may also include a hardwired power supply, a battery power supply, or a combination thereof (not shown). The power supply can provide power to the system 565 to enable the components of the system 565 to operate.

[0175] Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 565. Alternatively, the individual modules can be individually positioned or placed in various combinations of semiconductor platforms according to user needs. Although various embodiments have been described above, it should be understood that they are presented merely as examples and not limitations. Therefore, the breadth and scope of the preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0176] Sample network environment

[0177] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be configured to: Figure 5A The processing system 500 and / or Figure 5B The exemplary system 565 may be implemented on one or more instances of the exemplary system 565 , for example, each device may include similar components, features, and / or functionality of the processing system 500 and / or the exemplary system 565 .

[0178] The multiple components of the network environment can communicate with each other via a network, which can be wired, wireless, or both. The network can include multiple networks or networks of networks. As an example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connections.

[0179] Compatible network environments may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server(s) may be implemented on any number of client devices.

[0180] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include software supporting the software layer and / or a framework for one or more applications of the application layer. The software or application may include network-based service software or applications, respectively. In an embodiment, one or more of the client devices may use web-based service software or applications (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open source software web application framework, such as a distributed file system that may be used for large-scale data processing (e.g., "big data").

[0181] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these individual functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be distributed across states, regions, countries, the world, etc.). If the connection to the user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0182] The client device(s) may include Figure 5A The example processing system 500 and / or Figure 5BAt least some of the components, features, and functionality of the exemplary system 565. By way of example and not limitation, the client device may be embodied as a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a vessel, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of the depicted devices, or any other suitable device.

[0183] Machine Learning

[0184] Deep neural networks (DNNs) developed on processors such as the PPU 400 are already being used in a variety of use cases: from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, constantly learning, getting smarter, and delivering more accurate results faster over time. A child, initially taught by an adult to correctly identify and classify various shapes, eventually becomes able to recognize shapes without any tutoring. Similarly, deep learning or neural learning systems need to be trained in object recognition and classification to become smarter and more efficient at recognizing basic objects, occluded objects, and assigning context to objects.

[0185] At the simplest level, neurons in the human brain examine the various inputs they receive, assign a level of importance to each of these inputs, and pass outputs to other neurons for processing. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs representing various features of the object it is being trained to recognize and classify, and each of these features is assigned a certain weight based on its importance in defining the object's shape.

[0186] Deep neural network (DNN) models consist of multiple layers of connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.), which can be trained with large amounts of input data to solve complex problems quickly and accurately. In one example, the first layer of a DNN model breaks down an input image of a car into its components and looks for basic patterns (such as lines and angles). The second layer assembles the lines to find higher-level patterns, such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final layers generate labels for the input image, identifying the model of a specific car brand.

[0187] Once trained, a DNN can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on check deposits at an ATM, identifying images of friends in photos, providing movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human speech in real time.

[0188] During training, data flows through the DNN in a forward propagation phase until a prediction is produced, indicating the label corresponding to the input. If the neural network does not correctly label an input, the error between the correct and predicted labels is analyzed, and weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels that input and other inputs in the training dataset. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplications and additions supported by the PPU 400. Inference is a less computationally intensive and latency-sensitive process than training, in which a trained neural network is applied to new inputs it has not seen before to perform image classification, detect emotions, identify suggestions, recognize and translate speech, and generally reason about new information.

[0189] Neural networks rely heavily on matrix math operations, and complex, multi-layer networks require significant floating-point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, the PPU 400 is a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0190] Furthermore, images generated using one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for recognizing objects and environments in the real world. Such images can include scenes of roads, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or real-world environments. Such images can be used to train, test, or certify DNNs employed in machines or robots for manipulating, handling, or modifying physical objects in the real world. Furthermore, such images can be used to train, test, or certify DNNs used in autonomous vehicles to navigate and move the vehicle through the real world. Furthermore, images generated using one or more of the techniques disclosed herein can be used to convey information to users of such machines, robots, and vehicles.

[0191] Figure 5CComponents of an exemplary system 555 that can be used to train and utilize machine learning, according to at least one embodiment, are shown. As will be discussed, various components can be provided by various combinations of computing devices and resources, or a single computing system, which can be under the control of a single entity or multiple entities. Further, various aspects can be triggered, initiated, or requested by different entities. In at least one embodiment, training of a neural network can be directed by a provider associated with a provider environment 506, while in at least one embodiment, training can be requested by a client or other user accessing the provider environment through a client device 502 or other such resource. In at least one embodiment, training data (or data to be analyzed by a trained neural network) can be provided by a provider, a user, or a third-party content provider 524. In at least one embodiment, a client device 502 can be a vehicle or object that will be navigated on behalf of a user; for example, the vehicle or object can submit requests for assistance navigating the device and / or receive instructions for assistance navigating the device.

[0192] In at least one embodiment, the request can be submitted across at least one network 504 to be received by the provider environment 506. In at least one embodiment, the client device can be any suitable electronic and / or computing device that enables a user to generate and send such a request, such as, but not limited to, a desktop computer, a laptop computer, a computer server, a smartphone, a tablet computer, a game console (portable or otherwise), a computer processor, computing logic, and a set-top box. The (one or more) networks 504 may include any suitable network for transmitting the request or other such data, such as the Internet, an intranet, an Ethernet network, a cellular network, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), an ad hoc network with direct wireless connections between peers, etc.

[0193] In at least one embodiment, a request may be received at interface layer 508, which in this example may forward data to training and inference manager 532. Training and inference manager 532 may be a system or service comprising hardware and software for managing requests and servicing data or content corresponding to the request. In at least one embodiment, training and inference manager 532 may receive a request to train a neural network and may provide the requested data to training module 512. In at least one embodiment, training module 512 may select an appropriate model or neural network to use (if not specified by the request) and may use the relevant training data to train the model. In at least one embodiment, the training data may be a batch of data stored in training data repository 514, received from client device 502, or obtained from third-party provider 524. In at least one embodiment, training module 512 may be responsible for the training data. The neural network may be any suitable network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN). Once the neural network is trained and successfully evaluated, the trained neural network may be stored in model repository 516, which may store different models or networks for users, applications, services, etc. In at least one embodiment, there may be multiple models for a single application or entity, as may be utilized based on a number of different factors.

[0194] In at least one embodiment, at a later point in time, a request for content (e.g., a path determination) or data determined or influenced at least in part by a trained neural network may be received from client device 502 (or another such device). The request may include, for example, input data to be processed using the neural network to obtain one or more inferences or other output values, classifications, or predictions, or for at least one embodiment, the input data may be received by interface layer 508 and directed to inference module 518, although a different system or service may also be used. In at least one embodiment, if not already locally stored in inference module 518, inference module 518 may obtain a suitable trained network, such as a trained deep neural network (DNN) discussed herein, from model repository 516. Inference module 518 may provide data as input to the trained network, which may then generate one or more inferences as output. This may include, for example, a classification of an instance of the input data. In at least one embodiment, the inferences may then be transmitted to client device 502 for display or other communication to the user. In at least one embodiment, user context data may also be stored in a user context data repository 522, which may include data about the user that can be used as input to the network when generating inferences or determining data to return to the user after obtaining an instance. In at least one embodiment, related data, including at least some of the input or inference data, may also be stored in a local database 534 for use in processing future requests. In at least one embodiment, a user can use account information or other information to access resources or functionality of the provider environment. In at least one embodiment, if permitted and available, user data may also be collected and used to further train the model to provide more accurate inferences for future requests. In at least one embodiment, requests may be received through a user interface to a machine learning application 526 executing on the client device 502, and results may be displayed through the same interface. The client device may include resources such as a processor 528 and memory 562 for generating requests and processing results or responses, as well as at least a data storage element 552 for storing data for the machine learning application 526.

[0195] In at least one embodiment, processor 528 (or the processor of training module 512 or inference module 518) will be a central processing unit (CPU). However, as mentioned, resources in such environments may utilize GPUs to process at least some types of requested data. With thousands of cores, GPUs (such as PPU 400) are designed to handle massively parallel workloads and have therefore become popular in deep learning for training neural networks and generating predictions. While using GPUs for offline builds has enabled faster training of larger and more complex models, generating predictions offline means that input features cannot be used at the time of request, or predictions must be generated for all permutations of features and stored in lookup tables to service real-time requests. If the deep learning framework supports CPU mode and the model is small and simple enough to execute feedforward on a CPU with reasonable latency, a service on a CPU instance can host the model. In this case, training can be done offline on the GPU, and inference can be done in real time on the CPU. If a CPU approach is not feasible, the service can run on a GPU instance. However, because GPUs have different performance and cost characteristics than CPUs, running a service that offloads runtime algorithms to a GPU may require designing it differently from a CPU-based service.

[0196] In at least one embodiment, video data may be provided from client device 502 for augmentation in provider environment 506. In at least one embodiment, video data may be processed for augmentation on client device 502. In at least one embodiment, video data may be streamed from third-party content provider 524 and augmented by third-party content provider 524, provider environment 506, or client device 502. In at least one embodiment, video data may be provided from client device 502 for use as training data in provider environment 506.

[0197] In at least one embodiment, supervised and / or unsupervised training can be performed by the client device 502 and / or the provider environment 506. In at least one embodiment, a set of training data 514 (e.g., classified or labeled data) is provided as input to serve as training data. In one embodiment, the set of training data can be used to train one or more neural networks within the control variable neural network system 100.

[0198] In at least one embodiment, the training data may include instances of at least one type of object on which the neural network is to be trained, as well as information identifying objects of that type. In at least one embodiment, the training data may include a set of images, each image including a representation of an object of one type, wherein each image also includes or is associated with a label, metadata, classification, or other piece of information identifying the object of one type represented in the corresponding image. Various other types of data may also be used as training data, and may include text data, audio data, video data, and the like. In at least one embodiment, the training data 514 is provided as training input to the training module 512. In at least one embodiment, the training module 512 may be a system or service comprising hardware and software, such as one or more computing devices executing a training application for training a neural network (or other model or algorithm, etc.). In at least one embodiment, the training module 512 receives an instruction or request indicating the type of model to be used for training. In at least one embodiment, the model may be any suitable statistical model, network, or algorithm useful for these purposes, such as an artificial neural network, a deep learning algorithm, a learning classifier, a Bayesian network, and the like. In at least one embodiment, the training module 512 can select an initial model or other untrained model from an appropriate repository 516 and train the model using the training data 514, thereby generating a trained model (e.g., a trained deep neural network) that can be used to classify similar types of data or generate other such inferences. In at least one embodiment that does not use training data, an appropriate initial model can still be selected to train the input data of each training module 512.

[0199] In at least one embodiment, the model can be trained in a variety of different ways, as may depend in part on the type of model selected. In at least one embodiment, a training data set can be provided to a machine learning algorithm, where the model is a model artifact created by the training process. In at least one embodiment, each instance of the training data contains a correct answer (e.g., a classification) that may be referred to as a target or target attribute. In at least one embodiment, the learning algorithm finds patterns in the training data that map input data attributes to targets, answers to predict, and outputs a machine learning model that captures these patterns. In at least one embodiment, the machine learning model can then be used to obtain predictions for new data for which the target was not specified.

[0200] In at least one embodiment, the training and inference manager 532 can select from a set of machine learning models including binary classification, multi-class classification, generative, and regression models. In at least one embodiment, the type of model to be used can depend at least in part on the type of target to be predicted.

[0201] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device may be directly coupled to a system or processor that generates or renders images. In other embodiments, the display device may be indirectly coupled to the system or processor, such as via a network. Examples of such networks include the Internet, mobile telecommunications networks, WIFI networks, and any other wired and / or wireless network systems. When the display device is indirectly coupled, the images generated by the system or processor can be streamed to the display device via the network. Such a stream, for example, allows a video game or other application that renders an image to be executed on a server, a data center, or in a cloud-based computing environment, and the rendered image is transmitted and displayed on one or more user devices (such as computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separated from the server or data center. Therefore, the technology disclosed herein can be applied to enhance the images being streamed and enhance services that stream images, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0202] Example streaming system

[0203] Figure 6 is an example system diagram of a streaming system 605 according to some embodiments of the present disclosure. Figure 6 including (one or more) servers 603 (which may include Figure 5A The example processing system 500 and / or Figure 5B ), (one or more) client devices 604 (which may include components, features, and / or functions similar to the exemplary system 565 of Figure 5A The example processing system 500 and / or Figure 5B ), and network(s) 606 (which may be similar to the network(s) described herein). In some embodiments of the present disclosure, system 605 may be implemented.

[0204] In an embodiment, streaming system 605 is a game streaming system and server 604 is a game server. In system 605, for a game session, client device 604 may simply receive input data in response to input to input device 626, send the input data to server 603, receive encoded display data from server 603, and display the display data on display 624. In this way, more computationally intensive calculations and processing are offloaded to server(s) 603 (e.g., rendering(s)—specifically, ray or path tracing—for graphical output of the game session by GPU(s) 615 of server(s) 603). In other words, the game session is streamed from server 603 to client device(s) 604, thereby reducing the demand on client device(s) 604 for graphics processing and rendering.

[0205] For example, with respect to instantiation of a game session, client device 604 may display a frame of the game session on display 624 based on receiving display data from server 603. Client device 604 may receive input from one of input devices 626 and generate input data in response. Client device 604 may transmit the input data to server 603 via communication interface 621 and via network 606 (e.g., the Internet), and server 603 may receive the input data via communication interface 618. CPU 608 may receive the input data, process the input data, and send the data to GPU 615 so that GPU 615 generates a rendering of the game session. For example, the input data may represent a user's character moving in the game, firing a weapon, reloading, passing a ball, turning a vehicle, etc. Rendering component 612 may render the game session (e.g., representing the results of the input data), and rendering capture component 614 may capture the rendering of the game session as display data (e.g., as image data capturing a rendered frame of the game session). The rendering of the game session may include ray or path-traced lighting and / or shadow effects calculated using one or more parallel processing units (such as a GPU), which may further use one or more dedicated hardware accelerators or processing cores to perform the ray or path tracing techniques of the server 603. The encoder 616 may then encode the display data to produce encoded display data, and the encoded display data may be transmitted to the client device 604 via the network 606 via the communication interface 618. The client device 604 may receive the encoded display data via the communication interface 621, and the decoder 622 may decode the encoded display data to produce display data. The client device 604 may then display the display data via the display 624.

[0206] The embodiments of the present disclosure can be implemented in view of the following items.

[0207] 1. A computer-implemented method comprising:

[0208] dividing an input to the neural network model into parts, each part being sized for execution by a single thread block in a processing core;

[0209] loading the weights of the neural network model into a register file within a processor once to process the input;

[0210] independently processing the portions in parallel by a set of processing cores within the processor, wherein weights of a first layer of the neural network model are applied to each of the portions to generate an intermediate result for each portion;

[0211] storing the intermediate results of each portion in a memory shared among the set of processing cores; and

[0212] The intermediate result of each portion is processed by a subsequent layer of the neural network model to produce a subsequent intermediate result until the last layer of the neural network model generates an output.

[0213] 2. The computer-implemented method of clause 1 , wherein the set of processing cores and the memory are included within a graphics processing unit.

[0214] 3. The computer-implemented method of clause 1 , wherein the memory comprises at least one of a low-level cache, a shared on-chip memory, and a register.

[0215] 4. The computer-implemented method of clause 1 , wherein the neural network model implements a radiance cache for performing ray tracing.

[0216] 5. The computer-implemented method of clause 4, wherein the input is a three-dimensional (3D) position associated with a light transport path through a scene, and the output is a radiance prediction at the 3D position.

[0217] 6. The computer-implemented method of clause 5, further comprising:

[0218] generating an image of the scene corresponding to at least one camera using the radiosity prediction;

[0219] expanding the subset of light transport paths to generate additional radiation predictions; and

[0220] The weights are updated based on the additional radiance predictions, and the additional radiance predictions are propagated to the 3D positions for a subset of the light transport paths.

[0221] 7. A computer-implemented method according to claim 5, wherein each of the light transport paths terminates at the 3D location based on a first size of a first footprint of the light transport path at the 3D location compared to a second size of a second footprint of the light transport path at the visible 3D location.

[0222] 8. The computer-implemented method of clause 5, wherein the input comprises at least one of a material property, a view direction, or a normal vector associated with each of the 3D positions.

[0223] 9. The computer-implemented method of clause 4, wherein a filtered version of the weights is used to generate the radiance prediction at the 3D location.

[0224] 10. The computer-implemented method of clause 6, further comprising:

[0225] processing, by the radiance cache, a second 3D position associated with a second light transport path through the scene according to the updated weights to generate a second radiance prediction at the second 3D position; and

[0226] A second image of the scene is generated using the second radiance prediction.

[0227] 11. The computer-implemented method of clause 10, wherein at least one of the camera, lighting, geometry, and material is changed for the second light transport path compared to the light transport path.

[0228] 12. The computer-implemented method of clause 6, wherein at least a portion of the optical transmission path is associated with an additional camera.

[0229] 13. The computer-implemented method of clause 5, wherein the 3D position is generated by at least one of a rasterizer, a ray tracer, a ray stepper, a sphere tracer, and a cone tracer.

[0230] 14. The computer-implemented method of clause 1 , wherein the neural network model is trained on a server or in a data center and the output is streamed to a user device.

[0231] 15. The computer-implemented method of clause 1, wherein one or more of the steps of partitioning, loading, independently storing, and processing are performed within a cloud computing environment.

[0232] 16. The computer-implemented method of clause 1, wherein one or more of the steps of partitioning, loading, independent storage, and processing are performed on a server or in a data center, and the image is streamed to a user device.

[0233] 17. A computer-implemented method according to clause 1, wherein one or more of the steps of partitioning, loading, independent storage and processing are performed on a server or in a data center, and the neural network model is streamed to a user device.

[0234] 18. The computer-implemented method of clause 1, wherein the output is used to train, test, or certify a neural network employed in a machine, robot, or autonomous vehicle.

[0235] 19. The computer-implemented method of clause 1, wherein one or more of the steps of partitioning, loading, independently storing, and processing are performed on a virtual machine that includes a portion of a graphics processing unit.

[0236] 20. A system comprising:

[0237] Global memory, which stores weights;

[0238] A processor is connected to the global memory and executes a neural network model by:

[0239] dividing an input of the neural network model into portions, each portion sized for execution by a single thread block in a processing core;

[0240] loading the weights of the neural network model once into a register file within the processor to process the input;

[0241] independently processing the portions in parallel by a set of processing cores within the processor, wherein weights of a first layer of the neural network model are applied to each of the portions to generate an intermediate result for each portion;

[0242] storing the intermediate results of each portion in a memory shared among the set of processing cores; and

[0243] The intermediate result of each portion is processed by a subsequent layer of the neural network model to produce a subsequent intermediate result until the last layer of the neural network model generates an output.

[0244] 21. The system of clause 20, wherein the set of processing cores and the memory are included within a graphics processing unit.

[0245] 22. The system of clause 20, wherein the memory comprises at least one of a low-level cache, a shared on-chip memory, and a register.

[0246] 23. The system of clause 20, wherein the neural network model implements a radiance cache for performing ray tracing.

[0247] 24. A non-transitory computer-readable medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps:

[0248] Dividing the input of the neural network model into parts, each part is sized for execution by a single thread block in a processing core;

[0249] loading the weights of the neural network model into a register file within a processor once to process the input;

[0250] independently processing the portions in parallel by a set of processing cores within the processor, wherein weights of a first layer of the neural network model are applied to each of the portions to generate an intermediate result for each portion;

[0251] storing the intermediate results of each portion in a memory shared among the set of processing cores; and

[0252] The intermediate result of each portion is processed by a subsequent layer of the neural network model to produce a subsequent intermediate result until the last layer of the neural network model generates an output.

[0253] 25. The non-transitory computer-readable medium of clause 24, wherein the processor is a graphics processing unit.

[0254] 26. The non-transitory computer-readable medium of clause 24, wherein the memory comprises at least one of a low-level cache, a shared on-chip memory, and a register.

[0255] 27. The non-transitory computer-readable medium of clause 24, wherein the neural network model implements a radiance cache for performing ray tracing.

[0256] It should be noted that the techniques described herein may be embodied in executable instructions stored in a computer-readable medium, which are used by or in conjunction with a processor-based instruction execution machine, system, device, or apparatus. Those skilled in the art will appreciate that for some embodiments, different types of computer-readable media for storing data may be included. As used herein, "computer-readable media" includes one or more of any suitable media for storing executable instructions of a computer program so that an instruction execution machine, system, apparatus, or device can read (or fetch) instructions from the computer-readable medium and execute instructions for executing the described embodiments. Suitable storage formats include one or more of electronic formats, magnetic formats, optical formats, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable compact disks (CDs), portable digital video disks (DVDs), and the like.

[0257] It should be understood that the arrangement of the components shown in the drawings is for illustrative purposes and that other arrangements are possible. For example, one or more of the elements described herein may be implemented as an electronic hardware component in whole or in part. Other elements may be implemented in software, hardware, or a combination of software and hardware. Furthermore, some or all of these other elements may be combined, some may be omitted entirely, and additional components may be added while still achieving the functionality described herein. Therefore, the subject matter described herein may be embodied in many different variations, and all such variations are considered to be within the scope of the claims.

[0258] For ease of understanding theme described herein, many aspects have been described with respect to action sequence. Those skilled in the art will recognize that different actions can be performed by dedicated circuit or circuit, by the program instruction executed by one or more processors or by a combination of the two. The description of any action sequence herein is not intended to imply that the described particular order for performing this sequence must be followed. Unless otherwise indicated herein or context clearly conflict otherwise, all methods described herein can be performed in any suitable order.

[0259] In the context of describing a subject (particularly in the context of the following claims), the use of the terms "one" and "a kind of" and "said" and similar references should be interpreted as encompassing both the singular and the plural, unless otherwise indicated herein or clearly contradictory to the context. Unless otherwise indicated herein or clearly contradictory to the context, the use of the term "at least one" following one or more project lists (e.g., "at least one of A and B") should be interpreted as meaning an item selected from any combination of two or more of the listed items (A or B) or the listed items (A and B). In addition, the foregoing description is only for illustrative purposes, rather than for limiting purposes, because the scope of protection sought is limited by the claims set forth below and any equivalents thereof. The use of any and all examples or exemplary language (e.g., "such as") provided herein is only intended to better illustrate the subject and not to limit the scope of the subject, unless otherwise required. The use of the term "based on" and other similar phrases representing the conditions for producing a result in the claims and written description is not intended to exclude any other conditions for producing a result. The language in the specification should not be interpreted as indicating that any unclaimed element is essential to the practice of the claimed invention.

Claims

1. A computer-implemented method comprising: processing, by a neural network radiance cache model, three-dimensional (3D) locations associated with light transport paths through a scene to produce radiance predictions at the 3D locations, the radiance predictions being used to generate an image of the scene corresponding to at least one camera; extending the optical transmission path to generate an extended optical transmission path terminating at a terminal 3D position; processing the terminal 3D position to generate a second radiation prediction at the terminal 3D position; as well as Weights of the neural network radiance cache model are updated based on the second radiance prediction.

2. The computer-implemented method of claim 1 , wherein: The extended light transmission path is generated by extending the light transmission path from the 3D position.

3. The computer-implemented method of claim 2, wherein: The updated weights are also updated based on the second radiation prediction propagated to the 3D location.

4. The computer-implemented method of claim 1 , wherein: The extended light transmission path does not include the 3D position.

5. The computer-implemented method of claim 1 , wherein: The terminal 3D position is processed by the neural network radiation cache model to generate the second radiation prediction.

6. The computer-implemented method of claim 1 , wherein: The light transport path terminates at the 3D location based on a first size of a first footprint of the light transport path at the 3D location compared to a second size of a second footprint of the light transport path at the visible 3D location.

7. The computer-implemented method of claim 6, wherein: The extended light transmission path is terminated at the terminal 3D position based on a third size of a third footprint of the extended light transmission path at the terminal 3D position compared to the second size.

8. The computer-implemented method of claim 1 , wherein: The neural network radiance cache model also processes additional inputs to produce the radiance predictions, the additional inputs being at least one of a material property, a view direction, or a normal vector associated with the 3D position.

9. The computer-implemented method of claim 8, wherein: Prior to processing the 3D position and the additional input, the 3D position and the additional input are encoded into a higher dimensional space.

10. The computer-implemented method of claim 1 , wherein: A filtered version of the weights is used to generate the radiance prediction at the 3D location.

11. The computer-implemented method of claim 10, wherein: Updating the weights produces a weight update sequence, and the filtering is a moving average over the weight update sequence.

12. The computer-implemented method of claim 1 , wherein: The light transport path and additional light transport paths from additional cameras are traced to generate a light transport path for each pixel of the image, and further comprising: selecting a subset including the additional light transport paths and extending each additional light transport path in the subset to generate additional extended light transport paths.

13. The computer-implemented method of claim 1 , further comprising: processing, by the neural network radiance cache model, a second 3D position associated with a second light transport path through the scene according to the updated weights to generate a second radiance prediction at the second 3D position; as well as A second image of the scene is generated using the second radiance prediction.

14. The computer-implemented method of claim 13, wherein: At least one of the camera, lighting, geometry, and material is changed for the second light delivery path compared to the first light delivery path.

15. The computer-implemented method of claim 1 , wherein: The radiance prediction is multiplied by at least one of diffuse reflectance, specular reflectance, normal vector, roughness, albedo, bidirectional reflectance distribution function (BRDF), and phase function among material properties associated with the 3D position.

16. The computer-implemented method of claim 1 , wherein: The second 3D position for generating a second radiance prediction is importance sampled based on the 3D position.

17. The computer-implemented method of claim 1, wherein: The radiance prediction includes only ambient occlusion.

18. The computer-implemented method of claim 1, wherein: The neural network radiance cache model handles additional 3D positions associated with additional light transport paths from additional cameras.

19. The computer-implemented method of claim 1 , wherein: The 3D position is generated by at least one of a rasterizer, a ray tracer, a ray stepper, a sphere tracer, and a cone tracer.

20. The computer-implemented method of claim 1, wherein: The 3D positions are parameterized by 2D surface coordinates.

21. The computer-implemented method of claim 1 , wherein: The neural network radiance cache model is trained on a server or in a data center, and the image is streamed to the user device.

22. The computer-implemented method of claim 1, wherein: The neural network radiation cache model is trained on a server or in a data center and streamed to a user device.

23. The computer-implemented method of claim 1, wherein: One or more of the steps of processing the 3D position, generating the image, generating the extended optical transmission path, processing the terminal 3D position, and updating are performed within a cloud computing environment.

24. The computer-implemented method of claim 1, wherein: One or more of the steps of processing the 3D position, generating the image, generating the extended optical transmission path, processing the terminal 3D position, and updating are performed on a server or in a data center, and the image is streamed to a user device.

25. The computer-implemented method of claim 1 , wherein one or more of the steps of processing the 3D position, generating the image, generating the extended light transport path, processing the terminal 3D position, and updating are performed on a server or in a data center, and the neural network radiance cache is streamed to a user device.

26. The computer-implemented method of claim 1, wherein: The generated images are used to train, test, or certify a neural network employed in a machine, robot, or autonomous vehicle.

27. The computer-implemented method of claim 1, wherein: One or more of the steps of processing the 3D position, generating the image, generating the extended light transmission path, processing the terminal 3D position, and updating are performed on a virtual machine including a portion of a graphics processing unit.

28. A system comprising: a memory storing scene data; a processor connected to the memory and configured to generate an image of the scene by: processing, by a neural network radiance cache model, three-dimensional (3D) locations associated with light transport paths through a scene to produce radiance predictions at the 3D locations, the radiance predictions being used to generate an image of the scene corresponding to at least one camera; extending the optical transmission path to generate an extended optical transmission path terminating at a terminal 3D position; processing the terminal 3D position to generate a second radiation prediction at the terminal 3D position; as well as Weights of the neural network radiance cache model are updated based on the second radiance prediction.

29. The system of claim 28 further comprising terminating the light transport path at the visible 3D location based on a first size of a first footprint of the light transport path at the 3D location compared to a second size of a second footprint of the light transport path at the visible 3D location.

30. The system of claim 28, wherein: The neural network radiance cache model also processes additional inputs to produce the radiance predictions, the additional inputs being at least one of a material property, a view direction, or a normal vector associated with the 3D position.

31. The system of claim 28, wherein: A filtered version of the weights is used to generate the radiance prediction at the 3D location.

32. The system of claim 28, wherein the processing comprises: dividing the 3D position and additional 3D positions associated with additional light transport paths from additional cameras into portions, each portion sized for execution by a single thread block in the processing core; as well as The portions are independently processed in parallel by a set of processing cores within the processor, wherein the weights are applied to each of the portions to generate the radiance prediction at the 3D location and to generate additional radiance predictions at the additional 3D locations.

33. A non-transitory computer-readable medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: processing, by a neural network radiance cache model, three-dimensional (3D) locations associated with light transport paths through a scene to produce radiance predictions at the 3D locations, the radiance predictions being used to generate an image of the scene corresponding to at least one camera; extending the optical transmission path to generate an extended optical transmission path terminating at a terminal 3D position; processing the terminal 3D location to produce second reflected radiation at the terminal 3D location; as well as Weights of the neural network radiance cache model are updated based on the second radiance prediction.

34. The non-transitory computer readable medium of claim 33, wherein: The light transport path terminates at the visible 3D location based on a first size of a first footprint of the light transport path at the 3D location compared to a second size of a second footprint of the light transport path at the visible 3D location.

Images