Recovering a three-dimensional model from two-dimensional images

Abstract

Recovery of three-dimensional models from two-dimensional images is disclosed. A three-dimensional (3D) model of an object is recovered from two-dimensional (2D) images of the object. Each image in a set of 2D images includes the object captured from a different camera position, and a deformation of a base mesh defining the 3D model can be computed corresponding to each image. The 3D model can also include a texture map representing lighting and material properties of the 3D model. Recovery of the 3D model relies on analytical anti-aliasing to provide a link between pixel colors in the 2D images and geometry of the 3D model. A modular, differentiable renderer design yields high performance by leveraging existing, highly optimized hardware graphics pipelines to reconstruct the 3D model. A differentiable renderer renders images of the 3D model, and differences between the rendered images and reference images are backpropagated through the rendering pipeline to iteratively adjust the 3D model.

Description

[0001] CLAIM

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 066,425, filed August 17, 2020, entitled “Differential Rasterization for Facial Performance Capture,” and U.S. Provisional Application No. 63 / 070,736, filed August 26, 2020, entitled “Differential Rasterization for Facial Performance Capture.” The entire contents of which are hereby incorporated by reference herein. BACKGROUND

[0003] Inverse rendering is a technique for iteratively recovering shape, lighting, and material properties of a 3D model based on 2D images. Inverse rendering is challenging because the operations used to render a 3D model to produce a 2D image cannot simply be performed in reverse to produce a 3D model from a 2D image. Conventional inverse rendering techniques generally focus on image quality or performance. There is a need to address these and / or other issues associated with existing techniques. SUMMARY

[0004] Embodiments of the present disclosure relate to recovering a three-dimensional (3D) model from two-dimensional (2D) images. Systems and methods are disclosed that enable recovering a 3D model of an object from a set of 2D images of the object. Each image in the set of 2D images includes the object captured from a different camera position. The recovered 3D model can be represented as a single base mesh defined by vertices corresponding to positions in 3D space. A deformation of the 3D model corresponding to each image can be computed. In embodiments, the deformation is an offset of the vertex positions. The 3D model can also include a texture map representing lighting and material properties of the 3D model. The recovery of the 3D model relies on analytic antialiasing for providing a link between pixel colors in the 2D images and the geometry of the 3D model.

[0005] A method, computer-readable medium, and system for constructing a 3D model from 2D images are disclosed. An image of a 3D model defined by initial geometry is rendered, where an antialiasing operation performed by a rendering pipeline processes data associated with the geometry to antialias the image, and a pixel difference is computed based on the image and a reference image. The pixel difference is back-propagated through the antialiasing operation of the rendering pipeline to compute a geometry change corresponding to reducing the pixel difference, and the initial geometry is adjusted based on the geometry change to produce modified geometry defining the 3D model. BRIEF DESCRIPTION OF DRAWINGS

[0006] The present systems and methods for recovering a 3D model from 2D images are described in detail below with reference to the attached drawing figures, wherein:

[0007] FIG. 1A A conceptual diagram of a 3D model recovery system is shown, in accordance with an embodiment.

[0008] FIG. 1B Another conceptual diagram of a 3D model recovery system is shown, in accordance with an embodiment.

[0009] FIG. 1C A conceptual diagram of analytical anti-aliasing is shown, in accordance with an embodiment.

[0010] FIG. 1D Another conceptual diagram of analytical anti-aliasing is shown, in accordance with an embodiment.

[0011] FIG. 2A A block diagram of an example 3D model recovery system suitable for implementing some embodiments of the present disclosure is shown.

[0012] FIG. 2B Another block diagram of an example 3D model recovery system suitable for implementing some embodiments of the present disclosure is shown.

[0013] FIG. 2C Another block diagram of an example 3D model recovery system suitable for implementing some embodiments of the present disclosure is shown.

[0014] FIG. 3 A flow diagram of a method for 3D model recovery suitable for implementing some embodiments of the present disclosure is shown.

[0015] FIG. 4 An example parallel processing unit suitable for implementing some embodiments of the present disclosure is shown.

[0016] FIG. 5A A conceptual diagram of a processing system implemented using a PPU of FIG. 4 suitable for implementing some embodiments of the present disclosure.

[0017] FIG. 5B An example system in which various previously described embodiments of architecture and / or functionality can be implemented is shown.

[0018] FIG. 5C Components of an example system that can be used for training and with machine learning, in at least one embodiment, are shown.

[0019] FIG. 6A A conceptual diagram of a graphics processing pipeline implemented by a PPU of FIG. 4 suitable for implementing some embodiments of the present disclosure.

[0020] FIG. 6B An example game streaming system suitable for implementing some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] Systems and methods related to recovering a 3D model from 2D images are disclosed. A modular, differentiable renderer design yields high performance by leveraging existing, highly optimized hardware graphics pipelines to reconstruct a 3D model. In embodiments, one or more operations of the differentiable renderer are performed using a graphics processing unit (GPU) graphics pipeline, GPU general purpose compute cores, or any combination on a central processing unit (CPU). The differentiable renderer implements operations such as rasterizing a large number of triangles, attribute interpolation, filtering texture lookups, and user programmable shading and geometry processing all at high resolution. The recovered 3D model is accurate compared to conventional systems and can be generated by rendering analytically antialiased images of the 3D model and iteratively adjusting the 3D model by backpropagating the difference between the rendered images and the reference images through the rendering pipeline.

[0022] FIG. 1A A conceptual diagram of a 3D model recovery system 100 according to an embodiment is shown. A set of 2D images 110 of an object are captured from a variety of camera positions. The 3D model recovery system 100 uses the set of 2D images to construct a 3D model 130 of the object to refine an initial 3D model. The initial 3D model can be a sphere or other geometric shape. In embodiments, the 3D model 130 is a base model that is deformed to yield a particular 3D model corresponding to each of the 2D images in the set. In another embodiment, a complete 3D model 130 is constructed for each of one or more of the 2D images in the set. A set of texture maps 125 are global surface textures that define lighting and material properties 120 that can be applied to the 3D model 130. In embodiments, the set of texture maps 125 includes mip-mapped textures. An initial global surface texture can be a constant color and the color of each texture pixel in the initial global surface texture is adjusted by the 3D model recovery system 100 to yield the set of texture maps 125.

[0023] The goal of the 3D model recovery system 100 is to yield a 3D model 130 that, when rendered using the set of texture maps 125, yields rendered images that closely match the set of 2D images 110 of the object. The 3D model recovery system 100 uses the camera positions associated with the set of 2D images 110 of the object to render the 3D model 130. Unlike conventional rendering systems that have the goal of rendering high quality images, the 3D model recovery system 100 leverages rendering to implement the recovery of the 3D model 130. In other words, the rendered images are processed to determine and fine-tune the geometry that defines the 3D model. In embodiments, the geometry is defined by the positions of the vertices of a mesh that forms the 3D model 130. In embodiments, the mesh is defined by other types of primitives or representations. In one embodiment, the set of 2D images 110 of the object includes a video.

[0024] The recovery of the 3D model 130 can be used to perform markerless facial performance capture. Markerless means that there are no marker points placed on the subject in the set of 2D images 110. The recovery process constructs a 3D model 130 that reproduces the facial expressions visible in the different 2D images. The 3D model recovery enables the generation of new images of the 3D model for character animation, such as during game play or for use during film production.

[0025] More explanatory information will now be set forth regarding various optional architectures and features that can be implemented in accordance with the desires of the user to implement the foregoing framework. It is strongly noted that the following information is set forth for illustrative purposes and is not to be construed as limiting in any manner. Any of the following features can optionally be incorporated with or without excluding other features described.

[0026] FIG. 1B Another conceptual diagram of a 3D model recovery system 100 is shown in accordance with an embodiment. The 3D model recovery system 100 receives an initial 3D model 132 of a subject and an initial global texture (not shown). In an embodiment, the texture coordinates of the initial 3D model 132 are associated with each vertex that defines the initial 3D model 132, and the association between the vertices and the texture coordinates does not change even when the positions of the vertices are modified to produce a constructed 3D model 134. However, as the constructed 3D model 134 is modified, the content of the global texture is adjusted.

[0027] The 3D model recovery system 100 also receives a set of 2D images 110 of the subject, which can include a reference image 112. As previously described, the goal of the 3D model recovery system 100 is to find a global texture and a constructed 3D model 134 (e.g., per-image mesh) that, when rendered from the camera position 114 associated with the reference image 112, produces a rendered image 115 that matches the reference image 112. In an embodiment, the 3D model recovery system 100 compares the reference image 112 and the rendered image 115, determines the differences and computes an image-space loss. The image-space loss is then backpropagated through the rendering operations to adjust the initial 3D model 132 and produce the constructed 3D model 134. In contrast to conventional rendering systems that are configured to produce images from 3D geometry, the 3D model recovery system 100 includes a differentiable rendering pipeline. The differentiable rendering pipeline is capable of producing images from 3D geometry in a forward mode of operation, and is also capable of reconstructing 3D geometry from images in a reverse mode of operation. The rendering and backpropagation can be repeated for several different reference images to iteratively adjust the initial 3D model 132, morphing the initial 3D model 132 to correspond to the respective reference images.

[0028] In the context of the following description, the rendered image 115 and other images rendered by the 3D model recovery system 100 are anti-aliased images, in which an anti-aliasing operation processes the geometry data (e.g., vertex positions or primitives) associated with the initial 3D model 132 or base mesh to anti-alias the image. In one embodiment, the anti-aliasing operation is an analytical anti-aliasing operation that determines the shaded pixel values based on the geometry coverage after performing rasterization, visibility testing, and texture mapping.

[0029] FIG. 1C A conceptual diagram of analytical anti-aliasing is shown in accordance with an embodiment. The rendered geometry 136 forms a silhouette edge 138 of the object that is closer to the camera than the background or another surface occluded by the object. In an embodiment, an edge forms a silhouette if the edge has only one connecting triangle, or if the edge connects two triangles with the same winding (indicating that the two triangles are front-facing or back-facing). In an embodiment, an edge forms a silhouette if the triangles connected to the edge are on the same side of the edge as seen from the camera, regardless of the winding.

[0030] Silhouette edges provide useful information needed to accurately recover the 3D model of the object, as the shape of the object in the image plane can be extracted and used to adjust the geometry (e.g., vertex positions) of the 3D model in conjunction with the camera position. As a result, each reference 2D image and the corresponding rendered analytical anti-aliased image, as described further herein, provides additional information for improving the 3D model.

[0031] Each of the pixels 135 and 140 includes a single sample at the center of the pixel. Conventionally, the rendered geometry 136 is visible in the pixel 135 and not visible in the pixel 140, as the sample in the pixel 135 is covered by the rendered geometry 136 and the sample in the pixel 140 is not covered. As a result, when shaded, the pixels 135 and 140 appear as shaded pixels 141 and 142. For many different positions and orientations of the edge 138, the shaded pixels 141 and 142 look the same. Notably, the shaded pixels 141 and 142 only provide enough information to know that the intersection 137 of the edge 138 is somewhere along the horizontal segment between the two samples of the pixels 135 and 140. When the sample in the pixel 140 is covered or not covered, the transition is abrupt and discontinuous, rather than smooth or gradual, and does not accurately represent the rendered geometry 136.

[0032] Analytical anti-aliasing uses the geometry data (especially the edge 138 and the intersection 137) to compute the shaded pixels 143 and 144, rather than simply relying on whether the samples are covered or not to compute the shaded pixels. As FIG. 1CAs shown in the middle, pixel 144 is shaded corresponding to the portion of the rendered geometry 136 that covers it. As the intersection 137 moves closer to the sample in pixel 140, the contribution to the shaded value of pixel 144 from the rendered geometry 136 increases. Conversely, as the intersection 137 moves further from the sample in pixel 140, the contribution to the shaded value of pixel 144 from the rendered geometry 136 decreases. The same information used to determine the contribution of the rendered geometry 136 to the shaded pixels 143 and 144 in image space can be used to work backwards to calculate the gradient of the vertex position in 3D model space. Just as analytical antialiasing produces a more accurate image in terms of object visibility, the corresponding vertex gradient provides a more accurate adjustment to the 3D model.

[0033] In an embodiment, edge 138 passes between the centers of horizontally adjacent pixels 135 and 140, and is detected by pixels 135 and 140 having different triangle identifiers (IDs) that are rasterized into horizontally adjacent pixels 135 and 140. Pixels 135 and 140 can be processed together as a pair, and one of the following can occur. (a) Edge 138 intersects a line segment connecting the pixel centers at intersection 137 inside pixel 140, such that the color of pixel 135 is blended into pixel 140. (b) The intersection occurs inside pixel 135, so the blending is done in the opposite direction. To approximate the geometry of the surface between, the blending factor is a linear function of the position of the intersection - from zero at the midpoint to 50% at the pixel center. This particular analytical antialiasing method is differentiable, because the resulting pixel color is a continuous function of the position of the vertex of the rendered geometry 136. In an embodiment, more complex calculations can be performed to determine the blending factor, taking into account, for example, the orientation, length, and position of the endpoints of edge 138, to more accurately estimate the amount of rendered geometry 136 that covers pixel 140. In an embodiment, multiple edges can be considered when determining the blending factor. In one embodiment, multiple blending factors can be determined to enable blending between more than two pixels.

[0034] FIG. 1DAnother conceptual diagram of analytical antialiasing is shown. The rendered geometry 146 forms an outline edge 148 of an object. Each of the pixels 145 and 150 includes a single sample at the center of the pixel. Conventionally, the rendered geometry 146 is visible in the pixel 145 and not visible in the pixel 150 because the sample in the pixel 145 is covered by the rendered geometry 146 and the sample in the pixel 150 is not covered. As such, when shaded, the pixels 145 and 150 appear as shaded pixels 151 and 152. With conventional rasterization without analytical antialiasing (one sample per pixel, or even multiple samples per pixel), the visibility (i.e., which geometry is visible in each pixel) is discontinuous and piecewise constant - moving any vertex or edge 148 by an infinitesimal amount will not change which pixels (or samples) the rendered geometry 146 covers.

[0035] In contrast, analytical antialiasing uses the edges 148 and intersection 147 to compute the shaded pixels 153 and 154. The intersection 147 is inside the pixel 145, such that the color of the pixel 150 is blended into the pixel 145. As shown in FIG. 1D pixel 153 corresponds to being shaded partially covered by the rendered geometry 146, and the pixel 154 is not shaded because the rendered geometry 146 does not intersect the pixel 150.

[0036] Analytical antialiasing approximates the pixel integral (the average surface color inside the pixel) based on the position of the outline edge in the pixel. The output color of a pixel depends on the position of the vertices that define the outline edge in the pixel, among other things. In particular, the intersection of segments between pairs of pixels of different triangle IDs. The dependence is (piecewise) continuous and thus differentiable, and thus the gradient of the vertex positions will also reflect the change in the output pixel color due to how much more of the pixel is covered by the rendered geometry that defines the outline edge as a result of a change in the position of the vertices. The gradient will thus contain information about how a movement of a vertex affects the position of the outline edge in the rendered image. As shown in FIG. 1C and FIG. 1D As the outline edge of the rendered geometry moves closer to or further away from the center of the pixel, the color of the pixel changes. As such, the change in the color of the pixel provides information about the position of the vertices. In contrast, when using conventional rasterization, the color of the pixel only changes when the center is covered or uncovered, so very little information about the position of the vertices can be determined from the color. Thus, if the position of the outline edge of a traditionally rendered 3D model in image space is wrong, little or no information is available based on which vertices of the 3D model can be adjusted to make the outline edge appear closer to where it should be in the rendered image.

[0037] Given a set of 2D images of an object, 3D model recovery techniques can use analytical anti-aliasing to recover not only the shape of the object, but also the lighting and material properties of the 3D model. Analytical anti-aliasing is included in a differentiable rasterization pipeline that performs deferred shading to render the 3D model to produce a rendered image for each camera position associated with a 2D image. Specifically, analytical anti-aliasing is performed on the output of the deferred shading operations, receives shaded pixels, and takes as additional inputs geometry data, triangle IDs, and vertex positions and indices.

[0038] Analytical anti-aliasing can be achieved by first detecting potential visibility discontinuities by finding all adjacent horizontal and vertical pixel pairs with mismatched triangle IDs. For each potential discontinuity, the triangle associated with the surface closer to the camera is extracted as determined from the normalized device coordinates (NDC) depth computed during rasterization. The edges of the triangle are then examined to determine if either of the edges forms a silhouette and passes between the pixel centers of the pixel pair. For horizontal pixel pairs, only vertically oriented edges (|w c，1 ·y c，2 -w c，2 ·y c，1 |>|w c，1 ·x c，2 -w c，2 ·x c，1 |), and vice versa, where (x, y, w) are the x, y pixel coordinates in clip space. If the silhouette edge crosses the segment between the pixel centers, the blending weight is computed by determining the intersection point where the crossing occurs. The pixel color is then adjusted to reflect the approximate coverage of either surface in the pixel. This technique uses axis-aligned slabs to approximate the exact surface coverage of each pixel. Thus, the coverage estimate is only exact for perfect vertical and horizontal edges that extend beyond the pixel. However, for the purpose of recovering an accurate 3D model, the coverage estimate is a sufficiently good approximation for other (non-vertical, non-horizontal) intersections. In embodiments, more complex calculations can be performed to determine the blending factors taking into account, for example, the orientation, length, and position of the edge endpoints.

[0039] Given a 3D scene description in the form of geometry, materials, and camera and lighting models, rendering 2D images reduces to two computational problems: computing which parts of the 3D scene are visible in each pixel and what color the visible parts look like. A proper differentiable renderer must provide gradients for all parameters used in the process, e.g., lighting and material parameters and the contents of texture maps. In the context described below, it is useful to decompose the rendering process into the following form, where the final color I i ,y i ) of a pixel at screen coordinates (x iGiven by:

[0040] I i = filter x，y (shade(M(P(x, y)), lights))(x i , y i ) Equation (1)

[0041] Here, P(x, y) represents the world point visible at (continuous) image coordinates (x, y) after projection from 3D to 2D, and M(P) represents all spatially varying factors (texture mapping, normal vectors, etc.) living on the surface of the scene. The shading function typically models the light surface interaction. A 2D anti-aliasing filter, crucial for both image quality and differentiability, is applied to the shading result in continuous (x, y), and the final color is obtained by sampling the result at the pixel center (x, y).

[0042] FIG. 2A A block diagram of an example 3D model recovery system 100 suitable for implementing some embodiments of the present disclosure is shown. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted altogether. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location of hardware, firmware, and / or software. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For instance, various functions can be performed by a processor executing instructions stored in memory. Moreover, one of ordinary skill in the art will appreciate that any system that performs the operations of the 3D model recovery system 100 is within the scope and spirit of embodiments of the present disclosure.

[0043] The 3D model recovery system 100 includes a 3D model construction unit 200, a differentiable renderer 215, and an image space loss unit 245. The differentiable renderer 215 includes a rendering pipeline 205 and a backpropagation pipeline 260. A 3D model is rendered in a forward pass through the rendering pipeline 205 of the differentiable renderer 215 from a camera position to produce an anti-aliased image. The 3D model construction unit 200 provides a representation of the 3D model to the rendering pipeline 205 along with a reference camera position. In embodiments, the initial 3D model can be a basic mesh as simple as a cube or a sphere. In embodiments, the 3D model includes vertices in a 3D model space and attributes associated with the vertices. An initial surface texture map corresponding to the initial 3D model can be a uniform color.

[0044] The rendering pipeline 205 processes the 3D model, performing steps of transformation (from 3D to 2D), rasterization, interpolation, texture lookup, and anti-aliasing. The last stage in the rendering pipeline 205 performs analytical anti-aliasing to compute the visibility-dependent effects of geometric edges on the rendered image. When performed in reverse by the backpropagation pipeline 260, the analytical anti-aliasing operation determines how the gradient of pixel colors transfers to the gradient of the 3D model.

[0045] The image-space loss unit 245 determines an image-space (color) loss based on the per-pixel color difference between the anti-aliased image and a reference (target) image of the object associated with a reference camera position. This difference quantifies the accuracy of the 3D model of the object and represents a "loss." In embodiments, the difference is computed as a per-pixel mean squared error. In embodiments, the difference between high-dimensional embeddings of images is computed, e.g., using a pre-trained neural network. The image-space loss penalizes 3D model solutions in which the rendering does not match the reference image. However, the image-space loss unit 245 can be used with other loss functions to regularize the optimization. In embodiments, a Laplacian loss penalizes solutions in which the curvature of the mesh changes significantly compared to the base mesh, effectively encouraging the optimization to consider only physically reasonable solutions.

[0046] The backpropagation pipeline 260 receives the anti-aliased image gradients that indicate how the color of each pixel of the anti-aliased image affects the loss. The backpropagation pipeline 260 computes (1) the gradient of the aliased image input to the last stage of the rendering pipeline 205 (analytical anti-aliasing), and (2) the gradient of the vertex positions. The forward rendering, comparison to reference images, and backpropagation are performed for multiple camera positions to produce a surface texture map and geometry of a modified version of the initial 3D model or base mesh for each reference image. The result is a 3D model of the object and a corresponding surface texture map that, when rendered, matches the target images. The use of multiple camera positions enables the recovery of different parts of the object through analytical anti-aliasing due to the variety of rendered silhouette edges.

[0047] In embodiments, the texture map values (e.g., texture map coordinates and texture pixels) and vertex positions are potential variables that can be optimized to reduce loss, thereby improving the accuracy of the 3D model. Backpropagation is used to perform the determination of how to change the potential variables to achieve a reduction in loss. When the antialiasing image gradient is propagated through the backpropagation pipeline 260, the gradient of the output of each computational step is known (i.e., how a change in the output value of each computational step will affect the loss), so that the gradient of the input to the computational step can be determined (i.e., how a change in the input value of each computational step will affect the loss). The parameters computed by each step during the forward propagation through the rendering pipeline 105 can be provided to the backpropagation pipeline 260 for use in computing the corresponding per-stage gradient.

[0048] After the antialiasing image gradient is backpropagated through the entire backpropagation pipeline 260, the changes to the potential variables can be quantified as to how they affect the loss, and the potential variables can be adjusted in a direction in which the loss should be reduced. The visibility-dependent 3D model gradient computed during backpropagation indicates the effect of moving the vertex positions due to changes in fractional pixel coverage on the antialiasing image. The 3D model building unit 200 receives the 3D model gradient and adjusts the 3D model to reduce the loss.

[0049] Applying equation (1) to differentiable rendering, geometry, projection, and lighting can all be viewed as parameterized functions. The visible world points are affected by the parameterized geometry G and the parameterized projection by θ C . Similarly, the surface factors are parameterized by θ M and the light sources by θ L . In the simplest case, θ G and θ M can describe, for example, the vertex coordinates of a triangle mesh of fixed topology and the diffuse albedo stored at the vertices and interpolated to the triangle interiors. In embodiments, the 3D model representation is a complex parameterization that is computed by a deep learning model within the 3D model building unit 200 and input to the differentiable renderer 115. In the context described below, differentiable rendering includes computing the gradient of the scalar function L(I) of the rendered image I with respect to the scene parameters Note that this does not require computing the (very large) Jacobian matrix but only the ability to implement the multiplication with the transpose of the Jacobian (“backpropagation”), resulting in the final result by the chain rule:

[0050]

[0051] and similarly for the other parameter vectors.

[0052] Two main factors make the design of an efficient rendering algorithm challenging. First, the mapping P(x, y) between 3D model or world space points and screen or image coordinates is dynamic: it is affected by changes in both the scene geometry and the 3D to 2D projection. Moreover, the mapping is discontinuous due to occlusion boundaries. These two factors are also central to the difficulty of computing gradients for the backpropagation pipeline 260.

[0053] The differentiable renderer 215 can render 3D scenes that are complex in terms of geometric detail, occlusions, and appearance at high resolution. In one embodiment, the rendering is performed in real-time, and stages in the pipeline are parallelized to handle both geometric primitives and pixels. In embodiments, the differentiable renderer 215 includes modular, configurable, and programmable stages to enable easy construction of potentially complex custom rendering pipelines. In embodiments, the differentiable renderer 215 takes input geometry and texture maps (e.g., 3D models) in the form of tensors, allowing both to be parameterized in a freely chosen manner, and enables rendering primitives to be used as building blocks for complex learning systems.

[0054] The differentiable renderer 215 performs a first computation of the M(P(x, y)) term from equation (1) for each pixel, and stores the intermediate result in an image space regular grid. The grid is then consumed by the shading function to produce a shaded output grid, which is input to the analytical anti-aliasing filter in equation (1). Effectively, it is assumed that shading is constant with respect to the occlusion effect at the silhouette boundary, but not with respect to other effects on appearance.

[0055] FIG. 2B Another block diagram of another example 3D model recovery system 100 suitable for implementing some embodiments of the present disclosure is shown. The differentiable rendering pipeline 205 includes a vertex transformation unit 210, a rasterizer 220, an interpolation unit 230, a texture lookup 240, and an analytical anti-aliasing unit 250. In embodiments, the differentiable rendering pipeline 205 also includes a backpropagation pipeline 260. Details of the backpropagation pipeline 260 are shown in FIG. 2C In embodiments, the differentiable rendering pipeline 205 and the backpropagation pipeline 260 are combined, and the operations performed by the backpropagation pipeline 260 are performed by a combination of the vertex transformation unit 210, the rasterizer 220, the interpolation unit 230, the texture lookup 240, the analytical anti-aliasing unit 250.

[0056] The 3D model construction unit 200 adjusts the 3D model for each iteration of the differentiable renderer 215. In embodiments, the 3D model construction unit 200 includes a “morphing network” that takes a base mesh of the 3D model and a frame index representation and outputs vertex positions to render an anti-aliased image for a reference camera position. The morphing network is not necessarily a general-purpose neural network, but it can learn (i.e., be more amenable to optimization) more efficiently than having an array of vertex positions for each frame individually. The 3D model construction unit 200 also adjusts a global surface texture for each iteration of the differentiable renderer 215. As shown in FIG. 2, the 3D model output to the vertex transformation unit 210 includes vertices and attributes in 3D model space. FIG. 2B

[0057] The vertex transformation unit 210 performs world, view, and isometric perspective transformations to produce transformed vertices that are output to the rasterizer 220. The rasterizer 220 performs perspective subdivision and enables dynamic mapping between world coordinates and discrete pixel coordinates. Per-pixel auxiliary data can be stored in the form of barycentric coordinates and triangle IDs in forward pass through the rendering pipeline 205. Using barycentric and NDC depth (u, v, z c / w c ) as the base coordinate system allows easy coupling of shading and interpolation, as well as combining texture gradients with geometry gradients when backward pass through the backpropagation pipeline 260.

[0058] In embodiments, the rasterizer 220 consumes triangles with an array of vertex positions given as clip space isometric coordinates (x c ,y c ,z c ,w c ). The backpropagation pipeline 260 then computes the gradient of the loss L with respect to the clip space positions Differentials with respect to any higher-level parameterization can be performed externally to the 3D model recovery system 100.

[0059] In forward pass through the rendering pipeline 205, the rasterizer 220 outputs a 2D sample grid associated with the image being rendered, with each location storing a tuple (ID, u, v, z c / w c ​), where ID identifies the triangle that covers the sample, (u, v) are barycentric coordinates that specify the relative position along the triangle, and z / w corresponds to the depth in normalized device coordinates (NDC). In embodiments, a special ID is reserved for blank pixels. The barycentric coordinates serve as a convenient basis for interpolation and texture mapping calculations for downstream stages in the rendering pipeline 205. In embodiments, the NDC depth is only utilized by the subsequent analytical anti-aliasing unit 250 and does not propagate gradients. In embodiments, the rasterizer 220 outputs a 2x2 Jacobian matrix with barycentric coordinates relative to image coordinates for each pixel An auxiliary output buffer of the 2x2 Jacobian matrix

[0060] Within the rasterizer 220, rasterization can be performed by OpenGL using a hardware graphics pipeline. Using a hardware graphics pipeline ensures that the rasterization is accurate and, for example, does not have visibility leaks due to precision issues. Additionally, using a hardware graphics pipeline automatically provides proper view frustum clipping. The per-pixel 2x2 Jacobian matrix between barycentric and image coordinates can be obtained from an OpenGL fragment shader, which is computed from finite differences in a 2x2 pixel quad. Alternatively, the per-pixel 2x2 Jacobian matrix can be analytically computed for each pixel.

[0061] The interpolation unit 230 extends per-vertex data (i.e., vertex attributes) to pixel or image space, resulting in interpolated attributes. With the barycentrics computed by the rasterizer 220, the interpolation unit 230 accomplishes the mapping of the forward direction, which can also be used by the corresponding interpolation operations in the reverse propagation pipeline 260 to map from image space to NDC space.

[0062] Attribute interpolation is a standard part of the graphics pipeline. In particular, it requires the computation of a weighted sum of vertex attributes, where the weights are given by the barycentrics, thereby creating a mapping between pixels and attributes. In general, vertex attributes can be used for arbitrary purposes. However, one of the typical uses is to provide 2D coordinates for texture mapping. In addition to its general operation, the interpolation unit 230 also provides special support for computing the image space derivatives of the texture coordinates in the forward pass, which can later be used by the texture lookup 240 in determining the MIP-mapped filter footprint.

[0063] The interpolation unit 230 receives a vector of attributes A i associated with the i-th vertex, where the attribute index i is for the triangle that is visible in pixel (x, y) 0,1,2and the barycenter produced by the rasterizer 220 is u = u(x, y) and v = v(x, y). Given the output of the rasterizer (per-pixel triangle ID and barycenter), the interpolation vector A is defined as

[0064]

[0065] The implementation of the forward pass is straightforward. The image-space derivatives of the attributes marked as requiring derivatives are computed using the barycentric Jacobian matrix output by the rasterizer by where the last Jacobian matrix is easily derived from equation (2).

[0066] The texture lookup 240 receives the interpolated texture coordinate attribute and the image-space derivative of the texture coordinate attribute, and reads the texture pixel from the global surface texture stored in the 3D model construction unit 200. The texture lookup 240 uses the texture pixel and the image-space derivative to produce a shaded pixel. The backpropagation pipeline 260 can correctly backpropagate gradients by inputting both the texture coordinate and the content of the texture map (MIP-mapped) to the texture lookup.

[0067] When the texture lookup 240 performs texture mapping using a trilinear MIP-mapped texture fetch, the (continuous) MIP-mapped pyramid level (i.e., level of detail, LOD) is selected based on the incoming image-space derivative of the texture coordinate. Then, a trilinear interpolation is performed using the eight nearest texture pixels from that appropriate MIP pyramid level. The MIP level can be selected based on the texture-space length of the long axis of the sample footprint defined by the derivative. In an embodiment, the MIP level is computed by the interpolation unit 230 as part of the interpolation operation, where the necessary data is readily available. In one embodiment, the texture lookup 240 can be configured to perform texture mapping using bilinear, point sampling, or other types of MIP-mapped or non-MIP-mapped texture fetches.

[0068] The image comprising the shaded pixels output by the texture lookup 240 exhibits no aliasing within the surface (inside the rendered geometry). However, point sampling causes aliasing at visibility discontinuities, and more critically, cannot produce visibility-dependent gradients for adjusting the vertex positions of the 3D model. The analytical anti-aliasing unit 250 converts the discontinuities into a smooth variation from which the gradients can be computed. Note that the anti-aliasing can be performed only after shading, and thus must be implemented as a separate stage of the rendering pipeline 205, rather than as part of the rasterization.

[0069] The analytical anti-aliasing unit 250 receives the delayed shaded, transformed vertices, triangle IDs, and depth (z c / w c) produces a colored pixel, and outputs an anti-aliased image. Analytic anti-aliasing unit 250 detects visibility discontinuities and associates them with vertex positions as needed to compute gradients. In embodiments, potential visibility discontinuities are detected by finding all pairs of adjacent horizontal and vertical pixels with mismatched triangle IDs, and computing blending weights to adjust the pixel color of the profile edge that crosses between the centers of the detected pairs of adjacent horizontal and vertical pixels, as previously described in connection with FIG. 1C and FIG. 1D The results of the discontinuity analysis performed by analytic anti-aliasing unit 250 during forward pass can be stored to avoid repeated computation during backward pass in preparation for gradient computation during backward propagation. In embodiments, the results of the discontinuity analysis performed by analytic anti-aliasing unit 250 during forward pass are stored in a buffer 255. Buffer 255 can be implemented as a memory buffer in system memory 120 or as a register file in GPU 110. In embodiments, the results of the discontinuity analysis performed by analytic anti-aliasing unit 250 during forward pass are stored in buffer 255 in the form of a visibility map 255-A and a gradient map 255-B. Visibility map 255-A can be used to determine which pixels are affected by a visibility discontinuity. Gradient map 255-B can be used to determine the gradient of a visibility discontinuity.

[0070] FIG. 2C Another block diagram of an example 3D model reconstruction system 100 suitable for implementing some embodiments of the present disclosure is shown. Details of a backward propagation pipeline 260 are shown. One or more of the stages in backward propagation pipeline 260 can be integrated into the corresponding stages of rendering pipeline 205. Stages 250-B, 240-B, 230-B, 220-B, and 210-B in backward propagation pipeline 260 correspond to analytic anti-aliasing unit 250, texture lookup 240, interpolation unit 230, rasterizer 220, and vertex transformation unit 210, respectively.

[0071] Stage 250-B uses the stored results of the discontinuity analysis to compute gradients for each pixel pair that was analytically anti-aliased by analytic anti-aliasing unit 250 in the forward pass. The anti-aliased image gradient of a pixel is transferred to a vertex position gradient by determining how the two vertex positions of a profile edge affect the blending weights. The loss gradient with respect to the shaded pixel computed by stage 250-B is output to stage 240-B as a shaded pixel gradient. The loss gradient with respect to the vertex positions is output by stage 250-B to stage 210-B to transform from clip space to 3D model space.

[0072] Stage 240-B computes the loss gradient with respect to the attribute A and with respect to the texture pixels. Once the MIP map level is selected, the operations of the forward and backward passes are very similar to attribute interpolation: the eight closest texture pixels replace the three triangle vertices, and the three subtexture pixel coordinates that determine the exact position within the eight texture pixels collectively replace the barycentric coordinates. The texturing of the MIP map differs from attribute interpolation in its multiscale nature: the gradients are accumulated in the backward pass on the various levels of the MIP map pyramid. Since all MIP map levels of the MIP map pyramid are obtained from the texture at the finest level during the construction of the forward pass, the backward pass needs to be done by transposing the construction operation and flattening the gradient pyramid so that the gradients are specified densely at the finest level. Fortunately, this is easily achieved by starting at the coarsest level, recursively upsampling the result, and precisely adding the gradients from the next level (like collapsing a Laplacian pyramid). Stage 240-B provides the texture pixel gradients to the 3D model construction unit 200. The texture pixel gradients are used to adjust the texture pixels of the global surface texture mapping of the 3D model.

[0073] Stage 230-B is associated with the interpolation unit 230 and receives the per-pixel loss gradient with respect to the interpolated attribute Stage 230-B provides the attribute gradients to the 3D model construction unit 200. The attribute gradients can be used to adjust the texture coordinates of the global surface texture mapping of the 3D model. In an embodiment, the gradients with respect to the attribute tensor are computed as a tensor by scatter-add, which applies a simple Jacobian matrix to the per-pixel input gradient. By simple differentiation, the gradient with respect to the input barycentric coordinates computed by stage 230-B is given by:

[0074]

[0075] The rasterizer backward pass receives the gradient with respect to the barycentric coordinates output by the rasterizer for each pixel and computes the gradient for each input vertex The perspective mapping between barycentric coordinates and clip space positions is easily analytically differentiated, and the necessary output is computed by stage 220-B by the following equation:

[0076]

[0077] Equation (4) can be implemented as a dense operation on the output pixels that uses scatter-add operations to accumulate the gradient from the pixel to the correct vertex based on the triangle ID.

[0078] Stage 210-B is associated with the vertex variation unit 210 and transforms the clipping space gradient for the 3D model to the 3D model space to provide the 3D model gradient to the 3D model construction unit 100. In an embodiment, the 3D model construction unit 100 directly uses vertex positions to represent the 3D model geometry and adjusts the vertex positions based on the 3D model gradient. In another embodiment, the 3D model construction unit 100 represents the 3D model geometry as a set of weights for a morphing network and adjusts the weights based on the 3D model gradient. The morphing network applies the set of weights to a base mesh to produce the 3D model that can be rendered for each camera position.

[0079] In an embodiment, the 3D model restoration system 100 is configured to construct a 2D model in which the 3D model defined by the initial geometry is replaced by a 2D model defined by a polygonal 2D geometry in a 2D model space. When constructing the 2D model, the depth-dependent calculations can be omitted. In an embodiment, the restored 2D model represents a font or 2D clip art defined by a 2D mesh or a curved primitive.

[0080] FIG. 3 A flowchart of a method 300 for 3D model restoration suitable for implementing some embodiments of the present disclosure is shown. Each block of the method 300 described herein comprises a computational procedure that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be carried out by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on computer storage media. To name just a few, the method can be provided by a standalone application, a service or hosted service (either independently or in combination with another hosted service), or a plug-in to another product. Further, by way of example, the method 300 is described with respect to the 3D model restoration system 100 of FIGS. 1-2. However, this method can additionally or alternatively be performed by any system or any combination of systems including, but not limited to, the systems described herein. Moreover, one of ordinary skill in the art would understand that any system performing the method 300 is within the scope and spirit of embodiments of the present disclosure. FIG. 1A 、 FIG. 1B 、 FIG. 2A 、 FIG. 2B and FIG. 2C The method 300 is described with respect to the 3D model restoration system 100 of FIGS. 1-2. However, this method can additionally or alternatively be performed by any system or any combination of systems including, but not limited to, the systems described herein. Moreover, one of ordinary skill in the art would understand that any system performing the method 300 is within the scope and spirit of embodiments of the present disclosure.

[0081] At step 305, an image of the 3D model defined by the initial geometry is rendered, where an anti-aliasing operation performed by the rendering pipeline processes data associated with the geometry to anti-alias the image. In an embodiment, the initial geometry is defined by vertex positions or a plurality of positions. In another embodiment, the initial geometry is defined by weights applied to a base mesh. The weights can be applied to the base mesh by a morphing network to produce the 3D model.

[0082] In embodiments, the anti-aliasing operation is an analytical anti-aliasing operation that includes determining that a silhouette edge intersects a pair of adjacent pixels based on data associated with the geometric shape, computing a blending weight between the adjacent pixels, and adjusting the colors of the adjacent pixels according to the blending weight. In embodiments, the blending weight is computed based on the intersection between the adjacent pixels. In embodiments, a more complex computation is performed to determine the blending factor taking into account, for example, the orientation, length, and position of the edge endpoints. In embodiments, determining includes identifying a silhouette edge between a first rendered geometric shape associated with a first identifier and a second rendered geometric shape associated with a second identifier. In embodiments, the anti-aliasing operation approximates the pixel integral based on the position of the silhouette edge within the pixel.

[0083] At step 310, pixel differences are computed based on the image and the reference image. In embodiments, the pixel differences are computed as a per-pixel mean squared error by a loss function. In embodiments, the pixel differences are computed between high-dimensional embeddings of the image, for example, using a pre-trained neural network.

[0084] At step 315, the pixel differences are back-propagated through the anti-aliasing operations of the rendering pipeline to compute geometric shape changes that correspond to reducing the pixel differences. In embodiments, the pixel differences are used to compute a per-pixel gradient of the aliased image. In embodiments, the geometric changes include gradients of vertex positions. In embodiments, propagating the pixel differences further includes generating a surface texture corresponding to the 3D model. The surface texture represents lighting and / or material properties of the 3D model.

[0085] At step 320, the initial geometric shape is adjusted based on the geometric shape changes to produce a modified geometric shape that defines the 3D model. Steps 305, 310, 315, and 320 can be repeated for at least one additional reference image. In embodiments, the reference image and the at least one additional reference image are each associated with a different camera position. In embodiments, the image of the 3D model is rendered according to the camera position.

[0086] The primary goal of the 3D model recovery system 100 is to construct an accurate 3D model of an object. While the 3D model recovery system 100 can also render anti-aliased images of the object, the generation of high-quality images should be considered a secondary goal. The differentiable renderer 215 provides a framework for programmable shading and geometry processing, providing a high degree of user control as well as the ability to render high-resolution images of scenes composed of millions of geometric primitives. The differentiable renderer 215 provides a customized, high-performance implementation for the following operations: rasterization, attribute interpolation, texture filtering, and anti-aliasing and differentiation operations for constructing accurate 3D models.

[0087] Parallel processing architecture

[0088] FIG. 4A parallel processing unit (PPU) 400 according to an embodiment is shown. The PPU 400 can be used to implement the 3D model recovery system 100. The PPU 400 can be used to implement one or more of the 3D model construction unit 200, the differentiable renderer 215, the rendering pipeline 205, the backpropagation pipeline 260, and the image space loss unit 245 within the 3D model recovery system 100. In embodiments, a processor such as the PPU 400 can be configured to implement a neural network model. The neural network model can be implemented as software instructions executed by the processor, or in other embodiments, the processor can include a matrix of hardware elements configured to process a set of inputs (e.g., electrical signals representing values) to generate a set of outputs that can represent activations of the neural network model. In other embodiments, the neural network model can be implemented as a combination of software instructions and processing performed by the matrix of hardware elements. Implementing a neural network model can include determining a set of parameters of the neural network model through, for example, supervised or unsupervised training of the neural network model, and, or in the alternative, using the set of parameters to perform inference to process a novel set of inputs.

[0089] In one embodiment, the PPU 400 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 400 is a latency hiding architecture designed for parallel processing of many threads. A thread (i.e., an execution thread) is an instance of a set of instructions configured to be executed by the PPU 400. In one embodiment, the PPU 400 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data in order to generate two-dimensional (2D) image data for display on a display device. In other embodiments, the PPU 400 can be used to perform general purpose computations. Although one exemplary parallel processor is provided herein for purposes of illustration, it is specifically intended that the processor be illustrative of any processor, and that the processor be supplemented by, or substituted for, any processor without departing from the scope of the present disclosure.

[0090] One or more PPUs 400 can be configured to accelerate thousands of compute-intensive computing applications including deep learning systems and applications for autonomous vehicles, simulations, compute graphics (such as ray-tracing or path tracing), deep learning, high-precision voice, image, and video recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0091] As FIG. 4As shown, PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a work distribution unit 425, a hub 430, a crossbar (Xbar) 470, one or more general processing clusters (GPCs) 450, and one or more memory partition units 480. PPU 400 can be connected to a host processor and other PPUs 400 via one or more high-speed NVLinks 410. PPU 400 can be connected to a host processor and other peripherals via an interconnect 402. PPU 400 can also be connected to a local memory 404, which can include a number of memory devices. In one embodiment, the local memory can include a number of dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem in which a number of DRAM dies are stacked.

[0092] The NVLink 410 interconnect enables systems to scale by connecting one or more CPUs with one or more PPUs 400, supports cache coherence between the PPU 400 and the CPU, and enables CPU predication. Data and / or commands can be sent by the NVLink 410 to other units of the PPU 400 or from them, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In conjunction with the Xbar 470, the NVLink 410 can be used to transmit data between the various units within the PPU 400 and between the PPU 400 and other processing clusters. FIG. 5B The NVLink 410 is described in more detail.

[0093] The I / O unit 405 is configured to transmit and receive communications (e.g., commands, data, etc.) with a host processor (not shown) over the interconnect 402. The I / O unit 405 can communicate directly with the host processor via the interconnect 402 or can do so by

[0094] The I / O unit 405 decodes data packets received via the interconnect 402. In one embodiment, the data packets represent commands configured to cause the PPU 400 to perform various operations. The I / O unit 405 sends decoded commands, as specified by the commands, to various other units of the PPU 400. For example, some commands can be sent to the front-end unit 415. Other commands can be sent to the hub 430 or other units such as one or more copy engines, video encoders, video decoders, power management units, and like (not explicitly shown). In other words, the I / O unit 405 is configured to route communications between and among various logical units of the PPU 400.

[0095] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. The workload can include a number of instructions and data to be processed by those instructions. The buffer is a region of memory that is accessible (e.g., read / write) by both the host processor and the PPU 400. For example, the I / O unit 405 can be configured to access the buffer in a system memory connected to the interconnect 402 via memory requests transmitted over the interconnect 402. In one embodiment, the host processor writes the command stream to the buffer and then sends a pointer to the start of the command stream to the PPU 400. The front-end unit 415 receives the pointer to the one or more command streams. The front-end unit 415 manages the one or more streams, reading commands from the streams and forwarding the commands to various units of the PPU 400.

[0096] The front-end unit 415 is coupled to a scheduler unit 420, which is configured to schedule various GPCs 450 to process tasks defined by one or more streams. The scheduler unit 420 is configured to track state information related to various tasks managed by the scheduler unit 420. The state can indicate which GPC 450 a task is assigned to, whether the task is active or inactive, a priority associated with the task, and like.

[0097] The scheduler unit 420 is coupled to a work distribution unit 425, which is configured to distribute work to the GPCs 450 for execution. The work distribution unit 425 can track pending tasks received from the scheduler unit 420. In one embodiment, the work distribution unit 425 manages a pending task queue and an active task queue for each GPC 450. When a GPC 450 completes execution of a task, the task is evicted from the GPC's 450 active task queue and another task from the pending task queue is selected and dispatched to the GPC 450 for execution. If the active task on the GPC 450 has been idle, for example, while waiting for a data dependency to be resolved, then the active task can be evicted from the GPC 450 and returned to the pending task queue while another task is selected from the pending task queue and dispatched to the GPC 450 for execution.

[0098] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to execute on the host processor to schedule operations for execution on the PPU 400. In one embodiment, multiple compute applications are executed simultaneously by the PPU 400, and the PPU 400 provides isolation, quality of service (QoS), and independent address spaces for the multiple compute applications. An application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 400. The driver kernel outputs the tasks to one or more streams that are being processed by the PPU 400. Each task can include one or more groups of related threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can be executed in parallel. Cooperative threads can refer to a plurality of threads that execute instructions of a task and can exchange data through shared memory. The task can be assigned to one or more processing units within a GPC 450, and instructions are scheduled for execution by the processing units through at least one warp.

[0099] The work distribution unit 425 communicates with one or more GPCs 450 via the XBar 470. The XBar 470 is an interconnect network that couples many of the units of the PPU 400 to other units of the PPU 400. For example, the XBar 470 can be configured to couple the work distribution unit 425 to a particular GPC 450. Although not explicitly shown, one or more other units of the PPU 400 can also be connected to the XBar 470 via the hub 430.

[0100] Tasks are managed by a scheduler unit 420 and dispatched to GPCs 450 by a work distribution unit 425. GPCs 450 are configured to process tasks and generate results. The results can be consumed by other tasks within GPCs 450, routed to different GPCs 450 via XBar 470, or stored in memory 404. Results can be written to memory 404 via a memory partition unit 480, which implements a memory interface for reading data to and writing data from memory 404. Results can be transmitted to another PPU 400 or CPU via NV link 410. In an embodiment, PPU 400 includes a number U of memory partition units 480 equal to a number of separate and distinct memory devices that are coupled to PPU 400. Each GPC 450 can include a memory management unit to provide translations of virtual addresses into physical addresses, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit provides one or more translation lookaside buffers (TLBs) for instruction and data translation.

[0101] In one embodiment, memory partition unit 480 includes a raster operations (ROP) unit, a level 2 (L2) cache memory, and a memory interface to memory 404. The memory interface can implement a 32, 64, 128, 1024 bit data bus, etc. for high-speed data transfer. PPU 400 can be connected to up to Y memory devices, such as high bandwidth memory stacks or graphics double data rate, version 5, synchronous dynamic random access memory, or other types of persistent storage devices. In an embodiment, the memory interface implements an HBM2 memory interface and Y is equal to one half U. In an embodiment, the HBM2 memory stacks are located on the same physical package as PPU 400, providing significant power and area savings compared to a conventional GDDR5 SDRAM system. In an embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, with each HBM2 stack including two 128 bit channels per die (for a total of 8 channels) and a data bus width of 1024 bits.

[0102] In an embodiment, memory 404 supports single error correction double error detection (SECDED) error correction code (ECC) to protect data. ECC provides higher reliability for data-intensive applications that are sensitive to data corruption. Reliability is particularly important in large cluster computing environments in which PPU 400 processes very large data sets and / or runs applications for extended periods.

[0103] In an embodiment, PPU 400 implements a multi-level memory hierarchy. In one embodiment, memory partition unit 480 supports a unified memory to provide a single unified virtual address space for CPU and PPU 400 memory, enabling data sharing between virtual memory systems. In an embodiment, the frequency of PPU 400 accesses to memory located on other processors is tracked in order to ensure that memory pages which are frequently accessed by PPU 400 are moved to the physical memory of PPU 400. In one embodiment, NVLink 410 supports address translation services, allowing PPU 400 to access page tables of the CPU's memory directly and providing complete access to CPU memory by PPU 400.

[0104] In an embodiment, a copy engine transfers data between multiple PPUs 400 or between a PPU 400 and a CPU. The copy engine can generate a page fault for an address that is not mapped into a page table. Memory partition unit 480 can then service the page fault, map the address into a page table, after which the copy engine can perform the transfer. In a conventional system, for multiple copy engine operations between multiple processors, the memory is pinned (e.g., unpageable), greatly reducing the available memory. With hardware page faults, an address can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0105] Data from memory 404 or other system memory can be fetched by memory partition unit 480 and stored in L2 cache memory 460, which is on-chip and shared among different GPCs 450. As shown, each memory partition unit 480 includes a portion of the L2 cache memory associated with the corresponding memory 404. Lower level caches can then be implemented within each processing unit in GPC 450. For example, each processing unit in GPC 450 can implement a level one (LI) cache. The LI cache is private per processing unit and is not shared. L2 cache 460 is coupled to memory interface 470 and XBar 470, and data from the L2 cache can be fetched and stored into each of the LI caches for processing by the processing units.

[0106] In an embodiment, the processing units within each GPC 450 implement a SIMD (Single Instruction, Multiple Data) architecture wherein a set of threads (for example, a warp) is configured to process different elements of a computational dataset independently. All threads in a warperun the same program, but each thread can be at a different position in the program. A thread group, also referred to as a coherently scheduled thread (CST) group, can be conceptually organized as a number of warps, and each thread in the group can have access to its own private data segment. The threads in a CST group can be scheduled to execute on a number of different cores, and each thread can be given exclusive access to its own private data segment. Each thread can be assigned a unique thread ID. The threads in a thread group can exchange data through memory and through the exchange of messages.

[0107] A cooperative group is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling richer, more efficient parallel decomposition. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. Conventional programming models provide a single, simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (for example, the syncthreads() function). However, programmers often want to define groups of threads at a granularity smaller than a thread block and synchronize within the defined groups to enable higher performance, design flexibility, and software reuse in the form of collective group-wide function interfaces.

[0108] Cooperative groups enable programmers to explicitly define groups of threads at sub-block (for example, as small as a single thread) and multi-block granularities and perform collective operations such as synchronization on threads in a cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across a grid of thread blocks.

[0109] Each processing unit includes a large number (e.g., 128, etc.) of different processing cores (e.g., functional units) that can include single-precision, double-precision, and / or mixed-precision, fully pipelined, and that include floating-point and integer arithmetic logic units. In one embodiment, the floating-point arithmetic logic units implement the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the cores include 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0110] The tensor cores are configured to perform matrix operations. In particular, the tensor cores are configured to perform deep learning matrix operations, such as GEMM (matrix-matrix multiplication) for convolution operations during neural network training and inference. In one embodiment, each tensor core operates on 4x4 matrices and performs a matrix multiplication and accumulation operation D = A x B + C, where A, B, C, and D are 4x4 matrices.

[0111] In an embodiment, the matrix multiplication inputs A and B can be integer, fixed-point, or floating-point matrices, while the accumulation matrices C and D can be integer, fixed-point, or floating-point matrices of equal or higher bit-width. In an embodiment, the tensor cores operate on one, four, or eight-bit integer input data with 32-bit integer accumulation. An 8-bit integer matrix multiplication requires 1024 operations and results in a full-precision product, which is then accumulated with other intermediate products of the 8x8x16 matrix multiplication using 32-bit integer addition. In one embodiment, the tensor cores operate on 16-bit floating-point input data and 32-bit floating-point accumulation. A 16-bit floating-point multiplication requires 64 operations, resulting in a full-precision product, which is then accumulated with other intermediate products of the 4x4x4 matrix multiplication using 32-bit floating-point addition. In practice, the tensor cores are used to perform larger two-dimensional or higher-dimensional matrix operations built up from these smaller elements. An API, such as the CUDA 9 C++ API, exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use the tensor cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16x16 size matrices across all 32 threads of a warp.

[0112] Each processing unit also includes M special function units (SFUs) that perform special functions such as attribute evaluations, inverse square root, etc. In one embodiment, the SFUs can include tree traversal units configured to traverse hierarchical tree data structures. In one embodiment, the SFUs can include texture units configured to perform texture mapping filtering operations. In one embodiment, the texture units are configured to load a texture map (e.g., a 2D array of texture pixels) from memory 404 and sample the texture map to produce a sampled texture value for use in a shader program executed by the processing unit. In one embodiment, the texture map is stored in shared memory which can include or encompass the LI cache. The texture units implement texture operations such as filtering operations using mip maps (i.e., texture maps of different levels of detail). In one embodiment, each processing unit includes two texture units.

[0113] Each processing unit also includes N load / store units (LSUs) that implement load and store operations between shared memory and the register file. Each processing unit includes an interconnect network that connects each core to the register file and connects the LSUs to the register file, shared memory. In one embodiment, the interconnect network is a crossbar that can be configured to connect any core to any register in the register file, and to connect the LSUs to memory locations in the register file and shared memory.

[0114] The shared memory is an on-chip memory array that allows data storage and communication between processing units and threads within a processing unit. In one embodiment, the shared memory includes 128 KB of storage capacity and is in the path from each processing unit to the memory partition unit 480. The shared memory can be used for cache reads and writes. One or more of the shared memory, the LI cache, the L2 cache, and the memory 404 are backing stores.

[0115] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory accesses. This capacity can be used by a program as a cache that does not use shared memory. For example, if the shared memory is configured to use half the capacity, then texture and load / store operations can use the remaining capacity. The integration within the shared memory causes the shared memory to act as a high-throughput pipeline for streaming data, and at the same time provides high-bandwidth and low-latency access to frequently-reused data.

[0116] When configured for general-purpose parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, fixed function graphics processing units are bypassed, creating a simpler programming model. In the general-purpose parallel computing configuration, the work distribution unit 425 assigns and dispatches thread blocks directly to the processing units within the GPC 450. The threads execute the same program, use a unique thread ID in the computation to ensure each thread generates a unique result, use the processing units to execute the program and perform the computation, use shared memory to communicate between threads, and use the LSUs to read and write global memory through shared memory and the memory partition unit 480. When configured for general-purpose parallel computing, the processing units can also write commands to the scheduler unit 420 that the scheduler unit 420 can use to launch new work on the processing units.

[0117] The PPU 400 can each include and / or be configured to perform the functions of one or more processing cores and / or components thereof, such as a tensor core (TC), a tensor processing unit (TPU), a pixel visual core (PVC), a ray tracing (RT) core, a visual processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multi-processor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, etc.

[0118] The PPU 300 can be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smart phone (e.g., a wireless, hand-held device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a hand-held electronic device, etc. In one embodiment, the PPU 300 is contained on a single semiconductor die. In another embodiment, the PPU 300 is included on a system-on-a-chip (SoC) along with one or more other devices, such as an additional PPU 300, a memory 304, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.

[0119] In one embodiment, PPU 300 may be included on a graphics card that includes one or more memory devices 304. The graphics card may be configured to interface with a PCIe slot on a desktop computer's motherboard. In another embodiment, PPU 300 may be an integrated graphics processing unit (iGPU) or parallel processor included in a chipset on the motherboard. In another embodiment, PPU 400 may be implemented in reconfigurable hardware. In another embodiment, portions of PPU 400 may be implemented in reconfigurable hardware.

[0120] Exemplary Computing System

[0121] Systems with multiple GPUs and CPUs are being used across various industries as developers expose and exploit greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0122] FIG. 5A According to one embodiment, the FIG. 4 The exemplary system 500 may be configured to implement the 3D model recovery system 100 and / or FIG. 3 The processing system 500 includes a CPU 530, a switch 510, and a plurality of PPUs 400 and corresponding memories 404.

[0123] NVLink 410 provides a high-speed communication link between each PPU 400. FIG. 5B 4 shows a specific number of NVLink 410 and interconnect 402 connections, but the number of connections connected to each PPU 400 and CPU 530 can vary. Switch 510 interfaces between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLink 410 can be located on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols that interface between various different connections and / or links.

[0124] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and the CPU 530, and the switch 510 interfaces between the interconnect 402 and each PPU 400. The PPUs 400, memory 404, and interconnect 402 may be located on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), the interconnect 402 provides one or more communication links between each PPU 400 and the CPU 530, and the switch 510 interfaces between each PPU 400 using NVLink 410 to provide one or more high-speed communication links between the PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between the PPUs 400 and the CPU 530 through the switch 510. In yet another embodiment (not shown), the interconnect 402 provides one or more communication links directly between each PPU 400. One or more NVLink 410 high-speed communication links may be implemented as a physical NVLink interconnect or as an on-chip or on-die interconnect using the same protocol as NVLink 410 .

[0125] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that emulates on-chip operation and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 525 may be implemented as a circuit board substrate, and each of the PPU 400 and / or memory 404 may be a packaged device. In one embodiment, the CPU 530, switch 510, and parallel processing module 525 are located on a single semiconductor platform.

[0126] In one embodiment, the signaling rate of each NVLink 410 is 20 to 25 Gbit / s, and each PPU 400 includes six NVLink 410 interfaces (e.g., FIG. 5A As shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 Gbit / s in each direction, with six links providing 400 Gbit / s. When the CPU 530 also includes one or more NVLink 410 interfaces, the NVLink 410 can be used exclusively for FIG. 5A PPU to PPU communication shown, or some combination of PPU to PPU and PPU to CPU.

[0127] In one embodiment, the NVLink 410 allows direct loads / stores / atomic accesses from the CPU 530 to the memory 404 of each PPU 400. In one embodiment, the NVLink 410 supports coherency operations allowing data read from the memory 404 to be stored in the cache hierarchy of the CPU 530, reducing cache access latency for the CPU 530. In one embodiment, the NVLink 410 includes support for address translation services (ATS) allowing the PPU 400 to directly access page tables within the CPU 530. One or more NVLinks 410 can also be configured to operate in a low-power mode.

[0128] FIG. 5B An exemplary system 565 is shown in which various previously-embodied architectures and / or functionality can be implemented. The exemplary system 565 can be configured to implement the 3D model recovery system 100 and / or the method 300 shown in FIGS. FIG. 3

[0129] As shown, a system 565 is provided that includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 can directly or indirectly couple one or more of the following: a main memory 540, a network interface 535, one or more CPUs 530, one or more display devices 545, one or more input devices 560, a switch 510, and a parallel processing system 525. The communication bus 575 can be implemented using any suitable protocol, and can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The communication bus 575 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, HyperTransport, and / or another type of bus or link. In some embodiments, there are direct connections between components. For example, the CPU 530 can be directly connected to the main memory 540. Further, the CPU 530 can be directly connected to the parallel processing system 525. Where there are direct or point-to-point connections between components, the communication bus 575 can include a PCIe link used to perform the connection. In these examples, a PCI bus need not be included in the system 565.

[0130] Although FIG. 5B ​The various boxes of the computing device are shown connected by a communications bus 575, but this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component such as display device 545 can be considered an I / O component such as input device 560 (e.g., if the display is a touch screen). As another example, CPU 530 and / or parallel processing system 525 can include memory (e.g., main memory 540 can represent a storage device in addition to parallel processing system 525, CPU 530, and / or other components). In other words, FIG. 5C The computing device of FIG. 1 is illustrative. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “gaming console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of the computing device of FIG. 1. FIG. 5C The computing device of FIG. 1 is illustrative. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “gaming console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of the computing device of FIG. 1.

[0131] System 565 also includes main memory 540. Control logic (software) and data are stored in main memory 540, which can take the form of various computer-readable media. Computer-readable media can be any available media accessible by system 565. Computer-readable media can include both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0132] Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, main memory 540 can store computer readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by system 565. Computer storage media, as used herein, does not include signals per se.

[0133] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above can also be included within the scope of computer readable media.

[0134] The computer programs, when executed, enable the system 565 to perform various functions. The CPU(s) 530 can be configured to execute at least some of the computer-readable instructions to control one or more components of the system 565 to perform one or more of the methods and / or processes described herein. The CPU(s) 530 can each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing multiple software threads concurrently. The CPU(s) 530 can include any type of processors, and can include different types of processors depending on the type of system 565 being implemented (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of system 565, the processor can be an Advanced RISC Machines (ARM) processor implemented using reduced instruction set computing (RISC) or an x86 processor implemented using complex instruction set computing (CISC). The system 565 can include one or more CPU(s) 530 in addition to or in place of one or more microprocessors or co-processors such as mathematical co-processors.

[0135] In addition to or in place of the CPU(s) 530, the parallel processing module(s) 525 can be configured to execute at least some of the computer-readable instructions to control one or more components of the system 565 to perform one or more of the methods and / or processes described herein. The parallel processing module(s) 525 can be used by the system 565 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, the parallel processing module(s) 525 can be used for general-purpose computing on GPUs (GPGPU). In embodiments, the CPU(s) 530 and / or the parallel processing module(s) 525 can execute any combination of the methods, processes, and / or portions thereof discretely or jointly.

[0136] The system 565 also includes one or more input devices 560, parallel processing system 525, one or more display devices 545. The one or more display devices 545 can include a display (e.g., a monitor, a touchscreen, a television screen, a heads-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The one or more display devices 545 can receive data from other components (e.g., parallel processing system 525, CPU 530, etc.) and output the data (e.g., as images, video, sound, etc.).

[0137] The network interface 535 can enable the system 565 to be logically coupled to other devices, including input devices 560, one or more display devices 545, and / or other components, some of which can be built into (e.g., integrated with) the system 565. Illustrative input devices 560 include a microphone, a mouse, a keyboard, a joystick, a game pad, a game controller, a dish antenna, a scanner, a printer, a wireless device, etc. The input devices 560 can provide a natural user interface (NUI) that processes air gestures, speech, or other physiological inputs generated by a user. In some cases, inputs can be transmitted to an appropriate network element for further processing. A NUI can enable a user to interact with the displays associated with the system 565 in a way that is natural and intuitive, for example, by using gestures and / or speech recognition (as described in greater detail below). The system 565 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition. In addition, the system 565 can include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable detection of motion. In some examples, the output of the accelerometers or gyroscopes can be used by the system 565 to render immersive augmented reality or virtual reality.

[0138] Further, the system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through the network interface 535 for communication purposes. The system 565 can be included within a distributed network and / or cloud computing environment.

[0139] The network interface 535 can include one or more receivers, transmitters, and / or transceivers that enable the system 565 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The network interface 535 can include components and functionality to enable communication via any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0140] The system 565 can also include secondary storage (not shown). The secondary storage includes, for example, a hard disk drive and / or a removable storage drive representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash drive. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner. The system 565 can also include a hardwired power supply, a battery power supply, or a combination thereof (not shown). The power supply can provide power to the system 565 to enable the components of the system 565 to operate.

[0141] Each of the foregoing modules and / or devices can even be located on a single semiconductor platform. Alternatively, the different modules can be located on different semiconductor platforms that are communicatively coupled via appropriate circuits / interfaces. While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. Numerous changes to the embodiments can be made in accordance with the disclosure herein without departing from the spirit or scope of the disclosure. Thus, the breadth and scope of the preferred embodiments should not be limited by any of the above described exemplary embodiments, but should be defined in accordance with the following claims and their equivalents.

[0142] Example Network Environment

[0143] A network environment suitable for implementing embodiments of the present disclosure can include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) can be implemented on a processing system 500 and / or an example system 565 of FIG. 5A The processing system 500 and / or the example system 565 of FIG. 5B Each device can include similar components, features, and / or functionality of the processing system 500 and / or the example system 565, for example.

[0144] Components of a network environment can communicate with each other via one or more networks, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet, and / or the public switched telephone network (PSTN), and / or one or more private networks. Where a network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0145] A compatible network environment can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment) and one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein with respect to servers can be implemented on any number of client devices.

[0146] In at least one embodiment, a network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a work scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports one or more applications of a software layer and / or an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a type of free and open-source software web application framework, such as can be used for large-scale data processing (e.g., “big data”) using a distributed file system.

[0147] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functionality described herein (or one or more portions thereof). Any of these different functionalities can be distributed across multiple locations from central or core servers (e.g., one or more data centers that can be distributed across states, regions, countries, the Earth, and the like). Core servers can designate at least a portion of functionality to edge servers if the connection to a user (e.g., a client device) is relatively close to an edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0148] A client device can include FIG. 5A at least some of the components, features, and functions of the example processing system 500 and / or FIG. 5B The client device can be implemented as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a ship, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0149] Machine learning

[0150] Deep neural networks (DNNs) developed on processors such as the PPU 400 have been used for a variety of use cases: from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, learns continuously, gets smarter over time, and delivers more accurate results faster over time. A child is initially taught by an adult to correctly identify and classify various shapes, and eventually is able to identify shapes without any coaching. Similarly, a deep learning or neural learning system needs to be trained in object identification and classification in order to become more intelligent and efficient in identifying basic objects, occluded objects, and the like while also assigning context to the objects.

[0151] At the simplest level, a neuron in the human brain looks at various inputs received, assigns a level of importance to each of these inputs, and passes the output to other neurons for processing. An artificial neuron or perceptron is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs that represent various features of an object that the perceptron is being trained to identify and classify, and each of these features is given a certain weight based on the importance of that feature in defining the shape of the object.

[0152] Deep neural network (DNN) models include multiple layers of connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) that can be trained with large amounts of input data to quickly and accurately solve complex problems. In one example, the first layer of a DNN model breaks down an input image of a car into individual parts and looks for basic patterns such as lines and corners. The second layer assembles the lines to look for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the last few layers generate a label for the input image, identifying a specific make and model of car.

[0153] Once a DNN is trained, it can be deployed and used to recognize and classify objects or patterns in a process known as inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on checks deposited into an ATM machine, recognizing images of friends in a photograph, providing movie recommendations to over 50 million users, recognizing and classifying different types of cars, pedestrians, and road hazards in a self-driving car, or translating human speech in real-time.

[0154] During training, data flows through the DNN in a forward propagation phase until a prediction is made that indicates a label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels the input and other inputs in the training dataset. Training complex neural networks requires a large amount of parallel computing performance, including floating point multiplication and addition supported by PPU 400. Inference is less computationally intensive than training and is a latency-sensitive process in which a trained neural network is applied to new inputs it has not seen before to perform image classification, detect emotions, identify recommendations, recognize and translate speech, and generally infer new information.

[0155] Neural networks rely heavily on matrix math operations, and complex multi-layer networks require a large amount of floating point performance and bandwidth to be efficient and fast. With thousands of processing cores, optimized for matrix math operations, and delivering tens to hundreds of TFLOPS of performance, PPU 400 is a computing platform capable of delivering the performance needed for deep neural network-based artificial intelligence and machine learning applications.

[0156] Further, images generated applying one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for identifying objects and environments in the real world. Such images can include scenes of roads, factories, buildings, urban environments, rural environments, people, animals, and any other physical objects or real-world environments. Such images can be used to train, test, or certify DNNs employed in machines or robots to manipulate, process, or modify physical objects in the real world. Further, such images can be used to train, test, or certify DNNs employed in autonomous vehicles in order to navigate and move the vehicle in the real world. Additionally, images generated applying one or more of the techniques disclosed herein can be used to convey information to users of such machines, robots, and vehicles.

[0157] FIG. 5C Components of an example system 555 that can be used to train and utilize machine learning, in accordance with at least one embodiment, are shown. As will be discussed, various components can be provided by different combinations of computing devices and resources, or a single computing system, which can be under the control of a single entity or multiple entities. Further, aspects can be triggered, initiated, or requested by different entities. In at least one embodiment, training of a neural network can be instructed by a provider associated with a provider environment 506, while in at least one embodiment, training can be requested by a customer or other user accessing the provider environment by a client device 502 or other such resource. In at least one embodiment, training data (or data to be analyzed by a trained neural network) can be provided by a provider, a user, or a third-party content provider 524. In at least one embodiment, a client device 502 can be a vehicle or object that is navigating on behalf of a user, for example, which can submit requests and / or receive instructions to assist in navigation of the device.

[0158] In at least one embodiment, a request can be submitted across at least one network 504 to be received by a provider environment 506. In at least one embodiment, a client device can be any suitable electronic and / or computing device that enables a user to generate and send such a request, such as but not limited to a desktop computer, a notebook computer, a computer server, a smart phone, a tablet computer, a game console (portable or otherwise), a computer processor, computing logic, and a set-top box. Network 504 can include any suitable network for communicating requests or other such data, as can include the Internet, an intranet, an Ethernet, a cellular network, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), an ad hoc network of direct wireless connections between peers, and the like.

[0159] In at least one embodiment, a request can be received at interface layer 508, which can forward data to training and inference manager 532 in this example. Training and inference manager 532 can be a system or service that includes hardware and software for managing requests and servicing corresponding to data or content, in at least one embodiment, training and inference manager 532 can receive a request to train a neural network, and can provide requested data to training module 512. In at least one embodiment, training module 512 can select an appropriate model or neural network to use, if not specified by the request, and can train the model using relevant training data. In at least one embodiment, training data can be a batch of data stored in training data repository 514, received from client device 502, or obtained from third party provider 524. In at least one embodiment, training module 512 can be responsible for training data. A neural network can be any appropriate network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN). Once a neural network is trained and successfully evaluated, a trained neural network can be stored in, for example, model repository 516, which can store different models or networks for users, applications, or services, etc. In at least one embodiment, there can be multiple models for a single application or entity, as can be utilized based on a number of different factors.

[0160] In at least one embodiment, at a subsequent point in time, a request for content (e.g., path determination) or data determined or influenced at least in part by a trained neural network can be received from client device 502 (or another such device). This request can include, for example, input data that is to be processed using a neural network to obtain one or more inference or other output values, classifications, or predictions, or input data can be received by interface layer 508 and directed to inference module 518 for at least one embodiment, although different systems or services can be used. In at least one embodiment, if not already stored locally to inference module 518, inference module 518 can obtain an appropriate trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 516. Inference module 518 can provide data as input to the trained network, which can then generate one or more inferences as output. This can include, for example, a classification of an instance of input data. In at least one embodiment, the inferences can then be communicated to client device 502 for display to a user or other communication. In at least one embodiment, a user’s contextual data can also be stored to user contextual data repository 522, which can include data about a user that can be used as input to a network when generating inferences or determining data to return to a user after obtaining an instance. In at least one embodiment, related data that can include at least some of input or inference data can also be stored to local database 534 for processing of future requests. In at least one embodiment, a user can use account information or other information to access resources or functionality of a provider environment. In at least one embodiment, if permitted and available, user data can also be collected and used to further train models in order to provide more accurate inferences for future requests. In at least one embodiment, requests can be received through a user interface for a machine learning application 526 executing on client device 502, and results displayed through the same interface. A client device can include resources such as a processor 528 and memory 562 for generating requests and processing results or responses, as well as at least one data storage element 552 for storing data of machine learning application 526.

[0161] In at least one embodiment, processor 528 (or processor of training module 512 or inference module 518) will be a central processing unit (CPU). However, as mentioned, resources in such environments can utilize GPUs to process at least some types of data for requests. With thousands of cores, GPUs such as PPU 300 are designed to handle workloads that are substantially parallel, and thus have become popular in deep learning for training neural networks and generating predictions. While using GPUs for offline build has enabled training of larger and more complex models faster, offline generation of predictions means that input features cannot be used as requests come in, or predictions must be generated for all permutations of features and stored in a lookup table in order to service real-time requests. If a deep learning framework supports CPU mode and the model is small and simple enough for a feedforward to execute on a CPU with reasonable latency, then a service on a CPU instance can host the model. In this case, training can be done offline on a GPU, and inference can be done in real-time on a CPU. If a CPU approach is not feasible, then a service can run on a GPU instance. However, because GPUs have different performance and cost characteristics than CPUs, a service that runs runtime algorithms offloaded to a GPU can need to be designed differently than a CPU-based service.

[0162] In at least one embodiment, video data can be provided from client device 502 for augmentation in provider environment 506. In at least one embodiment, video data can be processed for augmentation on client device 502. In at least one embodiment, video data can be streamed from third party content provider 524 and augmented by third party content provider 524, provider environment 506, or client device 502. In at least one embodiment, video data can be provided from client device 502 for use as training data in provider environment 506.

[0163] In at least one embodiment, supervised and / or unsupervised training can be performed by client device 502 and / or provider environment 506. In at least one embodiment, a set of training data 514 (e.g., classified or labeled data) is provided as input to be used as training data. In at least one embodiment, training data can include instances of at least one type of object that a neural network is to be trained on, along with information identifying that type of object. In at least one embodiment, training data can include a set of images each including a representation of one type of object, where each image also includes or is associated with a label, metadata, classification, or other information identifying a type of object represented in the respective image. Various other types of data can also be used as training data, and can include textual data, audio data, video data, etc. In at least one embodiment, training data 514 is provided as training input to a training module 512. In at least one embodiment, training module 512 can be a system or service including hardware and software, such as one or more computing devices executing a training application for training a neural network (or other model or algorithm, etc.). In at least one embodiment, training module 512 receives instructions or requests indicating a type of model to be used for training, which in at least one embodiment can be any appropriate statistical model, network, or algorithm for such purposes, as can include artificial neural networks, deep learning algorithms, learning classifiers, Bayesian networks, etc. In at least one embodiment, training module 512 can select an initial model or other untrained model from an appropriate repository 516, and utilize training data 514 to train the model, thereby generating a trained model (e.g., a trained deep neural network) that can be used to classify similar types of data, or to generate other such inferences. In at least one embodiment in which training data is not used, an appropriate initial model can still be selected for training on input data per training module 512.

[0164] In at least one embodiment, a model can be trained in a variety of different ways, as can depend in part on a type of model selected. In at least one embodiment, a machine learning algorithm can be provided with a training dataset, where a model is a model artifact created by a training process. In at least one embodiment, each instance of training data contains a correct answer (e.g., a classification), which can be referred to as a target or target attribute. In at least one embodiment, a learning algorithm finds patterns in the training data that map input data attributes to the target, the answer to be predicted, and outputs a machine learning model that captures these patterns. In at least one embodiment, a machine learning model can then be used to obtain predictions about new data for which no target was specified.

[0165] In at least one embodiment, training and inference manager 532 can select from a set of machine learning models including binary classification, multi-class classification, generative, and regression models. In at least one embodiment, a type of model to use can depend at least in part on a type of target to be predicted.

[0166] Graphics processing pipeline

[0167] In one embodiment, PPU 400 includes a graphics processing unit (GPU). PPU 400 is configured to receive commands that specify processing of graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quads, triangle strips, etc. Typically, a primitive includes data that specifies a plurality of vertices (e.g., in a model space coordinate system) and attributes associated with each vertex of the primitive. PPU 400 can be configured to process the primitives to generate a frame buffer (e.g., pixel data for each of a plurality of pixels of a display).

[0168] An application writes model data (e.g., a set of vertices and attributes) for a scene to memory, such as system memory or memory 404. The model data defines each of the objects that can be visible on a display. The application then makes an API call to a driver kernel that requests the model data to be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations to process the model data. These commands can reference different shader programs to be implemented on processing units within PPU 400, including one or more of vertex shading, hull shading, domain shading, geometry shading, and pixel shading. For example, one or more of the processing units can be configured to execute a vertex shader program that processes a plurality of vertices defined by the model data. In one embodiment, different processing units can be configured to execute different shader programs at the same time. For example, a first subset of processing units can be configured to execute a vertex shader program while a second subset of processing units can be configured to execute a pixel shader program. The first subset of processing units processes the vertex data to produce processed vertex data and writes the processed vertex data to L2 cache 460 and / or memory 404. After the processed vertex data is rasterized (e.g., converted from three-dimensional data to two-dimensional data in screen space) to produce fragment data, the second subset of processing units executes the pixel shader to produce processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in memory 404. The vertex shader program and the pixel shader program can execute concurrently, processing different data from the same scene in a pipelined fashion until all model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then transferred to a display controller to be displayed on a display device.

[0169] FIG. 6A is implemented by a PPU 400 in accordance with one embodiment FIG. 4 Conceptual diagram of a graphics processing pipeline 600 implemented by the PPU 400 of The graphics processing pipeline 600 is an abstract flowchart of processing steps implemented to generate a 2D computer-generated image from 3D geometric data. As is known, a pipeline architecture can more efficiently perform long-latency operations by breaking the operations into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, the graphics processing pipeline 600 receives input data 601 passed from one stage of the graphics processing pipeline 600 to the next to generate output data 602. In one embodiment, the graphics processing pipeline 600 can represent a graphics processing pipeline defined by an API. Alternatively, the graphics processing pipeline 600 can be implemented in the context of the functionality and architecture of the previous figures and / or one or more any subsequent figures.

[0170] As shown in FIG. 6A The graphics processing pipeline 600 includes a pipeline architecture including multiple stages. These stages include, but are not limited to, a data assembly stage 610, a vertex shading stage 620, a primitive assembly stage 630, a geometry shading stage 640, a viewport scale, cull, and clip (VSCC) stage 650, a rasterization stage 660, a fragment shading stage 670, and a raster operations stage 680. In one embodiment, the input data 601 includes commands that configure the processing units to implement the stages of the graphics processing pipeline 600 and configure geometric primitives (e.g., points, lines, triangles, quads, triangle strips, or fans, etc.) to be processed by the stages. The output data 602 can include pixel data (i.e., color data) that is copied to a frame buffer in memory or other type of surface data structure.

[0171] The data assembly stage 610 receives input data 601 that specifies vertex data for high-order surfaces, primitives, etc. The data assembly stage 610 collects vertex data in temporary storage or a queue, such as by receiving a command from a host processor that includes a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then passed to the vertex shading stage 620 for processing.

[0172] The vertex shading stage 620 processes vertex data by performing a set of operations (e.g., a vertex shader or program) on each of the vertices. A vertex can be specified, for example, as a 4-coordinate vector (e.g., <x, y, z, w>) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normal, etc.). The vertex shading stage 620 can manipulate individual vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 620 performs operations on vertex coordinates or other vertex attributes associated with a vertex. These operations typically include lighting operations (e.g., modifying a color attribute of a vertex) and transformation operations (e.g., modifying a coordinate space of a vertex). For example, a vertex can be specified using coordinates in an object coordinate space, which are transformed by multiplying the coordinates by a matrix that converts the coordinates from the object coordinate space to a world space or a normalized-device-coordinate (NDC) space. The vertex shading stage 620 generates transformed vertex data that is passed to the primitive assembly stage 630.

[0173] The primitive assembly stage 630 collects vertices output by the vertex shading stage 620 and groups the vertices into geometric primitives for processing by the geometry shading stage 640. For example, the primitive assembly stage 630 can be configured to group every three consecutive vertices into a geometric primitive (e.g., a triangle) for passing to the geometry shading stage 640. In some embodiments, particular vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). The primitive assembly stage 630 passes the geometric primitives (e.g., a set of associated vertices) to the geometry shading stage 640.

[0174] The geometry shading stage 640 processes the geometric primitives by performing a set of operations (e.g., a geometry shader or program) on the geometric primitives. Tessellation operations can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 640 can tessellate each geometric primitive into a finer mesh of two or more geometric primitives for processing by the remainder of the graphics processing pipeline 600. The geometry shading stage 640 passes the geometric primitives to the viewport SCC stage 650.

[0175] In one embodiment, graphics processing pipeline 600 can perform processing operations in sequence at a stream multiprocessor and a vertex shading stage 620, a primitive assembly stage 630, a geometry shading stage 640, a fragment shading stage 670, and / or hardware / software internal operations associated therewith. Once the sequential processing operations are complete, in one embodiment, viewport SCC stage 650 can utilize the data. In one embodiment, primitive data processed by one or more of the stages in graphics processing pipeline 600 can be written into a cache (e.g., an LI cache, a vertex cache, etc.). In this case, in one embodiment, viewport SCC stage 650 can access the data in the cache. In one embodiment, viewport SCC stage 650 and rasterization stage 660 are implemented as fixed function circuitry.

[0176] Viewport SCC stage 650 performs viewport scaling, culling, and clipping of the geometric primitives. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of a viewer who is looking at the scene and defines a view frustum that encloses the objects of the scene. The view frustum can include a viewing plane, a back plane, and four clipping planes. Any geometric primitive that is completely outside of the view frustum can be culled (e.g., discarded) because it will not contribute to the final rendered scene. Any geometric primitive that is partially inside the view frustum and partially outside the view frustum can be clipped (e.g., converted to new geometric primitives that are enclosed within the view frustum). In addition, each geometric primitive can be scaled based on the depth of the view frustum. All potentially visible geometric primitives are then passed to rasterization stage 660.

[0177] Rasterization stage 660 converts 3D geometric primitives into 2D fragments (e.g., capable of being used for display, etc.). Rasterization stage 660 can be configured to set up a set of plane equations with the vertices of the geometric primitive from which various attributes can be interpolated. Rasterization stage 660 can also compute a coverage mask for a plurality of pixels that indicates whether one or more sample locations of the pixel intercept the geometric primitive. In one embodiment, a z-test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have already been rasterized. Rasterization stage 660 generates fragment data (e.g., interpolated vertex attributes associated with particular sample locations of each covered pixel) that is passed to fragment shading stage 670.

[0178] Fragment shading stage 670 processes the fragment data by performing a set of operations (e.g., a fragment shader or program) on each of the fragments. Fragment shading stage 670 can generate pixel data (e.g., color values) for the fragments, such as by performing lighting operations or sampling texture mappings using interpolated texture coordinates for the fragments. Fragment shading stage 670 generates pixel data that is sent to raster operations stage 680.

[0179] The raster operations stage 680 can perform various operations on the pixel data, such as performing alpha tests, stencil tests, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 680 has completed processing of the pixel data (e.g., output data 602), the pixel data can be written to a render object, such as a frame buffer, color buffer, etc.

[0180] It should be appreciated that one or more additional stages can be included in the graphics processing pipeline 600 in addition to or instead of one or more of the stages described above. Various implementations of an abstract graphics processing pipeline can implement different stages. Further, in some embodiments, one or more of the stages described above can be excluded from the graphics processing pipeline (such as the geometry shading stage 640). Other types of graphics processing pipelines are contemplated as being within the scope of the present disclosure. Further, any of the stages of the graphics processing pipeline 600 can be implemented by one or more specialized hardware units within a graphics processor, such as the PPU 400. Other stages of the graphics processing pipeline 600 can be implemented by programmable hardware units, such as processing units within the PPU 400.

[0181] The graphics processing pipeline 600 can be implemented via an application program executed by a host processor, such as a CPU. In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that can be utilized by an application program to generate graphics data for display. The device driver is a software program that includes a plurality of instructions that control the operation of the PPU 400. The API provides an abstraction for programmers that allows programmers to utilize specialized graphics hardware, such as the PPU 400, to generate graphics data without requiring the programmer to utilize the specific instruction set of the PPU 400. The application program can include API calls that are routed to the device driver of the PPU 400. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform operations at least partially by initiating operations on the PPU 400 utilizing an input / output interface between the CPU and the PPU 400. In one embodiment, the device driver is configured to utilize the hardware of the PPU 400 to implement the graphics processing pipeline 600.

[0182] Various programs can be executed within PPU 400 in order to implement the various stages of graphics processing pipeline 600. For example, a device driver can launch a kernel on PPU 400 to execute the vertex shading stage 620 on one or more processing units. The device driver (or an initial kernel executed by PPU 400) can also launch further kernels on PPU 400 to execute the other stages of the graphics processing pipeline 600, such as the geometry shading stage 640 and the fragment shading stage 670. Additionally, some of the stages of the graphics processing pipeline 600 can be implemented on fixed function hardware units such as a rasterizer or a data assembler implemented within PPU 400. It will be appreciated that the results of one kernel can be processed by one or more intermediate fixed function hardware units prior to being processed by a subsequent kernel on a processing unit.

[0183] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device can be directly coupled to the system or processor that produces or renders the images. In other embodiments, the display device can be indirectly coupled to the system or processor, e.g., via a network. Examples of such networks include the Internet, mobile telecommunication networks, WIFI networks, and any other wired and / or wireless networking systems. When the display device is indirectly coupled, the images produced by the system or processor can be streamed to the display device via the network. Such streaming allows, for example, a video game or other application that renders images to be executed on a server, in a data center, or in a cloud-based computing environment, and the rendered images will be transmitted and displayed on one or more user devices (such as computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separate from the server or data center. Thus, the techniques disclosed herein can be applied to enhance streamed images as well as to enhance services that stream images, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0184] Example game streaming system

[0185] FIG. 6B is an example system diagram of a game streaming system 605 in accordance with some embodiments of the present disclosure. FIG. 6B includes one or more game servers 603 (which can include components, features, and / or functionality similar to those of the example processing system 500 and / or FIG. 5A includes one or more game servers 603 (which can include components, features, and / or functionality similar to those of the example processing system 500 and / or FIG. 5B includes one or more game servers 603 (which can include components, features, and / or functionality similar to those of the example processing system 500 and / or FIG. 5A includes one or more game servers 603 (which can include components, features, and / or functionality similar to those of the example processing system 500 and / or FIG. 5BThe example system 565 can include one or more client devices 604 (which can be similar to one or more client devices described herein), one or more game servers 603 (which can be similar to one or more game servers described herein), and one or more networks 606 (which can be similar to one or more networks described herein). In some embodiments of the disclosure, the system 605 can be implemented.

[0186] In the system 605, for a game session, the one or more client devices 604 can receive input data in response to input to one or more input devices, transmit the input data to the one or more game servers 603, receive encoded display data from the one or more game servers 603, and display the display data on the display 624. As such, computationally more intensive computation and processing is offloaded to the one or more game servers 603 (e.g., rendering— specifically ray or path tracing— of graphical output of the game session is performed by one or more GPUs of the one or more game servers 603). In other words, the game session is streamed from the one or more game servers 603 to the one or more client devices 604, thereby reducing the requirements of the one or more client devices 604 for graphics processing and rendering.

[0187] For example, with respect to an instance of a game session, the client device 604 can display frames of the game session on the display 624 based on receiving display data from the one or more game servers 603. The client device 604 can receive input to one of the one or more input devices and generate input data in response. The client device 604 can transmit the input data to the one or more game servers 603 via the communication interface 621 and via the one or more networks 606 (e.g., the Internet), and the one or more game servers 603 can receive the input data via the communication interface 618. The CPU can receive the input data, process the input data, and transmit the data to the GPU, causing the GPU to produce a rendering of the game session. For example, the input data can represent movement of a user’s character in a race, firing a weapon, reloading, passing a ball, turning a vehicle, etc. The rendering component 612 can render the game session (e.g., representing the results of the input data), and the rendering capture component 614 can capture the rendering of the game session as display data (e.g., capturing image data of the rendered frames of the game session). The rendering of the game session can include lighting and / or shadow effects computed using ray or path tracing with one or more parallel processing units (such as GPUs), which can further employ one or more specialized hardware accelerators or processing cores to perform ray or path tracing techniques of the one or more game servers 603. The encoder 616 can then encode the display data to produce encoded display data, and the encoded display data can be transmitted to the client device 604 via the network 606 via the communication interface 618. The client device 604 can receive the encoded display data via the communication interface 621, and the decoder 622 can decode the encoded display data to generate display data. The client device 604 can then display the display data via the display 624.

[0188] It is noted that the techniques described herein can be implemented in executable instructions stored in a computer-readable medium for use by or in conjunction with a processor-based instruction execution machine, system, device, or apparatus. Those skilled in the art will appreciate that for some embodiments, different types of computer-readable media may be included for storing data. As used herein, "computer-readable medium" includes one or more of any suitable media for storing executable instructions of a computer program, such that an instruction execution machine, system, apparatus, or device can read (or obtain) instructions from the computer-readable medium and execute instructions for implementing the described embodiments. Suitable storage formats include one or more of electronic formats, magnetic formats, optical formats, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable compact disks (CDs), portable digital video disks (DVDs), and the like.

[0189] It should be understood that the arrangement of the components shown in the drawings is for illustrative purposes and that other arrangements are possible. For example, one or more of the elements described herein may be implemented as an electronic hardware assembly in whole or in part. Other elements may be implemented in software, hardware, or a combination of software and hardware. In addition, some or all of these other elements may be combined, some elements may be omitted entirely, and additional components may be added while still implementing the functions described herein. Thus, the subject matter described herein may be embodied in many different variations, and all such variations are contemplated to be within the scope of the claims.

[0190] For ease of understanding of the subject matter described herein, many aspects have been described with respect to action sequence. It will be appreciated by those skilled in the art that different actions can be performed by a dedicated circuit or circuit, by a program instruction executed by one or more processors, or by a combination of the two. The description of any action sequence herein is not intended to imply that the described particular order for executing the sequence must be followed. Unless otherwise indicated herein or context clearly contradicts, all methods described herein can be carried out in any suitable order.

[0191] The use of the terms “a” and “the” and similar referents in the context of describing the subject matter (especially in the context of the following claims) are to be interpreted in an inclusive sense, i.e., the inclusion of both singular as well as the plural or plural versions are to be read. The use of the term “at least one” followed by a list of one or more items (for example, “at least one of A and B”) is to be interpreted as meaning one item from the list of A or B or any combination of the items in the list A and B, unless otherwise indicated herein or clearly contradicted by context. Furthermore, the foregoing description is for the purpose of illustration only, and not for the purpose of limitation, as the scope of the sought protection is defined by the claims as set forth, and any equivalents thereof, below. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended merely to better illuminate examples of the subject matter and does not pose a limitation on the scope of the subject matter unless otherwise claimed. The use of the terms “based on” and other like phrases in the claims and written description is not intended to mean that the claimed subject matter is based on only a stated value, but rather “based on” is intended to mean that the claimed subject matter is based on at least one stated value, and is not meant to exclude any other circumstances. The language in this specification should not be interpreted to require that any undescribed element be an essential element of the applications as claimed.

Claims

1. A computer-implemented method for constructing a three-dimensional (3D) model, comprising: rendering an image of a 3D model defined by a base mesh, wherein an anti-aliasing operation performed by a rendering pipeline processes data associated with a geometry to anti-alias the image; determining, based on the data associated with the geometry, that a silhouette edge intersects a pair of adjacent pixels; computing an intersection point resulting from the intersection of a line segment connecting the pair of adjacent pixels with the silhouette edge, wherein the line segment connects a first sample location within a first pixel of the pair of adjacent pixels and a second sample location within a second pixel of the pair of adjacent pixels; coloring the pair of adjacent pixels based on the intersection point; computing a pixel difference based on the image and a reference image; backpropagating, through the anti-aliasing operation of the rendering pipeline, the pixel difference to compute a geometry change corresponding to a reduction in the pixel difference; adjusting, based on the geometry change, weights used to deform the base mesh, wherein the weights are applied to the base mesh by a deformation network to produce a geometry of the 3D model.

2. The computer-implemented method of claim 1, wherein the geometry is defined by vertex positions.

3. The computer-implemented method of claim 1, further comprising: repeating the rendering, the computing, the propagating, and the adjusting for at least one additional reference image.

4. The computer-implemented method of claim 3, wherein the reference image and the at least one additional reference image are each associated with a different camera position.

5. The computer-implemented method of claim 1, wherein propagating the pixel difference further comprises: producing a surface texture corresponding to the 3D model.

6. The computer-implemented method of claim 5, wherein the surface texture represents at least one of a lighting or material property of the 3D model.

7. The computer-implemented method of claim 1, wherein the anti-aliasing operation comprises: computing blending weights between the pair of adjacent pixels; and adjusting colors of the pair of adjacent pixels according to the blending weights. identifying the silhouette edge between a first rendered geometry associated with a first identifier and a second rendered geometry associated with a second identifier.

8. The computer-implemented method of claim 1, wherein the determining comprises:

9. The computer-implemented method of claim 1, wherein the anti-aliasing operation approximates a pixel integral based on a location of a silhouette edge within a pixel.

10. The computer-implemented method of claim 1, wherein at least one of the rendering, the computing, the propagating, and the adjusting steps are performed within a cloud computing environment.

11. The computer-implemented method of claim 1, wherein at least one of the rendering, the computing, the propagating, and the adjusting steps are performed on a server or in a data center to generate the image and the 3D model, and at least one of the image and the 3D model is streamed to a user device.

12. The computer-implemented method of claim 1, wherein at least one of the rendering, the computing, the propagating, and the adjusting steps are performed for training, testing, or certifying a neural network employed in a machine, a robot, or an autonomous vehicle. ​ 13. The computer-implemented method of claim 1, wherein at least one of the rendering, the computing, the propagating, and the adjusting steps are performed on a virtual machine comprising a portion of a graphics processing unit.

14. A system comprising: a processor configured to: render an image of a three-dimensional (3D) model defined by a base mesh, wherein an anti-aliasing operation performed by a rendering pipeline processes data associated with a geometry to anti-alias the image; determine, based on the data associated with the geometry, that a silhouette edge intersects a pair of adjacent pixels; compute an intersection point resulting from an intersection of a line segment connecting the pair of adjacent pixels with the silhouette edge, wherein the line segment connects a first sample location within a first pixel of the pair of adjacent pixels and a second sample location within a second pixel of the adjacent pixels; shade the pair of adjacent pixels based on the intersection point; compute a pixel difference based on the image and a reference image; back-propagate, through the anti-aliasing operation of the rendering pipeline, the pixel difference to compute a geometry change corresponding to a reduction in the pixel difference; adjust, based on the geometry change, weights used to deform the base mesh, wherein the weights are applied to the base mesh by a deformation network to produce a geometry of the 3D model.

15. The system of claim 14, wherein the geometry is defined by vertex positions.

16. The system of claim 14, wherein the processor is configured to repeat the rendering, the computing, the propagating, and the adjusting for at least one additional reference image.

17. The system of claim 14, wherein the processor is configured to propagate the pixel difference by producing a surface texture corresponding to the 3D model.

18. A non-transitory computer-readable medium storing computer instructions for three-dimensional (3D) model construction that, when executed by one or more processors, cause the one or more processors to perform the following steps: render an image of a 3D model defined by a base mesh, wherein an anti-aliasing operation performed by a rendering pipeline processes data associated with a geometry to anti-alias the image; determine, based on the data associated with the geometry, that a silhouette edge intersects a pair of adjacent pixels; compute an intersection point resulting from an intersection of a line segment connecting the pair of adjacent pixels with the silhouette edge, wherein the line segment connects a first sample location within a first pixel of the pair of adjacent pixels and a second sample location within a second pixel of the adjacent pixels; shade the pair of adjacent pixels based on the intersection point; compute a pixel difference based on the image and a reference image; back-propagate, through the anti-aliasing operation of the rendering pipeline, the pixel difference to compute a geometry change corresponding to a reduction in the pixel difference; adjust, based on the geometry change, weights used to deform the base mesh, wherein the weights are applied to the base mesh by a deformation network to produce a geometry of the 3D model.

Images