Joint shape and appearance optimization by topology sampling

Abstract

Disclosed is junction shape and appearance optimization by topological sampling, systems and methods presented implement optimization of 3D model representations that include the shape and appearance of specific 3D scenes or objects. An opaque 3D mesh (e.g., vertex positions and corresponding topology) and spatially-varying material attributes are jointly optimized based on image-space losses to match multiple image observations (e.g., reference images of a reference 3D scene or object). A geometric topology defines faces and / or cells in the opaque 3D mesh that are visible and can be randomly initialized and optimized by training based on the image-space losses. Applying a geometric topology to an opaque 3D mesh to learn the shape improves the accuracy and performance of silhouette edges compared to using a transparent mesh representation. The 3D model representation learns based on image-space differences without requiring an initial guess of the topology compared to methods that require an initial guess of the topology and / or an exhaustive test of possible geometric topologies.

Description

[0001] Claiming priority

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 195,301, filed June 1, 2021, entitled “Joint Shape and Appearance Optimization Through Monte Carlo Sampled Topologies,” the entire contents of which are incorporated herein by reference. Background Technology

[0003] Synthesizing images of objects with complex shapes and appearances is a core goal of computer graphics. The problem can be broken down into selecting a suitable representation for the shape and appearance of the object, modeling a 3D scene based on the selected representation, and finally efficiently rendering the scene. Creating a shape and appearance model for a specific 3D scene is essentially an inverse problem: finding a 3D model representation that, when rendered, will produce a 2D image that appears on demand. In multiple iterations, inverse rendering is used to iteratively recover the shape, lighting, and material properties of the 3D model based on a reference image of the 3D scene being modeled.

[0004] Existing techniques typically rely on an initial volume mesh or 3D model (cube, sphere, etc.) determined at least in part based on a reference image (or reference 3D model). Other techniques depend on training using all possible topologies of a unified volume mesh, which is costly in terms of execution time. For example, the depth stepping cube technique evaluates 140 possible topologies for each cube in a unified 3D mesh. Other techniques produce 3D models with inaccurate geometry. There is a need to address these and / or other problems associated with existing techniques. Summary of the Invention

[0005] Embodiments of this disclosure relate to the optimization of junction shape and appearance through topological sampling. Systems and methods are disclosed for optimizing 3D model representations that include the shape and appearance of a particular 3D scene or object. In one embodiment, an initial guess of the geometric topology using some knowledge of the 3D scene or object is not required. In one embodiment, the geometric topology defines the cells (e.g., 3D meshes, packed tetrahedrons) and / or the faces (e.g., triangles, quadrilaterals, etc.) of visible (e.g., enabled, active, included) volume meshes. The geometric topology can be randomly initialized and optimized through training based on image spatial loss. Applying the geometric topology to an opaque 3D mesh to learn the shape improves the accuracy and performance of contour edges compared to using a transparent mesh representation. The geometric topology is learned based on image spatial difference without initial guessing, compared to methods that require an initial guess of the topology and / or exhaustive testing of possible geometric topologies. Appearance-driven 3D model extraction has many uses, such as creating renderings of complex scenes at multiple levels of detail, conversions between different rendering systems, and even conversions between different geometric scene representations.

[0006] In one embodiment, the material properties of the spatial variations of the opaque 3D mesh (e.g., the vertex locations and the corresponding topology of the faces defining the connections between these vertex locations to form a surface or volume) and the 3D model representation are jointly optimized based on an image spatial loss to match multiple image observations (e.g., a reference image of the reference 3D scene or object). The 3D model representation is rendered by a differentiable renderer to produce a rendered image corresponding to the reference image. The image spatial loss is calculated based on computed probability values ​​associated with faces included in the volume mesh and the difference between the reference image and the rendered image. Higher probability values ​​correspond to a higher likelihood that the face is included in the 3D model representation. Therefore, faces or cells associated with probability values ​​of zero (or below a minimum threshold) can be removed from the 3D model representation (cells not inserted based on said probability values). The image spatial loss is backpropagated to update the parameters learned for generating the 3D model representation and the probability values.

[0007] A method, computer-readable medium, and system for optimizing the shape and appearance of a joint through topological sampling are disclosed. In one embodiment, a method for constructing a 3D model includes: receiving a 3D topological definition of faces of cells within a volume mesh and processing the 3D topology and the volume mesh according to parameters to predict a 3D model representation corresponding to a reference 3D model, wherein the 3D model representation includes a surface mesh. In one embodiment, the surface mesh is a list of vertex locations of the faces and a topology defining the connections between active faces of the cells. During optimization, the volume mesh is modified, and after optimization, the volume mesh is converted to a surface mesh by removing faces within the volume (only those faces on the surface mesh are retained). In one embodiment, faces in the volume mesh associated with learned probability values ​​below a threshold are removed to produce the surface mesh. In one embodiment, the surface mesh is defined by triangular faces. In one embodiment, the 3D model representation is rendered to produce a rendered image, and rays corresponding to the rendered image are transmitted through the 3D model representation to compute the probability value of the cell. In one embodiment, parameters are updated based on the probability value and the difference between the rendered image and a reference image of the reference 3D model to reduce the difference. Attached Figure Description

[0008] The system and method for shape and appearance optimization through topology sampling are described in detail below with reference to the accompanying drawings, wherein:

[0009] Figure 1A A conceptual diagram of the topology and probabilistic data used to learn the topology, according to an embodiment, is shown.

[0010] Figure 1B A conceptual diagram is shown, illustrating a 3D model representation, a learned probability density, a rendered 3D model representation, and a rendered reference object, according to an embodiment.

[0011] Figure 1C A conceptual diagram illustrating the probability density of learning, the rendered 3D model representation, and the learned surface mesh according to an embodiment is shown.

[0012] Figure 1D A conceptual diagram of a shape- and appearance-driven automated 3D modeling system according to an embodiment is shown.

[0013] Figure 1E A flowchart is shown of a method for shape and appearance-driven automated 3D modeling suitable for implementing some embodiments of this disclosure.

[0014] Figure 2A A flowchart is shown of a method for rendering probability values ​​of a sampled topology suitable for implementing some embodiments of this disclosure.

[0015] Figure 2BA flowchart is shown for a method for updating parameters based on probability value image spatial difference, suitable for implementing some embodiments of this disclosure.

[0016] Figure 2C A flowchart is shown for a method for topology importance sampling based on probability values, suitable for implementing some embodiments of this disclosure.

[0017] Figure 3A A conceptual diagram of coarse-to-fine layered training according to an embodiment is shown.

[0018] Figure 3B Conceptual diagrams of a learned surface mesh, a rendered 3D model representation, and a rendered reference 3D model, according to an embodiment, are shown.

[0019] Figure 3C A flowchart is shown of another method for shape and appearance-driven automated 3D modeling suitable for implementing some embodiments of this disclosure.

[0020] Figure 4 Example parallel processing units suitable for implementing some embodiments of this disclosure are shown.

[0021] Figure 5A For the purpose of implementing some embodiments of this disclosure, the use of Figure 4 A conceptual diagram of the processing system implemented by the PPU.

[0022] Figure 5B Exemplary systems in which various architectures and / or functions of various prior embodiments can be implemented are shown.

[0023] Figure 5C Components of an exemplary system that can be used to train and utilize machine learning in at least one embodiment are shown.

[0024] Figure 6A For the purpose of implementing some embodiments of this disclosure, the use of Figure 4 A conceptual diagram of the graphics processing pipeline implemented by the PPU.

[0025] Figure 6B An exemplary game streaming system suitable for implementing some embodiments of this disclosure is shown. Detailed Implementation

[0026] Systems and methods related to optimizing the shape and appearance of junctions via topological sampling are disclosed. The 3D model representation comprises the shape and appearance of a specific 3D scene or object from multiple image observations. This 3D model representation is generated by a 3D model generation unit that learns shape and appearance parameters using a reference image and randomized topology at each training iteration (e.g., step). In one embodiment, multiple randomized topologies are sampled to learn the shape and appearance parameters. The topology is specified by a mask indicating which faces (e.g., triangles, quadrilaterals, etc.) or cells in a volume mesh (e.g., a 3D mesh, a filled tetrahedron) are included in the topology (e.g., visible, enabled, active, covered). The vertex positions of the volume mesh are learned to define the shape of the 3D model representation of the topology. Appearance properties (e.g., spatially varying material properties such as color, normals, texture, roughness, metallicity, etc.) are also learned to define the appearance of the 3D model representation. The appearance properties can be used to render a surface mesh defined by the volume mesh to produce an image. The parameters used by the 3D model generation unit to generate a 3D model representation are updated based on the image spatial loss (e.g., pixel difference) calculated between the rendered image of the 3D model representation and a reference image of the 3D scene or object.

[0027] Figure 1A A conceptual diagram of a topology 105 according to an embodiment and probability densities 110 and 115 for learning the topology is shown. For illustrative purposes, 2D examples are described instead of 3D examples. A reference image 125 of a 2D object is used to learn the parameters needed to generate a 2D model representation that is visually identical or nearly identical to the reference image 125 when rendered.

[0028] Topology 105 identifies which cells in a 2D mesh 100 (e.g., a grid) are included in the 2D model representation. For example... Figure 1A As shown, each cell is a square in a two-dimensional array. In another embodiment, mesh 100 is a 3D structure, such as an array, and the cells are 3D geometries, such as cubes, tetrahedrons, etc. In one embodiment, topology 105 can be defined using a single bit for each cell, where the bit indicates whether the cell is included (enabled) in the 2D model representation or removed (disabled) from the 2D model representation. For topology 105, each included cell of mesh 100 is shown as solid black and removed cells of mesh 100 are shown as solid white. In one embodiment, topology 105 can be encoded as an "overlay" mask that includes bits for each cell. In one embodiment, topology 105 can be defined using a single bit for each face (e.g., overlay) of each cell, where the bit indicates whether the face is included (enabled) in the 2D model representation or removed (disabled) from the 2D model representation.

[0029] In one embodiment, topology 105 is randomly generated and does not depend on any information about reference image 125 or the object being modeled. In another embodiment, a different single topology 105 is used for each training iteration. Once the topology 105 is provided, a 2D model representation is predicted by applying the learned parameters to mesh 100 and topology 105. The 2D model representation includes the vertex positions and material properties of the enabled cells. The 2D model representation is rendered according to environmental conditions to produce a rendered image. The environmental conditions correspond to the camera position and lighting used to render reference image 125. When multiple reference images are available, training iterations can be completed for each reference image using topology 105 or different topologies.

[0030] For each training iteration, the loss is calculated based on the per-pixel image spatial difference between each reference image and the rendered image. To update the 2D model parameters (e.g., the learned parameters), the gradient corresponding to the loss is backpropagated (e.g., effectively reversed), which requires the renderer to be differentiable. Unfortunately, binary values, such as those encoding topology 105, are not differentiable. The probability density is not a function that complicates the topological representation to make it differentiable, but rather a continuous function computed for grid 100 and differentiable. For example, the probability of a particular unit included in the model representation can have several values ​​between 0 and 1 (inclusive).

[0031] The probability values ​​and the parameters used to calculate them (e.g., vertex positions and material parameters) are updated based on the calculated loss. In one embodiment, the probability value is associated with each cell of mesh 100. In another embodiment, the probability value is associated with each face of each cell of mesh 100. The probability values ​​in probability density 110 can be initialized to predetermined values ​​or random values ​​updated for each training iteration. Figure 1A As shown, each probability value is associated with a cell (or a face) and is initialized to 0.5, where a lower probability value (darker) indicates that the cell is unlikely to be active in the learned topology, while a higher probability value (lighter) indicates that the cell is more likely to be active in the learned topology.

[0032] For each pixel of the rendered image, rays are traced within the 2D plane of grid 100 to calculate a probability value per pixel. When grid 100 is a 3D grid, the rays are traced in 3D and are not limited to the 2D plane. Figure 1AAs shown, the first ray 102 intersects several uncovered cells before intersecting with the covered cell 112. The intersecting cells are shown in a diagonal pattern. The second ray 103 intersects one uncovered cell before intersecting with the covered cell 113. Because the grid 100 is opaque, the ray terminates when it intersects with a covered cell, or if the ray passes through the entire grid 100 without intersecting with a covered cell, then the ray terminates. In one embodiment, cell intersections are preserved to update probability values ​​during backpropagation. In another embodiment, cell intersections are not preserved; instead, the ray is traced back to update probability values ​​during backpropagation.

[0033] The rendered image is compared to reference image 125 to calculate the difference per pixel. For each pixel, this difference is multiplied by the probability value corresponding to that pixel to generate a product. These products are used to calculate the image layer loss. As training progresses and probability values ​​are updated to reduce the loss, pixels with higher probability values ​​contribute more to the image layer loss compared to pixels with lower probability values. After at least one training iteration, the updated probability density 115 shows that intersecting cells are updated to reduce the loss. Specifically, the probability values ​​of cells 112 and 114 corresponding to cells within objects in the reference image are increased, and the probability values ​​of intersecting cells not within objects are decreased. After several training iterations, the updated probability density 115 will converge to equal the learned topology 120 after optimization. In the learned topology 120, covered (or active) cells are indicated as black and empty cells are indicated as white. The learned topology 120 also represents the learned topology. The resolution increase of mesh 100 will cause the learned topology 120 to more closely replicate the fine contour details of objects in reference image 125.

[0034] Even when training relies on a single topology, such as topology 105 used for training, when using multiple reference images, the probability values ​​of cells not covered by topology 105 will increase, and as the loss computed for each of the multiple reference images is backpropagated to update the probability density corresponding to mesh 100, the probability values ​​of cells covered by topology 105 will decrease. Furthermore, the vertex locations of cells covered by topology 105 will change as training progresses to converge and approximate the shape of the object in reference image 125. Cells enabled by topology 105 are effectively deformed, thereby ensuring that the rendered image of the 3D model representation generated for mesh 100 and topology 105 matches the reference image 125. Similarly, the material properties of the 3D model representation also evolve with multiple training iterations.

[0035] Further illustrative information will now be provided regarding the various optional architectures and features that can be used to implement the aforementioned framework according to user expectations. It should be strongly noted that the following information is presented for illustrative purposes and should not be construed as restrictive in any way. Optionally, any of the following features may be incorporated, with or without excluding the other features described.

[0036] Figure 1B A conceptual diagram is shown, illustrating a 3D model representation 122, a learned probability density 126, a rendered 3D model representation 124, and a rendered reference object 158, according to one embodiment. The 3D model representation 122 defines the shape and appearance of a specific 3D scene or object from multiple image observations. The vertex positions of a volume mesh 121 (e.g., a dense 3D mesh) are deformed to define the shape of at least one topological 3D model representation 122. The topology controls which faces and / or cells of the volume mesh 121 are visible (e.g., active) and can be selected randomly or based on the probability density learned so far. The volume mesh 121 can be initialized as a uniform 3D mesh and learned during training. A surface mesh 132 can be extracted from the volume mesh 121. The shape of the surface mesh 132 is learned to closely match the shape of the rendered reference object 158. Appearance attributes 136 (e.g., spatially varying material parameters) are also learned to define the appearance of the 3D model representation 122. The 3D model representation 146 effectively includes the learned volume mesh 121, the learned surface mesh 132, and the learned appearance properties 136.

[0037] The 3D model representation 122 is rendered by a differentiable renderer to produce a rendered image corresponding to the reference image. For example... Figure 1B As shown, in the early stages of the training process, the rendered 3D model representation 124 is not very close to the rendered reference object 158. Based on the topology of the 3D model representation 122 used to learn the rendered representation 124, only a subset of cell faces are visible in the volume mesh 121, thus important parts of the object are missing or unknown. For example, 500 training iterations may have been performed to produce the volume mesh 121 associated with the rendered 3D model representation 124 and the learned probability density 126.

[0038] The volume mesh 121 is rendered by a probabilistic renderer based on the topology to produce a learned probability density 126. For example, the probability value calculated for each visible cell represents the probability that a ray will intersect with the cell, and each ray is traced until it intersects with cells (e.g., active cells) in the volume mesh 121 that are included in the topology (or the ray traverses the entire topology). In other words, the topology and the resulting learned surface mesh 132 are opaque, and tracing terminates at the first intersection.

[0039] The probability density 126 includes a probability value for each face or cell in the volume mesh. Lighter colors correspond to higher probability values, and darker colors correspond to lower probability values. Higher probability values ​​correspond to a higher likelihood that a cell in the volume mesh 121 will be included in the 3D model representation. Therefore, in one embodiment, faces or cells associated with a probability value of zero (or below a minimum threshold) can be removed from the 3D model representation 122. In one embodiment, faces and / or cells can be removed from the 3D model representation 122, but new faces and / or cells cannot be inserted or added to the 3D model representation 122. The probability density values ​​based on cells or faces can be converted into pixel probability values ​​for image spatial loss calculations.

[0040] Image spatial loss is calculated based on the learned probability density 126 and the per-pixel difference between the rendered 3D model representation 124 and the rendered reference object 158. The image spatial loss is backpropagated through a differentiable renderer, a probabilistic renderer, and a 3D model generation unit that predicts the 3D model representation 122 to update the learned parameters used by the probabilistic renderer and the 3D model generation unit.

[0041] Figure 1C A conceptual diagram illustrating the learned probability densities 134 and 144 according to one embodiment, rendered 3D model representations 128 and 138, a reference surface mesh 148, and a rendered reference object 158 ​​is shown. This is in contrast to the rendered 3D model representation and... Figure 1B The volume mesh 121 associated with the learned probability density 126 shown has undergone 1000 training iterations to produce a volume mesh 121 associated with the rendered 3D model representation 128 and the learned probability density 134. 2000 training iterations have been performed to produce a volume mesh 121 associated with the rendered 3D model representation 138 and the learned probability density 144. Compared to the reference surface mesh 148 and the rendered reference object 158, the surface mesh 132 and the rendered 3D model representation 138 accurately represent the shape and appearance of the object, respectively. As the probability values ​​in the learned probability densities 126, 134, and 144 are updated, more accurate values ​​are added (become lighter) and less accurate values ​​(become darker) are removed.

[0042] Figure 1D A shape and appearance-driven 3D modeling system 150 according to one embodiment is illustrated. The system 150 includes a 3D model generation unit 140 that receives a volume mesh 130 and topology 136 processed according to learned parameters 142 to generate a 3D model representation 122. In one embodiment, the 3D model generation unit 140 is a neural network model. The 3D model representation 122 includes a 3D surface mesh and appearance attributes for an object or scene. In one embodiment, the 3D model generation unit 140 extracts a surface triangle mesh from the volume mesh 130.

[0043] Volume mesh 130 comprises a dense array of geometric cells, where each cell is defined by the position of a vertex in 3D space forming a volume (e.g., a cube, tetrahedron, or other geometry). In one embodiment, volume mesh 130 is defined by other types of primitives or representations. In one embodiment, the initial volume mesh 130 is a uniform mesh of cubes or tetrahedrons. Through training, the vertex positions are adjusted, and as the 3D model representation 122 is learned, one or more vertices, cell faces, and / or cells can be removed. Volume mesh 130 can be subdivided as training progresses to improve the 3D model representation 122.

[0044] Topology generation unit 135 provides a topology 136 definition indicating the visibility or coverage of faces and / or cells in volume mesh 130. In one embodiment, one or more topologies used during training are Monte Carlo sampling topologies. In one embodiment, the learned probability density 156 is also used to generate one or more topologies through importance sampling.

[0045] The 3D model representation 122 includes the positions of learned vertices in the volume mesh 130, the surface mesh corresponding to the volume mesh 130, and appearance attributes 136. Appearance attributes may include normal maps, materials, texture maps, skinning weights, roughness, metallicity, etc. More specifically, texture maps can define lighting and material properties. The initial surface texture of the initial 3D model representation 122 can have a constant or random color, and the color of each texture pixel (texel) in the initial surface texture is adjusted based on the image spatial difference to produce the texture of the 3D model representation 122.

[0046] Environmental conditions 108 define the camera and light source positions for each reference image 154, and are used by a differentiable renderer 145 to generate each rendered image 144. The differentiable renderer 145 renders a 3D model representation 122 according to environmental conditions 108 to generate the rendered images 144. Specifically, the differentiable renderer 145 renders the surface mesh and appearance properties corresponding to the volume mesh 130. Each reference image 154 is generated by rendering the reference 3D model according to environmental conditions 108. Although the shape of the initial volume mesh 130 may not be very similar to the shape of the reference 3D model, after successful optimization, the learned shape of the volume mesh does indeed closely resemble the shape of the reference 3D model.

[0047] In one embodiment, one or more operations of the differentiable renderer 145 can be performed using a graphics processing unit (GPU) pipeline, any combination of GPU general-purpose computing cores, or on a central processing unit (CPU). The differentiable renderer 145 enables operations such as rasterizing a large number of triangles, attribute interpolation, filtered texture lookup, and user-programmable shading and geometry manipulation, all within a range of resolutions. The operations performed by the differentiable renderer 145 are differentiable compared to a conventional rendering pipeline, and the image space loss 162 can be backpropagated through the differentiable renderer 145 to iteratively adjust the 3D model representation 122. In some embodiments, rendering operations may include rasterization, ray tracing, and path tracing.

[0048] Probabilistic rendering unit 155 calculates a probability value per cell or per face based on environmental conditions 108, topology 136, and a volume (3D) mesh portion of the predicted or learned 3D model representation 122. Rays are stepped through the volume mesh 130 and terminate at the first intersection with a cell (or face) that is covered according to the topology (or, in the context of the scene, if no covered cell intersection exists). In the context of the following description, ray tracing or ray stepping through the 3D mesh is referred to as transport.

[0049] Image spatial loss unit 160 processes each reference image 157 and each corresponding rendered image 144 to produce image spatial loss 162. This image spatial loss 162 is backpropagated through differentiable renderer 145, probabilistic rendering unit 155, and 3D model generation unit 140 to reduce the difference between the reference image 157 and the rendered image 144. Backpropagation updates parameters 142 used by probabilistic rendering unit 155 and 3D model generation unit 140. Parameters can be updated for each rendered image 144 and its corresponding reference image 157. In one embodiment, a single topology can be used to render multiple rendered images 144, each associated with a different environmental condition 108. In another embodiment, multiple topologies can be used to render multiple rendered images 144 for the same environmental condition 108.

[0050] Learning success is measured by whether the rendered image 144 is visually identical or nearly identical to the reference image 157. In other words, the rendered image and the reference image are used to determine and fine-tune the geometry defining the 3D model representation 122 without requiring a direct comparison between the 3D model representation 122 and the reference 3D model. In fact, the representation of the reference 3D model can be completely different from the representation of 3D model representation 122. Since the only signal driving optimization is the difference between the rendered image and the reference image, this method is highly versatile and adaptable: it easily supports many different forward rendering models, such as normal maps, spatially varying bidirectional reflectivity distribution functions (BRDFs), translation maps, etc. Supervision solely through images is also key to the ability to easily convert between the rendering system and the scene representation.

[0051] In one embodiment, the generated 3D model representation 122 includes a triangular mesh with material encoded as texture to ensure that the 3D model representation 122 renders efficiently on modern graphics hardware and benefits from, for example, hardware-accelerated rasterization, ray tracing, and filtered texture lookup. The automatically generated 3D model representation 122 can be used for optimized level-of-detail generation to reduce aliasing, seamless mesh filtering, approximation of aggregate geometry, optimization of the junctions of shape and skinning weights to produce reduced animated geometry, transformations between shape representations, and transformations between rendering systems.

[0052] Existing techniques rely on training using all possible topologies. In contrast, a single topology 136 can be used to learn the 3D model representation 122 for each training step (iteration of loss calculation, backpropagation, and update of parameter 142). One reference image 157 can be used per training step, or multiple reference images can be used per training step. In each training step, the single topology 136 enables a subset of faces from the volume mesh 130 to create a rendered opaque surface mesh. If the entire volume mesh 130 is visible, only the exterior of the volume mesh 130 will be intersecting, and light will not penetrate into its interior. Implementing a subset of faces, particularly a subset of different combinations of faces, also produces more accurate and smoother contour edges.

[0053] Existing techniques do not use an opaque volume mesh 130, so light rays do not terminate at the first intersection point due to the varying levels of mesh transparency, and the contour edges appear smooth. These existing techniques are expensive in terms of execution time and / or geometric accuracy (because the mesh changes from transparent to opaque once training is complete). Furthermore, existing techniques typically rely on an initial volume mesh or 3D model (cube, sphere, etc.) input determined at least partially based on a reference image (or reference 3D model). In contrast, the same initial volume mesh 130 can be used to learn any 3D model representation 122.

[0054] Figure 1E A flowchart is shown of a method 170 for shape and appearance-driven automated 3D modeling suitable for implementing some embodiments of this disclosure. Each block of the method 170 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. The method can also be implemented as computer-usable instructions stored on a computer storage medium. The method can be provided by, for example, a standalone application, service, or hosting service (independently or in combination with another hosting service) or a plug-in to another product. Furthermore, method 170 is illustrated by way of example. Figure 1D The method is described using a shape and appearance-driven 3D modeling system 150. However, additionally or alternatively, the method can be performed by any single system, or any combination of systems (including, but not limited to, those described herein). Furthermore, those skilled in the art will understand that any system performing method 170 is within the scope and spirit of the embodiments of this disclosure.

[0055] In step 175, a 3D topology definition of the faces of the cells within the volume mesh is received. In one embodiment, the 3D topology definition is topology 136, and the volume mesh is volume mesh 130. In one embodiment, the volume mesh is opaque. In one embodiment, the volume mesh is initialized as a uniform 3D mesh. In one embodiment, the volume mesh is a dense volume mesh. In one embodiment, the topology definition is a multi-bit visibility mask that includes a single bit for each face of the cells within the volume mesh. In one embodiment, the topology definition is a multi-bit visibility mask that includes a single bit for each cell within the volume mesh, and the single bit is shared between the faces of the cells. The single bit can be asserted when a face or cell is visible and negated when a face or cell is not visible.

[0056] In step 180, the 3D topology and volume mesh are processed according to parameters to predict a 3D model representation corresponding to a reference 3D model, wherein the 3D model representation includes a surface mesh defined by the vertex positions of the elements and the connections between the vertex positions (e.g., vertices). In one embodiment, the surface mesh is opaque. In one embodiment, the faces of the surface mesh are triangles. In one embodiment, the 3D model representation includes spatially varying material properties (e.g., color, texture, normals, roughness, metallicity, etc.). In one embodiment, the 3D topology and volume mesh are processed by 3D model generation unit 140.

[0057] In step 185, the 3D model representation is rendered to produce a rendered image. In one embodiment, a differentiable renderer, such as differentiable renderer 145, produces the rendered image. In one embodiment, only faces or cells located within the visible volume mesh are rendered according to the topology. In one embodiment, steps 185 and 190 are performed in parallel. In one embodiment, the 3D model representation including opaque surface meshes is directly optimized, rather than optimizing the volume representation or semi-transparent representation.

[0058] In step 190, the ray transport corresponding to the rendered image is represented by a 3D model to calculate the probability value of the cell. In one embodiment, the same environmental conditions (e.g., camera position, light position, etc.) used to generate the reference image of the reference 3D model are used to render the rendered image and calculate the probability values. In one embodiment, the ray transport is represented by a 3D model for each pixel in the rendered image.

[0059] In one embodiment, a probability value is calculated for each face of a cell. In another embodiment, the probability value is calculated for each cell and shared by all faces of that cell. In one embodiment, the 3D model representation includes a predicted volume mesh processed along with a 3D topology to update the probability value for each face or cell intersecting with a ray. In one embodiment, the volume mesh is initialized as a uniform mesh, and the structure of the volume mesh (cell face vertex positions and connectivity) is modified for each training iteration to better approximate a reference object or scene. In one embodiment, ray propagation terminates when intersecting a visible cell or face, depending on the topology. In one embodiment, the probability value is calculated by a probabilistic rendering unit 155.

[0060] In step 195, parameters are updated based on probability values ​​and the difference between the rendered image and a reference image of the reference 3D model to reduce the difference. In one embodiment, these differences are backpropagated through a differentiable renderer 145 to update the parameters. In one embodiment, updating the parameters includes: calculating a pixel probability value for each pixel in the rendered image using the probability values, scaling the difference of that pixel using the probability values ​​for that pixel to compute a product, and summing the products to generate an image loss.

[0061] Steps 175, 180, 185, 190, and 195 include training iterations and may be repeated multiple times for additional 3D topology definitions. In one embodiment, a 3D topology definition and an additional 3D topology definition are randomly generated for each training iteration. In one embodiment, at least a portion of the 3D topology definition and the additional 3D topology definition is a Monte Carlo sampling topology. In one embodiment, at least one of the additional 3D topology definitions is determined based on probability values. In one embodiment, a volume mesh is subdivided for at least one of the additional 3D topology definitions.

[0062] In one embodiment, a single 3D topology definition is used across multiple training iterations, wherein additional images associated with different camera or lighting positions are rendered, additional rays corresponding to the additional rendered images are passed through the 3D model representation to compute additional probability values ​​for the units, and parameters are updated for each training iteration. The parameters are updated based on the additional probability values ​​and additional differences between the additional rendered images and additional reference images of the reference 3D model corresponding to the different camera or lighting positions to reduce these additional differences.

[0063] The details of the algorithm are described in the following paragraphs. Through training, a rendering x is generated using a 3D model representation predicted with unknown topology and a set of learned attributes θ (e.g., vertex positions and spatially varying material properties). i In comparison, reference rendering y i The visible differences between them are reduced. Let T represent a discrete set selected from the topology. The topology. The 3D model representation optimizes across multiple image observations i by minimizing an objective function (typically the l2 norm between pixel values ​​j in the image space) between rendered images:

[0064] arg min T，θ ∑ i ∑ j |x ij (T, θ)-y ij |2, Equation (1)

[0065] That is, optimizing the junction between the topology T and the parameter θ. The 3D model representation is rendered using a differentiable renderer R, so x i =R(T, θ, v) i ), where v i Represents the environmental conditions of view i, such as camera parameters and lighting.

[0066] To achieve gradient-based optimization on discrete topologies, the prediction of loss on the topological distribution is used as the optimization objective:

[0067]

[0068] Here, p(T|θ) is the probability of topology T. Compared to existing solutions that use loss based on the distance between 3D meshes, this loss function can be represented in image space, enabling optimization from image observations without knowing the reference 3D mesh.

[0069] By prematurely terminating ray propagation, linear complexity can be achieved in terms of how many m surfaces a ray intersects with. However, evaluating a unique set of topologies for each ray is often impractical. This can be addressed by unifying the sampling topology and using E...p(T|φ) [∑ j |x ij (T)-y ij |2] Unbiased Monte Carlo estimation is used to train 3D model generation unit 140 and probabilistic rendering unit 155, and constant topology can be used for each training iteration.

[0070] make A uniform sampling of the topology of the set T representing all topologies from a given grid is then used to make a Monte Carlo estimate of M samples:

[0071]

[0072] To elaborate further, the normalization constant It is omitted because it does not affect the minimization task.

[0073] Returning to the initial optimization task in Equation 2, we reintroduce the sum of training examples and substitute the Monte Carlo estimation from Equation 3 to produce...

[0074]

[0075] Note that in the last line, nested terms can be evaluated in any order. This means that the topology T i Random sampling can be performed in each training iteration, and Monte Carlo estimation can be viewed as adding additional training examples. The result is a practical and simple optimization process: in each training iteration, a single topology is randomly sampled, an image is rendered using that single topology, and the image loss is weighted by probability. In contrast, existing techniques, such as the Deep Marching Cubes described by Liao et al. in "Deep Marching Cubes: Learning Explicit Surface Representations" at CVPR 2018, evaluate all topologies in each training step.

[0076] The final objective function in Equation 4 leads to a simple training algorithm, as shown in Table 1. In each training step, a random topology is selected, the image is rendered using the mesh derived from that topology, ray probabilities are calculated, and the loss is updated. The gradient of the loss is used to update the face probabilities and other optimization parameters for calculating vertex positions and appearance attributes.

[0077] Table 1

[0078]

[0079] A key insight is pixel color x ij and probability of light The computation can be decoupled. This simple observation has powerful practical effects: the entire mechanism of differentiable rendering on triangular meshes can be reused while jointly learning the topology. Therefore, any off-the-shelf differentiable renderer can be used to compute pixel colors.

[0080] To calculate ray probabilities, a simple ray stepper can be used, as shown below. Figure 2A As shown in Table 2. The cumulative probability φ of the surface. k Similar to alpha compositing of semi-transparent triangles, but with the key difference that rays terminate at the first opaque intersection visible according to the topology. Differentiable rasterization is used to render color images using the renderColor shown in Table 1, so the topology can be constant for each training example. Maintaining a constant topology for the entire training example is not a strict requirement. For example, in one embodiment, differentiable ray tracing can be used to update the topology for each ray. However, using a single topology for each rendered image does lead to fast convergence, so increasing the complexity and computation required to change the topology on a per-ray basis may not benefit the accuracy or speed of 3D model building. Shape and appearance can be jointly optimized with the topology. However, care should be taken when optimizing vertex positions, as this affects ray probabilities and pixel colors. In one embodiment, dynamic vertex positions can be handled by updating the bounding volume hierarchy between each training iteration.

[0081] Figure 2A A flowchart of a method 200 for rendering probability values ​​for a sampled topology suitable for implementing some embodiments of the present disclosure is shown. Method 200 can be used to execute row 8 of the algorithm shown in Table 1. Method 200 can be performed for each pixel rendered in the reference image and / or the rendered image. In step 205, ray probability values ​​are initialized. Ray probability values ​​can be initialized to random values ​​or predetermined values.

[0082] In the following description of method 200, the intersection of a ray with one or more units is described. In another embodiment, the intersection point between the ray and a specific face of one or more units can be identified. In step 210, the probabilistic rendering unit 155 determines whether a unit in the volume mesh 130 intersects with a ray. If so, in step 215, the probabilistic rendering unit 155 then determines whether the intersecting unit is visible according to topology 136. If the intersecting unit is not visible, the probabilistic rendering unit 155 proceeds to step 225. Otherwise, in step 220, the probabilistic rendering unit 155 calculates the probability value of the intersecting and visible unit. Once a visible unit intersects with a ray, the transmission of the ray is completed and terminated, and in step 230, the probabilistic rendering unit 155 completes the probabilistic rendering of the ray.

[0083] In step 215, if the probabilistic rendering unit 155 determines from the topology that the intersecting cell is not visible, then in step 225, the probabilistic rendering unit 155 determines whether the transmission of the ray is complete. The transmission of the ray is complete when the ray has passed through the entire volume mesh 130 without intersecting with any visible cell. When the transmission is complete, in step 230, the probabilistic rendering unit 155 completes the probability generation for that ray. In step 225, if the transmission is not complete, the probabilistic rendering unit 155 returns to step 210 to determine whether the ray intersects with another cell in the volume mesh 130.

[0084] In one embodiment, Algorithm 2, as shown in Table 2, is used to calculate the probability value for each ray. Each ray is associated with a specific pixel.

[0085] Table 2

[0086]

[0087] Once all probability values ​​have been calculated and the rendered image 144 has been rendered, the image spatial loss unit 160 can calculate the image spatial loss 162.

[0088] Figure 2B A flowchart is shown for a method 240 for updating parameters based on probability values ​​of image spatial differences suitable for implementing some embodiments of the present disclosure. In step 245, image spatial loss unit 160 calculates the pixel differences between corresponding pixels in the rendered image 144 and the reference image 154. In step 250, image spatial loss unit 160 multiplies each pixel difference by its respective probability value to produce a product. In step 255, image spatial loss unit 160 sums the products to calculate image spatial loss 162. In step 260, image spatial loss unit 160 backpropagates image spatial loss 162 to update parameter 142.

[0089] In one embodiment, the backpropagation of the ray stepper algorithm shown in Table 2 provides the ray probability relative to the surface probability φ. k The gradient of . Let This represents the set of faces that intersect with the ray before the first hit in topology 136 (at face h). It includes the probability of the ray hitting the first time. for

[0090]

[0091] From equation (5), the partial derivative is given as:

[0092]

[0093] When optimizing vertex positions, partial derivatives are required. Where v kThis refers to the position of vertex k. Note that moving the vertex position changes visibility and introduces discontinuities. These visibility gradients should be handled carefully; this is a very active research topic with many solutions. In one embodiment, the visibility gradient can be approximated by screen-space anti-aliasing based on rasterization of the 3D model representation 122. The same screen-space anti-aliasing technique can be applied to ray probabilities to obtain...

[0094] In one embodiment, using the algorithm shown in Table 3, the learned surface probabilities φ can be used to determine the topology of important samples selected during training.

[0095] Table 3

[0096]

[0097] Importance sampling of the topology based on learned surface probabilities can improve convergence because training examples are concentrated in difficult regions and empty or solid spaces can be quickly eliminated. Note that even φ i Initialized to equal 0 At that time, optimization will also converge.

[0098] Figure 2C A flowchart is shown for a method 265 for importance sampling of topology based on probability values ​​suitable for implementing some embodiments of the present disclosure. In step 270, topology generation unit 135 generates topology 136 including visibility bits for each face of volume mesh 130. Steps 275 and 280 or 290 are performed for each face of topology 136. In step 275, topology generation unit 135 determines whether a face's topology value is less than its probability value; if so, the face's visibility bit is set in step 280. Otherwise, in step 290, the face's visibility bit is not set.

[0099] To promote well-formed meshes and simplify UV mapping, the optimization can be initialized with a dense 3D mesh, where a probability φ can be assigned to each mesh cell. Each cell can be subdivided into 12 triangles, two on each side, instead of relying on a step cube, where all triangles of the mesh cell share the cell's probability φ. k Compared to each triangle having a unique probability value, having all faces of an element share a single probability value presents a more constrained optimization problem. Importantly, it is assumed that the volume mesh 130 does not overlap or self-intersect, resulting in a manifold mesh.

[0100] Vertex position v kThis optimization allows the vertices of the volume mesh 130 to move to better match the object surface. The tangent space for a given topology can be inferred from vertices and faces, and a texture representing the tangent space normal map k can be trained, incorporating normal perturbations and material properties. n Texture diffuse reflectance k d and mirror parameter k s (That is, metallicity and roughness).

[0101] The algorithm shown in Table 1 can be applied hierarchically from coarse to fine resolution by periodically increasing the grid resolution during training. Intermediate results at coarser resolutions effectively serve as good initial guesses, thus significantly improving convergence.

[0102] Figure 3A The diagram illustrates a conceptual representation of coarse-to-fine layered training according to one embodiment. Training begins with a grid size of 16 units in each direction (x, y, and z) to generate a learning probability density 310. Compared to the learning probability density 310, the grid size is subdivided once more in each dimension to generate a learning probability density 320, thereby improving the smoothness and fidelity of the contour. The grid size is further subdivided twice more in each dimension to generate a learning probability density 330, further improving the smoothness and fidelity of the contour.

[0103] Figure 3B A conceptual diagram of a learned surface mesh 340, a rendered 3D model representation 345, and a rendered reference 3D model 350 according to an embodiment is shown. The learned surface mesh 340 includes faces on the surface of an optimized dense volume mesh. In one embodiment, vertices and faces within the dense volume mesh are removed to produce the learned surface mesh 340. In another embodiment, faces of the dense volume mesh with a learning probability below a threshold are removed to produce the learned surface mesh 340. In one embodiment, the learned surface mesh 340 includes opaque triangular faces, wherein the positions of the vertices are learned in an optimized topology. The 3D model representation 345 also includes learned material properties, and the rendered 3D model representation 345 looks almost identical to the rendered reference 3D model 350.

[0104] In one embodiment, the image spatial loss is backpropagated through the rendering pipeline to update the initial 3D model and improve the visual similarity between the image set and the reference image set. In another embodiment, the image spatial loss, as a gradient indicating the effect of moving mesh vertex positions and adjusting other parameters on the image set, is backpropagated. The operations performed by the rendering pipeline to generate the image set are differentiable, so gradients can be computed for backpropagation.

[0105] Figure 3CA flowchart is shown for another method 360 for shape and appearance-driven automated 3D modeling suitable for implementing some embodiments of this disclosure. Each block of the method 360 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. The method can also be implemented as computer-usable instructions stored on a computer storage medium. The method can be provided by (to name a few examples) a standalone application, service, or hosting service (independently or in combination with another hosting service) or a plug-in to another product. Furthermore, method 360 is addressed by way of example. Figure 1D The method is described using a shape and appearance-driven 3D modeling system 150. However, additionally or alternatively, the method can be performed by any single system, or any combination of systems (including, but not limited to, those described herein). Furthermore, those skilled in the art will understand that any system performing method 360 is within the scope and spirit of the embodiments of this disclosure.

[0106] In step 365, a 3D topology definition is received that identifies visible faces of cells within a volume mesh, defined by the vertex positions of the faces and the connections between the vertex positions. In one embodiment, the 3D topology definition is output by the topology generation unit 135 and identifies visible faces of cells within the volume mesh 130.

[0107] In step 370, the volume mesh is deformed according to parameters to produce an opaque surface mesh defined by the vertex locations of visible faces and the connections between these vertex locations, wherein the opaque surface mesh corresponds to the reference 3D model. In one embodiment, the 3D model representation 122 generated by the 3D model generation unit 140 includes a deformed volume mesh. In one embodiment, the volume mesh is a dense 3D mesh that is deformed to define the shape of the 3D model representation 122 with a 3D topology definition.

[0108] In step 375, parameters are updated to reduce the difference between the rendered image of the opaque surface mesh and the reference image of the reference 3D model. In one embodiment, parameter 142 is updated based on image spatial loss 162. In one embodiment, method 240 is used to update these parameters. In one embodiment, steps 370 and 375 are repeated for an additional 3D topology definition, wherein the additional 3D topology definition is different from the 3D topology definition. In one embodiment, the additional 3D topology definition is defined based on calculated probability values ​​of cells within the volume mesh, which indicate the likelihood that each cell will intersect with a ray corresponding to the reference image. In one embodiment, method 200 is used to calculate the probability values. In one embodiment, method 265 is used to generate the additional 3D topology definition.

[0109] End-to-end optimization of the shape and appearance of a triangle model is challenging due to changes in topology. Therefore, the optimization task is formulated on a topological distribution, enabling gradient-based optimization. Furthermore, a highly efficient implementation is achieved by estimating the expectation value on the distribution through ray tracing the topology sampled via Monte Carlo. Topological sampling techniques enable the joint learning of the topology, shape, and appearance of the triangle mesh from image observations. Moreover, by importance sampling the distribution of possible topologies, optimization can be achieved through inverse rendering using randomly sampled topologies for each training iteration.

[0110] Parallel processing architecture

[0111] Figure 4 A parallel processing unit (PPU) 400 according to an embodiment is illustrated. PPU 400 can be used to implement a shape and appearance-driven 3D modeling system 150. PPU 400 can be used to implement one or more of a topology generation unit 135, a 3D model generation unit 140, a probabilistic rendering unit 155, a differentiable renderer 145, or an image spatial loss unit 160. In one embodiment, a processor such as PPU 400 can be configured to implement a neural network model. The neural network model can be implemented as software instructions executed by the processor, or in other embodiments, the processor can include a matrix of hardware elements configured to process a set of inputs (e.g., electrical signals representing values) to generate a set of outputs that can represent activations of the neural network model. In other embodiments, the neural network model can be implemented as a combination of processing and software instructions executed by the hardware element matrix. Implementing a neural network model can include determining a set of parameters for the neural network model through, for example, supervised or unsupervised training of the neural network model, and, alternatively, performing inference using that parameter set to process new input sets.

[0112] In one embodiment, PPU 400 is a multi-threaded processor implemented on one or more integrated circuit devices. PPU 400 is a latency-hiding architecture designed to process many threads in parallel. A thread (e.g., an execution thread) is an instantiation of an instruction set configured to be executed by PPU 400. In one embodiment, PPU 400 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device. In other embodiments, PPU 400 may be used to perform general-purpose computing. While an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that such a processor is illustrated for illustrative purposes only, and any processor may be employed to complement and / or replace this processor.

[0113] One or more PPU 400s can be configured to accelerate thousands of high-performance computing (HPC), data center, cloud computing, and machine learning applications. PPU 400s can be configured to accelerate numerous deep learning systems and applications used in autonomous vehicles, simulations, computational graphics such as ray or path tracing, deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0114] like Figure 4 As shown, PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a job allocation unit 425, a hub 430, a crossbar (Xbar) 470, one or more general purpose processing clusters (GPCs) 450, and one or more memory partitioning units 480. PPU 400 can be connected to a host processor or other PPU 400 via one or more high-speed NVLink 410 interconnects. PPU 400 can be connected to a host processor or other peripheral devices via interconnect 402. PPU 400 can also be connected to local memory 404, which includes multiple memory devices. In one embodiment, local memory may include multiple dynamic random access memory (DRAM) devices. The DRAM devices may be configured as a high-bandwidth memory (HBM) subsystem, wherein multiple DRAM dies are stacked within each device.

[0115] The NVLink 410 interconnect enables the system to expand and include one or more PPUs 400 in conjunction with one or more CPUs, supporting cache coherency between the PPUs 400 and the CPU, as well as the CPU controller. Data and / or commands can be sent from or from the NVLink 410 to other units of the PPU 400 via hub 430, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 5B A more detailed description of the NVLink 410.

[0116] I / O unit 405 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 402. I / O unit 405 may communicate directly with the host processor via interconnect 402, or via one or more intermediate devices such as memory bridges. In one embodiment, I / O unit 405 may communicate with one or more other processors, such as one or more PPUs 400, via interconnect 402. In one embodiment, I / O unit 405 implements a Peripheral Component Interconnect High Speed ​​(PCIe) interface for communication via a PCIe bus, and interconnect 402 is a PCIe bus. In alternative embodiments, I / O unit 405 may implement other types of known interfaces for communication with external devices.

[0117] I / O unit 405 decodes data packets received via interconnect 402. In one embodiment, the data packets represent commands configured to cause PPU 400 to perform various operations. I / O unit 405 transmits the decoded commands to various other units of PPU 400 that these commands may specify. For example, some commands may be transmitted to front-end unit 415. Other commands may be transmitted to hub 430 or other units of PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 405 is configured to route communication between and among the various logical units of PPU 400.

[0118] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. The workload may include instructions and data to be processed by those instructions. The buffer is an area of ​​memory accessible (e.g., read / write) by both the host processor and the PPU 400. For example, I / O unit 405 may be configured to access a buffer in system memory connected to interconnect 402 via a memory request transmitted through interconnect 402. In one embodiment, the host processor writes a command stream to the buffer and then transmits a pointer to the start of the command stream back to the PPU 400. Front-end unit 415 receives pointers to one or more command streams. Front-end unit 415 manages one or more streams, reads commands from these streams, and forwards the commands to the respective units of the PPU 400.

[0119] Front-end unit 415 is coupled to scheduler unit 420, which configures various GPCs 450 to process tasks defined by one or more streams. Scheduler unit 420 is configured to track status information related to the various tasks managed by scheduler unit 420. Status can indicate which GPC 450 a task is assigned to, whether the task is active or inactive, the priority associated with the task, etc. Scheduler unit 420 manages the execution of multiple tasks on one or more GPCs 450.

[0120] Scheduler unit 420 is coupled to job allocation unit 425, which is configured to dispatch tasks for execution on GPC 450. Job allocation unit 425 can track several scheduled tasks received from scheduler unit 420. In one embodiment, job allocation unit 425 manages a pending task pool and an active task pool for each GPC 450. When GPC 450 completes the execution of a task, the task is evicted from the active task pool of GPC 450, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 450. If an active task on GPC 450 is idle, for example, while waiting for data dependencies to be resolved, then the active task can be evicted from GPC 450 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 450.

[0121] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API), enabling one or more applications executing on the host processor to be scheduled for operations to be performed on the PPU 400. In one embodiment, multiple computing applications are executed concurrently by the PPU 400, and the PPU 400 provides isolation, Quality of Service (QoS), and independent address spaces for the multiple computing applications. Applications may generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 400. The driver kernel outputs the tasks to one or more streams being processed by the PPU 400. Each task may include one or more associated thread groups, referred to herein as warps. In one embodiment, a warp includes 32 associated threads that can execute in parallel. Cooperative threads may refer to multiple threads that include instructions for performing tasks and can exchange data via shared memory. These tasks may be assigned to one or more processing units within the GPC 450, and instructions are scheduled for execution by at least one warp.

[0122] The work allocation unit 425 communicates with one or more GPCs 450 via XBar 470. XBar 470 is an interconnect network that couples a plurality of units of PPU 400 to other units of PPU 400. For example, XBar 470 can be configured to couple the work allocation unit 425 to a specific GPC 450. Although not explicitly shown, one or more other units of PPU 400 may also be connected to XBar 470 via hub 430.

[0123] Tasks are managed by scheduler unit 420 and dispatched to GPC 450 by job allocation unit 425. GPC 450 is configured to process tasks and generate results. Results can be consumed by other tasks within GPC 450, routed to different GPC 450 via XBar 470, or stored in memory 404. Results can be written to memory 404 via memory partitioning unit 480, which implements a memory interface for reading data from and writing data to memory 404. Results can be transferred to another PPU 400 or CPU via NVLink 410. In one embodiment, PPU 400 includes U number of memory partitioning units 480, equal to the number of individual and distinct memory devices coupled to memory 404 of PPU 400. Each GPC 450 may include a memory management unit to provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit provides one or more translation back buffers (TLBs) for performing virtual address to physical address translation in memory 404.

[0124] In one embodiment, memory partitioning unit 480 includes a raster operation (ROP) unit, a secondary (L2) cache, and a memory interface coupled to memory 404. The memory interface can implement 32-bit, 64-bit, 128-bit, or 1024-bit data buses for high-speed data transfer. PPU 400 can connect to up to Y memory devices, such as high-bandwidth memory stacks or graphics dual data rate, version 5, synchronous dynamic random access memory, or other types of persistent storage devices. In one embodiment, the memory interface implements an HBM2 memory interface, and Y equals half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 400, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies, and Y equals 4, wherein each HBM2 stack includes two 129-bit channels per die, for a total of eight channels, and the data bus width is 1024 bits.

[0125] In one embodiment, memory 404 supports single error correction double detection (SECDED) error correction codes (ECC) to protect data. ECC provides a high level of reliability for computing applications sensitive to data corruption. Reliability is particularly important where the PPU 400 is handling very large datasets and / or in large-scale cluster computing environments where applications run for extended periods.

[0126] In one embodiment, PPU 400 implements a multi-level memory hierarchy. In one embodiment, memory partitioning unit 480 supports unified memory that provides a single, unified virtual address space for the CPU and PPU 400 memory, allowing data sharing between virtual memory systems. In one embodiment, the frequency with which PPU 400 accesses memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 400 that is accessing these pages more frequently. In one embodiment, NVLink 410 supports address translation services, allowing PPU 400 to directly access the CPU's page table and providing PPU 400 with full access to the CPU's memory.

[0127] In one embodiment, the replication engine transfers data between multiple PPUs 400 or between a PPU 400 and a CPU. The replication engine can generate page faults for addresses that are not mapped to the page table. The memory partitioning unit 480 can then repair the page faults, mapping these addresses to the page table, after which the replication engine can perform the transfer. In conventional systems, memory is fixed (e.g., non-pageable) for multiple replication engine operations across multiple processors, significantly reducing available storage. In the event of a hardware page fault, addresses can be passed to the replication engine without concern for whether memory pages reside, and the replication process is transparent.

[0128] Data from memory 404 or other system memory can be fetched by memory partitioning unit 480 and stored in an on-chip L2 cache 460 shared among the various GPCs 450. As shown, each memory partitioning unit 480 includes a portion of the L2 cache associated with the corresponding memory 404. Low-level caches can then be implemented in various units within the GPC 450. For example, each processing unit within the GPC 450 can implement a Level 1 (L1) cache. The L1 cache is a private memory dedicated to a specific processing unit. The L2 cache 460 is coupled to memory interface 470 and XBar 470, and data from the L2 cache can be fetched and stored in each of the L1 caches for processing.

[0129] In one embodiment, the processing unit within each GPC 450 implements a SIMD (Single Instruction Multiple Data) architecture, where each thread in a thread group (e.g., a thread bundle) is configured to process a different data set based on the same set of instructions. All threads in the thread group execute the same instructions. In another embodiment, the processing unit implements a SIMT (Single Instruction Multiple Thread) architecture, where each thread in the thread group is configured to process a different data set based on the same set of instructions, but individual threads in the thread group are allowed to diverge during execution. In one embodiment, maintaining a program counter, call stack, and execution state for each thread bundle allows for concurrency between thread bundles and serial execution within a thread bundle when threads diverge. In another embodiment, maintaining a program counter, call stack, and execution state for each individual thread allows for equal concurrency among all threads within and between thread bundles. When maintaining an execution state for each individual thread, threads executing the same instructions can aggregate and execute in parallel for maximum efficiency.

[0130] Cooperative groups are a programming model for organizing groups of communicating threads. They allow developers to express the granularity at which threads are communicating, enabling richer and more efficient expressions of parallel decomposition. The Cooperative Startup API supports synchronization between blocks of threads executing parallel algorithms. Conventional programming models provide a single, simple construct for synchronizing cooperative threads: a barrier across all threads in a thread block (e.g., the `syncthreads()` function). However, programmers often prefer to define thread groups smaller than thread blocks in the form of a collective group-wide function interface, and synchronize within the defined group to allow for greater performance, design flexibility, and software reuse.

[0131] Collaboration groups enable programmers to explicitly define thread groups (as small as a single thread) at the sub-block and multi-block granularity, and perform collective operations such as synchronization on threads within the collaboration group. This programming model supports clean composition across software boundaries, allowing libraries and utility functions to be safely synchronized within their local contexts without having to make assumptions about aggregation. The collaboration group primitive allows for the implementation of new cooperative parallelism patterns, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire mesh of thread blocks.

[0132] Each processing unit comprises a large number (e.g., 128, etc.) of different processing cores (e.g., functional units), which may be fully pipelined, single-precision, double-precision, and / or mixed-precision, and include floating-point arithmetic logic units and integer arithmetic logic units. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the core comprises 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0133] Tensor cores are configured to perform matrix operations. Specifically, tensor cores are configured to perform deep learning matrix arithmetic, such as GEMM (matrix-matrix multiplication), for convolution operations during neural network training and inference. In one embodiment, each tensor core operates on a 4x4 matrix and performs matrix multiplication and accumulation operations, D = A × B + C, where A, B, C, and D are 4x4 matrices.

[0134] In one embodiment, matrix multiplication inputs A and B can be integer, fixed-point, or floating-point matrices, while accumulation matrices C and D can be integer, fixed-point, or floating-point matrices of equal or higher bit width. In one embodiment, the Tensor Core operates on 1-bit, 4-bit, or 8-bit integer input data using 32-bit integer accumulation. An 8-bit integer matrix multiplication requires 1024 operations and results in a full-precision product, which is then accumulated with other intermediate multiplications using 32-bit integer addition for an 8x8x16 matrix multiplication. In one embodiment, the Tensor Core operates on 16-bit floating-point input data using 32-bit floating-point accumulation. A 16-bit floating-point multiplication requires 64 operations and results in a full-precision product, which is then accumulated with other intermediate multiplications using 32-bit floating-point addition for a 4x4x4 matrix multiplication. In practice, the Tensor Core is used to perform operations on much larger two-dimensional or higher-dimensional matrices composed of these smaller elements. APIs such as the CUDA 9 C++ API expose specialized matrix loading, matrix multiplication and accumulation, and matrix storage operations for efficient use with Tensor Core from CUDA-C++ programs. At the CUDA level, the thread bundle-level interface takes a 16x16 matrix spanning all 32 threads of the thread bundle.

[0135] Each processing unit may also include M Special Function Units (SFUs) that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, an SFU may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, an SFU may include a texture unit configured to perform texture map filtering operations. In one embodiment, a texture unit is configured to load texture maps (e.g., a 2D texture array) from memory 404 and sample these texture maps to produce sampled texture values ​​for use by a shader program executed by the processing unit. In one embodiment, the texture maps are stored in shared memory that may include or include an L1 cache. The texture units use mip maps (e.g., texture maps with varying levels of detail) to implement texture operations such as filtering. In one embodiment, each processing unit includes two texture units.

[0136] Each processing unit also includes N Load Memory Units (LSUs) that implement load and store operations between shared memory and the register file. Each processing unit includes an interconnect network that connects each of the cores to the register file and connects the LSUs to the register file and shared memory. In one embodiment, the interconnect network is a crossbar switch that can be configured to connect any core to any register in the register file and connect the LSUs to memory locations in the register file and shared memory.

[0137] Shared memory is an on-chip memory array that allows data storage and communication between processing units and between threads within a processing unit. In one embodiment, the shared memory includes 128KB of storage capacity and is located on the path from each processing unit to memory partition unit 480. The shared memory can be used for caching reads and writes. One or more of the shared memory, L1 cache, L2 cache, and memory 404 serve as a backup cache.

[0138] Combining data caching and shared memory functionality into a single memory block provides optimal overall performance for both types of memory access. The capacity can be used as a cache by programs that do not use shared memory. For example, if shared memory is configured to use half its capacity, then texture and load / store operations can use the remaining capacity. Integration into shared memory allows it to function as a high-throughput conduit for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0139] When configured for general-purpose parallel computing, a simpler configuration can be used compared to graphics computing. Specifically, bypassing fixed-function graphics processing units (GPUs) creates a much simpler programming model. In this general-purpose parallel computing configuration, the work allocation unit 425 directly dispatches and assigns thread blocks to processing units within the GPC 450. Threads execute the same program using unique thread IDs during computation to ensure that each thread uses the executor program and the processing unit performing the computation, the shared memory for communication between threads, and the LSU (Local Subsystem for Memory) for reading and writing global memory via the shared memory and memory partitioning unit 480 to generate unique results. When configured for general-purpose parallel computing, processing units can also write commands, which the scheduler unit 420 can use to start new work on the processing units.

[0140] Each of the PPUs 400 may include one or more processing cores and / or components thereof, such as a Tensor Core (TC), a Tensor Processing Unit (TPU), a Pixel Vision Core (PVC), a Ray Tracing (RT) Core, a Vision Processing Unit (VPU), a Graphics Processing Cluster (GPC), a Texture Processing Cluster (TPC), a Streaming Multiprocessor (SM), a Tree Traversal Unit (TTU), an Artificial Intelligence Accelerator (AIA), a Deep Learning Accelerator (DLA), an Arithmetic Logic Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating-Point Unit (FPU), Input / Output (I / O) Elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect High Speed ​​(PCIe) Elements and / or the like, and / or be configured to perform its functions.

[0141] The PPU 400 may be included in desktop computers, laptop computers, tablet computers, servers, supercomputers, smartphones (e.g., wireless, handheld devices), personal digital assistants (PDAs), digital cameras, vehicles, head-mounted displays, handheld electronic devices, and the like. In one embodiment, the PPU 400 is implemented on a single semiconductor substrate. In another embodiment, the PPU 400 is included in a system-on-a-chip (SoC) along with one or more other devices such as an additional PPU 400, memory 404, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), and the like.

[0142] In one embodiment, PPU 400 may be included on a graphics card that includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on a desktop computer's motherboard. In yet another embodiment, PPU 400 may be an integrated graphics processing unit (iGPU) or a parallel processor included in the chipset of the motherboard. In yet another embodiment, PPU 400 may be implemented in reconfigurable hardware. In yet another embodiment, a portion of PPU 400 may be implemented in reconfigurable hardware.

[0143] Exemplary computing system

[0144] As developers expose and leverage more parallelism in applications such as artificial intelligence computing, systems with multiple GPUs and CPUs are being used across various industries. High-performance GPU-accelerated systems with tens to thousands of computing nodes are being deployed in data centers, research facilities, and supercomputers to solve increasingly complex problems. With the increasing number of processing devices within high-performance systems, communication and data transmission mechanisms need to be scaled to support the increased bandwidth.

[0145] Figure 5A According to one embodiment, the use Figure 4 A conceptual diagram of the processing system 500 implemented by the PPU 400. The exemplary system 565 can be configured to implement a shape and appearance-driven 3D modeling system 150 and / or Figure 1E Method 170 is shown in the figure. The processing system 500 includes a CPU 530, a switch 510, multiple PPUs 400, and various memories 404.

[0146] The NVLink 410 provides a high-speed communication link between each PPU 400. Although Figure 5B The diagram illustrates a specific number of NVLink 410 and interconnect 402 connections, but the number of connections to each PPU 400 and CPU 530 can vary. Switch 510 forms an interface between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLink 410 can reside on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols to interface between various connections and / or links.

[0147] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and CPU 530, and switch 510 forms an interface between interconnect 402 and each PPU 400. PPU 400, memory 404, and interconnect 402 may reside on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 and CPU 530, and switch 510 uses NVLink 410 to form an interface between each PPU 400 to provide one or more high-speed communication links between PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between PPU 400 and CPU 530 via switch 510. In yet another embodiment (not shown), interconnect 402 directly provides one or more communication links between each PPU 400. One or more of the NVLink 410 high-speed communication links can be implemented as physical NVLink interconnects or on-chip or die-on interconnects using the same protocol as the NVLink 410.

[0148] In the context of this specification, a single semiconductor platform can refer to a single semiconductor-based integrated circuit fabricated on a bare die or chip. It should be noted that the term "single semiconductor platform" can also refer to a multi-chip module with increased connectivity, simulating on-chip operation and representing a significant improvement over conventional bus implementations. Of course, different circuits or devices can also be located individually within the semiconductor platform or in various combinations thereof, as desired by the user. Alternatively, the parallel processing module 525 can be implemented as a circuit board substrate, and each PPU 400 and / or memory 404 can be a packaged device. In one embodiment, the CPU 530, switch 510, and parallel processing module 525 reside on a single semiconductor platform.

[0149] In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits per second, and each PPU400 includes six NVLink 410 interfaces (e.g., Figure 5A As shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 gigabits per second in each direction, and the six links provide 400 gigabits per second. NVLink 410 can be used as follows: Figure 5A It is used exclusively for PPU-to-PPU communication, or for a combination of PPU-to-PPU and PPU-to-CPU when the CPU 530 also includes one or more NVLink 410 interfaces.

[0150] In one embodiment, NVLink 410 allows direct load / store / atomic access from CPU 530 to memory 404 of each PPU 400. In one embodiment, NVLink 410 supports coherent operation, allowing data read from memory 404 to be stored in the cache hierarchy of CPU 530, reducing cache access latency of CPU 530. In one embodiment, NVLink 410 includes support for Address Translation Service (ATS), allowing PPU 400 to directly access page tables within CPU 530. One or more of NVLink 410 may also be configured to operate in low-power mode.

[0151] Figure 5B An exemplary system 565 is illustrated, in which various architectures and / or functions of various prior embodiments can be implemented. The exemplary system 565 can be configured to implement a shape and appearance-driven 3D modeling system 150. Figure 1E Method 170 and / or shown in the figure Figure 3C Method 360 is shown in the figure.

[0152] As shown in the figure, a system 565 is provided, which includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 may directly or indirectly couple one or more of the following devices: main memory 540, network interface 535, CPU 530, display device 545, input device 560, switch 510, and parallel processing system 525. The communication bus 575 may be implemented using any suitable protocol and may represent one or more links or buses, such as address bus, data bus, control bus, or combinations thereof. The communication bus 575 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect High Speed ​​(PCIe) bus, HyperTransport, and / or another type of bus or link. In some embodiments, direct connections exist between components. As an example, CPU 530 may be directly connected to main memory 540. Furthermore, CPU 530 may be directly connected to parallel processing system 525. In cases where there is a direct or point-to-point connection between components, the communication bus 575 may include a PCIe link that implements the connection. In these examples, the PCI bus need not be included in the system 565.

[0153] Although using lines Figure 5CThe different blocks are shown connected via a communication bus 575, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component such as a display device 545 can be considered an I / O component, such as an input device 560 (e.g., if the display is a touchscreen). As another example, the CPU 530 and / or the parallel processing system 525 may include memory (e.g., main memory 540 may represent storage devices other than the parallel processing system 525, the CPU 530, and / or other components). In other words, Figure 5C The term "computing device" is merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all of these are expected to fall under [the relevant category]. Figure 5C Within the scope of computing devices.

[0154] System 565 also includes main memory 540. Control logic (software) and data are stored in main memory 540, which can take the form of a variety of computer-readable media. Computer-readable media can be any available medium that can be accessed by system 565. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. For example and without limitation, computer-readable media can include computer storage media and communication media.

[0155] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, main memory 540 may store computer-readable instructions such as an operating system (e.g., representing programs and / or program elements). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic cartridges, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that may be used to store desired information and that can be accessed by system 565. When used herein, computer storage media does not include the signal itself.

[0156] Computer storage media may include computer-readable instructions, data structures, program modules, or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information transport medium. The term "modulated data signal" may refer to a signal whose characteristics are set or altered in a manner that encodes information into that signal. For example, and without limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0157] When executed, the computer program enables system 565 to perform various functions. CPU 530 may be configured to execute at least some of computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Each of CPUs 530 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing numerous software threads simultaneously. Depending on the type of system 565 implemented, CPU 530 may include any type of processor and may include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of system 565, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, system 565 may include one or more CPUs 530.

[0158] In addition to or alternatively to CPU 530, parallel processing module 525 may be configured to execute at least some of computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Parallel processing module 525 may be used by system 565 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, parallel processing module 525 may be used for general-purpose computing on a GPU (GPGPU). In embodiments, CPU 530 and / or parallel processing module 525 may execute any combination of methods, processes, and / or portions thereof, discretely or jointly.

[0159] System 565 also includes input device 560, parallel processing system 525, and display device 545. Display device 545 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. Display device 545 may receive data from other components (e.g., parallel processing system 525, CPU 530, etc.) and output that data (e.g., images, video, sound, etc.).

[0160] Network interface 535 enables system 565 to be logically coupled to other devices, including input device 560, display device 545, and / or other components, some of which may be embedded (e.g., integrated into) system 565. Illustrative input device 560 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite antennas, scanners, printers, wireless devices, etc. Input device 560 can provide a natural user interface (NUI) that processes user-generated air gestures, voice, or other physiological input. In some instances, input can be transmitted to appropriate network elements for further processing. NUI can enable speech recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head-eye tracking, and touch recognition associated with the display of system 565 (described in more detail below). System 565 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. In addition, system 565 may include an accelerometer or gyroscope that allows motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by system 565 to render immersive augmented reality or virtual reality.

[0161] Furthermore, system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) via network interface 535 for communication purposes. System 565 can be included in a distributed network and / or cloud computing environment.

[0162] Network interface 535 may include one or more receivers, transmitters, and / or transceivers that enable system 565 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. Network interface 535 may include components and functions that allow communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0163] System 565 may also include an auxiliary storage device (not shown). The auxiliary storage device includes, for example, a hard disk drive and / or a removable storage drive representing a floppy disk drive, magnetic tape drive, compact disc drive, digital versatile disc (DVD) drive, recording device, Universal Serial Bus (USB) flash memory. The removable storage drive reads from and / or writes to the removable storage unit in a well-known manner. System 565 may also include a hard-wired power supply, a battery power supply, or a combination thereof (not shown). This power supply can supply power to System 565 to enable the components of System 565 to operate.

[0164] Each of the aforementioned modules and / or devices may even reside on a single semiconductor platform to form system 565. Alternatively, various different modules may be placed individually or located in various combinations of semiconductor platforms as desired by the user. Although various different embodiments have been described above, it should be understood that they are given by way of example only and without limitation. Therefore, the breadth and scope of preferred embodiments should not be limited to any of the exemplary embodiments described above, but should be defined only by the following claims and their equivalents.

[0165] Example network environment

[0166] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage devices (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 5A Processing system 500 and / or Figure 5B Implemented on one or more instances of the exemplary system 565, for example, each device may include similar components, features and / or functions of the processing system 500 and / or the exemplary system 565.

[0167] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or a combination of both. A network can include multiple networks or networks of networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks—such as the Internet, and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. In cases where the network includes wireless telecommunications networks, components such as base stations, communication towers, or even access points (along with other components) can provide wireless connectivity.

[0168] A compatible network environment may include one or more peer-to-peer network environments—in which case the server may not be included in the network environment—and one or more client-server network environments—in which case one or more servers may be included in the network environment. In a peer-to-peer network environment, the functionality described herein regarding the server can be implemented on any number of client devices.

[0169] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework of one or more applications supporting the software layer and / or application layer. The software or application may include web-based service software or applications, respectively. In embodiments, one or more client devices may use the web-based service software or application (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software web application framework type that can be used for large-scale data processing (e.g., "big data").

[0170] A cloud-based network environment can provide cloud computing and / or cloud storage to implement the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., a central or core server in one or more data centers, which may be distributed across states, regions, countries, globally, etc.). If the connection to the user (e.g., client device) is relatively close to the edge server, then the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0171] Client devices may include Figure 5B Example processing system 500 and / or Figure 5C At least some of the components, features, and functions of the exemplary system 565. For example and without limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronics device, workstation, edge device, any combination of these defined devices, or any other suitable device.

[0172] Machine Learning

[0173] Deep neural networks (DNNs) developed on processors such as the PPU 400 have been used in a wide variety of use cases, from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technique that models the neural learning process of the human brain, which continuously learns, becomes smarter, and delivers more accurate results faster over time. Just as a child is initially taught by adults to correctly identify and classify various shapes, eventually becoming able to identify shapes without any guidance, a deep learning or neural learning system needs to be trained in object recognition and classification so that it becomes smarter and more efficient at identifying basic objects, occluded objects, and so on, while also attaching context to objects.

[0174] At its simplest level, neurons in the human brain receive various inputs, assigning a level of importance to each of these inputs, and the output is passed to other neurons to make a response. Artificial neurons, or perceptrons, are the most basic model of neural networks. In one example, a perceptron can receive one or more inputs representing various features of objects that the perceptron is being trained to recognize and classify, and each of these features is assigned a weight based on its importance in defining the shape of the object.

[0175] Deep neural network (DNN) models consist of multiple layers with numerous connections (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.), which can be trained on massive amounts of input data to solve complex problems quickly and with high accuracy. In one example, the first layer of a DNN model breaks down an input image of a car into different segments and looks for basic patterns such as lines and angles. The second layer assembles these lines to look for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final few layers generate labels for the input image that identify the model of a specific car brand.

[0176] Once trained, a DNN can be deployed and used to identify and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include identifying handwritten digits on a check deposited into an ATM, identifying images of friends in a photograph, delivering movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human language in real time.

[0177] During training, data flows through the DNN in the forward propagation phase until a prediction is produced indicating the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and weights are adjusted for each feature during the backpropagation phase until the DNN correctly labels the input as well as other inputs in the training dataset. Training complex neural networks requires significant parallel computing power, including floating-point multiplication and addition supported by a PPU 400. Inference is less computationally intensive than training and is a latency-sensitive process where the trained neural network is applied to new inputs it has not seen before for tasks such as image classification, sentiment detection, label recommendation, language recognition and translation, and typically infers new information.

[0178] Neural networks rely heavily on matrix operations, and complex, multi-layered networks require massive amounts of floating-point performance and bandwidth for both efficiency and speed. Leveraging thousands of processing cores optimized for matrix operations and delivering tens to hundreds of TFLOPS of performance, the PPU 400 is a computing platform capable of providing the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0179] Furthermore, images generated using one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for recognizing real-world objects and environments. Such images can include driveways, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or scenes of real-world environments. Such images can be used to train, test, or certify DNNs used in machines or robots to manipulate, process, or modify real-world physical objects. Additionally, such images can be used to train, test, or certify DNNs used in autonomous vehicles to navigate and move vehicles in the real world. Furthermore, images generated using one or more of the techniques disclosed herein can be used to communicate information to users of such machines, robots, and vehicles.

[0180] Figure 5C Components of an example system 555, which can be used to train and utilize machine learning according to at least one embodiment, are illustrated. As will be discussed, various components can be provided by a single computing system or various combinations of computing devices and resources, which may be under the control of a single entity or multiple entities. Furthermore, aspects may be triggered, initiated, or requested by different entities. In at least one embodiment, the training of the neural network may be guided by a vendor associated with vendor environment 506, while in at least one embodiment, training may be requested by a customer or other user who can access the vendor environment through client device 502 or other such resources. In at least one embodiment, training data (or data to be analyzed by the trained neural network) may be provided by a vendor, user, or third-party content provider 524. In at least one embodiment, client device 502 may be, for example, a vehicle or object to be navigated on behalf of a user, who can submit requests and / or receive instructions that aid in device navigation.

[0181] In at least one embodiment, a request can be submitted via at least one network 504 for receipt by a vendor environment 506. In at least one embodiment, the client device can be any suitable electronic and / or computing device that enables a user to generate and send such requests, such as, but not limited to, desktop computers, laptop computers, computer servers, smartphones, tablets, game consoles (portable or otherwise), computer processors, computing logic, and set-top boxes. One or more networks 504 can include any suitable network for transmitting requests or other such data, such as the Internet, intranet, Ethernet, cellular network, local area network (LAN), wide area network (WAN), personal area network (PAN), self-organizing network providing direct wireless connectivity between peers, etc.

[0182] In at least one embodiment, a request may be received at interface layer 508, which in this example may forward data to training and inference manager 532. Training and inference manager 532 may be a system or service including hardware and software for managing services and requests corresponding to data or content. In at least one embodiment, training and inference manager 532 may receive a request to train a neural network and may provide data for the request to training module 512. In at least one embodiment, if the request is not specified, training module 512 may select an appropriate model or neural network to use and may train the model using the associated training data. In at least one embodiment, training data may be a batch of data stored in training data repository 514, received from client device 502, or obtained from third-party vendor 524. In at least one embodiment, training module 512 may be responsible for training the data. The neural network may be any suitable network, such as a recurrent neural network (RNN) or convolutional neural network (CNN). Once the neural network is trained and successfully evaluated, the trained neural network may be stored in, for example, model repository 516, which may store different models or networks for users, applications, or services, etc. In at least one embodiment, there may be multiple models for a single application or entity, which can be utilized based on multiple different factors.

[0183] In at least one embodiment, at a subsequent point in time, a request for content (e.g., path determination) or data that is at least partially determined or influenced by a trained neural network can be received from client device 502 (or another such device). This request may include, for example, input data to be processed using the neural network to obtain one or more inference or other output values, classifications, or predictions. Alternatively, in at least one embodiment, the input data may be received by interface layer 508 and directed to inference module 518, although different systems or services may also be used. In at least one embodiment, if not already locally stored in inference module 518, inference module 518 may obtain a suitably trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 516. Inference module 518 may provide data as input to the trained network, which may then generate one or more inferences as outputs. This may, for example, include the classification of instances of input data. In at least one embodiment, the inference may then be transmitted to client device 502 for display to a user or for other communication with the user. In at least one embodiment, user context data may also be stored in a user context data repository 522, which may include data about the user that can be used as network input to generate inference or determine data returned to the user after obtaining an instance. In at least one embodiment, relevant data, including at least some of the input or inference data, may also be stored in a local database 534 for processing future requests. In at least one embodiment, the user may use account information or other information to access resources or functions of the vendor environment. In at least one embodiment, user data may also be collected and used to further train the model, if permitted and available, to provide more accurate inference for future requests. In at least one embodiment, requests to a machine learning application 526 executed on a client device 502 may be received via a user interface, and the results may be displayed via the same interface. The client device may include resources such as a processor 528 and a memory 562 for generating requests and processing results or responses, and at least one data storage element 552 for storing data for the machine learning application 526.

[0184] In at least one embodiment, processor 528 (or the processor of training module 512 or inference module 518) will be a central processing unit (CPU). However, as described above, resources in such an environment can utilize GPUs to process data for at least some types of requests. GPUs such as the PPU 300, with thousands of cores, are designed to handle large amounts of parallel workloads and have therefore become popular in deep learning for training neural networks and generating predictions. While using GPUs for offline building allows for faster training of larger, more complex models, offline prediction generation means that request-time input features cannot be used, or predictions must be generated for all features and stored in a lookup table for real-time service requests. If the deep learning framework supports CPU mode and the model is small and simple enough that the feedforward can be performed on the CPU with reasonable latency, then a service on a CPU instance can host the model. In this case, training can be done offline on the GPU and inference can be performed in real-time on the CPU. If the CPU approach is not feasible, the service can run on a GPU instance. However, due to the different performance and cost characteristics of GPUs compared to CPUs, running a service that offloads runtime algorithms to the GPU may require it to be designed differently from a CPU-based service.

[0185] In at least one embodiment, video data can be provided from client device 502 for enhancement in vendor environment 506. In at least one embodiment, the video data can be processed for enhancement on client device 502. In at least one embodiment, the video data can be streamed from third-party content provider 524 and enhanced by third-party content provider 524, vendor environment 506, or client device 502. In at least one embodiment, video data can be provided from client device 502 for use as training data in vendor environment 506.

[0186] In at least one embodiment, supervised and / or unsupervised training may be performed by client device 502 and / or vendor environment 506. In at least one embodiment, a set of training data 514 (e.g., classified or labeled data) is provided as input for use as training data. In at least one embodiment, the training data may include instances of at least one type of object for which the neural network is to be trained, and information identifying that object type. In at least one embodiment, the training data may include a set of images, each image including a representation of an object of a type, wherein each image also includes, or is associated with, tags, metadata, classification, or other information identifying or identifying the type of object represented in the corresponding image. Various other types of data may also be used as training data, which may include text data, audio data, video data, and so on. In at least one embodiment, training data 514 is provided as training input to training module 512. In at least one embodiment, training module 512 may be a system or service including hardware and software, such as one or more computing devices executing a training application for training a neural network (or other model or algorithm, etc.). In at least one embodiment, training module 512 receives instructions or requests indicating the type of model to be used for training. In at least one embodiment, the model can be any suitable statistical model, network, or algorithm useful for such a purpose, which may include artificial neural networks, deep learning algorithms, learning classifiers, Bayesian networks, etc. In at least one embodiment, training module 512 may select an initial model or other untrained models from an appropriate repository and train the model using training data 514 to generate a trained model (e.g., a trained deep neural network) that can be used to classify similar types of data or generate other such inference. In at least one embodiment where training data is not used, an initial model can still be selected to train on the input data of each training module 512.

[0187] In at least one embodiment, the model can be trained in several different ways, which may depend in part on the type of model chosen. In at least one embodiment, a training dataset can be provided to a machine learning algorithm, wherein the model is a model artifact created through a training process. In at least one embodiment, each instance of the training data includes the correct answer (e.g., classification) that may be referred to as a target or target attribute. In at least one embodiment, the learning algorithm finds patterns in the training data that map input data attributes to the target—the answer to be predicted—and the machine learning model is the output that captures these patterns. In at least one embodiment, the machine learning model can then be used to obtain predictions for new data without a specified target.

[0188] In at least one embodiment, the training and inference manager 532 may select from a set of machine learning models, including binary classification, multi-class classification, generative, and regression models. In at least one embodiment, the type of model to be used may depend at least in part on the type of target to be predicted.

[0189] Graphics processing pipeline

[0190] In one embodiment, PPU 400 includes a graphics processing unit (GPU). PPU 400 is configured to receive commands specifying a shader program for processing graphics data. Graphics data can be defined as a set of primitives such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying the number of vertices used for that primitive (e.g., in a model-space coordinate system) and attributes associated with each vertex of that primitive. PPU 400 can be configured to process graphics primitives to generate framebuffers (e.g., pixel data for each of the pixels in a display).

[0191] The application writes model data (such as attributes and vertex sets) for the scene into memory such as system memory or memory 404. The model data defines each of the objects that may be visible on the display. The application then makes API calls to the driver kernel, requesting that the model data be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations that process the model data. These commands may reference different shader programs to be implemented on the processing units within the PPU 400, including one or more vertex shaders, shell shaders, domain shaders, geometry shaders, and pixel shaders. For example, one or more processing units may be configured to execute a vertex shader program that processes a number of vertices defined by the model data. In one embodiment, these different processing units may be configured to execute different shader programs concurrently. For example, a first subset of processing units may be configured to execute a vertex shader program, while a second subset of processing units may be configured to execute a pixel shader program. The first subset of processing units processes the vertex data to produce processed vertex data and writes the processed vertex data into L2 cache 460 and / or memory 404. After the processed vertex data is rasterized (e.g., transformed from 3D data to 2D data in screen space) to produce fragment data, a second subset of processing units executes pixel shaders to produce processed fragment data, which is then mixed with other processed fragment data and written to the frame buffer in memory 404. Vertex shader and pixel shader programs can execute concurrently, pipelinedly processing different data from the same scene until all model data for that scene has been rendered to the frame buffer. The contents of the frame buffer are then transferred to the display controller for display on the display device.

[0192] Figure 6A According to one embodiment, by Figure 4 A conceptual diagram of a graphics processing pipeline 600 implemented by a PPU 400. The graphics processing pipeline 600 is an abstract flowchart of the processing steps implemented to generate 2D computer-generated images from 3D geometric data. It is well known that pipeline architectures can more efficiently perform long-latency operations by dividing operations into multiple stages, where the output of each stage is coupled to the input of the next successive stage. Therefore, the graphics processing pipeline 600 receives input data 601 passed from one stage of the graphics processing pipeline 600 to the next stage to generate output data 602. In one embodiment, the graphics processing pipeline 600 may represent a system... The graphics processing pipeline is defined by the API. As an option, the graphics processing pipeline 600 can be implemented within the context of the functionality and architecture of previous figures and / or one or more subsequent figures.

[0193] like Figure 6A As shown, the graphics processing pipeline 600 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 610, a vertex shading stage 620, a primitive assembly stage 630, a geometry shading stage 640, a viewport scale, cull, and clip (VSCC) stage 650, a rasterization stage 660, a fragment shading stage 670, and a raster operation stage 680. In one embodiment, input data 601 includes commands that configure processing units to implement the stages of the graphics processing pipeline 600 and configure geometric primitives (e.g., points, lines, triangles, quadrilaterals, triangular strips, or sectors, etc.) to be processed by these stages. Output data 602 may include pixel data (e.g., color data) that is copied to a framebuffer or other type of surface data structure in memory.

[0194] The data assembly stage 610 receives input data 601, which specifies vertex data for higher-order surfaces, primitives, etc. The data assembly stage 610 collects vertex data from a temporary storage device or queue, for example, by receiving a command from the host processor including a pointer to a buffer in memory and reading vertex data from that buffer. The vertex data is then transferred to the vertex shading stage 620 for processing.

[0195] The vertex shading stage 620 processes vertex data by performing a set of operations (e.g., a vertex shader or procedure) once for each vertex. A vertex can be specified, for example, as a 4-coordinate vector (e.g., color, texture coordinates, surface normals, etc.) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normals, etc.).<x,y,z,w> The vertex shading stage 620 can manipulate individual vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 620 performs operations on the vertex coordinates or other vertex attributes associated with a vertex. Such operations typically include lighting operations (e.g., modifying the vertex's color attributes) and transformation operations (e.g., modifying the vertex's coordinate space). For example, a vertex can be specified using coordinates in object coordinate space, which is transformed by multiplying the coordinates by a matrix that converts the coordinates from object coordinate space to world space or normalized-device-coordinate (NCD) space. The vertex shading stage 620 generates transformed vertex data that is then passed to the primitive assembly stage 630.

[0196] Primitive assembly stage 630 collects the vertices output by vertex shading stage 620 and groups the vertices into geometric primitives for processing by geometry shading stage 640. For example, primitive assembly stage 630 can be configured to group every three consecutive vertices into geometric primitives (e.g., triangles) for transmission to geometry shading stage 640. In some embodiments, specific vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). Primitive assembly stage 630 transmits geometric primitives (e.g., sets of associated vertices) to geometry shading stage 640.

[0197] The geometry shading stage 640 processes geometry primitives by performing a set of operations (e.g., geometry shaders or procedures) on them. Tessellation operations can generate one or more geometry primitives from each geometry primitive. In other words, the geometry shading stage 640 can subdivide each geometry primitive into a finer mesh of two or more geometry primitives for processing by the rest of the graphics processing pipeline 600. The geometry shading stage 640 then passes the geometry primitives to the viewport SCC stage 650.

[0198] In one embodiment, the graphics processing pipeline 600 can operate within a streaming multiprocessor, and the vertex shading stage 620, primitive assembly stage 630, geometry shading stage 640, fragment shading stage 670, and / or associated hardware / software can sequentially perform processing operations. In one embodiment, the viewport SCC stage 650 can utilize data once the sequential processing operations are complete. In one embodiment, the metadata processed by one or more stages in the graphics processing pipeline 600 can be written to a cache (e.g., L1 cache, vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 650 can access the data in the cache. In one embodiment, the viewport SCC stage 650 and the rasterization stage 660 are implemented as a fixed-function circuit system.

[0199] In the Viewport SCC stage 650, viewport scaling, culling, and clipping of geometry are performed. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of the viewer viewing the scene and defines the viewing frustum that surrounds the objects in the scene. The viewing frustum may include the viewing plane, the back plane, and four clipping planes. Any geometry completely outside the viewing frustum can be culled (e.g., discarded) because it will not contribute to the final rendered scene. Any geometry partially inside and partially outside the viewing frustum can be clipped (e.g., transformed into new geometry enclosed within the viewing frustum). Furthermore, each geometry can be scaled based on the depth of the viewing frustum. All potentially visible geometry is then passed to the rasterization stage 660.

[0200] Rasterization stage 660 converts 3D geometric primitives into 2D fragments (e.g., fragments usable for display). Rasterization stage 660 can be configured to use the vertices of the geometric primitives to establish a set of planar equations from which various properties can be interpolated. Rasterization stage 660 can also compute an overlay mask of multiple pixels, indicating whether one or more sample locations of the pixels occlude the geometric primitives. In one embodiment, a z-test can also be performed to determine whether the geometric primitives are occluded by other geometric primitives that have already been rasterized. Rasterization stage 660 generates fragment data (e.g., interpolated vertex properties associated with specific sample locations of each occluded pixel), which is then passed to fragment shading stage 670.

[0201] Fragment shading stage 670 processes fragment data by performing a set of operations (e.g., fragment shader or procedure) on each fragment. Fragment shading stage 670 may generate pixel data (e.g., color values) for the fragment by performing lighting operations or sampling a texture map using interpolated texture coordinates of the fragment. The pixel data generated by fragment shading stage 670 is then passed to raster operation stage 680.

[0202] The rasterization stage 680 can perform various operations on the pixel data, such as performing alpha tests, stencil tests, and mixing the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the rasterization stage 680 has completed processing the pixel data (e.g., output data 602), the pixel data can be written to a rendering target, such as a frame buffer, color buffer, etc.

[0203] It should be understood that, in addition to or in place of one or more of the aforementioned stages, one or more additional stages may be included in the graphics processing pipeline 600. Various implementations of the abstract graphics processing pipeline can implement different stages. Furthermore, in some embodiments, one or more of the aforementioned stages may be excluded from the graphics processing pipeline (such as geometry shading stage 640). Other types of graphics processing pipelines are contemplated within the scope of this disclosure. Additionally, any stage of the graphics processing pipeline 600 may be implemented by one or more dedicated hardware units within a graphics processor, such as a PPU 400. Other stages of the graphics processing pipeline 600 may be implemented by programmable hardware units, such as processing units within the PPU 400.

[0204] The graphics processing pipeline 600 can be implemented via an application executed by a host processor such as a CPU. In one embodiment, the device driver can implement an application programming interface (API), which defines various functions that can be utilized by the application to generate graphics data for display. The device driver is a software program that includes multiple instructions controlling the operation of the PPU 400. The API provides an abstraction for the programmer, allowing the programmer to utilize dedicated graphics hardware such as the PPU 400 to generate graphics data without requiring the programmer to utilize the specific instruction set of the PPU 400. The application can include API calls to the device driver routed to the PPU 400. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform operations at least in part by initiating operations on the PPU 400 using the input / output interface between the CPU and the PPU 400. In one embodiment, the device driver is configured to utilize the hardware of the PPU 400 to implement the graphics processing pipeline 600.

[0205] Various programs can be executed within the PPU 400 to implement the various stages of the graphics processing pipeline 600. For example, a device driver can launch a kernel on the PPU 400 to execute the vertex shading stage 620 on a processing unit (or multiple processing units). The device driver (or the initial kernel executed by the PPU 400) can also launch other kernels on the PPU 400 to execute other stages of the graphics processing pipeline 600, such as the geometry shading stage 640 and the fragment shading stage 670. Additionally, some of the stages of the graphics processing pipeline 600 can be implemented on fixed-unit hardware such as a rasterizer or data assembler implemented within the PPU 400. It should be understood that the results from one kernel can be processed by one or more intermediate fixed-function hardware units before subsequent kernel processing on the processed unit.

[0206] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device may be directly coupled to the system or processor that generated or rendered the image. In other embodiments, the display device may be indirectly coupled to the system or processor, for example, via a network. Examples of such networks include the Internet, mobile telecommunications networks, Wi-Fi networks, and any other wired and / or wireless networking systems. When the display device is indirectly coupled, images generated by the system or processor can be streamed to the display device over the network. Such streaming allows, for example, video games or other applications with rendered images to execute on servers, data centers, or cloud-based computing environments, and the rendered images are transmitted and displayed on one or more user devices (e.g., computers, video game consoles, smartphones, other mobile devices, etc.) physically separate from the server or data center. Therefore, the techniques disclosed herein can be applied to enhance streamed images and services that stream images, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0207] Example game streaming system

[0208] Figure 6B This is a schematic diagram of an example system 605 for a game streaming system according to some embodiments of the present disclosure. Figure 6B Including game server 603 (which may include and Figure 5A Example processing system 500 and / or Figure 5B (Similar components, features and / or functions to exemplary system 565), client 604 (which may include similar ... Figure 5A Example processing system 500 and / or Figure 5B The exemplary system 565 has similar components, features, and / or functions to the network 606 (which may be similar to the network described herein). In some embodiments of this disclosure, system 605 may be implemented.

[0209] In system 605, for a game session, client device 604 can simply receive input data in response to input from an input device, send the input data to game server 603, receive encoded display data from game server 603, and display the display data on monitor 624. In this way, computationally intensive computation and processing are offloaded to game server 603 (e.g., rendering of the game session's graphics output, especially ray or path tracing, is performed by the GPU of game server 603). In other words, the game session is streamed from game server 603 to client device 604, thereby reducing the demands on client device 604 for graphics processing and rendering.

[0210] For example, regarding the instantiation of a game session, client device 604 can display frames of the game session on display 624 based on display data received from game server 603. Client device 604 can receive input from one of the input devices and generate input data in response. Client device 604 can send the input data to game server 603 via communication interface 621 and through network 606 (e.g., the Internet), and game server 603 can receive the input data via communication interface 618. CPU can receive the input data, process the input data, and send the data to GPU, which causes GPU to generate a rendering of the game session. For example, the input data can represent the movement of a user character in the game, such as firing a weapon, reloading, passing a ball, turning a vehicle, etc. Rendering component 612 can render the game session (e.g., representing the result of the input data), and rendering capture component 614 can capture the rendering of the game session as display data (e.g., image data as frames of the captured game session rendering). The rendering of a game session may include lighting and / or shadow effects computed using one or more parallel processing units of the game server 603 (e.g., a GPU, which may further employ one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques). The encoder 616 can then encode the display data to generate encoded display data, which can be sent to the client device 604 via the network 606 through the communication interface 618. The client device 604 can receive the encoded display data via the communication interface 621, and the decoder 622 can decode the encoded display data to generate display data. The client device 604 can then display the display data via the display 624.

[0211] It should be noted that the techniques described herein may include executable instructions stored in a computer-readable medium for use by or in conjunction with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will appreciate that, for some embodiments, various different types of computer-readable media may be included for storing data. When used herein, “computer-readable medium” includes one or more of any suitable media for storing executable instructions of a computer program, such that an instruction execution machine, system, apparatus, or device may read (or retrieve) the instructions from the computer-readable medium and execute those instructions to implement the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable compact discs (CDs), portable digital video discs (DVDs), and the like.

[0212] It should be understood that the arrangement of components shown in the accompanying drawings is for illustrative purposes, and other arrangements are possible. For example, one or more of the elements described herein may be implemented wholly or partially as electronic hardware components. Other elements may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other elements may be combined, some may be omitted entirely, and additional components may be added while still achieving the functionality described herein. Therefore, the subject matter described herein can be implemented in many different variations, and all such variations are contemplated to be within the scope of the claims.

[0213] To facilitate understanding of the topics described herein, many aspects are described in sequence of actions. Those skilled in the art will recognize that various actions can be performed by dedicated circuitry or circuit systems, by program instructions executed by one or more processors, or by a combination of both. The description of any sequence of actions herein is not intended to imply that a particular order in which the actions described for execution must be followed. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context.

[0214] In the context of describing the subject matter (especially in the context of the claims below), the use of the terms “a,” “an,” “this,” and similar designations should be interpreted to cover both the singular and the plural, unless otherwise specified herein or obviously contradicted by the context. The use of the term “at least one” (e.g., at least one of A and B) followed by a list of one or more items should be interpreted to mean one item selected from the listed items (A or B), or any combination of two or more of the listed items (A and B), unless otherwise specified herein or obviously contradicted by the context. Furthermore, the foregoing description is for illustrative purposes only and not for limiting purposes, as the scope of protection sought is defined by the claims set forth thereafter with their equivalents. The use of any and all example or exemplary language provided herein (e.g., “such as”) is intended merely to better describe the subject matter and does not constitute a limitation on the scope of the subject matter, unless otherwise stated. The use of “based on,” and other similar phrases indicating the conditions leading to the result, in both the claims and the written description, is not intended to exclude any other conditions leading to that result. The language in the description should not be interpreted as indicating that any unclaimed element is essential to the implementation of the claimed invention.

Claims

1. A computer-implemented method for constructing a three-dimensional (3D) model, comprising: Receive a 3D topology definition, which identifies the faces of cells within a volume mesh; The 3D topology definition and the volume mesh are processed according to parameters to predict a 3D model representation corresponding to a reference 3D model, wherein the 3D model representation includes a surface mesh defined by the vertex positions of the faces and the connections between the vertex positions; Render the 3D model representation to produce a rendered image; The ray transmission corresponding to the rendered image is represented by the 3D model to calculate the probability value of the unit; as well as The parameters are updated based on the probability value and the difference between the rendered image and the reference image of the reference 3D model to reduce the difference.

2. The computer-implemented method of claim 1, wherein the surfaces of the surface mesh are opaque.

3. The computer-implemented method according to claim 1, further comprising: Render additional images associated with different camera positions or lighting positions; The additional ray transmission corresponding to the additional rendered image is represented by the 3D model to calculate the additional probability value of the unit; as well as The parameters are updated based on the additional probability value and the additional difference between the additional rendered image and the additional reference image of the reference 3D model corresponding to the different camera positions or lighting positions, in order to reduce the additional difference.

4. The computer-implemented method according to claim 1, wherein the 3D model represents material properties including spatial variations.

5. The computer-implemented method of claim 1, wherein the rendered image is generated using a differentiable renderer, and the difference is backpropagated through the differentiable renderer to update the parameters.

6. The computer-implemented method of claim 1, wherein each probability value is associated with a different facet of each of the units.

7. The computer-implemented method of claim 1, wherein each probability value is associated with all faces of each element in the unit.

8. The computer-implemented method of claim 1, wherein the processing, rendering, transmission, and updating include training iterations and further include repeating the training iterations for an additional 3D topology definition.

9. The computer-implemented method of claim 8, wherein the 3D topology definition and the additional 3D topology definition are randomly generated for each training iteration.

10. The computer-implemented method of claim 8, wherein at least one of the additional 3D topology definitions is determined based on the probability value.

11. The computer-implemented method of claim 8, wherein the volume mesh is subdivided for at least one of the additional 3D topology definitions.

12. The computer-implemented method of claim 1, wherein the volume mesh is initialized as a uniform 3D mesh.

13. The computer-implemented method of claim 1, wherein the rendering and transmission are performed in parallel.

14. The computer-implemented method of claim 1, wherein the update comprises: The probability value is used to calculate the pixel probability value for each pixel in the rendered image; The product is calculated by scaling the difference for that pixel with the probability value for each pixel; as well as The product is summed to generate the image loss.

15. The method of claim 1, wherein the volume mesh is a dense volume mesh.

16. A system comprising: The processor constructs 3D models in the following ways: Receive a 3D topology definition, which identifies the faces of cells within a volume mesh; The 3D topology definition and the volume mesh are processed according to parameters to predict a 3D model representation corresponding to a reference 3D model, wherein the 3D model representation includes a surface mesh defined by the vertex positions of the faces and the connections between the vertex positions; Render the 3D model representation to produce a rendered image; The ray transmission corresponding to the rendered image is represented by the 3D model to calculate the probability value of the unit; as well as The parameters are updated based on the probability value and the difference between the rendered image and the reference image of the reference 3D model to reduce the difference.

17. The system of claim 16, wherein the surfaces of the surface mesh are opaque.

18. The system of claim 16, wherein the rendered image is generated using a differentiable renderer, and the difference is backpropagated through the differentiable renderer to update the parameters.

19. The system of claim 16, wherein the processor is further configured to update the parameters in the following manner: The probability value is used to calculate the pixel probability value for each pixel in the rendered image; The product is calculated by scaling the difference for that pixel with the probability value for each pixel; and The product is summed to generate the image loss.

20. A non-transitory computer-readable medium storing computer instructions for constructing a three-dimensional 3D model, the computer instructions, when executed by one or more processors, causing the one or more processors to perform the following steps: Receive a 3D topology definition, which identifies the faces of cells within a volume mesh; The 3D topology definition and the volume mesh are processed according to parameters to predict a 3D model representation corresponding to a reference 3D model, wherein the 3D model representation includes a surface mesh defined by the vertex positions of the faces and the connections between the vertex positions; Render the 3D model representation to produce a rendered image; The ray transmission corresponding to the rendered image is represented by the 3D model to calculate the probability value of the unit; as well as The parameters are updated based on the probability value and the difference between the rendered image and the reference image of the reference 3D model to reduce the difference.

21. The non-transitory computer-readable medium of claim 20, wherein the face of the surface mesh is opaque.

22. A computer-implemented method for constructing a three-dimensional (3D) model, comprising: Receive a 3D topology definition, which identifies visible faces of cells within a volume mesh, the volume mesh being defined by the vertex positions of the faces and the connections between the vertex positions; The volume mesh is deformed according to parameters to predict an opaque surface mesh defined by the vertex positions of the visible faces and the connections between those vertex positions, wherein the opaque surface mesh corresponds to a reference 3D model; and The parameters are updated based on the difference between the rendered image of the opaque surface mesh and the reference image of the reference 3D model to reduce the difference.

23. The computer-implemented method according to claim 22, further comprising: The deformation and the update are repeated for an additional 3D topology definition, wherein the additional 3D topology definition is different from the original 3D topology definition.

24. The computer-implemented method of claim 23, wherein the additional 3D topology definition is defined based on calculated probability values ​​of the cells within the volume mesh, the calculated probability values ​​indicating the likelihood that each cell will intersect with a ray corresponding to the reference image.

Images