Neural network control variables

Abstract

The present invention discloses neural network control variables. Monte Carlo and quasi-Monte Carlo integration are simple numerical methods for solving complex integral problems, such as evaluating financial derivatives or synthesizing realistic images through light transport simulations. The disadvantage of directly applying (quasi) Monte Carlo integration is the relatively slow convergence rate, which manifests itself as a high error in the Monte Carlo estimator. Neural control variables can be used to reduce the error of the parametric (quasi) Monte Carlo integration, providing a more accurate solution in a shorter time. The neural network system has sufficient approximation power to estimate the integral and is an efficient evaluation. The efficiency stems from the use of a first neural network that derives the integral of the control variable and uses a normalized flow to model the shape of the control variable.

Description

[0001] Claiming priority

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 992,717, filed on March 20, 2020, entitled “Neural Control Variates Using Normalizing Flows,” which is incorporated herein by reference in its entirety. Background Art

[0003] Monte Carlo and quasi-Monte Carlo integration are simple numerical methods for solving complex integral problems. Examples of complex integral problems include evaluating financial derivatives or synthesizing realistic images through light transport simulations. (Quasi-) Monte Carlo is particularly attractive when there is no analytical solution to the integral problem and classical numerical algorithms (e.g. finite elements) fail due to the curse of dimensionality or lack of smoothness of the integrand.

[0004] The main drawback of direct application of Monte Carlo integration is the relatively slow convergence rate, which manifests itself in high variance for Monte Carlo estimators and high error for quasi-Monte Carlo estimators. Therefore, methods have been developed to improve computational efficiency. Among the most commonly used methods, there are techniques focusing on careful placement of samples, such as oppositional sampling, stratification, quasi-random sampling, or importance sampling. A complementary approach to further reduce variance is to exploit the concept of hierarchical integration or control variables.

[0005] Reducing the variance by control variables is equivalent to using an approximate solution of the integral corrected by an estimate of the approximation error. The numerical efficiency of estimating the integral depends on the scaled control variables. Since poorly chosen control variables can reduce efficiency, early research focused on efficient and accurate estimation of the scaling factor α. Although the optimal variance-minimizing value of α is known, numerically estimating α may introduce bias if it is done using samples related to the samples used for the actual estimation. The core challenge in applying the control variable method is to find a good approximation to the integrand that is efficient to integrate in terms of computation and / or time. There is a need to solve these problems and / or other problems associated with the prior art. Summary of the invention

[0006] The application of control variables has been explored in many fields, mainly in financial mathematics and operations research. Control variables can be used to reduce the estimation error generated by Monte Carlo or quasi-Monte Carlo integration. Compared with traditional systems (such as the one described above), the control variable generated by a neural network satisfies two properties: it approximates the integrand well and its integral can be calculated efficiently.

[0007] Embodiments of the present disclosure relate to neural network control variables. Systems and methods for parameterizing and learning control variables that have large approximation capabilities while still having efficiently computable integrals are disclosed. A first neural network is used to predict a first parameter integral G(y) of a first function g(x,y) based on an input y, and a second function integral to unity is used to predict a first parameter integral G(y) of a first function g(x,y) based on an input y. Approximate the shape of a first function, wherein parameters of a second function are computed by at least one neural network. Compute the first function as a product of a first parameter integral and a second function, and use the product to estimate a difference integral. Combine the difference integral and the first parameter integral to estimate a solution to a second parameter integral F(y), and use the solution to approximate a linear operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present system and method for controlling variables using a neural network are described in detail below with reference to the accompanying drawings, wherein:

[0009] Figure 1A Calculation of the integral F of the function f(x) using the control variable g(x) according to one embodiment is shown.

[0010] Figure 1B A reference light field, a light field predicted by a control variable neural network system, and an image of a difference integration are shown according to one embodiment.

[0011] Figure 1C A block diagram of a control variable neural network system according to one embodiment is shown.

[0012] Figure 2A A flow chart of a method for generating control variables using a neural network system suitable for implementing some embodiments of the present disclosure is shown.

[0013] Figure 2B A graph showing a control variable integral G and coefficients a predicted by a control variable neural network system and a probability of selection according to one embodiment.

[0014] Figure 3A is a conceptual diagram illustrating the use of neural control variables for light transport simulation, according to one embodiment.

[0015] Figure 3B Path vertices and the corresponding induced area expansion are shown according to one embodiment.

[0016] Figure 3C A flow chart of a method for generating an image of a 3D scene according to one embodiment is shown.

[0017] Figure 4 An example parallel processing unit suitable for implementing some embodiments of the present disclosure is shown.

[0018] Figure 5A is used Figure 4 A conceptual diagram of a PPU-implemented processing system suitable for implementing some embodiments of the present disclosure.

[0019] Figure 5B Exemplary systems are shown in which the various architecture and / or functionality of various previous embodiments may be implemented.

[0020] Figure 5C Components of an exemplary system that can be used to train and utilize machine learning in at least one embodiment are shown. DETAILED DESCRIPTION

[0021] Systems and methods related to neural network control variables are disclosed. A neural control variable (NCV) is a control variable predicted by a neural network system that can be used to perform unbiased variance reduction in parametric Monte Carlo integration or error reduction in parametric quasi-Monte Carlo integration when solving an integral equation. The neural network system implements a normalized flow to approximate the shape of an integrand and derive solutions to the integral equation. The neural network system parameterizes and learns a control variable with large approximation power but still with an effectively computable integral. The integral predicted by a first neural network is combined with a parameterized normalized flow implemented using one or more second neural networks to parameterize the control variable. Furthermore, in one embodiment, the first and second neural networks are trained using only a noisy estimate of the integrand.

[0022] The integral of the light transport equation used to render realistic images, particularly for indirect illumination via path tracing (tracing rays through a scene), can be estimated. Using control variables calculated by a neural network system can improve the efficiency of path tracing. Conventionally, the number of paths traced is increased to reduce noise in the rendered image. Using control variables calculated by a neural network system can result in less noise with the same number of paths, or the same noise with a smaller number of paths.

[0023] Using control variables to reduce variance can make use of an approximate solution to the integral that can be corrected by an estimate of the approximation error. This principle is given by the following identity:

[0024]

[0025] Instead of integrating the original function f to obtain the solution F, we use an α-scaling approximation G. The approximation G corresponds to integrating a (different) function g (the control variable) over the same domain D, i.e. The approximation error is corrected by adding the integral of the difference f(x)-α·g(x); making the right-hand side of equation (1) equal to the left-hand side.

[0026] Figure 1AThe use of control variables 102g(x) to calculate the integral F of function 101f(x) according to one embodiment is shown. The integral F of function 101f(x) is calculated with the help of control variables 102g(x) by using a known or easily calculated integral G plus the integral of difference 104. The overall variance can be reduced by minimizing the variance of the difference using an α-scaled control variable (where α = Cov(f,g) / Var(g)).

[0027] The numerical efficiency of estimating the right side of equation (1) relative to estimating the original integral can depend on the scaled control variable 103, which makes the integration easier. For example, the difference 104 may expose less variance than the difference 105 of the unscaled control variable 102. As long as the function or integrand f(x) is uncorrelated with the control variable g(x), the difference of the integral will usually be smoother. In practice, a scaling factor α of the intensity of the control application control variable 102 should be derived from the correlation of the two functions. In short, the successful application of the control variable requires that the control variable approximates the integrand well enough and allows efficient estimation and calculation of the integral G and the scaling α. In the context of the following description, the integrand f(x) and the control variable g(x) may be referred to as f and g, respectively.

[0028] As described further in the paper, machine learning can be used to infer the control variable g(x) from the observation of f(x). It is often difficult to normalize the output of a neural network. Therefore, the output of the neural network is used to parameterize a transformation that can be used to deform the function without changing the integral of the function. Using transformations allows learning to learn by constructing a normalized function. Such a model is called a normalized flow.

[0029] Since the control variables are learned, the key challenge becomes to (efficiently) allow computing their integrals The control variable can be expressed in the form of ) and integral G to avoid the difficulty of calculating G, so that, for example The shape and integral can be modeled independently. In one embodiment, the integral G is inferred using a first neural network, and the coefficients α are inferred using another neural network.

[0030] In one embodiment, to calculate the shape of the control variable A custom variant of the normalized stream is used. In contrast to a standard neural network, the normalized stream is capable of representing a normalized function. In one embodiment, a set of neural networks including at least one neural network is used to infer the parameters of the normalized stream. A normalizing flow is a technique for mapping an arbitrary distribution to a base distribution; for example, the normal distribution. The mapping is formally obtained by chaining a series of infinitesimal transformations, hence the name flow.

[0031] Figure 1B An image of a reference light field 150, a neural light field 160, and an error (difference integral) 155 is shown according to one embodiment. A control variable neural network system can be used to predict the neural light field 160 and estimate the error 155 using path tracing. Ideally, the tracing of paths ends at the light source that opens the door because these paths illuminate the scene. Paths that bounce around in the scene without reaching the light source will introduce noise to the rendered image. The control variable neural network system can be trained to learn the neural light field 160 so that fewer paths, especially important paths, are tracked to produce higher quality images. Using the example of light transport, the error 155 is a difference integral, and the error (difference integral) 155 can be summed with G (e.g., the predicted neural light field 160) to produce F (e.g., the true or reference light field 150). The error (difference integral) 155 is the integral of the difference between the integrand f(x) and the control variable g(x), where g(x) can be scaled by α.

[0032] To ensure that the predicted light field 160 is accurate, the error (difference integral) 155 should be rebound. The challenge is that the error (difference integral) is analytically unknown and can therefore be estimated using path tracing. To minimize noise, the control variables in the difference integral are approximated as the product terms of the rendering equations used to produce the reference light field 150. The control variables are learned and then integrated to compute the neural light field G, e.g. is very complex. Therefore, as mentioned before, the control variable is expressed as its integral G and its shape The shape always integrates to the value 1, so no matter what the shape is, multiplying the shape by G equals both sides of the neural light field equation 1 Probability density functions (PDFs) are non-negative normalized functions that can be integrated to unity and can be represented by a normalized flow controlled by a trained neural network.

[0033] Figure 1CA block diagram of a control variable neural network system 100 according to one embodiment is shown. It should be understood that this arrangement and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components and implemented in any suitable combination and location. The various functions described herein as being performed by an entity may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. In addition, one of ordinary skill in the art will appreciate that any system that performs the operations of the control variable neural network system 100 is within the scope and spirit of the embodiments of the present disclosure.

[0034] The control variable neural network system 100 receives input 115 and generates integral estimate 140. The control variable neural network system 100 reduces the variance of the estimated parameter integral F through machine learning. In one embodiment, the input 115 is the parameter y. Figure 1B Example of light transport where F(y) is the reflected radiation field and the parameter y represents the reflection position and direction.

[0035] The relationship between F, the integral estimate 140, and g(x), the control variable (scaling factor α omitted), is: F(y) = ∫f(x,y)dx = ∫f(x,y)+g(x,y)-g(x,y)dx, where g(x,y)-g(x,y) = 0. Then, F(y) = ∫g(x,y)dx+∫f(x,y)-g(x,y)dx. The integral of the control variable is G(y) = ∫g(x,y)dx, yielding

[0036] F(y)=G(y)+∫f(x,y)-g(x,y)dx. Equation (2)

[0037] In many applications, especially in computer graphics, the functions F(y) and f(x,y) may have infinite variations and lack smoothness. Therefore, the model used to approximate the function needs to be flexible and highly expressive. In one embodiment, a set of optimizable parameters θ is used. g The one or more neural networks driven by the control variables are modeled.

[0038] Solving F(y) using equation (2) requires G(y) and g(x,y). Learning the control variables g(x,y) and computing the integral G(y) from g(x,y) is difficult and / or impractical (except for perhaps very simple models). Learning the control variables g(x,y) and the integral G(y) separately is difficult because the equality between G(y) and the integral of the control variables should be preserved. Therefore, the control variable neural network model 125 learns the integral G(y), and the normalized flow neural network model 110 learns the parameters of the normalizing function (e.g., PDF) applied by the shape calculation 120 to generate the normalized flow

[0039] The normalized flow is the normalized shape of G(y) and has the property of integrating to one, so G(y) can be calculated as:

[0040]

[0041] The first neural network model, the integral prediction G(y) of the control variable neural network model 125. G(y) is exactly the integral of g(x,y) and a set of second neural networks, which include the normalized flow neural network model 110, the shape calculation 120 predicts the shape Multiply 130 to calculate G(y) and The product of is equal to the control variable g(x,y).

[0042] The control variable is calculated by product 130 as the product of G(y) and the normalized flow. Equation (2) can be evaluated by integral function calculation 135 using the calculated control variable (and integral G(y)) to estimate the solution of the integral function F(y). Specifically, the integral of the difference between f(x) and g(x, y) is summed with G(y) to produce F(y). In one embodiment, the integral of the control variable and the shape of the control variable are independently learned by the integral of the control variable neural network model 125 and the normalized flow neural network model 110.

[0043] In order to use equations (1) and (2), the integration of the control variable neural network model 125 should allow for the control variables g(x, y; θ g ) and its integral G(y) = ∫g(x, y; θ g )dx. This turns out to be the key challenge. Modeling g using a neural network may be expressive enough, but computing the integral G will require some form of numerical integration, which requires multiple forward passes to evaluate g(x, y; θ g ); the cost is too high.

[0044] By restricting g to a function of known integrals, the need for multiple passes of the integral can be avoided. In one embodiment, g is restricted to a normalized function that integrates to unity. It is still possible to match an arbitrary integrand by scaling the normalized function by a (learned) factor. Thus, the parameter controls the variable

[0045]

[0046] is defined as the product of two components: the parametric normalization function and the parameter scalar value G(y;θ G ). In the context described below, and G are called the shape and integral of the control variable, respectively, and they are each parameterized by a set of parameters; The advantage of decomposing into separate parameter sets is that computing the integral G is equivalent to evaluating the integral of the control variable neural network model 125 once, rather than performing g(x, y; θ) which requires a large number of neural network evaluations. g ) is an expensive numerical integration.

[0047] In one embodiment, the integral value G(y; θ) output by the integral of the control variable neural network model 125 can be G ) is raised to a power, thereby allowing the integral of the control variable neural network model 125 to output a high dynamic range of values ​​while operating numerically better within a low dynamic range. In one embodiment, the combination of a always positive power representation and a normalized flow for the shape of the control variable constrains the control variable to be a non-negative function; negative values ​​are excluded by design.

[0048] Now, more illustrative information about various optional architectures and features that can be used to implement the aforementioned framework will be given according to the needs of the user. It should be particularly noted that the following information is provided for illustrative purposes and should not be interpreted as limiting in any way. Any of the following features can be incorporated into or not exclude other features described as needed.

[0049] Figure 2A A flow chart of a method 200 for generating control variables using a neural network system suitable for implementing some embodiments of the present disclosure is shown. Each block of the method 200 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method can also be embodied as computer usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, a service or a hosted service (standalone or in combination with other hosted services), or a plug-in for another product, to name a few. In addition, by way of example, with respect to Figure 1CThe method 200 is described with reference to the control variable neural network system 100. However, the method may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein. In addition, a person of ordinary skill in the art will understand that any system that performs the method 200 is within the scope and spirit of the embodiments of the present disclosure.

[0050] At step 205, a first parameter integral G(y) of a first function g(x,y) is predicted based on input y using a first neural network. In one embodiment, the first parameter integral is a scaled parameter integral α(y)G(y). In one embodiment, the first neural network is an integral of a control variable neural network model 125.

[0051] Since the control variable may not match f exactly, in one embodiment, the control variable is weighted by the control variable coefficient α that controls its contribution. As mentioned earlier, the optimal variance-minimizing value of α(y) is known to be Cov(f(x,y),g(x,y)) / Var(g(x,y)). However, computing the optimal value, which generally varies with y, can be very expensive in practice. Therefore, a coefficient neural network α(y;θ) can be used instead. α ) models the coefficients, which are trained to output the appropriate contribution of the control variable according to the parameter y. The loss function used to optimize the coefficient neural network can be used to minimize the variance of the Monte Carlo estimate or the error of the Quasi-Monte Carlo estimate.

[0052] In step 210, a second function integrating to unity is used Approximate the shape of the first function, wherein the parameters of the second function are calculated by at least one neural network. In one embodiment, the at least one neural network is a normalized flow neural network model 110. The normalized flow is a computational graph representing a differentiable multi-dimensional composite mapping for transforming probability density. The mapping includes L bijective warping functions Therefore, it is also bijective overall. The warping function Density changes due to variable change formula

[0053]

[0054] where p is the probability density, x∈x is the warped independent variable, is the warped output, is the Jacobian matrix of h at x.

[0055] The density change caused by a chain of L such warps can be obtained by invoking the chain rule. This yields the following product of the absolute values ​​of the Jacobian:

[0056]

[0057] where x1 = x. The i-th term in the product represents the absolute value of the Jacobian of the i-th warp relative to the output of the i-1-th warp.

[0058] Transformed variables It is often referred to as the latent variable in the latent space L. Its distribution is related to the distribution of the input variables by combining equations (4) and (5):

[0059]

[0060] Latent variables The distribution of is usually chosen to be simple and easy to sample. In one embodiment, a uniform distribution over the unit hypercube is used

[0061] To achieve high modeling power, neural normalization flows exploit parametric warps driven by the outputs of a neural network. To allow modeling correlations across dimensions, the outputs of individual warps need to be fed into the neural network to condition subsequent warps in the flow. In the context of probabilistic modeling, two general approaches are autoregressive flows and coupled flows. Both of these approaches produce flows that (i) are reversible, (ii) avoid the cubic cost of computing the determinant of the dense Jacobian, and (iii) avoid the need to differentiate through the neural network to compute the correlation terms in the Jacobian.

[0062] To generate the control variable shape Efficient reversibility of the flow is not required, since modeling the control variable shape only requires evaluating the flow in one direction. However, it is still possible to utilize the previously proposed autoregressive formulation to ensure a tractable Jacobian. Furthermore, in the case of learning multiple densities (specifically, multiple channels of control variables), models such as the normalized flow neural network model 110 can be further accelerated.

[0063] In one embodiment, a unit hypercube having the same dimension as g is used as the latent space L.

[0064] The normalized control variables are then modeled as

[0065]

[0066] It is worth noting that the product on the right hand side is normalized by construction: the probability density Normalized by definition, each warp in the flow only redistributes the density without changing the overall mass. This is to ensure that during training The key to keep normalization.

[0067] In one embodiment, the warping in the normalizing flow assumes an autoregressive structure: the output x of the i-th warping I+1 for dimension d in only depends on the previous dimensions in the input x I :

[0068]

[0069] where the superscript <d denotes the previous dimensions, and represents the network parameters. This ensures an easily computable Jacobian determinant, which is calculated as the product of the diagonal terms in the Jacobian matrix of h. The diagonal terms are specific to the transformation h used. In one embodiment, an autoregressive structure is used in the normalizing flow, and an independent network is used to infer the warping for each dimension. Having an independent network for each dimension can simplify the implementation, and importantly, can facilitate network sharing when using multi-channel control variables for prediction.

[0070] Many integration problems run simultaneously on multiple potentially related channels (e.g., the RGB (red, green, blue) channels). To minimize the variance for each channel, it is advantageous to use a separate control variable for each channel rather than sharing one control variable across all channels. A straightforward solution is to instantiate different normalizing flows for each channel. Unfortunately, per-channel flows lead to a linear increase in the computational cost as the number of channels increases. Instead, the cost can be kept largely constant by sharing the corresponding neural networks across channels. However, since network sharing introduces cross-channel correlations, e.g., the red dimension can affect the green dimension, special care must be taken to correctly constrain the model.

[0071] Connecting only the inputs of the k-th network across per-channel flows and detecting the neural network to generate the parameters for warping the dimensions in all n channels is problematic because it corresponds to predicting a single normalized (n×D)-dimensional function. Instead, separately normalized D-dimensional functions can be implemented, e.g., in the case of instantiating different flows for each channel to ensure per-channel separate normalization of the control variables.

[0072] Note that since the channels can only affect each other after the first sub-flow, even when sharing the network across channels, the first sub-flow will separately produce a normalization function. This can be easily verified by examining the nD×nD Jacobian matrix constructed for all dimensions in all channels. The matrix will have a block-diagonal structure, where each d×d block corresponds to the Jacobian matrix of one of the channels. All entries outside the diagonal blocks will be zero. This observation allows for network sharing as long as only one sub-flow is used to model each channel of the control variable shape. The multi-channel normalizing flow represents the per-channel normalized form of the spectral decomposition of the control variable g

[0073] Return to Figure 2A , wherein in step 215, the first function g(x,y) is calculated as the first parameter integral G(y) and the second function In one embodiment, the first function is calculated by the product 130. In one embodiment, the first parameter integral is scaled by a parameter scaling factor α(y).

[0074] At step 220, the product is used to estimate the differential integral ∫f(x,y)-g(x,y)dx. The differential integral or the integral of the difference between f(x) and the control variable g(x) is the error. For example, as previously described in conjunction with Figure 1B As described above, error 155 is added to G (e.g., neural light field 160) to produce F (e.g., reference light field 150). In one embodiment, a heuristic is used to select a subset of inputs whose differential integrals are set to zero. In other words, for the subset of inputs, the differential integral is not calculated. In one embodiment, the heuristic compares a selected threshold to at least one of an exact or approximate area of ​​a path differential of a light transport path at a vertex associated with the input. In one embodiment, parameters or weights of the first neural network and at least one neural network are determined by minimizing the variance of an estimate of the differential integral using a Monte Carlo algorithm.

[0075] At step 225, the differential integral and the first parameter integral are combined to estimate a solution to the second parameter integral F(y)=G(y)+∫f(x,y)-g(x,y)dx. In one embodiment, the product and the first parameter integral are both scaled by a parameter scaling factor α(y). In one embodiment, the parameter scaling factor is calculated by a second neural network, and the weights of the second neural network are obtained by minimizing the variance of the estimated differential integral using a Monte Carlo algorithm. In one embodiment, the parameter scaling factor minimizes the variance of the estimated differential integral using a Monte Carlo algorithm.

[0076] In one embodiment, importance sampling of x proportional to the absolute difference |f(x,y)-g(x,y)| is used to estimate the differential integral. In one embodiment, a combination of one or more PDFs is selected to be used as the sampling distribution. The PDFs can be combined according to learned probabilities. For example, to probabilistically sample and neural importance sampling p NI To choose between them, we can train the parameter probability selection neural network c(x; θ C ) to approximate p NI In one embodiment, c(x; θ c) is optimized simultaneously with the control variables and the PDF model to achieve a balance between uniform and neural importance sampling at any point during training. The final PDF may be:

[0077] where θ p

[0078] : =θ c ∪θ NI

[0079] In one embodiment, importance sampling is performed using a normalized flow parameterized by at least one neural network, such as the normalized flow neural network model 110. In one embodiment, by minimizing the relative L2 error between G(y) and F(y), and f(z,y) / ∫f(x,y)dx, and minimizing the KL divergence between |f(x,y)-g(x,y)| / ∫|f(x,y)-g(x,y)|dx and the importance sampling distribution to determine the weights of the first neural network and the at least one neural network.

[0080] At step 230, the solution is used to approximate a linear operator. In one embodiment, the linear operator generates an image; the image represents a 3D scene rendered from the viewpoint. To generate the image, neural network system 100 receives vertices of segments of a light transport path traced from the viewpoint into the 3D scene, wherein at least one of the first neural network and the at least one neural network predicts values ​​at vertices of the light transport path.

[0081] In one embodiment, the second parameter integral F(y) represents a value of at least one of a light field, radiance, intensity, radiance, irradiance, flux, spectral radiance, field radiance, and volume radiance for a particular input (y). In one embodiment, the input y to the first neural network is hit information associated with a vertex of a light transport path.

[0082] In one embodiment, the first neural network and the at least one neural network are trained using a light transport path. In one embodiment, the first neural network is trained using at least a first value of an input, the input comprising the first value and at least a second value of the input. In one embodiment, this can be accomplished using a Q-learning reinforcement technique such as described in K. Dahm and A. Keller, 2017. Learning Light Transport the Reinforced Way. CoRR abs / 1701.07403 (2017). http: / / arxiv.org / abs / 1701.07403.

[0083] The technique of predicting neural control variables to estimate linear operators (such as integrals) can be applied to Monte Carlo and quasi-Monte Carlo integration. The technique of calculating control variables can also be used in Bayesian statistics, nuclear physics (Feynman integrals and particle transport), reinforcement learning, light transport simulation, medical simulation, 5G / 6G communication technology, solving any high-dimensional integral. This technique may be used with neural importance sampling.

[0084] When applied to light transport simulations, the neural control variant is able to match the state-of-the-art performance of other unbiased methods while providing a means to develop higher performing, practical solutions. Physically based image synthesis relies on estimating scattered radiance

[0085]

[0086] It leaves the surface point x in the direction ω, where f s is the bidirectional scattering distribution function (BSDF), Li is the distance from the direction ω i The radiation arriving at x, while γ is the shortening angle.

[0087] Scattered radiation L s (x, ω) corresponds to the parameter integral F(y), where y ≡ (x, ω), which is called the query position. The domain of integration and the integral variable are the unit sphere and the incident direction, respectively, that is, and x≡ω i , refer to equations (1) and (2), respectively. The goal is to reduce the estimation variance by using parameter control variables. The integral component is used as an approximation of the scattered radiation, i.e. G(x,ω;θ G )≈L s (x,ω), and the shape component Approximately normalized integrand.

[0088] The learned light field approximation is of sufficient quality for higher order bounces, so in one embodiment, error correction can be omitted, resulting in a significant reduction in noise at the expense of an insignificant visible bias. Using the example of light transport, the error is a difference integral, and as previously described, the error is added to G (e.g., the predicted light field) to produce F (e.g., the true light field). The error is the integral of the difference between f(x) and the control variable g(x).

[0089] Neural control variables can be used for simulations of light transport governed by Fredholm integral equations of the second kind. Such integral equations are notoriously difficult to solve efficiently due to their recursive nature, often requiring high-dimensional samples in the form of multi-vertex transport paths obtained using, for example, path tracing. Control variables have two fascinating advantages over conventional techniques that focus only on placing samples. First, control variables reduce the number of constructed path vertices, since the difference integral typically carries less energy than the original integral. Therefore, the learned scattered radiation can be used as an approximation of the true scattered radiation, terminating the path earlier. Heuristics can be used to minimize the resulting deviations. Second, control variables simply support spectrally resolved path tracing by using a different g for each spectral band (e.g., color channel). As mentioned earlier, to avoid the computational overhead of using potentially many control variables, a normalized flow can be implemented that can represent multiple (ideally correlated) control variables at once. As a result, the spectral noise typical of importance sampling for only scalar distributions is largely suppressed.

[0090] Figure 2B 235 and scaling factor α 240 predicted by the control variable neural network system 100 and a graph of the selection probability 245 according to one embodiment. The selection probability can be determined by the probability selection neural network c(x; θ c )predict. Figure 2B The images shown in provide a visualization of the contribution of each component of the trainable control variable to the integral equation for light transport at the principal vertex of each pixel (the first non-incremental interaction). The control variable integral approximates the scattered light field L very well. s (x,ω). Where the integral or shape of the control variables is inaccurate, the learned α scaling factor weights down the contribution of the control variables to the unbiased estimator. Finally, the learned selection probabilities are mixed between bidirectional scattering directional function (BSDF) sampling and residual neural importance sampling (NIS) to minimize the variance. Figure 2B As shown, smooth surfaces (e.g., windows and mirrors) within the dashed boundary region of selection probability 245 tend to favor BSDF sampling, while rougher surfaces generally favor residual NIS.

[0091] Figure 3Ais a conceptual diagram showing the use of neural control variables for light transport simulation according to one embodiment. When applied to light transport simulation, the neural control variable algorithm learns an approximation G of the scattered light field and corrects the approximation error by estimating the difference between the original integral function f and the corresponding learned control variable g. The neural control variable algorithm couples g and G so that g is always accurately integrated to G. To further reduce noise, importance sampling can be applied to the absolute difference integral using the learned PDF p. A heuristic can be provided by using the approximate G to terminate the path early. Early termination of the path reduces the average path length and eliminates most of the remaining noise. In one embodiment, given the same amount of time, the mean absolute percentage error (MAPE) of the unbiased technique using neural importance sampling with neural control variables is reduced by 29%, while the biased algorithm (using the early path termination heuristic) reduces the MAPE by another 40%. This error reduction is roughly equivalent to a 10-fold increase in efficiency.

[0092] When used for light transport, in one embodiment, the first neural network learns the amount of light reflected from the first surface intersection to trace fewer paths. This technique works very well when only the first segment of the path needs to be traced. In other embodiments, the first neural network learns the amount of light reflected in subsequent bounces. For example, if the first vertex is a perfect mirror, the control variables for the next intersection along the path will be calculated. In one embodiment, importance sampling is used at the first segment corresponding to the vertex to determine the direction of subsequent segments of the light transport path.

[0093] In one embodiment, a first neural network is trained to learn the light field of a 3D scene using noisy light paths (e.g., not 2D images). The trained first neural network can then predict the value of the light field toward the camera based on hit information (e.g., 3D position, view direction, surface normal, etc.) for the primary ray (i.e., the first segment) or for subsequent rays.

[0094] The recursive estimation of the radiance is terminated when the path escapes the scene or hits a blackbody radiator that does not scatter light. Since the integral component of the control variables approximates the scattered light field well in many cases, sometimes the path may be truncated early, resulting in a biased radiance estimate. In particular, the neural scattered light field G can be terminated early on non-specular surfaces. However, terminating too early may produce images with errors manifested as low-frequency variations and blurring. The correction term can be ignored using the standard, that is, approximating L by the neural light field G. s .

[0095] In one embodiment, the criterion measures the random area expansion of the path vertices. Once the area expansion becomes large enough, the path can be terminated and the neural light field G can be used instead.

[0096] Figure 3B FIG. 2 shows a path vertex x and the corresponding induced area expansion according to one embodiment. The direction ω is sampled using p(Ω|x, ω) at the path vertex x, resulting in an area expansion around the next path vertex x' of

[0097]

[0098] where γ' is the angle of incidence at x'. The cumulative area spread at the nth path vertex is the convolution of the spreads induced at all previous vertices. An isotropic Gaussian spread is assumed with variance and parallel surfaces, the convolution can be approximated by summing the square roots of the area extensions at consecutive vertices and re-squaring them:

[0099]

[0100] The accumulated area expansion is compared to the pixel footprint projected onto the primary vertex x1. In one embodiment, a path terminates as G(x,ω) if the projected pixel footprint is 10,000 times smaller than the accumulated area expansion of the path - roughly corresponding to a 100-pixel-wide image-space filter. i ). Otherwise, the unbiased control variable is used to recursively evaluate the heuristic at the next path vertex. In other embodiments, different sizes or thresholds of the spatial area occupied by the projected pixels are used to control termination. Heuristic path termination shortens the average path length and eliminates a large amount of noise at the expense of a small amount of visible bias.

[0101] Figure 3C A flow chart of a method 320 for generating an image of a 3D scene is shown according to one embodiment. Each block of the method 320 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method can also be embodied as computer usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. In addition, by way of example, with respect to Figure 1C The control variable neural network system 100 of the present invention describes the method 320. However, the method can be additionally or alternatively performed by any system or any combination of systems, including but not limited to those described herein. In addition, it will be understood by a person skilled in the art that any system that performs the method 320 is within the scope and spirit of the embodiments of the present disclosure.

[0102] At step 325, the first neural network predicts an approximation of the reflected light field of the 3D scene based on the hit data of the vertices of the light transport path from the camera to the 3D scene. In one embodiment, the integral prediction G (e.g., Figure 1B In one embodiment, the approximation of the reflected light field is scaled by alpha (α), which is calculated by at least one of the first neural network and the second neural network.

[0103] At step 330, the error between the approximation of the reflected light field of the 3D scene and the correct reflected light field is estimated by tracing a secondary path from the position corresponding to the hit data to produce an error estimate. Figure 1B As shown, the error 155 is the integral of the difference between the integrand f(x) and the control variable g(x), where g(x) can be scaled by α. The approximated reflected light field is independent of the position (e.g., location and orientation) of the camera and can therefore be used to generate images of the 3D scene from different camera positions. In one embodiment, importance sampling proportional to the error estimate is used to select the direction in which to track the secondary path.

[0104] In one embodiment, importance sampling is performed using a normalized flow parameterized by at least one neural network. For example, Figure 1C The normalized flow neural network model 110 shown is used to perform the normalized flow. At step 335, the error estimate and the approximation are combined to generate an image of the 3D scene viewed from the camera.

[0105] The control variable neural network system 100 can be optimized in a reinforcement learning manner during the rendering process: the vertices of the traced path are used to optimize the neural network while driving the variance reduction. Therefore, the neural network drives the variance reduction of its own training data and the final image.

[0106] Similar to Monte Carlo and Quasi-Monte Carlo methods for high-dimensional integrals, neural networks are particularly useful in high-dimensional approximations. In contrast to conventional techniques, errors or occasional deviations from the ground truth solution may be observed. Using control variables, speckling, contrast reduction, or highlight blurring can be corrected. Specifically, an estimate of the difference between the correct solution and the correct solution is added to the approximation to recover an unbiased result where the error only appears as noise. When the control variables are designed appropriately, the residual integral The energy is usually greater than the original integral Neural importance sampling can perform importance sampling on any integrand, including residual integrals. The control variable neural network system 100 can use multiple sampling techniques to characterize the advantages of each method, as long as all trainable parameters are optimized together.

[0107] Neural control variables can be used to reduce variance in parameter Monte Carlo integrals and reduce errors in parameter quasi-Monte Carlo integrals. The control variable neural network system 100 has sufficient approximation capabilities and is highly efficient to evaluate. The efficiency results from using a first neural network to infer the integral of the control variable and using a normalized flow to model the shape of the control variable. The proportionality coefficient can be learned and used to reduce noise. Neural importance sampling can be used to estimate correction terms and further reduce integration errors. Parameter trainable control variables can be used in conjunction with a trainable importance sampler (e.g., neural importance sampling) to produce better results than using each technique alone. Unbiased application of neural control variables provides efficiency gains over traditional techniques, while biased algorithms greatly improve efficiency. Neural control variables allow data-driven, high-quality neural approximations to be combined with accurate physics-based integrators that can be used to correct errors as needed.

[0108] Parallel processing architecture

[0109] Figure 4 A parallel processing unit (PPU) 400 is shown according to one embodiment. The PPU 400 may be used to implement the control variable neural network system 100.

[0110] In one embodiment, PPU 400 is a multithreaded processor implemented on one or more integrated circuit devices. PPU 400 is a multithreaded processor implemented on one or more integrated circuit devices. PPU 400 is a latency-hiding architecture designed for processing many threads in parallel. A thread (i.e., an execution thread) is an instance of an instruction set configured to be executed by PPU 400. In one embodiment, PPU 400 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, PPU 400 can be used to perform general-purpose calculations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that the processor is described only for illustrative purposes, and any processor can be used to supplement and / or replace the processor.

[0111] One or more PPUs 400 can be configured to accelerate thousands of high-performance computing (HPC), data center, cloud computing, and machine learning applications. PPU 400 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, simulations, computational graphics (e.g., ray or path tracing), deep learning, high-precision speech, image and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analysis, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0112] like Figure 4 As shown, the PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a work distribution unit 425, a hub 430, a crossbar switch (Xbar) 470, one or more general processing clusters (GPCs) 450, and one or more memory partition units 480. The PPU 400 can be connected to a host processor or other PPUs 400 via one or more high-speed NVLink 410 interconnects. The PPU 400 can be connected to a host processor or other peripheral devices via an interconnect 402. The PPU 400 can also be connected to a local memory 404 including multiple storage devices. In one embodiment, the local memory may include multiple dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high bandwidth memory (HBM) subsystem in which multiple DRAM dies are stacked within each device.

[0113] The NVLink 410 interconnect enables the system to scale and include one or more PPUs 400 in conjunction with one or more CPUs, supporting cache coherency between the PPU 400 and the CPU, and CPU mastering. Data and / or commands can be sent by NVLink 410 through hubs 430430 to or from other units of the PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 5B NVLink 410 is described in more detail.

[0114] I / O unit 405 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 402. I / O unit 405 may communicate directly with the host processor via interconnect 402, or through one or more intermediate devices (such as a memory bridge). In one embodiment, I / O unit 405 may communicate with one or more other processors (e.g., one or more PPUs 400) via interconnect 402. In one embodiment, I / O unit 405 implements a peripheral component interconnect express (PCIe) interface for communicating over a PCIe bus, and interconnect 402 is a PCIe bus. In alternative embodiments, I / O unit 405 may implement other types of known interfaces for communicating with external devices.

[0115] The I / O unit 405 decodes packets received via the interconnect 402. In one embodiment, the packets represent commands configured to cause the PPU 400 to perform various operations. The I / O unit 405 sends the decoded commands to various other units of the PPU 400 as specified by the commands. For example, some commands may be sent to the front end unit 415. Other commands may be sent to the hub 430 or other units of the PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, the I / O unit 405 is configured to route communications between and among the various logical units of the PPU 400.

[0116] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. The workload may include many instructions and data to be processed by those instructions. A buffer is an area in memory that is accessible (e.g., read / write) by both the host processor and the PPU 400. For example, the I / O unit 405 may be configured to access a buffer in a system memory connected to the interconnect 402 via a memory request transmitted through the interconnect 402. In one embodiment, the host processor writes a command stream to the buffer and then sends a pointer to the start of the command stream to the PPU 400. The front end unit 415 receives a pointer to one or more command streams. The front end unit 415 manages one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 400.

[0117] The front end unit 415 is coupled to a scheduler unit 420, which configures the various GPCs 450 to process the tasks defined by one or more streams. The scheduler unit 420 is configured to track state information related to the various tasks managed by the scheduler unit 420. The state may indicate which GPC 450 the task is assigned to, whether the task is active or inactive, a priority associated with the task, etc. The scheduler unit 420 manages the execution of multiple tasks on one or more GPCs 450.

[0118] Scheduler unit 420 is coupled to work distribution unit 425, which is configured to dispatch tasks for execution on GPC 450. Work distribution unit 425 may keep track of a number of scheduled tasks received from scheduler unit 420. In one embodiment, work distribution unit 425 manages a pending task pool and an active task pool for each GPC 450. When a GPC 450 completes execution of a task, the task is evicted from the active task pool of GPC 450, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 450. If an active task on GPC 450 has become idle, such as while waiting for a data dependency to be resolved, then the active task may be evicted from GPC 450 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 450.

[0119] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 400. In one embodiment, multiple computing applications are executed simultaneously by the PPU 400, and the PPU 400 provides isolation, quality of service (QoS), and independent address spaces for multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 400. The driver kernel outputs the task to one or more streams being processed by the PPU 400. Each task can include one or more related thread groups, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can be executed in parallel. A cooperative thread can refer to multiple threads that include instructions to execute a task and can exchange data through a shared memory. Tasks can be assigned to one or more processing units within the GPC 450, and instructions are scheduled to be executed by at least one warp.

[0120] Work distribution unit 425 communicates with one or more GPCs 450 via XBar 470. XBar 470 is an interconnect network that couples many units of PPU 400 to other units of PPU 400. For example, XBar 470 may be configured to couple work distribution unit 425 to a particular GPC 450. Although not explicitly shown, one or more other units of PPU 400 may also be connected to XBar 470 via hub 430.

[0121] Tasks are managed by a scheduler unit 420 and assigned to GPCs 450 by a work distribution unit 425. GPCs 450 are configured to process tasks and generate results. Results may be consumed by other tasks within GPC 450, routed to other GPCs 450 via XBar 470, or stored in memory 404. Results may be written to memory 404 via a memory partition unit 480, which may implement a memory interface for reading data from or writing data to memory 404. Results may be transferred to another PPU 400 or CPU via NVLink 410. In one embodiment, PPU 400 includes a number U of memory partition units 480, which is equal to the number of separate and distinct memory devices of memory 404 coupled to PPU 400. Each GPC 450 may include a memory management unit to provide virtual address to physical address translation, memory protection, and memory request arbitration. In one embodiment, the memory management unit provides one or more translation lookaside buffers (TLBs) for performing translation of virtual addresses to physical addresses in memory 404.

[0122] In one embodiment, the memory partition unit 480 includes a raster operation (ROP) unit, a second level (L2) cache, and a memory interface coupled to the memory 404. The memory interface can implement a 32, 64, 128, 1024-bit data bus, etc. for high-speed data transmission. The PPU 400 can be connected to up to Y memory devices, such as a high bandwidth memory stack or a graphics double data rate version 5, a synchronous dynamic random access memory, or other types of persistent storage. In one embodiment, the memory interface implements an HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as the PPU 400, which can save a lot of power and area compared to traditional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory tube cores, and Y is equal to 4, and each HBM2 stack includes two 128-bit channels per tube core for a total of 8 channels and 1024 bits of data bus width.

[0123] In one embodiment, memory 404 supports single error correction double error detection (SECDED) error correction code (ECC) to protect data. ECC provides higher reliability for computing applications that are sensitive to data corruption. Reliability is particularly important in large cluster computing environments where PPU 400 processes very large data sets and / or long-running applications.

[0124] In one embodiment, the PPU 400 implements a multi-level memory hierarchy. In one embodiment, the memory partition unit 480 supports unified memory to provide a single unified virtual address space for the CPU and PPU 400 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of accesses by the PPU 400 to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 400 that accesses the page more frequently. In one embodiment, the NVLink 410 supports address translation services that allow the PPU 400 to directly access the CPU's page tables and provide full access to the CPU memory by the PPU 400.

[0125] In one embodiment, the copy engine transfers data between multiple PPUs 400 or between a PPU 400 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 480 can then service the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, fixed memory (e.g., non-pageable) is operated for multiple copy engines between multiple processors, which significantly reduces the available memory. Due to hardware paging faults, addresses can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0126] Data from memory 404 or other system memory may be retrieved by memory partition unit 480 and stored in L2 cache 460, which is located on-chip and shared between various GPCs 450. As shown, each memory partition unit 480 includes a portion of the L2 cache associated with the corresponding memory 404. Lower levels of cache may then be implemented in various units within GPC 450. For example, each processing unit within GPC 450 may implement a level 1 (L1) cache. An L1 cache is a dedicated memory dedicated to a particular processing unit. L2 cache 460 is coupled to memory interface 470 and XBar 470, and data may be retrieved from the L2 cache and stored in each L1 cache for processing.

[0127] In one embodiment, the processing unit within each GPC 450 implements a SIMD (single instruction, multiple data) architecture, in which each thread in a group of threads (e.g., a warp) is configured to process different data sets based on the same instruction set. All threads in the thread group execute the same instruction. In another embodiment, the processing unit implements a SIMT (single instruction, multiple thread) architecture, in which each thread in the thread group is configured to process different data sets based on the same instruction set, but in which individual threads in the thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, so that when threads within a warp diverge, concurrency between the warp and serial execution within the warp is achieved. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, so that equal concurrency is achieved between all threads within and between warps. When the execution state is maintained for each individual thread, threads executing the same instruction can be converged and executed in parallel to achieve maximum efficiency.

[0128] Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling the expression of richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. Conventional programming models provide a single simple structure for synchronizing cooperative threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than the thread block granularity and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0129] Cooperative Groups enables programmers to explicitly define thread groups at sub-block (e.g., as small as a single thread) and multi-block granularity and perform collective operations such as synchronization on threads in a cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative Groups primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks.

[0130] Each processing unit includes a large number (e.g., 128, etc.) of different processing cores (e.g., functional units), which can be fully pipelined, single-precision, double-precision, and / or mixed-precision, and include a floating-point arithmetic logic unit and an integer arithmetic logic unit. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the core includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0131] The tensor cores are configured to perform matrix operations. Specifically, the tensor cores are configured to perform deep learning matrix operations, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0132] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be 16-bit floating point or 32-bit floating point matrices. The tensor cores operate on 16-bit floating point input data as well as 32-bit floating point accumulations. The 16-bit floating point multiplication requires 64 operations, producing full-precision products, which are then accumulated using 32-bit floating point additions with other intermediate products of the 4×4×4 matrix multiplication. In practice, tensor cores are used to perform larger two-dimensional or higher dimensional matrix operations built from these smaller elements. APIs (such as the CUDA 9 C++ API) expose specialized matrix loads, matrix multiplications and accumulations, and matrix storage operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that the 16×16 dimensional matrix spans all 32 threads of the warp.

[0133] Each processing unit may also include M special function units (SFUs) that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFU may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFU may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texture pixels) from memory 404 and sample the texture map to generate sampled texture values ​​for use in a shader program executed by the processing unit. In one embodiment, the texture map is stored in a shared memory, which may include or include an L1 cache. The texture unit implements texture operations, such as filtering operations using a mip map (e.g., a texture map with different levels of detail). In one embodiment, each processing unit includes two texture units.

[0134] Each processing unit also includes N load-store units (LSUs) that implement load and store operations between the shared memory and the register file. Each processing unit includes an interconnect network that connects each core to the register file and the LSU to the register file (shared memory). In one embodiment, the interconnect network is a crossbar switch that can be configured to connect any core to any register in the register file and to connect the LSU to memory locations in the register file and the shared memory.

[0135] Shared memory is an array of on-chip memory that allows data storage and communication between processing units and between threads within a processing unit. In one embodiment, shared memory includes 128KB of storage capacity and is located in the path from each processing unit to memory partition unit 480. Shared memory can be used to cache reads and writes. One or more of shared memory, L1 cache, L2 cache, and memory 404 is a backing store.

[0136] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory accesses. This capacity can be used by programs as a cache that does not use the shared memory. For example, if the shared memory is configured to use half of the capacity, texture and load / store operations can use the remaining capacity. Integration in shared memory enables the shared memory to be used as a high-throughput pipeline for streaming data while providing high-bandwidth and low-latency access to frequently reused data.

[0137] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, the fixed-function graphics processing unit is bypassed, thereby creating a simpler programming model. In the general parallel computing configuration, the work distribution unit 425 assigns and distributes thread blocks directly to the processing units in the GPC 450. Threads execute the same program using a unique thread ID in the calculation to ensure that each thread generates a unique result. One or more processing units are used to execute programs and perform calculations, shared memory is used to communicate between threads, and LSUs are used to read and write global memory through shared memory and memory partition unit 480. When configured for general parallel computing, the processing unit can also write commands that the scheduler unit 420 can use to start new work on the processing unit.

[0138] Each PPU 400 may include and / or be configured to perform the functions of one or more processing cores and / or components thereof, for example, a ray tracing (RT) core, a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application specific integrator circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0139] The PPU 400 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 400 is included on a single semiconductor substrate. In another embodiment, the PPU 400 is included on a system on a chip (SoC) along with one or more other devices (such as an additional PPU 400, a memory 404, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0140] In one embodiment, PPU 400 may be included on a graphics card that includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on a motherboard of a desktop computer. In yet another embodiment, PPU 400 may be an integrated graphics processing unit (iGPU) or a parallel processor included in a chipset of a motherboard. In yet another embodiment, PPU 400 may be implemented in reconfigurable hardware. In yet another embodiment, portions of PPU 400 may be implemented in reconfigurable hardware.

[0141] Example computing system

[0142] Systems with multiple GPUs and CPUs are used in a variety of industries as developers expose and exploit more parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0143] Figure 5A According to one embodiment, the use Figure 4A conceptual diagram of a processing system 500 implemented by a PPU 400. An exemplary system 565 may be configured to implement Figure 2A The processing system 500 includes a CPU 530, a switch 510, and each of the plurality of PPUs 400 and a corresponding memory 404.

[0144] NVLink 410 provides a high-speed communication link between each PPU 400. Figure 5B 402 connections, but the number of connections connected to each PPU 400 and CPU 530 may vary. Switch 510 interfaces between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLink 410 may be located on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols that interface between various different connections and / or links.

[0145] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and CPU 530, and switch 510 interfaces between interconnect 402 and each PPU 400. PPU 400, memory 404, and interconnect 402 may be located on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 and CPU 530, and switch 510 interfaces between each PPU 400 using NVLink 410 to provide one or more high-speed communication links between PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between PPU 400 and CPU 530 through switch 510. In yet another embodiment (not shown), interconnect 402 provides one or more communication links directly between each PPU 400. One or more NVLink 410 high-speed communication links may be implemented as a physical NVLink interconnect or an on-chip or on-die interconnect using the same protocol as NVLink 410.

[0146] In the context of this specification, a single semiconductor platform may refer to a unique single semiconductor-based integrated circuit manufactured on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that simulates on-chip operations and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Optionally, the parallel processing module 525 may be implemented as a circuit board substrate, and each of the PPU 400 and / or memory 404 may be a packaged device. In one embodiment, the CPU 530, switch 510, and parallel processing module 525 are located on a single semiconductor platform.

[0147] In one embodiment, the signaling rate of each NVLink 410 is 20 to 25 Gbit / s, and each PPU 400 includes six NVLink 410 interfaces (e.g., Figure 5A As shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 Gbit / s in each direction, with six links providing 400 Gbit / s. When the CPU 530 also includes one or more NVLink 410 interfaces, the NVLink 410 can be used exclusively for example Figure 5A PPU to PPU communication as shown, or some combination of PPU to PPU and PPU to CPU.

[0148] In one embodiment, NVLink 410 allows direct load / store / atomic access from CPU 530 to memory 404 of each PPU 400. In one embodiment, NVLink 410 supports coherency operations, allowing data read from memory 404 to be stored in the cache hierarchy of CPU 530, reducing cache access latency of CPU 530. In one embodiment, NVLink 410 includes support for address translation services (ATS), allowing PPU 400 to directly access page tables within CPU 530. One or more NVLinks 410 may also be configured to operate in a low power mode.

[0149] Figure 5B An exemplary system 565 is shown in which various architectures and / or functions of various previous embodiments may be implemented. The exemplary system 565 may be configured to implement Figure 2A The method 200 shown in FIG.

[0150] As shown, a system 565 is provided, which includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 can directly or indirectly couple one or more of the following devices: main memory 540, network interface 535, one or more CPUs 530, one or more display devices 545, one or more input devices 560, switch 510 and parallel processing system 525. The communication bus 575 can be implemented using any suitable protocol and can represent one or more links or buses, such as an address bus, a data bus, a control bus or a combination thereof. The communication bus 575 can include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standard association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, a hypertransport (HyperTransport) and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, the CPU 530 can be directly connected to the main memory 540. In addition, one or more CPUs 530 can be directly connected to the parallel processing system 525. In the case where there is a direct or point-to-point connection between components, the communication bus 575 can include a PCIe link to perform the connection. In these examples, the PCI bus does not need to be included in the system 565.

[0151] although Figure 5C The various blocks of are shown as being connected via communication bus 575 with wires, but this is not intended to be limiting and is only for clarity. For example, in some embodiments, presentation components, such as one or more display devices 545, can be considered I / O components, such as one or more input devices 560 (e.g., if the display is a touch screen). As another example, one or more CPUs 530 and / or parallel processing system 525 may include memory (e.g., main memory 540 may represent a storage device in addition to parallel processing system 525, CPU 530, and / or other components). In other words, Figure 5C The computing devices described herein are illustrative only. No distinction is made between categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “gaming console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, all of which are described herein. Figure 5C within the range of computing devices.

[0152] The system 565 also includes a main memory 540. Control logic (software) and data are stored in the main memory 540, which can take the form of various computer-readable media. Computer-readable media can be any available media that can be accessed by the system 565. Computer-readable media can include volatile and non-volatile media and removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0153] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, and / or other data types. For example, main memory 540 may store computer readable instructions (e.g., representing one or more programs such as an operating system and / or one or more program elements), and computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information and that may be accessed by system 565. As used herein, computer storage media itself does not include signals.

[0154] Computer storage media may embody computer readable instructions, data structures, program modules and / or other data types in a modulated data signal (such as a carrier wave or other transport mechanism), and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a way that it encodes information. By way of example and not limitation, computer storage media may include wired media (such as a wired network or a direct wired connection), and wireless media (such as acoustic, RF, infrared and other wireless media). Combinations of any of the above should also be included within the scope of computer readable media.

[0155] The computer program enables the system 565 to perform various functions when executed. One or more CPUs 530 may be configured to execute at least some of the computer-readable instructions to control one or more components of the system 565 to perform one or more of the methods and / or processes described herein. One or more CPUs 530 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing multiple software threads simultaneously. One or more CPUs 530 may include any type of processor, and may include different types of processors (e.g., processors for mobile devices with fewer cores and processors for servers with more cores) depending on the type of system 565 implemented. For example, depending on the type of system 565, the processor may be an advanced RISC machine (ARM) processor implemented using reduced instruction set computing (RISC) or an x86 processor implemented using complex instruction set computing (CISC). In addition to one or more microprocessors or auxiliary coprocessors (e.g., math coprocessors), the system 565 may also include one or more CPUs 530.

[0156] In addition to or alternatively to one or more CPUs 530, the parallel processing module 525 may be configured to execute at least some of the computer-readable instructions to control one or more components of the system 565 to perform one or more of the methods and / or processes described herein. The system 565 may use the parallel processing module 525 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, the parallel processing module 525 may be used for general-purpose computing on a GPU (GPGPU). In an embodiment, one or more CPUs 530 and / or the parallel processing module 525 may perform any combination of methods, processes, and / or portions thereof, either discretely or in conjunction.

[0157] The system 565 also includes one or more input devices 560, a parallel processing system 525, and one or more display devices 545. The one or more display devices 545 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof) speakers and / or other presentation components. The one or more display devices 545 may receive data from other components (e.g., the parallel processing system 525, one or more CPUs 530, etc.) and output data (e.g., as images, video, sound, etc.).

[0158] The network interface 535 can enable the system 565 to be logically coupled to other devices, including input devices 560, one or more display devices 545, and / or other components, some of which can be built into (e.g., integrated into) the system 565. Illustrative input devices 560 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. The input device 560 can provide a natural user interface (NUI) to process air gestures, voice, or other physiological inputs generated by a user. In some cases, the input can be transmitted to an appropriate network element for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and adjacent to the screen, air gestures, head and eye tracking, and related touch recognition (described in more detail below) associated with the display of the system 565. The system 565 can include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations of these, for gesture detection and recognition. Additionally, system 565 may include an accelerometer or gyroscope capable of detecting motion (e.g., as part of an inertial measurement unit (IMU)). In some examples, system 565 may use the output of the accelerometer or gyroscope to render an immersive augmented reality or virtual reality.

[0159] In addition, the system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) for communication via the network interface 535. The system 565 can be included in a distributed network and / or cloud computing environment.

[0160] The network interface 535 may include one or more receivers, transmitters, and / or transceivers that enable the system 565 to communicate with other computing devices via an electronic communication network including wired and / or wireless communications. The network interface 535 may include components and functionality to enable communication over any of a number of different networks, such as a wireless network (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., communicating over Ethernet or InfiniBand), a low power wide area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0161] The system 565 may also include an auxiliary memory (not shown). The auxiliary memory 610 includes, for example, a hard disk drive and / or a removable storage drive, which represents a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, a universal serial bus (USB) flash memory. The removable storage drive reads and / or writes to the removable storage unit in a well-known manner. The system 565 may also include a hardwired power supply, a battery power supply, or a combination thereof (not shown). The power supply can provide power to the system 565 to enable the components of the system 565 to operate.

[0162] Each of the aforementioned modules and / or devices can even be placed on a single semiconductor platform to form system 565. Alternatively, the various modules can also be placed independently or in various combinations of semiconductor platforms according to the needs of the user. Although various embodiments have been described above, it should be understood that they are only exemplary and not restrictive. Therefore, the breadth and scope of the preferred embodiments should not be limited by any of the above exemplary embodiments, but should only be defined in accordance with the appended claims and their equivalents.

[0163] Example network environment

[0164] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be configured to communicate with the client device, server, or other device types. Figure 5A The processing system 500 and / or Figure 5B 565 - for example, each device may include similar components, features and / or functionality of processing system 500 and / or exemplary system 565.

[0165] The components of the network environment can communicate with each other via one or more networks (which can be wired, wireless, or both). The network can include multiple networks or networks of networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, and even access points (among other components) can provide wireless connections.

[0166] Compatible network environments may include one or more peer-to-peer network environments, in which case the server may not be included in the network environment, and one or more client-server network environments, in which case one or more servers may be included in the network environment. In a peer-to-peer network environment, the functionality described herein with respect to one or more servers may be implemented on any number of client devices.

[0167] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting software at the software layer and / or one or more applications at the application layer. The software or application may include network-based service software or applications, respectively. In an embodiment, one or more client devices may use network-based service software or applications (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open source software web application framework, such as a framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0168] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these various functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be distributed across states, regions, countries, the world, etc.). If the connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers can assign at least a portion of the functionality to one or more edge servers. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0169] One or more client devices may include Figure 5B The example processing system 500 and / or Figure 5CAt least some of the components, features, and functions of the exemplary system 565. By way of example and not limitation, the client device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smart phone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a boat, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, a device, a consumer electronic device, a workstation, an edge device, any combination of the depicted devices, or any other suitable device.

[0170] Machine Learning

[0171] Deep Neural Networks (DNNs) developed on processors such as the PPU 400 have been used in a variety of use cases: from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, constantly learning, constantly getting smarter, and delivering more accurate results faster over time. A child is initially taught by an adult to correctly identify and classify various shapes, and eventually is able to recognize shapes without any coaching. Similarly, deep learning or neural learning systems need to be trained in object recognition and classification in order to become smarter and more efficient in recognizing basic objects, occluded objects, etc., while also assigning context to objects.

[0172] At the simplest level, neurons in the human brain look at the various inputs they receive, assign importance levels to each of these inputs, and pass outputs to other neurons for processing. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs that represent various features of an object that the perceptron is being trained to recognize and classify, and each of these features is given a certain weight based on the importance of that feature when defining the shape of the object.

[0173] Deep neural network (DNN) models include multiple layers of many connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.), which can be trained with large amounts of input data to solve complex problems quickly and accurately. In one example, the first layer of a DNN model breaks down an input image of a car into its parts and looks for basic patterns (such as lines and angles). The second layer assembles lines to look for higher-level patterns, such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the last few layers generate labels for the input image, identifying the model of a specific car brand.

[0174] Once a DNN is trained, it can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on check deposits in ATM machines, identifying images of friends in photos, providing movie recommendations to over 50 million users, recognizing and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human speech in real time.

[0175] During training, data flows through the DNN in a forward propagation phase until a prediction is produced, which indicates a label corresponding to the input. If the neural network does not correctly label an input, the error between the correct label and the predicted label is analyzed, and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels that input and other inputs in the training data set. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplications and additions supported by the PPU 400. Inference is less computationally intensive than training and is a latency-sensitive process in which a trained neural network is applied to new inputs it has not seen before to classify images, detect emotions, identify suggestions, recognize and translate speech, and generally reason about new information.

[0176] Neural networks rely heavily on matrix math operations, and complex multi-layer networks require a lot of floating point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, the PPU 400 is a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0177] In addition, images generated using one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for recognizing objects and environments in the real world. Such images can include scenes of roads, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or real environments. Such images can be used to train, test, or certify DNNs used in machines or robots to manipulate, process, or modify physical objects in the real world. In addition, such images can be used to train, test, or certify DNNs used in autonomous vehicles to navigate and move vehicles in the real world. In addition, images generated using one or more of the techniques disclosed herein can be used to convey information to users of these machines, robots, and vehicles.

[0178] Figure 5CComponents of an exemplary system 555 that can be used to train and utilize machine learning in accordance with at least one embodiment are shown. As will be discussed, various components can be provided by various combinations of computing devices and resources or a single computing system that can be under the control of a single entity or multiple entities. In addition, aspects can be triggered, initiated, or requested by different entities. In at least one embodiment, the training of the neural network can be directed by a provider associated with the provider environment 506, while in at least one embodiment, the training of the neural network can be requested by a customer or other user who can access the provider environment through a client device 502 or other such resource. In at least one embodiment, the training data (or data to be analyzed by the trained neural network) can be provided by the provider, the user, or the third-party content provider 524. In at least one embodiment, the client device 502 can be a vehicle or object to be navigated on behalf of a user, for example, which can submit requests and / or receive instructions to facilitate navigation of the device.

[0179] In at least one embodiment, the request can be submitted via at least one network 504 to be received by the provider environment 506. In at least one embodiment, the client device can be any suitable electronic and / or computing device that enables a user to use to generate and send such a request, such as, but not limited to, a desktop computer, a laptop computer, a computer server, a smart phone, a tablet computer, a game console (portable or otherwise), a computer processor, computing logic, and a set-top box. The one or more networks 504 may include any suitable network for transmitting requests or other such data, such as the Internet, an intranet, an Ethernet network, a cellular network, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), an ad hoc network of direct wireless connections between peers, and the like.

[0180] In at least one embodiment, in this example, the request may be received at the interface layer 508, which may forward the data to the training and reasoning manager 532, in this example. The training and reasoning manager 532 may be a system or service including hardware and software for managing requests and serving corresponding data or content, and in at least one embodiment, the training and reasoning manager 532 may receive a request to train a neural network and may provide data to request to the training module 512. In at least one embodiment, the training module 512 may select an appropriate model or neural network to be used if the request is not specified, and may use the relevant training data to train the model. In at least one embodiment, the training data may be a batch of data stored in the training data repository 514, received from the client device 502, or obtained from the third-party provider 524. In at least one embodiment, the training module 512 may be responsible for the training data. The neural network may be any appropriate network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN). Once the neural network is trained and successfully evaluated, the trained neural network may be stored in, for example, a model repository 516, which may store different models or networks for users, applications, or services, etc. Multiple models for a single application or entity can be utilized based on a number of different factors.

[0181] In at least one embodiment, at a later point in time, a request for content (e.g., path determination) or data determined or affected at least in part by a trained neural network may be received from the client device 502 (or another such device). The request may include, for example, input data to be processed using the neural network to obtain one or more inferences or other output values, classifications, or predictions, or for at least one embodiment, the input data may be received by the interface layer 508 and directed to the reasoning module 518, although a different system or service may also be used. In at least one embodiment, if not already stored locally in the reasoning module 518, the reasoning module 518 may obtain a suitable training network from the model repository 516, such as a trained deep neural network (DNN) discussed herein. The reasoning module 518 may provide data as input to the training network, and may then generate one or more inferences as output. For example, this may include a classification of the input data instance. In at least one embodiment, the inference may then be sent to the client device 502 for display or other communication to the user. In at least one embodiment, context data for the user may also be stored in a user context data repository 522, which may include data about the user, which may be used as input to the network when generating inferences or determining data returned to the network after acquiring an instance. In at least one embodiment, related data that may include at least some of the input or inference data may also be stored in a local database 534 for processing future requests. In at least one embodiment, the user may use account information or other information to access resources or functions of the provider environment. In at least one embodiment, if allowed and available, user data may also be collected and used for further training models to provide more accurate inferences for future requests. In at least one embodiment, a request for a machine learning application 526 executed on a client device 502 may be received through a user interface, and the results may be displayed through the same interface. The client device may include resources such as a processor 528 and a memory 562 for generating requests and processing results or responses, and at least one data storage element 552 for storing data for the machine learning application 526.

[0182] In at least one embodiment, the processor 528 (or the processor of the training module 512 or the inference module 518) will be a central processing unit (CPU). However, as described above, resources in such environments may utilize GPUs to process data for at least some types of requests. GPUs with thousands of cores (such as PPU 400) are designed to handle massively parallel workloads and, as a result, have become popular in deep learning for training neural networks and generating predictions. Although using GPUs for offline building can train larger and more complex models faster, generating predictions offline means that request-time input features cannot be used, or predictions must be generated for all feature permutations and stored in a lookup table to use real-time requests. If the deep learning framework supports CPU mode and the model is small and simple enough to be executed feed-forward on the CPU with reasonable latency, then a service on a CPU instance can host the model. In this case, training can be performed offline on the GPU, while inference can be performed in real time on the CPU. If the CPU approach is not feasible, the service can run on a GPU instance. However, because GPUs have different performance and cost characteristics than CPUs, running a service that offloads runtime algorithms to a GPU may require that its design be different from that of a CPU-based service.

[0183] In at least one embodiment, video data may be provided from a client device 502 for enhancement in a provider environment 506. In at least one embodiment, video data may be processed for enhancement on the client device 502. In at least one embodiment, data may be streamed from a third party content provider 524 and enhanced by the third party content provider 524, the provider environment 506, or the client device 502. In at least one embodiment, video data may be provided from a client device 502 for use in a provider environment 506 as training data.

[0184] In at least one embodiment, supervised and / or unsupervised training can be performed by client device 502 and / or provider environment 506. In at least one embodiment, a set of training data 514 (e.g., classified or labeled data) is provided as input to be used as training data. In one embodiment, the set of training data can be used to train one or more neural networks within control variable neural network system 100.

[0185] In at least one embodiment, the training data may include instances of at least one type of object for which the neural network is to be trained, and information identifying the type of object. In at least one embodiment, the training data may include a set of images, each image including a representation of a type of object, wherein each image also includes a label, metadata, classification, or other information identifying the object or other information identifying the type of object represented in the corresponding image. Various other types of data may also be used as training data, and may include text data, audio data, video data, etc. In at least one embodiment, the training data 514 is provided as training input to the training module 512. In at least one embodiment, the training module 512 may be a system or service including hardware and software, such as one or more computing devices executing a training application, for training a neural network (or other model or algorithm, etc.). In at least one embodiment, the training module 512 receives an instruction or request indicating the type of model to be used for training. In at least one embodiment, the model may be any suitable statistical model, network, or algorithm useful for such a purpose, and may include an artificial neural network, a deep learning algorithm, a learning classifier, a Bayesian network, etc. In at least one embodiment, the training module 512 can select an initial model or other untrained model from an appropriate repository 516 and train the model using the training data 514, thereby generating a model that can be trained (e.g., a trained deep neural network) that can be used to classify similar types of data, or generate other such inferences. In at least one embodiment that does not use training data, according to the training module 512, an appropriate initial model can still be selected to train the input data.

[0186] In at least one embodiment, the model can be trained in a variety of different ways, which can depend in part on the type of model selected. In at least one embodiment, a set of training data can be provided to a machine learning algorithm, where the model is a model artifact created by the training process. In at least one embodiment, each instance of the training data contains a correct answer (e.g., a classification), which can be referred to as a target or target attribute. In at least one embodiment, a learning algorithm finds patterns in the training data that map input data attributes to targets, the answers to be predicted, and outputs a machine learning model that captures these patterns. In at least one embodiment, the machine learning model can then be used to obtain predictions for new data for which a target is not specified.

[0187] In at least one embodiment, the training and inference manager 532 can select from a set of machine learning models including binary classification, multi-classification, generative, and regression models. In at least one embodiment, the type of model to be used can depend at least in part on the type of target to be predicted.

[0188] The image generated by applying one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device can be directly coupled to a system or processor that generates or renders an image. In other embodiments, the display device can be indirectly coupled to the system or processor, for example, via a network. Examples of such networks include the Internet, mobile telecommunications networks, WIFI networks, and any other wired and / or wireless networking systems. When the display device is indirectly coupled, the image generated by the system or processor can be streamed to the display device via the network. This streaming allows, for example, a video game or other application that renders an image to be executed in a server, data center, or cloud-based computing environment, and the rendered image will be transmitted and displayed on one or more user devices (e.g., computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separated from the server or data center. Therefore, the technology disclosed herein can be applied to enhance streaming images and enhance streaming image services, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0189] Note that the techniques described herein may be embodied in executable instructions stored in a computer-readable medium for use or in conjunction with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will appreciate that for some embodiments, various types of computer-readable media may be included to store data. As used herein, "computer-readable media" includes one or more of any suitable media for storing executable instructions of a computer program, so that an instruction execution machine, system, apparatus, or device can read (or obtain) instructions from the computer-readable medium and execute instructions for performing the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer floppy disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; optical storage devices include portable compact disks (CDs), portable digital video disks (DVDs), etc.

[0190] It should be understood that the arrangement of components shown in the drawings is for illustrative purposes, and other arrangements are possible. For example, one or more elements described herein may be implemented in whole or in part as electronic hardware components. Other elements may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other elements may be combined, some other elements may be omitted entirely, and additional components may be added while still achieving the functions described herein. Therefore, the subject matter described herein may be embodied in many different variations, and all such variations are considered to be within the scope of the claims.

[0191] In order to promote the understanding of the subject matter described herein, many aspects are described according to the action sequence. It will be appreciated by those skilled in the art that various actions can be performed by a dedicated circuit or circuit, by a program instruction executed by one or more processors, or by a combination of the two. The description of any action sequence herein is not intended to imply that the specific order described for executing the sequence must be followed. Unless otherwise noted herein or clearly contradictory to the context, all methods described herein can be performed in any suitable order.

[0192] The use of the terms "a", "an" and "the" and similar references in the context of describing the subject matter (particularly in the context of the appended claims) should be interpreted as covering both the singular and the plural. Unless otherwise indicated herein or clearly contradicted by the context, it is plural. The term "at least one" followed by a list of one or more items (e.g., "at least one of A and B") should be understood to refer to one item selected from the listed items (A or B) or any combination of two or more listed items (A and B), unless otherwise indicated herein or clearly contradicted by the context. In addition, the foregoing description is for illustrative purposes only and not for limiting purposes, as the scope of protection sought is defined by the claims described below and any equivalents thereof. Unless otherwise required, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the subject matter and does not limit the scope of the subject matter. In the claims and written description, the use of the term "based on" and other similar phrases to indicate the conditions for producing a result is not intended to exclude any other conditions for producing the result. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the claimed invention.

Claims

1. A computer-implemented method comprising: predicting a first parameter integral of a first function based on an input using a first neural network; approximating the shape of the first function using a second function that integrates to unity, wherein parameters of the second function are computed by at least one neural network; calculating the first function as the product of the first parameter integral and the second function; estimating a differential integral using the product; combining the difference integral and the first parameter integral to estimate a solution to a second parameter integral; approximating a linear operator using the solution; as well as An image of a three-dimensional scene is synthesized using the linear operator.

2. The computer-implemented method of claim 1, wherein the second function is a normalized flow.

3. The computer-implemented method of claim 1 , wherein combining the product and the first parameter integral comprises: Both the product and the first parametric integral are scaled by a parametric scaling factor.

4. The computer-implemented method of claim 3, wherein the parameter scaling coefficients are calculated by a second neural network and the weights of the second neural network are obtained by minimizing the variance of the estimated difference integral using a Monte Carlo algorithm.

5. The computer-implemented method of claim 3, wherein the parameter scaling factor minimizes an estimate of a variance of the differential integral using a Monte Carlo algorithm.

6. The computer-implemented method of claim 1, wherein the second parameter integral represents a value of at least one of a light field, radiance, intensity, radiance, irradiance, flux, spectral radiance, field radiance, and volume radiance for a particular input y.

7. The computer-implemented method of claim 6, wherein the input y of the first neural network is hit information associated with vertices of a light transport path.

8. The computer-implemented method of claim 7, wherein: The linear operator generates an image; as well as The image represents a three-dimensional scene rendered from a viewpoint, the method further comprising: receiving vertices of segments of a light transport path from a viewpoint to the three-dimensional scene, Wherein at least one of the first neural network and the at least one neural network predicts values ​​at vertices of the light transport path, wherein the values ​​are used to generate the image.

9. The computer-implemented method of claim 7, further comprising: A heuristic is used to select a subset of inputs for which the differential integral is set to zero.

10. The computer-implemented method of claim 9, wherein the heuristic compares the selected threshold to at least one of an exact or approximate area of ​​a path differential of the light transport path at the vertex associated with the input.

11. The computer-implemented method of claim 7, further comprising: Importance sampling is used at a first segment corresponding to the vertex to determine a direction of a subsequent segment of the light transport path.

12. The computer-implemented method of claim 6, wherein the first neural network and the at least one neural network are trained using an optical transmission path.

13. The computer-implemented method of claim 1, wherein the first neural network is trained using at least a first value of the input, the input comprising the first value and at least one second value of the input.

14. The computer-implemented method of claim 1 , further comprising: The error is estimated using importance sampling of the input of the first function that is proportional to the absolute difference between the initial function and the first function.

15. The computer-implemented method of claim 14, wherein the importance sampling is performed using a normalization flow parameterized by the at least one neural network.

16. The computer-implemented method of claim 14, wherein the step of: and f(x,y) / ∫f(x,y)dx, and minimizing the KL divergence between |f(x,y)-g(x,y)| / ∫|f(x,y)-g(x,y)|dx and the importance sampling distribution to determine the weights of the first neural network and the at least one neural network, wherein f(x,y) represents the initial function, g(x,y) represents the first function, represents the second function, x and y in f(x,y) represent the input of the initial function, and x and y in g(x,y) represent the input of the first function.

17. The computer-implemented method of claim 1, wherein weights of the first neural network and the at least one neural network are determined by minimizing a variance of the estimate of the differential integral using a Monte Carlo algorithm.

18. The computer-implemented method of claim 1, wherein the steps of predicting, approximating, calculating, estimating, combining, and approximating are performed in a cloud computing environment.

19. The computer-implemented method of claim 1, wherein the steps of predicting, approximating, calculating, estimating, combining, and approximating are performed on a server or in a data center to generate an image, and the image is streamed to a user device.

20. The computer-implemented method of claim 1, wherein the steps of predicting, approximating, calculating, estimating, combining, and approximating are performed to generate images for use in training, testing, or proving a neural network employed in a machine, robot, or autonomous vehicle.

21. The computer-implemented method of claim 1, wherein the steps of predicting, approximating, calculating, estimating, combining, and approximating are performed on a virtual machine that includes a portion of a graphics processing unit.

22. A system comprising: The processor is configured as: implementing a first neural network configured to predict a first parameter integral of a first function based on an input; approximating the shape of the first function using a second function that integrates to unity, wherein parameters of the second function are computed by at least one neural network; calculating the first function as the product of the first parameter integral and the second function; estimating a differential integral using the product; combining the difference integral and the first parameter integral to estimate a solution to a second parameter integral; approximating a linear operator using the solution; as well as An image of a three-dimensional scene is synthesized using the linear operator.

23. The system of claim 22, wherein the second function is a normalized flow.

24. The system of claim 22, wherein the product and the first parameter integral are combined by scaling both the product and the first parameter integral by a parameter scaling factor.

25. The system of claim 22, wherein the second parameter integral represents a value of at least one of a light field, radiance, intensity, radiance, irradiance, flux, spectral radiance, field radiance, and volume radiance for a particular input.

26. The system of claim 22, wherein weights of the first neural network and the at least one neural network are determined by minimizing a variance of the estimate of the differential integral using a Monte Carlo algorithm.

27. A non-transitory computer readable medium storing computer instructions which, when executed by one or more processors, cause the one or more processors to perform the following steps: predicting a first parameter integral of a first function based on an input using a first neural network; approximating the shape of the first function using a second function that integrates to unity, wherein parameters of the second function are computed by at least one neural network; calculating the first function as the product of the first parameter integral and the second function; estimating a differential integral using the product; combining the difference integral and the first parameter integral to estimate a solution to a second parameter integral; approximating a linear operator using the solution; as well as An image of a three-dimensional scene is synthesized using the linear operator.

28. The non-transitory computer readable medium of claim 27, wherein the second function is a normalized flow.

29. The non-transitory computer readable medium of claim 27, wherein combining the product and the first parameter integral comprises: Both the product and the first parametric integral are scaled by a parametric scaling factor.

30. The non-transitory computer-readable medium of claim 27, wherein the second parameter integral represents a value of at least one of a light field, radiance, intensity, radiance, irradiance, flux, spectral radiance, field radiance, and volume radiance for a particular input y.

31. The non-transitory computer readable medium of claim 27, wherein weights of the first neural network and the at least one neural network are determined by minimizing a variance of the estimate of the differential integral using a Monte Carlo algorithm.

32. A computer-implemented method comprising: Predicting, by a first neural network, an approximation of a reflected light field of the three-dimensional scene based on hit data of vertices of a light transport path from a camera to the three-dimensional scene; estimating an error between an approximation of a reflected light field and a correct reflected light field for the three-dimensional scene by tracing secondary paths from locations corresponding to the hit data to produce an error estimate; combining the error estimate and the approximation to produce the correct reflected light field for the three-dimensional scene; and An image of the three-dimensional scene viewed from the camera is synthesized using the correctly reflected light field.

33. The computer-implemented method of claim 32, further comprising: The approximation of the reflected light field is scaled by α calculated by at least one of the first neural network and the second neural network.

34. The computer-implemented method of claim 32, wherein importance sampling proportional to the error estimate is used to select a direction to trace the secondary path.

35. The computer-implemented method of claim 34, wherein the importance sampling is performed using a normalization flow parameterized by at least one neural network.

Images