Light Importance Caching Using Spatial Hashing in Real-Time Ray Tracing Applications

Abstract

To make it possible to determine light contribution information and to cache the information for use in rendering image frames for a scene.SOLUTION: In at least one embodiment, a spatial hash data structure can be used to split a scene into regions such as octahedral voxels. Using cast light rays, an average light contribution can be computed for each individual voxel. Those light values can then be used to build a cumulative distribution function for each voxel. This function can be used to select which lights to sample for a given frame during rendering. The sampling for a region or voxel can be based at least in part upon the number of contributing lights for that region, as well as the relative contributions of those lights. Such an approach can be very bandwidth and cache efficient, while providing high image quality.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] As the quality of display devices continues to improve, along with user expectations, the quality of the displayed content must continually improve. [Background technology]

[0002] This may include tasks such as realistically lighting objects in the scene being rendered. In complex scenes with many light sources, including reflective or refractive surfaces, resource capacity may be insufficient to accurately represent all the lighting in the scene, especially for applications that require high frame rates.

[0003] Various embodiments according to the present disclosure will now be described with reference to the drawings. [Brief description of the drawings]

[0004] [Figure 1A] FIG. 1 illustrates ray tracing of an object in a scene, according to at least one embodiment. [Figure 1B] FIG. 1 illustrates ray tracing of an object in a scene, according to at least one embodiment. [Figure 1C] FIG. 1 illustrates ray tracing of an object in a scene, according to at least one embodiment. [Diagram 2] FIG. 13 illustrates ray tracing of both view and shadow rays, according to at least one embodiment. [Figure 3A] FIG. 13 illustrates ray tracing using octahedral voxels, according to at least one embodiment. [Figure 3B] FIG. 13 illustrates ray tracing using octahedral voxels, according to at least one embodiment. [Figure 4] FIG. 2 illustrates an example process for rendering a frame of a scene, according to at least one embodiment. [Diagram 5]FIG. 1 illustrates a process for performing ray tracing during rendering, according to at least one embodiment. [Figure 6] FIG. 1 illustrates an example image generation system including light importance caching, in accordance with at least one embodiment. [Figure 7] FIG. 1 illustrates an exemplary data center system in accordance with at least one embodiment. [Figure 8] FIG. 1 illustrates a computer system according to at least one embodiment. [Figure 9] FIG. 1 illustrates a computer system according to at least one embodiment. [Figure 10] A diagram illustrating at least a portion of a graphics processor according to at least one embodiment and according to one or more embodiments. [Figure 11] A diagram illustrating at least a portion of a graphics processor according to at least one embodiment and according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Approaches according to various embodiments can overcome various deficiencies in existing approaches to image generation. In particular, various embodiments can provide for determining and caching importance information for one or more light sources in a scene. Importance information can be determined for individual regions of a scene that may be associated with individual voxels and can include directional data. This importance information can be used to determine the amount of sampling to perform for these different regions based at least in part on factors such as the number of important lights and the determined importance of those lights. Such approaches can provide significantly improved bandwidth and cache efficiency while providing high image quality compared to approaches that sample all of the many light sources or approximate the sampling of all of the light sources.

[0006] Various real-time and offline rendering systems may use a virtual camera view of a three-dimensional scene to generate a sequence of images or video frames. The scene may include various types of static or dynamic objects and one or more light sources for illuminating these objects. The objects may include any type of foreground or background object that may be visually represented in an image or other digital representation. The light source may also be any suitable source of light or illumination that may affect one or more of these scene objects, the light having different colors or intensities. At least some of these scene objects may also be reflective or refractive, thereby affecting the illumination from one or more of these light sources or may act as separate light sources in some circumstances.

[0007] FIG. 1A illustrates an exemplary view 100 that may be generated for such a scene, according to various embodiments. In this example, the view 100 is a virtual camera's view of a virtual three-dimensional (3D) environment, resulting in a two-dimensional (2D) representation of that environment from a determined viewpoint. In this particular view or image frame, there are three objects 108, 110, 112, referred to herein as scene objects. These scene objects may include any objects that may be represented in a scene, which may include background or foreground objects, player avatars in a game, dynamic or static objects, etc. These objects may have different lighting characteristics and may include at least a portion of an opaque, transparent, reflective, transmissive, or refractive object, among other such options. The objects may also have different surface characteristics or textures that may affect their lighting characteristics. There are also three light sources 102, 104, 106 in the scene. These light sources may be any object or element capable of generating some type of lighting, which may include light sources such as the sun, lamps, neon signs, etc. In order to render this view of the scene in a realistic manner, the light from the various light sources 102, 104, 106 should illuminate the scene objects 108, 110, 112 in a manner similar to how those scene objects would be illuminated in their real-world environment, which may include not only direct lighting but also aspects such as reflections and shadow creation.

[0008] One technique for making such lighting determinations involves ray tracing, a rendering technique that uses algorithms to trace the path of light from a light source and then simulates how the light interacts in a computer-generated environment with scene objects that the light ray "hits" or intersects. Ray tracing can provide realistic shadows and reflections, as well as accurate translucency and scattering. Ray tracing also has the advantage that it can be performed on a graphics processing unit (GPU), such as in NVIDIA's GeForce RTX graphics cards, to perform lighting determinations fast enough to provide accurate rendering in real time for high frame rate applications.

[0009] A view 130 of this scene, shown in FIG. 1B, illustrates exemplary light rays from two light sources 102, 106. Although there may be at least one ray cast from each potential light source in the scene at a potential pixel location, such an approach may be very resource intensive. Thus, ray tracing approaches often involve selecting or "sampling" some of these rays per frame to determine updated lighting information that may be approximated based on these samples, and at least possibly based on previous lighting information from previous frames. FIG. 1B illustrates a selection of exemplary rays from light sources 102 and 106. As illustrated, the light rays from each of these light sources strike points along different objects or portions of those objects based at least in part on the relative positions of those light sources and objects in the scene.

[0010] Techniques according to various embodiments can utilize this information to enable more optimal sampling of light rays for a frame or scene. For example, light rays from light source 102 hit all three objects in the scene, but not to the right of scene object 112. Furthermore, scene object 112 blocks light rays from light source 102 from hitting a large portion of the frame to the right of scene object 112. Light rays from light source 102 hit primarily the top and right of scene object 108 (for convenience, reference numbers are carried over between figures), potentially all of scene object 110, and the top and left of scene object 112. Region 132 of the scene can then be determined, which includes object surfaces that may be hit by light rays from light source 102, and thus light source 102 is a relatively more important (higher contributing) light source. Similarly, light rays from light source 106 strike the right side of scene object 112, but scene object 112 blocks those rays from striking scene objects 108 and 110. Thus, a region 134 of the scene may be defined in which light source 106 has a relatively high importance. As can be seen, light source 102 may not contribute much to the illumination in region 134 because light rays from that source are mostly blocked by objects in region 134. Similarly, light source 106 may not contribute much to region 132 because light rays from that source are likewise mostly blocked by objects in region 132. Techniques according to various embodiments may utilize such importance information in determining which rays to sample for a frame or portion of a frame, since it may be more beneficial to sample only those light rays of light sources that provide a greater contribution to a given portion or region.

[0011] In at least one embodiment, rays may be sampled only for light sources that are determined to be at least somewhat important to a region. This may include up to a maximum number of important light sources in some embodiments. In other embodiments, one or more rays may be sampled per frame for light sources that have little or no determined contribution to the region to account for scene changes. For example, object motion may cause a ray to pass through the region or a ray to be reflected into the region. Furthermore, because the contribution determination is determined using sampling, it is not necessary to analyze each potential ray, and as a result, the contribution calculation may be determined using incomplete data and a light source may actually have undiscovered importance to a region. Such an approach allows the influence of lighting to be determined without allocating significant resources to light sources that are generally less important to a region or portion of a scene.

[0012] 1C shows sample rays cast from light source 104. In this example, the rays from light source 104 directly strike portions of scene objects 108 and 112. As shown, rays 162 reflected from scene object 108 also strike scene object 110. In such a situation, light source 104 may contribute to each region of the scene, and thus rays may be sampled for all regions. However, in at least some embodiments, the contributions determined for different regions may vary, so the amount or frequency of sampling for the rays of light source 104 may vary from region to region.

[0013] In at least one embodiment, there may also be different types of rays sampled for a given scene. As shown in view 200 of FIG. 2, the scene has a single light source 204 and a single scene object 208. A two-dimensional image 206 of the three-dimensional scene is rendered from the viewpoint of a virtual camera 202, which may, in at least some embodiments, be moved or oriented around the scene in different frames. The image will correspond to a plane that intersects with the virtual camera view at a determined location. To render image 206, the renderer must determine which portions of the scene object 208 are visible from the camera view in order to determine the content to be represented in the image. Additionally, the renderer must determine the illumination of each such portion in order to render the object realistically in relation to the light source. As shown, this may include determining at least two types of rays. The first type of rays, referred to herein as view rays, correspond to points or regions on the scene object 208 that are visible in the virtual view and that are illuminated (directly or indirectly) by the light source 204. A second type of ray, referred to herein as a shadow ray, corresponds to a point or region in a scene, such as on the ground or floor in an exemplary environment, where an object, such as scene object 208, casts a shadow based at least in part on light cast from light source 204. In this case, a shadow ray 212 from light source 204 to scene object 208 is blocked by object 208 and therefore does not hit this shadow region. However, by extending the ray, it can be determined that a shadow may be cast at that location by scene object 208 relative to light source 204. Thus, the ray may be determined not only for direct illumination, but also for shadows, reflections, refractions, diffusion, and other such aspects of illumination. These different types of rays may be sampled at similar or different sampling rates or frequencies, and in some embodiments may be sampled at least in part based on their relative importance to the scene or region.

[0014] A virtual three-dimensional environment can be composed of multiple individual voxels in a 3D grid. In various rendering systems, each voxel comprises a virtual cube, triangle, octahedron, or other such shaped or geometric object associated with a particular portion of the scene. In ray tracing, a ray can be cast into an individual voxel to identify any object (e.g., a triangular mesh or higher order primitive) that the ray hits, and the intersection information can be stored for use in rendering. However, in various instances, it may be desirable to capture directional data for the voxel as well. For example, there may be material properties for the voxel, such that one light source is more important (contributes more) from a first direction, while another light source is more important from a second direction. In such instances, it may be desirable to utilize voxels that can capture at least some of this dimension. In at least one embodiment, octahedral voxels can be used for the objects in the scene, as shown in view 300 of FIG. 3A. In at least some embodiments, other shapes, such as tetrahedral voxels, can be used as well. As shown in Figure 3A, a ray from a light source 302 may strike at least two different faces of an octahedral voxel 304 that is visible in the virtual camera view. Also, as shown, there is at least one face of the voxel 304 on which the light source 302 does not strike the voxel 304. Such directional information may help to better determine the importance of the light source 302 relative to the voxel. In at least some embodiments, an octahedral shape may be chosen because it provides a simple and efficient way to determine the contributions from various light sources at different locations or directions in a scene.

[0015] The exemplary view 350 of FIG. 3B shows an octahedral voxel 352 having four sides or faces (A, B, C, D) visible in a virtual camera view. This may be, for example, a voxel representing a thin wall or a portion of an object with a first light source 354 on one side and a second light source 356 on the other side. As shown, the first light source 354 is important to two faces (A, C) and the second light source 356 is important to the other two faces (B, D). In such a case, the importance information may be determined at least in part based on directionality, with each light source being determined to be important only for a respective range of directions or number of faces. In at least one embodiment, the rendering interface may provide the user with an option to turn on or off the octahedral voxel representation of the 3D environment.

[0016] Each of these voxels can correspond to a respective spatial hash. There may be different rays that hit different parts of this same spatial hash. Using octahedral voxels to represent this hash provides a relatively simple way to determine and store directional light information for the same spatial hash. Such an approach allows both shadow rays and view (or proximity) rays to be traced or projected into these voxels from different light sources and directions, and to store this information for rendering. This information may also be useful in determining the relative importance and / or contribution of each light source to a given voxel, set of voxels, or region of a scene or image. However, it should be noted that the use of octahedral spatial hashes (maps) allows for the association of positions and directions in a scene with locations in memory where light contribution information can be stored. However, as discussed above, sampling all possible rays of a scene where ray sampling may be utilized, especially for non-point light sources, may be too inefficient and resource intensive in many situations. Instead of selecting rays randomly or using pseudo-random sampling, other sampling techniques or algorithms can be utilized, such as quasi-Monte Carlo based methods. Such methods can provide for the generation of low discrepancy sequences that can be converted into light paths. Such techniques can utilize deterministic low discrepancy sequences to generate path segments. To generate light path segments, the components of the i-th vector of the low discrepancy sequence can be split into at least two sets, which can then be used to track the i-th camera and light path segment.

[0017] Path segments according to at least one embodiment may be determined by first selecting an origin, such as a light source, and then selecting a direction to trace the ray. At the first intersection with a scene surface, another decision may be made regarding the path terminus and possibly the direction of scattering to trace the next ray. This process may be repeated as necessary. An exemplary approach for determining light paths utilizes shadow rays and determines whether the endpoints of each path segment of the shadow ray are visible. While shadow rays are sufficient for most diffuse surfaces, shadow rays may not be efficient for light paths that include specular-diffuse-specular segments, such as light that is reflected by a mirror onto a diffuse surface and reflected back by a mirror. To address this inefficiency, photon trajectories may be connected by proximity to the camera path segments, which may help to efficiently capture the contribution of shadow rays, also known as photon mapping. Using such an approach for path spatial filtering, various light path segments may be generated by following rays (or photon trajectories) from a selected light source and tracing paths from a virtual camera. Then, if the endpoints of these path segments are mutually visible with respect to the shadow ray, or are sufficiently close (e.g., close enough), then these endpoints are connected. In at least one embodiment, path spatial filtering can be used in conjunction with these connection techniques. i Contribution to i is the contribution to the vertices in the determined region

number

number

number

[0018] To ensure consistency, the size of the region should shrink with increasing number of samples n. In at least one embodiment, this may involve decreasing the radius r of the spherical region. Photon mapping can then be used to connect the endpoints of path segments that are less than the specified radius. Since the radius r(n) is in fact equivalent to the tangent points of the shadow rays in the limit, a degree of consistency can be achieved by decreasing the radius r(n) as the number of sampled light paths n increases. As with stochastic progressive photon mapping, successive batches of light transport paths can be processed. Depending on the low discrepancy sequence used, some block sizes may be preferable over others.

[0019] At the very least, progressive path-space filtering is a fast and consistent variance reduction technique that complements shadow rays and progressive photon mapping, so such techniques are beneficial in that even a sparse set of samples can provide enough information to produce a high-quality image. Various techniques can be used to select different vertices of the light path, but one choice is the first vertex along the path from the camera where the optical properties are known to be sufficiently diffuse. A low discrepancy sequence can be transformed into sample path space in successive batches of light transport paths, and for each path, one selected tuple is stored for path-space filtering. Since memory consumption is proportional to the batch size, it is relatively easy to determine the maximum natural number given the size of the tuples and the maximum size of the memory block.

[0020] In at least one embodiment, this light data can be temporarily stored or cached for use in generating an image of the scene. In at least one embodiment, radiance samples can be cached and interpolated to increase the efficiency of light transport simulations implemented in a rendering engine. Beyond radiance interpolation, path spatial filtering can efficiently filter across discontinuities such as fine geometry. Such filtering overcomes excessive trajectory division to reduce noise in cached samples, which allows for efficient path tracing. Additionally, through this computation, the presence of consistency-related artifacts within a frame can be reduced without individual computation. Repeating the averaging process within a batch of light transport paths can lead to further significant speedups, but may result in some blurring of lighting details.

[0021] Such a method may have consistency without weighting, but when the radius is large, x i The resulting image may appear overly blurred because contributions that could not be collected in the first step are now included in the average. To reduce this temporal artifact of light leakage and to include more contributions in the average by utilizing a larger radius, we use orbital division to determine how likely it is that vertex x i Contribute to si+jWeights can be determined to evaluate how well a given feature is created. Blur techniques can be used to perform blurring between geometries or between textures. When blurring between geometries, the similarity of a surface normal to other surface normals can be determined by their dot product. When blurring between textures, the image can be optimal if the optical surface properties are evaluated by the vertices for the contributions included in the average. For non-diffuse surfaces, such as glossy surfaces, these properties depend at least in part on the viewing direction, which can be stored with the vertices. Some of this additional memory can be saved if the direction is implicitly known to be similar, for example, to the direction of the query position xi as seen directly from the camera. For blurred shadows, considering a point light source, its visibility when viewed from different vertices can be the same or different. To avoid blurring sharp shadow boundaries, contributions can only be included for identical visibility. For ambient occlusion and lighting with environment maps, blurring can be reduced by comparing the lengths of the individual rays cast within the hemisphere at these vertices by thresholding their difference.

[0022] Such path spatial filtering techniques can be implemented on top of any sampling-based rendering algorithm and can be implemented with little additional overhead. Progressive algorithms can efficiently reduce variance and can be guaranteed to converge without persistent artifacts due to consistency. Such spatial filtering, or spatial hashing, techniques can be used to divide the screen into voxels, such as the octahedral voxels described with respect to Figures 3A and 3B. Such techniques are suitable for applications such as real-time ray tracing, and in at least some embodiments can be performed solely on one or more GPUs, while overcoming various deficiencies with prior art techniques. Such techniques can also be advantageous, at least in part, because they do not require additional functionality for sampling or tracing beyond functionality already present in various ray tracers. Instead, the process can collect statistics on rays cast into the scene to determine the importance of each light source to a particular portion of the scene and determine how much sampling to perform for that portion of the scene based on factors such as scene visibility and material properties. Such techniques may require many fewer samples than those required by prior techniques such as ReSTIR, especially in the case of world space sampling rather than screen space sampling. Prior techniques require many additional samples, which often result in approximations being performed instead, which may reduce the image quality of these prior techniques. Instead, techniques according to various embodiments may determine which light sources are important to various regions of a scene, cache information about the same, and generate actual samples with the light of the light sources based at least in part on the relative importance of the light sources. In at least some embodiments, at least two steps may be used to determine these samples, including determining the importance of the light sources and then using an analytical solution to select or "pick" samples from those light sources in the scene.In at least some embodiments, this can be performed using multiple importance sampling (MIS).

[0023] However, as discussed above, determining which light sources are important in a scene can be difficult for complex or dynamic scenes. For example, there may be only a few rays from a light source that impact a surface due to occlusion or positioning or motion within the scene. Sampling only a subset of the possible rays of a scene may result in rays from a light source to an object not being sampled for many frames, which may cause the light source to be calculated to be inappropriately less important to that object or region than it should be. In some embodiments, the sampling patterns utilized between two regions of a scene may be significantly different, resulting in grid-like artifacts on the boundaries of various voxels when tracing rays there. To address at least this type of artifact, the boundaries between voxels can be blurred by jittering the locations at which data is sampled. Such jittering or offset-based sampling can effectively turn these voxel-like artifacts into noise, which can help improve overall image quality. In such an approach, the probability distribution function (PDF) for the light selection can be kept consistent from frame to frame, resulting in a gradually changing noise pattern as the light source moves through the scene. In at least one embodiment, normal directions can be jittered as well, for similar reasons.

[0024] As discussed above, the importance value in at least some embodiments may be determined at the voxel level. The size of these voxels may not be regular; rather, the octahedral voxels of the scene may be arranged as voxels of potentially varying size, such as voxels that increase in size with distance from the virtual camera of the scene. In various embodiments, light information about all light rays already cast into the scene may be collected for each voxel, and an average light contribution is calculated at each voxel. In at least one embodiment, this contribution data may correspond to radiance data, or the amount of directional power from each light source that hits that voxel. In at least one embodiment, a light source, such as a spherical light source, may be treated as a single light source, at least for purposes of importance determination, independent of the multiple triangles or other geometric components that make up that source, although other light source abstractions may be used within the scope of various embodiments as well. In at least some embodiments, any object emitting light in the scene may be treated as a light source.

[0025] These average values ​​can then be used to generate a cumulative distribution function (CDF) for light selection or culling. A discretized CDF can be generated for each frame or for every n frames, where n can be selected to be a value such as 2, 4, 8, or 10 based on factors such as performance and resource availability. Such an approach can help amortize the cost of constructing the CDF. The average contribution of each light source at each voxel can be used to construct the CDF to determine the relative importance of these light sources to these voxels for each frame or for every n frames. The CDF for a real-valued random variable X is given by: F X (x) = P(X≦x) where the right hand side of this equation represents the probability that the value of the random variable X is less than or equal to x. The operations used to construct these functions, such as for determining the prefix sums and reduction sums, may involve high performance computing primitives that can be executed in parallel on various GPUs, and thus may be executed serially or in parallel in different embodiments. Various other performance enhancements may be utilized as well, such as for constructing these CDFs on voxels in parallel for voxels and lights, or for constructing CDFs for groups of voxels instead of individual voxels.

[0026] In various embodiments, the data for the CDF may take the form of an array of floating point numbers between 0 and 1, with every next successive number being greater than or equal to the previous number. A binary search may then be performed using this array. Each time a light source is sampled, the process may access the CDF of this voxel and perform a binary search, for example using a random number. The index in this array that corresponds to the binary search may then be utilized. Importance information may also be used to guide adaptive sampling. For example, a greater amount or fraction of sampling or shadow ray casting may be allocated to areas with multiple important lights to more accurately capture important light information without expending extra effort on areas where there may only be one or two important light sources and where it is less important to cast a large number of shadow rays or capture as many samples as possible. Thus, such an approach allows fewer shadow rays to be cast in areas where there may only be one or two important lights, while more shadow rays may be cast on areas of the scene where there appear to be more important lights. For potential performance improvements, the CDF can be compressed into a unorm. Instead of working with 32-bit floating point numbers, a lower precision can be used which does not bias the results. Instead of storing the CDF values ​​as floating point numbers, the values ​​can be stored as 8-bit values, which can save bandwidth and cache storage.

[0027] In some embodiments, identifying a smaller number of more important lights may enable the use of additional solutions or techniques that would otherwise be impractical to use. For example, linearly transformed cosines (LTC) based techniques may be utilized for a smaller number of light sources. Additionally, knowing which lights are most important to a region and how many are important may be useful for guided adaptive sampling, so that the shadow ray budget is not utilized uniformly across the entire rendered image, but rather, a portion of the budget is more heavily focused on where the effort is more important. In at least one embodiment, the PVD may be utilized to evaluate the variance, such as to perform light culling. Light culling may effectively thin out or remove from consideration lights that are not important to a region, and retain lights that are important or "hero" lights for that region. Evaluations, such as LTC evaluations, which may be relatively expensive, may then be performed only on the important lights. The numerical values ​​in the CDF, which may include statistical and standard deviation information, may then be used to determine when more shadow rays are needed due to high variance, in at least one embodiment.

[0028] In at least one embodiment, an attempt may be made to ensure that a PDF that receives a value of zero, indicating that a light source is not important to a region or voxel, has a chance of obtaining a non-zero value when that light source may in fact have some importance to that region or voxel. As discussed above, sampling techniques may miss rays that pass only from the light to the object through a small aperture or as a result of motion in the scene. It may be desirable to provide a way to capture this importance that may not have been captured over a period of random sampling. In at least one embodiment, this may be done by casting a number of rays at random lights in the scene, regardless of their importance or those that are considered relatively less important, for each frame or subset of frames. This provides an opportunity to sample these light sources for rays that may have hit the object or region, allowing the respective PDF to have a non-zero value as a result of a hit being detected. In some embodiments, this may include casting a small number of shadow rays at a number of randomly (and uniformly) picked light sources for each voxel, for each frame or every n frames. Thus, a small budget of shadow rays can be used to attempt to find light interactions previously missed by sampling techniques. Another approach is to ensure that the PDF never has a value of 0, so that some contribution from light sources is always allowed. In at least one embodiment, this can include setting a minimum allowable PDF value that is assigned to light sources.

[0029] The PDF is determined based at least in part on the number of lights in the scene. For continuous functions, the probability density function (PDF) is the probability that the value of the variable is x. For discrete distributions, the PDF is f(x)=P[X=x] In at least one embodiment, the PDF corresponds to a fraction of the lights in the scene, multiplied by a fraction of the shadow rays that cast from that spatial hash or voxel. In at least one embodiment, there may be a refresh of at least a subset of the light values ​​for every frame. New information obtained for a frame may be used to update the data structure that stores the average radiance. Once the calculations for a frame are finished, a new CDF is constructed, giving new probabilities for a given light source. In at least one embodiment, the number of entries may be fixed, and a replacement policy may be used to minimize the amount of memory required for these operations. Such an approach may store information only for a maximum number of the most important lights. In other approaches, light information may be stored only for lights with importance values ​​that meet or exceed a minimum importance threshold.

[0030] As mentioned above, in at least one embodiment, jitter-based blending can be used when sampling voxels. Other blending techniques that may be related to PDF animation can be used as well. This can include, for example, linear interpolation animation between PDFs. When a new PDF is obtained at the end of a frame, the old PDF is present from the previous frame, and it may be desirable to animate that transition. For such purposes, the PDF may be animated. Other techniques can move away from average-based techniques and instead utilize maximum-based techniques. As mentioned above, there may be lights that hit a voxel very rarely, or only in a small percentage of rays, and so their effect can be minimized by the average-based technique. However, if these rays have high radiance values, it may be desirable to preserve that information. A maximum-based technique can preserve information about the maximum contribution returned for a light at any voxel, with a corresponding blending technique between high and low PDFs such that the radiance of the high PDF can be preserved, without over-sampling that light due to its relatively low overall importance.

[0031] In some embodiments, a partial CDF may be generated and stored for use. A partial CDF may be constructed for a given CDF only for a fixed number, selection, or subset of the lights in a scene. In one example, a random variable may be sampled and compared to a threshold b. If the value of this random sample is greater than b, a light may be selected, for example, from a fixed-size list. This selection may be Probability=(1-b)*PartialPDFValue+b*(1 / n) This can be performed using a partial CDF with a probability given by b*(1 / n), where n is the total number of lights, and the PartialPDFValue is constructed using subtraction from the partial CDF. If the random sample is lower than b, a light may be selected uniformly from all lights with a probability given by b*(1 / n). To update the structure, for all lights selected uniformly for a frame, the maximum contribution from the light can be accumulated, and the light is selected in a fixed-size structure to replace with the maximum contribution. In at least one embodiment, the replacement can be implemented by selecting a random slot to replace, or by another method such as picking the lowest PDF, rolling a die to determine whether to replace it, etc.

[0032] FIG. 4 illustrates an example process 400 for rendering a frame or image of a scene that can be performed by various embodiments. It should be understood that there may be additional, fewer, or alternative steps performed in a similar or alternative order, or at least some in parallel, within the scope of various embodiments, unless otherwise noted, for the above and other processes presented herein. In this example, a number of rays are sampled 402 for multiple light sources and objects in the scene. Data from this sampling can be used to determine importance values ​​of these light sources for various regions of the scene 404. Various techniques can be used to divide the scene into regions, which may include dividing the regions into multiple octahedral voxels of various sizes. These importance values ​​can be cached 406, such as in GPU memory. In at least one embodiment, a probability value for picking each light source can be stored in the cache, and the probability value can be calculated using the importance values. For a frame to be rendered, which may be part of a series of frames, a number of rays to sample (from all possible rays that may be cast by all possible light sources) may be selected 408 based at least in part on these cached importance values. For a given region or voxel, the number of rays to sample may be based at least in part on the number of significant light sources and their relative importance. Once rays are selected, sampling may be performed 410 on the selected rays. This information may be used to update cached data for the scene. The light information obtained from these sampled rays may also be used to render the frame 412 to realistically illuminate objects in the scene based on factors such as the placement, shape, and radiance of these light sources relative to the objects in the scene. Other types of regions may be selected similarly, which may relate to tiles of uniform or different sizes.

[0033] FIG. 5 illustrates another example process 500 for rendering a frame that may be performed by at least one embodiment. This example process may represent one or more implementation options for a more general process such as process 400 illustrated in FIG. 4. In this example, a scene to be rendered is determined 502. This may include, for example, determining a virtual three-dimensional environment and objects and light sources located in or near the environment. In this example, the objects of the scene (including both foreground and background objects) may optionally be divided into a set of octahedral voxels, although other types of regions may be used within the scope of various embodiments. In some embodiments, spatial hashing allows for the determination of voxel identifiers or locations in memory for location and, if available during rendering, direction. In this environment, rays (e.g., shadow rays) may be cast for multiple light sources. For each voxel, an average light contribution from these rays may be calculated 508. This may include directional considerations in at least some embodiments. These average values ​​may be used to construct a cumulative distribution function (CDF) for light selection for the current frame in the sequence to be rendered for the scene. This CDF may then be used to select lights to sample during rendering of the current frame 512. These selected lights may then be sampled 514, and the light information from these sampled lights, as well as, in at least one embodiment, cached light information from previous frames or samplings, may be used to render the current frame 516. Unless there are no more frames to render for this sequence, the next frame may be selected 518 and the process may continue with another sampling using the updated CDF.

[0034] In at least one embodiment, aspects of the rendering and presentation of frames of content can be performed in various locations, such as on the client device, by a third party provider, or in the cloud, among other such options. Figure 6 illustrates example components 600 that can be used to generate, provide, and present such content, according to various embodiments. In at least one embodiment, a client device 602 can generate content for a session, such as a gaming session or a video viewing session, using components of a content application 604 on the client device 602 and data stored locally on the client device. In at least one embodiment, a content application 624 (e.g., a gaming or streaming media application) executing on the content server 620 can initiate a session associated with at least a client device 602 that can utilize a session manager and user data stored in a user database 634, determine content 632 via the content manager 626, render the content 632 using a rendering engine 628 if necessary for this type of content or platform, and transmit the content 632 to the client device 602 using a transmission manager 622 suitable for sending via download, streaming, or another such transmission channel. In at least one embodiment, a client device 602 that receives this content can provide this content to a corresponding content application 604, which may additionally or alternatively include a rendering engine 608 for rendering at least a portion of this content for presentation via the client device 602, such as video content via a display 606 and audio, such as voice and music, via at least one audio playback device, such as speakers or headphones.In at least one embodiment, at least a portion of this content may already be stored on, rendered on, or accessible to the client device 602 such that transmission over the network 640 is not required for at least that portion of the content, such as where the content may have been previously downloaded or stored locally on a hard drive or optical disk. In at least one embodiment, this content may be transferred to the client device 602 from the server 620 or the content database 634 using a transmission mechanism such as data streaming. In at least one embodiment, at least a portion of this content may be obtained or streamed from another source, such as a third party content service 650, which may also include a content application 652 for generating or providing the content. In at least one embodiment, some of this functionality may be performed using multiple computing devices, or multiple processors in one or more computing devices, which may include a combination of a CPU and a GPU.

[0035] In at least one embodiment, the content application 624 includes a content manager 626 that can determine or analyze the content before it is sent to the client device 602. In at least one embodiment, the content manager 626 can also include or work with other components that can generate, modify, or enhance the content to be provided. In at least one embodiment, this can include a rendering engine 628 for rendering the content, such as the aliased content, at a first resolution. In at least one embodiment, an upsampling or scaling component 630 can generate at least one additional version of the image at a different resolution, higher or lower, and can perform at least some processing, such as anti-aliasing. In at least one embodiment, a blending component 632, which can include at least one neural network, can perform blending on one or more of the images with respect to one or more previous images, as described herein. In at least one embodiment, the content manager 626 can then select an image or video frame of the appropriate resolution to send to the client device 602. In at least one embodiment, the content application 604 on the client device 602 may also include components such as a rendering engine 608 such that any or all of this functionality may additionally or alternatively be performed on the client device 602. In at least one embodiment, the content application 652 on the third party content serving system 650 may also include such functionality. In at least one embodiment, the location where at least some of this functionality is performed may be configurable or may depend on factors such as the type of client device 602 or the availability of a network connection with adequate bandwidth, among other such factors.In at least one embodiment, the upsampling module 630 or the blending module 632 may include one or more neural networks to perform or assist this function, and these neural networks (or at least the network parameters of those networks) may be provided by the content server 620 or a third party system 650. In at least one embodiment, the system for content generation may include any suitable combination of hardware and software in one or more locations. In at least one embodiment, the generated image or video content in one or more resolutions may be provided to or made available to other client devices 660, such as for download or streaming from a media source that stores a copy of the image or video content. In at least one embodiment, this may include transmitting images of game content for a multiplayer game, where different client devices may display the content at different resolutions, including one or more super-resolutions.

[0036] Data Center 7 illustrates an example data center 700 in which at least one embodiment may be used. In at least one embodiment, the data center 700 includes a data center infrastructure layer 710, a framework layer 720, a software layer 730, and an application layer 740.

[0037] 7, in at least one embodiment, a data center infrastructure layer 710 may include a resource orchestrator 712, grouped computing resources 714, and node computing resources ("node CRs") 716(1)-716(N), where "N" represents any positive integer. In at least one embodiment, the node CRs 716(1)-716(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state drives or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and cooling modules. In at least one embodiment, one or more of the nodes CR 716(1)-716(N) may be a server having one or more of the computing resources described above.

[0038] In at least one embodiment, the grouped computing resources 714 may include separate groups of node CRs housed in one or more racks (not shown), or multiple racks housed in a data center at various graphical locations (also not shown). The separate groups of node CRs in the grouped computing resources 714 may include grouped compute resources, network resources, memory resources, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs including CPUs or processors may be grouped in one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, the one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0039] In at least one embodiment, the resource orchestrator 712 may configure or otherwise control one or more nodes CR 716(1)-716(N) and / or grouped computing resources 714. In at least one embodiment, the resource orchestrator 712 may include a software design infrastructure ("SDI") management entity for the data center 700. In at least one embodiment, the resource orchestrator may include hardware, software, or some combination thereof.

[0040] In at least one embodiment, as shown in FIG. 7 , framework layer 720 includes a job scheduler 722, a configuration manager 724, a resource manager 726, and a distributed file system 728. In at least one embodiment, framework layer 720 may include a framework for supporting software 732 in software layer 730 and / or one or more applications 742 in application layer 740. In at least one embodiment, software 732 or applications 742 may each include web-based service software or applications, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, framework layer 720 may be a type of free and open source software web application framework, such as, but not limited to, Apache Spark® (hereinafter “Spark”), which can use distributed file system 728 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 722 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of the data center 700. In at least one embodiment, the configuration manager 724 may be capable of configuring different tiers, such as the software tier 730, as well as the framework tier 720 including Spark and a distributed file system 728 to support large scale data processing. In at least one embodiment, the resource manager 726 may be capable of managing clustered or grouped computing resources that are mapped or allocated to support the distributed file system 728 and the job scheduler 722. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 714 in the data center infrastructure tier 710.In at least one embodiment, resource manager 726 may coordinate with resource orchestrator 712 to manage these mappings or allocated computing resources.

[0041] In at least one embodiment, software 732 included in software layer 730 may include software used by at least a portion of nodes CR 716(1)-716(N), grouped computing resources 714, and / or distributed file system 728 of framework layer 720. The one or more types of software may include, but are not limited to, Internet web page searching software, email virus scanning software, database software, and streaming video content software.

[0042] In at least one embodiment, the applications 742 included in the application layer 740 may include one or more types of applications used by at least a portion of the nodes CRs 716(1)-716(N), the grouped computing resources 714, and / or the distributed file system 728 of the framework layer 720. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive compute, and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0043] In at least one embodiment, any of configuration manager 724, resource manager 726, and resource orchestrator 712 may implement any number and type of self-correcting actions based on any amount and type of data obtained in any technically feasible manner. In at least one embodiment, the self-correcting actions may enable data center operators of data center 700 to avoid determining potentially poor configurations and eliminate underutilized and / or underperforming portions of the data center.

[0044] In at least one embodiment, data center 700 may include tools, services, software, or other resources for training one or more machine learning models or predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to data center 700. In at least one embodiment, a trained machine learning model corresponding to the one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 700 by using the weight parameters calculated by one or more techniques described herein.

[0045] In at least one embodiment, the data center may use a CPU, application specific integrated circuit (ASIC), GPU, FPGA, or other hardware to perform training and / or inference using the resources described above. Additionally, one or more of the software and / or hardware resources described above may be configured as a service to enable a user to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0046] These components can be used to improve image quality during image generation by caching light importance information and using that information to determine which rays to sample for an image or video frame.

[0047] Computer Systems 8 is a block diagram illustrating an exemplary computer system, which may be a system having interconnected devices and components formed with a processor that may include an execution unit for executing instructions, a system on a chip (SoC), or some combination of these 800, according to at least one embodiment. In at least one embodiment, computer system 800 may include components such as, without limitation, a processor 802 for using an execution unit that includes logic for executing algorithms for processing data in accordance with the present disclosure, such as in the embodiments described herein. In at least one embodiment, computer system 800 may include a processor such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors available from Intel Corporation, Santa Clara, Calif., although other systems may be used (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.). In at least one embodiment, computer system 800 may run a version of the WINDOWS® operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (e.g., UNIX® and Linux®), embedded software, and / or graphical user interfaces may be used.

[0048] Embodiments may be used in other devices, such as portable devices and embedded applications. Some examples of portable devices include cellular phones, Internet protocol devices, digital cameras, personal digital assistants ("PDAs"), and portable PCs. In at least one embodiment, embedded applications may include microcontrollers, digital signal processors ("DSPs"), systems-on-chips, network computers ("NetPCs"), set-top boxes, network hubs, wide area network ("WAN") switches, or any other system capable of executing one or more instructions according to at least one embodiment.

[0049] In at least one embodiment, computer system 800 may include, without limitation, a processor 802, which may include one or more execution units 808 for performing training and / or inference of machine learning models according to the techniques described herein. In at least one embodiment, computer system 800 is a single processor desktop or server system, while in other embodiments, computer system 800 may be a multiprocessor system. In at least one embodiment, processor 802 may include, without limitation, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combination of instruction sets, or any other processor device, such as a digital signal processor. In at least one embodiment, processor 802 may be coupled to a processor bus 810, which may transmit data signals between processor 802 and other components within computer system 800.

[0050] In at least one embodiment, the processor 802 may include, without limitation, a level 1 ("L1") internal cache memory ("cache") 804. In at least one embodiment, the processor 802 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory may be external to the processor 802. Other embodiments may include a combination of both internal and external cache depending on the particular implementation and needs. In at least one embodiment, the register file 806 may store different types of data in various registers, including, without limitation, integer registers, floating point registers, status registers, and an instruction pointer register.

[0051] In at least one embodiment, processor 802 also includes an execution unit 808, including, without limitation, logic for performing integer and floating point operations. In at least one embodiment, processor 802 may also include a microcode ("u-code") read only memory ("ROM") that stores microcode for certain macroinstructions. In at least one embodiment, execution unit 808 may include logic for a packed instruction set 809. In at least one embodiment, by including packed instruction set 809 in the instruction set of a general purpose processor along with associated circuitry for executing the instructions, operations used by many multimedia applications may be performed using packed data in general purpose processor 802. In one or more embodiments, many multimedia applications may be accelerated and executed more efficiently by performing operations on packed data using the full width of the processor's data bus, thereby eliminating the need to transfer smaller units of data between the processor's data bus to perform one or more operations on one data element at a time.

[0052] In at least one embodiment, the execution unit 808 may also be used in a microcontroller, an embedded processor, a graphics device, a DSP, and other types of logic circuits. In at least one embodiment, the computer system 800 may include, without limitation, a memory 820. In at least one embodiment, the memory 820 may be implemented as a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or other memory device. In at least one embodiment, the memory 820 may store instructions 819 and / or data 821 represented by data signals that may be executed by the processor 802.

[0053] In at least one embodiment, a system logic chip may be coupled to the processor bus 810 and the memory 820. In at least one embodiment, the system logic chip may include, without limitation, a memory controller hub ("MCH") 816, and the processor 802 may communicate with the MCH 816 via the processor bus 810. In at least one embodiment, the MCH 816 may provide a high bandwidth memory path 818 to the memory 820 for storing instructions and data, and for storing graphics commands, data, and textures. In at least one embodiment, the MCH 816 may route data signals between the processor 802, the memory 820, and other components of the computer system 800, and may bridge data signals between the processor bus 810, the memory 820, and the system I / O interface 822. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 816 may be coupled to memory 820 via a high-bandwidth memory path 818, and the graphics / video card 812 may be coupled to the MCH 816 via an Accelerated Graphics Port ("AGP") interconnect 814.

[0054] In at least one embodiment, computer system 800 may use system I / O 822, which is a proprietary hub interface bus for coupling MCH 816 to I / O controller hub ("ICH") 830. In at least one embodiment, ICH 830 may provide direct connectivity to some I / O devices via a local I / O bus. In at least one embodiment, local I / O bus may include, without limitation, a high-speed I / O bus for connecting peripherals to memory 820, chipset, and processor 802. Examples may include, without limitation, an audio controller 829, a firmware hub ("flash BIOS") 828, wireless transceiver 826, data storage 824, legacy I / O controller 823 including user input and keyboard interface 825, serial expansion port 827 such as Universal Serial Bus ("USB"), and network controller 834. Data storage 824 may include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0055] In at least one embodiment, Figure 8 illustrates a system including interconnected hardware devices or "chips," while in other embodiments, Figure 8 may illustrate an exemplary system-on-a-chip ("SoC"). In at least one embodiment, the devices may be interconnected with proprietary interconnects, standard interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of computer system 800 may be interconnected using a compute express link (CXL) interconnect.

[0056] These components can be used to improve image quality during image generation by caching light importance information and using that information to determine which rays to sample for an image or video frame.

[0057] 9 is a block diagram illustrating an electronic device 900 for utilizing a processor 910, according to at least one embodiment. In at least one embodiment, electronic device 900 may be, for example, without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0058] In at least one embodiment, system 900 may include a processor 910 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices, without limitation. In at least one embodiment, processor 910 is coupled using a bus or interface, such as an I°C bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advance Technology Attachment (“SATA”) bus, a Universal Serial Bus (“USB”) (versions 1, 2, 3), or a Universal Asynchronous Receiver / Transmitter (“UART”) bus. In at least one embodiment, FIG. 9 illustrates a system including interconnected hardware devices or "chips," while in other embodiments, FIG. 9 may illustrate an exemplary system-on-a-chip ("SoC"). In at least one embodiment, the devices illustrated in FIG. 9 may be interconnected with proprietary interconnects, standard interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of FIG. 9 may be interconnected using a Compute Express Link (CXL) interconnect.

[0059] In at least one embodiment, FIG. 9 illustrates a display 924, a touch screen 925, a touch pad 930, a Near Field Communications unit ("NFC") 945, a sensor hub 940, a thermal sensor 946, an Express Chipset ("EC") 935, a Trusted Platform Module ("TPM") 938, a BIOS / firmware / flash memory ("BIOS,FW flash") 922, a DSP 960, a drive 920, such as a Solid State Disk ("SSD") or a Hard Disk Drive ("HDD"), a wireless local area network unit ("WLAN") 950, a Bluetooth unit 952, a Wireless Wide Area Network unit ("WWAN") 956, a Global Positioning System (GPS), a Bluetooth 952, ... The memory may include a USB 3.0 System unit 955, a camera, such as a USB 3.0 camera ("USB 3.0 Camera") 954, and / or a Low Power Double Data Rate ("LPDDR") memory unit ("LPDDR3") 915, implemented, for example, to the LPDDR3 standard. Each of these components may be implemented in any suitable manner.

[0060] In at least one embodiment, other components may be communicatively coupled to the processor 910 via the components described above. In at least one embodiment, an accelerometer 941, an ambient light sensor ("ALS") 942, a compass 943, and a gyroscope 944 may be communicatively coupled to the sensor hub 940. In at least one embodiment, a thermal sensor 939, a fan 937, a keyboard 946, and a touch pad 930 may be communicatively coupled to the EC 935. In at least one embodiment, a speaker 963, a headphone 964, and a microphone ("mic") 965 may be communicatively coupled to an audio unit (audio codec and class d amplifier) ​​962, which may be communicatively coupled to the DSP 960. In at least one embodiment, the audio unit 964 may include, for example, without limitation, an audio coder / decoder ("codec") and a class D amplifier. In at least one embodiment, a SIM card ("SIM") 957 may be communicatively coupled to the WWAN unit 956. In at least one embodiment, components such as the WLAN unit 950 and the Bluetooth unit 952, as well as the WWAN 956, may be implemented in a Next Generation Form Factor ("NGFF").

[0061] These components can be used to improve image quality during image generation by caching light importance information and using that information to determine which rays to sample for an image or video frame.

[0062] 10 is a block diagram of a processing system according to at least one example embodiment. In at least one embodiment, system 1000 includes one or more processors 1002 and one or more graphics processors 1008 and may be a single processor desktop system, a multiprocessor workstation system, or a server system having multiple processors 1002 or processor cores 1007. In at least one embodiment, system 1000 is a processing platform integrated into a system-on-chip (SoC) integrated circuit for use in a mobile, portable, or embedded device.

[0063] In at least one embodiment, the system 1000 may include or be incorporated into a server-based gaming platform, a game console including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, the system 1000 is a mobile phone, a smart phone, a tablet computing device, or a mobile Internet device. In at least one embodiment, the processing system 1000 may also include, be coupled to, or be integrated into a wearable device, such as a smart watch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device. In at least one embodiment, the processing system 1000 is a television or set-top box device having one or more processors 1002 and a graphical interface generated by one or more graphics processors 1008.

[0064] In at least one embodiment, the one or more processors 1002 each include one or more processor cores 1007 for processing instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of the one or more processor cores 1007 is configured to process a particular instruction set 1009. In at least one embodiment, the instruction set 1009 may facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing via very long instruction words (VLIW). In at least one embodiment, the processor cores 1007 each may process a different instruction set 1009, which may include instructions that facilitate emulation of other instruction sets. In at least one embodiment, the processor cores 1007 may also include other processing devices, such as digital signal processors (DSPs).

[0065] In at least one embodiment, the processor 1002 includes a cache memory 1004. In at least one embodiment, the processor 1002 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared among various components of the processor 1002. In at least one embodiment, the processor 1002 also uses an external cache (e.g., a level 3 (L3) cache or a last level cache (LLC)) (not shown), which may be shared among the processor cores 1007 using known cache coherence techniques. In at least one embodiment, the processor 1002 further includes a register file 1006, which may include different types of registers (e.g., integer registers, floating point registers, status registers, and an instruction pointer register) for storing different types of data. In at least one embodiment, the register file 1006 may include general purpose registers or other registers.

[0066] In at least one embodiment, the one or more processors 1002 are coupled to one or more interface buses 1010 to transmit communication signals, such as address, data, or control signals, between the processors 1002 and other components in the system 1000. In at least one embodiment, the interface bus 1010 may be a processor bus, such as, in one embodiment, a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface 1010 is not limited to a DMI bus, but may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), a memory bus, or other types of interface buses. In at least one embodiment, the processor 1002 includes an integrated memory controller 1016 and a platform controller hub 1030. In at least one embodiment, the memory controller 1016 facilitates communication between memory devices and other components of the system 1000, while the platform controller hub (PCH) 1030 provides connectivity to I / O devices via a local I / O bus.

[0067] In at least one embodiment, the memory device 1020 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or any other memory device having suitable performance to serve as process memory. In at least one embodiment, the memory device 1020 may operate as system memory for the system 1000 to store data 1022 and instructions 1021 for use when the one or more processors 1002 execute applications or processes. In at least one embodiment, the memory controller 1016 is also coupled to an optional external graphics processor 1012, which may communicate with one or more graphics processors 1008 within the processor 1002 to perform graphics and media operations. In at least one embodiment, a display device 1011 may be connected to the processor 1002. In at least one embodiment, the display device 1011 may include one or more of an internal display device, such as a mobile electronic device or laptop device, or an external display device attached via a display interface (e.g., a display port, etc.). In at least one embodiment, the display device 1011 may include a head mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0068] In at least one embodiment, the platform controller hub 1030 allows peripherals to connect to the memory device 1020 and the processor 1002 via a high-speed I / O bus. In at least one embodiment, the I / O peripherals include, but are not limited to, an audio controller 1046, a network controller 1034, a firmware interface 1028, a wireless transceiver 1026, a touch sensor 1025, and a data storage device 1024 (e.g., hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1024 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensor 1025 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 1026 may be a WiFi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1028 enables communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 1034 may enable network connectivity to a wired network. In at least one embodiment, a high performance network controller (not shown) couples to the interface bus 1010. In at least one embodiment, the audio controller 1046 is a multi-channel high definition audio controller. In at least one embodiment, the system 1000 includes an optional legacy I / O controller 1040 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to the system.In at least one embodiment, platform controller hub 1030 may also connect to one or more universal serial bus (USB) controllers 1042 for connecting input devices, such as a keyboard and mouse 1043 combination, a camera 1044, or other USB input devices.

[0069] In at least one embodiment, instances of memory controller 1016 and platform controller hub 1030 may be integrated into a separate external graphics processor, such as external graphics processor 1012. In at least one embodiment, platform controller hub 1030 and / or memory controller 1016 may be external to one or more processors 1002. For example, in at least one embodiment, system 1000 may include external memory controller 1016 and platform controller hub 1030, which may be configured as a memory controller hub and a peripheral controller hub in a system chipset that communicates with processor 1002.

[0070] These components can be used to improve image quality during image generation by caching light importance information and using that information to determine which rays to sample for an image or video frame.

[0071] 11 is a block diagram of a processor 1100 having one or more processor cores 1102A-1102N, an integrated memory controller 1114, and an integrated graphics processor 1108, according to at least one embodiment. In at least one embodiment, the processor 1100 may include a smaller number of additional cores, including additional core 1102N, represented by a dashed box. In at least one embodiment, each of the processor cores 1102A-1102N includes one or more internal cache units 1104A-1104N. In at least one embodiment, each processor core also has access to one or more shared cache units 1106.

[0072] In at least one embodiment, the internal cache units 1104A-1104N and the shared cache unit 1106 represent a cache memory hierarchy within the processor 1100. In at least one embodiment, the cache memory units 1104A-1104N may include at least one level of instruction and data cache within each processor core, as well as one or more levels of shared mid-level cache, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, where the highest level of cache before external memory is classified as an LLC. In at least one embodiment, cache coherence logic maintains coherency between the various cache units 1106 and 1104A-1104N.

[0073] In at least one embodiment, the processor 1100 may also include a set of one or more bus controller units 1116 and a system agent core 1110. In at least one embodiment, the one or more bus controller units 1116 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1110 provides management functions for the various processor components. In at least one embodiment, the system agent core 1110 includes one or more integrated memory controllers 1114 for managing access to various external memory devices (not shown).

[0074] In at least one embodiment, one or more of the processor cores 1102A-1102N include support for simultaneous multithreading. In at least one embodiment, the system agent core 1110 includes components for coordinating and operating the cores 1102A-1102N during multithreaded processing. In at least one embodiment, the system agent core 1110 may further include a power control unit (PCU), which includes logic and components for coordinating the power state of one or more of the processor cores 1102A-1102N and the graphics processor 1108.

[0075] In at least one embodiment, the processor 1100 further includes a graphics processor 1108 for performing graphics processing operations. In at least one embodiment, the graphics processor 1108 couples to a shared cache unit 1106 and to a system agent core 1110 that includes one or more integrated memory controllers 1114. In at least one embodiment, the system agent core 1110 also includes a display controller 1111 for directing output of the graphics processor to one or more coupled displays. In at least one embodiment, the display controller 1111 may also be a separate module coupled to the graphics processor 1108 via at least one interconnect or may be integrated within the graphics processor 1108.

[0076] In at least one embodiment, a ring-based interconnect unit 1112 is used to couple the internal components of processor 1100. In at least one embodiment, alternative interconnect units such as a point-to-point interconnect, a switched interconnect, or other techniques may be used. In at least one embodiment, graphics processor 1108 couples to ring interconnect 1112 via I / O link 1113.

[0077] In at least one embodiment, the I / O link 1113 represents at least one of a variety of I / O interconnects, including an on-package I / O interconnect that facilitates communication between various processor components and a high performance embedded memory module 1118, such as an eDRAM module. In at least one embodiment, each of the processor cores 1102A-1102N and the graphics processor 1108 use the embedded memory module 1118 as a shared last level cache.

[0078] In at least one embodiment, the processor cores 1102A-1102N are homogenous cores that execute a common instruction set architecture. In at least one embodiment, the processor cores 1102A-1102N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 1102A-1102N execute a common instruction set, while one or more other of the processor cores 1102A-1102N execute a subset of the common instruction set, or a different instruction set. In at least one embodiment, the processor cores 1102A-1102N are heterogeneous in terms of micro-architecture, where one or more cores with relatively higher power consumption are combined with one or more cores with lower power consumption. In at least one embodiment, the processor 1100 can be implemented on one or more chips or as an SoC integrated circuit.

[0079] These components can be used to improve image quality during image generation by caching light importance information and using that information to determine which rays to sample for an image or video frame.

[0080] Other variations are within the scope of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrative examples thereof have been shown in the drawings and have been described in detail above. However, there is no intention to limit the disclosure to the particular disclosed form or forms, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure as defined by the appended claims.

[0081] The use of the terms "a" and "an" and "the" and similar referents in the context of describing the disclosed embodiments (particularly in the context of the claims that follow) should be construed to cover both the singular and the plural and should not be construed as defining terms, unless otherwise stated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (meaning "including, but not limited to"), unless otherwise stated. The term "connected," when referring to a physical connection without modification, is to be construed as being partially or completely contained within, attached to, or joined to one another, even if there is something intervening. The recitation of ranges of values ​​herein is merely intended to serve as a shorthand method of individually referring to each separate value falling within the range, unless otherwise stated herein and unless each separate value is incorporated into the specification as if it were individually recited herein. Use of the term "set" (e.g., "set of items") or "subset" should be construed as a non-empty collection having one or more members, unless otherwise indicated or constrained by context. Further, unless otherwise indicated or constrained by context, the term "subset" of a corresponding set does not necessarily refer to a strict subset of the corresponding set, and a subset and a corresponding set may be equivalent.

[0082] Conjunctions such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C" are understood in the context in which they are generally used to indicate that an item, term, etc. is A, B, or C, or a non-empty subset of any of the sets A, B, and C, unless specifically stated otherwise or clearly contradicted by context. For example, in the illustrative example of a set having three members, the conjunctive phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctives do not generally imply that a given embodiment requires the presence of at least one A, at least one B, and at least one C, respectively. Further, unless specifically stated otherwise or clearly stated by context, the term "plurality" refers to a plurality (e.g., "a plurality of items" refers to multiple items). A plurality is at least two items, but may be more if indicated explicitly or by context. Further, unless specifically stated otherwise or clearly stated otherwise by context, the phrase "based on" means "based at least in part on" and not "based only on."

[0083] The operations of the processes described herein may be performed in any suitable order unless otherwise stated herein or clearly contradicted by context. In at least one embodiment, a process such as the processes described herein (or variations and / or combinations thereof) is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed under the control of one or more computer systems configured with executable instructions and collectively executed on one or more processors, by hardware, or by a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., propagating transitory electrical or electromagnetic transmissions), but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver of a transitory signal. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer readable storage media that store executable instructions (or have other memory for storing executable instructions) that, when executed by (i.e., as a result of being executed by) one or more processors of a computer system, cause the computer system to perform the operations described herein. The set of non-transitory computer readable storage media, in at least one embodiment, comprises a plurality of non-transitory computer readable storage media, where one or more of the individual non-transitory storage media of the plurality of non-transitory computer readable storage media does not contain all of the code, but the plurality of non-transitory computer readable storage media collectively store all of the code.In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors, e.g., a non-transitory computer-readable storage medium stores the instructions, a main central processing unit ("CPU") executes some instructions, and a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of a computer system have separate processors, and the different processors execute different subsets of the instructions.

[0084] Thus, in at least one embodiment, a computer system is configured to implement one or more services that, singly or collectively, perform the operations of the processes described herein, and such computer system is configured with applicable hardware and / or software that enables the operations to be performed. Further, a computer system that implements at least one embodiment of the present disclosure may be a single device, or in another embodiment, a distributed computer system comprising multiple devices that operate in different ways, such that the distributed computer system performs the operations described herein such that no single device performs all the operations.

[0085] Any examples provided herein, or the use of exemplary language (e.g., "such as"), are intended only to further clarify embodiments of the disclosure and do not limit the scope of the disclosure unless otherwise stated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0086] All references cited in this specification, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference was individually indicated to be incorporated by reference and was set forth in its entirety herein.

[0087] In the specification and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms may not be intended as synonyms for each other. Rather, in certain instances, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still interlock or interact with each other.

[0088] Unless specifically noted otherwise, terms such as "processing," "computing," "calculating," or "determining" throughout the specification refer to the acts and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data represented as physical quantities, such as electronic quantities, in the registers and / or memory of the computing system into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display device of the computing system.

[0089] Similarly, the term "processor" may refer to any device, or part of a device, that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" may be a CPU or a GPU. A "computing platform" may comprise one or more processors. A "software" process as used herein may include software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Also, each process may refer to multiple processes for executing instructions serially or in parallel, continuously or intermittently. The terms "system" and "method" are used interchangeably herein to the extent that a system may embody one or more methods and a method may be considered a system.

[0090] As used herein, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. Obtaining, acquiring, receiving, or inputting analog or digital data may be accomplished in various ways, such as receiving the data as a parameter of a function call or a call to an application programming interface. In some implementations, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be accomplished by transferring the data over a serial or parallel interface. In other implementations, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be accomplished by transferring the data over a computer network from a providing entity to an acquiring entity. Reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data may be accomplished by transferring the data as an input or output parameter of a function call, a parameter of an application programming interface, or an inter-process communication mechanism.

[0091] Although the above discussion describes example implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Additionally, while a specific distribution of roles is defined for purposes of discussion, various functions and roles may be distributed and divided in different manners depending on the circumstances.

[0092] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. determining light information relating to a first set of light rays cast on one or more light sources in the virtual environment; determining values ​​of the one or more light sources for a plurality of spatial regions of the virtual environment based at least in part on the light information; selecting a second set of rays to sample for the one or more light sources for the multiple spatial regions based at least in part on the value, the second set of rays being selected such that light sources with higher values ​​have a greater number of rays sampled for the light source; sampling the second set of rays to obtain updated illumination information for the plurality of spatial regions; Rendering an image of the virtual environment using the updated lighting information.

4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , further comprising: determining the plurality of spatial regions of the virtual environment using a spatial hashing algorithm, the spatial regions also providing directional information.

3. 3. The computer-implemented method of claim 2, wherein the spatial hashing algorithm segments the virtual environment into a plurality of spatial regions corresponding to a plurality of octahedral voxels, the plurality of octahedral voxels being one or more sizes.

4. The computer-implemented method of claim 3 , further comprising determining a value of the selected light source for each octahedral voxel based at least in part on directional information regarding the selected light source.

5. using the values ​​to construct a cumulative distribution function (CDF) for the plurality of spatial regions; caching selection probability data determined according to the CDF for the plurality of spatial regions, the selection probability data being used to select the second set of rays; The computer-implemented method of claim 1 , further comprising:

6. The computer-implemented method of claim 5 , further comprising updating the CDF for the plurality of spatial regions for at least a subset of images of a sequence of images of the virtual environment.

7. The computer-implemented method of claim 1, further comprising a step of selecting a maximum number of selectable light sources, the light sources having the highest said values, from which the second set of light sources is sampled.

8. The computer-implemented method of claim 1 , further comprising determining the values ​​of the one or more light sources for the multiple spatial regions based at least in part on an average light contribution of individual light sources to individual spatial regions.

9. The computer-implemented method of claim 1 , further comprising causing one or more light sources whose determined value is less than the threshold to be sampled for sampling for one or more subsequent images to be rendered.

10. 1. A system comprising: A processor; When executed by the processor, the system further comprises: determining light information regarding a first set of light rays cast on one or more light sources in the virtual environment; determining values ​​of the one or more light sources for a plurality of spatial regions of the virtual environment based at least in part on the light information; selecting a second set of rays to sample for the one or more light sources for the multiple spatial regions based at least in part on the value, wherein the second set of rays is selected such that light sources with higher values ​​have a greater number of rays sampled for the light source; sampling the second set of rays to obtain updated illumination information for the plurality of spatial regions; and Rendering an image of the virtual environment using the updated lighting information. a memory containing instructions for performing Including, the system.

11. The instructions, when executed, cause the system to: The system of claim 10 , further comprising: determining the plurality of spatial regions of the virtual environment using a spatial hashing algorithm.

12. 12. The system of claim 11, wherein the spatial hashing algorithm segments the virtual environment into a plurality of spatial regions defined by a plurality of octahedral voxels, the plurality of octahedral voxels being one or more sizes.

13. The instructions, when executed, cause the system to: The system of claim 11 , further comprising: determining a value of a selected light source for each octahedral voxel based at least in part on directional information regarding the selected light source.

14. The instructions, when executed, cause the system to: constructing a cumulative distribution function (CDF) for the plurality of spatial regions using the values; caching selection probability data determined according to the CDF for the plurality of spatial regions, the selection probability data being used to select the second set of rays; and The system of claim 10 , further comprising:

15. The instructions, when executed, cause the system to: The system of claim 10 , further comprising: updating a cumulative distribution function (CDF) for the plurality of spatial regions for at least a subset of images of a sequence of images of the virtual environment.

16. The instructions, when executed, cause the system to: The system of claim 10 , further comprising: determining the values ​​of the one or more light sources for the multiple spatial regions based at least in part on an average light contribution of each light source to each spatial region.

17. The system further comprises: A system for performing graphical rendering operations; A system for performing a simulation operation; A system for performing simulation operations to test or validate an autonomous machine application, A system for performing deep learning operations, A system implemented using edge devices; A system incorporating one or more virtual machines (VMs); A system implemented at least partially within a data center; or A system implemented at least in part using cloud computing resources The system of claim 10 , comprising at least one of:

18. When executed by one or more processors, the one or more processors are determining light information regarding a first set of light rays cast on one or more light sources in the virtual environment; determining values ​​of the one or more light sources for a plurality of spatial regions of the virtual environment based at least in part on the light information; selecting a second set of rays to sample for the one or more light sources for the multiple spatial regions based at least in part on the value, wherein the second set of rays is selected such that light sources with higher values ​​have a greater number of rays sampled for the light source; sampling the second set of rays to obtain updated illumination information for the plurality of spatial regions; and Rendering an image of the virtual environment using the updated lighting information. A non-transitory computer-readable storage medium containing instructions to cause

19. The instructions, when executed, cause the one or more processors to:

20. The non-transitory computer-readable storage medium of claim 18, further comprising: determining the plurality of spatial regions of the virtual environment using a spatial hashing algorithm, the plurality of spatial regions being octahedral voxels of one or more sizes.

20. The instructions, when executed, cause the one or more processors to:

20. The non-transitory computer-readable storage medium of claim 19, further comprising: determining a value of the selected light source for each octahedral voxel based at least in part on directional information regarding the selected light source.

Images