Method and system for inverse transform sampling by ray tracing

Abstract

The present disclosure relates to inverse transform sampling by ray tracing. High quality image rendering can be partially achieved by using inverse transform sampling to direct samples to more important areas (e.g., areas with higher luminance values) to reduce noise and improve convergence. Inverse transform sampling can be more efficiently implemented by reformulating the tree traversal unit, which can be accelerated, as a ray tracing problem. A geometric grid can be generated based on a set of cumulative distribution functions (CDFs) for individual rows and columns of pixels in a texture, and individual rays can be traced against the grid, with higher intersection probabilities at points with greater importance (e.g., higher luminance values). A probability distribution function for importance sampling can be derived by analyzing the partial derivatives of the CDF geometry at the intersection locations.

Description

BACKGROUND

[0001] As display technology continues to improve, it is correspondingly desirable to improve the quality of content to be presented on these displays. For content such as games or animations, this can involve techniques such as ray tracing to determine color values for individual pixels of rendered image or video content. For real-time ray tracers, for example, image data streams can be generated for online games, and performance is important to avoid latency issues while still providing high quality content. Attempts to reduce noise or differences between image or video frames are often limited by a given performance budget, which often results in selection of less effective noise reduction strategies than might be possible with more effective but more costly strategies. BRIEF DESCRIPTION OF DRAWINGS

[0002] Various embodiments according to the present disclosure will be described with reference to the drawings, in which:

[0003] Figure 1A And 1B An example image that can be processed according to at least one embodiment is shown;

[0004] Figure 2A And 2B Probabilities and cumulative distribution functions that can be determined from image or texture data according to various embodiments are shown;

[0005] Figure 3A , 3B 3C and 3D show representations of a texture for ray tracing based sampling according to at least one embodiment;

[0006] Figure 4 An example rendering pipeline for rendering image content according to at least one embodiment is shown;

[0007] Figure 5 A process for determining lighting or shading information for an image according to at least one embodiment is shown;

[0008] Figure 6 Components of a system for generating and / or transmitting image data according to at least one embodiment are shown;

[0009] Figure 7A Inference and / or training logic is shown in FIG. 10 that can be used in various examples;

[0010] Figure 7B Inference and / or training logic is shown in FIG. 10 that can be used in various examples;

[0011] Figure 8 An example data center system according to at least one embodiment is shown;

[0012] Figure 9A computer system according to at least one embodiment is shown;

[0013] Figure 10 A computer system according to at least one embodiment is shown;

[0014] Figure 11 At least a portion of a graphics processor according to one or more embodiments is shown;

[0015] Figure 12 At least a portion of a graphics processor according to one or more embodiments is shown;

[0016] Figure 13 This is an example data flow diagram of an advanced computing pipeline according to at least one embodiment;

[0017] Figure 14 This is a system diagram of an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment; and

[0018] Figure 15A and Figure 15B A data flow diagram of the process for training a machine learning model according to at least one embodiment is shown, as well as a client-server architecture for enhancing annotation tools using a pre-trained annotation model. Detailed Implementation

[0019] Methods according to various embodiments can provide for the generation of image data. In particular, various embodiments can provide for the efficient generation of high-quality images from components such as textures and assets, where the textures can provide illumination sources for one or more assets in a scene to be rendered. To make such illumination or shading processes efficient, various aspects of the process can be mapped to hardware units that can provide hardware-based acceleration. Methods according to at least one embodiment can determine one or more cumulative distribution functions of a texture, and a geometric representation can be constructed from this cumulative distribution, such that one or more rays can be traced against this geometry for sampling in a hardware-accelerated manner. Large emissive textures can be used, which can increase the detail and overall realism of the scene, providing an effective source of ambient lighting. Significant sampling of these large emissive textures can help reduce the amount of noise in the image, which could be introduced, for example, if these textures were sampled naively, since most of the incident radiation might come from a small fraction of the texture's texels, and equally probable random sampling might result in these small, influential texels being sampled very little.

[0020] In at least one embodiment, importance sampling of large light textures can be performed using inverse transform sampling. While traditional inverse transform sampling can be successfully utilized, such sampling performance can be unreliable due to the unpredictability of binary search. Further, traditional inverse transform sampling does not allow control over whether or how much to prioritize variance reduction or performance. Methods in accordance with at least one embodiment can utilize ray traversal searches, such as a single hardware-accelerated ray traversal search over cumulative probability geometry, rather than traditional binary searches performed in texture space. Using ray traversal searches can leverage ray tracing hardware, which can significantly improve the performance of importance sampling. Such importance sampling performance improvements have been shown to translate to improved rendering performance, up to 70% in one example, while achieving similar variance reduction to traditional inverse transform sampling. Further, the cumulative probability geometry of intersections can be modified to prioritize performance over variance reduction.

[0021] Figure 1A An example image 100 to be generated is shown. In this example, there are two main components that make up this image. A background texture is used, which provides a view of one or more background objects of a scene, and a foreground object 104 representing an object, such as a character avatar, non-player character, or scene object, in this case a boat. In various embodiments, information useful for lighting or shading of assets or objects will be included in the texture. For simplicity, the term "lighting" as used here generally refers to determining the amount of light incident on an object or asset surface, or color values resulting from such lighting, even though other terms such as "shading" can be more commonly used for particular applications or use cases. To make the generated image appear more realistic when displayed, the rendering process can determine aspects of lighting, reflections, shadows, and caustic patterns that can result from one or more light sources in the scene. In this example, the background objects in the example texture can influence the lighting or color of the boat 104 in the scene. Based on factors such as the position and objects in the scene, there will be some objects that are more able to influence the lighting of various objects in the scene than others. In this example, if rendered realistically, the sun 102 will provide the majority of the lighting in the scene, while the sky region can provide at least some amount of lighting and the ground or trees can provide little lighting. If the brightness values of individual pixels are analyzed, etc., these brightness values will represent the amount of lighting provided by various portions of a given texture. As an example, Figure 1B An example image 100 to be generated is shown. In this example, there are two main components that make up this image. A background texture is used, which provides a view of one or more background objects of a scene, and a foreground object 104 representing an object, such as a character avatar, non-player character, or scene object, in this case a boat. In various embodiments, information useful for lighting or shading of assets or objects will be included in the texture. For simplicity, the term "lighting" as used here generally refers to determining the amount of light incident on an object or asset surface, or color values resulting from such lighting, even though other terms such as "shading" can be more commonly used for particular applications or use cases. To make the generated image appear more realistic when displayed, the rendering process can determine aspects of lighting, reflections, shadows, and caustic patterns that can result from one or more light sources in the scene. In this example, the background objects in the example texture can influence the lighting or color of the boat 104 in the scene. Based on factors such as the position and objects in the scene, there will be some objects that are more able to influence the lighting of various objects in the scene than others. In this example, if rendered realistically, the sun 102 will provide the majority of the lighting in the scene, while the sky region can provide at least some amount of lighting and the ground or trees can provide little lighting. If the brightness values of individual pixels are analyzed, etc., these brightness values will represent the amount of lighting provided by various portions of a given texture. As an example, Figure 1Aa low resolution version 150 of the texture, for simplicity of explanation, using a low resolution. The color values of these pixels correspond to the relative brightness or other aspect of light of these pixels, which can be referred to as a type of "importance" value of the object or scene lighting. As shown, the pixel 152 corresponding to the position of the sun in the texture is shown as the brightest pixel, or the pixel of greatest importance for lighting. The pixels 154 corresponding to the sky region are not as bright, indicating that their contribution to the overall lighting of the scene is generally less than the sun, and thus have a lower importance from a lighting perspective. The pixels 156 corresponding to the ground or terrain region are even less bright, indicating that their contribution to the lighting of objects in the scene is even less, and their importance from a lighting perspective is even lower.

[0022] When rendering a scene, one method by which a highly realistic image can be generated is by sampling lighting contributions by tracing rays from the screen into the 3D scene. When these traced rays hit objects in the 3D scene, shadow rays can be traced from those intersection locations on the 3D objects to another object to which the emission texture is applied. In many instances, an artist will construct a sphere that encloses the entire 3D scene, with the emission texture mapped to the sphere using a latitude / longitude projection. The emission texture sphere can then illuminate the scene from various directions. In other scenes, the artist can instead apply the emission texture to 3D objects in the scene. For example, a 3D quadrilateral face can have a "lava" texture applied, with the lava texture emitting an orange light. Performing a binary search to identify pixels in a background image to trace a ray during classic inverse transform sampling can result in a large amount of computation for each image or video frame, which can prove too expensive in many cases, particularly for real-time rendering of images that can need to be generated at frame rates of approximately 60 Hz, 90 Hz, or higher, and often need to be performed using client devices that can have limited rendering or computational capabilities. Thus, methods according to various embodiments can instead trace a subset of these rays for each image or frame to determine lighting, color, and other such values. In many embodiments, this involves sampling different subsets of pixels or rays for different frames, as using the same subset for each frame can result in missing contributions of some pixels or overemphasizing contributions of other pixels.

[0023] In selecting pixels or rays for sampling, it can be beneficial to concentrate these samples on pixels or locations that are more important to the lighting of the object or from a lighting perspective. Thus, methods according to at least some embodiments can utilize methods such as importance sampling. Importance sampling is an approximation method that can help reduce variance in many cases, as compared to traditional acceptance or rejection sampling.

[0024] In at least one embodiment, a geometric representation of an emissive texture can be generated based at least in part on the probability distributions determined for those textures. As one example, Figure 2A A set of distributions 200 that can be determined in one dimension (ID) is shown. A row 202 of example ID luminance values is used. From this distribution, a probability distribution 204 can be computed. As shown, this distribution indicates that the highest lighting importance comes from the middle region of the row, whose width (or standard deviation around the mean) corresponds to the relative luminance of the pixels in that row. A cumulative distribution function 206 based on this probability distribution function can be generated. As shown, this cumulative function has the steepest slope at the most important locations when plotted. This property is suitable for ray-tracing based sampling.

[0025] For example, consider Figure 2B A view 250 of these distributions is shown. Due to the relative steepness of the curves at different luminance or importance regions, random rays will have a higher probability of hitting or intersecting a point on the cumulative distribution curve that corresponds to a higher luminance or importance. As shown in this example, a majority (here, 60%) of the pixels that rays can be projected into will hit a pixel in the most important region (e.g., the sun region). Then, a majority of these rays will intersect the curve at a location that corresponds to a minority (here, 40%) of the pixels, which also correspond to the most important pixels. Thus, this approach increases the probability that a ray will intersect a pixel with a relatively high luminance or importance value.

[0026] For example, to use this approach for 2D, 3D, 4D, or multi-modal images, this approach can be extended beyond one-dimensional data. In some embodiments, slices of 3D data can be determined to reduce the 3D case to a set of 2D cases. Figure 3A and 3B An example approach that can be used to interpret a low resolution texture as previously shown in Figure 1B In this example, the luminance values can first be analyzed using a probability distribution function for each row. For each row, as part of the conditional distribution, the set of probabilities for individual pixels can sum to 1. For rows 302, 304, 306 that have relatively similar luminance or importance values, the distribution function will show similar values for all pixels in a given row. For rows 308 that have a significant variation in luminance or importance, such as a row that includes one or more pixels that correspond to the sun, the relative importance of the pixels will be high, here represented in white, while the other pixels in the row will be low, represented in dark. In a range of importance normalized between 0 and 1, where 1 represents the highest importance and 0 represents the lowest importance, a white pixel can correspond to a value of 1 and a black pixel can correspond to a value of 0. A similar analysis can be performed on the columns of this low resolution texture based at least in part on the values in that column.

[0027] Alternatively, a cumulative distribution function (CDF) can be determined for this texture, such as... Figure 3B Image 330 is shown. In this example, a cumulative distribution has been determined for each row. Since the pixel values ​​in rows 302, 304, and 306 are substantially similar across rows, the cumulative distribution will have a slope that increases substantially linearly from the minimum importance value to the maximum importance value. However, for row 380 with varying brightness or importance, the cumulative function will have a significant slope change near the point of highest brightness or importance or within the standard deviation of the mean of the probability distribution curve. As shown, the change in importance value is much more drastic near the brightness or highest importance region. In this approach, a probability is assigned to each row such that the sum of the row probabilities is 1, as part of a marginal distribution. This marginal CDF can then be inverted, and the CDF can be randomly sampled to select a row to redirect samples to. Using that row, the sampling process can invert the corresponding CDF of that row with random values ​​to select the column to which samples are redirected. In various embodiments, the user, application, or entity may choose to geometrically encode the inverse CDF using width, height, depth, and / or corresponding geometry and metadata.

[0028] The advantages of this method are Figure 3C It is shown in a 360-degree perspective view. In this view, it is relative to... Figure 3B Rotated, but still displaying the same rows, the importance or brightness value of each pixel location is represented as a height from 0 to 1 in addition to color. While this example is illustrated graphically for illustrative purposes, it should be understood that these functions can be mathematically calculated, and these graphs or views are not necessary for implementing aspects of the various embodiments presented herein. To perform an importance sampling, a set of random rays can be traced to determine intersections with the geometric grid or representation, and the texture pixels corresponding to these intersections can be used to determine the lighting of the corresponding image or video frame. In at least some embodiments, this may involve tracing multiple rays at random (or otherwise determined, selected, or sampled) locations and determining intersections. As shown in the figure. Figure 3CAs shown, the amount of "intersectable" area for a given pixel will correspond to the change in importance at or near that pixel location, corresponding to the slope of the cumulative distribution function at that location. As shown, the pixel location in the corresponding texture that corresponds to the sun or most important pixel has significantly more intersectable area than other pixels in this view, and thus the pixel with the highest importance has the highest probability of a random ray 362 intersecting it. In this particular view, the pixel corresponding to the sun has approximately 6 times the likelihood of intersecting any other pixel, as it has the most impact on the illumination of the objects in this scene. Thus, random sampling or tracing of rays will have a significantly higher probability of resulting in a sample of this most important pixel. For higher resolution images where there can be many pixels that correspond to a light source such as the sun, this can increase the likelihood that one or more or possibly many of these rays will sample the pixel related to the sun for each frame.

[0029] In at least some embodiments, it can be desirable to smooth the geometry grid. A smoother grid can require fewer triangles or geometric shapes to represent, which can require less time to trace rays against the grid and test for intersections. Figure 3D One such geometry grid 390 is shown in FIG. 3B. In this example, smoothing has been applied to the geometry grid, which can take into account the values or importance of surrounding pixels. As shown, even after smoothing, tracing a ray 392 will still have the highest probability of hitting a location corresponding to the sun-related pixel.

[0030] Figure 4Components of an example rendering pipeline 400 that can be used to render an image are shown in accordance with various embodiments. In this example, an application 402 runs on a central processing unit (CPU) 402, where the application includes instructions that can be stored in system memory 404 and executed by the CPU. For example, the application can be a video game or animation application or a process that provides data about an image to be rendered. In this example, data for rendering an image can be provided to a graphics processing unit (GPU) 410 through an application programming interface (API) runtime 406 or other such interface mechanism. As noted above, for at least certain types of rendering or tasks for which a GPU can provide improved performance relative to a CPU, particularly for a large number of small parallel tasks such as can be used to render an image, particularly where hardware acceleration can be applied to at least some of these tasks. Instructions can be stored in GPU memory 412 until they are selected or scheduled for execution. In this example, data and instructions can be passed to one or more shaders 414, which can include one or more vertex shading components 416 for adding effects to objects in a scene or environment (typically a 3D environment) by determining vertex data for the objects in the scene, which can then perform various mathematical operations on the object vertex data. In this example, the vertex data is passed to one or more geometry components 418, which can perform various tasks such as at least some of those described herein. In this example, this can include tasks such as performing model and view transformations, performing vertex shading and lighting, performing data projection, performing clipping or culling of geometry-based data, and determining appropriate scene mapping, among others. For shading or lighting tasks described herein that can be based at least in part on a cumulative distribution function, these tasks can be performed within the shaders 414 of one or more GPUs on a single computing device or distributed across multiple devices. After performing these various geometry-based tasks, the resulting data can be passed to shading components 420, which can perform tasks such as single pixel shading in order to generate output image data for various pixels. This data can then be cached in one or more buffers 424 in (or external to) GPU memory 422 (which can be the same as or separate from GPU memory 412) until this information is transferred for presentation via at least one display 430 or other such mechanism, which can be attached to or included in at least one computing device or system, which can be the same computing device or system as includes CPU 402 and GPU 410. This process can be performed for each image to be generated, which can constitute a sequence of video frames to be presented via display 430.As discussed elsewhere herein, the display 430 is not limited to traditional video display devices such as televisions, monitors, or touchscreens, but can also include projectors, VR / AR / MR headsets, wearable displays, holographic displays, and the like. As will be appreciated. Figure 6 As discussed in greater detail, such components can be included in a client device for which video is to be displayed, a server transmitting content to a client device, or a third-party system generating image data on behalf of a client or server device, among other such options.

[0031] Figure 5 An example process 500 for generating an image that can be performed in accordance with various embodiments is shown. It will be appreciated that for this and other processes discussed herein, additional, fewer, or alternative steps can be performed in similar or alternative order, or at least partially in parallel, within the scope of various embodiments, unless otherwise specifically noted. Further, while this process is described with respect to image generation, it will be appreciated that the advantages of this importance sampling approach can also be advantageously used for other applications or uses. In this example, at least one texture is received 502 that is to be used to illuminate one or more assets in an image to be rendered. In other embodiments or examples, assets or objects other than textures can be used to provide representations of background objects, light sources, and other such features of an image to be generated. In this example, the texture itself can be an image or image dataset that can include color, pixel, luminance, or other such values for individual locations in the texture, which can correspond to individual pixel locations. For each pixel location in the texture, an importance value can be determined 504 that corresponds at least to a luminance value for that pixel. Other values can also be used, such as can include color values or texture properties, that can be used to determine the degree to which an object represented by that pixel, when rendered, affects the appearance of another object or pixel included in the image. As previously noted, a pixel in the texture that corresponds to the sun can have a higher importance value than a pixel that corresponds to a dark night sky, as they have a greater impact on the illumination of objects in the scene.

[0032] In this example, a set of cumulative distribution functions (CDFs) can be determined 506 based at least in part on these importance values for computing a texture. In some embodiments, a set of probability distributions can be determined for individual rows, columns, or other subsets of the texture, and the CDFs can be determined from these individual probability distributions. The set of CDFs can then be used to generate 508 a geometric grid or other geometric representation. This can include, for example, a two- or three-dimensional grid in which points on the grid with higher importance values have a higher probability of intersecting a random ray trajectory than points with lower importance values. Once generated, a set of random rays can be traced with respect to this geometric grid, and one or more intersection points (or grid triangles, etc.) of these rays with this grid can be determined 510. Each intersection point can correspond to a pixel location on the grid, can be determined using the distance of the traced ray, and this pixel can be used to determine lighting information for a current image to be generated. An image can then be rendered 512 or otherwise generated that uses colors, luminances, pixels, or other values from the texture pixels corresponding to the intersection points to determine lighting for assets in the scene. This approach allows for fast sampling of pixels for shading processes, for example, with increased probability of sampling for pixels with higher importance to lighting or shading processes determinations. More generally, this approach can also provide memory-efficient, high-performance, high-quality inverse transform sampling for one-, two-, and three-dimensional datasets. Inverse transform sampling can be beneficially used for tasks such as Monte Carlo ray tracing, as it can be used to redirect samples to “important” regions to greatly reduce noise and improve convergence, for example, by redirecting a uniform random distribution of locations within a range to a grid or set of samples based on the probability of each sample. In at least one embodiment, the PDF for Monte Carlo importance sampling can be derived by analyzing the partial derivative of the CDF geometry at the intersection location, as the derivative of the CDF is the PDF, as discussed in more detail elsewhere herein.

[0033] Various aspects of the embodiments discussed above can benefit from additional details, at least with respect to how these aspects are implemented for certain applications or use cases. For example, there can be multiple ways to represent the geometry of a given texture. For example, there can be instances in which a user wants to perform inverse transform sampling on a one-dimensional CDF. This CDF can be geometrically represented as a linear band geometry, where the depth encodes the domain of the CDF function, and the height encodes the range of the CDF function. To invert this CDF, this process can trace a ray with respect to this height field, for example, with the ray origin being a random height between 0 and 1, and the returned distance can encode the inverse-transformed sample. Since the CDF is the inverse derivative of the PDF, the slope of the height field at the intersection location can be returned as the probability of the sample occurring.

[0034] As another example, using the same one-dimensional case mentioned earlier, this CDF can be geometrically encoded by creating a planar geometry where the width of the ribbon segments encodes the inverse CDF. This process can then randomly trace a ray against this geometry and use the primitive ID to determine the corresponding sample and probability. For both cases, adjacent primitives can be merged together to save memory and improve performance by using gradient magnitude measurements. However, by representing this data using primitives that are compatible with ray traversal units, the search against this geometry can be sped up, further improving performance. These different geometric encoding strategies can be combined together to implement multi-dimensional inverse transform sampling. For example, the ray origin can be initialized so that one axis is set to a constant and the other two axes are driven using two random variables. The direction of this ray can be set to be orthogonal to the two random axes, pointing towards the grid used to represent the multiple CDFs.

[0035] A grid can be represented using a series of triangular ribbons, one ribbon assigned to each row in the grid to be sampled. These ribbons can run along the same axis as the earlier ray. Each ribbon can be subdivided into segments, one segment per column at most, and elevated to match the corresponding row "conditional" CDF. The height of these ribbons can be made to span the possible range of ray values along the second random axis, the first segment starting with a height matching the lowest possible random axis value and ending with a height matching the maximum possible random axis value, the elevation of the intermediate segments being proportional to their corresponding conditional CDF values. The width of these ribbons follows the inverse of a single row "marginal" CDF, the more probable rows having wider ribbons, and vice versa, and all ribbons together span the possible range of ray values along the first randomized axis. In this configuration, the aforementioned ray, once traced, hits the grid at some location.

[0036] When the aforementioned ray is traced against this grid, the first randomized ray axis can effectively select the row according to the intersecting ribbon ID, avoiding the first binary search. The second randomized ray axis can make the ray rise with the ribbon, causing the intersection distance to either grow or shrink. This hit distance effectively selects the column, avoiding the second binary search. The probability of the selected row can be found using the ribbon ID, and the probability of the selected column can be found using the slope of the ribbon along the ray direction axis.

[0037] As another example of multi-dimensional reformulation, a user can construct a planar geometry in which the width and height of a quadrilateral spanning two dimensions encodes inverse condition and marginal CDFs. This approach differs from the earlier reformulation in that distance is no longer used to encode the conditional CDF domain. The ray source can similarly be randomly modified to sample this planar geometry, where intersecting primitives and corresponding per-primitive data can be used to redirect samples. For three-dimensional importance sampling, the 2D quadrilateral mentioned earlier can be substituted for a 3D hexahedron, where the hexahedron can be sampled using an epsilon-length ray or hardware-accelerated point query, and from there the relevant per-primitive sample redirection data needed for the inverse transform sampling can be looked up.

[0038] For any of these or other such reformulations, the CDF grid can be simplified to improve traversal performance while preserving important CDF features. This can involve analyzing the CDF’s inflection points and selecting a subset of those inflection points that are most influential. In at least some embodiments, it can also be beneficial to transform this geometry to a range between 0 and 1 for all dimensions, as in at least some cases it has been found that this can improve sample inverse performance with tree-traversal units. Otherwise, proxy geometries with very different scales along different axes can cause internal nodes to grow due to boundary quantization, e.g., leading to unnecessary intersection tests.

[0039] As mentioned earlier, when integrating a function f, importance sampling can utilize a modified form of the Monte Carlo estimator in which the sampled value f(X i ) is divided by its corresponding sampling probability, as can be given by:

[0040]

[0041] This modified estimator allows for redirection of samples to more influential areas, where p(X i ) effectively serves as a correction factor for this redirection.

[0042] To use this modified estimator for texture importance sampling, a process can first determine and / or arrange the influence of each texel (or pixel, etc.) in a given image, which in at least one embodiment can be determined by converting the image from linear RGB to grayscale luminance L. This gives a single scalar value for each texel that can be used to estimate how much light a particular texel will emit. Such a process can consider texels with high luminance to be more important, and vice versa, by computing a joint probability distribution p(X,Y) from these luminance values, where the term “joint” is used to refer to the fact that both X and Y coordinates are used to look up the probability of a texel, as follows:

[0043]

[0044] Once this joint probability distribution is determined, the next step can be to further break these probabilities into marginal and conditional probabilities that can be used for importance sampling of rows and columns, respectively.

[0045] To importance sample a row of interest, a marginal probability p(Y) can be computed for each row. This marginal probability can describe the overall probability of a row and can allow for sampling of rows and columns separately, typically stored in the margins of the joint probability table, which can be given by, for example:

[0046]

[0047] Then, to importance sample a column within a previously sampled row, a conditional probability p(X|Y) can be computed for each texel within that row. The word “conditional” here means that this probability depends on the occurrence of another event. In this example, given that Y has been chosen, the probability of sampling X needs to be computed, as follows:

[0048]

[0049] Sample redirection can then be performed in at least some embodiments as appropriate. This can involve randomly selecting from a probability distribution to match the probability of selecting a sample to the probability previously assigned to that sample. This importance sampling process should be efficient in various implementations, as one process can acquire many of these samples in a given ray-tracer. This can be performed using a process such as inverse transform sampling, which can transform these probability distributions to implement an efficient sampling algorithm, as discussed elsewhere herein. More specifically, in at least some embodiments, the previous probability distribution function (PDF) can be converted to a cumulative distribution function (CDF). The cumulative distribution function evaluated at some location X returns the probability that a sample less than or equal to X has occurred. In this description, the PDF is denoted using lower case p, and the CDF is denoted using upper case P, which can be given as:

[0050]

[0051] Since CDFs can be used as running sums of the positive probability values, the values of these functions will generally only go up, never down. Thus, CDFs can be inverted. This ability to invert CDFs can be beneficial for tasks such as importance sampling, as this inversion process can map a uniform distribution into a distribution where highly influential samples are more likely to be sampled, and less influential samples are less likely to be sampled. In practice, this inversion can be performed numerically, as these PDFs can be discrete arrays derived from luminances, rather than from an analytically invertible equation. Fortunately, this numerical inversion can be done using, for example, a fast O(log(n)) binary search, as follows:

[0052]

[0053] In practice, a single marginal CDF P(Y) can be computed and stored, along with the conditional CDFs P(X|Y) for each row, and the original marginal, conditional, and joint PDFs discarded to safe memory. During the sampling process, two uniform random numbers e1 and e2 can be generated. From here, a binary search can be used to determine the first Y coordinate of the marginal CDF where P(Y) > e1. Using this Y coordinate, the conditional CDF can be selected for that Y, and a binary search again performed to determine the X coordinate where P(X|Y) > e2. Still, the probability of drawing that sample needs to be computed, as this probability can serve as a correction factor. Since these CDFs are running sums of the original PDFs, the original probability densities in X and Y can be computed by simple subtraction: p(X|Y) = P(X|Y) - P(X-1|Y) and p(Y) = P(Y) - P(Y-1). From there, the joint PDF p(X,Y) can be computed as p(X,Y) = p(X|Y)p(Y).

[0054] The inverse transform sampling method presented above has been shown to have near-optimal running time complexity of O(log(N) + log(M)), where N and M represent the texture dimensions. However, there are a few methods that can further optimize this method in order to gather more samples under the same performance budget, while also improving memory efficiency. More specifically, the implementation constants associated with inverse transform sampling can still be relatively high, especially for real-time and interactive applications. As mentioned previously, binary search is not optimal, as it adopts a memory access pattern that prevents prefetching data that can be accessed in the next iteration. Row and column accesses also depend on random numbers, making these accesses incoherent. To further improve performance and memory efficiency, various techniques presented in this paper can reduce the size of these implementation constants by simplifying the CDFs, while also using representations that are compatible with ray tracing hardware to improve search performance.

[0055] Within the CDF, many of the neighboring probabilities can have similar values. For example, if the texture contains a smooth gradient of the sky, the brightness between neighboring pixels will be similar, and therefore, the neighboring probabilities will also be similar. The similarity of neighboring pixels can then be ordered based at least in part on their proximity of their corresponding probabilities. Using this measure of similarity, it is possible to preferentially merge highly similar pixels, and it can be attempted to avoid merging neighboring pixels with very different probabilities. Structurally, this measure of similarity allows for a tradeoff between importance sampling accuracy on the one hand and data locality on the other hand. In practice, this merging can not always be straightforward. By merging neighboring values together, the CDF goes from being structured to being unstructured, and it can be somewhat difficult to search efficiently in these unstructured data. For one such strategy, the CDF can be interpreted geometrically as a height field. For each address in the CDF array, the associated probability can be interpreted as a "height" ranging from 0 to 1. From this, a geometric grid can be constructed to represent this height field using linear strips.

[0056] Using this unstructured geometric reformulation, the search for the inverse transform sample can be implemented by tracing one or more rays. To sample the CDF geometry, a uniform random number ξ ∈ (0, 1) can be generated, but instead of using a binary search to search for this random number, the random number can be used to control the height of a given ray. This ray can be traced into the CDF height field, where a traversal is used to facilitate the search, as follows:

[0057]

[0058] o = (0, ε, 0)

[0059]

[0060] By randomizing the y-offset of the origin and aligning the ray to the x-axis, the ray can be traced into the height field with a positive slope, such that the intersection distance t represents the inverse x-coordinate.

[0061] To compute the corresponding sample probability "correction factor" for the Monte Carlo estimator, the derivative of the CDF can be computed at the sample location, since the derivative of the CDF is the PDF to be sampled. Since the height field geometry consists of linear segments, this derivative can be determined by computing the slope of the hit linear segment. So far, it can be necessary to trace a ray to facilitate the first search over the marginal CDF P(Y), and then trace a second ray to search over the conditional CDF P(X|Y). However, with some modifications, it is possible to search both CDFs simultaneously using only one ray.

[0062] Instead of representing each CDF geometry as a separate band geometry, the available third dimension can be utilized along with the hit group records associated with the bands to simultaneously search both the marginal distribution and the conditional distribution. As previously described, each conditional CDF can use a simplified band geometry representation, where the height of the band (e.g., Y in this example) can be used to encode the probability of a given row of columns (e.g., X in this example). In at least one embodiment, the inverse of the marginal CDF can be obtained by varying the thickness and position of those bands along the third dimension (e.g., Z in this example), storing the corresponding “projected” row identifier (ID) and row probability in the band hit group records that can be obtained from the most recent hit procedure.

[0063] By using the band width to represent the marginal probability and placing these bands side by side, the marginal CDF height can be projected onto the Z axis, effectively encoding the inverse of the marginal CDF. To recover the “projected” address of each marginal CDF segment, the hit record data corresponding to the hit geometry can be determined. Rows with a marginal probability of 0 (indicating that the brightness of that row is zero) will get a zero thickness, and therefore will not be hit by a ray at all. Conversely, rows with a high marginal probability have a very wide thickness, making it more likely that these bands will be hit.

[0064] When tracing a ray against this combined geometry, the direction of the ray can be towards the positive inclination of the CDF, but the origin of the ray can be set to o = (0, e1, e2), where e2 is a random number used to sample the marginal distribution, and e1 is a random number used to sample the conditional CDF. From this ray, the sampled y coordinate can be obtained from the marginal distribution through the row ID associated with the hit band, and then the x coordinate is obtained as before through the returned hit distance t.

[0065] In at least some embodiments, it is important to reduce the implementation constant while preserving the variance reduction by simplifying the condition and marginal CDFs. Rather than merging adjacent probabilities together in a local range, the method according to various embodiments can collect a subset of texels deemed to be at least minimally important in a global range. These important texels can be referred to as control points, as these control points can be used to generate the CDF geometry. One control point can be constructed for each texel in a given texture, and these control points are inserted into a list. For each control point, the value of the conditional CDF at that coordinate can be stored, as well as the first and second order partial derivatives of the conditional CDF with respect to the row direction X. The array of control points can then be sorted in descending order by their second order derivatives. Sorting by second order derivative can cause "spike" control points with high influence to appear early in the list. Control points with smaller second order derivative values will appear later in the list, indicating that the adjacent texels of these control points are at least somewhat similar, and these control points can be discarded to simplify the geometric representation. This sorted list can be truncated, such that all control points except for N control points are discarded.

[0066] These resulting control points represent the row-wise CDFs, each with a minimum value of 0 and a maximum value of 1. By discarding control points globally, it is possible that a row CDF can not have a control point associated with it. To account for this possibility, the x-coordinates of all control points can first be incremented by 1, such that these control points are centered on the vertices of the texel grid, rather than the cells. Next, each row can be forced to be represented by inserting a control point at x=0 with a probability of P(0|Y)=0, and at x=n with a control point if they do not exist, using a function value of P(n|Y)=1. After sorting and truncating the control point list by second order derivative, these control points can be reordered, first by their x-coordinates, and then by their y-coordinates, the latter being a stable sort to preserve the order in x. These operations can effectively generate an unstructured list of CDFs sorted by row, each row represented by at least two control points.

[0067] Such a process can result in the CDF being simplified to points that the control points can use to generate a geometry representation, such as a geometry mesh. Rays can be traced against this geometry and traversal performance can be improved by using hardware acceleration, such as using real-time ray tracing acceleration, such as through NVIDIA’s RT Core as part of its RTX ray tracing technology. At least in some implementations, the ray tracing core is a tree traversal unit that can perform fast and efficient bounding volume hierarchy (BVH) traversal in, for example, three dimensions. The BVH can be constructed on 3D triangles of the color bands, rather than 2D lines, for ray tracing hardware. For each unstructured CDF row, a band-like geometry shape can be generated as described herein such that consecutive CDF control points in a row are connected by triangular planar quadrilaterals. As previously described, the respective marginal CDF values for each row can be used to adjust the thickness and position of these bands in three dimensions in a row-by-row fashion.

[0068] Various additional optimizations can also be used. For example, the dimensions of the CDF geometry can be normalized, as this can result in more consistent bounding box sizes and greatly improved traversal performance, such as 45-50% improvement in the datasets used for testing. Correction for this normalized scale can be performed after sampling. Finally, if it is detected that adjacent rows each contain only two control points, performance can be further improved by merging these adjacent rows together, such as 20-30% improvement on these datasets. Thus, a band can span multiple rows. The rows spanned by a band can be uploaded along with the triangle barycentric coordinates used to determine which row in the functional zone of a ray sample through the hit group record of the band geometry.

[0069] As discussed above, the various methods presented herein are lightweight enough to be performed in real-time on a client device such as a personal computer or game console. Such processing can be performed on content generated on the client device or received from an external source, such as streaming content received over at least one network. The source can be any appropriate source, such as a game host, a streaming media provider, a third-party content provider, or other client devices and other such options. In some cases, the processing and / or rendering of the content can be performed by one of these other devices, systems, or entities and then provided to the client device (or another such recipient) for presentation or other similar use.

[0070] As an example, Figure 6An example network configuration 600 that can be used to provide, generate, modify, encode, and / or transmit content is shown. In at least one embodiment, a client device 602 can generate or receive content for a session using components of a content application 604 on the client device 602 and data stored locally on the client device. In at least one embodiment, a content application 624 (e.g., an image generation or editing application) executing on a content server 620 (e.g., a cloud server or edge server) can initiate a session associated with at least client device 602, as can utilize a session manager and user data stored in a user database 634, and can cause content 632 to be determined by a content manager 626. An image content application 630 can obtain image, asset, and / or texture data for a scene or environment and work with a rendering engine 628 or other such components to generate an image-based representation of the scene or environment. At least a portion of this content can be transmitted to the client device 602 using an appropriate transport manager 622 for delivery by download, streaming, or another such transport channel. An encoder can be used to encode and / or compress at least some of this data prior to transmission to the client device 602. In at least one embodiment, content 632 can include video or image data for a scene. In at least one embodiment, a client device 602 receiving such content can provide the content to a corresponding content application 604, which can also or instead include a graphical user interface 610, a rendering engine 612, or an image generation application 614 or process for generating, modifying, or rendering image data received to or generated on the client device 602. A decoder can also be used to decode data received over a network 640 for presentation by the client device 602, such as presentation of image or video content by a display 606 and audio such as sound and music by at least one audio playback device 608 such as a speaker or headphones. In at least one embodiment, at least some of this content can have been stored on the client device 602, rendered on the client device 602, or accessible to the client device 602 such that at least this portion of the content does not require transmission over the network 640, such as in the case that the content can have been previously downloaded or stored locally on a hard drive or optical disc. In at least one embodiment, a transmission mechanism such as a data stream can be used to transmit this content from the server 620 or content database 634 to the client device 602. In at least one embodiment, at least a portion of this content can be obtained or streamed from another source such as a third-party content service 660, which can also include a content application 662 for generating or providing content.In at least one embodiment, portions of functionality can be performed using multiple computing devices or multiple processors within one or more computing devices such as a combination of CPUs and GPUs.

[0071] In this example, the client devices can include any appropriate computing devices such as desktop computers, notebook computers, set-top boxes, streaming devices, game consoles, smartphones, tablet computers, VR headsets, AR eyewear, wearable computers, or smart televisions. Each client device can submit requests across at least one wired or wireless network which can include the Internet, an Ethernet network, a local area network (LAN), or a cellular network among other such options. In this example, the requests can be submitted to an address associated with a cloud provider which can operate or control one or more electronic resources under a cloud provider environment such as can include a data center or a server farm. In at least one embodiment, the requests can be received or processed by at least one edge server located at an edge of a network and outside of at least one security layer associated with the cloud provider environment. In this way, latency can be reduced by enabling client devices to interact with servers that are closer in distance while also improving security of resources in the cloud provider environment.

[0072] In at least one embodiment, such a system can be used to perform graphics rendering operations. In other embodiments, such a system can be used for other purposes such as to provide image or video content for testing or validating autonomous machine applications or to perform deep learning operations. In at least one embodiment, such a system can be implemented using edge devices or can incorporate one or more virtual machines (VMs). In at least one embodiment, such a system can be implemented at least partially in a data center or at least partially using cloud computing resources.

[0073] Inference and training logic

[0074] Figure 7A Inference and / or training logic 715 are used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 715 can be used in Figure 7A and / or Figure 7B Details regarding inference and / or training logic 715 are provided below in conjunction with FIGS. 7A and / or 7B.

[0075] In at least one embodiment, inference and / or training logic 715 can include, without limitation, code and / or data storage 701 for storing forward and / or output weights and / or input / output data, and / or other parameters of neurons or layers of a neural network configured in aspects of one or more embodiments that are trained and / or used for inferencing. In at least one embodiment, training logic 715 can include or be coupled to code and / or data storage 701 for storing graph code or other software to control timing and / or order, where weight and / or other parameter information is loaded to configure logic, including integer and / or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which that code corresponds. In at least one embodiment, code and / or data storage 701 stores weight parameters and / or input / output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inferencing using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data storage 701 can be included with other on-chip or off-chip data storage, including a processor’s Ll, L2, or L3 cache or system memory.

[0076] In at least one embodiment, any portion of code and / or data storage 701 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 701 can be cache memory, dynamic random accessible memory (“DRAM”), static random accessible memory (“SRAM”), non-volatile memory (such as flash memory), or other storage. In at least one embodiment, a choice of whether code and / or data storage 701 is internal or external to a processor, e.g., or comprised of DRAM, SRAM, flash or some other storage type, can depend on available storage space on-chip or off-chip, latency requirements of training and / or inferencing functions being performed, batch size of data used in inferencing and / or training of a neural network, or some combination of these factors.

[0077] In at least one embodiment, inference and / or training logic 715 can include, without limitation, code and / or data storage 705 for storing backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, code and / or data storage 705 stores weight parameters and / or input / output data of each layer of a neural network trained or used in conjunction with one or more embodiments during backward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, training logic 715 can include or be coupled to code and / or data storage 705 for storing graph code or other software to control timing and / or order, where weight and / or other parameter information is loaded to configure logic, including integer and / or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which that code corresponds. In at least one embodiment, any portion of code and / or data storage 705 can be included with other on-chip or off-chip data storage, including a processor’s Ll, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and / or data storage 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 705 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., Flash memory), or other storage. In at least one embodiment, a choice of whether code and / or data storage 705 is internal or external to a processor, e.g., whether it is made up of DRAM, SRAM, Flash memory, or some other storage type, depends on whether available storage is on-chip or off-chip, latency requirements of training and / or inferencing functions being performed, batch size of data being used in inference and / or training of a neural network, or some combination of these factors.

[0078] In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 can be separate storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 can be the same storage structure. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 can be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of code and / or data storage 701 and code and / or data storage 705 can be included with other on-chip or off-chip data storage, including a processor’s Ll, L2, or L3 cache or system memory.

[0079] In at least one embodiment, inference and / or training logic 715 can include, without limitation, one or more arithmetic logic units (“ALUs”) 710 (including integer and / or floating point units) for performing logical and / or mathematical operations based, at least in part, on training and / or inference code (e.g., graph code) or instructions therefrom. In at least one embodiment, results of such operations can produce activations (e.g., output values from layers or neurons within a neural network) stored in activation storage 720 that are functions of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, activations stored in activation storage 720 are generated by ALUs 710 executing linear algebraic and / or matrix-based mathematics in response to executing instructions or other code, where weight values stored in code and / or data storage 705 and / or code and / or data storage 701 are used as operands along with other values such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which can be stored in code and / or data storage 705 or code and / or data storage 701 or other on-chip or off-chip storage.

[0080] In at least one embodiment, one or more ALUs 710 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment one or more ALUs 710 can be external to a processor or other hardware logic device or circuit using them (e.g., a co-processor). In at least one embodiment, one or more ALUs 710 can be included within execution units of a processor, or otherwise included in a group of ALUs accessible by execution units of a processor, which can be within a same processor or distributed between different types of processors (e.g., central processing units, graphics processing units, fixed function units, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 can be on a same processor or other hardware logic device or circuit, while in another embodiment they can be in different processors or other hardware logic devices or circuits or some combination of same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 720 can be included with other on-chip or off-chip data storage, including a processor’s LI, L2, or L3 cache or system memory. Moreover, inference and / or training code can be stored with other code accessible to a processor or other hardware logic or circuitry, and can be fetched and / or processed using fetch, decode, schedule, execute, exit, and / or other logic circuitry of a processor.

[0081] In at least one embodiment, the active memory 720 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 720 may be wholly or partially located inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory 720 is internal to or external to the processor may depend on the available on-chip or off-chip storage, the latency requirements for training and / or inference functions, the batch size of data used in inference and / or training the neural network, or some combination of these factors. For example, it may include DRAM, SRAM, flash memory, or other memory types. In at least one embodiment, Figure 7A The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as those from Google. Processing unit, from Graphcore TM Inference processing unit (IPU) or from Intel Corp. (e.g., "LakeCrest") processor. In at least one embodiment, Figure 7A The inference and / or training logic 715 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware such as field programmable gate array (“FPGA”)

[0082] Figure 7B Inference and / or training logic 715 according to at least one or more embodiments is illustrated. In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, hardware logic, wherein computational resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM Inference processing unit (IPU) or from Intel Corp. (e.g., "LakeCrest") processor. In at least one embodiment, Figure 7BThe inference and / or training logic 715, as shown in FIG. 7B, can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware or other hardware, such as field programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 715 includes, without limitation, code and / or data storage 701 and code and / or data storage 705, which can be used to store code (e.g., graph code), weight values and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In Figure 7B In at least one embodiment, each of code and / or data storage 701 and code and / or data storage 705 is associated with a dedicated computing resource, such as computing hardware 702 and computing hardware 706, respectively. In at least one embodiment, each of computing hardware 702 and computing hardware 706 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) only on information stored in code and / or data storage 701 and code and / or data storage 705, respectively, the results of which are stored in activation storage 720.

[0083] In at least one embodiment, each of code and / or data storage 701 and 705 and corresponding computing hardware 702 and 706, respectively, correspond to different layers of a neural network, such that activations resulting from one “storage / computing pair 701 / 702” of code and / or data storage 701 and computing hardware 702 are provided as input to the next “storage / computing pair 705 / 706” of code and / or data storage 705 and computing hardware 706 in order to reflect the conceptual organization of a neural network. In at least one embodiment, each storage / computing pair 701 / 702 and 705 / 706 can correspond to more than one neural network layer. In at least one embodiment, additional storage / computing pairs (not shown) can be included in inference and / or training logic 715 after or in parallel with storage / computing pairs 701 / 702 and 705 / 706.

[0084] Data Center

[0085] Figure 8 An example data center 800 that can use at least one embodiment is shown. In at least one embodiment, data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830 and an application layer 840.

[0086] In at least one embodiment, as Figure 8As shown, the data center infrastructure layer 810 can include a resource orchestrator 812, grouped computing resources 814, and node computing resources (“node C.R.s”) 816(1)-816(N), where “N” represents any positive integer. In at least one embodiment, node C.R.s 816(1)-816(N) can include, but are not limited to, any number of central processing units (“CPUs” or “processors”), including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc., memory devices (e.g., dynamic random access memory), storage devices (e.g., solid state or disk drives), network input / output (“NWI / O”) devices, network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more node C.R.s of node C.R.s 816(1)-816(N) can be a server having one or more of the above-described computing resources.

[0087] In at least one embodiment, grouped computing resources 814 can include individual groups of node C.R.s housed within one or more racks (not shown), or housed within a number of racks (also not shown) within various geographic locations. Individual groups of node C.R.s within grouped computing resources 814 can include groups of computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s including CPUs or processors can be grouped within one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, one or more racks can also include any number of power modules, cooling modules, and network switches, in any combination.

[0088] In at least one embodiment, resource orchestrator 812 can configure or otherwise control one or more node C.R.s 816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, resource orchestrator 812 can include a software design infrastructure (“SDI”) management entity for data center 800. In at least one embodiment, resource orchestrator 108 can comprise hardware, software, or some combination thereof.

[0089] In at least one embodiment, as Figure 8As shown, the framework layer 820 includes a job scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, the framework layer 820 can include a framework that supports software layer 830 software 832 and / or one or more application programs 842 of application layer 840. In at least one embodiment, software 832 or application programs 842 can include web-based service software or application programs, respectively, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 820 can be, but is not limited to, a type of free and open-source software web application framework such as Apache Spark™ (hereinafter “Spark”) that can utilize the distributed file system 828 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 832 can include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 800. In at least one embodiment, the configuration manager 824 can be capable of configuring different layers, such as the software layer 830 and the framework layer 820 including Spark and the distributed file system 828 for supporting large-scale data processing. In at least one embodiment, the resource manager 826 can be capable of managing clustered or grouped computing resources mapped to or allocated for supporting the distributed file system 828 and the job scheduler 822. In at least one embodiment, the clustered or grouped computing resources can include grouped computing resources 814 on the data center infrastructure layer 810. In at least one embodiment, the resource manager 826 can coordinate with the resource orchestrator 812 to manage these mapped or allocated computing resources.

[0090] In at least one embodiment, software 832 included in the software layer 830 can include software used by at least a portion of the node C.R.s 816(1)-816(N), the grouped computing resources 814, and / or the distributed file system 828 of the framework layer 820. One or more types of software can include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0091] In at least one embodiment, one or more application programs 842 included in application layer 840 can include one or more types of application programs used by at least portions of node C.R.s 816(1)-816(N), grouped computing resources 814, and / or distributed file system 828 of framework layer 820. One or more types of application programs can include, but are not limited to, any number and type of genomics applications, cognitive computing and machine learning applications including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0092] In at least one embodiment, any of configuration manager 824, resource manager 826, and resource orchestrator 812 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible fashion. In at least one embodiment, self-modification actions can mitigate poor configuration decisions made by data center operators of data center 800 and can avoid underutilization and / or poor performing portions of a data center.

[0093] In at least one embodiment, data center 800 can include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information in accordance with one or more embodiments described herein. For example, in at least one embodiment, a machine learning model can be trained by computing weight parameters according to a neural network architecture using software and computing resources described above with respect to data center 800. In at least one embodiment, using weight parameters computed by one or more training techniques described herein, a trained machine learning model corresponding to one or more neural networks can be used to infer or predict information using resources described above with respect to data center 800.

[0094] In at least one embodiment, a data center can use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inference using resources described above. Moreover, one or more software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference such as image recognition, speech recognition, or other artificial intelligence services.

[0095] Inference and / or training logic 715 are used to perform inferencing and / or training operations associated with one or more embodiments. Inference and / or training logic 715 are used to perform inferencing and / or training operations associated with one or more embodiments described herein using one or more examples of a neural network, of a support Figure 7A and / or Figure 7BDetails are provided regarding the inference and / or training logic 715. In at least one embodiment, the inference and / or training logic 715 can be implemented in the system. Figure 8 Used in systems for reasoning or predicting operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0096] Such components can be used to render images using ray-tracing-based importance sampling, which can be accelerated by hardware.

[0097] Computer System

[0098] Figure 9 This is a block diagram illustrating an exemplary computer system according to at least one embodiment. The exemplary computer system may be a system of interconnected devices and components, a system-on-a-chip (SoC), or some combination thereof formed with a processor, which may include an execution unit to execute instructions. In at least one embodiment, according to this disclosure, such as the embodiments described herein, computer system 900 may include, but is not limited to, components such as processor 902, whose execution unit includes logic to execute algorithms for process data. In at least one embodiment, computer system 900 may include a processor, such as one available from Intel Corporation of Santa Clara, California. Processor family, Xeon TM , XScale TM and / or StrongARM TM , Core TM or Nervana TM A microprocessor may be used, although other systems (including PCs, engineering workstations, set-top boxes, etc.) with other microprocessors may also be used. In at least one embodiment, computer system 900 may execute a version of the Windows operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (such as UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0099] Embodiments can be used in other devices such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, embedded applications can include a microcontroller, a digital signal processor ("DSP"), a system on a chip, a network computer ("NetPC"), a set-top box, a network hub, a wide area

[0100] In at least one embodiment, computer system 900 can include, but is not limited to, processor 902, which can include, but is not limited to, one or more execution units 908 to perform machine learning model training and / or inferencing according to techniques described herein. In at least one embodiment, computer system 900 is a single processor desktop or server system, but in another embodiment, computer system 900 can be a multiprocessor system. In at least one embodiment, processor 902 can include, but is not limited to, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combo of instruction sets, or any other processor device, such as a digital signal processor. In at least one embodiment, processor 902 can be coupled to a processor bus 910 that can transmit data signals between processor 902 and other components in computer system 900.

[0101] In at least one embodiment, processor 902 can include, but is not limited to, level 1 ("Ll") internal cache memory ("cache") 904. In at least one embodiment, processor 902 can have a single -level internal cache or multi-level internal cache. In at least one embodiment, cache memory can reside in the processor 902's external. Other embodiments can include a combination of internal and external caches based on specific implementation and requirements. In at least one embodiment, register file 906 can store different types of data within various registers including, but not limited to, integer registers, floating point registers, status registers, and instruction pointer registers.

[0102] In at least one embodiment, execution unit 908 includes, without limitation, logic to perform integer and floating-point operations, including bit- wide operations. In at least one embodiment, processor 902 can also include a microcode (“ucode”) read only memory (“ROM”) that stores microcode for certain macro instructions. In at least one embodiment, execution unit 908 can also include logic to handle a packed instruction set (e.g., 64-bit data instructions). In at least one embodiment, by including this type of instruction set within a general-purpose

[0103] In at least one embodiment, execution unit 908 can also be used in a microcontroller, embedded processor, graphics device, DSP, and other types of logic circuits. In at least one embodiment, computer system 900 can include, without limitation, memory 920. In at least one embodiment, memory 920 can be implemented as a Dynamic Random Access Memory (“DRAM”) device, a Static Random Access Memory (“SRAM”) device, a flash memory device, or other memory device. In at least one embodiment, memory 920 can store instruction(s) 919 and / or data 921 represented by data signals that can be executed by processor 902.

[0104] In at least one embodiment, a system logic chip can be coupled to processor bus 910 and memory 920. In at least one embodiment, system logic chip can include, without limitation, a memory controller hub (“MCH”) 916 and processor 902 can communicate with MCH 916 via processor bus 910. In at least one embodiment, MCH 916 can provide a high bandwidth memory path 918 to memory 920 for instruction and data storage and for storage of graphics commands, data, and textures. In at least one embodiment, MCH 916 can also initiate data

[0105] In at least one embodiment, computer system 900 can use system I / O 922, which is a proprietary hub interface bus to couple MCH 916 to I / O controller hub (“ICH”) 930. In at least one embodiment, ICH 930 can provide a direct connection to some I / O devices and indirectly through the local I / O bus. In at least one embodiment, local I / O bus can include, without limitation, a high-speed I / O bus for connecting peripherals to memory 920, chipset, and processor 902. Examples can include, without limitation, audio controller 929, firmware hub (“Flash BIOS”) 928, wireless transceiver 926, data storage 924, legacy I / O controller 923 containing user input and keyboard interfaces, serial expansion port 927 (e.g., Universal Serial Bus (“USB”) port), and network controller 934. Data storage 924 can include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0106] In at least one embodiment, Figure 9 A system is shown that includes interconnected hardware devices or “chips,” while in other embodiments, Figure 9An exemplary system on a chip (SoC) can be shown. In at least one embodiment, devices can be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of computer system 900 are interconnected using a compute express link (CXL) interconnect.

[0107] Inference and / or training logic 715 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 715 are provided below in conjunction with FIGS. 7L and / or 7M. Figure 7A and / or Figure 7B Details regarding inference and / or training logic 715 are provided below in conjunction with FIGS. 7L and / or 7M. In at least one embodiment, inference and / or training logic 715 can be used in a system that uses neural network training operations, neural network functions and / or architectures, or neural network use cases described herein to perform inferencing or prediction operations based, at least in part, on weight parameters calculated using neural network training operations. Figure 9 Details regarding inference and / or training logic 715 are provided below in conjunction with FIGS. 7L and / or 7M. In at least one embodiment, inference and / or training logic 715 can be used in a system that uses neural network training operations, neural network functions and / or architectures, or neural network use cases described herein to perform inferencing or prediction operations based, at least in part, on weight parameters calculated using neural network training operations.

[0108] Such components can be used to render images with ray-tracing based importance sampling, which can be accelerated by hardware.

[0109] Figure 10 is a block diagram illustrating an electronic device 1000 for utilizing a processor 1010, in accordance with at least one embodiment. In at least one embodiment, electronic device 1000 can be, for example and without limitation, a laptop, a tower server, a rack server, a blade server, a laptop computer, a desktop computer, a tablet computer, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0110] In at least one embodiment, system 1000 can include, without limitation, a processor 1010 communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices. In at least one embodiment, processor 1010 is coupled using a bus or interface, such as an 1C bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advanced Technology Attachment (“SATA”) bus, a Universal Serial Bus (“USB”) (versions 1, 2, 3), or a Universal Asynchronous Receiver / Transmitter (“UART”) bus. In at least one embodiment, processor 1010 is coupled to a memory 1020, which can include, without limitation, a volatile memory, a non-volatile memory, a cache, a buffer, a flash memory, a register, or any combination thereof. Figure 10 In at least one embodiment, system 1000 is illustrated as including hardware devices or “chips” that are interconnected instead of, or in addition to, a bus. In at least one embodiment, system 1000 includes, without limitation, a processor 1010, a memory 1020, a storage 1030, an input / output (I / O) circuit 1040, a power supply circuit 1050, a radio frequency (RF) circuit 1060, and an external interface circuit 1070. Figure 10 An exemplary system on a chip (SoC) can be shown. In at least one embodiment, Figure 10 In at least one embodiment, devices shown in FIG. 1 can be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, Figure 10One or more components are interconnected using Computational Fast Link (CXL) interconnects.

[0111] In at least one embodiment, Figure 10 It may include a display 1024, a touch screen 1025, a touchpad 1030, a near field communication unit (“NFC”) 1045, a sensor hub 1040, a thermal sensor 1046, a fast chipset (“EC”) 1035, a trusted platform module (“TPM”) 1038, a BIOS / firmware / flash (“BIOS, FW Flash”) 1022, a DSP 1060, a drive 1020 (e.g., a solid-state drive (“SSD”) or a hard disk drive (“HDD”)), a wireless local area network unit (“WLAN”) 1050, a Bluetooth unit 1052, a wireless wide area network unit (“WWAN”) 1056, a global positioning system (GPS) 1055, a camera (“USB 3.0 camera”) 1054 (e.g., a USB 3.0 camera) and / or a low-power double data rate (“LPDDR”) memory unit (“LPDDR3”) 1015 implemented in, for example, the LPDDR3 standard. These components can each be implemented in any suitable way.

[0112] In at least one embodiment, other components may be communicatively coupled to processor 1010 via the components described above. In at least one embodiment, accelerometer 1041, ambient light sensor (“ALS”) 1042, compass 1043, and gyroscope 1044 may be communicatively coupled to sensor hub 1040. In at least one embodiment, thermal sensor 1039, fan 1037, keyboard 1036, and touchpad 1030 may be communicatively coupled to EC 1035. In at least one embodiment, speaker 1063, earphone 1064, and microphone (“mic”) 1065 may be communicatively coupled to audio unit (“audio codec and Class D amplifier”) 1062, which in turn may be communicatively coupled to DSP 1060. In at least one embodiment, audio unit 1062 may include, for example, but not limited to, audio encoder / decoder (“codec”) and Class D amplifier. In at least one embodiment, SIM card (“SIM”) 1057 may be communicatively coupled to WWAN unit 1056. In at least one embodiment, components such as WLAN unit 1050, Bluetooth unit 1052, and WWAN unit 1056 can be implemented as next-generation form factor (NGFF).

[0113] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. The following is combined with... Figure 7A and / or Figure 7BDetails regarding the inference and / or training logic 715 are provided. In at least one embodiment, inference and / or training logic 715 can be used in the system to infer or predict operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein. Figure 10 operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0114] Such components can be used to render images with ray-tracing based importance sampling, which can be accelerated by hardware.

[0115] Figure 11 is a block diagram of a processing system in accordance with at least one embodiment. In at least one embodiment, system 1100 includes one or more processor(s) 1102 and one or more graphics processor(s) 1108, and can be a single processor desktop system, a multiprocessor workstation system, or a server system having many processor(s) 1102 or processor core(s) 1107. In at least one embodiment, system 1100 is a processing platform incorporated within a system on a chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices.

[0116] In at least one embodiment, system 1100 can include or be coupled to a graphics processing unit (GPU) or GPUs 1108. In at least one embodiment, system 1100 can include, couple to, or be part of a server system, mobile device, a handheld device, or a desktop computer. In at least one embodiment, system 1100 is a mobile phone, a smart phone, a tablet, or a phablet. In at least one embodiment, system 1100 is a gaming console, a media console, a handheld game console, or an online game console. In at least one embodiment, system 1100 includes, couples to, or is part of a wearable device, such as a smart watch, smart glasses, a virtual reality device, or an augmented reality device.

[0117] In at least one embodiment, one or more processors 1102 each include one or more processor cores 1107 to process instructions which, when executed, perform operations for system and user software. In at least one embodiment, each of the one or more processor cores 1107 is configured to process a specific instruction set 1109. In at least one embodiment, instruction set 1109 can facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computing via a Very Long Instruction Word (VLIW). In at least one embodiment, processor cores 1107 can each process a different instruction set 1109, which can include instructions to facilitate emulation of other instruction sets. In at least one embodiment, processor core 1107 can include other processing devices, such as a digital signal processor (DSP).

[0118] In at least one embodiment, processor 1102 includes cache memory 1104. In at least one embodiment, processor 1102 can have single-level or multi-level internal caches. In at least one embodiment, cache memory is shared among multiple components of processor 1102. In at least one embodiment, processor 1102 also uses an external cache, which can be shared by processor cores 1107, such as a third level (L3) cache or last level cache (LLC) (not shown), which can be shared between processor cores 1107 using known cache coherency techniques. In at least one embodiment, processor 1102 additionally includes a register file 1106 that can include different types of registers such as integer registers, floating point registers, status registers, and instruction pointer registers.

[0119] In at least one embodiment, one or more processors 1102 are coupled with one or more interface buses 1110 for passing control information and receiving data signals between processor 1102 and other components of system 1100. In at least one embodiment, interface bus 1110 can be versions of the Peripheral Component Interconnect (PCI) bus, including PCI, PCI Extended (PCI-X), and PCI Express (PCIe). In at least one embodiment, interface bus 1110 is not a PCI bus implementation, and can include one or more bus implementations including, but not limited to, Industry Standard Architecture (ISA), Micro Channel Architecture (MCA), Digital Electronic Bus (DEB), MicroBlaze, Advanced eXtensible Interface (AXI), Advanced Microcontroller Bus Architecture (AMBA), and / or any other bus implementation that provides for communication to one or more processors 1102.

[0120] In at least one embodiment, memory devices 1120 can be Dynamic Random Access Memory (DRAM) devices, Static Random Access Memory (SRAM) devices, flash memory devices, and / or other memory devices. In at least one embodiment, memory devices 1120 can be used as main memory for processor 1102, to store instructions and data used by the one or more processors 1102 for execution. In at least one embodiment, memory controller 1116 is used to facilitate communication between one or more processors 1102 and memory devices 1120. In at least one embodiment, one or more memory devices 1120 can be used for storage of data and / or instructions for one or more processors 1102. In at least one embodiment, memory controller 1116 can be a separate component from processor 1102, or memory controller 1116 can be integrated into processor 1102. In at least one embodiment, memory controller 1116 can be implemented as a set of instructions executed by one or more processors 1102 or implemented as hardware, or implemented as a combination of instructions and hardware.

[0121] In at least one embodiment, platform controller hub 1130 enables peripherals coupled to bridge 1122 to interact with a processor and / or each other over high-speed I / O buses 1120 and 1110. In at least one embodiment, I / O peripherals include, but are not limited to, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, a touch sensor 1125, a data storage device 1124 (e.g., solid-state drive (SSD), floppy drive, optical drive, etc.). In at least one embodiment, data storage device 1124 can communicate via a storage interface (e.g., SATA) or via a peripheral bus, such as a Peripheral Component Interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, touch sensor 1125 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, wireless transceiver 1126 can be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, firmware interface 1128 enables communication with system firmware, and can be, for example, a unified extensible firmware interface (UEFI). In at least one embodiment, network controller 1134 can enable network connectivity to one or more private or public networks. In at least one embodiment, a high-performance network controller (not shown) is coupled to interface bus 1110. In at least one embodiment, audio controller 1146 is a multi-channel high definition audio controller. In at least one embodiment, processing system 1100 includes an optional legacy I / O controller 1140 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to system 1100. In at least one embodiment, platform controller hub 1130 can also connect to one or more Universal Serial Bus (USB) controllers 1142 that connect into input devices such as keyboard and mouse 1143 combination, camera 1144, or other USB input devices.

[0122] In at least one embodiment, memory controller 1116 and instances of platform controller hub 1130 can be integrated into a discrete external graphics processor, such as external graphics processor 1112. In at least one embodiment, platform controller hub 1130 and / or memory controller 1116 can be external to one or more processor(s) 1102. For example, in at least one embodiment, system 1100 can include an external memory controller 1116 and platform controller hub 1130, which can be configured as a memory controller hub and a peripheral controller hub in a system-on-a-chip (SoC) that can communicate with processor(s) 1102.

[0123] Inference and / or training logic 715 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 715 are provided below in conjunction with FIGS. 7L and / or 7M. Figure 7A and / or Figure 7BDetails regarding the inference and / or training logic 715 are provided. In at least one embodiment, some or all of inference and / or training logic 715 can be incorporated with graphics processor 1100. For example, in at least one embodiment, the training and / or inference techniques described herein can use one or more ALUs embodied in a graphics processor. Further, in at least one embodiment, the inference and / or training operations described herein can be accomplished with logic other than that shown. Figure 7A or Figure 7B In at least one embodiment, weight parameters can be stored in on-chip or off-chip memory and / or registers (shown or not) that configure ALUs of a graphics processor to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0124] Such components can be used to render images with ray-tracing based importance sampling, which can be accelerated by hardware.

[0125] Figure 12 is a block diagram of a processor 1200 having one or more processor cores 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208, according to at least one embodiment. In at least one embodiment, processor 1200 can include additional cores, up to and including an additional core 1202N represented by a dashed lined in at least one embodiment. In at least one embodiment, each processor core 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core can also include access to one or more shared cache units 1206.

[0126] In at least one embodiment, internal cache units 1204A-1204N and shared cache unit 1206 represent a cache memory hierarchy within processor 1200. In at least one embodiment, cache memory units 1204A-1204N can include at least one level of cache memory such as level one (LI), level two (L2), level three (L3), level four (L4), or other levels of cache, within each processor core 1202A-1202N and shared level cache(s) 1206, where the highest level of cache memory prior to main memory is referred to as the LLC. In at least one embodiment, cache coherence logic maintains coherency for the various cache units 1206 and 1204A-1204N.

[0127] In at least one embodiment, processor 1200 also includes a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, one or more bus controller units 1216 manage a set of peripheral buses, such as one or more PCI or PCIe buses. In at least one embodiment, system agent core 1210 provides management functionality for various processor components. In at least one embodiment, system agent core 1210 includes one or more integrated memory controllers 1214 to manage access to various external memory devices (not shown), including support for data bus protocols such as DDR SDRAM.

[0128] In at least one embodiment, one or more processor cores 1202A-1202N include support to run in multiple threads simultaneously. In at least one embodiment, system agent core 1210 includes components for coordination and operation of cores 1202A-1202N during multi-threaded processing. In at least one embodiment, system agent core 1210 can additionally include a power control unit (PCU), including logic and components to govern one or more power states of processor cores 1202A-1202N and graphics processor 1208.

[0129] In at least one embodiment, processor 1200 also includes graphics processor 1208, which can be configured to perform a graphics processing applications or a graphics processing- intensive applications. In at least one embodiment, graphics processor 1208 couples with shared cache unit 1206 and system agent core 1210, including one or more integrated memory controllers 1214. In at least one embodiment, system agent core 1210 also includes a display controller 1211 for driving one or more coupled displays to present graphics processor output to a

[0130] In at least one embodiment, ring based interconnect unit 1212 is used to couple the internal components of processor 1200. In at least one embodiment, an alternative interconnect unit can be used, such as a point-to-point interconnect, a switched interconnect, or other technology. In at least one embodiment, graphics processor 1208 couples with ring interconnect 1212 via I / O link 1213.

[0131] In at least one embodiment, I / O link 1213 represents at least one of a variety of I / O interconnects, including packaged I / O interconnects that facilitate communication between various processor components and high-performance embedded memory module 1218 (e.g., eDRAM module). In at least one embodiment, each of processor cores 1202A-1202N and graphics processor 1208 uses embedded memory module 1218 as a shared last-level cache.

[0132] In at least one embodiment, processor cores 1202A-1202N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, processor cores 1202A-1202N are heterogeneous in terms of instruction set architecture (ISA), with one or more processor cores 1202A-1202N executing a common instruction set, while one or more other processor cores 1202A-1202N execute a subset of the common instruction set or a different instruction set. In at least one embodiment, processor cores 1202A-1202N are heterogeneous in terms of microarchitecture, with one or more cores having relatively high power consumption coupled to one or more power cores having lower power consumption. In at least one embodiment, processor 1200 may be implemented on one or more chips or implemented as a SoC integrated circuit.

[0133] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. The following is combined with... Figure 7A and / or Figure 7B Details regarding the inference and / or training logic 715 are provided. In at least one embodiment, some or all of the inference and / or training logic 715 may be incorporated into the processor 1200. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs embodied in... Figure 12 The graphics processor 1512, graphics core 1202A-1202N, or other components are used. Furthermore, in at least one embodiment, the inference and / or training operations described herein can use, except... Figure 7A or Figure 7B The logic is performed using logic other than that shown. In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown), which configure the ALU of the graphics processor 1200 to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0134] Such components can be used to render images using importance sampling based on ray tracing, which can be accelerated by hardware.

[0135] Virtualization computing platform

[0136] Figure 13 is an example data flow diagram of a process 1300 of generating and deploying image processing and inference pipelines, in accordance with at least one embodiment. In at least one embodiment, process 1300 can be deployed for use with imaging devices, processing devices, and / or other device types at one or more facilities 1302. Process 1300 can be executed within a training system 1304 and / or a deployment system 1306. In at least one embodiment, training system 1304 can be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for deployment system 1306. In at least one embodiment, deployment system 1306 can be configured to offload processing and computing resources in a distributed computing environment to reduce infrastructure requirements of facilities 1302. In at least one embodiment, one or more applications in a pipeline can use or call services (e.g., inference, visualization, computation, AI, etc.) of deployment system 1306 during application execution.

[0137] In at least one embodiment, some applications used in an advanced processing and inference pipeline can use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, machine learning models can be trained at facilities 1302 using data 1308 (e.g., imaging data) generated at facilities 1302 (and stored on one or more picture archiving and communication systems (PACS) servers at facilities 1302), can be trained using imaging or sequencing data 1308 from another or more facilities, or a combination thereof. In at least one embodiment, training system 1304 can be used to provide applications, services, and / or other resources to generate working, deployable machine learning models for deployment system 1306.

[0138] In at least one embodiment, model registry 1324 can be supported by object storage, which can support versioning and object metadata. In at least one embodiment, object storage can be accessed from within a cloud platform through, for example, an application programming interface (API) compatible with cloud 1426) storage. In at least one embodiment, machine learning models within model registry 1324 can be uploaded, listed, modified, or deleted by developers or partners of systems that interact with API. In at least one embodiment, API can provide access to methods that allow users with appropriate credentials to associate models with applications, such that models can be executed as part of execution of containerized instantiations of applications. Figure 14

[0139] In at least one embodiment, training pipeline 1404( Figure 14 ​) can include instances in which facility 1302 is training their own machine learning model, or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1308 generated by imaging devices, sequencing devices, and / or other types of devices can be received. In at least one embodiment, once imaging data 1308 is received, AI-assisted annotation 1310 can be used to help generate annotations corresponding to imaging data 1308 to be used as ground truth data for a machine learning model. In at least one embodiment, AI-assisted annotation 1310 can include one or more machine learning models (e.g., a convolutional neural network (CNN)) that can be trained to generate annotations corresponding to certain types of imaging data 1308 (e.g., from certain devices). In at least one embodiment, AI-assisted annotation 1310 can then be used directly, or can be adjusted or fine-tuned using annotation tools to generate ground truth data. In at least one embodiment, AI-assisted annotation 1310, labeled clinical data 1312, or a combination thereof can be used as ground truth data to train a machine learning model. In at least one embodiment, a trained machine learning model can be referred to as output model 1316, and can be used by deployment system 1306, as described herein.

[0140] In at least one embodiment, training pipeline 1404( Figure 14) can include situations in which facility 1302 needs a machine learning model for performing one or more processing tasks for one or more applications in deployment system 1306, but facility 1302 can not currently have such a machine learning model (or can not have a model that is optimized, efficient, or effective for this purpose). In at least one embodiment, an existing machine learning model can be selected from model registry 1324. In at least one embodiment, model registry 1324 can include machine learning models that are trained to perform a variety of different inferencing tasks on imaging data. In at least one embodiment, the machine learning models in model registry 1324 can have been trained on imaging data from different facilities (e.g., facilities located remotely from facility 1302). In at least one embodiment, a machine learning model can have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a particular location, the training can be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or limits the transfer of the imaging data offsite. In at least one embodiment, once a model is trained, or partially trained, at a location, the machine learning model can be added to model registry 1324. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be used in model registry 1324. In at least one embodiment, a machine learning model can then be selected from model registry 1324 (and referred to as output model 1316), and can be used in deployment system 1306 to perform one or more processing tasks for one or more applications of the deployment system.

[0141] In at least one embodiment, training pipeline 1404( Figure 14In at least one embodiment, scenario can include facility 1302 that requires a machine learning model for performing one or more processing tasks for deploying one or more applications in deployment system 1306, but facility 1302 can not currently have such a machine learning model (or can not have an optimized, efficient, or effective model). In at least one embodiment, due to population differences, robustness of training data used to train a machine learning model, diversity of training data anomalies, and / or other issues with training data, a machine learning model selected from model registry 1324 can not be fine-tuned or optimized for imaging data 1308 generated at facility 1302. In at least one embodiment, AI assisted annotation 1310 can be used to help generate annotations corresponding to imaging data 1308 for use as ground truth data to train or update a machine learning model. In at least one embodiment, labeled clinical data 1312 can be used as ground truth data to train a machine learning model. In at least one embodiment, retraining or updating a machine learning model can be referred to as model training 1314. In at least one embodiment, model training 1314 (e.g., AI assisted annotation 1310, labeled clinical data 1312, or a combination thereof) can be used as ground truth data to retrain or update a machine learning model. In at least one embodiment, a trained machine learning model can be referred to as output model 1316 and can be used by deployment system 1306, as described herein.

[0142] In at least one embodiment, deployment system 1306 can include software 1318, services 1320, hardware 1322, and / or other components, features, and functionality. In at least one embodiment, deployment system 1306 can include a software “stack” such that software 1318 can be built on top of services 1320, and can use services 1320 to perform some or all processing tasks, and services 1320 and software 1318 can be built on top of hardware 1322 and use hardware 1322 to perform processing, storage, and / or other computing tasks of deployment system. In at least one embodiment, software 1318 can include any number of different containers, where each container can execute an instantiation of an application. In at least one embodiment, each application can perform one or more processing tasks in a high-level processing and inference pipeline (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.). In at least one embodiment, a high-level processing and inference pipeline can be defined based on a selection of different containers desired or required to process imaging data 1308 (e.g., to convert output back to a usable data type, in addition to receiving and configuring containers for use by each container with imaging data for use and / or use by facility 1302 after processing through the pipeline. In at least one embodiment, a combination of containers within software 1318 (e.g., which make up a pipeline) can be referred to as a virtual instrument (as described in greater detail herein), and a virtual instrument can utilize services 1320 and hardware 1322 to perform some or all processing tasks of applications instantiated in containers.

[0143] In at least one embodiment, a data processing pipeline can receive input data (e.g., imaging data 1308) in a specific format in response to an inference request (e.g., a request from a user of deployment system 1306). In at least one embodiment, input data can represent one or more images, videos, and / or other data representations generated by one or more imaging devices. In at least one embodiment, data can be pre-processed as part of a data processing pipeline to prepare data for processing by one or more applications. In at least one embodiment, post-processing can be performed on output of one or more inference tasks or other processing tasks of a pipeline to prepare output data for a next application and / or to prepare output data for transmission and / or use by a user (e.g., in response to an inference request). In at least one embodiment, inference tasks can be performed by one or more machine learning models, such as trained or deployed neural networks, which can include output models 1316 of training system 1304.

[0144] In at least one embodiment, the tasks of the data processing pipeline can be encapsulated in containers, each container representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing a machine learning model. In at least one embodiment, containers or applications can be published to a private (e.g., limited access) area of ​​a container registry (described in more detail herein), and trained or deployed models can be stored in a model registry 1324 and associated with one or more applications. In at least one embodiment, an image of an application (e.g., a container image) can be used in the container registry, and once a user selects an image from the container registry for deployment in the pipeline, that image can be used to generate containers for instantiation of the application for use by the user's system.

[0145] In at least one embodiment, a developer (e.g., a software developer, clinician, physician, etc.) can develop, publish, and store an application (e.g., as a container) for performing image processing and / or inference on provided data. In at least one embodiment, a software development kit (SDK) associated with the system can be used to perform development, publication, and / or storage (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application can be tested locally using the SDK (e.g., at a first facility, testing data from a first facility), the SDK serving as a system (e.g.,...). Figure 14 System 1400 may support at least some services 1320. In at least one embodiment, since DICOM objects may contain one to hundreds of images or other data types, and due to variations in the data, the developer may be responsible for managing (e.g., setting up constructs for preprocessing built into the application, etc.) the extraction and preparation of incoming data. In at least one embodiment, once verified by system 1400 (e.g., for accuracy), the application becomes available in the container registry for user selection and / or implementation to perform one or more processing tasks on data at the user's facility (e.g., a second facility).

[0146] In at least one embodiment, the developer can then share the application or container over a network for the system (e.g., Figure 14of the system 1400) by users. In at least one embodiment, completed and validated applications or containers can be stored in a container registry, and related machine learning models can be stored in a model registry 1324. In at least one embodiment, a requesting entity (which provides an inference or image processing request) can browse the container registry and / or model registry 1324 to obtain applications, containers, datasets, machine learning models, etc., select a desired combination of elements to include in a data processing pipeline, and submit an image processing request. In at least one embodiment, a request can include input data necessary to perform the request (and, in some examples, data related to a patient), and / or can include a selection of applications and / or machine learning models to be executed in processing the request. In at least one embodiment, a request can then be passed to one or more components of the deployment system 1306 (e.g., a cloud) to perform processing of the data processing pipeline. In at least one embodiment, processing by the deployment system 1306 can include referencing elements (e.g., applications, containers, models, etc.) selected from the container registry and / or model registry 1324. In at least one embodiment, once results are generated through the pipeline, the results can be returned to a user for review (e.g., for review in a viewing application suite executed on a local, on-premises workstation or terminal).

[0147] In at least one embodiment, to help process or execute applications or containers in a pipeline, services 1320 can be utilized. In at least one embodiment, services 1320 can include computing services, artificial intelligence (Al) services, visualization services, and / or other service types. In at least one embodiment, services 1320 can provide functionality that is common to one or more applications in software 1318, and thus functionality can be abstracted as a service that can be called or utilized by applications. In at least one embodiment, functionality provided by services 1320 can run dynamically and more efficiently, while also allowing applications to process data in parallel (e.g., using Figure 14the parallel computing platform 1430) to scale well. In at least one embodiment, not every application that requires the same functionality provided by a shared service 1320 must have a corresponding instance of the service 1320, but rather the service 1320 can be shared among and between various applications. In at least one embodiment, as a non-limiting example, a service can include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service can be included, which can provide machine learning model training and / or retraining capabilities. In at least one embodiment, a data augmentation service can be further included, which can provide GPU-accelerated data (e.g., DICOM, RIS, CIS, REST-compliant, RPC, raw, etc.) extraction, resizing, scaling, and / or other augmentations. In at least one embodiment, a visualization service can be used, which can add image rendering effects (e.g., ray tracing, rasterization, de-noising, sharpening, etc.) to add realism to two-dimensional (2D) and / or three-dimensional (3D) models. In at least one embodiment, a virtual instrument service can be included, which provides beamforming, segmentation, inference, imaging, and / or support to other applications within a pipeline of a virtual instrument.

[0148] In at least one embodiment, where the services 1320 include an AI service (e.g., an inference service), as part of execution of an application, one or more machine learning models can be executed by invoking (e.g., as an API call) the inference service (e.g., inference server) to execute one or more machine learning models or processing thereof. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, the application can invoke the inference service to execute the machine learning model for performing one or more processing operations associated with the segmentation task. In at least one embodiment, software 1318 implementing an advanced processing and inference pipeline, including a segmentation application and an anomaly detection application, can be pipelined, as each application can invoke the same inference service to perform one or more inference tasks.

[0149] In at least one embodiment, hardware 1322 can include a GPU, CPU, graphics card, AI / deep learning system (e.g., an AI supercomputer such as NVIDIA’s DGX), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 can be used to provide efficient, specially-built support for software 1318 and services 1320 in deployment system 1306. In at least one embodiment, GPU processing can be implemented for local processing within an AI / deep learning system, in a cloud system, and / or in other processing components of deployment system 1306 (e.g., at facility 1302) to improve efficiency, accuracy, and performance of image processing and generation. In at least one embodiment, software 1318 and / or services 1320 can be optimized for GPU processing, by way of non-limiting example with respect to deep learning, machine learning, and / or high performance computing. In at least one embodiment, at least some of computing environments of deployment system 1306 and / or training system 1304 can be executed in a data center with GPU-optimized software (e.g., a hardware and software combination of NVIDIA’s DGX system), one or more supercomputers, or high performance computer systems. In at least one embodiment, hardware 1322 can include any number of GPUs that can be called upon to perform data processing in parallel, as described herein. In at least one embodiment, a cloud platform can also include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, an AI / deep learning supercomputer and / or GPU-optimized software (e.g., as provided on NVIDIA’s DGX system) can be used as a hardware abstraction and scaling platform to execute a cloud platform (e.g., NVIDIA’s NGC). In at least one embodiment, a cloud platform can integrate an application container clustering system or orchestration system (e.g., KUBERNETES) across multiple GPUs to enable seamless scaling and load balancing.

[0150] Figure 14 is a system diagram of an example system 1400 for generating and deploying imaging deployment pipelines in accordance with at least one embodiment. In at least one embodiment, system 1400 can be used to implement processes 1300 and / or other processes of FIG. 13, including advanced processing and inference pipelines. In at least one embodiment, system 1400 can include training system 1304 and deployment system 1306. In at least one embodiment, training system 1304 and deployment system 1306 can be implemented using software 1318, services 1320, and / or hardware 1322, as described herein. Figure 13

[0151] ​In at least one embodiment, system 1400 (e.g., training system 1304 and / or deployment system 1306) can be implemented in a cloud computing environment (e.g., using cloud 1426). In at least one embodiment, system 1400 can be implemented locally (with respect to a medical service facility), or as a combination of cloud computing resources and local computing resources. In at least one embodiment, access to APIs in cloud 1426 can be limited to authorized users by instituting security measures or protocols. In at least one embodiment, security protocols can include network tokens that can be signed by an authentication (e.g., AuthN, AuthZ, Gluecon, etc.) service and can carry appropriate authorization. In at least one embodiment, APIs (described herein) of a virtual instrument or other instances of system 1400 can be limited to a set of public IPs that have been vetted or authorized for interaction.

[0152] In at least one embodiment, various components of system 1400 can communicate information between each other using any of a plurality of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communication between facilities and components of system 1400 (e.g., for sending inference requests, for receiving results of inference requests, etc.) can be communicated through one or more data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0153] In at least one embodiment, similar to training pipelines 1302 described herein with respect to Figure 13 In at least one embodiment, training system 1304 can execute training pipeline 1404. In at least one embodiment, where deployment system 1306 is to use one or more machine learning models in deployment pipeline 1410, training pipeline 1404 can be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1406 (e.g., without retraining or updating). In at least one embodiment, as a result of training pipeline 1404, output models 1316 can be generated. In at least one embodiment, training pipeline 1404 can include any number of processing steps, such as but not limited to conversion or adaptation of imaging data (or other input data). In at least one embodiment, different training pipelines 1404 can be used for different machine learning models used by deployment system 1306. In at least one embodiment, similar to training pipeline 1302 described herein with respect to Figure 13 Training pipeline 1404 of a first example described herein with respect to Figure 13 Training pipeline 1404 of a second example described herein with respect to Figure 13The training pipeline 1404 of the third example described can be used for the third machine learning model. In at least one embodiment, any combination of tasks within the training system 1304 can be used depending on the requirements of each corresponding machine learning model. In at least one embodiment, one or more machine learning models can already be trained and ready for deployment, so the training system 1304 can not perform any processing on the machine learning model and the one or more machine learning models can be implemented by the deployment system 1306.

[0154] In at least one embodiment, the output model 1316 and / or the pre-trained model 1406 can include any type of machine learning model, depending on implementation or embodiment. In at least one embodiment and without limitation thereto, machine learning models used by the system 1400 can include using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k- nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptrons, long / short-term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0155] In at least one embodiment, the training pipeline 1404 can include AI-assisted annotation, as described herein with respect to at least Figure 15BIn at least one embodiment, labeled clinical data 1312 (e.g., traditional annotations) can be generated by any number of techniques. In at least one embodiment, labels or other annotations can be generated in a drawing program (e.g., annotation program), a computer aided design (CAD) program, a labeling program, another type of application suitable for generating annotations or labels for ground truth, and / or can be hand drawn, in some examples. In at least one embodiment, ground truth data can be synthetically generated (e.g., from computer models or renderings), realistically generated (e.g., designed and generated from real world data), automatically generated by a machine (e.g., using feature analysis and learning to extract features from data and then generate labels), manually annotated (e.g., by a marker or annotation specialist defining locations of labels), and / or combinations thereof. In at least one embodiment, for each instance of imaging data 1308 (or other data types used by machine learning models), there can be corresponding ground truth data generated by training system 1304. In at least one embodiment, AI assisted annotation can be performed as part of deployment pipeline 1410; in addition to or instead of AI assisted annotation included in training pipeline 1404. In at least one embodiment, system 1400 can include a multi-tiered platform that can include a software tier of diagnostic applications (or other application types) (e.g., software 1318) that can perform one or more medical imaging and diagnostic functions. In at least one embodiment, system 1400 can be communicatively coupled to (e.g., via encrypted links) a PACS server network of one or more facilities. In at least one embodiment, system 1400 can be configured to access and reference data from a PACS server to perform operations such as training machine learning models, deploying machine learning models, image processing, inferencing, and / or other operations.

[0156] In at least one embodiment, software tier can be implemented as a secure, encrypted, and / or authenticated API through which applications or containers can be invoked (e.g., called) from an external environment (e.g., facility 1302). In at least one embodiment, applications can then call or execute one or more services 1320 to perform computing, AI, or visualization tasks associated with respective applications, and software 1318 and / or services 1320 can utilize hardware 1322 to perform processing tasks in an efficient and effective manner.

[0157] In at least one embodiment, deployment system 1306 can execute deployment pipelines 1410. In at least one embodiment, deployment pipelines 1410 can include any number of applications that can be sequential, non-sequential, or otherwise applied to imaging data (and / or other data types) - including AI assisted annotation - generated by imaging devices, sequencing devices, genomics devices, etc., as described above. In at least one embodiment, a deployment pipeline 1410 for an individual device can be referred to as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.), as described herein. In at least one embodiment, there can be more than one deployment pipeline 1410 for a single device, depending on information desired from data generated by a device. In at least one embodiment, where anomalies are desired to be detected from MRI machines, there can be a first deployment pipeline 1410, and where image enhancement is desired from output of MRI machines, there can be a second deployment pipeline 1410.

[0158] In at least one embodiment, image generation applications can include processing tasks that include use of machine learning models. In at least one embodiment, a user can wish to use their own machine learning model, or select a machine learning model from model registry 1324. In at least one embodiment, a user can implement their own machine learning model or select a machine learning model for inclusion in an application that performs a processing task. In at least one embodiment, applications can be selectable and customizable, and by defining a construction of an application, deployment and implementation of an application for a particular user is presented as a more seamless user experience. In at least one embodiment, by leveraging other features of system 1400 (e.g., services 1320 and hardware 1322), deployment pipelines 1410 can be more user friendly, provide easier integration, and produce more accurate, efficient, and timely results.

[0159] In at least one embodiment, deployment system 1306 can include a user interface 1414 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications to include in deployment pipelines 1410, arrange applications, modify or change applications or parameters or constructions thereof, use and interact with deployment pipelines 1410 during setup and / or deployment, and / or otherwise interact with deployment system 1306. In at least one embodiment, although not shown with respect to training system 1304, user interface 1414 (or a different user interface) can be used to select models for use in deployment system 1306, for selecting models for training or retraining in training system 1304, and / or for otherwise interacting with training system 1304.

[0160] In at least one embodiment, in addition to application orchestration system 1428, pipeline manager 1412 can also be used to manage interactions between applications or containers of deployment pipeline 1410 and services 1320 and / or hardware 1322. In at least one embodiment, pipeline manager 1412 can be configured to facilitate interactions from application to application, from application to service 1320, and / or from application or service to hardware 1322. In at least one embodiment, although illustrated as included in software 1318, this is not intended to be limiting, and in some examples (e.g., as illustrated in FIG. 13B), pipeline manager 1412 can be included in services 1320. In at least one embodiment, application orchestration system 1428 (e.g., Kubernetes, DOCKER, etc.) can include a container orchestration system that can group applications into containers as logical units for orchestration, management, scaling, and deployment. In at least one embodiment, by associating applications (e.g., reconstituted applications, split applications, etc.) from deployment pipeline 1410 with individual containers, each application can execute in a self-contained environment (e.g., at kernel level) to improve speed and efficiency. Figure 14

[0161] ​In at least one embodiment, each application and / or container (or image thereof) can be separately developed, modified, and deployed (e.g., a first user or developer can develop, modify, and deploy a first application, a second user or developer can develop, modify, and deploy a second application separate from first user or developer), which can allow for focus and attention to tasks of a single application and / or container without being impeded by tasks of another application or container. In at least one embodiment, pipeline manager 1412 and application coordination system 1428 can facilitate communication and collaboration between different containers or applications. In at least one embodiment, as long as intended inputs and / or outputs of each container or application are known to system (e.g., based on construction of application or container), application coordination system 1428 and / or pipeline manager 1412 can facilitate communication between and among each application or container and sharing of resources. In at least one embodiment, as one or more applications or containers in deployment pipeline 1410 can share same services and resources, application coordination system 1428 can coordinate, load balance, and determine sharing of services or resources between and among various applications or containers. In at least one embodiment, a scheduler can be used to track resource needs of applications or containers, current or planned use of these resources, and resource availability. Accordingly, in at least one embodiment, a scheduler can allocate resources to different applications and among and between applications, taking into account needs and availability of system. In some examples, a scheduler (and / or other components of application coordination system 1428) can determine resource availability and distribution based on constraints imposed on system (e.g., user constraints), such as quality of service (QoS), urgency of data output (e.g., to determine whether to perform real-time processing or delayed processing), etc.

[0162] In at least one embodiment, services 1320 utilized by and shared by applications or containers in deployment system 1306 can include compute services 1416, AI services 1418, visualization services 1420, and / or other service types. In at least one embodiment, applications can invoke (e.g., execute) one or more services 1320 to perform processing operations for applications. In at least one embodiment, applications can utilize compute services 1416 to perform supercomputing or other high performance computing (HPC) tasks. In at least one embodiment, one or more compute services 1416 can be utilized to perform parallel processing (e.g., using parallel computing platform 1430) to process data substantially simultaneously by one or more applications and / or one or more tasks of a single application. In at least one embodiment, parallel computing platform 1430 (e.g., NVIDIA’s CUDA) can implement general purpose computing on GPUs (GPGPU) (e.g., GPU 1422). In at least one embodiment, software layers of parallel computing platform 1430 can provide access to virtual instruction sets and parallel computing elements of GPUs to execute compute kernels. In at least one embodiment, parallel computing platform 1430 can include memory, and in some embodiments, memory can be shared between and among multiple containers, and / or between and among different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls can be generated for multiple containers and / or multiple processes within a container to use same data for shared memory segments of parallel computing platform 1430 (e.g., where multiple different stages of an application or applications are processing same information). In at least one embodiment, rather than copying data and moving data to different locations in memory (e.g., read / write operations), same data in same location in memory can be used for any number of processing tasks (e.g., at same time, at different times, etc.). In at least one embodiment, as data is used to generate new data as a result of processing, this information of new location of data can be stored and shared between various applications. In at least one embodiment, location of data and location of updated or modified data can be part of how to understand a definition of a payload in a container.

[0163] In at least one embodiment, AI services 1418 can be utilized to perform inferencing services for executing machine learning models associated with an application (e.g., task is to perform one or more processing tasks for an application). In at least one embodiment, AI services 1418 can utilize AI system 1424 to execute machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inferencing tasks. In at least one embodiment, an application of deployment pipeline 1410 can use one or more output models 1316 of a self-training system 1304 and / or other models of an application to perform inferencing on imaging data. In at least one embodiment, two or more examples of inferencing can be available using an application coordination system 1428 (e.g., a scheduler). In at least one embodiment, a first category can include a high priority / low latency path, which can implement a higher service level agreement, such as for performing inferencing on urgent requests in emergency situations, or for radiologists in a diagnostic process. In at least one embodiment, a second category can include a standard priority path, which can be used for requests that can not be urgent or can have analysis performed at a later time. In at least one embodiment, application coordination system 1428 can allocate resources (e.g., services 1320 and / or hardware 1322) based on a priority path for different inferencing tasks of AI services 1418.

[0164] In at least one embodiment, shared memory may be installed into AI service 1418 in system 1400. In at least one embodiment, shared memory may operate as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a set of API instances of deployment system 1306 may receive the request and may select one or more instances (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request may be fed into a database, and if not already in the cache, a machine learning model may be located from model registry 1324. A verification step may ensure that an appropriate machine learning model is loaded into the cache (e.g., shared memory), and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not already running or there are not enough instances of the application, a scheduler (e.g., the scheduler of pipeline manager 1412) may be used to start the application referenced in the request. In at least one embodiment, if an inference server has not yet been started to execute the model, an inference server may be started. Any number of inference servers may be started for each model. In at least one embodiment, in a pull model that clusters inference servers, the model can be cached whenever load balancing is favorable. In at least one embodiment, the inference servers can be statically loaded into the corresponding distributed servers.

[0165] In at least one embodiment, an inference server running in a container can be used to perform inference. In at least one embodiment, an instance of the inference server can be associated with a model (and optionally multiple versions of the model). In at least one embodiment, if an instance of the inference server does not exist when a request to perform inference on the model is received, a new instance can be loaded. In at least one embodiment, when the inference server is started, a model can be passed to the inference server, allowing the same container to be used to serve different models, as long as the inference server runs as different instances.

[0166] In at least one embodiment, during application execution, an inference request for a given application can be received and a container (e.g., an instance hosting an inference server) can be loaded (if not already loaded) and a launcher can be invoked. In at least one embodiment, pre-processing logic in a container can load, decode, and / or perform any additional pre-processing on incoming data (e.g., using CPU and / or GPU). In at least one embodiment, once data is ready for inference, a container can infer on data as needed. In at least one embodiment, this can include a single inference call on one image (e.g., a hand X-ray), or can require inference on hundreds of images (e.g., a chest CT). In at least one embodiment, an application can summarize results before completion, which can include, without limitation, a single confidence score, a pixel-level segmentation, a voxel-level segmentation, generating a visualization, or generating text to summarize results. In at least one embodiment, different priorities can be assigned for different models or applications. For example, some models can have real-time (TAT less than 1 minute) priority, while other models can have lower priority (e.g., TAT less than 10 minutes). In at least one embodiment, model execution time can be measured from a requesting authority or entity, and can include cooperative network traversal time as well as execution time of an inference service.

[0167] In at least one embodiment, transfer of requests between service 1320 and inference applications can be hidden behind a software development kit (SDK), and robust transfer can be provided through queues. In at least one embodiment, requests will be placed in queues through an API for individual application / tenant ID combinations, and SDK will pull requests from queues and provide requests to applications. In at least one embodiment, a name of a queue can be provided in an environment from which SDK will pick up queues. In at least one embodiment, asynchronous communication through queues can be useful because it can allow any instance of an application to pick up work when it is available. Results can be transferred back through queues to ensure no data loss. In at least one embodiment, queues can also provide an ability to split work, because highest priority work can go into a queue that connects to most instances of an application, while lowest priority work can go into a queue that connects to a single instance that processes tasks in order of receipt. In at least one embodiment, an application can run on GPU-accelerated instances that are spawned in cloud 1426, and inference service can perform inference on GPUs.

[0168] In at least one embodiment, visualization service 1420 can be utilized to generate visualizations for viewing application and / or deployment pipeline 1410 output. In at least one embodiment, visualization service 1420 can utilize GPU 1422 to generate visualizations. In at least one embodiment, visualization service 1420 can implement rendering effects such as ray tracing to generate higher quality visualizations. In at least one embodiment, visualizations can include, without limitation, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slices, virtual reality displays, augmented reality displays, etc. In at least one embodiment, virtualization environment can be used to generate virtual interactive displays or environments (e.g., virtual environments) for interaction by system users (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, visualization service 1420 can include internal visualizers, movie and / or other rendering or image processing capabilities or functionality (e.g., ray tracing, rasterization, internal optics, etc.).

[0169] In at least one embodiment, hardware 1322 can include GPU 1422, AI system 1424, cloud 1426, and / or any other hardware used to execute training system 1304 and / or deployment system 1306. In at least one embodiment, GPU 1422 (e.g., NVIDIA’s TESLA and / or QUADRO GPU) can include any number of GPUs that can be used to perform processing tasks for any features or functionality of computing service 1416, AI service 1418, visualization service 1420, other services, and / or software 1318. For example, for AI service 1418, GPU 1422 can be used to perform pre-processing on imaging data (or other data types used by machine learning models), perform post-processing on outputs of machine learning models, and / or perform inference (e.g., to execute machine learning models). In at least one embodiment, cloud 1426, AI system 1424, and / or other components of system 1400 can use GPU 1422. In at least one embodiment, cloud 1426 can include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1424 can use GPUs, and one or more AI systems 1424 can be used to perform cloud 1426 (or at least portions of tasks that are deep learning or inference). Likewise, although hardware 1322 is shown as discrete components, this is not intended to be limiting, and any component of hardware 1322 can be combined with, or utilized by, any other component of hardware 1322.

[0170] In at least one embodiment, AI system 1424 can include a purpose-built computing system (e.g., a supercomputer or HPC) configured for inferencing, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, AI system 1424 (e.g., NVIDIA’s DGX) can include software (e.g., a software stack) that can use multiple GPUs 1422 to perform split-GPU optimizations in addition to CPUs, RAM, storage, and / or other components, features, or functions. In at least one embodiment, one or more AI systems 1424 can be implemented in cloud 1426 (e.g., in a data center) to perform some or all of AI-based processing tasks of system 1400.

[0171] In at least one embodiment, cloud 1426 can include a GPU-accelerated infrastructure (e.g., NVIDIA’s NGC) that can provide a GPU-optimized platform for performing processing tasks of system 1400. In at least one embodiment, cloud 1426 can include AI system 1424 for performing one or more AI-based tasks of system 1400 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1426 can integrate with application orchestration system 1428 that utilizes multiple GPUs to enable seamless scaling and load balancing between and among applications and services 1320. In at least one embodiment, cloud 1426 can be responsible for performing at least some services 1320 of system 1400, including compute services 1416, AI services 1418, and / or visualization services 1420, as described herein. In at least one embodiment, cloud 1426 can perform batched inference (e.g., perform NVIDIA’s TENSORRT), provide an accelerated parallel computing API and platform 1430 (e.g., NVIDIA’s CUDA), execute application orchestration system 1428 (e.g., KUBERNETES), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality cinematic effects), and / or can provide other functionality for system 1400.

[0172] Figure 15A A dataflow graph for process 1500 for training, retraining, or updating a machine learning model is shown, in accordance with at least one embodiment. In at least one embodiment, process 1500 can be performed using, as a non-limiting example, NVIDIA’s Figure 14the system 1400 to perform the process 1500. In at least one embodiment, the process 1500 can utilize the services 1320 and / or hardware 1322 of the system 1400, as described herein. In at least one embodiment, a refined model 1512 generated by the process 1500 can be executed by the deployment system 1306 for one or more containerized applications in the deployment pipeline 1410.

[0173] In at least one embodiment, the model training 1314 can include retraining or updating the initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data (such as the customer dataset 1506), and / or new ground truth data associated with the input data). In at least one embodiment, to retrain or update the initial model 1504, an output or loss layer of the initial model 1504 can be reset or deleted, and / or replaced with an updated or new output or loss layer. In at least one embodiment, the initial model 1504 can have previously fine-tuned parameters (e.g., weights and / or biases) that are preserved from previous training, so the training or retraining 1314 can not take as long or require as much processing as training a model from scratch. In at least one embodiment, during the model training 1314, by resetting or replacing the output or loss layer of the initial model 1504, parameters of the new dataset can be updated and re-tuned as predictions are generated on the new customer dataset 1506 (e.g., image data 1308) based on loss calculations associated with accuracy of the output or loss layer. Figure 13 In at least one embodiment, the model training 1314 can include retraining or updating the initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data (such as the customer dataset 1506), and / or new ground truth data associated with the input data). In at least one embodiment, to retrain or update the initial model 1504, an output or loss layer of the initial model 1504 can be reset or deleted, and / or replaced with an updated or new output or loss layer. In at least one embodiment, the initial model 1504 can have previously fine-tuned parameters (e.g., weights and / or biases) that are preserved from previous training, so the training or retraining 1314 can not take as long or require as much processing as training a model from scratch. In at least one embodiment, during the model training 1314, by resetting or replacing the output or loss layer of the initial model 1504, parameters of the new dataset can be updated and re-tuned as predictions are generated on the new customer dataset 1506 (e.g., image data 1308) based on loss calculations associated with accuracy of the output or loss layer.

[0174] In at least one embodiment, the pre-trained model 1406 can be stored in a data store or registry (e.g., a model registry 1308) for use by other systems or applications. In at least one embodiment, the pre-trained model 1406 can be stored in a data store or registry (e.g., a model registry 1308) for use by other systems or applications. Figure 13model registry 1324). In at least one embodiment, pre-trained models 1406 can have been trained, at least in part, at one or more facilities other than the facility executing process 1500. In at least one embodiment, to protect the privacy and rights of patients, subjects, or customers of different facilities, pre-trained models 1406 can have been trained locally using locally generated customer or patient data. In at least one embodiment, pre-trained models 1406 can be trained using cloud 1426 and / or other hardware 1322, but confidential, privacy protected patient data can not be transferred to, used by, or accessed by any component of cloud 1426 (or other non-local hardware). In at least one embodiment, if pre-trained models 1406 are trained using patient data from more than one facility, pre-trained models 1406 can have been individually trained for each facility before training on patient or customer data from another facility. In at least one embodiment, customer or patient data from any number of facilities can be used to train pre-trained models 1406 locally and / or externally, such as in a data center or other cloud computing infrastructure, for example, in cases where customer or patient data has been de-identified (e.g., by waiver, for experimental use, etc.), or where customer or patient data is included in a public dataset.

[0175] In at least one embodiment, when selecting an application to use in deployment pipeline 1410, a user can also select a machine learning model for use with the particular application. In at least one embodiment, a user can not have a model to use, so the user can select a pre-trained model 1406 to use with the application. In at least one embodiment, pre-trained models 1406 can not be optimized for generating accurate results on a customer dataset 1506 of a user’s facility (e.g., based on patient diversity, demographics, types of medical imaging devices used, etc.). In at least one embodiment, pre-trained models 1406 can be updated, retrained, and / or fine-tuned for use at individual facilities before being deployed into deployment pipeline 1410 for use with one or more applications.

[0176] In at least one embodiment, a user can select a pre-trained model 1406 to update, retrain, and / or fine-tune, and the pre-trained model 1406 can be referred to as an initial model 1504 for training system 1304 in process 1500. In at least one embodiment, a customer dataset 1506 (e.g., imaging data, genomic data, sequencing data, or other data types generated by equipment at a facility) can be used to perform model training 1314 (which can include, without limitation, transfer learning) on initial model 1504 to generate a refined model 1512. In at least one embodiment, ground truth data corresponding to customer dataset 1506 can be generated by training system 1304. In at least one embodiment, ground truth data can be generated at least in part by a clinician, scientist, physician, practitioner at a facility (e.g., as labeled clinical data 1312 in FIG. 13B). Figure 13

[0177] In at least one embodiment, AI-assisted annotation 1310 can be used in some examples to generate ground truth data. In at least one embodiment, AI-assisted annotation 1310 (e.g., implemented using an AI-assisted annotation SDK) can utilize a machine learning model (e.g., a neural network) to generate suggested or predicted ground truth data for a customer dataset. In at least one embodiment, a user 1510 can use annotation tools within a user interface (graphical user interface (GUI)) on computing device 1508.

[0178] In at least one embodiment, user 1510 can interact with GUI via computing device 1508 to edit or fine-tune annotations or automated annotations. In at least one embodiment, a polygon editing feature can be used to move vertices of a polygon to more precise or fine-tuned locations.

[0179] In at least one embodiment, once customer dataset 1506 has associated ground truth data, the ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) can be used during model training 1314 to generate refined model 1512. In at least one embodiment, customer dataset 1506 can be applied to initial model 1504 any number of times, and ground truth data can be used to update parameters of initial model 1504 until an acceptable level of accuracy is reached for refined model 1512. In at least one embodiment, once refined model 1512 is generated, refined model 1512 can be deployed within one or more deployment pipelines 1410 at a facility for performing one or more processing tasks with respect to medical imaging data.

[0180] ​In at least one embodiment, a refined model 1512 can be uploaded to pre-trained models 1406 in model registry 1324 for selection by another facility. In at least one embodiment, his process can be completed at any number of facilities such that refined model 1512 can be further refined any number of times on new data sets to generate more general purpose models.

[0181] Figure 15B is an example illustration of a client-server architecture 1532 for augmenting annotation tools with pre-trained annotation models, in accordance with at least one embodiment. In at least one embodiment, an AI-assisted annotation tool 1536 can be instantiated based on client-server architecture 1532. In at least one embodiment, annotation tools 1536 in an imaging application can assist radiologists, for example, in identifying organs and abnormalities. In at least one embodiment, an imaging application can include software tools that help a user 1510 identify a few extreme points on a particular organ of interest in a raw image 1534 (e.g., in a 3D MRI or CT scan), for example, and receive automatic annotation results for all 2D slices of a particular organ. In at least one embodiment, results can be stored as training data 1538 in a data store and used as ground truth data for training, for example, but not by way of limitation. In at least one embodiment, when a computing device 1508 sends extreme points for AI-assisted annotation 1310, a deep learning model, for example, can receive that data as input and return an inference result that segments an organ or abnormality. In at least one embodiment, a pre-instantiated annotation tool (e.g., AI-assisted annotation tool 1536B in Figure 15B AI-assisted annotation tool 1536B) can be augmented by making API calls (e.g., API call 1544) to a server, such as an annotation helper server 1540, which can include a set of pre-trained models 1542 stored in an annotation model registry, for example. In at least one embodiment, an annotation model registry can store pre-trained models 1542 (e.g., machine learning models, such as deep learning models) that are pre-trained to perform AI-assisted annotation on a particular organ or abnormality. In at least one embodiment, these models can be further updated by using a training pipeline 1404. In at least one embodiment, as new labeled clinical data 1312 is added, a pre-installed annotation tool can be improved over time.

[0182] Such components can be used to render images with ray-tracing based importance sampling, which can be hardware accelerated.

[0183] Automated techniques

[0184] An example system architecture for an autonomous vehicle, in accordance with at least one embodiment. In at least one embodiment, each of components, features, and systems of a vehicle include a connection via a bus. In at least one embodiment, a bus can include, without limitation, a CAN data interface (also referred to herein as a “CAN bus”). In at least one embodiment, a CAN bus can be a network within a vehicle that is used to help control various features and functions of a vehicle, such as actuation of brakes, acceleration, braking, steering, windshield wipers, etc. In at least one embodiment, a bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). In at least one embodiment, a bus can be read to find steering wheel angle, ground speed, engine revolutions per minute (“RPM”), button positions, and / or other vehicle status indicators. In at least one embodiment, a bus can be an ASIL B compliant CAN bus.

[0185] In at least one embodiment, in addition to or instead of CAN, FlexRay and / or Ethernet can be used. In at least one embodiment, there can be any number of buses, which can include, without limitation, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses that use different protocols. In at least one embodiment, two or more buses can be used to perform different functions, and / or can be used for redundancy. For example, a first bus can be used for collision avoidance functions, while a second bus can be used for actuation control. In at least one embodiment, each bus can communicate with any component of a vehicle, and two or more buses can communicate with the same component. In at least one embodiment, any number of system on chips (“SoCs”), each controller, and / or each computer within a vehicle can have access to the same input data (e.g., input from sensors of a vehicle), and can be connected to a common bus, such as a CAN bus.

[0186] In at least one embodiment, a vehicle can include one or more controllers, such as those described herein with respect to Figure 1A In at least one embodiment, a controller can be used for a variety of functions. In at least one embodiment, a controller can be coupled to any of various other components and systems of a vehicle, and can be used for control of a vehicle, artificial intelligence of a vehicle, infotainment of a vehicle, and / or the like.

[0187] In at least one embodiment, a vehicle can include any number of SoCs. Each SoC can include, without limitation, a central processing unit (“CPU”), a graphics processing unit (“GPU”), a processor, a cache, an accelerator, a data store, and / or other components and features not shown. In at least one embodiment, SoCs can be used to control vehicles in a variety of platforms and systems. For example, in at least one embodiment, SoCs can be combined in systems (e.g., systems of vehicles) with high definition (“HD”) maps that can obtain map refreshes and / or updates from one or more servers via an interface over a network.

[0188] In at least one embodiment, a CPU can include a CPU cluster or CPU complex (referred to herein as a “CCPLEX”). In at least one embodiment, a CPU can include multiple cores and / or level two (“L2”) caches. For example, in at least one embodiment, a CPU can include eight cores in a coherent multi-processor configuration. In at least one embodiment, a CPU can include four dual-core clusters with each cluster having a dedicated L2 cache (e.g., 2 MB L2 cache). In at least one embodiment, a CPU (e.g., CCPLEX) can be configured to support simultaneous cluster operation such that any combination of CPU’s clusters can be active at any given time.

[0189] In at least one embodiment, one or more CPUs can implement power management capabilities including, but not limited to, one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when a core is not actively executing instructions due to executing a wait for interrupt (“WFI”) / wait for event (“WFE”) instruction; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated. In at least one embodiment, a CPU can further implement an enhanced algorithm for managing power states in which allowed power states and expected wake-up times are specified and hardware / microcode determines the best power state for a core, cluster, and CCPLEX to enter. In at least one embodiment, a processing core can support a simplified power state entry sequence in software in which work is offloaded to microcode.

[0190] In at least one embodiment, GPU can include an integrated GPU (also referred to herein as “iGPU”). In at least one embodiment, GPU can be programmable and efficient for parallel workloads. In at least one embodiment, GPU can use an enhanced tensor instruction set in at least one embodiment. In at least one embodiment, GPU can include one or more streaming microprocessors, where each streaming microprocessor can include a level one (“LI”) cache (e.g., LI cache with at least 96 KB storage capacity), and two or more streaming microprocessors can share an L2 cache (e.g., L2 cache with 512 KB storage capacity). In at least one embodiment, GPU can include at least eight streaming microprocessors. In at least one embodiment, GPU can use a compute application programming interface (“API”). In at least one embodiment, GPU can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0191] In at least one embodiment, one or more of GPU can be power-optimized for best performance in automotive and embedded use cases. For example, in one embodiment, GPU can be fabricated on a finned field effect transistor (“FinFET”). In at least one embodiment, each streaming microprocessor can incorporate multiple mixed-precision processing cores divided into four processing blocks, for example, but not limited to, 64 PF32 cores and 32 PF64 cores. In at least one embodiment, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor Cores for deep learning matrix arithmetic, a level zero (“LO”) instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. In at least one embodiment, streaming microprocessor can include independent parallel integer and floating point data paths to provide efficient execution of workloads through a mix of computation and address computation. In at least one embodiment, streaming microprocessor can include independent thread scheduling capabilities to enable finer-grain synchronization and cooperation among parallel threads. In at least one embodiment, streaming microprocessor can include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

[0192] In at least one embodiment, one or more GPUs can include high bandwidth memory (HBM) and / or 16 GB HBM2 memory subsystems to provide, in some examples, a peak memory bandwidth of approximately 900 GB / sec. In at least one embodiment, in addition to or instead of HBM memory, synchronous graphics random access memory (“SGRAM”) can be used, such as graphics double data rate type five synchronous random access memory (“GDDR5”).

[0193] In at least one embodiment, GPUs can include unified memory technology. In at least one embodiment, address translation services (“ATS”) support can be used to allow GPUs to directly access CPU page tables. In at least one embodiment, when GPU memory management unit (“MMU”) experiences a miss, an address translation request can be sent to a CPU. In response, in at least one embodiment, a CPU can look up a virtual to physical mapping for an address in a page table and transmit a translation back to a GPU. In at least one embodiment, unified memory technology can allow a single unified virtual address space for memory for both CPUs and GPUs, simplifying GPU programming and porting applications to GPUs.

[0194] In at least one embodiment, GPUs can include any number of access counters that can track how frequently a GPU is accessing memory of other processors. In at least one embodiment, access counters can help ensure that memory pages are moved to physical memory of a processor that most frequently accesses the pages, improving efficiency of memory ranges shared between processors.

[0195] In at least one embodiment, one or more SoCs can include any number of caches, including those described herein. For example, in at least one embodiment, a cache can include a level three (“L3”) cache that is available to both CPUs and GPUs (e.g., it is connected to both CPUs and GPUs). In at least one embodiment, a cache can include a write-back cache that can track state of lines, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, an L3 cache can include 4 MB or more, depending on embodiment, although smaller cache sizes can be used.

[0196] In at least one embodiment, one or more SoC can include one or more accelerators (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, SoC can include a hardware acceleration cluster that can include optimized hardware accelerators and / or large on-chip memory. In at least one embodiment, large on-chip memory (e.g., 4MB SRAM) can enable hardware acceleration cluster to accelerate neural networks and other computations. In at least one embodiment, hardware acceleration cluster can be used to supplement GPU and offload some tasks from GPU (e.g., freeing up more cycles of GPU for performing other tasks). In at least one embodiment, accelerators can be used for target workloads (e.g., perception, convolutional neural networks (“CNNs”), recurrent neural networks (“RNNs”), etc.) that are stable enough to change for acceleration. In at least one embodiment, CNNs can include region-based or region with convolutional neural networks (“RCNNs”) and fast RCNNs (e.g., as used for object detection) or other types of CNNs.

[0197] In at least one embodiment, accelerators (e.g., hardware acceleration cluster) can include a deep learning accelerator (“DLA”). DLA can include, without limitation, one or more tensor processing units (“TPUs”) that can be configured to provide an additional 100 trillion operations per second for deep learning applications and inferencing. In at least one embodiment, TPU can be an accelerator configured to perform and optimized for image processing functions (e.g., for CNNs, RCNNs, etc.). DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. In at least one embodiment, design of DLA can provide higher performance per mm than typical general purpose GPUs, and often greatly exceeds performance of CPUs. In at least one embodiment, TPU can perform a variety of functions including single instance convolution functions, such as supporting INT8, INT16, and FP16 data types for features and weights, and post-processor functions. In at least one embodiment, DLA can quickly and efficiently perform neural networks, especially CNNs, on processed or unprocessed data for any of a variety of functions including, without limitation: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and identification and detection using data from microphones; CNNs for face recognition and vehicle owner identification using data from camera sensors; and / or CNNs for safety and / or safety related events.

[0198] In at least one embodiment, a DLA can perform any function of a GPU, and by using an inference accelerator, for example, a designer can target a DLA or GPU for any function. For example, in at least one embodiment, a designer can concentrate processing and floating point operations of a CNN on a DLA, and leave other functions to a GPU and / or other accelerators.

[0199] In at least one embodiment, an accelerator (e.g., a hardware acceleration cluster) can include a programmable vision accelerator (“PVA”), which can alternatively be referred to herein as a computer vision accelerator. In at least one embodiment, a PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (“ADAS”), autonomous driving, augmented reality (“AR”) applications, and / or virtual reality (“VR”) applications. A PVA can provide a balance between performance and flexibility. For example, in at least one embodiment, each PVA can include, for example and without limitation, any number of reduced instruction set computer (“RISC”) cores, direct memory access (“DMA”), and / or any number of vector processors.

[0200] In at least one embodiment, a RISC core can interact with an image sensor (e.g., an image sensor of any camera described herein), an image signal processor, and / or the like. In at least one embodiment, each RISC core can include any number of memories. In at least one embodiment, a RISC core can use any of a number of protocols, depending on embodiment. In at least one embodiment, a RISC core can execute a real-time operating system (“RTOS”). In at least one embodiment, a RISC core can be implemented using one or more integrated circuit devices, application specific integrated circuits (“ASICs”), and / or memory devices. For example, in at least one embodiment, a RISC core can include an instruction cache and / or a tightly coupled RAM.

[0201] In at least one embodiment, a DMA can enable components of a PVA to access system memory independently of a CPU. In at least one embodiment, a DMA can support any number of features for providing optimizations to a PVA, including, without limitation, support for multi-dimensional addressing and / or circular addressing. In at least one embodiment, a DMA can support up to six or more dimensions of addressing, which can include, without limitation, a block width, a block height, a block depth, a horizontal block stride, a vertical block stride, and / or a depth stride.

[0202] In at least one embodiment, vector processors can be programmable processors that can be designed to efficiently and flexibly execute programming of computer vision algorithms and provide signal processing capabilities. In at least one embodiment, a PVA can include one PVA core and two vector processing subsystem partitions. In at least one embodiment, a PVA core can include a processor subsystem, DMA engines (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, a vector processing subsystem can operate as a main processing engine of a PVA and can include a vector processing unit (“VPU”), an instruction cache, and / or a vector memory (e.g., “VMEM”). In at least one embodiment, a VPU can include a digital signal processor, such as a single instruction, multiple data (“SIMD”), very long instruction word (“VLIW”) digital signal processor. In at least one embodiment, a combination of SIMD and VLIW can improve throughput and speed.

[0203] In at least one embodiment, each vector processor can include an instruction cache and can be coupled to a dedicated memory. As a result, in at least one embodiment, each vector processor can be configured to execute independently of other vector processors. In at least one embodiment, vector processors included in a particular PVA can be configured to employ data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA can execute the same computer vision algorithm but on different regions of an image. In at least one embodiment, vector processors contained in a particular PVA can simultaneously execute different computer vision algorithms on the same image, or even different algorithms on consecutive images or portions of images. In at least one embodiment, any number of PVAs can be included in a hardware acceleration cluster, and any number of vector processors can be included in each PVA, among other things. In at least one embodiment, a PVA can include additional error-correcting code (“ECC”) memory to enhance security of an overall system.

[0204] In at least one embodiment, an accelerator (e.g., hardware acceleration cluster) can include an on-chip computer vision network and static random access memory (“SRAM”) for providing high bandwidth, low latency SRAM for accelerators. In at least one embodiment, on-chip memory can include at least 4 MB of SRAM, including, for example and without limitation, 8 field-programmable memory blocks accessible by both PVA and DLA. In at least one embodiment, each pair of memory blocks can include an advanced peripheral bus (“APB”) interface, configuration circuitry, a controller, and a multiplexer. In at least one embodiment, any type of memory can be used. In at least one embodiment, PVA and DLA can access memory via a backbone that provides PVA and DLA with high-speed access to memory. In at least one embodiment, a backbone can include an on-chip computer vision network that interconnects PVA and DLA to memory (e.g., using APB).

[0205] In at least one embodiment, an on-chip computer vision network can include an interface that determines that both PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. In at least one embodiment, an interface can provide separate phases and separate lanes for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transmission. In at least one embodiment, an interface can comply with International Organization for Standardization (“ISO”) 26262 or International Electrotechnical Commission (“IEC”) 61508 standards, although other standards and protocols can be used.

[0206] In at least one embodiment, one or more SoCs can include real-time ray tracing hardware accelerators. In at least one embodiment, real-time ray tracing hardware accelerators can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) to generate real-time visualizations simulations, for radar signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR systems, for general wave propagation simulation, for comparison with LIDAR data for localization and / or other functions, and / or for other uses.

[0207] In at least one embodiment, accelerators (e.g., hardware accelerator clusters) have a wide range of uses for autonomous driving. In at least one embodiment, a PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. In at least one embodiment, capabilities of a PVA are well suited for algorithm domains that require predictable processing, low power, and low latency. In other words, PVAs perform well on semi-dense or dense regular computations, even on small data sets that require predictable run-time with low latency and low power. In at least one embodiment, autonomous vehicles, such as vehicles, PVAs are designed to run classic computer vision algorithms because they are efficient at object detection and integer math operations.

[0208] For example, in accordance with at least one embodiment of technology, a PVA is used to perform computer stereo vision. In at least one embodiment, a semi-global matching based algorithm can be used in some examples, although this is not intended to be limiting. In at least one embodiment, applications for level 3-5 autonomous driving use dynamic motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). In at least one embodiment, a PVA can perform computer stereo vision functions on inputs from two monocular cameras.

[0209] In at least one embodiment, a PVA can be used to perform dense optical flow. For example, in at least one embodiment, a PVA can process raw RADAR data (e.g., using a 4D fast Fourier transform) to provide processed RADAR data. In at least one embodiment, a PVA is used for time-of-flight depth processing, e.g., by processing raw time-of-flight data to provide processed time-of-flight data.

[0210] In at least one embodiment, a DLA can be used to run any type of network to enhance control and driving safety, including, for example and without limitation, a neural network that outputs a confidence measure for each object detection. In at least one embodiment, a confidence can be expressed or interpreted as a probability, or provide a relative “weight” of each detection compared to other detections. In at least one embodiment, a confidence enables a system to make further decisions about which detections should be treated as true positive detections rather than false positive detections. For example, in at least one embodiment, a system can set a threshold for confidence and treat only detections that exceed a threshold as true positive detections. In embodiments using automatic emergency braking (“AEB”) systems, false positive detections would result in a vehicle automatically performing an emergency brake, which is obviously undesirable. In at least one embodiment, a highly confident detection can be considered a trigger for AEB. In at least one embodiment, a DLA can run a neural network for regression of a confidence value. In at least one embodiment, a neural network can take as its input at least some subset of parameters, such as a bounding box size, a ground plane estimate obtained (e.g., from another subsystem), output from a MU sensor, a vehicle bearing, distance, 3D position estimate of an object obtained from a neural network and / or other sensors (e.g., LIDAR sensors or RADAR sensors), etc.

[0211] In at least one embodiment, one or more SoCs can include one or more data stores (e.g., memory). In at least one embodiment, a data store can be on-chip memory of a SoC, which can store neural networks to be executed on a GPU and / or a DLA. In at least one embodiment, a data store can be large enough in capacity to store multiple instances of a neural network for redundancy and safety. In at least one embodiment, a data store can include an L2 or L3 cache.

[0212] In at least one embodiment, one or more SoCs can include any number of processors (e.g., embedded processors). In at least one embodiment, a processor can include a boot and power management processor that can be a dedicated processor and subsystem to handle boot power and management functions and related security enforcement. In at least one embodiment, a boot and power management processor can be part of a SoC boot sequence and can provide run-time power management services. In at least one embodiment, a boot power and management processor can provide clock and voltage programming, assist system low power state transitions, manage SoC heat and temperature sensors, and / or manage SoC power states. In at least one embodiment, each temperature sensor can be implemented as a ring oscillator whose frequency is proportional to temperature, and a SoC can use a ring oscillator to detect temperature of a CPU, GPU, and / or an accelerator. In at least one embodiment, if a temperature is determined to exceed a threshold, a boot and power management processor can enter a temperature fault routine and put a SoC into a lower power state and / or put a vehicle into a driver safe stop mode (e.g., bring a vehicle to a safe stop).

[0213] In at least one embodiment, a processor can further include a set of embedded processors that can function as an audio processing engine. In at least one embodiment, an audio processing engine can be an audio subsystem that is capable of comprehensive hardware support for multi-channel audio through a number of interfaces, as well as a broad and flexible range of audio I / O interfaces. In at least one embodiment, an audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0214] In at least one embodiment, a processor can further include an always-on processor engine that can provide necessary hardware features to support low-power sensor management and wake-up use cases. In at least one embodiment, an always-on processor engine can include, but is not limited to, a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0215] In at least one embodiment, processor can also include a safety cluster engine that includes, without limitation, a dedicated processor subsystem to handle automotive application safety management. In at least one embodiment, safety cluster engine can include, without limitation, two or more processor cores, tightly coupled RAM, support peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, in at least one embodiment, two or more cores can operate in lockstep mode and function as a single core with comparison logic to detect any differences between their operations. In at least one embodiment, processor can also include a real-time camera engine that can include, without limitation, a dedicated processor subsystem to handle real-time camera management. In at least one embodiment, processor can also include a high dynamic range signal processor that can include, without limitation, an image signal processor that is a hardware engine that is part of a camera processing pipeline.

[0216] In at least one embodiment, processor can include a video image compositor that can be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required for video playback applications to produce the final image of a player window. In at least one embodiment, video image compositor can perform lens distortion correction on wide-angle cameras, surround cameras, and / or in-cabin monitoring camera sensors. In at least one embodiment, in-cabin monitoring camera sensors are preferably monitored by a neural network running on another instance of SoC, configured to identify in-cabin events and respond accordingly. In at least one embodiment, in-cabin systems can perform, without limitation, lip reading to activate cellular service and place a phone call, dictate an email, change a destination of a vehicle, activate or change an infotainment system and settings of a vehicle, or provide voice-activated web surfing. In at least one embodiment, certain functionality is available to a driver when a vehicle is operating in autonomous mode, otherwise disabled.

[0217] In at least one embodiment, video image compositor can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in at least one embodiment where motion occurs in a video, spatial information is appropriately weighted by noise reduction, reducing a weight of information provided by adjacent frames. In at least one embodiment, where an image or portion of an image does not include motion, temporal noise reduction performed by video image compositor can use information from a previous image to reduce noise in a current image.

[0218] In at least one embodiment, a video image compositor can also be configured to perform stereoscopic correction on input stereoscopic lens frames. In at least one embodiment, a video image compositor can also be used for user interface composition when an operating system desktop is in use, and does not require a GPU to continuously render new surfaces. In at least one embodiment, a video image compositor can be used to offload a GPU to improve performance and responsiveness when a GPU is powered on and actively rendering 3D.

[0219] In at least one embodiment, one or more SoCs can also include a mobile industry processor interface (“MIPI”) camera serial interface for receiving video and input from a camera, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functionality. In at least one embodiment, one or more SoCs can also include one or more input / output controllers that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0220] In at least one embodiment, one or more SoCs can also include a wide range of peripheral interfaces to enable communication with peripherals, audio encoders / decoders (“codecs”), power management, and / or other devices. SoCs can be used to process data from cameras (e.g., over a gigabit multimedia serial link and an Ethernet connection), sensors (e.g., LIDAR sensors, RADAR sensors, etc., which can be over an Ethernet connection), data from a bus (e.g., speed of a vehicle, steering wheel position, etc.), data from GNSS sensors (e.g., over an Ethernet or CAN bus connection), etc. In at least one embodiment, one or more SoCs can also include dedicated high-performance mass storage controllers that can include their own DMA engines and can be used to free up CPUs from routine data management tasks.

[0221] In at least one embodiment, SoCs can be end-to-end platforms with flexible architectures spanning automation levels 3-5, providing a comprehensive functional safety architecture that leverages and effectively utilizes the diversity and redundancy of computer vision and ADAS technologies for flexible, reliable driving software stacks and deep learning tools. In at least one embodiment, SoCs can be faster, more reliable, and even more energy efficient and space saving than traditional systems. For example, in at least one embodiment, accelerators, in conjunction with CPUs, GPUs, and data storage, can provide a fast, efficient platform for level 3-5 autonomous vehicles.

[0222] In at least one embodiment, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages such as C programming language to perform a variety of processing algorithms across a multitude of vision data. However, in at least one embodiment, CPUs often cannot meet performance requirements of many computer vision applications, such as those related to execution time and power consumption. In at least one embodiment, many CPUs are unable to execute complex object detection algorithms in real-time, which are used in vehicle ADAS applications and practical 3-5 level autonomous vehicles.

[0223] Embodiments described herein allow multiple neural networks to be executed simultaneously and / or sequentially, and allow results to be combined together to enable 3-5 level autonomous driving functionality. For example, in at least one embodiment, a CNN executed on a DLA or discrete GPU (e.g., GPU) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs that a neural network has not been specifically trained for. In at least one embodiment, a DLA can also include a neural network capable of recognizing, interpreting, and providing a semantic understanding of a symbol and passing that semantic understanding to a path planning module running on a CPU Complex.

[0224] In at least one embodiment, multiple neural networks can be run simultaneously, such as 3, 4, or 5 levels of driving. For example, in at least one embodiment, a warning sign consisting of a “warning: flashing light indicates icy conditions” and a light can be interpreted independently or collectively by multiple neural networks. In at least one embodiment, the sign itself can be recognized as a traffic sign by a first deployed neural network (e.g., a neural network that has been trained), and the text “flashing light indicates icy conditions” can be interpreted by a second deployed neural network, which informs a vehicle’s path planning software (preferably executed on a CPU Complex) that icy conditions exist when a flashing light is detected. In at least one embodiment, the flashing light can be recognized by a third deployed neural network running over multiple frames, informing a vehicle’s path planning software that a flashing light exists (or does not exist). In at least one embodiment, all three neural networks can be run simultaneously, such as within a DLA and / or on a GPU.

[0225] In at least one embodiment, a CNN for facial recognition and owner recognition can use data from a camera sensor to recognize the presence of an authorized driver and / or owner of a vehicle. In at least one embodiment, when an owner approaches a driver’s door and turns on a light, a sensor processing engine that is always on can be used to unlock a vehicle, and in a safe mode, the vehicle can be disabled when the owner leaves the vehicle. In this way, a SoC provides security against theft and / or carjacking.

[0226] In at least one embodiment, a CNN for emergency vehicle detection and identification can use data from microphones to detect and identify emergency vehicle sirens. In at least one embodiment, a SoC uses a CNN to classify environmental and urban sounds, as well as to classify visual data. In at least one embodiment, a CNN running on a DLA is trained to identify a relative closing speed of an emergency vehicle (e.g., by using Doppler effect). In at least one embodiment, a CNN can also be trained to identify emergency vehicles specific to a local region in which a vehicle is operating, as identified by GNSS sensors. In at least one embodiment, when operating in Europe, a CNN will seek to detect European sirens, when in the United States, a CNN will seek to identify only North American sirens. In at least one embodiment, once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, slow vehicle down, pull to the side of the road, stop and / or idle vehicle until emergency vehicle passes, with assistance of ultrasonic sensors.

[0227] In at least one embodiment, a vehicle can include a CPU (e.g., discrete CPU or dCPU) that can be coupled to a SoC via a high-speed interconnect (e.g., PCIe). For example, in at least one embodiment, a CPU can include an X86 processor. A CPU can be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and a SoC, and / or monitoring a status and health of a controller and / or an on-chip infotainment system (“infotainment SoC”), for example.

[0228] In at least one embodiment, a vehicle can include a GPU (e.g., discrete GPU or dGPU) that can be coupled to a SoC via a high-speed interconnect (e.g., NVLINK by NVIDIA). In at least one embodiment, a GPU can provide additional artificial intelligence functionality, such as by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on input from sensors of a vehicle (e.g., sensor data).

[0229] In at least one embodiment, vehicle can further include a network interface that can include, without limitation, a wireless antenna (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). In at least one embodiment, network interface can be used to enable wireless connectivity over the Internet with a cloud (e.g., with a server and / or other network devices), with other vehicles, and / or with computing devices (e.g., a client device of a passenger). In at least one embodiment, to communicate with other vehicles, a direct link can be established between vehicle and other vehicles and / or an indirect link can be established (e.g., across a network and over the Internet). In at least one embodiment, a direct link can be provided using a vehicle-to-vehicle communication link. An inter-vehicle communication link can provide vehicle with information about vehicles in a vicinity of vehicle (e.g., vehicles in front of, to the side of, and / or behind vehicle). In at least one embodiment, the foregoing functionality can be part of a cooperative adaptive cruise control functionality of vehicle.

[0230] In at least one embodiment, network interface can include a SoC that provides modulation and demodulation functionality and enables controller to communicate over a wireless network. In at least one embodiment, network interface can include a radio frequency front end to upconvert from baseband to radio frequency and downconvert from radio frequency to baseband. In at least one embodiment, frequency conversion can be performed in any technically feasible way. For example, frequency conversion can be performed through well-known processes and / or using a superheterodyne process. In at least one embodiment, radio frequency front end functionality can be provided by a separate chip. In at least one embodiment, network interface can include wireless functionality to communicate over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocol.

[0231] In at least one embodiment, vehicle can also include a data store that can include, without limitation, off-chip (e.g., off-chip SoC) storage. In at least one embodiment, data store can include, without limitation, one or more storage elements including RAM, SRAM, dynamic random access memory (“DRAM”), video random access memory (“VRAM”), flash memory, hard disks, and / or other components and / or devices that can store at least one bit of data.

[0232] In at least one embodiment, vehicle can further include a GNSS sensor (e.g., a GPS and / or assisted GPS sensor) to assist in mapping, perception, occupancy grid generation, and / or path planning functionality. In at least one embodiment, any number of GNSS sensors can be used, including, for example and without limitation, using a GPS with an Ethernet-to-serial (e.g., RS-232) bridge using a USB connector.

[0233] In at least one embodiment, vehicle can further include RADAR sensors. RADAR sensors can be used by vehicle for long-range vehicle detection, even in darkness and / or adverse weather conditions. In at least one embodiment, RADAR functional safety level can be ASIL B. RADAR sensors can use CAN and / or FlexRay bus (e.g., to transmit data generated by RADAR sensors) for control and access to object tracking data, and in certain examples can access Ethernet for access to raw data. In at least one embodiment, multiple RADAR sensor types can be used. For example and without limitation, RADAR sensors can be suitable for front, rear, and side RADAR use. In at least one embodiment, one or more RADAR sensors are pulse Doppler RADAR sensors.

[0234] In at least one embodiment, RADAR sensors can include different configurations, such as long-range with narrow field of view, short-range with wide field of view, short-range side coverage, etc. In at least one embodiment, long-range radar can be used for adaptive cruise control functionality. In at least one embodiment, long-range radar system can provide a wide field of view, such as over 250 m range, achieved through two or more independent scans. In at least one embodiment, RADAR sensors can help distinguish between static and moving objects, and can be used by ADAS system for emergency brake assist and forward collision warning. Sensors included in long-range radar system can include, without limitation, a single-base multi-mode radar with multiple (e.g., six or more) fixed radar antennas and high-speed CAN and FlexRay interfaces. In at least one embodiment, using six antennas, a central four antennas can create focused beam patterns aimed at recording the vehicle’s surroundings at higher speeds while minimizing interference from adjacent lane traffic. In at least one embodiment, other two antennas can expand the field of view, which can quickly detect vehicles entering or leaving the vehicle’s lane.

[0235] In at least one embodiment, as an example, mid-range radar system can include a range of up to 160 m (front) or 80 m (rear), and up to 42 degrees (front) or 150 degrees (rear). In at least one embodiment, short-range radar system can include, without limitation, any number of radar sensors designed to be mounted at both ends of rear bumper. When mounted at both ends of rear bumper, in at least one embodiment, RADAR sensor system can produce two beams that continuously monitor the vehicle’s rear and sides for blind spots. In at least one embodiment, short-range radar system can be used in ADAS system for blind spot detection and / or lane change assist.

[0236] In at least one embodiment, a vehicle can also include ultrasonic sensors. Ultrasonic sensors can be positioned in front, rear, and / or sides of a vehicle, can be used for parking assistance and / or to create and update an occupancy grid. In at least one embodiment, multiple ultrasonic sensors can be used, and different ultrasonic sensors can be used for different detection ranges (e.g., 2.5m, 4m). In at least one embodiment, ultrasonic sensors can operate at a functional safety level of ASIL B.

[0237] In at least one embodiment, a vehicle can include LIDAR sensors. LIDAR sensors can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, LIDAR sensors can be functional safety level ASIL B. In at least one embodiment, a vehicle can include multiple LIDAR sensors (e.g., two, four, six, etc.) that can use Ethernet (e.g., provide data to a Gigabit Ethernet switch).

[0238] In at least one embodiment, LIDAR sensors can be capable of providing a list of objects and their distances for a 360-degree field of view. In at least one embodiment, for example, a commercially available LIDAR sensor can have an advertised range of approximately 100m, with a precision of 2cm-3cm, and support, for example, a 100Mbps Ethernet connection. In at least one embodiment, one or more flush LIDAR sensors can be used. In such embodiments, LIDAR sensors can be implemented as small devices that can be embedded in front, rear, sides, and / or corners of a vehicle. In at least one embodiment, LIDAR sensors, in such embodiments, can provide up to 120 degrees horizontal and 35 degrees vertical field of view, with a range of 200m even for low reflectivity objects. In at least one embodiment, front-facing LIDAR sensors can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0239] In at least one embodiment, a laser radar technology can also be used, such as a 3D flash lidar. A 3D flash lidar uses a laser flash as a transmission source that illuminates a vehicle’s surroundings, up to about 200 m away. In at least one embodiment, a flash lidar unit includes, without limitation, a receiver that records laser pulse transmission time and reflected light on each pixel, which in turn corresponds to a distance from a vehicle to an object. In at least one embodiment, a flash lidar can allow for a highly accurate and distortion-free image of a surrounding environment to be generated at each laser flash. In at least one embodiment, four flash lidar sensors can be deployed, one on each side of a vehicle. In at least one embodiment, a 3D flash lidar system includes, without limitation, a solid-state 3D staring array lidar camera that has no moving parts other than a fan (e.g., a non-scanning lidar device). In at least one embodiment, a flash lidar device can use a class I (eye-safe) laser pulse of 5 nanoseconds per frame, and can capture reflected laser in the form of a 3D range point cloud and co-registered intensity data.

[0240] In at least one embodiment, a vehicle can further include an IMU sensor. In at least one embodiment, an IMU sensor can be located at a center of a rear axle of a vehicle, in at least one embodiment. In at least one embodiment, an IMU sensor can include, without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In at least one embodiment, such as in a six-axis application, an IMU sensor can include, without limitation, an accelerometer and a gyroscope. In at least one embodiment, such as in a nine-axis application, an IMU sensor can include, without limitation, an accelerometer, a gyroscope, and a magnetometer.

[0241] In at least one embodiment, an IMU sensor can be implemented as a miniature, high-performance GPS-aided inertial navigation system (“GPS / INS”) that combines micro-electromechanical systems (“MEMS”) inertial sensors, high-sensitivity GPS receivers, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. In at least one embodiment, an IMU sensor can enable a vehicle to estimate heading by directly observing and correlating changes in velocity from a GPS to an IMU sensor without requiring input from a magnetic sensor. In at least one embodiment, an IMU sensor and a GNSS sensor can be combined in a single integrated unit.

[0242] In at least one embodiment, a vehicle can include microphones placed in and / or around a vehicle. In at least one embodiment, microphones can be used for emergency vehicle detection and identification, among other things.

[0243] In at least one embodiment, a vehicle can also include any number of camera types including stereo cameras, wide-view cameras, infrared cameras, surround cameras, long-range cameras, mid-range cameras, and / or other camera types. In at least one embodiment, cameras can be used to capture image data around an entire periphery of a vehicle. In at least one embodiment, a type of camera used depends on a vehicle. In at least one embodiment, any combination of camera types can be used to provide necessary coverage around a vehicle. In at least one embodiment, a number of cameras can vary depending on an embodiment. For example, in at least one embodiment, a vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, or other number of cameras. As an example but not by way of limitation, cameras can support Gigabit Multimedia Serial Link (“GMSL”) and / or Gigabit Ethernet. In at least one embodiment, each camera is described in greater detail earlier herein.

[0244] In at least one embodiment, a vehicle can also include a vibration sensor. In at least one embodiment, a vibration sensor can measure vibrations of a component of a vehicle, such as an axle. For example, in at least one embodiment, a change in vibration can be indicative of a change in a road surface. In at least one embodiment, when two or more vibration sensors are used, a difference between vibrations can be used to determine a friction or slip of a road surface (e.g., when a vibration difference is between a power driven axle and a freely rotating axle).

[0245] In at least one embodiment, a vehicle can include an ADAS system. In some examples, an ADAS system can include, without limitation, a SoC. In at least one embodiment, an ADAS system can include, without limitation, an autonomous / adaptive / automatic cruise control (“ACC”) system, a cooperative adaptive cruise control (“CACC”) system, a forward collision warning (“FCW”) system, an automatic emergency braking (“AEB”) system, a lane departure warning (“LDW”) system, a lane-keeping assist (“LKA”) system, a blind-spot warning (“BSW”) system, a rear cross-traffic warning (“RCTW”) system, a collision warning (“CW”) system, a lane centering (“LC”) system, and / or any number and combination of other systems, features, and / or functionality.

[0246] In at least one embodiment, an ACC system can use RADAR sensors, LIDAR sensors, and / or any number of cameras. In at least one embodiment, an ACC system can include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, a longitudinal ACC system monitors and controls distance to a vehicle directly ahead of the vehicle and automatically adjusts the vehicle’s speed to maintain a safe distance from the vehicle ahead. In at least one embodiment, a lateral ACC system performs distance keeping and suggests lane changes for the vehicle when necessary. In at least one embodiment, lateral ACC is relevant to other ADAS applications such as LC and CW.

[0247] In at least one embodiment, a CACC system uses information from other vehicles that can be received from other vehicles via a wireless link or indirectly through a network connection (e.g., over the Internet) via a network interface and / or wireless antenna. In at least one embodiment, a direct link can be provided by a vehicle-to-vehicle (“V2V”) communication link, while an indirect link can be provided by an infrastructure-to-vehicle (“I2V”) communication link. Generally speaking, a V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of and in the same lane as the vehicle), while an I2V communication concept provides information about traffic further ahead. In at least one embodiment, a CACC system can include one or both of I2V and V2V sources of information. In at least one embodiment, given information about vehicles ahead of a given vehicle, a CACC system can be more reliable, and it has potential to improve traffic flow smoothness and reduce road congestion.

[0248] In at least one embodiment, an FCW system is designed to alert a driver to a hazard so that the driver can take corrective action. In at least one embodiment, an FCW system uses a front-facing camera and / or RADAR sensors that are coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or a vibrating component. In at least one embodiment, an FCW system can provide a warning, for example, in the form of a sound, a visual warning, a vibration, and / or a quick braking pulse.

[0249] In at least one embodiment, an AEB system detects an impending forward collision with another vehicle or other object and can automatically apply brakes if a driver does not take corrective action within a specified time or distance parameter. In at least one embodiment, an AEB system can use front-facing camera and / or RADAR sensors coupled to dedicated processors, DSPs, FPGAs, and / or ASICs. In at least one embodiment, when an AEB system detects a hazard, an AEB system will typically first alert a driver to take corrective action to avoid a collision, and if a driver does not take corrective action, an AEB system can automatically apply brakes in an effort to prevent or at least minimize the impact of a predicted collision. In at least one embodiment, an AEB system can include technologies such as dynamic brake support and / or crash imminent braking.

[0250] In at least one embodiment, an LDW system provides visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert a driver when a vehicle crosses lane markings. In at least one embodiment, an LDW system does not activate when a driver indicates an intentional departure from a lane by activating a turn signal. In at least one embodiment, an LDW system can use a front-facing camera coupled to dedicated processors, DSPs, FPGAs, and / or ASICs that are electrically coupled to driver feedback such as display, speaker, and / or vibrator components. In at least one embodiment, an LKA system is a variation of an LDW system. If a vehicle begins to depart from a lane, an LKA system provides steering input or braking to correct the vehicle.

[0251] In at least one embodiment, a BSW system detects and warns a driver when a vehicle is in a blind spot zone of a car. In at least one embodiment, a BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. In at least one embodiment, a BSW system can provide additional warnings when a driver uses a turn signal. In at least one embodiment, a BSW system can use rear-side cameras and / or radar sensors coupled to dedicated processors, DSPs, FPGAs, and / or ASICs that are electrically coupled to driver feedback such as display, speaker, and / or vibrator components.

[0252] In at least one embodiment, a RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside of a rear camera range when a vehicle is backing up. In at least one embodiment, a RCTW system includes an AEB system to ensure that vehicle brakes are applied to avoid a collision. In at least one embodiment, a RCTW system can use one or more rear-facing radar sensors coupled to dedicated processors, DSPs, FPGAs, and / or ASICs that are electrically coupled to driver feedback such as display, speaker, and / or vibrator components.

[0253] In at least one embodiment, traditional ADAS systems can be prone to false positive results, which can be annoying and distracting to a driver, but are typically not catastrophic because traditional ADAS systems alert the driver and allow the driver to decide whether the safety condition is actually present and take appropriate action. In at least one embodiment, in the event of a result conflict, the vehicle itself decides whether to pay attention to results from a primary computer or a secondary computer (e.g., a first controller or a second controller). For example, in at least one embodiment, an ADAS system can be a secondary and / or auxiliary computer for providing perception information to a secondary computer plausibility module. In at least one embodiment, a secondary computer plausibility monitor can run redundant different software on hardware components to detect faults in perception and dynamic driving tasks. In at least one embodiment, output from an ADAS system can be provided to a supervisory MCU. In at least one embodiment, if output from a primary computer and a secondary computer conflict, the supervisory MCU determines how to reconcile the conflict to ensure safe operation.

[0254] In at least one embodiment, a primary computer can be configured to provide a confidence score to a supervisory MCU indicating a primary computer’s confidence in a selected result. In at least one embodiment, if a confidence score exceeds a threshold, the supervisory MCU can follow the primary computer’s indication regardless of whether a secondary computer provides a conflicting or inconsistent result. In at least one embodiment, in the event that a confidence score does not satisfy a threshold and a primary computer and a secondary computer indicate different results (e.g., conflict), the supervisory MCU can arbitrate between the computers to determine an appropriate result.

[0255] In at least one embodiment, a supervisory MCU can be configured to run one or more neural networks trained and configured to determine conditions under which a secondary computer provides false alarms based at least in part on output from a primary computer and a secondary computer. In at least one embodiment, a neural network in a supervisory MCU can learn when to trust output of a secondary computer and when not to. For example, in at least one embodiment, when a secondary computer is a radar-based FCW system, one or more neural networks in a supervisory MCU can learn when the FCW system identifies metal objects that are not actually dangerous, such as drain grates or manhole covers that trigger an alert. In at least one embodiment, when a secondary computer is a camera-based LDW system, a neural network in a supervisory MCU can learn to override LDW when a bicyclist or pedestrian is present and lane departure is actually the safest maneuver. In at least one embodiment, a supervisory MCU can include at least one of a DLA or a GPU suitable for running a neural network with associated memory. In at least one embodiment, a supervisory MCU can include and / or be included as a component of a SoC.

[0256] In at least one embodiment, an ADAS system can include a secondary computer that performs ADAS functions using traditional computer vision rules. In at least one embodiment, a secondary computer can use classic computer vision rules (if-then), and the presence of a neural network in a supervisory MCU can improve reliability, safety, and performance. For example, in at least one embodiment, diversification and intentional non-identity make the overall system more fault-tolerant, especially to faults caused by software (or software-hardware interface) functions. For example, in at least one embodiment, if there is a software bug or error in software running on a primary computer, and different software code running on a secondary computer provides the same overall result, then a supervisory MCU can have more confidence that the overall result is correct, and an error in software or hardware on the primary computer does not result in a significant error.

[0257] In at least one embodiment, an output of an ADAS system can be fed to a perception block of a primary computer and / or a dynamic driving task block of a primary computer. For example, in at least one embodiment, if an ADAS system indicates that a forward collision warning is issued due to an object immediately ahead, then a perception block can use that information in identifying the object. In at least one embodiment, a secondary computer can have its own neural network that is trained and thus reduces a risk of false positives, as described herein.

[0258] In at least one embodiment, vehicle can also include an infotainment SoC (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, in at least one embodiment, infotainment system can not be a SoC and can include, without limitation, two or more discrete components. In at least one embodiment, infotainment SoC can include, without limitation, a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., navigation systems, rear parking space assistance, radio data system, vehicle related information (e.g., fuel level, total distance traveled, brake fluid level, oil level, doors open / close, air filter information, etc.)) to a vehicle. For example, infotainment SoC can include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, car, car entertainment, WiFi, steering wheel audio controls, hands-free voice controls, heads-up display (“HUD”), HMI display, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. In at least one embodiment, infotainment SoC can also be used to provide information (e.g., visually and / or audibly) to a user of a vehicle, such as information from ADAS systems, autonomous driving information (e.g., planned vehicle operations, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0259] In at least one embodiment, infotainment SoC can include any number and type of GPU functionality. In at least one embodiment, infotainment SoC can communicate with other devices, systems, and / or components of a vehicle over a bus (e.g., CAN bus, Ethernet, etc.). In at least one embodiment, infotainment SoC can be coupled to a supervisory MCU such that GPU of infotainment system can perform some autonomous driving functions in the event of a failure of a primary controller (e.g., a primary and / or backup computer of a vehicle). In at least one embodiment, infotainment SoC can place a vehicle in a driver safe park mode, as described herein.

[0260] In at least one embodiment, vehicle can also include an instrument cluster (e.g., digital dashboard, electronic dashboard, digital instrument cluster, etc.). In at least one embodiment, instrument cluster can include, without limitation, a controller and / or supercomputer (e.g., discrete controller or supercomputer). In at least one embodiment, instrument cluster can include, without limitation, any number and combination of gauges such as speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seat belt warning light, park brake warning light, engine malfunction light, auxiliary restraint system (e.g., airbag) information, lighting controls, safety system controls, navigation information, etc. In some examples, information can be displayed and / or shared between infotainment SoC and instrument cluster. In at least one embodiment, instrument cluster can be included as part of infotainment SoC, and vice versa.

[0261] Inference and / or training logic 715 are used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 715 are described in greater detail below in conjunction with FIGS. 7B and 7C. Figure 7A In at least one embodiment, inference and / or training logic 715 can be used in system to infer or predict operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0262] A system for communicating between cloud-based servers and autonomous vehicles, according to at least one embodiment. In at least one embodiment, system can include, without limitation, servers, a network, and any number and type of vehicles, including vehicles. In at least one embodiment, servers can include, without limitation, a plurality of GPUs (A)-(H) (collectively referred to herein as GPUs), PCIe switches (A)-(D) (collectively referred to herein as PCIe switches), and / or CPUs (A)-(B) (collectively referred to herein as CPUs). GPUs, CPUs, and PCIe switches can be interconnected with high-speed interconnects such as, without limitation, NVLink interface developed by NVIDIA and / or PCIe connections. In at least one embodiment, GPUs are connected via NVLink and / or NVSwitch SoC and GPUs and PCIe switches are connected via PCIe interconnects. In at least one embodiment, while eight GPUs, two CPUs, and four PCIe switches are illustrated, this is not intended to be limiting. In at least one embodiment, each server can include any number of GPUs, CPUs, and / or PCIe switches in any combination. For example, in at least one embodiment, servers can each include eight, sixteen, thirty-two, and / or more GPUs.

[0263] In at least one embodiment, a server can receive image data representing an image from a vehicle over a network, the image showing an unexpected or changing road condition, such as a road construction that has recently begun. In at least one embodiment, a server can transmit a neural network, an updated neural network, and / or map information, including but not limited to information regarding traffic and road conditions, to a vehicle over a network. In at least one embodiment, updates to map information can include, but are not limited to, updates to high definition maps, such as information regarding construction sites, potholes, detours, flooding, and / or other obstacles. In at least one embodiment, a neural network, an updated neural network, and / or map information can have resulted from new training and / or experience represented in data received from any number of vehicles in an environment, and / or based at least in part on training performed at a data center (e.g., using a server and / or other servers).

[0264] In at least one embodiment, a server can be used to train a machine learning model (e.g., a neural network) based at least in part on training data. In at least one embodiment, training data can be generated by vehicles, and / or can be generated in simulations (e.g., using a game engine). In at least one embodiment, any amount of training data is labeled (e.g., a neural network associated benefits from supervised learning) and / or undergoes other pre-processing. In at least one embodiment, any amount of training data is not labeled and / or pre-processed (e.g., where a neural network associated does not require supervised learning). In at least one embodiment, once a machine learning model is trained, a machine learning model can be used by vehicles (e.g., transmitted to vehicles over a network, and / or a machine learning model can be used by a server to remotely monitor vehicles.

[0265] In at least one embodiment, a server can receive data from vehicles and apply the data to a latest real-time neural network for real-time intelligent inference. In at least one embodiment, a server can include a deep learning supercomputer and / or a GPU-driven specialized AI computer, such as a DGX and DGX Station machine developed by NVIDIA. However, in at least one embodiment, a server can include a deep learning infrastructure using a CPU-driven data center.

[0266] In at least one embodiment, a server’s deep learning infrastructure can be capable of fast, real-time inference and can use this capability to assess and vouch for the health of processors, software, and / or related hardware. For example, in at least one embodiment, a deep learning infrastructure can receive periodic updates from a vehicle, such as a sequence of images and / or objects that a vehicle has located in that sequence of images (e.g., via computer vision and / or other machine learning object classification techniques). In at least one embodiment, a deep learning infrastructure can run its own neural network to identify objects and compare them to objects identified by a vehicle, and if results do not match and the deep learning infrastructure concludes that AI in a vehicle is malfunctioning, a server can send a signal to a vehicle instructing a vehicle’s failsafe computer to assume control, notify passengers, and complete a safe parking operation.

[0267] In at least one embodiment, a server can include a GPU and one or more programmable inference accelerators (e.g., NVIDIA’s TensorRT 3). In at least one embodiment, a combination of GPU-powered servers and inference acceleration can make real-time responses possible. In at least one embodiment, servers powered by CPUs, FPGAs, and other processors can be used for inference, for example, in cases where performance is less critical. In at least one embodiment, inference and / or training logic 715 are used to perform one or more embodiments. Details regarding inference and / or training logic 715 are provided below in conjunction with FIGS. 7A and / or 7B. Figure 7A and / or 7B provide.

[0268] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the disclosure to the specific form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure, as defined in the appended claims.

[0269] Unless otherwise indicated or contradicted by context, the use of the terms "a" and "an" and "the" and similar referents in the context of describing the disclosed embodiments (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated or contradicted by context. The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (meaning "including, but not limited to") unless otherwise noted or contradicted by context. The term "connected" (when used without modification) is to be construed as partly or wholly encompassed, attached to, or joined together, even if there are some intervening items. Unless otherwise indicated herein, a reference to a range of values herein is merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, and each separate value is incorporated in the specification as if it were individually recited herein. Unless otherwise indicated or contradicted by context, the use of the term "subset" (e.g., "set of items") or "subcollection" is to be construed as a non-empty set of one or more members. Also, unless otherwise indicated or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but rather the subset and the corresponding set can be equal.

[0270] Unless explicitly indicated otherwise or contradicted by context, conjunction language such as phrases in the form "at least one of A, B, and C" or "at least one of A, B, and C" is to be construed in context as generally used to mean that the item, term, etc. can be A or B or C, or any non-empty subset of the set of A and B and C. For example, in the illustrative example of a set having three members, the conjunction phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunction language is not generally intended to imply that certain embodiments require the existence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise indicated or contradicted by context, the term "plurality" denotes a plural state (e.g., "a plurality of items" denotes a plurality of items). The number of items in a plurality of items is at least two, but can be more if explicitly indicated or indicated by context. Furthermore, unless otherwise indicated or clear from context, the phrase "based on" means "based at least in part on" rather than "based solely on."

[0271] The operations of a process described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. In at least one embodiment, a process, such as those described herein (or variations and / or combinations thereof) is performed under the control of one or more computer systems configured with executable instructions to perform the operations of the process, and the process is implemented as code (e.g., executable instructions, one or more computer programs or one or more applications) that is collectively executed by a hardware or combination of hardware and / or software. In at least one embodiment, the code is stored on a computer-readable storage medium, such as a computer program product, which is readable by a computer system including one or more processors. In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electric or electromagnetic transmission) but includes non-transitory data storage circuitry (e.g., buffers, cache, and queues). In at least one embodiment, the code (e.g., executable instructions or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) having stored thereon executable instructions that, as a result of being executed by one or more processors of a computer system (i.e., as a result of being executed), cause the computer system to perform operations described herein. In at least one embodiment, a set of non-transitory computer-readable storage media includes multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media in the multiple non-transitory computer-readable storage media lack all of the code, with the multiple non-transitory computer-readable storage media collectively storing the entire code. In at least one embodiment, executable instructions are executed by different processors, e.g., a non-transitory computer-readable storage medium stores instructions and a main central processing unit (“CPU”) executes some instructions, while a graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of a computer system have separate processors and different processors execute different subsets of the instructions.

[0272] Accordingly, in at least one embodiment, a computer system is configured to implement one or more services that individually or collectively perform operations of processes described herein, and such a computer system is configured with applicable hardware and / or software to enable implementation of the operations. Moreover, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment is a distributed computer system including multiple devices operating in different manners such that the distributed computer system performs operations described herein and such that a single device does not perform all of the operations.

[0273] The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate embodiments of the disclosure and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0274] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0275] In the description and claims, the terms "coupled" and "connected," along with derivatives thereof, can be used. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, "connected" or "coupled" can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" can also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other.

[0276] Unless specifically stated otherwise, it can be appreciated that throughout the specification, terms such as "processing," "computing," "calculating," "determining," or the like, refer to the action and / or processes of a computer or computing system, or similar electronic

[0277] In a similar manner, the term "processor" can refer to any device or portion of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities such as tasks, threads, and intelligent agents that perform work over time. Likewise, each process can refer to multiple processes to sequentially or concurrently execute instructions, either continuously or intermittently. The terms "system" and "method" can be used interchangeably herein so long as a system can embody one or more methods and a method can be considered a system.

[0278] In this document, obtaining, accessing, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine can be referenced. Analog and digital data can be obtained, accessed, received, or inputted in a variety of ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In some implementations, the process of obtaining, accessing, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In another implementation, the process of obtaining, accessing, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an accessing entity via a computer network. Providing, outputting, transmitting, sending, or presenting analog or digital data can also be referenced. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be accomplished by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0279] Although the above discussion discloses example implementations of the described technology, other architectures can be utilized and are intended to fall within the scope of the present disclosure. Moreover, although a specific division of responsibilities has been defined above for purposes of discussion, various functions and responsibilities can be distributed and divided in different ways depending on circumstances.

[0280] Further, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: receiving a texture to be used in computing lighting data corresponding to one or more assets in an image to be rendered; determining, for individual pixel locations in the texture, an importance value corresponding to at least luminance of the respective pixel; determining, based at least in part on the importance values, a set of cumulative distribution functions (CDFs) for the texture; generating a geometric grid representing the texture using the set of CDFs; determining intersection points of one or more traced rays with the geometric grid, wherein points in the geometric grid corresponding to higher importance values have higher probabilities of intersection; and rendering the image by using, at least in part, pixels in the texture corresponding to the intersection points to represent lighting corresponding to the one or more assets.

2. The computer-implemented method of claim 1, wherein the importance values correspond to grayscale luminance values determined for respective pixels of the texture.

3. The computer-implemented method of claim 1, further comprising: randomly selecting paths for the one or more traced rays to intersect the geometric grid.

4. The computer-implemented method of claim 1, further comprising: generating the geometric grid using a series of triangular strips corresponding to rows or columns of pixels in the texture to be sampled.

5. The computer-implemented method of claim 1, further comprising: determining, from a set of joint probability distributions for the texture, a first set of marginal probabilities for an importance sample row of the texture and a second set of conditional probabilities for an importance sample column of the texture.

6. The computer-implemented method of claim 1, wherein determining at least the intersection points of the one or more traced rays with the geometric grid is performed using hardware acceleration.

7. The computer-implemented method of claim 1, wherein tracing of a single ray provides importance sampling of multiple dimensions of the geometric grid simultaneously.

8. The computer-implemented method of claim 1, wherein probability distribution functions used for Monte Carlo-based importance sampling are derived based at least in part on one or more partial derivatives of the geometric grid at one or more of the intersection points.

9. The computer-implemented method of claim 1, wherein the set of CDFs are determined based at least in part on probability distributions of the importance values for individual rows and columns of the texture.

10. An importance sampling system comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the importance sampling system to: determine importance values for pixel locations in an image; generate a geometric representation of the image using a set of cumulative distribution functions (CDFs), the set of CDFs determined using the importance values; determine intersection points of one or more rays with the geometric representation; and provide a set of pixel locations from the image corresponding to the intersection points. ​ ​ 11. The importance sampling system of claim 10, wherein the importance values correspond to grayscale luminance values determined for respective pixels of the image.

12. The importance sampling system of claim 10, wherein the instructions, if executed, further cause the importance sampling system to: generate the geometric representation using a series of triangular zones corresponding to rows or columns of pixels in the image.

13. The importance sampling system of claim 10, wherein the instructions, if executed, further cause the importance sampling system to: determine, from the set of joint probability distributions for the image, a first set of marginal probabilities for a row of importance samples of the image and a second set of conditional probabilities for a column of importance samples of the image.

14. The importance sampling system of claim 10, wherein determining at least the intersection of the one or more rays with the geometric representation is performed using hardware acceleration.

15. The importance sampling system of claim 10, wherein the importance sampling system comprises at least one of: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using edge devices; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

16. A non-transitory computer-readable storage medium comprising instructions that, if executed by one or more processors, cause the one or more processors to: receive a texture to be used for lighting one or more assets in an image to be rendered; for individual pixel locations in the texture, determine an importance value corresponding to at least luminance of the respective pixel; determine, based at least in part on the importance values, a set of cumulative distribution functions (CDFs) for the texture; generate a geometric grid representing the texture using the set of CDFs; determine intersection points of one or more tracing rays with the geometric grid, wherein points in the geometric grid corresponding to higher importance values have higher probabilities of intersection; and render the image by at least partially lighting the one or more assets using pixels in the texture corresponding to the intersection points.

17. The non-transitory computer-readable storage medium of claim 16, wherein the importance values correspond to grayscale luminance values determined for respective pixels of the texture.

18. The non-transitory computer-readable storage medium of claim 16, wherein the instructions, if executed, further cause the one or more processors to: generate the geometric grid using a series of triangular zones corresponding to rows or columns of pixels in the texture to be sampled.

19. The non-transitory computer-readable storage medium of claim 16, wherein the instructions, if executed, further cause the one or more processors to: determining, from the set of joint probability distributions of the texture, a first set of marginal probabilities of importance sample rows of the texture and a second set of conditional probabilities of importance sample columns of the texture.

20. The non-transitory computer-readable storage medium of claim 16, wherein at least determining the intersection of the one or more traced rays with the geometric grid is performed using hardware acceleration.

Images