Spatiotemporal noise masks and sampling using vectors for image processing and light transport simulation systems and applications

Abstract

The present disclosure relates to spatio-temporal noise masks and sampling using vectors for image processing and light transport simulation systems and applications, and in particular to apparatuses, systems, and techniques for generating blue noise masks for real-time image rendering and augmentation. In at least one embodiment, a vector-valued noise mask is generated and applied to one or more images to generate one or more augmented images for image processing (e.g., real-time image rendering). In at least one embodiment, the noise mask includes a vector value per pixel and is capable of handling the temporal domain (e.g., adding time to the spatial domain) to improve image quality when rendering images over multiple frames.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 196,116, filed June 2, 2021, entitled “Noise Masks for Image Processing,” the contents of which are hereby incorporated herein by reference in their entirety. Technical Field

[0003] At least one embodiment relates to processing resources for performing and facilitating real-time image rendering and enhancement. For example, a processor or computing system generates a blue noise texture that can process vector values, wherein the blue noise texture can be used for real-time image rendering and enhancement. Background Technology

[0004] Image processing techniques, such as imaging rendering and enhancement, can utilize significant memory, time, or computational resources, especially when the processing is performed in real time. The amount of memory, time, or computational resources used for image enhancement can be improved. Attached Figure Description

[0005] Figure 1 An example of a process for performing pixel swapping during blue noise mask generation, according to at least one embodiment, is shown;

[0006] Figure 2 The process for generating a blue noise mask according to at least one embodiment is shown;

[0007] Figure 3 A process for generating a blue noise mask including importance sampling, according to at least one embodiment, is illustrated;

[0008] Figure 4 A blue noise mask and its frequency distribution according to at least one embodiment are shown;

[0009] Figure 5 A blue noise mask using importance sampling and its frequency distribution are shown according to at least one embodiment;

[0010] Figure 6A An example of ambient masking used in conjunction with various noise masks is shown;

[0011] Figure 6B An example of environmental occlusion for important sampling is illustrated, according to one embodiment, using a combination of various noise masks.

[0012] Figure 7A The performance of various noise masks, including scalar noise masks that are converted into vector noise masks, is shown according to one embodiment.

[0013] Figure 7B The performance of various noise masks, including a layered noise mask, according to one embodiment is illustrated;

[0014] Figure 7C The performance of various noise masks, including a noise mask having multiple values ​​per pixel, is illustrated according to one embodiment.

[0015] Figure 8 A process for analyzing the temporal layering of a noise mask according to one embodiment is shown.

[0016] Figure 9A An example of a process for generating a blue noise mask that is optimal in both spatial and temporal use, according to at least one embodiment, is shown;

[0017] Figure 9B An example of a process for generating a three-dimensional mask for both spatial and temporal use for a framework, according to at least one embodiment, is shown;

[0018] Figure 10 Exemplary images are shown illustrating the use of a blue noise mask that is optimal in both spatial and temporal terms, according to at least one embodiment;

[0019] Figure 11 The above is illustrated according to at least one embodiment. Figure 3 The three types of blue noise masks mentioned in the text were compared using Fourier analysis to generate frequency results;

[0020] Figure 12 The convergence rate of the 1D function according to at least one embodiment is shown;

[0021] Figure 13 The DFT of a 2D projection of a 64x64x16x16 4D blue noise mask according to at least one embodiment is shown;

[0022] Figure 14 An autocorrelation image illustrating blue noise texture is shown according to at least one embodiment;

[0023] Figure 15 A 2Dx1D spatiotemporal blue noise mask with various sigma / axis is shown according to at least one embodiment;

[0024] Figure 16 It is shown that the generation time according to at least one embodiment is a function of the number of pixels in the blue noise mask and approximately follows y = x 2 A graph of the curve;

[0025] Figure 17Random transparency using various types of noise is shown according to at least one embodiment;

[0026] Figure 18 Convergence rates in random alpha of various types of noise according to at least one embodiment are shown;

[0027] Figure 19 The illustration shows color mixing prior to quantization of various types of noise to 1 bit per color channel according to at least one embodiment;

[0028] Figure 20 A graph showing the convergence rate in color mixing of various types of noise according to at least one embodiment is shown.

[0029] Figure 21 Four steps are shown in which noise is used to randomly offset the start portion of the ray travel for each pixel ray travel according to at least one embodiment;

[0030] Figure 22 A graph showing the convergence rate of light-traveling fog with various types of noise according to at least one embodiment is presented;

[0031] Figure 23 The diagram illustrates the use of noise to layer 16 samples of line segments for each pixel through a participating medium, according to at least one embodiment.

[0032] Figure 24 A graph showing the convergence rate of light-traveling fog with various types of noise according to at least one embodiment is presented;

[0033] Figure 25 The diagram illustrates, according to at least one embodiment, the use of two independent noise streams to generate the x and y components of a 2D vector mapped to a cosine-weighted hemisphere for each pixel's single ambient occlusion (AO) sample.

[0034] Figure 26 This illustrates how AO convergence relates to various types of noise according to at least one embodiment;

[0035] Figure 27 Images using one or more of a 2D blue noise mask, a 3D blue noise mask, a spatiotemporal blue noise mask, and a 2DGR blue noise mask according to at least one embodiment are shown.

[0036] Figure 28 An image using Sobol sequence offset is shown according to at least one embodiment;

[0037] Figure 29The Heitz & Belcour technique, according to at least one embodiment, uses staggered gradient noise and a stylized grayscale image for a noise pattern target.

[0038] Figure 30 A graph showing convergence in Monte Carlo integral, leakage integral, and convergent leakage integral according to at least one embodiment is shown;

[0039] Figure 31 This demonstrates how a threshold mask according to at least one embodiment can produce point sets of any density;

[0040] Figure 32 This illustrates how a set of threshold points, according to at least one embodiment, maintains its desired frequency on an axis group;

[0041] Figure 33 Five cumulative frames of pixels sampled from an image using a non-uniform importance map according to at least one embodiment are shown, such that pixels oriented toward the center are more likely to be sampled.

[0042] Figure 34 It is shown that white noise according to at least one embodiment may have redundant sampled pixels for each frame, and that spatial blue noise removes spatially redundant pixels over time, and 2Dx1D spatiotemporal blue noise also removes them over time;

[0043] Figure 35A The inference and / or training logic according to at least one embodiment is illustrated;

[0044] Figure 35B The inference and / or training logic according to at least one embodiment is illustrated;

[0045] Figure 36 The training and deployment of a neural network according to at least one embodiment are illustrated;

[0046] Figure 37 An example data center system according to at least one embodiment is shown;

[0047] Figure 38A A chip-level supercomputer according to at least one embodiment is illustrated;

[0048] Figure 38B A rack-mounted supercomputer according to at least one embodiment is illustrated;

[0049] Figure 38C A rack-mounted supercomputer according to at least one embodiment is shown;

[0050] Figure 38D A supercomputer at the entire system level according to at least one embodiment is shown;

[0051] Figure 39 This is a block diagram illustrating a computer system according to at least one embodiment;

[0052] Figure 40 This is a block diagram illustrating a computer system according to at least one embodiment;

[0053] Figure 41 A computer system according to at least one embodiment is shown;

[0054] Figure 42 A computer system according to at least one embodiment is shown;

[0055] Figure 43A A computer system according to at least one embodiment is shown;

[0056] Figure 43B A computer system according to at least one embodiment is shown;

[0057] Figure 43C A computer system according to at least one embodiment is shown;

[0058] Figure 43D A computer system according to at least one embodiment is shown;

[0059] Figure 43E and Figure 43F A shared programming model according to at least one embodiment is shown;

[0060] Figure 44 An exemplary integrated circuit and a related graphics processor according to at least one embodiment are shown.

[0061] Figure 45A and Figure 45B An exemplary integrated circuit and an associated graphics processor according to at least one embodiment are shown.

[0062] Figure 46A and Figure 46B Additional exemplary graphics processor logic according to at least one embodiment is shown;

[0063] Figure 47 A computer system according to at least one embodiment is shown;

[0064] Figure 48A A parallel processor according to at least one embodiment is shown;

[0065] Figure 48B A partitioning unit according to at least one embodiment is shown;

[0066] Figure 48CA processing cluster according to at least one embodiment is shown;

[0067] Figure 48D A graphics multiprocessor according to at least one embodiment is shown;

[0068] Figure 49 A multi-graphics processing unit (GPU) system according to at least one embodiment is illustrated;

[0069] Figure 50 A graphics processor according to at least one embodiment is shown;

[0070] Figure 51 It is a block diagram illustrating a processor microarchitecture for a processor according to at least one embodiment;

[0071] Figure 52 A deep learning application processor according to at least one embodiment is shown;

[0072] Figure 53 A block diagram of an example neuromorphic processor is shown according to at least one embodiment;

[0073] Figure 54 At least a portion of a graphics processor according to one or more embodiments is shown;

[0074] Figure 55 At least a portion of a graphics processor according to one or more embodiments is shown;

[0075] Figure 56 At least a portion of a graphics processor according to one or more embodiments is shown;

[0076] Figure 57 A block diagram of a graphics processing engine of a graphics processor is shown according to at least one embodiment;

[0077] Figure 58 It is a block diagram of at least a portion of a graphics processor core according to at least one embodiment;

[0078] Figure 59A and Figure 59B The diagram illustrates thread execution logic according to at least one embodiment, which includes an array of processing elements of a graphics processor core.

[0079] Figure 60 A parallel processing unit (“PPU”) according to at least one embodiment is shown;

[0080] Figure 61 A general-purpose processing cluster (“GPC”) according to at least one embodiment is illustrated;

[0081] Figure 62A memory partition unit of a parallel processing unit (“PPU”) according to at least one embodiment is shown;

[0082] Figure 63 A streaming multiprocessor according to at least one embodiment is illustrated;

[0083] Figure 64 This is an example data flow diagram of an advanced computing pipeline according to at least one embodiment;

[0084] Figure 65 This is a system diagram of an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment;

[0085] Figure 66 Example illustrations include an advanced computing pipeline 6510A for processing imaging data according to at least one embodiment;

[0086] Figure 67A Includes example data flow diagrams of virtual instruments supporting ultrasound equipment according to at least one embodiment;

[0087] Figure 67B Includes example data flow diagrams of virtual instruments supporting CT scanners according to at least one embodiment;

[0088] Figure 68A A data flow diagram of a process for training a machine learning model according to at least one embodiment is shown;

[0089] Figure 68B This is an example illustration of a client-server architecture that utilizes a pre-trained annotation model to enhance an annotation tool according to at least one embodiment;

[0090] Figure 69 A software stack of a programming platform according to at least one embodiment is shown;

[0091] Figure 70 The illustration shows an embodiment according to at least one of the embodiments. Figure 69 The CUDA implementation of the software stack;

[0092] Figure 71 The illustration shows an embodiment according to at least one of the embodiments. Figure 69 The ROCm implementation method of the software stack;

[0093] Figure 72 The illustration shows an embodiment according to at least one of the embodiments. Figure 69 The OpenCL implementation of the software stack;

[0094] Figure 73 Software supported by a programming platform according to at least one embodiment is shown;

[0095] Figure 74 A method for using at least one embodiment is shown. Figures 69-72 Compiled code executed on the programming platform;

[0096] Figure 75 A multimedia system according to at least one embodiment is shown;

[0097] Figure 76 A distributed system according to at least one embodiment is shown;

[0098] Figure 77 An oversampling neural network according to at least one embodiment is shown;

[0099] Figure 78 An architecture of an oversampling neural network according to at least one embodiment is shown;

[0100] Figure 79 An example of streaming using an oversampling neural network according to at least one embodiment is shown;

[0101] Figure 80 Examples of simulations using an oversampled neural network according to at least one embodiment are shown; and

[0102] Figure 81 An example of a device using an oversampling neural network according to at least one embodiment is shown. Detailed Implementation

[0103] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of at least one embodiment. However, it will be apparent to those skilled in the art that the inventive concept can be practiced without one or more of these specific details.

[0104] Blue noise masks (e.g., blue noise sampling masks, blue noise textures) are used in image rendering algorithms to provide random numbers at each pixel level, resulting in patterns of random noise that are perceptibly better than white noise. Blue noise masks are generally limited to high frequencies, making them more thoroughly removed with low-pass filters (making denoising with blurring much more effective).

[0105] Conventional techniques for generating noise masks for frames typically do not consider how time affects mask generation and application. Unfortunately, image rendering processes that do not incorporate consideration of pixel changes over time into the generation and application of noise masks (e.g., blue noise masks) can produce visually unpleasant images for the observer.

[0106] Therefore, the time axis should be considered in real-time image rendering. In some embodiments, sampling of the image based on texture improvements over time improves image quality when viewed in motion and can also be applied to temporal filtering methods such as Temporal Anti-aliasing (TAA), Deep Learning Supersampling (DLSS), and Intel's XeSS. Various methods exist for animate blue noise masks over time, but there is a trade-off between quality on the spatial axes (e.g., the image spatial axes, x and y axes) and quality on the time axis.

[0107] In one embodiment, vector spatiotemporal blue noise masks (e.g., vector-valued blue noise masks) are valuable in any animation scenario currently using blue noise textures because they are a solution to the problem of animating blue noise masks. In contrast to scalar spatiotemporal blue noise masks, vector spatiotemporal blue noise masks store the vector value (e.g., a vector) for each pixel and have a high-quality blue noise spectrum in space, while also having a blue noise spectrum over time.

[0108] In at least one embodiment, one or more circuits (which may be part of one or more processors in a computer system) generate a blue noise mask that stores a vector value for each pixel and is spatially and temporally optimally blue. One or more circuits (implementing software operations) can generate these vector-space-time blue noise masks as a set of N blue noise textures, such that each texture stores a vector value for each pixel, is a separate, suitable blue noise, and demonstrates the blue noise over time for each pixel. The suitable blue noise may contain a sufficient amount of higher frequencies and a low amount of lower frequencies.

[0109] In one embodiment, the vector spatiotemporal blue noise mask is created by modifying the Blue Noise Mixing and Sampling (BNDS) algorithm. The BNDS algorithm is used to generate an N-dimensional blue noise mask composed of vector values ​​(e.g., a blue noise mask with multidimensional values ​​as pixel values). In an embodiment, the BNDS algorithm can be modified to produce a noise pattern that simultaneously addresses desired spatial and temporal constraints for real-time image rendering. In one embodiment, the modified BNDS algorithm receives one or more images with pixel data, where the pixel data includes N-dimensional (e.g., three or more dimensions, one of which is time) data, where N represents the number of dimensions. In an embodiment, the multiple dimensions are represented by d. In an embodiment, the image data (e.g., image data) can be processed like a function (e.g., an image function), where input (e.g., pixel values ​​indicating location) produces output (e.g., pixel values ​​indicating color, intensity, reflectance properties). In some embodiments, the pixel data can be stored as any of the following: scalar values ​​(e.g., scalars, single values), vector values ​​(e.g., vectors, arrays of values), multidimensional values, N-dimensional values, N-dimensional vectors, d-dimensional vectors, and / or higher-dimensional values.

[0110] According to various embodiments, the techniques described herein are aimed at generating vector spatiotemporal blue noise masks for real-time image rendering and enhancement. For example, a vector spatiotemporal blue noise mask can be a three-dimensional mask, where two dimensions correspond to space (e.g., x and y coordinates) and one dimension corresponds to time. In embodiments, the vector spatiotemporal blue noise mask is used to benefit various applications or techniques of image rendering, including color mixing, random transparency, region light sampling, and volume rendering. Furthermore, the vector spatiotemporal blue noise mask can be applied to a variety of sampling techniques, including, for example, soft shadows and path tracing, random alpha, and color mixing.

[0111] The techniques described herein also pertain to generating vector spatiotemporal blue masks that can use vectors as pixel values ​​and can handle the temporal domain, for example, adding time to the spatial (image) domain to improve image quality when rendering an image across multiple frames (e.g., time) and multidimensional (e.g., more than one dimension) pixel values. In embodiments, the vector spatiotemporal blue mask can use vectors or unit vectors. In embodiments, the vector spatiotemporal blue noise mask is at least partially based on vector values ​​associated with pixels.

[0112] In embodiments, for image rendering, image enhancement can be achieved by integrating over multiple samples of each pixel across the coverage duration or other dimensions, while still spatially preserving the characteristics of the blue noise error. Human perception (and some computational evidence) performs a certain amount of implicit integration over time, especially at high frame rates, and these cases typically provide values ​​to a well-sampled pattern over time without any explicit filtering. Therefore, a two-dimensional (2D) blue noise pattern is used for each frame, which is well-distributed over time at each pixel and converges (e.g., quickly) for Monte Carlo integration (e.g., numerical integration using random numbers, using random sampling for integration). Accordingly, the techniques described herein relate to systems and methods for generating vector spatiotemporal blue noise masks based on vector values. In embodiments, the vector spatiotemporal blue noise mask is generalized to arbitrary dimensions for use in higher dimensions.

[0113] In at least one embodiment, a time slice corresponds to a specific moment or unit of time. More generally, a slice may also be a layer of a specific dimension or multiple dimensions (e.g., based on the coordinates of pixels falling within a plane having a specific dimension or multiple dimensions). In embodiments, the frame enables each two-dimensional (2D) slice of a 3D image (in the spatial domain) to be associated with blue noise properties, such that each pixel includes a one-dimensional sampling property in the temporal dimension. In embodiments, acquiring a slice from a higher spatial dimension results in an image in a lower-dimensional space. Thus, a 2D slice may be an image taken from a 3D image or an object. In one embodiment, a 3D image over time may be represented by 2D slices (e.g., an XY plane in the spatial domain) and a Z-axis representing the temporal domain.

[0114] In an embodiment, the BNDS algorithm has an energy function used to find regions of blank space in the image to place the next pixel within. In another embodiment, the energy function determines which one or more vector values ​​(e.g., position coordinates, on or off values, color values, intensity, angle, direction) should be assigned to which pixels to better represent blue noise. In one embodiment, one or more modifications can be made to the energy function of the BNDS algorithm. This energy function makes it so that pixels influence each other in the energy field only if they come from the same slice (e.g., frame) or if pixels are the same pixel at different time points. Thus, each 2D slice of the vector-valued blue noise mask, which is also blue-3D blue noise in both space and time, will be good 2D blue noise, and each pixel corresponding to the blue noise mask will become 1D blue noise over time. It is important to note that if a single 2D blue noise texture is used for each slice, then each pixel will obtain the same result, and no new samples may be provided for integration; on the other hand, if an independently generated 2D blue noise texture is used for each frame, each pixel can become a sequence of white noise over time. Errors can be hidden as blue noise, but are made smaller by being able to converge to the correct results more effectively. In this embodiment, the vector spatiotemporal blue noise mask has blue noise in space, which allows it to provide better image results over time than white noise.

[0115] In one embodiment, another modification is specifying which axes should be grouped together into N-dimensional blue noise to extend the application of the BNDS algorithm to higher dimensions. This allows the techniques described herein to extend beyond spatiotemporal blue noise to four-dimensional (4D) spatiotemporal depth blue noise. This is useful, for example, when rendering fog, but it can also be generalized to any dimension and any grouping of those dimensions that a particular rendering algorithm might require. It is possible to create custom random numbers in this way to provide blue noise error in screen space while achieving faster convergence for the rendering algorithm.

[0116] Blue noise distributions are well-suited to human perception and minimize unwanted low-frequency noise. Sets of blue noise points are often referred to as blue noise masks or blue noise textures. In image rendering, this typically involves integrating samples over multiple frames to amortize rendering costs, or equivalently, multiple samples collected per frame. Therefore, in one embodiment, the techniques described herein achieve various technical advantages, including (but not limited to) using a 2D blue noise pattern that produces samples well-distributed over time at pixels during animation, converges (e.g., quickly) for Monte Carlo integration, while still preserving spatial blue noise characteristics. Some spatial blue noise methods applied at each frame independently produce results that temporally represent the white noise spectrum, and therefore converge slowly for integration across time and are unstable when temporally filtered.

[0117] Therefore, the technique described herein is an extension of the BNDS algorithm involving the reformulation of the energy function. The vector spatiotemporal blue noise mask disclosed herein can lead to visually pleasing error patterns, fast convergence speed, and increased stability when time-filtered. In some embodiments, the technique described herein can also be extended to higher dimensions, as it provides unique sampling characteristics for time integration. By applying the technique described herein, improvements can be achieved in a variety of applications such as color mixing, random transparency, low-sampling-count environment occlusion, and volume rendering.

[0118] In at least one embodiment, the techniques described herein achieve various technical advantages, including, but not limited to, improved real-time image rendering and enhancement in applications using rendering algorithms that require per-pixel random vectors, and any location quantization, because it enables very good color mixing (perceptually good from a filtering point of view, and also more accurate spatial and temporal averages of small regions of pixels compared to the actual average of the unquantized source data), which masks the fact that low-bit counting is used. This is useful for reducing memory usage for geometry buffers (G-buffers), render targets, textures, etc.

[0119] While masks can be generated for two-dimensional and three-dimensional applications, the techniques described herein are applicable to multidimensional applications (e.g., greater than 3, 6, 7, etc.). Furthermore, the techniques are not limited to having only one dimension as the time dimension; rather, the techniques described herein can generate multidimensional masks (e.g., 7 dimensions), where one dimension is time, or no dimension is time. For example, the generated mask could involve a 7-dimensional mask where the first three axes (e.g., dimensions) represent 3D blue noise, the next two axes (e.g., dimensions) represent 2D blue noise, and the last two axes (e.g., dimensions) represent 1D blue noise. In at least one embodiment, the computer implementation can select the grouping of dimensions when generating the mask.

[0120] In at least one embodiment, after one or more circuits generate and apply a blue noise mask during image rendering, one or more circuits apply other types of filtering operations to the rendering process, such as red noise filtering, bandpass filtering, or other types of noise filtering (e.g., frequency attenuation filters or other denoising methods).

[0121] In embodiments, a vector spatiotemporal blue noise mask may be part of an image rendering process that includes other methods for generating spatiotemporal blue noise masks, such as scalar spatiotemporal blue noise masks. The image rendering process may use a combination of vector spatiotemporal blue noise masks and other blue noise masks (e.g., scalar blue noise masks or scalar spatiotemporal blue noise masks). Vector spatiotemporal blue noise masks generally share the advantages and quality of other spatiotemporal blue noise masks, including those generated using modified empty-aggregate algorithms. In at least one embodiment, the process includes applying a modified BNDS algorithm to generate a blue noise mask that processes vector values, and the process includes applying the modified empty-aggregate algorithm separately to scalar values ​​(e.g., as part of a different rendering step when a video game provides scalar values ​​as part of a scene rendering operation).

[0122] Figure 1 A process 100 according to at least one embodiment is illustrated, in which pixel swapping is performed on a texture. Pixel swapping is an element of the BNDS algorithm, which creates a blue noise mask of vector values. In the embodiment, blue noise is partially implemented when the total energy of the mask is minimized, which will be discussed in more detail below. In at least one embodiment, one or more computer systems are based on computer-executable instructions and code (e.g., computer-executable instructions, one or more computer programs or one or more applications, Computing Unified Device Architecture (CUDA) code) that are jointly executed on one or more processors via hardware, software or a combination thereof, and as combined Figure 9A Further discussion may include some or all of the execution process 100 (or any other process described herein, or variations and / or combinations thereof).

[0123] exist Figure 1 The top left side is texture 102, which may be referred to as a “3x3” texture, and is shown in its initial state (e.g., initial texture, initial version). Texture 102 comprises nine pixels (P1, P2, P3, P4, P5, P6, P7, P8, P9) and a total texture energy E0. Each pixel in texture 102 has an energy value (e.g., pixel energy value) determined by its relationship to another pixel in the texture (e.g., distance, similarity of values). In one embodiment, pixel swaps are performed (e.g., swapping one or more values ​​of one pixel with the values ​​of another pixel) to determine whether the total energy of the texture can be reduced, wherein reducing the total energy of the texture is associated with adjusting the characteristics of the texture from white noise to blue noise. In an embodiment, the total energy of the texture is calculated according to equation (1):

[0124]

[0125] Where E(M) is the total energy of the texture (e.g., a mask), p and q are pixels, p i and qi They are the integer coordinates of those pixels, p s and q s It is the d-dimensional sample value associated with those pixels, σ i and σ s These are configurable parameters (e.g., energy decay parameters, intensity decay parameters, Gaussian values) that control energy decay over distance and in Gaussian space (e.g., a uniform distribution of random samples). In an embodiment, σ i The energy or intensity is controlled to decrease (e.g., diminish) with distance. In some embodiments, the energy decay parameter corresponds to a Gaussian blur function, where d refers to the dimension, and d / 2 is an exponent that corrects for the difference in average distance between points in the d-dimensional sample space and the image space (e.g., 2D image space). In embodiments, σ i =2.1 and σ s =1. In at least one embodiment, the farther apart two pixels are and the less similar they are, the lower the pixel energy of each pixel. In the embodiment, the distance used by the energy function is calculated circumferentially on all axes, which means that individual texture slices are spatially well-stitched, but the temporal quality is also temporally well-stitched, with no seams when time starts at zero.

[0126] In an embodiment, the energy function of the BNDS algorithm is used in three dimensions to generate a vector-space-time blue noise texture storing vector values ​​for each pixel. In an embodiment, the energy function of the BNDS algorithm is adjusted such that if pixels used in the energy function come from the same two-dimensional texture slice or if they are the same pixel at different time points, then the energy function only returns non-zero energy values. This adjustment to the energy function results in N textures being spatially blue (e.g., exhibiting the characteristics of blue noise) and blue along the z-axis (e.g., the time axis) in each pixel. In an embodiment, unit vectors may be used with the energy function of the BNDS algorithm. Unit vectors can be used when direction vectors are needed during image rendering. Non-unit vectors can be used when N-dimensional random numbers are needed, such as points in space. In an embodiment, the vector-space-time blue noise texture provides random values ​​(e.g., random vectors representing rays) to be input into a function (such as an image function), the details of which may be unknown to the image rendering process, throughout this disclosure and in conjunction with at least Figure 3 The image rendering process will be discussed further.

[0127] In at least some embodiments, pixel swap 104 occurs between P2 and P9 to create texture 106. The total texture energy E1 of texture version 106 is calculated to be below E0 using a modified BNDS energy function, and therefore, pixel swap 104 is accepted. Next, pixel swap 108 occurs between P4 and P7 to create texture 110. The total texture energy E2 of texture 110 is calculated to be above E1, and therefore, pixel swap 108 is rejected. Next, pixel swap 112 occurs between P3 and P6 to create texture version 114. In an example scenario, the total texture energy E3 of texture version 114 is calculated to be below E1, and therefore pixel swap 112 is accepted. In at least one embodiment, pixel pairs are repeatedly swapped until the total energy of the texture is minimized. In at least one embodiment, pixel pairs are repeatedly randomly picked and swapped until the total energy of the texture is minimized. In at least one embodiment, the structure, framework, or algorithm used to determine which pixel pairs to swap is based on previous pixel swaps. In at least one embodiment, the number of swaps performed (including rejected swaps) is determined by formula (2):

[0128] Quantity of exchanges = (W) 2 xH 2 xD 2 ) / 1000 (2)

[0129] Where W is the width of the texture in pixels, H is the height of the texture in pixels, and D is the depth of the texture in pixels. In another embodiment, the number of exchanges (including rejected exchanges) is determined by formula (3):

[0130] The number of exchanges = (W) 2 xH 2 xD 2 ) / 100000 (3)

[0131] In one embodiment, pixel swapping is used to determine the minimum possible energy of the texture, starting with a 100% chance of swapping, which results in a higher total energy for the texture. At the end of the swapping, there is a 0% chance of swapping, which also results in a higher total energy for the texture. In another embodiment, pixel swapping may not cause a change in the total texture energy.

[0132] Figure 2An example process 200 for creating a vector spatiotemporal blue noise texture using pixel swapping, according to one embodiment, is depicted. One or more circuits or one or more processors may execute part or all of process 200. Process 200 includes receiving a texture 202, calculating the energy value of the texture 204, swapping pixel pairs 206, recalculating the energy value of the texture 208, calculating whether the swap results in a lower energy value for the texture 210, and if so, holding the swap 212. If not, the swap is revoked 214, and a texture is received at step 202 to restart the process. If holding the swap 212 is performed, it is calculated whether the texture has reached its minimum energy value 216, and if so, a texture is generated as output 218, and process 200 ends. If the texture has not yet reached its minimum energy value, a texture is received at step 202 to restart the process.

[0133] In an embodiment, the texture received in step 202 may initially be as described throughout this disclosure and in combination with at least Figure 1 The white noise texture discussed (e.g., uniform texture, uniform mask). In at least one embodiment, the received texture may be a template texture or a base texture, which is a starting texture (or initial texture) for generating a texture with blue noise, red noise, or other color noise characteristics. In at least one embodiment, the initial texture may include a non-uniform distribution of pixels.

[0134] In an embodiment, calculating the energy value of texture 204 includes applying a modified energy function from the BNDS algorithm, which at least combines... Figure 1 Further discussion.

[0135] In an embodiment, swapping pixel pairs 206 includes swapping the pixel values ​​of two pixels, such as combining... Figure 1 Further discussion is needed. The choice of which two pixels to swap can be performed in a pseudo-random manner.

[0136] In this embodiment, except for the two swapped pixels, the energy value of the texture calculated in step 208 is the same as the energy value of the texture calculated in step 204, and the combination... Figure 1 Further discussion.

[0137] In an embodiment, if a lower energy value is determined in step 210 due to pixel swapping, the texture retains swap 212. If a higher energy value is determined in step 210 due to pixel swapping, the texture rejects the swap and restores the pixel values ​​to their values ​​before swap 214. At least in combination Figure 1 Further discussion on energy values.

[0138] In an embodiment, determining whether a texture has pixels or pixel values ​​arranged such that the texture has the lowest possible energy is similar to simulated annealing in which process 200 has discovered the global minimum energy of the texture. Combined with Figure 1 Further discussion is needed to determine the minimum possible energy of the texture.

[0139] When the output texture 218 is generated, the output texture has its minimum possible energy applied to the image data and therefore exhibits blue noise throughout this disclosure and in combination with at least Figure 1 This will be discussed further. The output texture is a vector spatiotemporal blue noise mask. In one embodiment, the output texture can be applied to image data containing vectors, points on a grid, or objects with more than three dimensions.

[0140] Figure 3 A process 300 for rendering an image (e.g., an image function) using importance sampling and a vector spatiotemporal blue noise mask, according to one embodiment, is described. In at least one embodiment, the importance sampling and vector spatiotemporal blue noise mask generation occurs prior to the image rendering process (e.g., thus preparing the mask for application when the image is to be rendered). This can be combined as described above. Figure 2 The discussed process 200 executes process 300. One or more circuits or one or more processors may execute part or all of process 300. Process 300 includes the steps of performing importance sampling 302, performing pixel swapping 304, and rendering an image 306. Process 300 computes a modified energy function using pixel swapping on an initially non-uniform texture (e.g., white noise created using uniform random sampling). Conversely, the initial texture is non-uniform (e.g., a texture that has already been importance-sampled, a texture created using non-uniform random sampling). In an embodiment, the spatiotemporal blue noise texture exhibits a non-uniform histogram (e.g., a graphical representation of pixel intensity), which allows for importance sampling.

[0141] In one embodiment, the start image rendering process 302 can be executed to calculate, for each pixel, how much light is reflected toward the virtual camera at a given surface point of the object (e.g., an image function). In one embodiment, a pixel may correspond to a vector, a point on a grid, or an object with more than three dimensions. However, calculating all the light reflected toward the virtual camera for each pixel may be computationally too expensive for real-time rendering.

[0142] In this embodiment, importance sampling 302 is performed to determine the direction of light that best approximates or represents the sum of all light reflected toward the virtual camera. Importance sampling is a type of Monte Carlo integration (e.g., Monte Carlo sampling), which at least combines... Figure 4-5 , Figure 19 , Figure 22 and Figure 25 Further discussion. Importance sampling can be performed in one or more dimensions. In one or more embodiments, importance sampling can be performed in many dimensions using multidimensional integrals and multidimensional values. In an embodiment, a probability density function (PDF) is used to generate random values ​​to determine the optimal direction of light during importance sampling. The PDF can provide random scalar or vector values ​​to determine which samples to take during importance sampling. The PDF can be determined in part by knowing where most light will be reflected away from the object to be sampled (e.g., knowing where the object is most reflective). The PDF can contain values ​​that result in an approximate shape of the object (e.g., the shape of the output of an image function) or that result in importance sampling values ​​that determine the approximate location of one or more light sources. In an embodiment, the PDF can include samples in low-value regions of the PDF (e.g., light that contributes less to a pixel) and samples in high-value regions of the PDF (e.g., light that contributes more to a pixel), and the samples are weighted by multiplying each sample by the inverse of the PDF. In an embodiment, the PDF is associated with each pixel and stored in an alpha channel (e.g., a data component of an image file). In an embodiment, the alpha channel of the texture stores the PDF as a percentage between the minimum and maximum PDF. PDF can be calculated from pixel values ​​in a texture, for example, by performing a dot product where importance sampling is cosine-hemisphere weighted, or by dividing the pixel value by a normalized value passed as a shader constant. Importance sampling can be performed on pixels, where each pixel corresponds to a random variable, and where the sampled pixel corresponds to the value of a probability density function that determines the pixel, and where sampling includes reducing the variance of the samples by skewing the samples toward regions of higher energy based on an energy function. In embodiments, importance sampling can use any combination of techniques including GGX (e.g., a normal distribution of an ellipsoid), two-way scattering distribution function (BSDF), two-way reflectance distribution function (BRDF), Smith shadow mask, cosine lobe sampling, or some combination thereof, and as combined Figure 25 Further discussion is needed.

[0143] In at least one embodiment, after one or more circuits or processors perform an importance sampling operation 302 on an image function (e.g., generating pixel values ​​from regions of higher importance in the image function), one or more circuits or processors perform a pixel swap 304 to ensure or improve the blue noise of the pixels generated by the importance sampling. At least in combination Figure 1 and 2 Further discussion includes pixel swapping involving the number of swap iterations. One or more circuits or processors perform pixel swapping in conjunction with a modified BNDS function, and at least in conjunction with... Figure 1 and 2 Further discussion.

[0144] Once pixel swapping operation 304 is complete and a vector spatiotemporal blue noise texture is generated, image 306 is rendered. In an embodiment, the image is presented on a display while it is being rendered. In at least one embodiment, image rendering 306 occurs after importance sampling 302 and pixel swapping 304, wherein one or more circuits perform importance sampling and pixel swapping to provide an importance-sampled blue noise mask, and then one or more circuits use the mask to render the image. Image rendering 306 may occur for multidimensional pixels (such as vectors, e.g., 2D, 3D, 4D, etc.). The result may be a multidimensional photorealistic image representing an image function of a multidimensional object.

[0145] In at least one embodiment, one or more circuits use process 200 and / or 300 to receive a texture (e.g., a white noise texture) including pixels; calculate a texture energy value of the texture based on an energy function (e.g., using equation (1), wherein the energy function is based on the distance between a pair of pixels, a first configurable parameter and a second configurable parameter, and wherein the pixel energy value of each pixel in the pixel pair is non-zero because the pixel pair is in the same two-dimensional layer or the pixel pair has the same coordinates at different time slices); swap the positions of pixels from the pixel pair in the texture until the energy value of the texture reaches a minimum energy value based on the energy function (e.g., according to equation (1)); generate an output texture including output pixels based on the texture with the minimum energy value to be applied to image data; and render an output image on multiple frames based on applying the texture to the one or more images (e.g., applying the output texture to a video game scene to render frames and images as part of a video game).

[0146] Figure 4 A representation of a spatiotemporal blue noise texture with vector values ​​in a table according to one embodiment is depicted. Figure 4 The type of representation depicted in the text can be a combination of at least Figure 1-3 The results of combinations of some or all of the techniques discussed. Pixel swapping using an adjusted energy function from the BNDS algorithm has been applied to 128x128x64 textures, and slices of those textures and their Discrete Fourier Transforms (DFTs) (e.g., representations of frequencies and their components) are depicted in Table 1. At least in combination Figure 4 and Figures 6A-6BFurther discussion of the DFT. In the embodiment, the DFT is averaged to show the expected spectrum in addition to the golden ratio animated blue noise, which highlights two ways in which it destroys spatial frequencies at a specific frame number. In the embodiment, it is desirable to obtain blue noise characteristics in each spatial 2D slice in order to provide a noise sequence that is improved over white noise along the time axis. Row XY[0] shows a spatiotemporal blue noise texture slice of vector values ​​of a one-dimensional vector (Vec1), its unit vector (Unit Vec1), a two-dimensional vector (Vec2), its unit vector (Unit Vec2), a three-dimensional vector (Vec3), and its unit vector (Unit Vec3). The texture slice depicted in row XY[0] represents blue noise. The type of representation shown in row XY[0] can be viewed as a blue noise storage of vectors of different dimensions. Row DFT(XY) shows the DFT in a two-dimensional layer of the spatiotemporal blue noise texture slice of the corresponding vector values ​​shown in row XY[0]. The DFT depicted in DFT(XY) shows lower values ​​of high frequencies (e.g., energy) and higher values ​​of low frequencies, indicating blue noise. Row DFT(XZ) shows the DFT of the texture along the z-axis (e.g., the time axis) corresponding to the texture slice shown in row XY[0]. The DFT depicted in row DFT(XZ) shows the blue noise over time, where lower frequencies show higher values. The DFT of Table 1 is computed for each color channel. The type of representation of the blue noise texture shown in Table 1 can be applied to vectors of different types and sizes and can be considered as blue noise storage of vectors of different dimensions and their DFTs.

[0147] Figure 5 A representation of the vector spatiotemporal blue noise texture is depicted according to the importance samples in Table 2 of one embodiment. Figure 4 The type of representation depicted in the text can be a combination of at least Figure 1-4 The results of a combination of some or all of the techniques discussed. The spatiotemporal blue noise texture of the vector values ​​of importance sampling can also be referred to as the spatiotemporal blue noise texture of the vector values ​​of importance sampling. The first row of Table 2 (cosine-weighted hemispherical Unit Vec3) depicts slices of the spatiotemporal texture of the vector values ​​of importance sampling (in the column texture [0]), which includes three-dimensional vectors along different axes and their corresponding DFTs (DFT(XY) and DFT(ZY)). The blue noise texture in the first row is based on the texture of importance sampling of the hemispherical unit, where importance sampling includes cosine-weighted sampling. In an embodiment, importance sampling may be applied to a texture that has been adjusted to represent blue noise. In an embodiment, importance sampling may be applied to the texture before it is adjusted to represent blue noise. The texture slices in the first row of Table 2 and their corresponding DFTs represent blue noise, where the DFTs represent low frequencies with higher magnitudes in space and time.

[0148] The second row of Table 2 (HDR Skybox Importance Sampling Unit Vec3) depicts slices of the spatiotemporal texture (in column texture [0]) of the importance sample vector values, including three-dimensional vectors along different axes and their corresponding DFTs (DFT(XY) and DFT(ZY)). The blue noise texture in the second row is a texture based on the importance sampling of the High Dynamic Range Skybox (HDR Skybox) (e.g., a representation of a virtual environment stored as a cuboid). The texture slices and their corresponding DFTs in the second row of Table 2 exhibit blue noise, where the DFTs represent low frequencies with higher magnitudes in both space and time.

[0149] Figure 6A The image depicts an image generated by rendering an ambient occlusion effect in a 3D scene using three different types of noise textures according to one embodiment. Figure 6A and 6B The images depicted in the text can be combined with at least Figure 1-5 The result is a combination of some or all of the techniques discussed. In an embodiment, ambient occlusion is a shading rendering technique used to calculate how much light is reflected from each point when exposed to ambient lighting, and will be discussed further below. Figure 6A The three images described herein, as part of a uniform sampling environment occlusion process, take four samples per pixel and use three different texture sampling techniques, one for each image—white noise, blue noise, and spatiotemporal blue noise (STBN). In this embodiment, the spatiotemporal blue noise is a vector value. Of the three images, the image using white noise produces the worst discriminative detail of the scene, while the image using STBN produces the best discriminative detail of the scene.

[0150] Figure 6B The image depicts an image generated by rendering ambient occlusion effects in a 3D scene using three different types of noise textures and importance sampling according to one embodiment. In this embodiment, spatiotemporal blue noise is a vector value. Figure 6B The three images depicted, as part of a uniform sampling environment occlusion process, sample four samples per pixel while using a cosine-weighted hemispherical importance sampling unit vector (e.g., determining the more important angle of a sample from which it is represented, in part by the angle represented by the cosine value). In the embodiment, the blue noise and STBN textures have cosine-weighted hemispherical vectors stored in their textures, which are transformed into tangent space using TBN (tangent, bitangent, normal) basis matrices. Of the three images, the image using white noise produces the worst discriminative detail of the scene, while the image using STBN produces the best discriminative detail of the scene.

[0151] Figure 7A The performance (e.g., convergence) of various noise masks according to one embodiment, including a vector spatiotemporal blue noise mask, is described. Figures 7A to 7C As shown, the x-axis has frames, the y-axis has root mean square error (RMSE), white noise is represented by hollow circles, STBN Morton is represented by triangles, STBN Hilbert is represented by squares, and STBN vectors are represented by solid circles. Figure 7A The result type described in -C can be a combination of at least Figure 1-6A This is the result of a combination of some or all of the techniques discussed in 6B. In an embodiment, a scalar blue noise mask can be transformed into a vector blue noise mask by inputting scalar values ​​into a space-filling curve function (e.g., Morton curve, Hilbert curve) to output pixel values ​​as vectors (e.g., curve inversion). Applying curve inversion to a scalar blue noise mask results in a vector-value mask that retains the same properties as the scalar blue noise mask. Curve inversion can be used with other types of dithering masks or other scalar-value (e.g., grayscale) noise patterns (e.g., Bayer matrix), staggered gradient noise, and stylized noise patterns. Figure 7A This is a graph (e.g., a convergence graph) depicting the root mean square error (RMSE) (e.g., the amount of error generated by the noise mask during image rendering) of various noise textures of size 32x32x64, all of which sample trigonometric functions. The trigonometric functions are sampled using two-dimensional frames of size 32x32. As depicted, the RMSE decreases with increasing sampled frames. In the embodiment, and as depicted, the vector spatiotemporal blue noise texture transformed from a scalar noise texture with a Hilbert curve performs better than the scalar noise texture transformed by a Hilbert curve. STBNs generated by either the Morton curve or the Hilbert curve are not superior to vector spatiotemporal blue noise masks generated by using the BNDS algorithm and a modified energy function, such as further combined... Figure 1 The subject of discussion.

[0152] Figure 7B The performance (e.g., convergence) of various noise masks comprising several vector-valued blue noise masks according to one embodiment is described. One noise mask is a time-layered blue noise mask (Vector BNxStrat). At least... Figure 8 The layering of textures is further discussed. In the embodiment, this is achieved by using at least a combination of Figure 1 Further discussion involves pixel swapping to create a noise mask, but the energy function of the BNDS algorithm is modified so that it returns a non-zero value if two pixels share the same z-coordinate (e.g., are located on the same 2D slice of the texture) and the swap between those two pixels does not worsen the layering of the texture along the z-axis. Figure 7B Charts and combinations Figure 7AThe charts discussed are similar, showing that once all samples are collected, the noise mask (Vector BNxStrat), which is spatially blue and temporally layered, performs well.

[0153] Figure 7C A graph is shown illustrating the performance (e.g., convergence) of various noise masks, including a vector spatiotemporal blue noise mask containing multiple values ​​for each pixel. In some cases, such as when rendering multiple samples per pixel, it is necessary to have more than one spatiotemporal blue noise value per pixel. Adding more than one value per pixel can be accomplished by reading the pixels of the noise texture and adding fixed offset values ​​(e.g., fixed coordinate values ​​in the x-direction and y-direction) to generate new pixels at new locations.

[0154] In this embodiment, more than one spatiotemporal blue noise value per pixel can be created by adding a rank-1 matrix (e.g., a set of points with a periodic structure and low dissimilarity sequence in N-dimensional space) to each pixel. In this embodiment, applying the rank-1 matrix to each pixel of a scalar blue noise mask with a path length having a golden ratio value or the square root of two achieves optimal results. Figure 7C The graph shows that an added two-dimensional blue noise mask with a rank-1 lattice (vector 2DBN+R2) can perform better than a uniform spatiotemporal blue noise mask (albeit more irregular).

[0155] Figure 8 A process 800 for determining whether pixel swapping between layers should be maintained, according to one embodiment, is described. Process 800 may be combined with at least... Figure 1-5 Any combination of techniques discussed in 6A-B and 7A-C. Process 800 includes collecting (e.g., acquiring) pixel values ​​802 of pixels located at the same xy coordinates in each layer of the texture, said pixel values ​​being the layering of the texture along the z-axis.

[0156] In step 804, the number of values ​​collected during step 802 is counted. In an implementation, the ideal number of values ​​collected within each layer is 1. In at least one embodiment, if layering is performed over time, layering works best when each pixel at a two-dimensional location is located within one layer.

[0157] In step 806, the number of counted values ​​in each layer is subtracted from the total number of counts. In an embodiment, if the ideal number of values ​​in each layer is 1, then the number to be subtracted is 1, and the result of the subtraction is a value that helps determine whether pixel swapping between layers should be maintained, as discussed further in the following steps. In an embodiment, the value produced by the subtraction is referred to as the error value.

[0158] In step 808, each error value is squared. As part of steps 810 and 812 below, the squaring operation at least partially explains layers that do not contain pixel values ​​when determining whether pixel swapping should be maintained. In an embodiment, if a layer does not contain pixel values, the error value will be -1, and this value will be counted towards the summation operation performed in step 810.

[0159] In step 810, the squared error values ​​are summed to create a total error value for fitting the layers to the texture when the texture is pixel-dependent. In one embodiment, the sum of the squared error values ​​is associated with the number of times the layer contains more than one pixel value or contains no pixel value at all.

[0160] In step 812, the total error value of the pixel-related layer is compared with the total error value of the layer before the pixel swap. In an embodiment, if the total error value has increased after the pixel swap, the layer has deteriorated, and the pixel swap is rejected. If the total error value has decreased after the pixel swap, the layer has improved (e.g., changed). If the total error value remains the same after the pixel swap, the layer has neither improved nor deteriorated, and the swap can be accepted to avoid reversing one or more computational steps of the pixel swap.

[0161] In combination Figure 9A The following discussion, along with the accompanying figures, can be applied to vector spatiotemporal blue noise masks as disclosed herein, and more generally to other types of spatiotemporal blue noise masks, including scalar spatiotemporal blue noise masks generated at least in part by a modified empty-clustering algorithm.

[0162] Figure 9AAn example of a process 900 for generating a blue noise mask optimal for both spatial and temporal use, according to at least one embodiment, is illustrated. In at least one embodiment, some or all of process 900 (or any other process described herein, including those combining BNDS algorithms, importance sampling, lattice and hierarchical structures, or variations and / or combinations thereof) is executed under the control of one or more computer systems (e.g., computing devices) configured with computer-executable instructions, and is implemented as code (e.g., computer-executable instructions, one or more computer programs, or one or more applications, Computing Unified Device Architecture (CUDA) code) jointly executed by hardware, software, or a combination thereof on one or more processors. In at least one embodiment, the code is stored in the form of a computer program on a computer-readable storage medium comprising a plurality of computer-readable instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transient computer-readable medium. In at least one embodiment, not only transient signals (e.g., propagating transient electrical or electromagnetic transmissions) are used to store at least some of the computer-readable instructions that can be used to execute process 900.

[0163] In at least one embodiment, the non-transient computer-readable medium does not necessarily include non-transient data storage circuitry (e.g., buffers, caches, and queues) within the transceiver of transient signals. In at least one embodiment, process 900 is executed at least partially on a computer system, such as those described elsewhere in this disclosure. In at least one embodiment, process 900 is executed by one or more circuits to calculate the motion of one or more pixels in the first region of the image based at least partially on the motion of one or more pixels in a second region of the image that overlaps with the first region.

[0164] In at least one embodiment, the system performing at least a portion of process 900 includes executable code for generating a blue noise mask that is optimal both spatially and temporally (e.g., a spatiotemporal blue noise mask). The spatiotemporal blue noise mask can be generated as a set of N blue noise textures, each texture individually having good blue noise (e.g., containing a high amount of higher frequencies and a low amount of lower frequencies), and each pixel individually also having blue noise over time. This can provide the desired quality without compromise on either the spatial or temporal axis. In one embodiment, 902 one or more images are obtained from a computing device, camera, etc. In one embodiment, one or more images may be part of a game application, wherein a real-time image rendering algorithm is used to display the images when the game application is executed by the computing device. The computing device may include one or more graphics cards that use deep learning to upscale lower-resolution images to higher resolution for display on a computing screen. In an example embodiment, 64 3A spatiotemporal blue noise mask with a resolution of (64x64x64) can also be created. Additionally, a 64x64 resolution spatiotemporal blue noise mask can be created. 2 64 3 A 3D blue noise mask and 64 independent 2D blue noise masks. In one embodiment, a spatiotemporal mask is also created by using a single 2D blue noise mask and adding a golden ratio to each of the 63 frames to produce 64 different masks. In another embodiment, one or more processors of a computing device execute instructions to apply the spatiotemporal blue noise mask to one or more of the acquired images for real-time image rendering 904.

[0165] In one embodiment, one or more computing devices perform the generation of dimensions [d0, d1, ..., d]. n An algorithm for a blue noise mask M is described. The algorithm may require each pixel to store Boolean logic specifying whether the pixel is activated (emitting energy into an energy field), and an integer index specifying the order in which the pixel is activated. The order in which pixels are activated defines the final output color of the pixels, where the first pixel to be activated is black, and the last pixel to be activated is white.

[0166] In some instances, multiple two-dimensional blue noise masks can be used for high quality in the spatial domain; however, each pixel may also individually require a high-quality sampling sequence over time. Therefore, in embodiments, one or more computing devices can execute one or more algorithms to generate a three-dimensional blue noise mask. In embodiments, the BNDS algorithm can be reformulated such that it is driven by a new energy function as shown in Equation 1 above. In embodiments, instead of executing the formula in two dimensions, it is executed in three dimensions, and the energy function is constrained in two ways. The energy can be non-zero if two pixels in the energy function are located in the same two-dimensional layer, or if two pixels have the same (x, y) coordinates. A first condition ensures that each two-dimensional layer can have blue noise characteristics, and a second condition ensures that each pixel can have blue noise characteristics over time. Without the first condition, each pixel will be blue noise on the time axis, but will be independent of each other and spatially white noise. Without the second condition, each z-plane slice will be independent, and the result will be white noise along the time axis. Without the constraint that one of these conditions must be met, the result will be three-dimensional blue noise, which is not well distributed along the spatial or temporal axes (spatiotemporally), but rather well distributed within the 3D volume. In the embodiment, pixels in the three-dimensional spatiotemporal blue noise texture are represented as p = (p xy ,p z )=(p x ,p y ,p z ). Figure 9BAn example of process 906 for generating a 3D mask for use in both space and time is shown, where two dimensions correspond to space (e.g., x and y coordinates) and one dimension corresponds to time. While 3D masks can be generated, N-dimensional masks can also be generated, as explained in receiving operation 908. In at least one embodiment, process 906 is integrated into process 900. In at least one embodiment, some or all of process 906 (or any other process described herein, or variations and / or combinations thereof) is executed under the control of one or more computer systems configured with computer-executable instructions and is implemented as code (e.g., computer-executable instructions, one or more computer programs, or one or more application programs, CUDA code) jointly executed by hardware, software, or a combination thereof on one or more processors. In at least one embodiment, a system includes a memory storing instructions that, when executed by one or more processors, cause the system to execute the instructions to perform process 906. In at least one embodiment, the code is stored in the form of a computer program on a computer-readable storage medium, the computer program comprising a plurality of computer-readable instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transient computer-readable medium. In at least one embodiment, not only transient signals (e.g., propagating transient electrical or electromagnetic transmissions) are used to store at least some computer-readable instructions that can be used to execute process 906. In at least one embodiment, the non-transient computer-readable medium does not necessarily include non-transient data storage circuitry (e.g., buffers, caches, and queues) within the transceiver of the transient signal. In at least one embodiment, process 906 is executed at least partially on a computer system such as those described elsewhere in this disclosure. In at least one embodiment, process 906 begins with a receive operation 908 and continues with a calculation operation 910.

[0167] In receiving operation 908, the system or computing device receives pixel data with three dimensions corresponding to one or more images. In one embodiment, the system or computing device receives the pixel data based on sampling one or more images from one or more images obtained in step 902 of process 900. In another embodiment, the system or computing device receives the pixel data from another device or another process (e.g., from an application running on another device). Receiving the pixel data may include receiving N-dimensional data of the pixels. For example, receiving the pixel data may include receiving spatial pixel data (e.g., x and y coordinates) and temporal data, where the spatial data corresponds to two dimensions and the temporal data corresponds to the temporal dimension of each pixel.

[0168] At calculation operation 910, the system or computing device performing process 906 calculates energy values ​​for some pixels of one or more images received in receiving operation 908. In at least one embodiment, the energy value of a pixel can be calculated according to Equation 1. In at least one embodiment, the energy value corresponds to an intensity value, where the intensity value indicates the intensity of a pixel, for example, how much a pixel stands out relative to other pixels (e.g., when it is activated). As shown in Equation 1, the energy value is based on the coordinates of at least some pixels (e.g., pixels p and q), the distance between at least some pixels (e.g., pixels p and q), an energy attenuation parameter, and Gaussian space. When determining the distance between at least some pixels, the distance between a pair of pixels can be used. In an embodiment, when determining the distance between at least some pixels, multiple pixel pairs (e.g., pixels p and q, where q can be any adjacent pixel for p) can be determined. When determining the energy value of a pixel, the distance between pixels can be calculated circularly. In one embodiment, the energy value is calculated for each pixel. In one embodiment, the system or computing device calculates the energy values ​​of some pixels based on determining the relevant processing portion of the image.

[0169] Using the modified Equation 1, there are typically two constraints that determine the energy value. For example, the energy value can be non-zero if two pixels in the energy function are in the same 2D layer, or if two pixels have the same (x, y) coordinates. The first condition ensures that each 2D layer can have blue noise characteristics, and the second condition ensures that each pixel can have blue noise characteristics over time. If the calculated pixel energy does not satisfy these two constraints, the system or computing device can set the energy value to zero. For example, if a pixel and another pixel are not in the same 2D layer or do not have the same coordinates (e.g., at different time slices), then the system or computing device can set the energy value of at least some pixels to zero.

[0170] To generate a 3D mask, in mask operation 912, the system or computing device generates the mask based on the calculated energy values ​​from computation operation 910. Generating the 3D mask may be part of another digital image processing algorithm (e.g., BNDS algorithm, spatial aggregation algorithm, color mixing, or error diffusion), where the image processing algorithm uses an energy function and corresponding energy values ​​of the pixel data as determined in computation operation 910. In one embodiment, because mask operation 912 considers energy according to Equation 1, the generated mask produces a blue noise mask that provides optimal visual results for human perception, for example, as part of a video game, video, or other digital video process. The 3D mask can be considered a spatiotemporal mask of blue noise vectors.

[0171] In operation 914, the system or computing device provides one or more output images based on applying a 3D mask from masking operation 912 to the one or more images, a sample version of the one or more images, or a processed version of the one or more images. In one embodiment, the one or more output images may be part of a game application, wherein a real-time image rendering algorithm is used to display the images when the game application is executed by the computing device. For example, the rendered output images may be part of an image generation pipeline that includes ray tracing or path tracing. In one embodiment, one or more processors of the computing device execute instructions to apply a 3D mask to one or more images for real-time image rendering.

[0172] Process 906 can be integrated with other image processing techniques. In embodiments, process 906 can be integrated into sampling as part of image processing in motion and temporal filtering methods such as temporal anti-aliasing (TAA) and deep learning supersampling (DLSS). As part of DLSS, process 906 may include applying temporal image upscaling to the one or more images, wherein the upscaling is based on neural network inference from a lower resolution image. In at least one embodiment, a spatiotemporal mask is applied before, after, or both before and after image processing associated with TAA and DLSS.

[0173] Similarly, process 906 can be integrated into color mixing, random transparency, area light sampling, volume rendering, path tracing, and / or random alpha image processing techniques. Furthermore, the operation of process 906 can be repeated (e.g., for multiple images) or performed in a different order as part of another digital image processing algorithm. For example, process 906 can be performed as part of a sampling algorithm. In at least one embodiment, processes 900 and 906 can be applied to video or video game content. The video or video game includes a sequence of images (e.g., frames) that can be displayed at a frequency (e.g., frame rate), where a single video frame is an image. Furthermore, a video frame refers to video information, while an audio frame refers to audio information, and video frames can be processed synchronously with or separately from audio frames.

[0174] Figure 10An exemplary image is shown illustrating the use of a blue noise mask that is optimal in both space and time, according to at least one embodiment. In the embodiment, the blue noise mask provides a way for the system to hide noise and errors. This is useful in real-time rendering where computational resources are limited to making noise completely disappear (which is the motivation for denoising). Although a blue noise mask does not produce less noise and errors than white noise, it does arrange itself in a more visually pleasing, less perceptible, and easier-to-denoise manner. For example, a three-dimensional vector blue noise mask may correspond to the red, green, and blue (RGB) components of a pixel, and the mask may be applied to an image to produce an output image with blue noise. Blue noise has various uses in rasterization and ray tracing. Figure 10 As shown, at the top, blue noise and white noise are used to render grayscale image points as black and white. The top blue noise is much less noisy and looks more like the source image, although it has the same amount of error as the white noise image below it. At the bottom, two noises are used to mix the color images before they are quantized to one bit per color channel. Both images can contain eight colors: red, green, blue, yellow, cyan, magenta, black, and white, and have the same amount of error as the source image, but the blue noise version at the top has better image quality.

[0175] The pointillist case can be obvious when ray tracing and capturing less than one ray per pixel. A black point can be treated as a pixel when choosing to capture a ray and selecting white or blue noise will produce the same kind of result in 3D rendering. The color mixing case occurs when encoding data in the buffer. Being able to use a single bit per color channel instead of the usual 8 bits per color channel means 3 bits are used for color instead of 24, meaning that only 12% of the previous bit depth can be used to represent the data.

[0176] Figure 11 Comparative frequency results generated on three types of blue noise masks using Fourier analysis according to at least one embodiment are shown. That is, in Figure 4 The discrete Fourier transform (DFT) of the 2D projections of various blue noise masks is shown. The frequency comparison results can be performed using procedure 100 (see [link]). Figure 1 ) and / or process 106 (see Figure 1 This is generated. In an embodiment, the spatiotemporal blue noise mask has blue noise in space, thus enabling it to provide better image results than white noise on the z-axis (time). In an embodiment, the DFT is averaged to show the desired spectrum, in addition to the golden ratio animated blue noise, which highlights two ways it destroys spatial frequencies at specific frame numbers. In an embodiment, it is desirable to obtain blue noise characteristics in each spatial 2D slice in order to provide a noise sequence that is better than a white noise sequence along the time axis. Figure 11As shown, a spatiotemporal blue noise mask, as described in this paper, provides both features simultaneously by having 2D blue noise characteristics on the XY plane and adding the blue noise characteristics to the Z-axis.

[0177] In the embodiments, although the DFT indication is blue both spatially and temporally using a spatiotemporal blue noise mask as described herein, the convergence rate of the blue noise over time is increased compared to other alternative methods for animating blue noise (this is in Figure 12 (This is shown in more detail below). Since time integration is equivalent to integrating multiple samples within the same frame, solving it in one domain is equivalent to solving it in another. Time integration often uses leak integrators instead of Monte Carlo integrators.

[0178] like Figure 11 As shown, the right two columns also illustrate that if spatiotemporal blue noise is offset along the time axis, it can have the same convergence properties and is actually asymptotically continuous at any index, while also being circumferentially continuous. This circumferential / asymptotic nature of the time axis can be a powerful property used in temporal anti-aliasing (TAA) style temporal integration and filtering algorithms. In those algorithms, each pixel is asymptotically integrating the integrand over each frame, but when an individual pixel considers its history to be no longer valid due to occlusion changes or similar factors, the pixel will effectively discard its history and restart the integration.

[0179] Using an animated blue noise mask to drive the integral of those pixels means that the global sequence is driving all pixels. Most progressive sequences will only give a progressive sequence starting from index 0 (the exception to this is the Sobol sequence, which is progressive for all powers of 2-sized segments). This is problematic because global sequence-driven sampling for individual pixels discards their history at arbitrary points in time, and those pixels will start sampling at arbitrary positions in the sampling sequence.

[0180] In the case of continuous / progressive spatiotemporal blue noise on the timeline, after rejecting history at any frame number, each pixel can receive the benefit of starting at the beginning of the progressive sequence without the overhead of having to trace the index of each pixel to make this happen. Furthermore, history rejection is typically not a discrete event but a continuous operation, such as clamping historical data to the minimum and maximum of the colors seen in the local neighborhood of the newly rendered pixel values. In some cases, the sampling index is reset, while in others it is not. Nevertheless, the progressive sequence from any index implies whether a pixel has rejected its history, and taking a sample is a good thing, meaning it also handles this continuous history rejection case.

[0181] 4D Blue Noise Mask Analysis

[0182] In this embodiment, the two 4D blue noise masks are configured as follows: 2Dx1Dx1D and 2Dx2D, each with a size of 64x64x16x16. In this embodiment, it is possible to... Figure 13 The frequency analysis shown illustrates the expected frequency behavior for each pair of axes in a 2D DFT. In this embodiment, two masks represent 2D blue noise in the XY plane, but are different under all other projections. A 2Dx1Dx1D blue noise mask can represent 1D blue noise on the Z and W axes under all projections (including the ZW plane, where they all exist and are shown in a cross pattern). On the other hand, 2Dx2D blue noise represents white noise for all other projections except the ZW plane, where it represents 2D blue noise.

[0183] From observations, the generation time of the blue noise mask is a function of the total pixel count, regardless of how those pixels are partitioned across dimensions, such as... Figure 16 As shown in the diagram, where n is the number of pixels. Doubling the number of pixels in a blue noise mask will roughly take four times longer to generate the mask. Blue noise masks can be stored as single-channel 8-bit textures. The chart below shows some texture sizes and their byte sizes as examples. Due to good tiling on each axis, smaller textures, such as 64x64x16 (64KB) and 64x64x16x16 (1MB) for spatiotemporal blue noise, are sufficient for image rendering. Actual sizes for spatiotemporal blue noise and 4D versions are marked with an asterisk (*) and in bold.

[0184]

[0185] In an embodiment, the algorithm used to generate the spatiotemporal blue noise mask can be configured to specify different dimensions for each axis (see [link to implementation]). Figure 13 ) and different energies of sigma (see Figure 15 Furthermore, although all axes are circularly continuous, if this is not desired, it is possible to select features for each axis by non-circularly calculating the distance on that axis.

[0186] When using blue noise masks, multiple independent masks may be required. For example, when using diffuse and specular buffers for color mixing that are later combined via addition, the same blue noise mask may not be used repeatedly for both buffers because it will increase the difference between pixels when pixels are added together (already using the same mixing mode on each buffer). In some instances, the system may generate and load two independent masks, but these masks can consume more memory than needed, especially if each different color buffer in the rendering pipeline requires an independent blue noise mask. This number can even be dynamic or unbounded, which would be even more problematic.

[0187] An alternative way to approximate independent blue noise sources is by offsetting, where a blue noise mask is read for each desired independent blue noise source. Figure 14 The diagram illustrates the autocorrelation of a blue noise mask, showing that small offsets read from the blue noise texture can lead to correlation or anticorrelation, but larger offsets will result in uncorrelated values. This is because blue noise is correlated at small distances but uncorrelated at large distances, as illustrated in the autocorrelation diagram.

[0188] To generalize this to wanting N distinct, independent data sources, you might need N points on the texture, which are almost always farthest from each other. In other words, these points should have low variance. Since star-shaped variance is not a circular measurement, it can be measured circularly. If the number of independent data sources needed is unknown beforehand, an asymptotically circular sequence of low variance can provide any number of points with this characteristic.

[0189] Since higher-dimensional blue noise mask calculations take longer and require more memory to store them in order to obtain an N-dimensional mask, the system can approximate it by starting with an N-1 dimensional mask, reading the values ​​of the first N-1 axes, and then multiplying the last dimension index by the golden ratio, adding it to the mask value, and using a modulus to keep it between 0 and 1.

[0190] N(a0,a1,…,a n )=(N(a0,a1,…,a n-1 )+φa n-1 )mod 1 (5)

[0191] This is illustrated by comparing spatiotemporal blue noise with 2D blue noise shown through a golden ratio animation, and also by comparing 2Dx1Dx1D blue noise with spatiotemporal blue noise using the golden ratio to add a fourth dimension. While this may compromise the spatial frequency, it does demonstrate convergence and can help in creating temporal and memory usage. Although other irrational numbers exist to form other rank 1 lattices, which could also be used here, they have lower quality for sampling, and this method can only sum groups of 1D axes.

[0192] In this embodiment, a lower-quality, higher-dimensional group is added. For example, interleaved gradient noise or a z-sampler can be used to add a 2D group because they are ways to transform 2D integer coordinates into scalars with desired properties on a 2D plane. This scalar can be added to the value read from the blue noise mask, and the modulus can again be used to make it between 0 and 1.

[0193] Figure 12The convergence rate of an example function according to at least one embodiment is shown. That is, Figure 12 The convergence rates of functions with x∈[0,1] using time axes of various mask types are shown, illustrating Monte Carlo and leakage integrals. Functions can be used with higher-dimensional vectors (e.g., 2D, 3D, 4D, etc.). In at least one embodiment, hierarchical sampling shows that potentially better convergence rates exist if only the 1D time axis is considered without considering the 2D plane of screen space. The offset plots in the two columns on the right show that starting integration from indices other than 0 does not affect the results, and that spatiotemporal blue noise is asymptotic from any index and continuous when reaching the end of the sequence and restarting at index 0. The VanDer Corput base (VDC) does not exhibit the characteristics shown by the unstable accuracy with a low sample size.

[0194] Figure 13 The DFT of a 2D projection of a 64x64x16x16 4D blue noise mask according to at least one embodiment is shown. For clarity, Figure 13 The projections described herein are averaged to show the expected spectrum. In at least one embodiment, process 900 (see [link to documentation]) can be used. Figure 9A ) or process 906 (see Figure 9B (A portion of) is used to generate the DFT.

[0195] Figure 14 An image according to at least one embodiment is shown to illustrate the autocorrelation of a blue noise texture. In the embodiment, neighboring objects can have very different values, which results in ripples of correlation (red / white) and anticorrelation (blue / black) at small offset centers, but rapidly decays to decorrelation values ​​(white / gray).

[0196] Figure 15 A 2Dx1D spatiotemporal blue noise mask according to at least one embodiment is shown, with each axis having various sigma.

[0197] Figure 16 It is shown that the generation time according to at least one embodiment is a function of the number of pixels in the blue noise mask and approximately follows y = x 2 The graph shows the curve. Doubling the pixel count can roughly quadruple the processing time.

[0198] In this embodiment, random transparency is the process of randomly selecting whether to accept or ignore samples based on the transparency level of the material. Complex algorithms have been developed using alternative methods, but the core idea of ​​randomly accepting or rejecting pixels can remain the same. In this embodiment, the spatiotemporal blue noise mask described herein uses very low sample counts and low computational cost (single texture read and comparison), thus providing the same blue noise distribution error in screen space as 2D blue noise, but converging faster than other methods used by 2D blue noise. Random transparency is useful in situations such as delayed lighting that stores information about how pixels are shaded rather than the shading result itself, and storing multiple or arbitrary numbers of layers to subsequently compute the appropriate transparency is impractical. Random transparency is also useful in contexts requiring path tracing of a single sample per ray vertex, and focusing solely on the average pixel value is correct for things like semi-transparency, rather than incurring computational and memory costs to compute the semi-transparency of a single sample. Random transparency works by generating a random number ξ∈[0,1] and comparing that random number with the opacity α∈[0,1] of the material. If ξ is greater than α, the sample is discarded. When white noise random numbers are used on ξ, if performed an infinite number of times, the percentage of pixels surviving the test will match α, but for a small number of samples, it has a large variance both spatially and temporally. In at least one embodiment, a vector mask (e.g., a blue noise mask that can process vectors) is used to handle multiple depth levels of random transparency.

[0199] Conversely, using a 2D blue noise mask will make the percentage of surviving pixels spatially more accurate for a smaller number of samples, which will also randomize the surviving pixels, but space them roughly evenly. However, as previously mentioned, methods for animating blue noise over time either change the spatial blue noise or turn it into white noise over time, thus causing poor convergence when taking multiple samples per frame or integrating over multiple frames. Using a spatiotemporal blue noise mask as described herein (where each individual frame is good blue noise, but each pixel is also a good sampling sequence over time) means that individual frames will have surviving pixels (which are spatially distributed blue noise), and also means that each frame will have very different surviving pixels, thus allowing for better convergence over time or on multiple samples within a single frame. Figure 17 The rendering comparison is shown in the image, and... Figure 18 The convergence rate is shown in the figure.

[0200] Figure 17 Random transparency using various types of noise is illustrated according to at least one embodiment. In the embodiment, Figure 17One sample is shown for each pixel. The top image is the original frame, where the bottom image is Gaussian blurred with 2σ. In this embodiment, the spatiotemporal blue noise is spatially as good as 2D blue noise and better than the golden ratio animated blue noise.

[0201] Mixed colors

[0202] Color mixing is the process of adding a small amount of noise to data before quantizing it to obtain a noisy result rather than quantization artifacts. This can be used to hide striping artifacts that would otherwise occur due to reduced bit depth, thus allowing for less memory to be used while attempting to maintain image quality. Color mixing causes pixels to be randomly rounded up or down when quantized, where the probability of rounding toward a quantization level is based on how far the value is from that level. If quantizing consecutive values ​​x∈[0,1] yields n distinct values, then the quantized value is obtained. Random numbers ξ∈[0,1) can be used in the following equation:

[0203]

[0204] In at least one embodiment, a vector mask (e.g., a blue noise texture that can be processed by vectors) provides a random number for each color channel.

[0205] When white noise is used for color mixing, the result is a white noise pattern. If blue noise is used instead, the result is more visually pleasing to humans or displays, while also having a more accurate average value over small pixel areas in space. When spatiotemporal blue noise is used for color mixing, the result may be spatial blue noise, but it may also be temporal blue noise, where each pixel can have a more accurate average value over smaller samples over time when animated. Figure 11 The rendering comparison is shown in the image, and... Figure 20 The convergence rate is shown in the figure.

[0206] Figure 19 The image illustrates color mixing before quantization to 1 bit per color channel using various types of noise, according to at least one embodiment. The top image is the original frame, and the bottom image is Gaussian blurred with 2 sigma. Figure 19 As shown, spatiotemporal blue noise is as good as 2D blue noise in terms of space and is improved compared to the blue noise of golden ratio animation.

[0207] Figure 20 A graph showing the convergence rate in color mixing of various types of noise according to at least one embodiment is presented. In the embodiment, the golden ratio animated blue noise converges faster than spatiotemporal blue noise, but spatially varies in frequency.

[0208] Ray-Marched Participating Medium with Spatiotemporal Blue Noise

[0209] In one embodiment, one or more computing devices execute an algorithm to render a single scattering heterogeneous participating medium with a very low sample count. This is a different type of algorithm from random transparency or color mixing because it demonstrates how blue noise masks can be applied to arbitrary rendering problems. According to one embodiment, while more complex algorithms exist for rendering participating media, the technique described herein is simple, performance-optimized, produces good results with very low sample counts, and works with rasterization or ray tracing. In the embodiment, the algorithm is run after the major hits have been shaded and the surface depth is known. The surface depth d can be the length of a line segment along the camera ray r that must be integrated. In the embodiment, this line segment is sampled at n evenly spaced locations, where the space between each sample is... Units. Then, the position p of sample s∈Z[0,n-1] s Calculated as:

[0210]

[0211] At each sampling point p s At a certain location, the fog density field F is sampled to obtain the density f. s This is assumed to be the density for the entire step of the distance.

[0212] f s =F(p s )

[0213] Still in p s Evaluate the light visibility function V to obtain the visibility value v for all light i∈I. s,i ∈[0,1]. Visibility value v s,i It could be a binary value similar to when shooting a single ray of light towards the light source, or it could be a more continuous value similar to using a filter closer to a percentage to read the shadow image, or it could come from shooting multiple shadow ray samples.

[0214] v s,i =V(p) s,i )

[0215] To calculate individual fog samples c s The color of the fog in the shadow determines the fog color c. unlit The shadow of the fog illuminated by light i and the color of the fog c lit,i Fog color can be calculated or provided. Visibility value v s,i Can be multiplied by c lit,i To obtain the contribution of that light. Add all the illumination contributions and then calculate c. unlit Add to the result to obtain that sample c s The final color of the fog.

[0216]

[0217] To calculate the opacity of the sample o s The usual beer law absorption formula can be used, with density f and step distance d.

[0218] o s =e -df

[0219] When performing integration, the cumulative result r can be initialized with the shadow surface color p, and then moving backward from the surface toward the camera, the color and opacity of the fog sample are calculated, and the usual alpha blending operation is applied to the cumulative result.

[0220] r0 = p

[0221] r s =r s-1 (1-o s )+c s o s

[0222] Running the algorithm with low values ​​of n samples along the line segment causes noticeable banding. Much like the case of color mixing, random numbers can be used to replace the banding with noise. In this embodiment, each primary hit sample (e.g., each pixel) is offset by a random value ξ ∈ [0; 1) to each sample point p. s The positions of the samples are still evenly spaced; they are only shifted forward or backward in depth.

[0223]

[0224] White noise was used to obtain screen-space white noise results. 2D blue noise was used to improve the error pattern. Spatiotemporal blue noise was used to obtain a screen-space blue noise error pattern, and the error magnitude was smaller. Figure 21 The rendering result can be seen in [the image / image], and [the image / image] can be seen in [the image / image]. Figure 22 The convergence graph can be seen in the image.

[0225] Figure 21 An example output generated according to at least one embodiment is shown, which uses noise to randomly offset the starting portion of the ray travel by 4 ray travel steps per pixel. The top image is the original frame, and the bottom image is a depth-aware Gaussian blur with 2 sigma.

[0226] Figure 22 A graph showing the convergence rate of ray-traveling fog with various types of noise according to at least one embodiment is presented. In the embodiment, only 4 ray-traveling steps are performed per pixel.

[0227] Participating medium with 2Dx1Dx1D blue noise

[0228] In one embodiment, one or more computing devices can execute an algorithm utilizing a 2Dx1Dx1D blue noise mask, where a previous algorithm utilized a 2Dx1D spatiotemporal blue noise mask. This algorithm can be used to indicate how many higher-dimensional blue noise masks can be used in a rendering algorithm. In both this algorithm and the previous algorithm, the goal is to integrate a single scattering participating medium. In the previous algorithm, samples at regular intervals are acquired along a line segment, and noise is used to cancel out the starting points of those samples in exchange for noise stripes. In this algorithm, the line segment can be divided into n evenly spaced portions, but instead of using only a single random offset for the entire sampling sequence, this algorithm reads the random offset for each sample. Then, n random values ​​ξ are obtained. s ∈[0,1), and the sampling position p s The following can be calculated:

[0229]

[0230] The rest of the algorithms remain the same. Figure 23 The rendering result can be seen in [the image / image], and [the image / image] can be seen in [the image / image]. Figure 24 The convergence plot can be seen in the image. This reformulation changes it from a ray-tracing technique to a hierarchical sampling technique, and if this is compared to spatiotemporal blue noise convergence, it improves for the same sample count.

[0231] Figure 23 The diagram illustrates the use of noise to layer 16 samples of line segments for each pixel through a participating medium, according to at least one embodiment. The top image is the original frame, and the bottom imager is a depth-aware Gaussian blur using 2 sigma.

[0232] Figure 24 A graph showing the convergence rate of ray-traveling fog with various types of noise according to at least one embodiment is presented. In the embodiment, only 4 ray-traveling steps are performed per pixel.

[0233] Ray-traced ambient occlusion (AO)

[0234] In this embodiment, ray tracing AO is another algorithm that can be used. In this embodiment, AO uses a 2D vector for each pixel to acquire each AO sample. In the techniques described herein, the algorithm generates a blue noise mask with scalar values ​​for each entry rather than each vector. In one or more embodiments, multiple independent streams of scalar values ​​can be derived from a single blue noise mask by a readout offset having approximately the maximum distance of each stream. Thus, in the ray tracing AO algorithm, this extension can be used as an independent spatiotemporal blue noise data stream for each axis. While other, more complex ray tracing and rasterization AO algorithms exist, this is intended to give high-quality results with low sample counts, such as 1 ray tracing sample per pixel (spp) – or even lower if running at sub-full resolution.

[0235] In this embodiment, the ray tracing AO algorithm runs after the principal ray strikes position p, and the surface normal n is known. N random 3D unit vectors ξ can be generated. i Add it to the surface normal n and normalize it to obtain N cosine-weighted hemispherical samples v. i The hemisphere is oriented towards the surface normal n.

[0236]

[0237] In the embodiment, each v i It is used to take a ray from position p to obtain the direction of the hit distance d. Because AO is a local shading phenomenon, the hit length can be limited to d. max The maximum value of scene dependence. The AO shadow value a of this ray. i This can be calculated as a percentage of how far the ray traveled relative to its maximum distance, which is attributed to closer hits that cause more occlusion and thus shadowing. This also allows for more information per sample than a binary hit or miss result, resulting in lower amplitude noise.

[0238]

[0239] Then you can adjust the AO shadow value a. i Calculate the average to give the combined AO shadow value 'a', which can be used as the shadow term in the lighting equation.

[0240]

[0241] If independent random numbers are used to generate ξ i For each component, the result will be white noise error. If 2D blue noise is used, the noise is spatially eliminated, and if spatiotemporal blue noise is used, the AO data acquires the desired sampling characteristics over time. Figure 25 The rendering result can be seen in [the image / image], and [the image / image] can be seen in [the image / image]. Figure 26The convergence graph can be seen in the image.

[0242] Spatiotemporal blue noise in HeitzBelcour technology

[0243] In embodiments, blue noise masks tend to show benefits when used in algorithms employing scalars, such as point drawing, color mixing, and ray travel as the participating medium. They also show benefits when used in simpler graphics algorithms that want vectors rather than scalars (such as AO in ray tracing) by employing multiple independent noise streams for each axis. In previous methods, blue noise masks also tend to stop working when sample counting or dimensionality increases (e.g., path tracing). However, in some embodiments, the techniques described herein can be used as extensions of algorithms for generating blue noise masks in path tracing (e.g., the Heitz & Belcoour technique). For example, in the Heitz & Belcoour technique, there may be a seed value per pixel generated by any desired means, which is used to render the result for each pixel using any desired algorithm and sampling sequence. After this rendering is complete, the Heitz & Belcoour technique can break the screen down into smaller 4x4-order segments and sort the pixels in each segment from darkest to brightest. Furthermore, the Heitz & Belcoour technique can break down cross-screen stitched blue noise textures into identical small parts and sort them. These two sorted lists serve as a mapping of how to exchange the seeds used for rendering previous frames, so that if rendered again, the result will be closer to the blue noise. An R2 low-difference sequence can be used to offset readings into this blue noise texture each frame, resulting in each frame having 2D blue noise values ​​that are largely unrelated to the previous frame. This gives a spatially blue noise result, but provides white noise over time. By combining the techniques described in this paper, the Heitz & Belcoour technique can provide spatiotemporal blue noise results, thus preserving the spatial quality of the blue noise while obtaining the desired sampling characteristics over time. Figure 28 The rendering results are shown in [the image], and [the image] is also shown in [the image]. Figure 30 The convergence plot is shown in the figure. Therefore, it should be possible to achieve virtually any target error pattern, such as the desired staggered gradient noise for better use in time-anti-aliasing scenarios.

[0244] Figure 25 The diagram illustrates the use of two independent noise streams to generate the x and y components of a 2D vector mapped to a cosine-weighted hemisphere for a single AO sample per pixel, according to at least one embodiment. The top image is the original frame, and the bottom image is a depth-aware Gaussian blurred with 2 sigma.

[0245] Figure 26The method of ambient occlusion (AO) convergence according to at least one embodiment is shown to be related to various types of noise.

[0246] Figure 27 Images using one or more of a 2D blue noise mask, a 3D blue noise mask, a spatiotemporal blue noise mask, and a 2DGR blue noise mask, according to at least one embodiment, are shown. In the embodiments, images from... Figure 27 The image shows the result rendered using Monte Carlo integral rendering, where each pixel has four samples.

[0247] Figure 28 An image offset using a Sobol sequence is shown according to at least one embodiment. In the embodiment, the Sobol sequence is offset by vec2 from each mg type of each frame. In the embodiment, the top image is the original frame, and the bottom image is a depth-aware Gaussian blur with 2 sigma.

[0248] Figure 29 The Heitz & Belcour technique, according to at least one embodiment, uses interleaved gradient noise and a stylized grayscale image for a noise pattern target. These images are rendered using standard path tracing rendering code, but the seeds used to randomize each pixel are reordered to give a rendering result as the target image.

[0249] Figure 30 A graph showing convergence in Monte Carlo integral, leakage integral, and convergent leakage integral according to at least one embodiment is shown.

[0250] In this embodiment, the spatiotemporal blue noise has the property of being thresholded to a certain percentage, such that a corresponding percentage of pixels will survive, and the surviving pixels will be distributed within the constraints of the dimensional group in a blue noise sample pattern. More specifically, if all pixels in a spatiotemporal 2Dx1D blue noise mask are thresholded to 10%, each 2D XY slice of the mask will show approximately 10% pixel survival, and they will be blue noise distributed (randomized but roughly uniformly spaced). Furthermore, viewing each pixel in isolation along the 1D Z-axis produces a 1D image where approximately 10% of the pixels will also survive, and they will also be blue noise distributed. These properties can be extended to any grouping of dimensions and sub-dimensions that generate the mask.

[0251] An example use case for this feature is in situations where an importance map for sparse ray tracing can be performed within a scene. Approximately how many rays are needed per frame can be defined, and this per-pixel count, along with a per-pixel random number, is used to determine whether a pixel should emit a ray (per frame). When using random numbers that are spatially and temporally white, clustering and gaps occur, resulting in non-uniform and redundant sampling both spatially and temporally. When using a folded book of independent 2D blue noise textures, the results improve spatially, but redundant sampling still exists over time. When using a spatiotemporal blue noise mask, both time and space are sampled more uniformly because the noise pattern can be the desired blue noise pattern in screen space, but for the same number of frames, more unique pixels will have rays emitted for them, thus maximizing the unique information received per frame, per ray. Figure 34 The graph shows the unique pixel count over time, and Figure 33 The image can also be seen visually in the middle.

[0252] Figure 31 This illustrates how a threshold mask, according to at least one embodiment, can form a point set of any density. That is, in Figure 31 In this study, 1024 blue noise samples were compared to different levels of thresholding using a 128x128x10 2Dx1D blue noise mask with the Best Transmission (BNOT) sample. BNOT is spatially much higher quality but has a fixed density and does not provide temporal processing, thus forcing the individual sample sets to become white noise over time.

[0253] Figure 32 This illustrates how the threshold point set, according to at least one embodiment, maintains its desired spectrum on the axis group. That is, Figure 32 The DFT of the 2D projection of a 64x64x64 2Dx1D blue noise mask with a 1 / 8 threshold is shown to illustrate how the set of threshold points preserves the mask’s inherent blue noise spectrum.

[0254] In this embodiment, a modification of the BNDS algorithm is described, wherein a blue noise mask of any dimension with blue noise characteristics limited to a set of subspace axes is generated. In this embodiment, these blue noise masks can be used in various low-sample-count rendering algorithms that aim to obtain the desired blue noise error pattern while converging faster than other methods using blue noise masks. In this embodiment, these blue noise masks may have a threshold that brings these properties into the blue noise sampling domain.

[0255] Figure 33Five cumulative frames of pixels sampled from an image are shown, according to at least one embodiment, using a non-uniform importance map to make pixels oriented towards the center more likely to be sampled. While both 2D blue noise and spatiotemporal blue noise have desired sampling patterns in space, spatiotemporal blue noise samples more unique pixels in a shorter number of frames.

[0256] Since importance sampling is a subject largely inconsistent with the use of specific sample patterns, blue noise itself more often happens to retain the desired properties when undergoing a distortion function. In embodiments, these blue noise masks can be extended to not only have the desired projection for each axis group, but also allow them to have a specific distribution for each axis group. This makes it possible to generate blue noise in the distorted space, meaning that the blue noise is not corrupted in any way, and important sampled PDFs can be baked into them. While some PDFs may be very specialized for their use and therefore may not bake as expected (such as HDRI skyline images), other PDFs will be more reused, such as GGX for specular reflections.

[0257] Figure 34 It is shown that white noise according to at least one embodiment can have redundant sampled pixels for each frame, and that spatial blue noise removes spatially redundant pixels over time, and 2Dx1D spatiotemporal blue noise also removes them over time.

[0258] While the techniques described herein involve blue noise masks, other colors of noise (e.g., red noise) can also be used to improve real-time image rendering and enhancement.

[0259] Reasoning and training logic

[0260] Figure 35A Inference and / or training logic 3515 is illustrated for performing inference and / or training operations associated with one or more embodiments. The following is in conjunction with... Figure 35A and / or Figure 35B Details regarding inference and / or training logic 3515 are provided. In at least one embodiment, inference and / or training logic 3515 can be implemented using process 900 (see...). Figure 9A ) or process 906 (see Figure 9B This can be achieved, for example, by using DLSS to render images. In at least one embodiment, the inference and / or training logic 2815 performs some or all of the processes 200 and / or 800.

[0261] In at least one embodiment, inference and / or training logic 3515 may include, but is not limited to, code and / or data storage 3501 for storing forward and / or output weights and / or input / output data, and / or other parameters configuring neurons or layers of a neural network trained for and / or used for inference in one or more embodiments. In at least one embodiment, training logic 3515 may include or be coupled to code and / or data storage 3501 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic, including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, code and / or data storage 3501 stores weight parameters and / or input / output data of each layer of a neural network trained or used in one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inference using one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 3501 may be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0262] In at least one embodiment, any portion of the code and / or data storage 3501 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 3501 may be a cache memory, dynamic random-addressable memory (“DRAM”), static random-addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 3501 is internal or external to the processor, for example, or composed of DRAM, SRAM, flash memory, or some other storage type, may depend on the available on-chip or off-chip storage space, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0263] In at least one embodiment, the inference and / or training logic 3515 may include, but is not limited to, code and / or data storage 3505 to store backpropagation and / or output weights and / or input / output data neural networks corresponding to neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, during training and / or inference using one or more embodiments, the code and / or data storage 3505 stores weight parameters and / or input / output data for each layer of a neural network trained or used in one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 3515 may include or be coupled to the code and / or data storage 3505 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic including integer and / or floating-point units (collectively referred to as Arithmetic Logic Units (ALUs)).

[0264] In at least one embodiment, code (such as graph code) causes the architecture of the neural network corresponding to that code to load weights or other parameter information into the processor ALU. In at least one embodiment, any portion of the code and / or data storage 3505 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 3505 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 3505 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice between the code and / or data storage 3505 being internal or external to the processor, for example, whether it consists of DRAM, SRAM, flash memory, or some other type of storage, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the data batch size used in the inference and / or training of the neural network, or some combination of these factors.

[0265] In at least one embodiment, code and / or data storage 3501 and code and / or data storage 3505 may be separate storage structures. In at least one embodiment, code and / or data storage 3501 and code and / or data storage 3505 may be the same storage structure. In at least one embodiment, code and / or data storage 3501 and code and / or data storage 3505 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data storage 3501 and code and / or data storage 3505 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0266] In at least one embodiment, the inference and / or training logic 3515 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 3510 (including integer and / or floating-point units) for performing logical and / or mathematical operations at least in part based on or instructed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from layers or neurons within a neural network) stored in activation storage 3520, which are functions of input / output and / or weight parameter data stored in code and / or data storage 3501 and / or code and / or data storage 3505. In at least one embodiment, activation is activated in response to execution instructions or other code, and linear algebraic and / or matrix-based mathematical generation performed by ALU 3510 is stored in activation storage 3520, wherein weight values ​​stored in code and / or data storage 3505 and / or code and / or data storage 3501 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, and any or all of these can be stored in code and / or data storage 3505 or code and / or data storage 3501 or other on-chip or off-chip storage.

[0267] In at least one embodiment, one or more processors or other hardware logic devices or circuits include one or more ALUs 3510, while in another embodiment, one or more ALUs 3510 may be located outside the processor or other hardware logic device or the circuitry that uses them (e.g., a coprocessor). In at least one embodiment, one or more ALUs 3510 may be included within an execution unit of a processor, or otherwise included in a group of ALUs accessible by the execution unit of the processor, which may be within the same processor or distributed among different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, code and / or data storage 3501, code and / or data storage 3505, and activation storage 3520 may share a processor or other hardware logic device or circuitry, while in another embodiment, they may be located in different processors or other hardware logic devices or circuitry, or in some combination of the same and different processors or other hardware logic devices or circuitry. In at least one embodiment, any portion of activation storage 3520 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be retrieved and / or processed using the processor’s fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0268] In at least one embodiment, the active memory 3520 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 3520 may be wholly or partially located inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory 3520 is internal to or external to the processor may depend on the available on-chip or off-chip storage, the latency requirements for training and / or inference functions, the batch size of data used in inference and / or training the neural network, or some combination of these factors. For example, it may include DRAM, SRAM, flash memory, or other memory types.

[0269] In at least one embodiment, Figure 35A The inference and / or training logic 3515 shown can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 35AThe inference and / or training logic 3515 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware or other hardware (such as field programmable gate array (“FPGA”)).

[0270] Figure 35B Inference and / or training logic 3515 according to at least one embodiment is illustrated. In at least one embodiment, the inference and / or training logic 3515 may include, but is not limited to, hardware logic, wherein computational resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 35B The inference and / or training logic 3515 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 35B The inference and / or training logic 3515 shown can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., field-programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 3515 includes, but is not limited to, code and / or data storage 3501 and code and / or data storage 3505, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 35B In at least one embodiment shown, each of code and / or data storage 3501 and code and / or data storage 3505 is associated with dedicated computing resources (e.g., computing hardware 3502 and computing hardware 3506), respectively. In at least one embodiment, each of computing hardware 3502 and computing hardware 3506 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) only on the information stored in code and / or data storage 3501 and code and / or data storage 3505, respectively, and the results of the function execution are stored in activation memory 3520.

[0271] In at least one embodiment, each of the code and / or data storage 3501 and 3505 and the corresponding computing hardware 3502 and 3506 corresponds to a different layer of the neural network, such that activations obtained from one “store / computation pair 3501 / 3502” of the code and / or data storage 3501 and computing hardware 3502 provide input as input to the next “store / computation pair 3505 / 3506” of the code and / or data storage 3505 and computing hardware 3506, in order to reflect the conceptual organization of the neural network. In at least one embodiment, each store / computation pair 3501 / 3502 and 3505 / 3506 may correspond to more than one neural network layer. In at least one embodiment, additional store / computation pairs (not shown) may be included in the inference and / or training logic 3515 after or in parallel with the store / computation pairs 3501 / 3502 and 3505 / 3506.

[0272] Neural network training and deployment

[0273] Figure 36 Training and deployment of a deep neural network according to at least one embodiment are illustrated. In at least one embodiment, an untrained neural network 3606 is trained using a training dataset 3602. In at least one embodiment, the untrained neural network 3606 can be implemented using process 900 or process 906 (see [link to documentation]). Figure 9B For example, DLSS or other neural network operations are used to render images. In at least one embodiment, the training framework 3604 is a PyTorch framework, while in other embodiments, the training framework 3604 is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training frameworks. In at least one embodiment, the training framework 3604 trains an untrained neural network 3606 and enables it to be trained using the processing resources described herein to generate a trained neural network 3608. In at least one embodiment, the weights may be randomly selected or pre-trained using a deep belief network. In at least one embodiment, training may be performed in a supervised, partially supervised, or unsupervised manner.

[0274] In at least one embodiment, supervised learning is used to train an untrained neural network 3606, wherein the training dataset 3602 includes inputs paired with desired outputs for input, or wherein the training dataset 3602 includes inputs with known outputs and the neural network 3606 is manually graded output. In at least one embodiment, the untrained neural network 3606 is trained in a supervised manner, and inputs from the training dataset 3602 are processed, and the resulting outputs are compared with a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through the untrained neural network 3606. In at least one embodiment, a training framework 3604 adjusts the weights controlling the untrained neural network 3606. In at least one embodiment, the training framework 3604 includes tools for monitoring the degree to which the untrained neural network 3606 converges to a model (e.g., a trained neural network 3608) adapted to generate the correct answer (e.g., result 3614) based on input data (e.g., a new dataset 3612). In at least one embodiment, the training framework 3604 repeatedly trains the untrained neural network 3606 while adjusting the weights to improve the output of the untrained neural network 3606 using a loss function and tuning algorithm (e.g., stochastic gradient descent). In at least one embodiment, the training framework 3604 trains the untrained neural network 3606 until the untrained neural network 3606 reaches the desired accuracy. In at least one embodiment, the trained neural network 3608 can then be deployed to implement any number of machine learning operations.

[0275] In at least one embodiment, unsupervised learning is used to train an untrained neural network 3606, wherein the untrained neural network 3606 attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised learning training dataset 3602 will include input data without any associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 3606 can learn groupings within the training dataset 3602 and can determine how each input relates to the untrained dataset 3602. In at least one embodiment, unsupervised training can be used to generate a self-organizing graph in a trained neural network 3608, which is capable of performing operations useful for reducing the dimensionality of the new dataset 3612. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows the identification of data points in the new dataset 3612 that deviate from the normal patterns of the new dataset 3612.

[0276] In at least one embodiment, semi-supervised learning can be used, a technique in which a mixture of labeled and unlabeled data is included in the training dataset 3602. In at least one embodiment, the training framework 3604 can be used to perform incremental learning, for example, through transfer learning techniques. In at least one embodiment, incremental learning enables the trained neural network 3608 to adapt to a new dataset 3612 without forgetting the knowledge injected into the trained neural network 3608 during initial training.

[0277] Data Center

[0278] Figure 37 An example data center 3700 that can be used with at least one embodiment is shown. In at least one embodiment, the data center 3700 includes a data center infrastructure layer 3710, a framework layer 3720, a software layer 3730, and an application layer 3740. The data center 3700 can implement process 900 or process 906 (see...). Figure 9B ).

[0279] In at least one embodiment, such as Figure 37 As shown, the data center infrastructure layer 3710 may include a resource coordinator 3712, packet computing resources 3714, and node computing resources (“nodes CR”) 3716(1)-3716(N), where “N” represents a positive integer (which may be an integer “N” different from the integers used in other diagrams). In at least one embodiment, nodes CR 3716(1)-3716(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), memory storage devices 3718(1)-3718(N) (e.g., dynamic read-only memory, solid-state drives, or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more nodes CR 3716(1)-3716(N) may be servers having one or more of the aforementioned computing resources.

[0280] In at least one embodiment, the grouped computing resource 3714 may include individual groups (not shown) of node CRs housed within one or more racks, or a plurality of racks (also not shown) housed within data centers in various geographical locations. In at least one embodiment, the individual groups of node CRs within the grouped computing resource 3714 may include computing, networking, memory, or storage resources that can be configured or allocated to support groups of one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, the one or more racks may also include any number of power modules, cooling modules, and network switches, in any combination.

[0281] In at least one embodiment, resource coordinator 3712 may be configured or otherwise control one or more nodes CR3716(1)-3716(N) and / or grouped computing resources 3714. In at least one embodiment, resource coordinator 3712 may include a software design infrastructure (“SDI”) management entity for data center 3700. In at least one embodiment, resource coordinator 3512 may include hardware, software, or some combination thereof.

[0282] In at least one embodiment, such as Figure 37 As shown, framework layer 3720 includes a job scheduler 3722, a configuration manager 3724, a resource manager 3726, and a distributed file system 3728. In at least one embodiment, framework layer 3720 may include a framework of software 3732 supporting software layer 3730 and / or one or more applications 3742 supporting application layer 3740. In at least one embodiment, software 3732 or application 3742 may respectively include web-based service software or applications, such as services or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, framework layer 3720 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark, which can utilize distributed file system 3728 for large-scale data processing (e.g., "big data"). TM(Hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 3722 may include a Spark driver to facilitate the scheduling of workloads supported by various layers of the data center 3700. In at least one embodiment, the configuration manager 3724 may be able to configure different layers, such as the software layer 3730 and the framework layer 3720, which includes Spark and a distributed file system 3728 for supporting large-scale data processing. In at least one embodiment, the resource manager 3726 is able to manage cluster or group computing resources mapped to or allocated to support the distributed file system 3728 and the job scheduler 3722. In at least one embodiment, the cluster or group computing resources may include group computing resources 3714 on the data center infrastructure layer 3710. In at least one embodiment, the resource manager 3726 may coordinate with the resource coordinator 3712 to manage these mapped or allocated computing resources.

[0283] In at least one embodiment, the software 3732 included in the software layer 3730 may include software used by at least a portion of the nodes CR3716(1)-3716(N), the grouped computing resources 3714, and / or the distributed file system 3728 of the framework layer 3720. In at least one embodiment, one or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0284] In at least one embodiment, one or more applications 3742 included in application layer 3740 may include one or more types of applications used by at least a portion of nodes CR3716(1)-3716(N), grouped computing resources 3714, and / or the distributed file system 3728 of framework layer 3720. In at least one embodiment, one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, applications, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0285] In at least one embodiment, any of the configuration manager 3724, resource manager 3726, and resource coordinator 3712 can perform any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. In at least one embodiment, self-modification actions can mitigate potentially poor configuration decisions by data center operators of data center 3700 and can prevent underutilization and / or poor performance of the data center.

[0286] In at least one embodiment, data center 3700 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model can be trained by calculating weight parameters based on a neural network architecture using the software and computing resources described above with respect to data center 3700. In at least one embodiment, information can be inferred or predicted using trained machine learning models corresponding to one or more neural networks using the resources described above with respect to data center 3700 by using weight parameters calculated through one or more training techniques described herein.

[0287] In at least one embodiment, the data center may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, or other hardware to utilize the aforementioned resources to perform training and / or inference. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as a service to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0288] Inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, the inference and / or training logic 3515 can be implemented in the system. Figure 37 Used in this context for inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0289] Supercomputing

[0290] The following figures illustrate, but are not limited to, exemplary supercomputer-based systems that can be used to implement at least one embodiment.

[0291] In at least one embodiment, a supercomputer can refer to a hardware system exhibiting substantial parallelism and comprising at least one chip, wherein the chips in the system are interconnected via a network and housed in a hierarchically organized enclosure. In at least one embodiment, a large hardware system filling a machine room with several racks is a specific example of a supercomputer, each rack containing several board / rack modules, each board / rack module containing several chips all interconnected via a scalable network. In at least one embodiment, a single rack of such a large hardware system is another example of a supercomputer. In at least one embodiment, a single chip exhibiting substantial parallelism and comprising several hardware components can also be considered a supercomputer because as feature size can be reduced, the amount of hardware that can be incorporated into a single chip can also increase.

[0292] Figure 38A A chip-level supercomputer according to at least one embodiment is illustrated. In at least one embodiment, within an FPGA or ASIC chip, the main computation is executed within a finite state machine (3804) called a thread unit. In one embodiment, the supercomputer 3800 may implement process 900 (see...). Figure 9A ) or process 906 (see Figure 9B In at least one embodiment, a task and synchronization network (3802) connects to a finite state machine and is used to schedule threads and perform operations in the correct order. In at least one embodiment, a memory network (3806, 3810) is used to access a multi-level cache hierarchy (3808, 3812) partitioned on the chip. In at least one embodiment, a memory controller (3816) and an off-chip memory network (3814) are used to access off-chip memory. In at least one embodiment, an I / O controller (3818) is used for cross-chip communication when the design is not suitable for a single logic chip.

[0293] Figure 38B A supercomputer at the rack module level is illustrated according to at least one embodiment. In at least one embodiment, within the rack module, there are multiple FPGA or ASIC chips (3820) connected to one or more DRAM cells (3822) constituting the main accelerator memory. In at least one embodiment, each FPGA / ASIC chip is connected to its adjacent FPGA / ASIC chip using a wide on-board bus with differential high-speed signaling (3824). In at least one embodiment, each FPGA / ASIC chip is also connected to at least one high-speed serial communication cable.

[0294] Figure 38C A rack-mounted supercomputer according to at least one embodiment is shown. Figure 38DA supercomputer at the entire system level is illustrated according to at least one embodiment. In at least one embodiment, reference is made to... Figure 38C and Figure 38D High-speed serial optical or copper cables (3826, 3828) are used to implement scalable, potentially incomplete, hypercube networks between rack modules within a rack and across racks throughout the system. In at least one embodiment, one of the FPGA / ASIC chips in the accelerator is connected to the host system via a PCI-Express connection (3830). In at least one embodiment, the host system includes a host microprocessor (3834) running the software portion of an application and a memory consisting of one or more host memory DRAM cells (3832) aligned with the memory on the accelerator. In at least one embodiment, the host system may be a standalone module on one of the racks or may be integrated with one of the modules of the supercomputer. In at least one embodiment, a cubic-connected loop topology provides communication links to create a hypercube network for a large supercomputer. In at least one embodiment, a group of FPGA / ASIC chips on a rack module may act as a single hypercube node, increasing the total number of external links per group compared to a single chip. In at least one embodiment, a group comprises chips A, B, C, and D on a rack module having an internal wide differential bus connecting A, B, C, and D in a toroidal organization. In at least one embodiment, there are 12 serial communication cables connecting the rack module to the outside world. In at least one embodiment, chip A on the rack module is connected to serial communication cables 0, 1, and 2. In at least one embodiment, chip B is connected to cables 3, 4, and 5. In at least one embodiment, chip C is connected to cables 6, 7, and 8. In at least one embodiment, chip D is connected to cables 9, 10, and 11. In at least one embodiment, the entire group {A, B, C, D} constituting the rack module can form a hypercube node within a supercomputer system, with up to 2^12 = 4096 rack modules (16384 FPGA / ASIC chips). In at least one embodiment, for chip A to send a message on link 4 of group {A, B, C, D}, the message must first be routed to chip B, which has an onboard differential wide bus connection. In at least one embodiment, messages arriving at group {A, B, C, D} (i.e., arriving at B) on link 4, destined for chip A, must also first be routed to the correct destination chip (A) within group {A, B, C, D}. In at least one embodiment, parallel supercomputer systems of other sizes can also be implemented.

[0295] Computer System

[0296] Figure 39This is a block diagram illustrating an exemplary computer system according to at least one embodiment. The exemplary computer system may be a system of interconnected devices and components, a system-on-a-chip (SOC), or some combination thereof formed with a processor, which may include an execution unit to execute instructions. In at least one embodiment, according to this disclosure, such as the embodiments described herein, computer system 3900 may include, but is not limited to, components such as processor 3902, whose execution unit includes logic to execute algorithms for process data. In at least one embodiment, computer system 3900 may include a processor, such as those available from Intel Corporation of Santa Clara, California. Processor family, Xeon TM , XScale TM and / or StrongARM TM , Core TM or Nervana TM A microprocessor may be used, although other systems (including PCs, engineering workstations, set-top boxes, etc.) with other microprocessors may also be used. In at least one embodiment, computer system 3900 may execute a version of the Windows operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (such as UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0297] The embodiments can be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol (IP) devices, digital cameras, personal digital assistants (“PDAs”), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor (“DSP”), a system-on-a-chip (SoC), a network computer (“NetPC”), a set-top box, a network hub, a wide area network (“WAN”) switch, or any other system that can execute one or more instructions according to at least one embodiment.

[0298] In at least one embodiment, the computer system 3900 may include, but is not limited to, a processor 3902, which may include, but is not limited to, one or more execution units 3908, to perform machine learning model training and / or inference according to the techniques described herein. In at least one embodiment, the computer system 3900 is a single-processor desktop or server system, but in another embodiment, the computer system 3900 may be a multiprocessor system. In at least one embodiment, the processor 3902 may include, but is not limited to, a Complex Instruction Set Computer (“CISC”) microprocessor, a Reduced Instruction Set Computing (“RISC”) microprocessor, a Very Long Instruction Word (“VLIW”) microprocessor, a processor implementing instruction set combination, or any other processor device, such as a digital signal processor. In at least one embodiment, the processor 3902 may be coupled to a processor bus 3910, which may transmit data signals between the processor 3902 and other components in the computer system 3900.

[0299] In at least one embodiment, processor 3902 may include, but is not limited to, a Level 1 (“L1”) internal cache memory (“cache”) 3904. In at least one embodiment, processor 3902 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory may reside external to processor 3902. Depending on specific implementation and requirements, other embodiments may also include a combination of internal and external caches. In at least one embodiment, register file 3906 may store different types of data in various registers, including but not limited to integer registers, floating-point registers, status registers, and instruction pointer registers.

[0300] In at least one embodiment, an execution unit 3908, including but not limited to logic for performing integer and floating-point operations, is also located within the processor 3902. In at least one embodiment, the processor 3902 may further include a microcode (“ucode”) read-only memory (“ROM”) for storing microcode of certain macro instructions. In at least one embodiment, the execution unit 3908 may include logic for processing a packaged instruction set 3909. In at least one embodiment, by including the packaged instruction set 3909 in the instruction set of a general-purpose processor, along with the associated circuitry for executing the instructions, the packaged data in the processor 3902 can be used to perform operations used by numerous multimedia applications. In at least one embodiment, many multimedia applications can be executed more quickly and efficiently by using the full width of the processor’s data bus to perform operations on the packaged data, which may eliminate the need to transfer smaller data units on the processor’s data bus to perform one or more operations on one data element at a time.

[0301] In at least one embodiment, the execution unit 3908 may also be used in a microcontroller, embedded processor, graphics device, DSP, and other types of logic circuitry. In at least one embodiment, the computer system 3900 may include, but is not limited to, memory 3920. In at least one embodiment, memory 3920 may be a dynamic random access memory (“DRAM”) device, a static random access memory (“SRAM”) device, a flash memory device, or another storage device. In at least one embodiment, memory 3920 may store instructions 3919 and / or data 3921 represented by data signals that can be executed by processor 3902.

[0302] In at least one embodiment, the system logic chip may be coupled to a processor bus 3910 and a memory 3920. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub (“MCH”) 3916, and the processor 3902 may communicate with the MCH 3916 via the processor bus 3910. In at least one embodiment, the MCH 3916 may provide a high-bandwidth memory path 3918 to the memory 3920 for instruction and data storage, as well as for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 3916 may initiate data signals between the processor 3902, the memory 3920, and other components in the computer system 3900, and bridge data signals between the processor bus 3910, the memory 3920, and the system I / O interface 3922. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, MCH 3916 can be coupled to memory 3920 via high-bandwidth memory path 3918, and graphics / video card 3912 can be coupled to MCH 3916 via Accelerated Graphics Port (“AGP”) interconnect 3914.

[0303] In at least one embodiment, the computer system 3900 may use the system I / O interface 3922 as a proprietary hub interface bus to couple the MCH 3916 to the I / O controller hub (“ICH”) 3930. In at least one embodiment, the ICH 3930 may provide direct connectivity to certain I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripheral devices to the memory 3920, chipset, and processor 3902. Examples may include, but are not limited to, an audio controller 3929, a firmware hub (“Flash BIOS”) 3928, a wireless transceiver 3926, a data storage 3924, a conventional I / O controller 3923 including a user input and keyboard interface 3925, a serial expansion port 3927 (e.g., a Universal Serial Bus (USB) port), and a network controller 3934. In at least one embodiment, the data storage 3924 may include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0304] In at least one embodiment, Figure 39 The illustration shows a system comprising interconnected hardware devices or "chips," while in other embodiments, Figure 39 The SoC can be shown. In at least one embodiment, Figure 39 The devices shown can be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of the computer system 3900 are interconnected using a Computational Fast Link (CXL) interconnect.

[0305] The inference and / or training logic 3515 is used to perform inference and / or training operations related to one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, the inference and / or training logic 3515 can... Figure 39 Used in systems for reasoning or predicting operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0306] Figure 40 This is a block diagram illustrating an electronic device 4000 for utilizing a processor 4010 according to at least one embodiment. In at least one embodiment, the electronic device 4000 may be, for example, but not limited to, a laptop computer, tower server, rack server, blade server, desktop computer, tablet computer, mobile device, telephone, embedded computer, or any other suitable electronic device.

[0307] In at least one embodiment, the electronic device 4000 may, but is not limited to, a processor 4010 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 4010 is coupled using a bus or interface, such as I... 2 C-bus, System Management Bus (“SMBus”), Low Pin Count (LPC) bus, Serial Peripheral Interface (“SPI”), High Definition Audio (“HDA”) bus, Serial Advanced Technology Accessory (“SATA”) bus, Universal Serial Bus (“USB”) (versions 1, 2, 3, etc.), or Universal Asynchronous Receiver / Transmitter (“UART”) bus. In at least one embodiment, Figure 40 The system shown includes interconnected hardware devices or "chips," while in other embodiments, Figure 40 An exemplary SoC can be shown. In at least one embodiment, Figure 40 The device shown can be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, Figure 40 One or more components are interconnected using Computational Fast Link (CXL) interconnects.

[0308] In at least one embodiment, Figure 40 It may include a display 4024, a touch screen 4025, a touchpad 4030, a near field communication unit (“NFC”) 4045, a sensor hub 4040, a thermal sensor 4046, a fast chipset (“EC”) 4035, a trusted platform module (“TPM”) 4038, a BIOS / firmware / flash (“BIOS, FW Flash”) 4022, a DSP 4060, a drive 4020 (e.g., a solid-state drive (“SSD”) or a hard disk drive (“HDD”)), a wireless local area network unit (“WLAN”) 4050, a Bluetooth unit 4052, a wireless wide area network unit (“WWAN”) 4056, a global positioning system (GPS) unit 4055, a camera (“USB 3.0 camera”) 4054 (e.g., a USB 3.0 camera), and / or a low-power double data rate (“LPDDR”) memory unit (“LPDDR3”) 4015 implemented in, for example, the LPDDR3 standard. These components can each be implemented in any suitable way.

[0309] In at least one embodiment, other components may be communicatively coupled to processor 4010 via the components described herein. In at least one embodiment, accelerometer 4041, ambient light sensor (“ALS”) 4042, compass 4043, and gyroscope 4044 may be communicatively coupled to sensor hub 4040. In at least one embodiment, thermal sensor 4039, fan 4037, keyboard 4036, and touchpad 4030 may be communicatively coupled to EC 4035. In at least one embodiment, speaker 4063, earphone 4064, and microphone (“mic”) 4065 may be communicatively coupled to audio unit (“audio codec and Class D amplifier”) 4062, which in turn may be communicatively coupled to DSP 4060. In at least one embodiment, audio unit 4062 may include, for example, but not limited to, audio encoder / decoder (“codec”) and Class D amplifier. In at least one embodiment, SIM card (“SIM”) 4057 may be communicatively coupled to WWAN unit 4056. In at least one embodiment, components such as WLAN unit 4050, Bluetooth unit 4052, and WWAN unit 4056 can be implemented as next-generation form factor (NGFF).

[0310] Inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, the inference and / or training logic 3515 can be implemented in the system. Figure 40 It is used in the context of reasoning or predicting operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0311] Figure 41 A computer system 4100 according to at least one embodiment is shown. In at least one embodiment, the computer system 4100 is configured to implement various processes and methods described throughout this disclosure.

[0312] In at least one embodiment, the computer system 4100 includes, but is not limited to, at least one central processing unit (“CPU”) 4102 connected to a communication bus 4110 implemented using any suitable protocol, such as PCI (“Peripheral Device Interconnect”), Peripheral Component Interconnect Express (“PCI-Express”), AGP (“Accelerated Graphics Port”), HyperTransport, or any other bus or point-to-point communication protocol. In at least one embodiment, the computer system 4100 includes, but is not limited to, main memory 4104 and control logic (e.g., implemented in hardware, software, or a combination thereof), and data may be stored in main memory 4104 in the form of random access memory (“RAM”). In at least one embodiment, a network interface subsystem (“Network Interface”) 4122 provides an interface to other computing devices and networks for receiving data using the computer system 4100 and transferring data to other systems.

[0313] In at least one embodiment, the computer system 4100 includes, but is not limited to, an input device 4108, a parallel processing system 4112, and a display device 4106, which may be implemented using conventional cathode ray tube (“CRT”), liquid crystal display (“LCD”), light-emitting diode (“LED”) display, plasma display, or other suitable display technologies. In at least one embodiment, user input is received from the input device 4108 (such as a keyboard, mouse, touchpad, microphone, etc.). In at least one embodiment, each of the modules described herein may reside on a single semiconductor platform to form the processing system.

[0314] Inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, the inference and / or training logic 3515 can be implemented in the system. Figure 41 It is used to perform inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architecture or neural network use cases described herein.

[0315] Figure 42A computer system 4200 according to at least one embodiment is illustrated. In at least one embodiment, the computer system 4200 includes, but is not limited to, a computer 4210 and a USB stick 4220. In at least one embodiment, the computer 4210 may include, but is not limited to, any number and type of processors (not shown) and memory (not shown). In at least one embodiment, the computer 4210 includes, but is not limited to, a server, a cloud instance, a laptop computer, and a desktop computer.

[0316] In at least one embodiment, the USB stick 4220 includes, but is not limited to, a processing unit 4230, a USB interface 4240, and USB interface logic 4250. In at least one embodiment, the processing unit 4230 can be any instruction execution system, apparatus, or device capable of executing instructions. In at least one embodiment, the processing unit 4230 can include, but is not limited to, any number and type of processing cores (not shown). In at least one embodiment, the processing unit 4230 includes an application-specific integrated circuit (“ASIC”) optimized to perform any amount and type of operations associated with machine learning. For example, in at least one embodiment, the processing unit 4230 is a tensor processing unit (“TPC”) optimized to perform machine learning inference operations. In at least one embodiment, the processing unit 4230 is a vision processing unit (“VPU”) optimized to perform machine vision and machine learning inference operations.

[0317] In at least one embodiment, the USB interface 4240 can be any type of USB connector or USB receptacle. For example, in at least one embodiment, the USB interface 4240 is a USB 3.0 Type-C receptacle for data and power. In at least one embodiment, the USB interface 4240 is a USB 3.0 Type-A connector. In at least one embodiment, the USB interface logic 4250 may include any amount and type of logic enabling the processing unit 4230 to interface with a device (e.g., computer 4210) via the USB connector 4240.

[0318] Inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. Details regarding inference and / or training logic 3515 are combined herein. Figure 35A And / or 35B is provided. In at least one embodiment, the inference and / or training logic 3515 can be provided in the system. Figure 42 The operation is used to infer or predict based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architecture, or neural network use cases as described herein.

[0319] Figure 43AAn exemplary architecture is illustrated in which multiple GPUs 4310(1)-4310(N) are communicatively coupled to multiple multi-core processors 4305(1)-4305(M) via high-speed links 4340(1)-4340(N) (e.g., bus / point-to-point interconnect, etc.). In at least one embodiment, the high-speed links 4340(1)-4340(N) support communication throughput of 4GB / s, 30GB / s, 80GB / s, or higher. In at least one embodiment, various interconnect protocols may be used, including but not limited to PCIe 4.0 or 5.0 and NVLink 2.0. In the various figures, “N” and “M” represent positive integers, the values ​​of which may vary from figure to figure.

[0320] Furthermore, in at least one embodiment, two or more GPUs 4310 are interconnected via high-speed links 4329(1)-4329(2), which can be implemented using a protocol / link similar to or different from that used for high-speed links 4340(1)-4340(N). Similarly, two or more multi-core processors 4305 can be connected via a high-speed link 4328, which can be a symmetric multiprocessor (SMP) bus operating at speeds of 20GB / s, 30GB / s, 120GB / s, or higher. Alternatively, similar protocols / links (e.g., via a common interconnect structure) can be used. Figure 43A This shows all communication between the various system components.

[0321] In at least one embodiment, each multi-core processor 4305 is communicatively coupled to processor memories 4301(1)-4301(M) via memory interconnects 4326(1)-4326(M), and each GPU 4310(1)-4310(N) is communicatively coupled to GPU memories 4320(1)-4320(N) via GPU memory interconnects 4350(1)-4350(N). In at least one embodiment, memory interconnects 4326 and 4350 may utilize similar or different memory access technologies. By way of example and not limitation, processor memories 4301(1)-4301(M) and GPU memories 4320 may be volatile memories, such as dynamic random access memory (DRAM) (including stacked DRAM), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high bandwidth memory (HBM), and / or may be non-volatile memories, such as 3D XPoint or Nano-RAM. In at least one embodiment, some portions of the processor memory 4301 may be volatile memory, while other portions may be non-volatile memory (e.g., using a two-level memory (2LM) hierarchy).

[0322] As described herein, although the various multi-core processors 4305 and GPUs 4310 can be physically coupled to specific memories 4301 and 4320 respectively, and / or can implement a unified memory architecture, in which the virtual system address space (also known as the “effective address” space) is distributed among the various physical memories. For example, processor memories 4301(1)-4301(M) can each contain 64GB of system memory address space, and GPU memories 4320(1)-4320(N) can each contain 32GB of system memory address space, resulting in a total addressable memory size of 256GB when M=2 and N=4. N and M may also be other values.

[0323] Figure 43B Additional details are shown regarding the interconnection between a multi-core processor 4307 and a graphics acceleration module 4346 according to an exemplary embodiment. In at least one embodiment, the graphics acceleration module 4346 may include one or more GPU chips integrated on a line card coupled to the processor 4307 via a high-speed link 4340 (e.g., PCIe bus, NVLink, etc.). In at least one embodiment, the graphics acceleration module 4346 may optionally be integrated on a package or chip having the processor 4307.

[0324] In at least one embodiment, the processor 4307 includes a plurality of cores 4360A-4360D, each core having a translation back cover buffer (“TLB”) 4361A-4361D and one or more caches 4362A-4362D. In at least one embodiment, the cores 4360A-4360D may include various other components (not shown) for executing instructions and processing data. In at least one embodiment, the caches 4362A-4362D may include level 1 (L1) and level 2 (L2) caches. Furthermore, one or more shared caches 4356 may be included in the caches 4362A-4362D and shared by the respective groups of cores 4360A-4360D. For example, one embodiment of the processor 4307 includes 24 cores, each core having its own L1 cache, twelve shared L2 caches, and twelve shared L3 caches. In this embodiment, two adjacent cores share one or more L2 and L3 caches. In at least one embodiment, the processor 4307 and the graphics acceleration module 4346 are connected to the system memory 4314, which may include... Figure 43A The processor memory in the memory is 4301(1)-4301(M).

[0325] In at least one embodiment, consistency of data and instructions stored in the various caches 4362A-4362D, 4356 and system memory 4314 is maintained via inter-core communication through the consistency bus 4364. In at least one embodiment, for example, each cache may have associated cache consistency logic / circuit to communicate via the consistency bus 4364 in response to the detection of a read or write to a particular cache line. In at least one embodiment, a cache snooping protocol is implemented via the consistency bus 4364 to snoop on cache accesses.

[0326] In at least one embodiment, the proxy circuit 4325 communicatively couples the graphics acceleration module 4346 to the coherence bus 4364, thereby allowing the graphics acceleration module 4346 to participate in the cache coherence protocol as a peer of cores 4360A-4360D. Specifically, in at least one embodiment, interface 4335 provides connectivity to the proxy circuit 4325 via high-speed link 4340, and interface 4337 connects the graphics acceleration module 4346 to high-speed link 4340.

[0327] In at least one embodiment, the accelerator integrated circuit 4336 provides cache management, memory access, context management, and interrupt management services for a plurality of graphics processing engines 4331(1)-4331(N) of the graphics acceleration module 4346. In at least one embodiment, the graphics processing engines 4331(1)-4331(N) may each include a separate graphics processing unit (GPU). In at least one embodiment, the graphics processing engines 4331(1)-4331(N) may optionally include different types of graphics processing engines within the GPU, such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and blit engines. In at least one embodiment, the graphics acceleration module 4346 may be a GPU having a plurality of graphics processing engines 4331(1)-4331(N), or the graphics processing engines 4331(1)-4331(N) may be individual GPUs integrated on a general-purpose package, line card, or chip.

[0328] In at least one embodiment, the accelerator integrated circuit 4336 includes a memory management unit (MMU) 4339 for performing various memory management functions, such as virtual-to-physical memory translation (also known as effective-to-real memory translation), and a memory access protocol for accessing system memory 4314. In at least one embodiment, the MMU 4339 may also include a translation back buffer (“TLB”) (not shown) for caching virtual / effective-to-physical / real address translations. In at least one embodiment, cache 4338 may store commands and data for efficient access by graphics processing engines 4331(1)-4331(N). In at least one embodiment, a fetch unit 4344 may be used to keep data stored in cache 4338 and graphics memory 4333(1)-4333(M) consistent with core caches 4362A-4362D, 4356 and system memory 4314. As previously mentioned, this task can be accomplished via proxy circuitry 4325 representing cache 4338 and graphics memory 4333(1)-4333(M) (e.g., sending updates related to the modification / access of cache lines on processor caches 4362A-4362D, 4356 to cache 4338 and receiving updates from cache 4338).

[0329] In at least one embodiment, a set of registers 4345 stores context data of threads executed by graphics processing engines 4331(1)-4331(N), and context management circuitry 4348 manages the thread context. For example, context management circuitry 4348 can perform save and restore operations to save and restore the context of individual threads during context switching (e.g., saving the first thread and storing the second thread so that the second thread can be executed by the graphics processing engine). For example, during context switching, context management circuitry 4348 can store the current register value in a designated area of ​​memory (e.g., identified by a context pointer). The register value can then be restored when returning to the context. In at least one embodiment, interrupt management circuitry 4347 receives and processes interrupts received from system devices.

[0330] In at least one embodiment, the virtual / effective address from the graphics processing engine 4331 is translated into a real / physical address in system memory 4314 via MMU 4339. In at least one embodiment, the accelerator integrated circuit 4336 supports multiple (e.g., 4, 8, 16) graphics accelerator modules 4346 and / or other accelerator devices. In at least one embodiment, the graphics accelerator module 4346 may be dedicated to a single application executing on processor 4307, or may be shared among multiple applications. In at least one embodiment, a virtualized graphics execution environment is presented, wherein the resources of the graphics processing engines 4331(1)-4331(N) are shared with multiple applications or virtual machines (VMs). In at least one embodiment, the resources may be subdivided into “slices” based on processing requirements and priorities associated with VMs and / or applications, which are allocated to different VMs and / or applications.

[0331] In at least one embodiment, the accelerator integrated circuit 4336 acts as a bridge to the system of the graphics acceleration module 4346 and provides address translation and system memory caching services. Additionally, in at least one embodiment, the accelerator integrated circuit 4336 can provide virtualization facilities for the host processor to manage the virtualization, interrupt, and memory management of the graphics processing engines 4331(1)-4331(N).

[0332] In at least one embodiment, since the hardware resources of the graphics processing engines 4331(1)-4331(N) are explicitly mapped to the real address space seen by the host processor 4307, any host processor can directly address these resources using valid address values. In at least one embodiment, a function of the accelerator integrated circuit 4336 is to physically separate the graphics processing engines 4331(1)-4331(N) so that they appear as independent units to the system.

[0333] In at least one embodiment, one or more graphics memories 4333(1)-4333(M) are coupled to each graphics processing engine 4331(1)-4331(N), and N = M. In at least one embodiment, the graphics memories 4333(1)-4333(M) store instructions and data processed by each graphics processing engine 4331(1)-4331(N). In at least one embodiment, the graphics memories 4333(1)-4333(M) may be volatile memories, such as DRAM (including stacked DRAM), GDDR memories (e.g., GDDR5, GDDR6), or HBM, and / or may be non-volatile memories, such as 3DXPoint or Nano-RAM.

[0334] In at least one embodiment, to reduce data traffic on the high-speed link 4340, a biasing technique can be used to ensure that the data stored in the graphics memory 4333(1)-4333(M) is the data most frequently used by the graphics processing engine 4331(1)-4331(N), and preferably data that the cores 4360A-4360D do not use (or at least do not use frequently). Similarly, in at least one embodiment, the biasing mechanism attempts to keep the data needed by the cores (and preferably not the graphics processing engine 4331(-1)-4331(N)) in the caches 4362A-4362D, 4356 and system memory 4314.

[0335] Figure 43C Another exemplary embodiment is shown, wherein the accelerator integrated circuit 4336 is integrated within the processor 4307. In this embodiment, the graphics processing engines 4331(1)-4331(N) communicate directly with the accelerator integrated circuit 4336 via a high-speed link 4340 through interfaces 4337 and 4335 (which may also be any form of bus or interface protocol). In at least one embodiment, the accelerator integrated circuit 4336 can perform operations related to... Figure 43B The described operation is similar. However, due to its close proximity to the coherence bus 4364 and caches 4362A-4362D, 4356, it may have higher throughput. In at least one embodiment, the accelerator integrated circuit supports different programming models, including a dedicated process programming model (without graphics acceleration module virtualization) and a shared programming model (with virtualization), which may include a programming model controlled by the accelerator integrated circuit 4336 and a programming model controlled by the graphics acceleration module 4346.

[0336] In at least one embodiment, graphics processing engines 4331(1)-4331(N) are dedicated to a single application or process under a single operating system. In at least one embodiment, a single application can funnel requests from other applications to graphics processing engines 4331(1)-4331(N), thereby providing virtualization within a VM / partition.

[0337] In at least one embodiment, graphics processing engines 4331(1)-4331(N) can be shared by multiple VM / application partitions. In at least one embodiment, the shared model can use a hypervisor to virtualize graphics processing engines 4331(1)-4331(N) to allow each operating system to access them. In at least one embodiment, for a single-partition system without a hypervisor, the operating system owns graphics processing engines 4331(1)-4331(N). In at least one embodiment, the operating system can virtualize graphics processing engines 4331(1)-4331(N) to provide access to each process or application.

[0338] In at least one embodiment, the graphics acceleration module 4346 or the individual graphics processing engine 4331(1)-4331(N) uses a process handle to select a process element. In at least one embodiment, the process element is stored in system memory 4314 and can be addressed using the effective address to real address translation techniques described herein. In at least one embodiment, the process handle may be an implementation-specific value provided to the host process when registering its context with the graphics processing engine 4331(1)-4331(N) (i.e., invoking system software to add the process element to the process element linked list). In at least one embodiment, the lower 16 bits of the process handle may be the offset of the process element in the process element linked list.

[0339] Figure 43D An exemplary accelerator integration slice 4390 is illustrated. In at least one embodiment, a "slice" includes a designated portion of the processing resources of an accelerator integrated circuit 4336. In at least one embodiment, the application is an effective address space 4382 in system memory 4314, which stores process element 4383. In at least one embodiment, process element 4383 is stored in response to a GPU call 4381 from an application 4380 executing on processor 4307. In at least one embodiment, process element 4383 contains the process state of the corresponding application 4380. In at least one embodiment, a job descriptor (WD) 4384 contained in process element 4383 may be a single job requested by the application, or it may contain a pointer to a job queue. In at least one embodiment, WD 4384 is a pointer to a job request queue in the effective address space 4382 of the application.

[0340] In at least one embodiment, the graphics acceleration module 4346 and / or the various graphics processing engines 4331(1)-4331(N) may be shared by all processes or a subset of processes in the system. In at least one embodiment, infrastructure may be included for setting process states and sending WD 4384 to the graphics acceleration module 4346 to begin operations in a virtualized environment.

[0341] In at least one embodiment, the dedicated process programming model is implementation-specific. In at least one embodiment, in this model, a single process owns either the graphics acceleration module 4346 or an individual graphics processing engine 4331. In at least one embodiment, when the graphics acceleration module 4346 is owned by a single process, the hypervisor initializes the accelerator integrated circuit 4336 for the owned partition; when the graphics acceleration module 4346 is assigned, the operating system initializes the accelerator integrated circuit 4336 for the owned process.

[0342] In at least one embodiment, during operation, the WD acquisition unit 4391 in the accelerator integration slice 4390 acquires the next WD 4384, which includes instructions for work to be performed by one or more graphics processing engines of the graphics acceleration module 4346. In at least one embodiment, data from the WD 4384 may be stored in register 4345 and used by the MMU 4339, interrupt management circuitry 4347, and / or context management circuitry 4348, as shown. For example, one embodiment of the MMU 4339 includes segment / page roaming circuitry for accessing segment / page tables 4386 within the OS virtual address space 4385. In at least one embodiment, the interrupt management circuitry 4347 may handle interrupt events 4392 received from the graphics acceleration module 4346. In at least one embodiment, when performing graphics operations, a valid address 4393 generated by graphics processing engines 4331(1)-4331(N) is translated into a real address by the MMU 4339.

[0343] In one embodiment, register 4345 is copied for each graphics processing engine 4331(1)-4331(N) and / or graphics acceleration module 4346, and register 4345 may be initialized by a hypervisor or operating system. In at least one embodiment, each of these copied registers may be included in accelerator integration slice 4390. Exemplary registers that may be initialized by a hypervisor are shown in Table 1.

[0344]

[0345]

[0346] Table 2 shows exemplary registers that can be initialized by the operating system.

[0347]

[0348] In at least one embodiment, each WD 4384 is specific to a particular graphics acceleration module 4346 and / or graphics processing engine 4331(1)-4331(N). In at least one embodiment, it contains all the information required for the graphics processing engine 4331(1)-4331(N) to complete its work, or it may be a pointer to a memory location where the application has set up a command queue for the work to be completed.

[0349] Figure 43E Additional details of an exemplary embodiment of the shared model are shown. This embodiment includes a hypervisor real address space 4398, in which a list of process elements 4399 is stored. In at least one embodiment, the hypervisor real address space 4398 can be accessed via a hypervisor 4396, which virtualizes the graphics acceleration module engine for an operating system 4395.

[0350] In at least one embodiment, the shared programming model allows all processes or subsets of processes from all partitions or subsets of partitions in the system to use the graphics acceleration module 4346. In at least one embodiment, there are two programming models in which the graphics acceleration module 4346 is shared by multiple processes and partitions, namely, time-slice sharing and graphics-oriented sharing.

[0351] In at least one embodiment, in this model, the hypervisor 4396 owns the graphics acceleration module 4346 and makes its functionality available to all operating systems 4395. In at least one embodiment, for the graphics acceleration module 4346 to support virtualization through the hypervisor 4396, the graphics acceleration module 4346 may comply with certain requirements, such as (1) the job requests of the application must be autonomous (i.e., no need to maintain state between jobs), or the graphics acceleration module 4346 must provide a context saving and recovery mechanism, (2) the graphics acceleration module 4346 guarantees that the job requests of the application are completed within a specified amount of time, including any conversion errors, or the graphics acceleration module 4346 provides the ability to preempt job processing, and (3) when operating in a directed shared programming model, fairness between the processes of the graphics acceleration module 4346 must be ensured.

[0352] In at least one embodiment, application 4380 needs to make system calls to operating system 4395 using the graphics acceleration module type, working descriptor (WD), permission mask register (AMR) value, and context save / restore region pointer (CSRP). In at least one embodiment, the graphics acceleration module type describes the target acceleration function for the system call. In at least one embodiment, the graphics acceleration module type can be a system-specific value. In at least one embodiment, the WD is specifically formatted for graphics acceleration module 4346 and can take the form of graphics acceleration module 4346 commands, valid address pointers to user-defined structures, valid address pointers to command queues, or any other data structure describing the work to be performed by graphics acceleration module 4346.

[0353] In at least one embodiment, the AMR value is the AMR state for the current process. In at least one embodiment, the value passed to the operating system is similar to that of the application that sets the AMR. In at least one embodiment, if the implementation of the accelerator integrated circuit 4336 (not shown) and the graphics acceleration module 4346 does not support the User Rights Mask Overwrite Register (UAMOR), the operating system may apply the current UAMOR value to the AMR value before passing the AMR in the hypervisor call. In at least one embodiment, the hypervisor 4396 may selectively apply the current Rights Mask Overwrite Register (AMOR) value before placing the AMR into the process element 4383. In at least one embodiment, CSRP is one of the registers 4345 that contains the effective address of a region in the effective address space 4382 of the application for the graphics acceleration module 4346 to save and restore the context state. In at least one embodiment, this pointer is optional if it is not necessary to save state between jobs or when a job is preempted. In at least one embodiment, the context save / restore region may be fixed system memory.

[0354] Upon receiving a system call, the operating system 4395 can verify that the application 4380 has been registered and granted permission to use the graphics acceleration module 4346. Then, in at least one embodiment, the operating system 4395 uses the information shown in Table 3 to invoke the hypervisor 4396.

[0355]

[0356]

[0357] In at least one embodiment, upon receiving a hypervisor call, the hypervisor 4396 verifies that the operating system 4395 has been registered and granted permission to use the graphics acceleration module 4346. Then, in at least one embodiment, the hypervisor 4396 adds the process element 4383 to a linked list of process elements of the corresponding graphics acceleration module 4346 type. In at least one embodiment, the process element may include the information shown in Table 4.

[0358]

[0359] In at least one embodiment, the hypervisor initializes multiple accelerator integration slice 4390 registers 4345.

[0360] like Figure 43F As shown, in at least one embodiment, a unified memory is used, which is addressable via a common virtual memory address space for accessing physical processor memories 4301(1)-4301(N) and GPU memories 4320(1)-4320(N). In this implementation, operations performed on GPUs 4310(1)-4310(N) utilize the same virtual / effective memory address space to access processor memories 4301(1)-4301(M) and vice versa, thereby simplifying programmability. In at least one embodiment, a first portion of the virtual / effective address space is allocated to processor memory 4301(1), a second portion to second processor memory 4301(N), a third portion to GPU memory 4320(1), and so on. In at least one embodiment, the entire virtual / effective memory space (sometimes referred to as the effective address space) is thus distributed across each of processor memory 4301 and GPU memory 4320, thereby allowing any processor or GPU to access that memory using a virtual address mapped to any physical memory.

[0361] In at least one embodiment, the bias / coherence management circuitry 4394A-4394E within one or more MMUs 4339A-4339E ensures cache coherence between the caches of one or more host processors (e.g., 4305) and the GPU 4310, and implements biasing techniques to indicate the physical memory in which certain types of data should be stored. In at least one embodiment, although in Figure 43F Several instances of bias / coherence management circuitry 4394A-4394E are shown, but bias / coherence circuitry can be implemented within the MMU of one or more host processors 4305 and / or within the accelerator integrated circuit 4336.

[0362] One embodiment allows GPU memory 4320 to be mapped as part of system memory and accessed using shared virtual memory (SVM) technology without suffering the performance drawbacks associated with full system cache coherence. In at least one embodiment, the ability to access GPU memory 4320 as system memory without the heavy overhead of cache coherence provides a favorable operating environment for GPU offloading. In at least one embodiment, this arrangement allows the host processor 4305 to software-set operands and access computation results without the overhead of conventional I / ODMA data copying. In at least one embodiment, such conventional copying includes driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, all of which are less efficient than simple memory accesses. In at least one embodiment, the ability to access GPU memory 4320 without cache coherence overhead can be critical to the execution time of offloaded computations. In at least one embodiment, for example, in cases with high streaming write memory traffic, cache coherence overhead can significantly reduce the effective write bandwidth seen by GPU 4310. In at least one embodiment, the efficiency of operand setting, the efficiency of result access, and the efficiency of GPU computation can play a role in determining the effectiveness of GPU offloading.

[0363] In at least one embodiment, the selection of GPU bias and host processor bias is driven by a bias tracker data structure. In at least one embodiment, for example, a bias table can be used, which may be a page-granular structure (e.g., controlled at the memory page level) comprising one or two bits of memory pages attached to each GPU. In at least one embodiment, with or without a bias cache (e.g., for caching frequently / recently used entries in the bias table) in GPU 4310, the bias table can be implemented across one or more stolen memory ranges of GPU memory 4320. Alternatively, in at least one embodiment, the entire bias table can be maintained within the GPU.

[0364] In at least one embodiment, prior to actual access to GPU memory, an access to the bias table entry associated with each access to GPU-attached memory 4320 is performed, resulting in the following operations: In at least one embodiment, a local request from GPU 4310 to find its page in the GPU bias is forwarded directly to the corresponding GPU memory 4320. In at least one embodiment, a local request from GPU to find its page in the host bias is forwarded to processor 4305 (e.g., via the high-speed link described herein). In at least one embodiment, a request from processor 4305 to find the requested page in the host processor bias completes a request similar to a normal memory read. Alternatively, a request for a page pointing to the GPU bias can be forwarded to GPU 4310. In at least one embodiment, if the GPU is not currently using the page, the GPU may subsequently migrate the page to the host processor bias. In at least one embodiment, the page bias state can be changed through a software-based mechanism, a hardware-assisted software mechanism, or, in limited cases, a purely hardware-based mechanism.

[0365] In at least one embodiment, a mechanism for changing the bias state employs an API call (e.g., OpenCL), which subsequently invokes the GPU's device driver. The device driver then sends a message (or enqueues a command descriptor) to the GPU, instructing the GPU to change the bias state and, in some migration, performs a cache refresh operation on the host. In at least one embodiment, the cache refresh operation is used for migration from the host processor 4305 bias to the GPU bias, but not for the reverse migration.

[0366] In one embodiment, cache coherence is maintained by temporarily rendering GPU bias pages that the host processor 4305 cannot cache. In at least one embodiment, to access these pages, the processor 4305 may request access from the GPU 4310, which may or may not immediately grant access. Therefore, in at least one embodiment, to reduce communication between the processor 4305 and the GPU 4310, it is beneficial to ensure that the GPU bias pages are pages needed by the GPU, not those needed by the host processor 4305, and vice versa.

[0367] Figure 44 Exemplary integrated circuits and associated graphics processors according to various embodiments described herein are illustrated, which may be manufactured using one or more IP cores. In addition to the illustrations, at least one embodiment may include other logic and circuitry, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0368] Figure 44This is a block diagram illustrating an exemplary system on a chip integrated circuit 4400 that may be fabricated using one or more IP cores according to at least one embodiment. In at least one embodiment, the integrated circuit 4400 includes one or more application processors 4405 (e.g., CPUs), at least one graphics processor 4410, and may additionally include an image processor 4415 and / or a video processor 4420, any of which may be a modular IP core. In at least one embodiment, the integrated circuit 4400 includes peripheral or bus logic including a USB controller 4425, a UART controller 4430, an SPI / SDIO controller 4435, and an I22S / I22C controller 4440. In at least one embodiment, the integrated circuit 4400 may include a display device 4445 coupled to one or more of a High Definition Multimedia Interface (HDMI) controller 4450 and a Mobile Industrial Processor Interface (MIPI) display interface 4455. In at least one embodiment, storage may be provided by a flash memory subsystem 4460, including flash memory and a flash memory controller. In at least one embodiment, a memory interface may be provided via a memory controller 4465 for accessing an SDRAM or SRAM memory device. In at least one embodiment, some integrated circuits also include an embedded security engine 4470.

[0369] Inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, the inference and / or training logic 3515 may be used in integrated circuit 4400 to infer or predict operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0370] Figures 45A-45B Exemplary integrated circuits and associated graphics processors according to various embodiments described herein are illustrated, which may be manufactured using one or more IP cores. In addition to the illustrations, at least one embodiment may include other logic and circuitry, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0371] Figures 45A-45B This is a block diagram illustrating an exemplary graphics processor used within a SoC according to embodiments described herein. Figure 45A An exemplary graphics processor 4510 of a system-on-a-chip according to at least one embodiment is shown, which can be manufactured using one or more IP cores. Figure 45BFurther exemplary graphics processor 4540 of a system-on-a-chip according to at least one embodiment is shown, which can be manufactured using one or more IP cores. In at least one embodiment, Figure 45A The graphics processor 4510 is a low-power graphics processor core. In at least one embodiment, Figure 45B The graphics processor 4540 is a higher-performance graphics processor core. In at least one embodiment, each graphics processor 4510, 4540 may be... Figure 44 A variant of the 4410 graphics processor.

[0372] In at least one embodiment, the graphics processor 4510 includes a vertex processor 4505 and one or more fragment processors 4515A-4515N (e.g., 4515A, 4515B, 4515C, 4515D to 4515N-1 and 4515N). In at least one embodiment, the graphics processor 4510 can execute different shader programs via separate logic, such that the vertex processor 4505 is optimized to perform operations for the vertex shader program, while one or more fragment processors 4515A-4515N perform fragment (e.g., pixel) shading operations for fragments or pixels or shader programs. In at least one embodiment, the vertex processor 4505 performs the vertex processing stage of the 3D graphics pipeline and generates primitive and vertex data. In at least one embodiment, one or more fragment processors 4515A-4515N use the primitive and vertex data generated by the vertex processor 4505 to generate a framebuffer for display on a display device. In at least one embodiment, one or more fragment processors 4515A-4515N are optimized to execute fragment shader programs as provided in the OpenGL API, which can be used to perform operations similar to those of pixel shader programs provided in the Direct 3D API.

[0373] In at least one embodiment, the graphics processor 4510 additionally includes one or more memory management units (MMUs) 4520A-4520B, one or more caches 4525A-4525B, and one or more circuit interconnects 4530A-4530B. In at least one embodiment, one or more MMUs 4520A-4520B provide virtual-to-physical address mappings for the graphics processor 4510, including for the vertex processor 4505 and / or fragment processors 4515A-4515N, which can reference vertex or image / texture data stored in memory, in addition to vertex or image / texture data stored in one or more caches 4525A-4525B. In at least one embodiment, one or more MMUs 4520A-4520B can be synchronized with other MMUs within the system, including with... Figure 44 One or more application processors 4405, image processors 4415, and / or video processors 4420 are associated with one or more MMUs, such that each processor 4405-4420 can participate in a shared or unified virtual memory system. In at least one embodiment, one or more circuit interconnects 4530A-4530B enable the graphics processor 4510 to be connected to other IP cores within the SoC via the SoC's internal bus or via a direct connection.

[0374] In at least one embodiment, the graphics processor 4540 includes one or more shader cores 4555A-4555N (e.g., 4555A, 4555B, 4555C, 4555D, 4555E, 4555F to 4555N-1 and 4555N), such as Figure 45B As shown, it provides a unified shader core architecture, where a single core or type or core can execute all types of programmable shader code, including shader program code for implementing vertex shaders, fragment shaders, and / or compute shaders. In at least one embodiment, the number of shader cores can vary. In at least one embodiment, the graphics processor 4540 includes an inter-core task manager 4545, which acts as a thread dispatcher to assign execution threads to one or more shader cores 4555A-4555N and a tile unit 4558 to accelerate tile-based rendering operations, where scene rendering operations are subdivided in image space, for example, to utilize local spatial consistency within the scene or optimize the use of internal caches.

[0375] Inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, the inference and / or training logic 3515 may be integrated into an integrated circuit. Figure 45A and / or Figure 45B The above is used for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functions or architectures, or neural network use cases described herein.

[0376] Figures 46A-46B Additional exemplary graphics processor logic according to embodiments described herein is illustrated. In at least one embodiment, Figure 46A It shows that it can be included in Figure 44 The graphics core 4600 within the graphics processor 4410, and in at least one embodiment, may be as follows: Figure 45B The unified shader cores shown are 4555A-4555N. Figure 46BA highly parallel general-purpose graphics processing unit (“GPGPU”) 4630 suitable for deployment on a multi-chip module is shown in at least one embodiment.

[0377] In at least one embodiment, the graphics core 4600 includes a shared instruction cache 4602, texture units 4618, and cache / shared memory 4620, which are common to the execution resources within the graphics core 4600. In at least one embodiment, the graphics core 4600 may include multiple slices 4601A-4601N or partitions of each core, and the graphics processor may include multiple instances of the graphics core 4600. In at least one embodiment, slices 4601A-4601N may include supporting logic, including local instruction caches 4604A-4604N, thread schedulers 4606A-4606N, thread dispatchers 4608A-4608N, and a set of registers 4610A-4610N. In at least one embodiment, slices 4601A-4601N may include a set of additional functional units (AFU 4612A-4612N), floating-point units (FPU 4614A-4614N), integer arithmetic logic units (ALU 4616A-4616N), address calculation units (ACU 4613A-4613N), double-precision floating-point units (DPFPU 4615A-4615N), and matrix processing units (MPU 4617A-4617N).

[0378] In at least one embodiment, the FPU 4614A-4614N can perform single-precision (32-bit) and half-precision (16-bit) floating-point operations, while the DPFPU 4615A-4615N performs double-precision (64-bit) floating-point operations. In at least one embodiment, the ALU 4616A-4616N can perform variable-precision integer operations with 8-bit, 16-bit, and 32-bit precision, and can be configured for mixed-precision operations. In at least one embodiment, the MPU 4617A-4617N can also be configured for mixed-precision matrix operations, including half-precision floating-point operations and 8-bit integer operations. In at least one embodiment, the MPU 4617-4617N can perform various matrix operations to accelerate machine learning application frameworks, including enabling support for accelerated generalized matrix-to-matrix multiplication (GEMM). In at least one embodiment, the AFU 4612A-4612N can perform additional logical operations not supported by floating-point or integer units, including trigonometric operations (e.g., sine, cosine, etc.).

[0379] Inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This is combined with... Figure 35A and / or Figure 35BDetails regarding inference and / or training logic 3515 are provided. In at least one embodiment, inference and / or training logic 3515 may be used in graphics core 4600 for inferring or predicting operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0380] Figure 46B A general-purpose processing unit (GPGPU) 4630 is illustrated in at least one embodiment, which can be configured to enable highly parallel computational operations to be performed by a set of graphics processing units. In at least one embodiment, the GPGPU 4630 can be directly linked to other instances of the GPGPU 4630 to create a multi-GPU cluster to improve the training speed for deep neural networks. In at least one embodiment, the GPGPU 4630 includes a host interface 4632 for connection to a host processor. In at least one embodiment, the host interface 4632 is a PCI Express interface. In at least one embodiment, the host interface 4632 may be a vendor-specific communication interface or communication structure. In at least one embodiment, the GPGPU 4630 receives commands from the host processor and uses a global scheduler 4634 to allocate execution threads associated with those commands to a set of compute clusters 4636A-4636H. In at least one embodiment, compute clusters 4636A-4636H share a cache memory 4638. In at least one embodiment, cache memory 4638 can be used as a higher-level cache within the cache memory of computing clusters 4636A-4636H.

[0381] In at least one embodiment, the GPGPU 4630 includes memories 4644A-4644B, which are coupled to the computing cluster 4636A-4636H via a set of memory controllers 4642A-4642B. In at least one embodiment, memories 4644A-4644B may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), which includes graphics double data rate (GDDR) memory.

[0382] In at least one embodiment, each of the computing clusters 4636A-4636H includes a set of graphics cores, for example... Figure 46AThe graphics core 4600 may include various types of integer and floating-point logic units that can perform computational operations across a range of precisions, including precisions suitable for machine learning computations. For example, in at least one embodiment, at least a subset of the floating-point units in each computing cluster 4636A-4636H may be configured to perform 16-bit or 32-bit floating-point operations, while different subsets of the floating-point units may be configured to perform 64-bit floating-point operations.

[0383] In at least one embodiment, multiple instances of the GPGPU 4630 can be configured as a computing cluster. In at least one embodiment, the communication used for synchronization and data exchange by the computing clusters 4636A-4636H varies between embodiments. In at least one embodiment, the multiple instances of the GPGPU 4630 communicate via a host interface 4632. In at least one embodiment, the GPGPU 4630 includes an I / O hub 4639 that couples the GPGPU 4630 to a GPU link 4640, enabling direct connection to other instances of the GPGPU 4630. In at least one embodiment, the GPU link 4640 is coupled to a dedicated GPU-to-GPU bridge, which enables communication and synchronization between the multiple instances of the GPGPU 4630. In at least one embodiment, the GPU link 4640 is coupled to a high-speed interconnect for sending and receiving data to and from other GPGPUs or parallel processors. In at least one embodiment, the multiple instances of the GPGPU 4630 reside in a separate data processing system and communicate via network devices accessible through the host interface 4632. In at least one embodiment, GPU link 4640 may be configured to enable connection to a host processor other than or as a replacement for host interface 4632.

[0384] In at least one embodiment, the GPGPU 4630 can be configured to train a neural network. In at least one embodiment, the GPGPU 4630 can be used within an inference platform. In at least one embodiment, when the GPGPU 4630 is used for inference, the GPGPU 4630 may include fewer compute clusters 4636A-4636H compared to when the GPGPU 4630 is used to train a neural network. In at least one embodiment, the memory technology associated with the memories 4644A-4644B can differ between inference and training configurations, wherein a higher bandwidth memory technology is dedicated to the training configuration. In at least one embodiment, the inference configuration of the GPGPU 4630 can support inference-specific instructions. For example, in at least one embodiment, the inference configuration can provide support for one or more 8-bit integer dot product instructions, which can be used during the inference operation of the deployed neural network.

[0385] Inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding inference and / or training logic 3515 are provided. In at least one embodiment, inference and / or training logic 3515 may be used in the GPGPU 4630 for inferring or predicting operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architecture, or neural network use cases described herein.

[0386] Figure 47 A block diagram of a computer system 4700 according to at least one embodiment is shown. In at least one embodiment, the computer system 4700 includes a processing subsystem 4701 having one or more processors 4702 and a system memory 4704 communicating via an interconnect path that may include a memory hub 4705. In at least one embodiment, the memory hub 4705 may be a separate component within a chipset component or may be integrated within one or more processors 4702. In at least one embodiment, the memory hub 4705 is coupled to an I / O subsystem 4711 via a communication link 4706. In at least one embodiment, the I / O subsystem 4711 includes an I / O hub 4707 that enables the computer system 4700 to receive input from one or more input devices 4708. In at least one embodiment, the I / O hub 4707 enables a display controller to provide output to one or more display devices 4710A, the display controller being included in one or more processors 4702. In at least one embodiment, one or more display devices 4710A coupled to the I / O hub 4707 may include local, internal, or embedded display devices.

[0387] In at least one embodiment, the processing subsystem 4701 includes one or more parallel processors 4712 coupled to the memory hub 4705 via a bus or other communication link 4713. In at least one embodiment, the communication link 4713 may use any of many standards-based communication link technologies or protocols, such as, but not limited to, PCI Express, or may be a vendor-specific communication interface or communication architecture. In at least one embodiment, one or more parallel processors 4712 form a computationally centralized parallel or vector processing system, which may include a large number of processing cores and / or processing clusters, such as multi-core integrated (MIC) processors. In at least one embodiment, one or more parallel processors 4712 form a graphics processing subsystem that can output pixels to one or more display devices 4710A coupled via an I / O hub 4707. In at least one embodiment, the parallel processors 4712 may also include a display controller and a display interface (not shown) to enable direct connection to one or more display devices 4710B.

[0388] In at least one embodiment, system storage unit 4714 may be connected to I / O hub 4707 to provide a storage mechanism for computer system 4700. In at least one embodiment, I / O switch 4716 may be used to provide an interface mechanism to enable connectivity between I / O hub 4707 and other components, such as network adapter 4718 and / or wireless network adapter 4719 which may be integrated into the platform, and various other devices that can be added via one or more additional devices 4720. In at least one embodiment, network adapter 4718 may be an Ethernet adapter or another wired network adapter. In at least one embodiment, wireless network adapter 4719 may include one or more of Wi-Fi, Bluetooth, Near Field Communication (NFC), or other network devices including one or more wireless devices.

[0389] In at least one embodiment, the computing system 4700 may include other components not explicitly shown, such as USB or other port connections, optical storage drives, video capture devices, etc., which may also be connected to the I / O hub 4707. In at least one embodiment, the interconnect can be implemented using any suitable protocol (e.g., a PCI-based protocol (e.g., PCI-Express) or other bus or point-to-point communication interface and / or protocol). Figure 47 The communication paths of the various components, such as NV-Link high-speed interconnect or interconnect protocols.

[0390] In at least one embodiment, one or more parallel processors 4712 include circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constituting a graphics processing unit (GPU). In at least one embodiment, the parallel processors 4712 include circuitry optimized for general-purpose processing. In at least one embodiment, components of the computing system 4700 may be integrated with one or more other system elements on a single integrated circuit. For example, in at least one embodiment, the parallel processor 4712, memory hub 4705, processor 4702, and I / O hub 4707 may be integrated into a system-on-a-chip (SoC) integrated circuit. In at least one embodiment, components of the computing system 4700 may be integrated into a single package to form a system-in-package (SIP) configuration. In at least one embodiment, at least a portion of the components of the computing system 4700 may be integrated into a multi-chip module (MCM) that can interconnect with other MCMs to a modular computer system.

[0391] Inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, the inference and / or training logic 3515 can... Figure 47 The system 4700 is used for reasoning or predicting operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0392] processor

[0393] Figure 48A A parallel processor 4800 according to at least one embodiment is illustrated. In at least one embodiment, various components of the parallel processor 4800 may be implemented using one or more integrated circuit devices, such as programmable processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In at least one embodiment, the illustrated parallel processor 4800 is according to an exemplary embodiment. Figure 47 The variant shown is a parallel processor 4712 with one or more parallel processors.

[0394] In at least one embodiment, the parallel processor 4800 includes a parallel processing unit 4802. In at least one embodiment, the parallel processing unit 4802 includes an I / O unit 4804 that enables communication with other devices, including other instances of the parallel processing unit 4802. In at least one embodiment, the I / O unit 4804 can be directly connected to other devices. In at least one embodiment, the I / O unit 4804 is connected to other devices using a hub or switch interface (e.g., a memory hub 4805). In at least one embodiment, the connection between the memory hub 4805 and the I / O unit 4804 forms a communication link 4813. In at least one embodiment, the I / O unit 4804 is connected to a host interface 4806 and a memory crossbar switch 4816, wherein the host interface 4806 receives commands for performing processing operations, and the memory crossbar switch 4816 receives commands for performing memory operations.

[0395] In at least one embodiment, when host interface 4806 receives a command buffer via I / O unit 4804, host interface 4806 can direct work operations to execute those commands to front end 4808. In at least one embodiment, front end 4808 is coupled to scheduler 4810, which is configured to assign commands or other work items to processing cluster array 4812. In at least one embodiment, scheduler 4810 ensures that processing cluster array 4812 is correctly configured and in an active state before tasks are assigned to clusters of processing cluster array 4812. In at least one embodiment, scheduler 4810 is implemented via firmware logic executed on a microcontroller. In at least one embodiment, the microcontroller-implemented scheduler 4810 can be configured to perform complex scheduling and work assignment operations at both coarse and fine granular levels, thereby enabling fast preemption and context switching of threads executing on processing array 4812. In at least one embodiment, host software can demonstrate workloads for scheduling on processing cluster array 4812 via one of multiple graphics processing paths. In at least one embodiment, the workload can then be automatically distributed on the processing array cluster 4812 by the scheduler 4810 logic within the microcontroller, which includes the scheduler 4810.

[0396] In at least one embodiment, the processing cluster array 4812 may include up to "N" processing clusters (e.g., clusters 4814A, 4814B to 4814N), where "N" represents a positive integer (which may be an integer "N" different from the integer "N" used in other diagrams). In at least one embodiment, each cluster 4814A-4814N of the processing cluster array 4812 can execute a large number of concurrent threads. In at least one embodiment, the scheduler 4810 may use various scheduling and / or work allocation algorithms to allocate work to the clusters 4814A-4814N of the processing cluster array 4812, which may vary depending on the workload generated by each type of program or computation. In at least one embodiment, scheduling may be handled dynamically by the scheduler 4810, or may be partially assisted by compiler logic during the compilation of program logic configured to be executed by the processing cluster array 4812. In at least one embodiment, the different clusters 4814A-4814N of the processing cluster array 4812 may be assigned to process different types of programs or to perform different types of computations.

[0397] In at least one embodiment, the processing cluster array 4812 can be configured to perform various types of parallel processing operations. In at least one embodiment, the processing cluster array 4812 is configured to perform general-purpose parallel computing operations. For example, in at least one embodiment, the processing cluster array 4812 may include logic for performing processing tasks, including filtering video and / or audio data, performing modeling operations, including physical operations, and performing data transformations.

[0398] In at least one embodiment, the processing cluster array 4812 is configured to perform parallel graphics processing operations. In at least one embodiment, the processing cluster array 4812 may include additional logic to support the execution of such graphics processing operations, including but not limited to texture sampling logic for performing texture operations, as well as tessellation logic and other vertex processing logic. In at least one embodiment, the processing cluster array 4812 may be configured to execute shader programs related to graphics processing, such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. In at least one embodiment, the parallel processing unit 4802 may transfer data from system memory via I / O unit 4804 for processing. In at least one embodiment, during processing, the transferred data may be stored in on-chip memory (e.g., parallel processor memory 4822) and then written back to system memory.

[0399] In at least one embodiment, when the parallel processing unit 4802 is used to perform graphics processing, the scheduler 4810 may be configured to divide the processing workload into tasks of approximately equal size to better distribute graphics processing operations among the multiple clusters 4814A-4814N of the processing cluster array 4812. In at least one embodiment, portions of the processing cluster array 4812 may be configured to perform different types of processing. For example, in at least one embodiment, a first portion may be configured to perform vertex shading and topology generation, a second portion may be configured to perform tessellation and geometry shading, and a third portion may be configured to perform pixel shading or other screen-space operations to produce a rendered image for display. In at least one embodiment, intermediate data generated by one or more of the clusters 4814A-4814N may be stored in a buffer to allow intermediate data to be transferred between the clusters 4814A-4814N for further processing.

[0400] In at least one embodiment, the processing cluster array 4812 may receive processing tasks to be executed via a scheduler 4810, which receives commands defining the processing tasks from a front end 4808. In at least one embodiment, the processing task may include an index of data to be processed, such as surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and commands defining how the data is processed (e.g., what program to execute). In at least one embodiment, the scheduler 4810 may be configured to acquire an index corresponding to a task, or may receive an index from the front end 4808. In at least one embodiment, the front end 4808 may be configured to ensure that the processing cluster array 4812 is configured to be active before initiating the workload specified by an incoming command buffer (e.g., a batch buffer, push buffer, etc.).

[0401] In at least one embodiment, each of one or more instances of the parallel processing unit 4802 may be coupled to the parallel processor memory 4822. In at least one embodiment, the parallel processor memory 4822 may be accessed via a memory crossbar switch 4816, which may receive memory requests from the processing cluster array 4812 and the I / O unit 4804. In at least one embodiment, the memory crossbar switch 4816 may be accessed via a memory interface 4818. In at least one embodiment, the memory interface 4818 may include a plurality of partition units (e.g., partition units 4820A, 4820B to 4820N), each of which may be coupled to a portion (e.g., a memory cell) of the parallel processor memory 4822. In at least one embodiment, the plurality of partition units 4820A-4820N are configured to be equal to the number of memory units, such that the first partition unit 4820A has a corresponding first memory unit 4824A, the second partition unit 4820B has a corresponding memory unit 4824B, and the Nth partition unit 4820N has a corresponding Nth memory unit 4824N. In at least one embodiment, the number of partition units 4820A-4820N may not be equal to the number of memory units.

[0402] In at least one embodiment, memory cells 4824A-4824N may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In at least one embodiment, memory cells 4824A-4824N may also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). In at least one embodiment, rendering targets such as frame buffers or texture maps may be stored across memory cells 4824A-4824N, allowing partitioning cells 4820A-4820N to write portions of each rendering target in parallel, to efficiently utilize the available bandwidth of the parallel processor memory 4822. In at least one embodiment, local instances of the parallel processor memory 4822 may be excluded to facilitate a unified memory design that combines system memory with local cache memory.

[0403] In at least one embodiment, any of the clusters 4814A-4814N of the processing cluster array 4812 can process data to be written to any memory cell 4824A-4824N within the parallel processor memory 4822. In at least one embodiment, the memory crossbar switch 4816 can be configured to transfer the output of each cluster 4814A-4814N to any partition cell 4820A-4820N or another cluster 4814A-4814N, and the clusters 4814A-4814N can perform further processing operations on the output. In at least one embodiment, each cluster 4814A-4814N can communicate with the memory interface 4818 via the memory crossbar switch 4816 to read from or write to various external memory devices. In at least one embodiment, the memory crossbar switch 4816 has a connection to a memory interface 4818 for communication with I / O unit 4804, and a connection to a local instance of parallel processor memory 4822, thereby enabling processing units within different processing clusters 4814A-4814N to communicate with system memory or other memory not local to parallel processing unit 4802. In at least one embodiment, the memory crossbar switch 4816 may use virtual channels to separate traffic flows between clusters 4814A-4814N and partition units 4820A-4820N.

[0404] In at least one embodiment, multiple instances of the parallel processing unit 4802 may be provided on a single add-in card, or multiple add-in cards may be interconnected. In at least one embodiment, different instances of the parallel processing unit 4802 may be configured to interoperate, even if the different instances have different numbers of processing cores, different numbers of local parallel processor memories, and / or other configuration differences. For example, in at least one embodiment, some instances of the parallel processing unit 4802 may include higher-precision floating-point units relative to other instances. In at least one embodiment, a system combining one or more instances of the parallel processing unit 4802 or the parallel processor 4800 may be implemented in various configurations and form factors, including but not limited to desktop, laptop, or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0405] Figure 48B This is a block diagram of a partitioning unit 4820 according to at least one embodiment. In at least one embodiment, the partitioning unit 4820 is... Figure 48AThis is an example of one of the partitioning units 4820A-4820N. In at least one embodiment, the partitioning unit 4820 includes an L2 cache 4821, a frame buffer interface 4825, and a ROP 4826 (raster operation unit). In at least one embodiment, the L2 cache 4821 is a read / write cache configured to perform load and store operations received from memory crossbar switches 4816 and ROP 4826. In at least one embodiment, the L2 cache 4821 outputs read misses and urgent write-back requests to the frame buffer interface 4825 for processing. In at least one embodiment, updates can also be sent to the frame buffer for processing via the frame buffer interface 4825. In at least one embodiment, the frame buffer interface 4825 communicates with memory cells in the parallel processor memory (such as...). Figure 48A It interacts with one of the memory cells 4824A-4824N (e.g., within the parallel processor memory 4822).

[0406] In at least one embodiment, ROP 4826 is a processing unit that performs raster operations such as stenciling, z-testing, blending, etc. In at least one embodiment, ROP 4826 then outputs processed graphics data stored in graphics memory. In at least one embodiment, ROP 4826 includes compression logic to compress depth or color data written to memory and decompress depth or color data read from memory. In at least one embodiment, the compression logic may be lossless compression logic utilizing one or more of a variety of compression algorithms. In at least one embodiment, the type of compression performed by ROP 4826 may vary based on the statistical characteristics of the data to be compressed. For example, in at least one embodiment, incremental color compression is performed based on depth and color data on a per-tile basis.

[0407] In at least one embodiment, ROP 4826 is included within each processing cluster (e.g., Figure 48A Clusters 4814A-4814N are used instead of partition units 4820. In at least one embodiment, read and write requests for pixel data are made via memory crossbar switch 4816 instead of pixel fragment data transfer. In at least one embodiment, the processed graphics data can be displayed on a display device (such as...). Figure 47 Displayed by one or more display devices 4710, routed by processor 4702 for further processing, or by... Figure 48A One of the processing entities within the parallel processor 4800 is routed for further processing.

[0408] Figure 48C This is a block diagram of a processing cluster 4814 within a parallel processing unit according to at least one embodiment. In at least one embodiment, the processing cluster is... Figure 48A An instance of one of the processing clusters 4814A-4814N. In at least one embodiment, the processing cluster 4814 can be configured to execute a number of threads in parallel, where a "thread" refers to an instance of a specific program executing on a particular set of input data. In at least one embodiment, a Single Instruction Multiple Data (SIMD) instruction issuing technique is used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In at least one embodiment, a Single Instruction Multiple Threading (SIMT) technique is used to support the parallel execution of a large number of generally synchronous threads, which uses a common instruction unit configured to issue instructions to a set of processing engines within each processing cluster.

[0409] In at least one embodiment, the operation of the processing cluster 4814 can be controlled by a pipeline manager 4832 that assigns processing tasks to the SIMT parallel processors. In at least one embodiment, the pipeline manager 4832... Figure 48A The scheduler 4810 receives instructions and manages the execution of these instructions via the graphics multiprocessor 4834 and / or texture unit 4836. In at least one embodiment, the graphics multiprocessor 4834 is an exemplary instance of a SIMT parallel processor. However, in at least one embodiment, the processing cluster 4814 may include various types of SIMT parallel processors with different architectures. In at least one embodiment, the processing cluster 4814 may include one or more instances of the graphics multiprocessor 4834. In at least one embodiment, the graphics multiprocessor 4834 can process data, and the data cross switch 4840 can be used to distribute the processed data to one of several possible destinations (including other shader units). In at least one embodiment, the pipeline manager 4832 can facilitate the distribution of processed data by specifying the destination of the processed data to be distributed via the data cross switch 4840.

[0410] In at least one embodiment, each graphics multiprocessor 4834 within the processing cluster 4814 may include the same set of functional execution logic (e.g., arithmetic logic units, load-memory units, etc.). In at least one embodiment, the functional execution logic may be configured in a pipelined manner, wherein new instructions may be issued before previous instructions complete. In at least one embodiment, the functional execution logic supports a variety of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, shift operations, and computation of various algebraic functions. In at least one embodiment, the same functional unit hardware may be used to perform different operations, and any combination of functional units may exist.

[0411] In at least one embodiment, instructions sent to the processing cluster 4814 constitute threads. In at least one embodiment, a group of threads executing across a set of parallel processing engines is a thread group. In at least one embodiment, the thread group executes a general program on different input data. In at least one embodiment, each thread within the thread group can be assigned to a different processing engine within the graphics multiprocessor 4834. In at least one embodiment, the thread group may include fewer threads than the number of processing engines within the graphics multiprocessor 4834. In at least one embodiment, when the number of threads included in the thread group is less than the number of processing engines, one or more processing engines may be idle during a loop that is processing the thread group. In at least one embodiment, the thread group may also include more threads than the number of processing engines within the graphics multiprocessor 4834. In at least one embodiment, when the thread group includes more threads than the number of processing engines within the graphics multiprocessor 4834, processing can be performed in consecutive clock cycles. In at least one embodiment, multiple thread groups can be executed simultaneously on the graphics multiprocessor 4834.

[0412] In at least one embodiment, the graphics multiprocessor 4834 includes an internal cache memory for performing load and store operations. In at least one embodiment, the graphics multiprocessor 4834 may forgo the internal cache and use a cache memory within the processing cluster 4814 (e.g., L1 cache 4848). In at least one embodiment, each graphics multiprocessor 4834 may also access partition units (e.g., Figure 48A The L2 cache is located within partition units 4820A-4820N, which are shared among all processing clusters 4814 and can be used to transfer data between threads. In at least one embodiment, the graphics multiprocessor 4834 can also access off-chip global memory, which may include one or more of local parallel processor memory and / or system memory. In at least one embodiment, any memory outside of the parallel processing unit 4802 can be used as global memory. In at least one embodiment, the processing cluster 4814 includes multiple instances of the graphics multiprocessor 4834, which can share common instructions and data that can be stored in the L1 cache 4848.

[0413] In at least one embodiment, each processing cluster 4814 may include a memory management unit (“MMU”) 4845 configured to map virtual addresses to physical addresses. In at least one embodiment, one or more instances of the MMU 4845 may reside in Figure 48AThe memory interface 4818 is located within the MMU 4845. In at least one embodiment, the MMU 4845 includes a set of page table entries (PTEs) for mapping virtual addresses to physical addresses of tiles and optionally to cache line indices. In at least one embodiment, the MMU 4845 may include an address translation back buffer (TLB) or a cache that may reside within the graphics multiprocessor 4834, the L1 cache 4848, or the processing cluster 4814. In at least one embodiment, physical addresses are processed to allocate surface data access locally for efficient request interleaving between partition units. In at least one embodiment, cache line indices may be used to determine whether a request for a cache line is a hit or a miss.

[0414] In at least one embodiment, the processing cluster 4814 can be configured such that each graphics multiprocessor 4834 is coupled to a texture unit 4836 to perform texture mapping operations, determine texture sample locations, read texture data, and filter texture data. In at least one embodiment, texture data is read as needed from an internal texture L1 cache (not shown) or from an L1 cache within the graphics multiprocessor 4834, and texture data is retrieved from an L2 cache, local parallel processor memory, or system memory. In at least one embodiment, each graphics multiprocessor 4834 outputs a processed task to a data crossbar switch 4840 to provide the processed task to another processing cluster 4814 for further processing or to store the processed task in an L2 cache, local parallel processor memory, or system memory via a memory crossbar switch 4816. In at least one embodiment, a preROP 4842 (pre-raster operation unit) is configured to receive data from the graphics multiprocessor 4834 and direct the data to a ROP unit, which can be associated with a partitioning unit (e.g., [missing information]). Figure 48A The PreROP 4842 unit is positioned together with the partitioning units 4820A-4820N. In at least one embodiment, the PreROP 4842 unit can perform optimizations for color blending, organize pixel color data, and perform address translation.

[0415] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding inference and / or training logic 3515 are provided. In at least one embodiment, inference and / or training logic 3515 may be used in a graphics processing cluster 4814 to perform inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0416] Figure 48DA graphics multiprocessor 4834 according to at least one embodiment is illustrated. In at least one embodiment, the graphics multiprocessor 4834 is coupled to a pipeline manager 4832 of a processing cluster 4814. In at least one embodiment, the graphics multiprocessor 4834 has an execution pipeline including, but not limited to, an instruction cache 4852, an instruction unit 4854, an address mapping unit 4856, a register file 4858, one or more general-purpose graphics processing unit (GPGPU) cores 4862, and one or more load / store units 4866. In at least one embodiment, the GPGPU cores 4862 and the load / store units 4866 are coupled to a cache memory 4872 and a shared memory 4870 via a memory and cache interconnect 4868.

[0417] In at least one embodiment, instruction cache 4852 receives a stream of instructions to be executed from pipeline manager 4832. In at least one embodiment, instructions are cached in instruction cache 4852 and dispatched to instruction unit 4854 for execution. In one embodiment, instruction unit 4854 may dispatch instructions as thread groups (e.g., thread bundles), assigning each thread of the thread group to a different execution unit within GPGPU core 4862. In at least one embodiment, instructions can access any local, shared, or global address space by specifying an address within a unified address space. In at least one embodiment, address mapping unit 4856 may be used to translate addresses in the unified address space into different memory addresses that can be accessed by load / store unit 4866.

[0418] In at least one embodiment, register file 4858 provides a set of registers for functional units of graphics multiprocessor 4834. In at least one embodiment, register file 4858 provides temporary storage for operands of data paths connected to functional units of graphics multiprocessor 4834 (e.g., GPGPU core 4862, load / store unit 4866). In at least one embodiment, register file 4858 is partitioned among each functional unit, such that a dedicated portion of register file 4858 is allocated to each functional unit. In at least one embodiment, register file 4858 is partitioned among different thread bundles being executed by graphics multiprocessor 4834.

[0419] In at least one embodiment, each of the GPGPU cores 4862 may include a floating-point unit (FPU) and / or an integer arithmetic logic unit (ALU) for executing instructions of the graphics multiprocessor 4834. In at least one embodiment, the GPGPU cores 4862 may be architecturally similar or may differ in architecture. In at least one embodiment, a first portion of the GPGPU core 4862 includes a single-precision FPU and an integer ALU, while a second portion of the GPGPU core includes a double-precision FPU. In at least one embodiment, the FPU may implement the IEEE 754-2008 standard for floating-point algorithms or enable variable-precision floating-point algorithms. In at least one embodiment, the graphics multiprocessor 4834 may additionally include one or more fixed-function or special-function units to perform specific functions, such as copying rectangles or pixel blending operations. In at least one embodiment, one or more of the GPGPU cores 4862 may also include fixed-function or special-function logic.

[0420] In at least one embodiment, the GPGPU core 4862 includes SIMD logic capable of executing a single instruction on multiple sets of data. In at least one embodiment, the GPGPU core 4862 can physically execute SIMD4, SIMD8, and SIMD16 instructions, and logically execute SIMD1, SIMD2, and SIMD32 instructions. In at least one embodiment, the SIMD instructions for the GPGPU core can be generated by a shader compiler at compile time, or automatically generated when executing a program written and compiled for a Single Program Multiple Data (SPMD) or SIMT architecture. In at least one embodiment, multiple threads of a program configured for a SIMT execution model can be executed using a single SIMD instruction. For example, in at least one embodiment, eight SIMD threads performing the same or similar operations can be executed in parallel using a single SIMD8 logic unit.

[0421] In at least one embodiment, the memory and cache interconnect 4868 is an interconnect network connecting each functional unit of the graphics multiprocessor 4834 to the register file 4858 and the shared memory 4870. In at least one embodiment, the memory and cache interconnect 4868 is a cross-switch interconnect that allows the load / store unit 4866 to perform load and store operations between the shared memory 4870 and the register file 4858. In at least one embodiment, the register file 4858 can operate at the same frequency as the GPGPU core 4862, resulting in very low latency for data transfer between the GPGPU core 4862 and the register file 4858. In at least one embodiment, the shared memory 4870 can be used to enable communication between threads executing on functional units within the graphics multiprocessor 4834. In at least one embodiment, the cache memory 4872 can be used, for example, as a data cache to cache texture data communicated between functional units and texture units 4836. In at least one embodiment, the shared memory 4870 can also be used as a program-managed cache. In at least one embodiment, in addition to the automatically cached data stored in cache memory 4872, the thread executing on GPGPU core 4862 can also programmatically store data in shared memory.

[0422] In at least one embodiment, a parallel processor or GPGPU, as described herein, is communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. In at least one embodiment, the GPU may be communicatively coupled to the host processor / core via a bus or other interconnect (e.g., high-speed interconnects such as PCIe or NVLink). In at least one embodiment, the GPU may be integrated as a core on a package or chip and communicatively coupled to the core via an internal processor bus / interconnect within the package or chip. In at least one embodiment, regardless of how the GPU is connected, the processor core may assign work to the GPU in the form of a sequence of commands / instructions contained in a job descriptor. In at least one embodiment, the GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

[0423] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. The following is in conjunction with... Figure 35A and / or Figure 35BDetails regarding inference and / or training logic 3515 are provided. In at least one embodiment, inference and / or training logic 3515 may be used in a graphics multiprocessor 4834 to perform inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0424] Figure 49 A multi-GPU computing system 4900 according to at least one embodiment is illustrated. In at least one embodiment, the multi-GPU computing system 4900 may include a processor 4902 coupled to a plurality of general-purpose graphics processing units (GPGPUs) 4906A-D via a host interface switch 4904. In at least one embodiment, the host interface switch 4904 is a PCI Express switch device that couples the processor 4902 to a PCI Express bus, through which the processor 4902 can communicate with the GPGPUs 4906A-D. In at least one embodiment, the GPGPUs 4906A-D may be interconnected via a set of high-speed point-to-point GPU-to-GPU links 4916. In at least one embodiment, the GPU-to-GPU links 4916 are connected to each of the GPGPUs 4906A-D via dedicated GPU links. In at least one embodiment, the P2P GPU links 4916 enable direct communication between each GPGPU 4906A-D without communication via the host interface switch 4904 to which the processor 4902 is connected. In at least one embodiment, when GPU-to-GPU traffic is directed to the P2P GPU link 4916, the host interface switch 4904 remains available for system memory access or, for example, communication with other instances of the multi-GPU computing system 4900 via one or more network devices. While in at least one embodiment, the GPGPUs 4906A-D are connected to the processor 4902 via the host interface switch 4904, in at least one embodiment, the processor 4902 includes direct support for the P2P GPU link 4916 and can be directly connected to the GPGPUs 4906A-D. In at least one embodiment, the multi-GPU computing system 4900 executes process 900 (see...). Figure 9A ) or process 906 (see Figure 9B ).

[0425] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35BDetails regarding inference and / or training logic 3515 are provided. In at least one embodiment, inference and / or training logic 3515 may be used in a multi-GPU computing system 4900 for performing inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0426] Figure 50 This is a block diagram of a graphics processor 5000 according to at least one embodiment. In at least one embodiment, the graphics processor 5000 includes a ring interconnect 5002, a pipeline front end 5004, a media engine 5037, and graphics cores 5080A-5080N. In at least one embodiment, the ring interconnect 5002 couples the graphics processor 5000 to other processing units, said processing units including other graphics processors or one or more general-purpose processor cores. In at least one embodiment, the graphics processor 5000 is one of many processors integrated within a multi-core processing system.

[0427] In at least one embodiment, the graphics processor 5000 receives multiple batches of commands via a ring interconnect 5002. In at least one embodiment, the incoming commands are interpreted by a command streamer 5003 in a pipeline front-end 5004. In at least one embodiment, the graphics processor 5000 includes scalable execution logic for performing 3D geometry processing and media processing via graphics cores 5080A-5080N. In at least one embodiment, for 3D geometry processing commands, the command streamer 5003 provides the commands to the geometry pipeline 5036. In at least one embodiment, for at least some media processing commands, the command streamer 5003 provides the commands to a video front-end 5034, which is coupled to a media engine 5037. In at least one embodiment, the media engine 5037 includes a video quality engine (VQE) 5030 for video and image post-processing, and a multi-format encoding / decoding (MFX) engine 5033 for providing hardware-accelerated media data encoding and decoding. In at least one embodiment, the geometry pipeline 5036 and the media engine 5037 each generate an execution thread for thread execution resources provided by at least one graphics core 5080.

[0428] In at least one embodiment, the graphics processor 5000 includes scalable thread execution resources featuring graphics cores 5080A-5080N (which may be modular and sometimes referred to as core slices), each graphics core having multiple sub-cores 5050A-5050N, 5060A-5060N (sometimes referred to as core sub-slices). In at least one embodiment, the graphics processor 5000 may have any number of graphics cores 5080A. In at least one embodiment, the graphics processor 5000 includes graphics cores 5080A having at least a first sub-core 5050A and a second sub-core 5060A. In at least one embodiment, the graphics processor 5000 is a low-power processor having a single sub-core (e.g., 5050A). In at least one embodiment, the graphics processor 5000 includes multiple graphics cores 5080A-5080N, each graphics core including a set of first sub-cores 5050A-5050N and a set of second sub-cores 5060A-5060N. In at least one embodiment, each of the first sub-cores 5050A-5050N includes at least a first set of execution units 5052A-5052N and media / texture samplers 5054A-5054N. In at least one embodiment, each of the second sub-cores 5060A-5060N includes at least a second set of execution units 5062A-5062N and samplers 5064A-5064N. In at least one embodiment, each of the sub-cores 5050A-5050N and 5060A-5060N shares a set of shared resources 5070A-5070N. In at least one embodiment, the shared resources include a shared cache memory and pixel operation logic.

[0429] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding inference and / or training logic 3515 are provided. In at least one embodiment, inference and / or training logic 3515 may be used in a graphics processor 5000 to perform inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0430] Figure 51This is a block diagram illustrating a microarchitecture for a processor 5100 according to at least one embodiment, the processor 5100 including logic circuitry for executing instructions. In at least one embodiment, the processor 5100 can execute instructions, including x86 instructions, ARM instructions, and special-purpose instructions for application-specific integrated circuits (ASICs). In at least one embodiment, the processor 5100 may include registers for storing packaged data, such as the 64-bit wide MMX registers used in Intel Corporation's Santa Clara, California-enabled MMX technology microprocessors. TM Registers. In at least one embodiment, MMX registers available in integer and floating-point forms can operate alongside packaged data elements accompanied by Single Instruction Multiple Data (“SIMD”) and Streaming SIMD Extensions (“SSE”) instructions. In at least one embodiment, a 128-bit wide XMM register associated with SSE2, SSE3, SSE4, AVX, or later (generally referred to as “SSEx”) technologies can hold such packaged data operands. In at least one embodiment, processor 5100 can execute instructions to accelerate machine learning or deep learning algorithms, training, or inference.

[0431] In at least one embodiment, processor 5100 includes an ordered front end (“front end”) 5101 to fetch instructions to be executed and prepare instructions for later use in the processor pipeline. In at least one embodiment, front end 5101 may include several units. In at least one embodiment, instruction prefetcher 5126 fetches instructions from memory and feeds the instructions to instruction decoder 5128, which in turn decodes or interprets the instructions. For example, in at least one embodiment, instruction decoder 5128 decodes the received instructions into one or more machine-executable so-called “micro-instructions” or “micro-operations” (also referred to as “micro-ops” or “uops”). In at least one embodiment, instruction decoder 5128 parses the instructions into opcodes and corresponding data and control fields, which can be used by the microarchitecture to perform operations according to at least one embodiment. In at least one embodiment, trace cache 5130 may assemble the decoded uops into a program-ordered sequence or trace in uops queue 5134 for execution. In at least one embodiment, when the trace cache 5130 encounters complex instructions, the microcode ROM 5132 provides the uops required to complete the operation.

[0432] In at least one embodiment, some instructions may be converted into a single micro-operation, while others require several micro-operations to complete the entire operation. In at least one embodiment, if more than four micro-operations are required to complete an instruction, the instruction decoder 5128 may access the microcode ROM 5132 to execute the instruction. In at least one embodiment, an instruction may be decoded into a small number of micro-operations for processing at the instruction decoder 5128. In at least one embodiment, if multiple micro-operations are required to complete the operation, the instruction may be stored in the microcode ROM 5132. In at least one embodiment, the trace cache 5130 references an entry point programmable logic array (“PLA”) to determine the correct micro-instruction pointer for reading a microcode sequence from the microcode ROM 5132 to complete one or more instructions, according to at least one embodiment. In at least one embodiment, after the microcode ROM 5132 has completed the micro-operation ordering of the instructions, the machine front end 5101 may resume fetching micro-operations from the trace cache 5130.

[0433] In at least one embodiment, the out-of-order execution engine (“out-of-order engine”) 5103 can prepare instructions for execution. In at least one embodiment, the out-of-order execution logic has multiple buffers to smooth and reorder the instruction stream to optimize performance as instructions descend the pipeline and are scheduled for execution. In at least one embodiment, the out-of-order execution engine 5103 includes, but is not limited to, an allocator / register renamer 5140, a memory micro-operation queue 5142, an integer / floating-point micro-operation queue 5144, a memory scheduler 5146, a fast scheduler 5102, a slow / general-purpose floating-point scheduler (“slow / general-purpose FP scheduler”) 5104, and a simple floating-point scheduler (“simple FP scheduler”) 5106. In at least one embodiment, the fast scheduler 5102, the slow / general-purpose floating-point scheduler 5104, and the simple floating-point scheduler 5106 are also collectively referred to as “micro-operation schedulers 5102, 5104, 5106”. In at least one embodiment, the allocator / register renamer 5140 allocates the machine buffers and resources required for each microinstruction to execute. In at least one embodiment, the allocator / register renamer 5140 renames logical registers to entries in a register file. In at least one embodiment, the allocator / register renamer 5140 also allocates entries for each micro-operation in one of two micro-operation queues, a memory micro-instruction queue 5142 for memory operations and an integer / floating-point micro-operation queue 5144 for non-memory operations, preceding the memory scheduler 5146 and micro-operation schedulers 5102, 5104, and 5106. In at least one embodiment, the micro-operation schedulers 5102, 5104, and 5106 determine when they are ready to execute a micro-operation based on the readiness of their dependent input register operand sources and the availability of the execution resource microinstructions required to complete their operations. In at least one embodiment, the fast scheduler 5102 may schedule on each half of the master clock cycle, while the slow / general-purpose floating-point scheduler 5104 and the simple floating-point scheduler 5106 may schedule once per master processor clock cycle. In at least one embodiment, micro-operation schedulers 5102, 5104, and 5106 arbitrate dispatch ports to schedule micro-operations for execution.

[0434] In at least one embodiment, execution block 5111 includes, but is not limited to, integer register file / tribute network 5108, floating-point register file / tribute network (“FP register file / tribute network”) 5110, address generation units (“AGU”) 5112 and 5114, fast arithmetic logic units (“fast ALU”) 5116 and 5118, slow arithmetic logic unit (“slow ALU”) 5120, floating-point ALU (“FP”) 5122, and floating-point move unit (“FP move”) 5124. In at least one embodiment, integer register file / tribute network 5108 and floating-point register file / bypass network 5110 are also referred to herein as “register files 5108, 5110”. In at least one embodiment, AGUs 5112 and 5114, fast ALUs 5116 and 5118, slow ALU 5120, floating-point ALU 5122, and floating-point movement unit 5124 are also referred to herein as "execution units 5112, 5114, 5116, 5118, 5120, 5122, and 5124". In at least one embodiment, execution block 5111 may include, but is not limited to, any number (including zero) and type of register files, branch networks, address generation units, and execution units (in any combination).

[0435] In at least one embodiment, register networks 5108, 5110 may be arranged between micro-operation schedulers 5102, 5104, 5106 and execution units 5112, 5114, 5116, 5118, 5120, 5122, and 5124. In at least one embodiment, integer register file / tribute network 5108 performs integer operations. In at least one embodiment, floating-point register file / tribute network 5110 performs floating-point operations. In at least one embodiment, each of register networks 5108, 5110 may include, but is not limited to, a tribute network that can bypass or forward recently completed results not yet written to a register file to a new dependent object. In at least one embodiment, register networks 5108, 5110 may communicate data with each other. In at least one embodiment, integer register file / tribute network 5108 may include, but is not limited to, two separate register files, one register file for low-order 32-bit data and a second register file for high-order 32-bit data. In at least one embodiment, the floating-point register file / branch network 5110 may include, but is not limited to, entries with a width of 128 bits, since floating-point instructions typically have operands with a width of 64 to 128 bits.

[0436] In at least one embodiment, execution units 5112, 5114, 5116, 5118, 5120, 5122, and 5124 can execute instructions. In at least one embodiment, register networks 5108 and 5110 store integer and floating-point data operation values ​​that the microinstructions need to execute. In at least one embodiment, processor 5100 can be, but is not limited to, any number of execution units 5112, 5114, 5116, 5118, 5120, 5122, and 5124, and combinations thereof. In at least one embodiment, floating-point ALU 5122 and floating-point move unit 5124 can perform floating-point, MMX, SIMD, AVX, and SSE or other operations, including specialized machine learning instructions. In at least one embodiment, floating-point ALU 5122 can be, but is not limited to, a 64-bit multiplication-64-bit floating-point divider to perform division, square root, and remainder micro-operations. In at least one embodiment, floating-point hardware can be used to process instructions involving floating-point values. In at least one embodiment, ALU operations can be passed to fast ALUs 5116 and 5118. In at least one embodiment, fast ALUs 5116 and 5118 can perform fast operations with an effective delay of half a clock cycle. In at least one embodiment, most complex integer operations are routed to slow ALU 5120, because slow ALU 5120 can include, but is not limited to, integer execution hardware for long-latency type operations, such as multipliers, shifters, flag logic, and branching. In at least one embodiment, memory load / store operations can be performed by ALUs 5112 and 5114. In at least one embodiment, fast ALU 5116, fast ALU 5118, and slow ALU 5120 can perform integer operations on 64-bit data operands. In at least one embodiment, fast ALU 5116, fast ALU 5118, and slow ALU 5120 can be implemented to support various data bit sizes, including sixteen, thirty-two, 128, 256, etc. In at least one embodiment, the floating-point ALU 5122 and the floating-point moving unit 5124 can be implemented to support a range of operands with various bit widths, for example, they can be combined with SIMD and multimedia instructions to operate on 128-bit wide packaged data operands.

[0437] In at least one embodiment, microinstruction schedulers 5102, 5104, and 5106 schedule dependent operations before the parent load completes execution. In at least one embodiment, since microinstructions can be speculatively scheduled and executed within processor 5100, processor 5100 may also include logic for handling memory misses. In at least one embodiment, if a data load miss occurs in the data cache, there may be a dependent operation running in the pipeline that temporarily deprives the scheduler of the correct data. In at least one embodiment, a replay mechanism tracks and re-executes instructions that use incorrect data. In at least one embodiment, it may be necessary to replay dependent operations and may allow independent operations to be completed. In at least one embodiment, the scheduler and replay mechanism of at least one embodiment of the processor may also be designed to capture instruction sequences used for text string comparison operations.

[0438] In at least one embodiment, "register" can refer to an onboard processor storage location that can be used as part of an instruction that identifies an operand. In at least one embodiment, a register can be one that can be used externally to the processor (from a programmer's perspective). In at least one embodiment, a register may not be limited to a particular type of circuit. Rather, in at least one embodiment, a register can store data, provide data, and perform the functions described herein. In at least one embodiment, the registers described herein can be implemented using a variety of different techniques via circuitry within the processor, such as dedicated physical registers, dynamically allocated physical registers renamed using register renaming, a combination of dedicated and dynamically allocated physical registers, etc. In at least one embodiment, an integer register stores 32-bit integer data. The register file of at least one embodiment also includes eight multimedia SIMD registers for encapsulating data.

[0439] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, part or all of the inference and / or training logic 3515 may be incorporated into execution block 5111 and other memories or registers shown or not shown. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs shown in execution block 5111. Furthermore, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALUs of execution block 5111 to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0440] Figure 52A deep learning application processor 5200 according to at least one embodiment is illustrated. In at least one embodiment, the deep learning application processor 5200 uses instructions, which, if executed by the deep learning application processor 5200, cause the deep learning application processor 5200 to perform some or all of the processes and techniques described herein. In at least one embodiment, the deep learning application processor 5200 is an application-specific integrated circuit (ASIC). In at least one embodiment, the application processor 5200 performs matrix multiplication operations or is "hardwired" into hardware as a result of executing one or more instructions or both. In at least one embodiment, the deep learning application processor 5200 includes, but is not limited to, a processing cluster 5210(1)-5210(12), an inter-chip link (“ICL”) 5220(1)-5220(12), an inter-chip controller (“ICC”) 5230(1)-5230(2), a second-generation high-bandwidth memory (“HBM2”) 5240(1)-5240(4), a memory controller (“Mem Ctrlr”) 5242(1)-5242(4), a high-bandwidth memory physical layer (“HBM PHY”) 5244(1)-5244(4), a management controller central processing unit (“management controller CPU”) 5250, a serial peripheral interface, internal integrated circuits and general-purpose input / output blocks (“SPI, I2C, GPIO”) 5260, a peripheral component interconnect fast controller and direct memory access block (“PCIe controller and DMA”) 5270, and a sixteen-channel peripheral component interconnect fast port (“PCI Express”). x 16”)5280.

[0441] In at least one embodiment, the processing cluster 5210 can perform deep learning operations, including inference or prediction operations based on weight parameters computed using one or more training techniques, including those described herein. In at least one embodiment, each processing cluster 5210 can include, but is not limited to, any number and type of processors. In at least one embodiment, the deep learning application processor 5200 can include any number and type of processing cluster 5210. In at least one embodiment, the inter-chip link 5220 is bidirectional. In at least one embodiment, the inter-chip link 5220 and the inter-chip controller 5230 enable multiple deep learning application processors 5200 to exchange information, including activation information generated from executing one or more machine learning algorithms embodied in one or more neural networks. In at least one embodiment, the deep learning application processor 5200 can include any number (including zero) and type of ICL 5220 and ICC 5230.

[0442] In at least one embodiment, the HBM2 5240 provides a total of 32GB of memory. In at least one embodiment, the HBM2 5240(i) is associated with both the memory controller 5242(i) and the HBM PHY 5244(i), where “i” is any integer. In at least one embodiment, any number of HBM2 5240s can provide any type and total amount of high-bandwidth memory and can be associated with any number (including zero) and type of memory controller 5242 and HBM PHY 5244. In at least one embodiment, any number and type of SPI, I2C, GPIO 5260, PCIe controller, and DMA 5270 and / or PCIe 5280 can be replaced with any number and type of blocks to implement any number and type of communication standards in any technically feasible manner.

[0443] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, the deep learning application processor is used to train a machine learning model (e.g., a neural network) to predict or infer information provided to the deep learning application processor 5200. In at least one embodiment, the deep learning application processor 5200 is used to infer or predict information based on a trained machine learning model (e.g., a neural network) that has been trained by another processor or system or by the deep learning application processor 5200. In at least one embodiment, the processor 5200 may be used to perform one or more neural network use cases described herein.

[0444] Figure 53This is a block diagram of a neuromorphic processor 5300 according to at least one embodiment. In at least one embodiment, the neuromorphic processor 5300 may receive one or more inputs from a source external to the neuromorphic processor 5300. In at least one embodiment, these inputs may be transmitted to one or more neurons 5302 within the neuromorphic processor 5300. In at least one embodiment, the neurons 5302 and their components may be implemented using circuitry or logic including one or more arithmetic logic units (ALUs). In at least one embodiment, the neuromorphic processor 5300 may include, but is not limited to, thousands upon thousands of instances of neurons 5302, but any suitable number of neurons 5302 may be used. In at least one embodiment, each instance of a neuron 5302 may include a neuron input 5304 and a neuron output 5306. In at least one embodiment, a neuron 5302 may generate an output that can be transmitted to the inputs of other instances of the neuron 5302. In at least one embodiment, the neuron input 5304 and the neuron output 5306 may be interconnected via synapses 5308.

[0445] In at least one embodiment, neuron 5302 and synapse 5308 may be interconnected, causing neuromorphic processor 5300 to operate to process or analyze information received by neuromorphic processor 5300. In at least one embodiment, neuron 5302 may send an output pulse (or “trigger” or “peak”) when the input received through neuron input 5304 exceeds a threshold. In at least one embodiment, neuron 5302 may sum or integrate the signal received at neuron input 5304. For example, in at least one embodiment, neuron 5302 may be implemented as a leaky integral-triggered neuron, wherein if the summation (referred to as “membrane potential”) exceeds a threshold, neuron 5302 may use a transfer function such as a sigmoid or threshold function to generate an output (or “trigger”). In at least one embodiment, the leaky integral-triggered neuron may sum the signal received at neuron input 5304 to a membrane potential and may apply an attenuation factor (or leak) to reduce the membrane potential. In at least one embodiment, a leaking integral-triggered neuron may trigger if multiple input signals are received at neuron input 5304 quickly enough to exceed a threshold (i.e., before the membrane potential decays too low to trigger). In at least one embodiment, neuron 5302 may be implemented using circuitry or logic that receives input, integrates the input to the membrane potential, and decays the membrane potential. In at least one embodiment, the input may be averaged, or any other suitable transfer function may be used. Furthermore, in at least one embodiment, neuron 5302 may include, but is not limited to, comparator circuitry or logic that generates an output spike at neuron output 5306 when the result of applying the transfer function to neuron input 5304 exceeds a threshold. In at least one embodiment, once neuron 5302 is triggered, it can ignore previously received input information by, for example, resetting the membrane potential to 0 or another suitable default value. In at least one embodiment, once the membrane potential is reset to 0, neuron 5302 may resume normal operation after a suitable period of time (or recovery period).

[0446] In at least one embodiment, neurons 5302 can be interconnected via synapses 5308. In at least one embodiment, synapses 5308 can be operated to transmit signals from the output of a first neuron 5302 to the input of a second neuron 5302. In at least one embodiment, neurons 5302 can transmit information on more than one instance of synapses 5308. In at least one embodiment, one or more instances of neuron outputs 5306 can be connected via instances of synapses 5308 to instances of neuron inputs 5304 in the same neuron 5302. In at least one embodiment, an instance of neuron 5302 that produces an output to be transmitted on the instance of synapse 5308 may be referred to as a "presynaptic neuron". In at least one embodiment, an instance of neuron 5302 that receives input transmitted via an instance of synapse 5308 may be referred to as a "postsynaptic neuron". In at least one embodiment, regarding various instances of synapse 5308, since an instance of neuron 5302 can receive input from one or more instances of synapse 5308 and can also transmit output through one or more instances of synapse 5308, a single instance of neuron 5302 can be both a "presynaptic neuron" and a "postsynaptic neuron".

[0447] In at least one embodiment, neurons 5302 may be organized into one or more layers. In at least one embodiment, each instance of neuron 5302 may have a neuron output 5306, which may fan out to one or more neuron inputs 5304 via one or more synapses 5308. In at least one embodiment, the neuron output 5306 of neuron 5302 in the first layer 5310 may be connected to the neuron input 5304 of neuron 5302 in the second layer 5312. In at least one embodiment, layer 5310 may be referred to as a "feedforward layer". In at least one embodiment, each instance of neuron 5302 in an instance of the first layer 5310 may fan out to each instance of neuron 5302 in the second layer 5312. In at least one embodiment, the first layer 5310 may be referred to as a "fully connected feedforward layer". In at least one embodiment, each instance of neuron 5302 in an instance of the second layer 5312 fan out to fewer than all instances of neuron 5302 in the third layer 5314. In at least one embodiment, the second layer 5312 may be referred to as a "sparsely connected feedforward layer". In at least one embodiment, neurons 5302 in the second layer 5312 may fan out to neurons 5302 in multiple other layers, including neurons 5302 fan out to the second layer 5312. In at least one embodiment, the second layer 5312 may be referred to as a "recurrent layer". In at least one embodiment, the neuromorphic processor 5300 may be any suitable combination of recurrent layers and feedforward layers, including but not limited to sparsely connected feedforward layers and fully connected feedforward layers.

[0448] In at least one embodiment, the neuromorphic processor 5300 may include, but is not limited to, a reconfigurable interconnect architecture or dedicated hardwired interconnects to connect synapses 5308 to neurons 5302. In at least one embodiment, the neuromorphic processor 5300 may include, but is not limited to, circuitry or logic that allows synapses to be assigned to different neurons 5302 as needed, depending on the neural network topology and neuron fan-in / fan-out. For example, in at least one embodiment, synapses 5308 may be connected to neurons 5302 using interconnect structures (such as on-chip networks) or via dedicated connections. In at least one embodiment, synaptic interconnects and their components may be implemented using circuitry or logic.

[0449] Figure 54A processing system according to at least one embodiment is illustrated. In at least one embodiment, system 5400 includes one or more processors 5402 and one or more graphics processors 5408, and may be a single-processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 5402 or processor cores 5407. In at least one embodiment, system 5400 is a processing platform incorporated within a system-on-a-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices. In at least one embodiment, system 4700 executes process 900 (see...). Figure 9A ) and / or process 906 (see Figure 9B ).

[0450] In at least one embodiment, system 5400 may include or be integrated into a server-based gaming platform, including a game console, mobile game console, handheld game console, or online game console, which are game and media consoles. In at least one embodiment, system 5400 is a mobile phone, smartphone, tablet computing device, or mobile internet device. In at least one embodiment, processing system 5400 may also include components coupled to or integrated into a wearable device, such as a smartwatch, smart glasses, augmented reality, or virtual reality device. In at least one embodiment, processing system 5400 is a television or set-top box device having one or more processors 5402 and a graphical interface generated by one or more graphics processors 5408.

[0451] In at least one embodiment, each of the one or more processors 5402 includes one or more processor cores 5407 for processing instructions that, when executed, perform operations against the system and user software. In at least one embodiment, each of the one or more processor cores 5407 is configured to process a specific instruction sequence 5409. In at least one embodiment, the instruction sequence 5409 may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation via Very Long Instruction Word (VLIW). In at least one embodiment, each processor core 5407 may process a different instruction sequence 5409, which may include instructions that facilitate the emulation of other instruction sequences. In at least one embodiment, the processor core 5407 may also include other processing devices, such as a digital signal processor (DSP).

[0452] In at least one embodiment, processor 5402 includes cache memory 5404. In at least one embodiment, processor 5402 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory is shared among various components of processor 5402. In at least one embodiment, processor 5402 also uses an external cache (e.g., a Level 3 (L3) cache or a last-level cache (LLC)) (not shown), which can be shared among processor cores 5407 using known cache coherence techniques. In at least one embodiment, processor 5402 further includes a register file 5406, which may include different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers). In at least one embodiment, register file 5406 may include general-purpose registers or other registers.

[0453] In at least one embodiment, one or more processors 5402 are coupled to one or more interface buses 5410 to transmit communication signals, such as address, data, or control signals, between the processors 5402 and other components in the system 5400. In at least one embodiment, the interface bus 5410 may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. In at least one embodiment, the interface bus 5410 is not limited to the DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, the processor 5402 includes an integrated memory controller 5416 and a platform controller hub 5430. In at least one embodiment, the memory controller 5416 facilitates communication between memory devices and other components of the processing system 5400, while the platform controller hub (PCH) 5430 provides connectivity to input / output (I / O) devices via a local I / O bus.

[0454] In at least one embodiment, memory device 5420 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or a device with suitable performance for use as processor memory. In at least one embodiment, memory device 5420 may be used as system memory of processing system 5400 to store data 5422 and instructions 5421 for use when one or more processors 5402 execute an application or process. In at least one embodiment, memory controller 5416 is also coupled to an optional external graphics processor 5412, which may communicate with one or more graphics processors 5408 of processor 5402 to perform graphics and media operations. In at least one embodiment, display device 5411 may be connected to processor 5402. In at least one embodiment, display device 5411 may include one or more internal display devices, such as in mobile electronic devices or laptop devices, or external display devices connected via a display interface (e.g., DisplayPort). In at least one embodiment, the display device 5411 may include a head-mounted display (HMD), such as a stereoscopic display device for virtual reality (VR) or augmented reality (AR) applications.

[0455] In at least one embodiment, the platform controller hub 5430 enables peripheral devices to connect to the storage device 5420 and the processor 5402 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, but are not limited to, an audio controller 5446, a network controller 5434, a firmware interface 5428, a wireless transceiver 5426, a touch sensor 5425, and a data storage device 5424 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 5424 may be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCIe). In at least one embodiment, the touch sensor 5425 may include a touchscreen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 5426 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or LTE transceiver. In at least one embodiment, the firmware interface 5428 enables communication with the system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, network controller 5434 may enable network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to interface bus 5410. In at least one embodiment, audio controller 5446 is a multi-channel high-definition audio controller. In at least one embodiment, processing system 5400 includes an optional legacy I / O controller 5440 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to system 5400. In at least one embodiment, platform controller hub 5430 may also be connected to one or more Universal Serial Bus (USB) controllers 5442 that connect input devices, such as a keyboard and mouse combination 5443, a camera 5444, or other USB input devices.

[0456] In at least one embodiment, instances of the memory controller 5416 and platform controller hub 5430 may be integrated into a discrete external graphics processor, such as external graphics processor 5412. In at least one embodiment, the platform controller hub 5430 and / or memory controller 5416 may be external to one or more processors 5402. For example, in at least one embodiment, system 5400 may include external memory controller 5416 and platform controller hub 5430, which may be configured as a memory controller hub and peripheral controller hub in a system chipset communicating with processor 5402.

[0457] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35BDetails regarding the inference and / or training logic 3515 are provided. In at least one embodiment, some or all of the inference and / or training logic 3515 may be incorporated into the graphics processor 5408. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs embodied in a 3D pipeline. Furthermore, in at least one embodiment, the inference and / or training operations described herein may use, in addition to Figure 35A or Figure 35B The logic is performed using logic other than that shown. In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown), which configure the ALU of the graphics processor 5408 to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0458] Figure 55 This is a block diagram of a processor 5500 having one or more processor cores 5502A-5502N, an integrated memory controller 5514, and an integrated graphics processor 5508 according to at least one embodiment. In at least one embodiment, the processor 5500 may include additional cores, up to and including additional cores 5502N indicated by dashed boxes. In at least one embodiment, each processor core 5502A-5502N includes one or more internal cache units 5504A-5504N. In at least one embodiment, each processor core may also access one or more shared cache units 5506. In at least one embodiment, the processor 5500 executes process 900 (see...). Figure 9A ) or process 906 (see Figure 9B ).

[0459] In at least one embodiment, internal cache units 5504A-5504N and shared cache unit 5506 represent a cache memory hierarchy within processor 5500. In at least one embodiment, cache memory units 5504A-5504N may include at least one level of instruction and data cache within each processor core and one or more levels of cache in a shared intermediate cache, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, wherein the highest level of cache preceding external memory is classified as LLC. In at least one embodiment, cache coherence logic maintains coherence between the various cache units 5506 and 5504A-5504N.

[0460] In at least one embodiment, the processor 5500 may further include a set of one or more bus controller units 5516 and a system agent core 5510. In at least one embodiment, the one or more bus controller units 5516 manage a set of peripheral buses, such as one or more PCI or PCIe buses. In at least one embodiment, the system agent core 5510 provides management functions for various processor components. In at least one embodiment, the system agent core 5510 includes one or more integrated memory controllers 5514 to manage access to various external memory devices (not shown).

[0461] In at least one embodiment, one or more processor cores 5502A-5502N include support for multi-threaded concurrent processing. In at least one embodiment, system agent core 5510 includes components for coordinating and operating cores 5502A-5502N during multi-threaded processing. In at least one embodiment, system agent core 5510 may additionally include a power control unit (PCU) including logic and components for regulating one or more power states of processor cores 5502A-5502N and graphics processor 5508.

[0462] In at least one embodiment, processor 5500 further includes a graphics processor 5508 for performing graph processing operations. In at least one embodiment, graphics processor 5508 is coupled to a shared cache unit 5506 and a system proxy core 5510 including one or more integrated memory controllers 5514. In at least one embodiment, system proxy core 5510 further includes a display controller 5511 for driving graphics processor outputs to one or more coupled displays. In at least one embodiment, display controller 5511 may also be a separate module coupled to graphics processor 5508 via at least one interconnect, or it may be integrated within graphics processor 5508.

[0463] In at least one embodiment, ring-based interconnect unit 5512 is used to couple internal components of processor 5500. In at least one embodiment, alternative interconnect units, such as point-to-point interconnects, switched interconnects, or other technologies, may be used. In at least one embodiment, graphics processor 5508 is coupled to ring interconnect 5512 via I / O link 5513.

[0464] In at least one embodiment, I / O link 5513 represents at least one of a variety of I / O interconnects, including packaged I / O interconnects that facilitate communication between various processor components and a high-performance embedded memory module 5518 (e.g., an eDRAM module). In at least one embodiment, each of the processor cores 5502A-5502N and the graphics processor 5508 uses the embedded memory module 5518 as a shared last-level cache.

[0465] In at least one embodiment, processor cores 5502A-5502N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, processor cores 5502A-5502N are heterogeneous in terms of instruction set architecture (ISA), with one or more processor cores 5502A-5502N executing a common instruction set, while one or more other processor cores 5502A-5502N execute a subset of the common instruction set or a different instruction set. In at least one embodiment, processor cores 5502A-5502N are heterogeneous in terms of microarchitecture, with one or more cores having relatively high power consumption coupled to one or more power cores having lower power consumption. In at least one embodiment, processor 5500 can be implemented on one or more chips or implemented as a SoC integrated circuit.

[0466] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, some or all of the inference and / or training logic 3515 may be incorporated into the processor 5500. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs embodied in... Figure 55 The 3D pipeline, graphics core 5502, shared functional logic, or other logic are included. Furthermore, in at least one embodiment, the inference and / or training operations described herein can use, except... Figure 35A or Figure 35B The logic is performed using logic other than that shown. In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown), which configure the ALU of processor 5500 to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0467] Figure 56This is a block diagram of a graphics processor 5600, which may be a discrete graphics processing unit or a graphics processor integrated with multiple processing cores. In at least one embodiment, the graphics processor 5600 communicates with registers on the graphics processor 5600 and commands placed in memory via a memory-mapped I / O interface. In at least one embodiment, the graphics processor 5600 includes a memory interface 5614 for accessing memory. In at least one embodiment, the memory interface 5614 is an interface to local memory, one or more internal caches, one or more shared external caches, and / or to system memory.

[0468] In at least one embodiment, the graphics processor 5600 further includes a display controller 5602 for driving display output data to the display device 5620. In at least one embodiment, the display controller 5602 includes a combination of hardware for one or more overlay planes of the display device 5620 and multi-layer video or user interface elements. In at least one embodiment, the display device 5620 may be an internal or external display device. In at least one embodiment, the display device 5620 is a head-mounted display device, such as a virtual reality (VR) display device or an augmented reality (AR) display device. In at least one embodiment, the graphics processor 5600 includes a video codec engine 5606 for encoding, decoding, or transcoding media into, from, or between one or more media encoding formats, including but not limited to Moving Picture Experts Group (MPEG) formats (e.g., MPEG-2), Advanced Video Coding (AVC) formats (e.g., H.264 / MPEG-4 AVC, and SMPTE 421M / VC-1), and Joint Picture Experts Group (JPEG) formats (e.g., JPEG) and MotionJPEG (MJPEG). In at least one embodiment, the graphics processor 5600 performs process 900 (see...). Figure 9A ) or process 906 (see Figure 9B ).

[0469] In at least one embodiment, the graphics processor 5600 includes a block image transfer (BLIT) engine 5604 to perform two-dimensional (2D) rasterizer operations, including, for example, bit boundary block transfer. However, in at least one embodiment, one or more components of a graphics processing engine (GPE) 5610 are used to perform 2D graphics operations. In at least one embodiment, the GPE 5610 is a computational engine for performing graphics operations, including three-dimensional (3D) graphics operations and media operations.

[0470] In at least one embodiment, GPE 5610 includes a 3D pipeline 5612 for performing 3D operations, such as rendering 3D images and scenes using processing functions that manipulate 3D primitive shapes (e.g., rectangles, triangles, etc.). In at least one embodiment, 3D pipeline 5612 includes programmable and fixed function elements that perform various tasks and / or generate execution threads to 3D / media subsystem 5615. While 3D pipeline 5612 can be used to perform media operations, in at least one embodiment, GPE 5610 also includes a media pipeline 5616 for performing media operations such as video post-processing and image enhancement.

[0471] In at least one embodiment, the media pipeline 5616 includes fixed-function or programmable logic units for performing one or more specialized media operations, such as video decoding acceleration, video deinterlacing, and video encoding acceleration, replacing or representing the video codec engine 5606. In at least one embodiment, the media pipeline 5616 also includes a thread generation unit for generating threads to execute on the 3D / media subsystem 5615. In at least one embodiment, the generated threads perform computations of media operations on one or more graphics execution units included in the 3D / media subsystem 5615.

[0472] In at least one embodiment, the 3D / media subsystem 5615 includes logic for executing threads generated by the 3D pipeline 5612 and the media pipeline 5616. In at least one embodiment, the 3D pipeline 5612 and the media pipeline 5616 send thread execution requests to the 3D / media subsystem 5615, which includes thread dispatch logic for arbitrating various requests and dispatching them to available thread execution resources. In at least one embodiment, the execution resources include an array of graphics execution units for processing 3D and media threads. In at least one embodiment, the 3D / media subsystem 5615 includes one or more internal caches for thread instructions and data. In at least one embodiment, the subsystem 5615 also includes shared memory, including registers and addressable memory, for sharing data among threads and storing output data.

[0473] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, some or all of the inference and / or training logic 3515 may be incorporated into the processor 5600. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs included in the 3D pipeline 5612. Furthermore, in at least one embodiment, the inference and / or training operations described herein may use, except for... Figure 35A or Figure 35B The logic other than that shown is used to perform this task. In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown), which configure the ALU of the graphics processor 5600 to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0474] Figure 57 This is a block diagram of a graphics processing engine 5710 of a graphics processor according to at least one embodiment. In at least one embodiment, the graphics processing engine (GPE) 5710 is... Figure 56 The version of GPE 5610 shown. In at least one embodiment, the media pipeline 5716 is optional and may not be explicitly included in the GPE 5710. In at least one embodiment, a separate media and / or image processor is coupled to the GPE 5710. In at least one embodiment, the graphics processing engine 5710 performs process 900 (see...). Figure 9A ) or process 906 (see Figure 9B ).

[0475] In at least one embodiment, GPE 5710 is coupled to or includes command stream converter 5703, which provides command streams to 3D pipeline 5712 and / or media pipeline 5716. In at least one embodiment, command stream converter 5703 is coupled to memory, which may be system memory, or one or more of internal cache memory and shared cache memory. In at least one embodiment, command stream converter 5703 receives commands from memory and sends the commands to 3D pipeline 5712 and / or media pipeline 5716. In at least one embodiment, the commands are instructions, primitives, or micro-operations retrieved from a circular buffer that stores commands for 3D pipeline 5712 and media pipeline 5716. In at least one embodiment, the circular buffer may further include a batch command buffer storing multiple commands in batches. In at least one embodiment, commands for 3D pipeline 5712 may further include references to data stored in memory, such as, but not limited to, vertex and geometry data for 3D pipeline 5712 and / or image data and memory objects for media pipeline 5716. In at least one embodiment, the 3D pipeline 5712 and the media pipeline 5716 process commands and data by performing operations or by dispatching one or more execution threads to the graphics core array 5714. In at least one embodiment, the graphics core array 5714 includes one or more graphics core blocks (e.g., one or more graphics cores 5715A, one or more graphics cores 5715B), each block including one or more graphics cores. In at least one embodiment, each graphics core includes a set of graphics execution resources, which include general-purpose and graphics-specific execution logic for performing graphics and computational operations, and fixed-function texture processing and / or machine learning and artificial intelligence acceleration logic, including... Figure 35A and Figure 35B The reasoning and / or training logic in 3515.

[0476] In at least one embodiment, the 3D pipeline 5712 includes fixed functions and programmable logic for processing one or more shader programs, such as vertex shaders, geometry shaders, pixel shaders, fragment shaders, compute shaders, or other shader programs, by processing instructions and dispatching execution threads to the graphics core array 5714. In at least one embodiment, the graphics core array 5714 provides a unified execution resource block for processing shader programs. In at least one embodiment, the multipurpose execution logic (e.g., execution units) within the graphics cores 5715A-5715B of the graphics core array 5714 includes support for various 3D API shader languages ​​and can execute multiple concurrently running threads associated with multiple shaders.

[0477] In at least one embodiment, the graphics core array 5714 further includes execution logic for performing media functions, such as video and / or image processing. In at least one embodiment, in addition to graphics processing operations, the execution unit also includes general-purpose logic programmable to perform parallel general-purpose computing operations.

[0478] In at least one embodiment, output data can be output to memory in a unified return buffer (URB) 5718, the output data being generated by a thread executing on the graphics core array 5714. In at least one embodiment, the URB 5718 can store data from multiple threads. In at least one embodiment, the URB 5718 can be used to send data between different threads executing on the graphics core array 5714. In at least one embodiment, the URB 5718 can also be used for synchronization between threads on the graphics core array 5714 and fixed-function logic within shared-function logic 5720.

[0479] In at least one embodiment, the graphics core array 5714 is scalable, such that it includes a variable number of graphics cores, each having a variable number of execution units based on the target power and performance level of the GPE 5710. In at least one embodiment, the execution resources are dynamically scalable, such that they can be enabled or disabled as needed.

[0480] In at least one embodiment, the graphics core array 5714 is coupled to shared function logic 5720, which includes multiple resources shared among the graphics cores in the graphics core array 5714. In at least one embodiment, the shared functions performed by the shared function logic 5720 are embodied in hardware logic units that provide dedicated supplementary functions to the graphics core array 5714. In at least one embodiment, the shared function logic 5720 includes, but is not limited to, a sampler unit 5721, a math unit 5722, and inter-thread communication (ITC) logic 5723. In at least one embodiment, one or more caches 5725 are included in or coupled to the shared function logic 5720.

[0481] In at least one embodiment, shared functionality is used if the demand for dedicated functionality is insufficient to be contained within the graphics core array 5714. In at least one embodiment, a single instance of the dedicated functionality is used in shared functionality logic 5720 and shared among other execution resources within the graphics core array 5714. In at least one embodiment, a specific shared functionality may be included within shared functionality logic 5726 within the graphics core array 5714, said specific shared functionality being widely used within shared functionality logic 5720 of the graphics core array 5714. In at least one embodiment, shared functionality logic 5726 within the graphics core array 5714 may include some or all of the logic within shared functionality logic 5720. In at least one embodiment, all logic elements within shared functionality logic 5720 may be replicated within shared functionality logic 5726 of the graphics core array 5714. In at least one embodiment, shared functionality logic 5720 is excluded to support shared functionality logic 5726 within the graphics core array 5714.

[0482] The inference and / or training logic 3515 is used to perform inference and / or training operations associated with one or more embodiments. This document combines... Figure 35A and / or Figure 35B Details regarding the inference and / or training logic 3515 are provided. In at least one embodiment, some or all of the inference and / or training logic 3515 may be incorporated into the graphics pro...

Claims

1. A computer-implemented method for generating textures, the method comprising: Receives textures consisting of two or more pixels; The texture energy value of the texture is calculated based on the energy function of the Blue Noise Mixing and Sampling (BNDS) algorithm. The energy function is based on the distance between pixel pairs in the two or more pixels and one or more configurable parameters. The texture energy value is based on the sum of the pixel energy values ​​of the pixels in the texture. The pixel energy value of each pixel in the pixel pair includes a non-zero value, which is the result of at least one of the following: the pixel pair is in the same multidimensional layer, or the pixel pair has the same coordinates at different time slices; Based on the energy function, it is determined whether to perform pixel swapping on pixels from pixel pairs in the texture. When it is determined to perform pixel swapping, the values ​​of the pixels in the pixel pair are swapped until the texture energy value reaches the minimum energy value. as well as An output texture, including output pixels, is generated based on the texture with the minimum energy value applied to the image data.

2. The computer-implemented method according to claim 1, wherein the received texture is a white noise texture.

3. The computer-implemented method of claim 1, wherein the energy function is further based on one or more sample values ​​associated with the pixel pair.

4. The computer-implemented method of claim 1, wherein at least one of the two or more pixels corresponds to at least one of the following: one or more vectors, one or more points on a grid, or one or more objects having more than three dimensions, and wherein the output texture is applied to the image data comprising vectors, points on a grid, or objects having more than three dimensions.

5. The computer-implemented method of claim 1, wherein before calculating the texture energy value of the texture, the method further comprises: Sampling is performed on one or more pixels corresponding to the image data, wherein each pixel corresponds to a random variable, wherein sampling of the one or more pixels corresponds to determining the value of a probability density function for the pixel, and wherein the sampling includes reducing the variance of the samples by skewing the samples toward regions of high energy based on the energy function.

6. The computer-implemented method of claim 1, wherein the one or more configurable parameters include a first configurable parameter and a second configurable parameter, the first configurable parameter including a Gaussian value, and the second configurable parameter including a Gaussian value different from the first configurable parameter.

7. The computer-implemented method according to claim 1, further comprising: Before calculating the texture energy value of the texture, one or more pixels corresponding to the image data are sampled, wherein each pixel corresponds to a random variable, and sampling the one or more pixels includes at least one of the following: determining the value of the probability density function of the pixel, or performing non-uniform random sampling.

8. The computer-implemented method according to claim 1, further comprising: Low-difference sequences are applied to the output texture to add an additional dimension to each output pixel.

9. The computer-implemented method according to claim 1, wherein, Generating the output texture includes: Input one or more scalar output pixel values ​​into the space fill curve function to output one or more pixel values ​​as a vector.

10. The computer-implemented method according to claim 1, wherein: Calculating the texture energy value of the texture further includes: - The received textures are layered over time to determine the pixel pairs in the same multidimensional layer; - Calculate the layer score based on the number of pixel values ​​in each layer of the pixel pair; and Determining whether to perform the pixel swap includes: - If the pixel swap increases the layer score, then the pixel swap is rejected.

11. The computer-implemented method according to claim 1, further comprising: After generating the output texture, a low-pass filter is applied to the image data.

12. The computer-implemented method of claim 1, wherein the pixel swapping further comprises: Select two or more random pixel pairs to swap the values ​​of the pixels in the two or more random pixel pairs.

13. A processor, comprising: One or more processing units are configured to perform multiple operations, said multiple operations including: Receives textures consisting of two or more pixels; The texture energy value of the texture is calculated based on the energy function of the Blue Noise Mixing and Sampling (BNDS) algorithm. The energy function is based on the distance between pixel pairs in the two or more pixels and one or more configurable parameters. The texture energy value is based on the sum of the pixel energy values ​​of the pixels in the texture. The pixel energy value of each pixel in the pixel pair includes a non-zero value, which is the result of at least one of the following: the pixel pair is in the same multidimensional layer, or the pixel pair has the same coordinates at different time slices; Based on the energy function, it is determined whether to perform pixel swapping on pixels from pixel pairs in the texture. When it is determined to perform pixel swapping, the values ​​of the pixels in the pixel pair are swapped until the texture energy value reaches the minimum energy value. Based on the texture generation with the minimum energy value to be applied to the image data, an output texture including the output pixels is generated; and The output image is rendered across multiple frames by applying the output texture to one or more images.

14. The processor of claim 13, wherein one or more pixels corresponding to the image data are sampled, wherein each pixel corresponds to a random variable, wherein sampling of one or more pixels corresponds to determining the value of a probability density function of the pixel, and wherein the sampling includes: The variance of the sample is reduced by skewing the sample toward a high-energy region based on the energy function.

15. The processor of claim 13, wherein the received texture is a white noise texture.

16. The processor of claim 13, wherein at least one of the two or more pixels corresponds to at least one of: one or more vectors, one or more points on a grid, or one or more objects having more than three dimensions, and wherein the output texture is applied to image data comprising vectors, points on a grid, or objects having more than three dimensions.

17. The processor of claim 13, wherein the operation further comprises: Sampling is performed on one or more pixels corresponding to the image data, wherein each pixel corresponds to a random variable, wherein sampling of the one or more pixels corresponds to determining the value of a probability density function for the pixel, and wherein the sampling includes reducing the variance of the samples by skewing cosine-weighted samples toward high-energy regions based on the energy function.

18. The processor of claim 13, wherein the operation further comprises: Apply a rank-1 matrix to add an additional dimension to the output texture.

19. The processor of claim 13, wherein generating the output texture comprises: Input one or more scalar output pixel values ​​into the space fill curve function to output one or more pixel values ​​as a vector.

20. A computer-readable storage medium having one or more instructions stored thereon, said instructions, when executed by one or more processors, causing one or more processors to perform the following operations: Receives textures consisting of two or more pixels; The texture energy value of the texture is calculated based on the energy function of the Blue Noise Mixing and Sampling (BNDS) algorithm. The energy function is based on the distance between pixel pairs in the two or more pixels and one or more configurable parameters. The texture energy value is based on the sum of the pixel energy values ​​of the pixels in the texture. The pixel energy value of each pixel in the pixel pair includes a non-zero value, which is the result of at least one of the following: the pixel pair is in the same multidimensional layer, or the pixel pair has the same coordinates at different time slices; Based on the energy function, it is determined whether to perform pixel swapping on pixels within pixel pairs in the texture. When it is determined to perform pixel swapping, the values ​​of pixels within pixel pairs are swapped until the texture reaches the minimum energy value. as well as An output texture, including output pixels, is generated based on the texture with the minimum energy value applied to the image data.

21. The computer-readable storage medium of claim 20, wherein: Calculating the texture energy value of the texture includes: layering the received texture over time to determine the pixel pairs within the same multidimensional layer; and Determining whether to perform the pixel swap includes rejecting the pixel swap if, based on multiple pixel values ​​found in the layers of the received texture, the pixel swap would result in an increase in the texture energy value.

22. The computer-readable storage medium of claim 20, wherein the received texture includes a time dimension.

23. The computer-readable storage medium of claim 20, further comprising: Sampling is performed on one or more pixels corresponding to the image data, wherein each pixel corresponds to a random variable, wherein sampling of the one or more pixels corresponds to determining the value of a probability density function for the pixel, and wherein the sampling includes reducing the variance of the samples by skewing the samples toward regions of high energy based on the energy function.

24. The computer-readable storage medium of claim 20, wherein the operation further comprises: If one pixel in a pixel pair is not in the same multidimensional layer or does not have the same coordinates, then the energy value of that pixel is set to zero.

25. The computer-readable storage medium of claim 20, wherein the one or more configurable parameters include a first configurable parameter and a second configurable parameter, the first configurable parameter including an energy attenuation parameter corresponding to how energy dissipates from a pixel, and the second configurable parameter corresponding to a normal distribution of energy.

26. The computer-readable storage medium of claim 20, wherein, Generating the output texture includes: Input one or more scalar output pixel values ​​into the space fill curve function to output one or more pixel values ​​as a vector.

27. The computer-readable storage medium of claim 20, wherein the operation further comprises: After generating the output texture, a low-pass filter is applied to the image data.

28. The computer-readable storage medium of claim 20, wherein the number of pixel swaps corresponds to the width, height, and depth of the received texture.

29. The computer-readable storage medium of claim 20, wherein the energy function has a non-zero value because the pixel pairs have the same z-coordinate.

Images