Universal scaled metadata layout for matrix multiplication and addition (MMA)

Abstract

The invention relates to a generic scaled metadata layout for matrix multiplication and addition (MMA). The present disclosure describes efficiently performing matrix multiplication and addition (MMA) operations using narrow operands. A narrow operand size (e.g., 8-bit / 6-bit / 4-bit operand) MMA operation utilizes scaled metadata to improve the accuracy of the MMA operation. Efficient layouts and uses thereof for scaling metadata in narrow operand size (MMA) operations are described. The proposed layout provides efficient storage and efficient use of scaled metadata.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to the following application:

[0003] U.S. patent application No. 18 / 449,381, titled “Method and Apparatus for Direct Convolution Calculation,” filed August 14, 2023;

[0004] U.S. Patent Application No. 18 / 458,638, entitled “Method and Apparatus for Weight-Stationary Direct Convolution Calculation,” filed August 14, 2023; and

[0005] U.S. patent application No. 18 / 484,790, titled “Low Precision Floating-Point Datapath in a Computer Processor,” was filed on October 11, 2023.

[0006] The entire contents of each of the above applications are incorporated herein by reference. Technical Field

[0007] The technology generally relates to parallel processing systems and, more particularly, to the organization and movement of data in matrix multiplication and addition (MMA) circuits within such processing systems. Background Art

[0008] Matrix multiplication and addition (MMA) is frequently performed in many demanding applications. For example, deep learning and machine learning applications for tasks such as image / video processing and natural language processing, scientific computing applications for tasks such as 3D simulation and signal processing, and graphics and video editing applications for tasks such as ray tracing, video editing, and processing, may make heavy use of the efficient MMA capabilities of parallel processing systems such as graphics processing units (GPUs).

[0009] As processing power increases and application demands grow, the energy consumption of computing platforms is also increasing. For example, the large number of processors deployed in modern graphics processing units (GPUs) for high-demand applications has led to the need to reduce the energy requirements of such processors.

[0010] The design and operation of the datapath at the core of a processor accounts for a significant portion of the total energy consumption of that processor and can be a particularly significant contributor in GPUs executing applications such as deep learning workloads that use a large number of datapaths in parallel. One mechanism for improving datapath energy efficiency is to utilize lower precision and alternative data formats (e.g., integer, INT8, floating point, logarithmic (Log), VS-Quant). This lower precision and alternative data formats can allow for narrower operand sizes, thereby reducing energy consumption.

[0011] Due to the lower precision of narrow operand sizes, using narrow operands on the data path may require making additional data (e.g., scaling information) available for computation. For large matrices subjected to MMA operations, this additional data can itself be large, and its movement through the GPU's memory hierarchy can be inefficient. Therefore, to more fully benefit from the energy savings of narrow operand sizes, improvements are needed to organize and move the additional data required to process narrow operands. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An MMA operation between an A matrix and a B matrix using a scaling factor according to some embodiments of the present disclosure is conceptually illustrated.

[0013] Figure 2A An example of a scaling metadata block data structure according to some embodiments of the present disclosure is schematically shown.

[0014] Figure 2B Schematically illustrates examples of scaling metadata block arrangements in different levels of a memory hierarchy in a computer system according to some embodiments of the present disclosure.

[0015] Figure 2C Schematically illustrates an example area on the A matrix or B matrix mapped by a scaling metadata block, a depth vector mapped to the A matrix or B matrix, and the scaling metadata vector according to some embodiments of the present disclosure.

[0016] Figure 3 An example of a logical organization of scaled metadata blocks corresponding to operand matrices of different sizes is shown according to some embodiments of the present disclosure.

[0017] Figure 4A shows an example mapping of scaling factors in a scaling metadata block mapped to an A matrix according to some embodiments of the present disclosure, Figure 4B shows the mapping to the B matrix, and Figure 4C The mapping to the A matrix or the B matrix is ​​shown.

[0018] Figure 5A flow chart illustrating a process for computing the product of an A matrix and a B matrix based on their operands and scaling metadata associated with the operands is shown.

[0019] Figure 6 An example microarchitecture of a data path of an example MMA circuit according to some embodiments of the present disclosure is shown.

[0020] Figure 7A and Figure 7B A portion of a processor (datapath processor) including a datapath configured to implement MMA operations is shown in accordance with some embodiments.

[0021] Figure 8 An example parallel processing unit (PPU) is shown in accordance with some embodiments.

[0022] Figure 9A According to some embodiments, Figure 8 The parallel processing units within the general processing cluster (GPC)

[0023] Figure 9B Shown Figure 8 An example memory partitioning unit of a parallel processing unit.

[0024] Figure 10A FIG. 1 shows a schematic diagram of a circuit with an MMA according to some embodiments. Figure 9A An example of a streaming multiprocessor (SM).

[0025] Figure 10B Conceptually illustrates a method for performing a multi-processor (M / M) process in accordance with some embodiments. Figure 10A The four sub-partitions implemented in the SM shown.

[0026] Figure 11A According to one embodiment, using Figure 8 An example conceptual diagram of a processing system implementing a PPU.

[0027] Figure 11B Block diagrams illustrating example systems in which the various architecture and / or functionality of the various previous embodiments may be implemented. DETAILED DESCRIPTION

[0028] As described above, using narrow operand sizes in the data path can reduce energy consumption, but may require providing additional data to the processor for calculations such as, for example, matrix multiplication of two matrices. The additional data may include scaling data to scale the narrow operands. Embodiments of the present disclosure provide a data layout for scaling metadata (e.g., a scaling factor that can be applied to an operand matrix) that the data path in the processor can efficiently use for matrix multiplication and addition (MMA) operations. When the scaling metadata is moved between multiple levels of a memory hierarchy in the processor, the layout can significantly reduce or eliminate additional data operations on the scaling metadata. In particular, the data layout disclosed herein provides improved energy efficiency in MMA calculations performed using narrow operand sizes.

[0029] Figure 1 An MMA operation between matrix A ("A matrix") 102 and matrix B ("B matrix") 104, used in the following description, is conceptually illustrated. Matrix A 102 includes M rows and K columns. Matrix B 104 includes K rows and N columns. The dot products of matrix A and matrix B (102 and 104) are accumulated in matrix C 106. The output of the MMA calculation is the final matrix C, represented as matrix D (also shown as 106).

[0030] The mechanism of narrow operand size as described above is to utilize an 8-bit floating-point format (FP8) that provides higher precision than an 8-bit integer (INT8) format. However, the higher accuracy of FP8 comes at the expense of lower energy efficiency than Int8. U.S. Application No. 18 / 484,790, entitled “LOW-PRECISION FLOATING-POINT DATAPATH IN A COMPUTER PROCESSOR”, filed by NVIDIA CORPORATION on October 11, 2023 (hereinafter referred to as “(NARROW FP APPLICATION) Narrow FP Application”), the entire contents of which are incorporated herein by reference, discloses a mechanism utilizing a multiplier unit configured to multiply two low-precision floating-point (FP) operands to generate an integer format (INT) result and an adder configured to add the integer result to a third operand. It also discloses utilizing a multiplier unit configured to multiply two low-precision (e.g., FP4) VS-Quant floating-point operands to generate a first INT result, an adder that reduces the first INT result to a second INT result, and logic that multiplies the second INT result by a low-precision floating-point scaling factor. The narrow FP application also describes utilizing a multiplier unit configured to add two low-precision VS-Quant (e.g., LOG4) floating-point operands to obtain a first sum, logic that converts part of the first sum to an INT result, and logic that converts the INT result to an INT product of the floating-point operands. Here, "low-precision floating point" refers to a floating-point data format that uses 8 bits or less (e.g., 4 / 6 / 8).

[0031] The mechanism described in the narrow FP application takes advantage of an energy-efficient floating-point data path microarchitecture with integer accumulation, and an enhancement mechanism for per-vector scaled quantization (VS-Quant) of low-precision floating-point parameters. Here, "FP" refers to the floating-point data format, INT refers to the integer data format, and INT32 refers to the 32-bit fixed-point number representation of the well-known integer data type. The example FP8 floating-point data format that can be used in the example embodiments of this document includes two encodings—E4M3 and E5M2, where the names encode the number of exponent (E) and mantissa (M) bits. For this data format, the term "mantissa" refers to the trailing valid bit field of the IEEE 754 standard (i.e., the bits that do not include the implicit leading 1 bit of a normal floating-point number).

[0032] Therefore, when using narrow operand sizes to compute MMA of A and B matrices (e.g., activation matrices and weight matrices, respectively, in some applications), the elements of A and B matrices represented using the narrow operand size can be multiplied by a scaling factor so that the computation can utilize the entire required range (e.g., the entire inference range). The scaling factor is stored separately from the A and B matrix operands and provided separately to the MMA circuit.

[0033] exist Figure 1 In FIG. 1 , matrix 108 contains the scaling factors for A matrix 102 and matrix 110 contains the scaling factors for B matrix 104. When scaling factors are used, the dot product of the A matrix and the B matrix is ​​further multiplied by the corresponding scaling factors. The scaling factors are referred to herein as "scaling metadata."

[0034] The scaling metadata or scaling factors for matrix A are provided as one or more scaling factors for each row of A. Similarly, the scaling metadata or scaling factors for matrix B are provided as one or more scaling factors for each column of B. In some embodiments, one scaling factor is provided for an entire row of operands of the A matrix or an entire column of operands of the B matrix. In some embodiments, each vector of the "scaling metadata vector length" along the rows of operands in the A matrix (i.e., in the K direction) or the columns of operands in the B matrix (i.e., in the K direction) can be multiplied by a corresponding scaling factor. The scaling metadata vector length defines the number of operands in the rows of the A matrix or the columns of the B matrix that are multiplied by each scaling factor. The data path depth is divided by the scaling metadata vector length to obtain the number of scaling factors specified by each vector in the row of A or column of B, whose length is equal to the data path depth.

[0035] Narrow operand size (e.g., 8-bit / 6-bit / 4-bit operand) MMA operations utilize scaling metadata to improve the accuracy of MMA operations. Conventional MMA operations can be defined as x,y =(from k=1 to A x,k *B k,y The sum of the "data path depth" of x,y , and the MMA operation using scaling metadata can be defined as: C x,y =((from k=1 to A x,k *B k,y The sum of the "data path depth" of the v * Scale B v The product of) + C x,y As described above, multiple scaling metadata values ​​may be used per datapath depth for an element of the A matrix or the B matrix.

[0036] For example, when 2 scaling metadata values ​​are used per data path depth (i.e., the "X2" format in this article), the result matrix of the MMA of the A matrix and the B matrix can be determined as (Equation 1): D[m][n] = (A[m][0]*B[0][n]+…+A[m][k / 2-1]*B[k / 2-1][n])*SF_A[m][0]*SF_B[n][0]+(A[m][k / 2]*B[k / 2][n]+…+A[m][k]*B[k][n])*SF_A[m][1]*SF_B[n][1]+C[m][n].

[0037] When 4 scaling metadata values ​​are used per datapath depth (i.e., the “X4” format herein), the result matrix of the MMA of the A matrix and the B matrix may be determined as (Equation 2): D[m][n] = (A[m][0]*B[0][n]+…+A[m][k / 4-1]*B[k / 4-1][n])*SF_A[m][0]*SF_B[n][0]+(A[m][k / 4]*B[k / 4][n]+…+A[m][k / 2-1]*B[k / 2-1][n])*SF_A[m][1]*SF_B[n][1]+(A[m][k / 2]*B[k / 2][n]+…+A[m][3k / 4-1]*B[3k / 4-1][n])*SF_A[m ][2]*SF_B[n][2]+(A[m][3k / 4]*B[3k / 4][n]+…+A[m][k]*B[k][n])*SF_A[m][3]*SF_B[n][3]+C[m][n].

[0038] In the above equations, A[][] and B[][] are operands from the operand matrix, and SF_A[] and SF_B[] are scaling factors.

[0039] Table 1 below shows an example correlation between the number of scaling factors per data path depth and the scaled metadata vector size for some example data path sizes according to some embodiments:

[0040]

[0041]

[0042] Simple solutions to the problem of using scaling metadata in MMA circuits may involve, for example: setting up different global memory layouts for the A matrix and the B matrix, thereby requiring software to write two copies of the scaling metadata, resulting in higher global memory usage and increased software complexity; high quantized global memory cost for storing scaling metadata, resulting in higher global memory usage and inefficient bandwidth usage; and high bandwidth usage when transferring scaling metadata to / from processing resources (e.g., stream processors (SMs), see Figure 9A ), resource usage is inefficient.

[0043] Embodiments of the present disclosure provide a scaling metadata block layout that enables efficient movement of scaling metadata within a computer system's memory hierarchy, minimizes quantization effects, reduces software complexity, and provides efficient use of memory storage. Figure 2A An example of a scaling metadata block data structure used to represent scaling metadata in an embodiment of the present disclosure is schematically shown.

[0044] Example scaling metadata block 202, scaling metadata block 204, and scaling metadata block 206, each of which is a 512-byte scaling metadata block (e.g., scaling factor). The scaling metadata block represents sufficient scaling metadata to include at most 128 rows of the A matrix or 128 columns of the B matrix, where the number of elements in the K dimension varies. Selecting a 128-row A matrix or a 128-column B matrix allows for a compact memory layout representation, does not require additional data movement / manipulation to move the scaling metadata around the memory hierarchy, and has low quantization overhead. Note that in some embodiments, the scaling metadata layout is quantized to 128 rows of the A matrix or 128 columns of the B matrix and varying numbers of elements in the K dimension.

[0045] According to the embodiments described herein, the scaled metadata block has a fixed size of 512 bytes and always contains enough data for 128 rows of the A matrix or 128 columns of the B matrix. The difference is that, depending on the number of scaling factors for each data path depth (e.g., 1, 2, or 4), the scaled metadata block supports different widths of the A matrix or B matrix. For example, considering a data path depth of 64, using X4, X2, and X1 formats, widths of 64, 128, and 256 (i.e., in the K direction) of the A matrix or B matrix can be covered by the scaled metadata block.

[0046] exist Figure 2AIn the illustration shown, scale metadata block 202 includes scale factors according to an "X1" format, i.e., an implementation of 1 scale factor per data path depth along the rows of the A matrix or the columns of the B matrix. Scale metadata block 204 includes scale factors for an "X2" format, i.e., an implementation of 2 scale factors per data path depth along the rows of the A matrix or the columns of the B matrix, and scale metadata block 206 includes scale factors for an "X4" format, i.e., an implementation of 4 scale factors per data path depth along the rows of the A matrix or the columns of the B matrix.

[0047] Figure 2A The scaling metadata block layout shown in provides a 512-byte block, regardless of whether the number of scaling factors per datapath depth along the rows of the A matrix or the columns of the B matrix is ​​1 (X1), 2 (X2), or 4 (X4), and regardless of the dimensions of the A matrix and the B matrix. Within the block, each scaling factor is represented in bytes.

[0048] The scaling metadata block 202 stores sufficient scaling metadata (ie, scaling factors) for 128 rows of the A matrix or 128 columns of the B matrix. For example, in a hardware architecture with a dot product width of 128, the scaling metadata block 202 includes scaling factors for 4 MMA operations.

[0049] Block 204, including two scaling factors per datapath depth along the rows of A or the columns of B, stores sufficient scaling metadata (i.e., ratio factors) for 128 rows of the A matrix or 128 columns of the B matrix. For example, in a hardware architecture with a dot product width of 128, the scaling metadata block 204, including two scaling factors per row or column, includes scaling factors for two MMA operations. That is, each dot product uses 2x128 scaling factors, providing enough scaling factors for two such dot products in 512 bytes.

[0050] Block 206, including four scaling factors per datapath depth along the rows of A or the columns of B, stores sufficient scaling metadata (i.e., scaling factors) for 128 rows of the A matrix or 128 columns of the B matrix. For example, in a hardware architecture with a dot product width of 128, the scaling metadata block 206, including four scaling factors per datapath depth along the rows of A or the columns of B, includes scaling factors for one MMA operation. That is, given a datapath depth of 128, at four scaling factors per datapath depth, each MMA operation requires 512 (i.e., 4x128) scaling factors, and thus a single MMA operation requires the entire 512 bytes of the block (e.g., each scaling factor is represented in bytes).

[0051] Table 2 below shows an example correlation between the number of scaling factors per data path depth, the scaled metadata vector length, and the size of the matrix A / B covered by the scaled metadata block, for some example data path depths, according to some embodiments:

[0052]

[0053]

[0054] like Figure 2A As shown in the above description, in the MMA calculation, the layout and use of the scaling metadata blocks of the A matrix and the B matrix are the same.

[0055] Another key aspect of the scaling metadata block (eg, 202, 204, and 206) in embodiments of the present disclosure is that it enables scaling metadata to be stored in the same layout at various levels of the memory hierarchy. Figure 2B As shown, for example, in computer system 210, scaling metadata block 202 (e.g., Figure 2A 216 ) can be stored in global memory (GMEM) 216, in shared memory (SMEM) 218 ​​located in a parallel processing unit (PPU) 212 of the computer system 210, and / or in device memory (e.g., identified herein as tensor memory or TMEM) 220 located in the MMA circuitry 215 of the PPU 212. The scaled metadata block 202 layout is defined such that it can be moved, for example, from GMEM to SMEM and / or TMEM by a data movement module 222 (e.g., a TMAU module described later) without requiring additional data manipulation. For example, a copy of block 202 may exist in GMEM, SMEM, and / or TMEM at some point during operation of the computer system. Within GMEM 216, all scaled metadata for either matrix A or matrix B may be stored in a plurality of scaled metadata blocks 217, with scaled metadata block 202 being one of the blocks in the plurality of scaled metadata blocks 217. In SMEM 218 and TMEM 220, corresponding multiple scaled metadata blocks 219 and 221, respectively, may be copies of portions of multiple scaled metadata blocks 217 in GMEM 216. For example, GMEM 216 may store scaled metadata for the entire A matrix of operands, scaled metadata for a subset of rows of the A matrix may be copied to SMEM 218, and the subset in SMEM may be copied to TMEM 220 for use by the MMA circuitry.

[0056] Figure 2C Schematically illustrates an example area in the A matrix or B matrix mapped by a scaling metadata block, a depth vector mapped to the A matrix or B matrix, and the scaling metadata vector according to some embodiments of the present disclosure.

[0057] In some embodiments, a plurality of scaled metadata blocks (also referred to as scaled metadata block data structures) are Figure 3 The illustrated or similar manner is arranged in the global memory of the computer system and in the device memory connected to the MMA circuit during the period when the computer system MMA circuit calculates the multiplication of the A matrix and the B matrix.

[0058] Each scaled metadata block may correspond to a region in one of the A matrix or the B matrix (e.g., region 234 in the A matrix 102 or region 234′ in the B matrix). That is, when a scaled metadata block is said to correspond to region 234, the scaling factors of the scaled metadata block are applied in the MMA calculation to scale the operands in that region of the A matrix or the B matrix.

[0059] Figure 2C A depth vector associated with the A matrix and the B matrix is ​​also conceptually illustrated. In some embodiments, the depth vector is defined based on the data path depth of the MMA circuit. For example, the depth vector is defined such that the length of the depth vector is the same as the data path depth of the MMA circuit. The data path depth can be the number of processing channels in the MMA circuit that can accept simultaneous inputs. For example, the data path depth can be the number of A matrix parameters or the number of B matrix parameters that can be simultaneously input to the MMA circuit.

[0060] Each scaling metadata block data structure may include s scaling factors for a corresponding region (e.g., 234, 234') defined by p rows and q columns in the A matrix or q rows and p columns in the B matrix. The value of q may be determined based on the length of the depth vector, s, p, and the scaling factor assignment associated with the depth vector length.

[0061] The scaling factor allocation can be specified as a number of scaling factors. For example, the scaling factor allocation can be the number of scaling factors per length of the vector length according to one of a variety of layouts. Example layouts that can be used in embodiments can include the above-mentioned X1, X2, and X4 layouts.

[0062] The value of q can be determined according to the relationship d*s / (p*r), where d is the length of the depth vector and r is the scaling factor assignment.

[0063] In one example, s=512, p=128, a first length of the depth vector is determined according to a data path of the MMA circuit, and r is determined based on the first vector length and according to one of the X1, X2, or X4 layouts.

[0064] In each of the multiple scaling metadata block data structures, the s scaling factors corresponding to the corresponding regions of the A matrix or the B matrix are arranged so that the scaling factor corresponding to a column of the A matrix is ​​interleaved with the scaling factors of one or more other columns of the A matrix, or the scaling factor corresponding to a row of the B matrix is ​​interleaved with the scaling factors of one or more other rows of the B matrix.

[0065] Interleaving may include arranging scaling factors of one or more other columns of the A matrix between scaling factors corresponding to a first set of consecutive rows in a column of the A matrix and scaling factors corresponding to a second set of consecutive rows in a column of the A matrix, or arranging scaling factors of one or more other rows of the B matrix between scaling factors corresponding to a first set of consecutive columns in a row of the B matrix and scaling factors corresponding to a second set of consecutive columns in a row of the B matrix. The one or more other columns of the A matrix may include r-1 other columns, or the one or more other rows of the B matrix may include r-1 other rows, where r is the scaling factor allocation.

[0066] In usually Figure 2B In many real-world applications running on the computer systems shown, the operand matrix of the MMA can be significantly larger than that of Figure 2A The 128x128 or 128x64 operand matrices discussed. An A matrix or B matrix that is larger than the area corresponding to the scaled metadata block (eg, 234, 234') will require multiple scaled metadata blocks. Figure 3 1 shows an example of logically organizing scaled metadata blocks (e.g., such as scaled metadata block 202, scaled metadata block 204, and scaled metadata block 206) in a memory (e.g., GMEM 216) to correspond to operand matrices of different sizes. On the left, the logical block numbers of the multiple scaled metadata blocks of X1 and X4 are shown, and on the right, the corresponding GMEM addresses for each layout of the logically numbered blocks are shown.

[0067] The logical number layout 302 and the corresponding address layout 304 in the top row of the layout show the organization of multiple scaling metadata blocks, which include scaling factors for the 768x512 dimension operand matrices in the K direction and the N direction respectively. Figure 2A 202 in the B matrix), each scaled metadata block includes the scaling factors for the blocks K=128 and N=128 in the operand matrix (taking into account the "N" of the B matrix). Therefore, the example operand matrix of 768x512 will require 4 scaled metadata blocks in the N direction (i.e., 512 / 128) and 6 scaled metadata blocks in the K direction (i.e., 768 / 128). Therefore, the 768x512 matrix requires 24 scaled metadata blocks in X1 format, and logically numbered from 0 to 23.

[0068] Layout 304 shows the address layout in GMEM corresponding to the logical number layout 302 of the scaled metadata block for the 768x512 operand matrix. Address layout 304 shows that logical block numbers 0, 1, 2, 3, 4, 5, ..., 23 are mapped to GMEM byte addresses (or address offsets) 0, 512, 1024, 1536, 2048, ..., 11776, illustrating the 512-byte offset between two consecutive scaled metadata blocks. Since the 768x512 operand matrix is ​​divisible by the scaled metadata block (providing the scaling factor for the 128x128 block in the X1 implementation), no operands are quantized at the edges of the operand matrix for the scaling factor.

[0069] The logical block layout 306 and corresponding address layout 308 illustrate a mapping of scaling metadata blocks of scaling factors in an X2 format including an 800x600 operand matrix. Neither 800 nor 600 is divisible by 128 (the M / N or K dimension of the operand block represented by the scaling metadata block). Therefore, in an example embodiment, in each dimension, the last block in the 800x600 operand matrix (i.e., the block after the last divisible block) is quantized with respect to the scaling factor mapping. For example, logical blocks 4, 9, 14, ..., 34 of scaling metadata may each overlap 88 (600-4x128) columns, and logical blocks 30, 31, 32, ..., 34 of scaling metadata may each overlap only 32 (800-6*128) rows. The quantization block 307 is in Figure 3 Indicated by shading in the layout.

[0070] Layout 308 shows an address layout in GMEM that corresponds to the logical numbering layout 306 of the scaled metadata blocks for the 800x600 operand matrix. Address layout 308 shows that scaled metadata blocks with logical block numbers 0, 1, 2, 3, 4, 5, 6, ... 34 are mapped to GMEM byte addresses (or address offsets) 0, 512, 1024, 1536, 2048, 2560, ... 17408, illustrating the 512-byte offset between two consecutive scaled metadata blocks.

[0071] Logical block layout 310 and corresponding address layout 312 illustrate the mapping of a scaled metadata block including a 900x700 operand matrix in an x4 format for scale factors. Neither 900 nor 700 is divisible by 128 (the scaled metadata block size). Therefore, in an example embodiment, the last block in each dimension (i.e., the block after the last evenly divisible block) is quantized. For example, logical blocks 4, 9, 14, ... 34 of scaled metadata may each include the remaining 60 (700 - 5x128) columns, and logical blocks 60, 61, 62, ... 64 of scaled metadata may each include the remaining 32 (900 - 12 * 64 (number of blocks * k-direction block size)) rows.

[0072] Layout 312 shows an address layout in GMEM corresponding to the logical number layout 310 of the scaled metadata blocks for the 900x700 operand matrix. Address layout 312 shows that scaled metadata of logical block numbers 0, 1, 2, 3, 4, 5, 6, ... 64 are mapped to scaled metadata of GMEM byte addresses (or address offsets) 0, 512, 1024, 1536, 2048, 2560, ... 32768, illustrating the 512-byte offset between two consecutive scaled metadata blocks.

[0073] and Figure 3 shows how to map multiple scaled metadata blocks to an operand matrix, Figure 4A and Figure 4B The mapping within each scaling metadata block is shown.

[0074] Figure 4A shows an example mapping of 512 bytes of a scaled metadata block to an A matrix, respectively, according to some embodiments of the present disclosure, and Figure 4B The mapping of a 512-byte scaled metadata block to a B matrix is ​​shown. The mapping shown is for an X1 format (e.g., scaled metadata block 202), where a scale factor is provided for each datapath depth. The mapping is shown as a 32x16 byte arrangement.

[0075] Figure 4AIt shows mapping of the X1 format scaling metadata for matrix A to 128 rows in M ​​dimensions (M=0 to 127), and for each row, to 4 columns in K dimensions (K=0 to 3). Thus, starting from the top left corner with index M0K0, from top to bottom and from left to right, the element indices of the first four columns are M0K0, M1K0, ..., M31K0 for the first column, M0K1, M1K1, ..., M31K1 for the second column, M0K2, M1K2, ..., M31K2 for the third column, M0K3, M1K3, ..., M31K3 for the fourth column, and in the same pattern, the second group of 4 columns have M32K0, M33K, ..., M63K0, M32K1, ..., M63K1, M64K2, M65K2, ..., M95K2, M96K3, M97K3, ..., M127K3, etc. It should be noted that the width of each scaled metadata element is 1 byte. As can be seen, the column pattern repeats every 4 bytes (e.g., columns starting with M0K0 and M32K0). X1 format, that is Figure 4A The layout format in provides a scaling factor of 1 for each column of B or row of A for a 128x128 block. Figure 4A The X1 layout in shows 4 scaling factors along the K dimension (K=0...3) and 128 scaling factors along the M dimension (M=0...127).

[0076] The scaled metadata layout in the scaled metadata block of the B matrix is ​​the same as the scaled metadata layout in the scaled metadata block of the A matrix shown in 4A. In the X1 format, the index M is replaced by N (N represents the column in the B matrix).

[0077] Figure 4B Scaling metadata for mapping the B matrix to 128 rows in N dimensions (N = 0 to 127) and 4 columns in K dimensions (K = 0 to 3) for each row is shown. Thus, starting from the upper left corner with index M0K0, from top to bottom, from left to right, the element indices of the first four columns are M0K0, M1K0, ..., M31K0 for the first column, M0K1, M1K1, ..., M31K1 for the second column, M0K2, M1K2, ..., M31K2 for the third column, M0K3, M1K3, ..., M31K3 for the fourth column, and in the same pattern, the second group of 4 columns have M32K0, M33K, ..., M63K0, M32K1, ..., M63K1, M64K2, M65K2, ..., M95K2, M96K3, M97K3, ..., M127K3, etc. It should be noted that each scaling metadata element is 1 byte wide. The X2 format provides 2 scaling factors for each datapath depth. Figure 4B It is shown that there are 2 scaling factors in K dimensions (K=0 and 1, 2 and 3) and 128 scaling factors in N dimensions.

[0078] Figure 4C The scaling metadata for the A matrix or B matrix is ​​shown mapped to 128 rows in M ​​dimensions or 128 columns in N dimensions (M or N = 0 to 127), and for each row / column, mapped to 4 columns / rows in K dimensions (K = 0 to 3). The X4 format provides four scaling factors for each datapath depth. Figure 4C It is shown that there is 1 scaling factor in K dimensions (K=0-3) and 32 scaling factors in M ​​or N dimensions. For example, therefore, Figure 4A 、 Figure 4B and Figure 4C This shows that X1, X2, and X4 have sufficient scaling metadata for 4, 2, and 1 dot products, respectively.

[0079] As can be seen from the arrangement of logical scaling metadata block 302 , logical scaling metadata block 304 , and logical scaling metadata block 306 , according to an example embodiment, quantization, if any, occurs only at the edges of the logical arrangement of scaling metadata blocks.

[0080] The scaled metadata block provided in the present disclosure enables scaled data for matrix multiplication to be retrieved from GMEM and copied to shared memory (SMEM) and from shared memory to device memory (e.g., TMEM) of an MMA circuit. In some embodiments, a first command (e.g., a UTMA* / UBLKP instruction) is used to move metadata from GMEM to SMEM, and a second command (e.g., a UTCCOPY instruction) is used to move metadata from SMEM to TMEM. In some embodiments, the block copy instruction can operate to move the scaled metadata based on the size of the scaled metadata block (e.g., 512 bytes). The MMA circuit can be configured to use the scaled metadata from TMEM. In some embodiments, the MMA circuit can be configured to use the scaled metadata from SMEM.

[0081] In some embodiments, the MMA circuitry may compute scaling metadata for the A and B matrices and write the computed scaling metadata to the register files of the associated SMs. Some embodiments may provide for storing the computed scaling metadata from the SM register files in SMEM as an arrangement of scaling metadata blocks and moving the scaling metadata to GMEM without further manipulation of the data.

[0082] The layout of scaled metadata blocks as defined in this disclosure enables speed-of-light (SOL) movement of scaled metadata in the memory hierarchy by eliminating the necessity to perform any additional data manipulation when copying scaled metadata between layers of the memory hierarchy.

[0083] Figure 5A flowchart 500 is shown of a process for computing the product of an A matrix and a B matrix based on their operands and scaling metadata associated with the operands. As described above, matrix multiplication can be used in many applications.

[0084] At 502, scaling metadata block data structures are stored in a global memory of a computer system, each scaling metadata block data structure including s scaling factors for a corresponding region defined by p rows and q columns in the A matrix or q rows and p columns in the B matrix. q is determined based on the depth vector length, s, p, and the scaling factor assignment associated with the depth vector length. Example techniques for determining q are described above. Scaling metadata for the A matrix and the B matrix can be stored in different multiple scaling metadata block data structures. For example, the storage can be initiated by software running on the CPU of the computer system.

[0085] As described above, within each scaled metadata block data structure, columns may be interleaved. Also as described above, the scaled metadata block data structures may have at least some scaled metadata block data structures covering regions in the A matrix or the B matrix where the number of operands is different than the number of operands in the other scaled metadata block data structures.

[0086] At 504, a copy of the plurality of scaled metadata block data structures from the global memory to the device memory of the MMA circuit in the computer system is performed. For example, during an MMA operation, a data movement module (e.g., the 222 module or the TMAU module discussed below) may read a group of scaled metadata block data structures and store the read group of scaled metadata block data structures directly to the device memory (e.g., TMEM) of the MMA circuit or after intermediate storage in SMEM.

[0087] At 508, a product calculation is performed in the MMA circuit, where the scaling metadata associated with the operands includes a plurality of scaling metadata block data structures read from device memory. In some embodiments, the scaling metadata block structures may be read from SMEM. Equation 1, Equation 2, or another equation may be used to calculate the product of the A matrix and the B matrix using the scaling factor.

[0088] In some example embodiments, such as with respect to Figure 6 and / or Figure 7A / Figure 7B The MMA circuit described above can be used for the above-mentioned MMA operations. In order to optimize the data movement along each processing channel of the MMA circuit, it is necessary to load the scaling metadata of multiple MMA instructions at once. The scaling block structure of the embodiment can package the scaling metadata in a format in which the scaling metadata of each dot product operation is interleaved along the K direction (i.e., the channel direction) in subsequent storage locations, such as Figures 4A to 4B The MMA circuit can then use the selector in the instruction to select the interleaved columns in the scaled metadata from TMEM (e.g., Figure 4A This further facilitates efficient movement of data from GMEM to TMEM and subsequent consumption of this scaled data by the MMA circuitry.

[0089] Depending on the number of scale elements per row / column represented in the scale metadata block, the packing pattern along the K dimension may vary depending on the number of scale factors specified for the MMA instruction.

[0090] The scaled metadata block layout of an embodiment allows K interleaving to be performed with 1 byte, 2 bytes, or 4 bytes without changing the overall layout. This facilitates software by avoiding the need for software to separately provide different interleaving modes.

[0091] The scaled metadata blocks defined in the example embodiments allow for bulk data movement from GMEM through SMEM to TMEM, which can keep the inputs of the MMA circuit fully utilized with minimal read-modify-write operations necessary when using smaller granularity loads (e.g., less than 128).

[0092] From another perspective, this also allows the cost of bulk movement of scaling metadata through the memory hierarchy to be amortized over several MMA operations.

[0093] It should also be noted that 512-byte blocks provide efficient storage utilization because, for example, all bytes in the block are fully utilized.

[0094] Embodiments of the present disclosure provide a compact memory layout for scaled metadata that is independent of the size of the scaled metadata vector. The scaled metadata layout of the embodiments efficiently utilizes the memory of each level of the memory hierarchy (eg, GMEM, SMEM, TMEM).

[0095] Embodiments provide a consistent storage layout for scaling metadata for the A matrix and the B matrix of an MMA circuit, regardless of the use of the scaling metadata. The scaling metadata for the A matrix and the B matrix have the same global memory layout, allowing the scaling metadata to be used as either the A matrix or the B matrix.

[0096] The ability to load scaling metadata without additional data manipulation or movement, which can degrade performance, increase software complexity, and increase power consumption. This provides efficient (e.g., 100% SOL) use of resources when transferring scaling metadata from global memory to tensor memory and from the Streaming Processor (SM) register file to global memory.

[0097] Figure 6 An example of using narrow operand sizes in the data path to further reduce precision in the data path of the vector multiplication unit 602 using the VS-Quant FP4 data format is shown. In this disclosure, "FP" refers to the floating point data format, INT refers to the integer data format, and INT32 refers to the well-known 32-bit fixed-point number representation of the integer data type. The exemplary embodiment may be described using INT32 accumulation, but the disclosed mechanism is also applicable to INT accumulation of other bit widths.

[0098] The example FP8 floating-point data format that can be used in the example embodiments of this document includes two encodings—E4M3 and E5M2, where the names encode the number of exponent (E) and mantissa (M) bits. For this data format, the term "mantissa" refers to the trailing significant bit field of the IEEE 754 standard (i.e., the bits excluding the implicit leading 1 bit of a normal floating-point number). The E4M3 encoding can be particularly suitable for representing weight and activation tensors (e.g., in deep learning applications), and the E5M2 encoding can be particularly suitable for representing gradient tensors.

[0099] Quantization enables efficient acceleration of deep neural networks by reducing model memory usage and leveraging low-cost integer math hardware units. Quantization uses scaling factors to map floating-point weights and activations in trained models to low-bitwidth integer values. Over-quantization, which reduces precision too aggressively, can lead to degraded model accuracy. When scaling factors are shared coarsely across many dimensions of each tensor, the effective precision of individual elements within the tensor is limited.

[0100] The following description of exemplary embodiments includes logic components such as exclusive OR (XOR), addition, complement, shift, truncation, alignment, etc. Unless otherwise specified, these components may be implemented in any number of ways known and understood in the art.

[0101] Figure 6 Depicted is an embodiment microarchitecture of a data path of a vector multiplication unit (MAC; eg, including MMA circuitry) that may be used in some embodiments of the present disclosure. Figure 6 The shown microarchitecture is described in a narrow FP application. This microarchitecture that exploits operand size can demonstrate improved area and energy efficiency over traditional schemes.

[0102] The depicted embodiment includes a data path for VS-Quant FP4 (e.g., E2M1 encoded) vectors and a per-vector FP4 (e.g., unsigned E3M1) scaling factor. Utilizing 4-bit precision and quantized 4-bit precision, the area and energy consumption of the vector multiply-accumulate unit logic can be significantly reduced. While the example embodiment specifically utilizes E2M1 (signed) and E3M1 (unsigned) formats, other FP4 and quantized FP4 formats can also be utilized (e.g., E1M2, E0M3, E3M0 vectors have E4M0, E2M2, E1M3, E0M4 per-vector scaling factors, depending on the constraints of the implementation). In addition, the bit width of the per-vector scaling factor can be increased to more than four bits (e.g., to eight bits), with a corresponding increase in logic size and / or complexity.

[0103] The VS-Quant FP4 data format consists of a vector of FP4 (E2M1, E1M2, E0M3) formatted elements and a scaling factor for FP4 or FP8 formatting applied to all elements in the vector. VS-Quant LOG4 consists of a vector of elements, each represented in a fixed-point exponent-only format (E3.0, E2.1, E1.2), and an exponent bias scaling factor applied to each element in the vector. Traditional block data formats such as Block FP and Microsoft-FP consist of a vector of fixed-point formatted elements and a scaling factor applied to the exponents of those elements only.

[0104] In the data path, the exponents of the two operands A and B are added (adder 604), the mantissas are multiplied (mantissa multiplier 606), and the sign bits are exclusive-ORed (XOR logic 608). The product mantissa MP is complemented based on the product sign bit SP (2's complement 610). The result is shifted and truncated based on the product exponent EP (shifter and truncation 612) to generate partial products.

[0105] The partial products from multiplication unit 602 are reduced (added) in adder 614. Post-processing of the reduced results is then performed. The reduced results are multiplied by the product of MSA and MSW (multipliers 616 and 618). MSA represents the mantissa scaling factor for the input activations of the neural network layer being processed, and MSW represents the mantissa scaling factor for the weights. Since the mantissa scaling factors are in E3M1 format, multiplier 616 includes a two-bit multiplier that produces a four-bit product. Multiplier 618 performs a four-bit multiplication on the output of the adder 614 layer.

[0106] The exponential scaling factors (ESA for activation and ESW for weight) are added (adder 620) and applied as a shift amount to the output of multiplier 418, and truncated appropriately (shifter and truncation 622). The result is added to a third 32-bit 2's complement operand (C, e.g., running sum) to produce the final output D (adder 624).

[0107] The use of FP4 format, per-vector scaling introduces only a small overhead to scale the dot product outputs and perform accumulation in INT32 format.

[0108] Figure 7A and Figure 7B 1 shows a portion of a processor (data path processor) including a data path configured to implement matrix operations according to some embodiments. In an example embodiment, the data path processor 710 can be located in a stream processor. For example, in Figure 9A In the illustrated streaming multiprocessor (SM) 940 , the plurality of processor cores 1050 may include one or more datapath processors 710 .

[0109] The portion of the processor 710 includes a plurality of datapath channels 702 , each of which includes a datapath processing circuit 704 configured to implement matrix operations.

[0110] Each datapath processing circuit 704 can be configured with a first memory component 706, such as, for example, a plurality of registers, and, optionally, a second memory component 708, such as, for example, auxiliary RAM. In the example of the convolution calculation described above, in order for the datapath processing circuit 704 in the datapath 702 to perform its portion of the MMA calculation, the weight matrix ("B matrix") elements are obtained from the first memory component 706, and the activation matrix ("A matrix") elements are obtained from a local buffer or SMEM (e.g., SM 940 shared memory 1070) via interface 714. The first memory component 706 is configured to receive the weight matrix elements from the SMEM via interface 712. In some embodiments, the SMEM from which the weight matrix elements are obtained and the SFEM from which the activation matrix elements are obtained are the same SMEM. For example, the SMEM 1070 on the SM 940 can be configured to obtain the weight matrix elements from global memory or L2 memory (e.g., Figure 10A 1090) receives data of the activation matrix and the weight matrix and provides the activation matrix to the datapath processing circuit 704 and the weight matrix to the second memory component 708 via the interconnection network 1080. The second memory component 708 can be connected to a register file, such as, for example, the register file 1020 of the SM 940, via the interface 716.

[0111] In one embodiment, the second memory component 708 is a local memory (sometimes referred to herein as "secondary memory," "tensor memory," or TMEM (e.g., TMEM 951)) configured to store data of the weight matrix in a manner that is separately accessible to each data path channel. Figure 9A An example optional TMEM 951 is shown in FIG.

[0112] In some embodiments, bus 714 can be used to share elements of the activation matrix between all data path channels. In some embodiments, the activation matrix elements can be obtained from SMEM via bus 714.

[0113] In the illustrated embodiment, a portion 710 of a datapath processor includes 32 datapath lanes 702, and multiple portions 710 (also referred to as "partitions" or "subpartitions") are connected so that they can operate as one datapath processor. For example, in one embodiment, a 128-element activation vector is used as a B operand for the datapath processor, such that each of the 128 elements of the activation vector is implemented as a B operand used by a corresponding one of the datapath lanes 702. U.S. Application Nos. 18 / 458,638 and 18 / 449,381 filed by NVIDIA CORPORATION describe 7A to 7B The entire contents of this application are incorporated herein by reference.

[0114] GPU Architecture Examples

[0115] Describes the Figures 1 to 6 An exemplary illustrative architecture of a PPU (e.g., GPU) described in the related text. The following information is for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be selectively combined with or without excluding other features described.

[0116] Figure 8A parallel processing unit (PPU) 800 is shown according to one embodiment, which may be included on a die of the PPU. In one embodiment, the PPU 800 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 800 is a latency-hidden architecture designed for processing many threads in parallel. A thread (e.g., an execution thread) is an instantiation of an instruction set configured to be executed by the PPU 800. In one embodiment, the PPU 800 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 800 can be used to perform general-purpose computations. In some other embodiments, the PPU 800 is configured to implement large neural networks in deep learning applications or other high-performance computing applications.

[0117] One or more PPUs 800 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPUs 800 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0118] like Figure 8 As shown, the PPU 800 includes an input / output (I / O) unit 805, a front-end unit 815, a scheduler unit 820, a work distribution unit 825, a hub 830, a crossbar switch (Xbar) 870, one or more general processing clusters (GPCs) 850, and one or more partition units 880. The PPU 800 can be connected to a host processor or other PPUs 800 via one or more high-speed NVLink 810 interconnects. The PPU 800 can be connected to a host processor or other peripheral devices via an interconnect 802. The PPU 800 can also be connected to a memory including a plurality of memory devices 804. In one embodiment, the memory 804 can include a plurality of dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem, in which multiple DRAM dies are stacked within each device.

[0119] The NVLink 810 interconnect enables the system to scale and include one or more PPUs 800 in conjunction with one or more CPUs, supporting cache coherency between the PPU 800 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 810 through the hub 830 to or from other units of the PPU 800, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 11A and Figure 11B NVLink 810 is described in more detail.

[0120] The I / O unit 805 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via the interconnect 802. The I / O unit 805 can communicate directly with the host processor via the interconnect 802, or through one or more intermediate devices (such as a memory bridge). In one embodiment, the I / O unit 805 can communicate with one or more other processors (e.g., one or more PPUs 800) via the interconnect 802. In one embodiment, the I / O unit 805 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating over a PCIe bus, and the interconnect 802 is a PCIe bus. In alternative embodiments, the I / O unit 805 can implement other types of known interfaces for communicating with external devices.

[0121] The I / O unit 805 decodes data packets received via the interconnect 802. In one embodiment, the data packets represent commands configured to cause the PPU 800 to perform various operations. The I / O unit 805 sends the decoded commands to various other units of the PPU 800 as specified by the commands. For example, some commands may be sent to the front end unit 815. Other commands may be sent to the hub 830 or other units of the PPU 800, such as one or more copy engines, video encoders, video decoders, a power management unit, etc. (not explicitly shown). In other words, the I / O unit 805 is configured to route communications between and among the various logical units of the PPU 800.

[0122] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 800 for processing. The workload may include many instructions and data to be processed by those instructions. A buffer is an area in memory that is accessible (i.e., read / write) by both the host processor and the PPU 800. For example, the I / O unit 805 can be configured to access a buffer in system memory connected to the interconnect 802 via a memory request transmitted through the interconnect 802. In one embodiment, the host processor writes a command stream into the buffer and then sends a pointer to the start of the command stream to the PPU 800. The front end unit 815 receives pointers to one or more command streams. The front end unit 815 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 800.

[0123] The front-end unit 815 is coupled to the scheduler unit 820, which configures the various GPCs 850 to process tasks defined by one or more streams. The scheduler unit 820 is configured to track state information related to the various tasks managed by the scheduler unit 820. The state may indicate which GPC 850 the task is assigned to, whether the task is active or inactive, the priority associated with the task, etc. The scheduler unit 820 manages the execution of multiple tasks on one or more GPCs 850.

[0124] Scheduler unit 820 is coupled to work distribution unit 825, which is configured to dispatch tasks for execution on GPCs 850. Work distribution unit 825 can track a number of scheduled tasks received from scheduler unit 820. In one embodiment, work distribution unit 825 manages a pending task pool and an active task pool for each GPC 850. The pending task pool can include a number of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular GPC 850. The active task pool can include a number of time slots (e.g., 4 time slots) for tasks being actively processed by GPC 850. When a GPC 850 completes execution of a task, the task is evicted from the active task pool of GPC 850, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 850. If an active task on GPC 850 has become idle, such as while waiting for a data dependency to be resolved, the active task may be evicted from GPC 850 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 850 .

[0125] Work distribution unit 825 communicates with one or more GPCs 850 via XBar (crossbar) 870. XBar 870 is an interconnect network that couples many units of PPU 800 to other units of PPU 800. For example, XBar 870 can be configured to couple work distribution unit 825 to a specific GPC 850. Although not explicitly shown, one or more other units of PPU 800 can also be connected to XBar 870 via hub 830.

[0126] Tasks are managed by the scheduler unit 820 and dispatched to the GPCs 850 by the work distribution unit 825. The GPCs 850 are configured to process tasks and generate results. The results can be consumed by other tasks within the GPC 850, routed to a different GPC 850 via the XBar 870, or stored in the memory 804. The results can be written to the memory 804 via the partition unit 880, which implements a memory interface for reading data from and writing data to the memory 804. The results can be sent to another PPU 804 or CPU via the NVLink 810. In one embodiment, the PPU 800 includes a number U of partition units 880, which is equal to the number of independent and different memory devices 804 coupled to the PPU 800. Figure 9B The partition unit 880 is described in more detail.

[0127] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 800. In one embodiment, multiple computing applications are executed simultaneously by the PPU 800, and the PPU 800 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 800. The driver kernel outputs the tasks to one or more streams being processed by the PPU 800. Each task can include one or more related thread groups, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can execute in parallel. Collaborating threads can refer to multiple threads that include instructions to perform tasks and can exchange data through shared memory (SMEM). According to some embodiments, threads, cooperative threads, and hierarchical groupings of threads (e.g., cooperative thread arrays (CTAs) and cooperative group arrays (CGAs)) are described in more detail in U.S. application Ser. No. 17 / 691,621, the entire contents of which are hereby incorporated by reference. According to some embodiments, SMEM is described in U.S. application Ser. No. 17 / 691,690, which is hereby incorporated by reference.

[0128] Figure 9A According to one embodiment, Figure 8 PPU 800 GPC 850. Figure 9A As shown, each GPC 850 includes multiple hardware units for processing tasks. In one embodiment, each GPC 850 includes a pipeline manager 910, a pre-raster operation unit (PROP) 915, a raster engine 925, a work distribution crossbar (WDX) 980, a memory management unit (MMU) 990, and one or more data processing clusters (DPCs) 920. It should be understood that Figure 9A The GPC 850 may include instead Figure 9A Other hardware units or other than the units shown in Figure 9A Other hardware units besides those shown in .

[0129] In one embodiment, the operation of GPC 850 is controlled by pipeline manager 910. Pipeline manager 910 manages the configuration of one or more DPCs 920 for processing tasks assigned to GPC 850. In one embodiment, pipeline manager 910 can configure at least one of one or more DPCs 920 to implement at least a portion of a graphics rendering pipeline, a neural network, and / or a compute pipeline. For example, with respect to the graphics rendering pipeline, DPC 920 can be configured to execute vertex shading programs on programmable streaming multiprocessors (SMs) 940. Pipeline manager 910 can also be configured to route packets received from work distribution units 925 to appropriate logic units within GPC 850. For example, some packets can be routed to fixed-function hardware units within PROP 915 and / or raster engine 925, while other packets can be routed to DPC 920 for processing by primitive engines 935 or SMs 940.

[0130] PROP unit 915 is configured to route data generated by raster engine 925 and DPC 920 to the raster operations (ROP) unit, in conjunction with Figure 9B The PROP unit 915 may also be configured to perform color blending optimizations, organize pixel data, perform address translations, and the like.

[0131] The GPC 850 includes a DPC 920, a controller (MPC) 930, a primitive engine 935, and one or more SMs 940. The MPC 930 controls the operation of the DPC 920 and routes data packets received from the pipeline manager 910 to the appropriate units in the DPC 920. For example, data packets associated with vertices may be routed to the primitive engine 935, which is configured to fetch vertex attributes associated with the vertices from the memory 804. Conversely, data packets associated with shading programs may be sent to the SM 940.

[0132] SM 940 includes a programmable streaming processor configured to process tasks represented by multiple threads. Each SM 940 is multi-threaded and configured to simultaneously execute multiple threads (e.g., 32 threads) from a specific thread group. In one embodiment, SM 940 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group (e.g., a warp) is configured to process a different data set based on the same instruction set. All threads in a thread group execute the same instructions. In another embodiment, SM 940 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group is configured to process a different data set based on the same instruction set, but in which individual threads in a thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within warps when threads within the warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby achieving equal concurrency between all threads within and between warps. When execution state is maintained for each individual thread, threads executing the same instruction can be converged and executed in parallel for maximum efficiency. Figure 10A SM 940 is described in more detail. Figure 10B Conceptually illustrates the process of performing a multi-processor (e.g., Figure 10A Four sub-partitions 1091-1094 implemented in the SM) shown in .

[0133] The MMU 990 provides an interface between the GPC 850 and the partition unit 880. The MMU 990 can provide virtual to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the MMU 990 provides one or more translation lookaside buffers (TLBs) for performing translations from virtual addresses to physical addresses in memory 804.

[0134] Figure 9B According to one embodiment, Figure 8 The memory partition unit 880 of the PPU 800. Figure 9BAs shown, the memory partition unit 880 includes a raster operations (ROP) unit 950, a level 2 (L2) cache 960, and a memory interface 970. The memory interface 970 is coupled to the memory 804. The memory interface 970 can implement a 32-, 64-, 128-, or 1024-bit data bus for high-speed data transfer. In one embodiment, the PPU 800 incorporates U memory interfaces 970, one for each pair of partition units 880, where each pair of partition units 880 is connected to a corresponding memory device 804. For example, the PPU 800 can be connected to up to Y memory devices 804, such as a high-bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0135] In one embodiment, memory interface 970 implements an HBM2 memory interface and Y is equal to half a U. In one embodiment, the HBM2 memory stack is located on the same physical package as the PPU 800, providing significant power and area savings compared to traditional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, where each die in the HBM2 stack includes two 128-bit channels, for a total of 8 channels and a data bus width of 824 bits.

[0136] In one embodiment, memory 804 supports single-error correction with double-error detection ("SECDED") error correction code ("ECC") to protect data. ECC provides increased reliability for computer applications that are sensitive to data corruption. Reliability is particularly important in large-scale cluster computing environments where PPU 800 processes very large data sets and / or long-running applications.

[0137] In one embodiment, the PPU 800 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 880 supports unified memory to provide a single unified virtual address space for the CPU and PPU 300 memory, thereby enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 800 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 800 where the pages are accessed more frequently. In one embodiment, NVLink 810 supports address translation services, allowing the PPU 800 to directly access the CPU's page tables and providing full access to the CPU's memory through the PPU 800.

[0138] In one embodiment, the copy engine transfers data between multiple PPUs 800 or between a PPU 800 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 880 can then service the page fault, mapping the address to a page table, after which the copy engine can perform the transfer. In conventional systems, memory is fixed (e.g., non-pageable) for multiple copy engine operations between multiple processors, significantly reducing the available memory. With hardware page faults, addresses can be passed to the copy engine regardless of whether the memory page is resident, and the copy process is transparent.

[0139] Data from memory 804 or other system memory can be retrieved by memory partition unit 880 and stored in L2 cache 960, which is located on-chip and shared between various GPCs 850. As shown, each memory partition unit 880 includes a portion of L2 cache 960 associated with the corresponding memory device 804. Then, low-level caches can be implemented in various units within GPC 850. For example, each SM 940 can implement a level 1 (L1) cache. The L1 cache is private memory dedicated to a specific SM 940. Data from L2 cache 960 can be retrieved and stored in each L1 cache for processing in the functional units of SM 940. L2 cache 960 is coupled to memory interface 970 and XBar 870.

[0140] The ROP unit 950 performs graphics raster operations associated with pixel color, such as color compression, pixel blending, etc. The ROP unit 950 also implements depth testing in conjunction with the raster engine 925, receiving the depth of the sample position associated with the pixel fragment from the culling engine of the raster engine 925. The depth is tested against the corresponding depth in the depth buffer of the sample position associated with the fragment. If the fragment passes the depth test for the sample position, the ROP unit 950 updates the depth buffer and sends the result of the depth test to the raster engine 925. It should be understood that the number of partition units 880 can be different than the number of GPCs 850, and therefore, each ROP unit 950 can be coupled to each GPC 850. The ROP unit 950 tracks the packets received from the different GPCs 850 and determines to which GPC 850 the results generated by the ROP unit 950 are routed via the Xbar 870. Although in Figure 9B In the embodiment, ROP unit 950 is included in memory partition unit 880, but in other embodiments, ROP unit 950 may be located outside of memory partition unit 880. For example, ROP unit 950 may be located in GPC 850 or another unit.

[0141] Figure 10AAccording to one embodiment, Figure 9A Streaming multiprocessor 940. Figure 10A As shown, SM 640 includes an instruction cache 1005, one or more scheduler units 1010 (K), a register file 1020, one or more processing cores 1050, one or more special function units (SFUs) 1052, one or more load / store units (LSUs) 1054, an interconnection network 1080, and a SMEM / L1 cache 1070.

[0142] As described above, the work distribution unit 825 schedules tasks for execution on the GPCs 850 of the PPU 800. Tasks are assigned to specific DPCs 920 within the GPCs 850 and, if the task is associated with a shader program, may be assigned to the SMs 940. The scheduler unit 1010(K) receives tasks from the work distribution unit 825 and manages the scheduling of instructions for one or more thread blocks assigned to the SMs 940. The scheduler unit 1010(K) schedules the thread blocks for execution as warps of parallel threads, where each thread block consists of at least one warp. In one embodiment, each warp includes 32 threads. The scheduler unit 1010(K) can manage multiple different thread blocks, assigning different thread blocks to different warps, and then dispatching instructions from multiple different cooperative groups to various functional units (e.g., cores 1050, SFUs 1052, and LSUs 1054) during each clock cycle.

[0143] The Cooperative Group Array (CGA) provides a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling the expression of richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. The conventional programming model provides a single simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than the thread block granularity and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0144] Cooperative group arrays enable programmers to explicitly define thread groups at sub-block (e.g., as small as a single thread) and multi-block granularity and perform collective operations on threads, such as synchronization in cooperative groups. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative group array primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across an entire grid of thread blocks. Hierarchical grouping of threads (e.g., cooperative thread arrays (CTAs) and cooperative group arrays (CGAs)) according to some embodiments is described in more detail in U.S. application Ser. No. 17 / 691,621, the entire contents of which are incorporated herein by reference.

[0145] The dispatch unit 1015 is configured to deliver instructions to one or more functional units. In this embodiment, the scheduler unit 1010(K) includes two dispatch units 1015, which enable scheduling of two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 1010(K) may include a single dispatch unit 1015 or additional dispatch units 1015.

[0146] Each SM 940 includes a register file 1020 that provides a set of registers for the functional units of the SM 940. In one embodiment, register file 1020 is partitioned between each functional unit so that each functional unit is allocated a dedicated portion of register file 1020. In another embodiment, register file 1020 is partitioned between different warps organized by the SM 940. Register file 1020 provides temporary storage for operands connected to the data paths of the functional units.

[0147] Each SM 940 includes multiple processing cores 1050. In one embodiment, the SM 940 includes a large number (e.g., 128, etc.) of different processing cores 1050. Each core 1050 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (FPU) and an integer arithmetic logic unit (ILU). In one embodiment, the FPU implements the IEEE 754-2008 standard for floating-point operations.

[0148] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 1050. Specifically, tensor cores are configured to perform deep learning matrix operations, such as convolution operations for neural network training and inference.

[0149] In some embodiments, the transposition hardware is included in the processing core 1050 or another functional unit (e.g., SFU 1052 or LSU 1054) and is configured to generate diagonally stored matrix data and / or generate the original matrix and / or the transposed matrix from the diagonally stored matrix data. The transposition hardware can be provided within the SMEM 1070 to load the register file 1020 of the SM 940 into the path.

[0150] Each SM 940 also includes a plurality of SFUs 1052 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, an SFU 1052 may include a tree traversal unit (e.g., TTU 943) configured to traverse a hierarchical tree data structure. In one embodiment, an SFU 1052 may include a texture unit (e.g., texture unit 942) configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texels) from memory 1004 and sample the texture map to generate sampled texture values ​​for use in shader programs executed by the SM 940. In one embodiment, the texture map is stored in the SMEM / L1 cache 970. The texture unit performs texture operations, such as filtering operations using mip maps (i.e., texture maps at different levels of detail). In one embodiment, each SM 940 includes two texture units.

[0151] Each SM 940 also includes multiple LSUs 1054 that perform load and store operations between the SMEM / L1 cache 1070 and the register file 1020. Each SM 940 includes an interconnect network 1080 that connects each functional unit to the register file 1020 and connects the LSUs 1054 to the register file 1020 and the SMEM / L1 cache 1070. In one embodiment, the interconnect network 1080 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 1020 and to connect the LSUs 1054 to memory locations in the register file and the SMEM / L1 cache 1070.

[0152] SMEM / L1 cache 1070 is an on-chip memory array that allows data storage and communication between SM 940 and primitive engine 935, as well as between threads in SM 940. In one embodiment, SMEM / L1 cache 1070 includes 128KB of storage capacity and is in the path from SM 940 to partition unit 1080. SMEM / L1 cache 1070 can be used to cache reads and writes. One or more of SMEM / L1 cache 1070, L2 cache 960, and memory 1004 are backing stores.

[0153] Combining data cache and SMEM functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used by programs as a cache that does not utilize SMEM. For example, if SMEM is configured to use half its capacity, texture and load / store operations can use the remaining capacity. Integration within SMEM / L1 cache 1070 enables SMEM / L1 cache 1070 to function as a high-throughput pipeline for streaming data, while providing both high-bandwidth and low-latency access to frequently reused data.

[0154] In the context of the present disclosure, an SM or "streaming multiprocessor" refers to a processor having an architecture as described in US Patent No. 7,447,873 to Nordquist, including improvements and advancements thereto, and as implemented, for example, in multiple generations of NVIDIA GPUs. For example, an SM may include multiple processing engines or cores configured to concurrently execute multiple threads arranged in multiple single instruction, multiple data (SIMD) groups (e.g., warps), wherein each thread in the same group of the SIMD groups executes the same data processing program comprising a sequence of instructions on different input objects, and different threads in the same group of the SIMD groups execute using different processing engines or cores. The SM may also typically provide (a) a local register file having multiple lanes, wherein each processing engine or core is configured to access a different subset of the lanes; and instruction issue logic configured to select one of the SIMD groups and issue one of the instructions of the same data processing program to each of the multiple processing engines in parallel, wherein each processing engine executes the same instruction in parallel with the other processing engines using a subset of the accessible local register file lanes. The SM typically further includes core interface logic that is configured to initiate execution of one or more SIMD groups. As shown in the figure, the architecture of such an SM is to provide fast local SMEM, thereby enabling data sharing / reuse and synchronization between all threads of a CTA executing on the SM.

[0155] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 9AThe fixed-function graphics processing unit is shown bypassed, creating a simpler programming model. In the general-purpose parallel computing configuration, work distribution unit 1025 assigns and distributes thread blocks directly to DPC 920. The threads in the block execute the same program, using unique thread IDs in computations to ensure each thread produces unique results, using SM 940 to execute the program and perform computations, using SMEM / L1 cache 1070 to communicate between threads, and using LSU 1054 to read and write global memory through SMEM / L1 cache 1070 and memory partition unit 1080. When configured for general-purpose parallel computing, SM 940 can also write commands that scheduler unit 1020 can use to start new work on DPC 920.

[0156] The PPU 800 as described in the present disclosure may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 800 is included on a single semiconductor substrate. In another embodiment, the PPU 800 is included on a system-on-chip (SoC) along with one or more other devices (such as an additional PPU 800, a memory 1004, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0157] Exemplary Computing System

[0158] Systems with multiple GPUs and CPUs are being used across various industries as developers expose and exploit greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0159] Figure 11B The use according to one embodiment includes Figure 8 A conceptual diagram of a processing system 1100 implemented by a PPU 800 of two or more PPUs 800. An exemplary system 1165 can be configured to implement the methods disclosed in this disclosure. The processing system 1100 includes a CPU 1130, a switch 1155, and each of the multiple PPUs and corresponding memory 1004. NVLink 1010 provides a high-speed communication link between each PPU. Although Figure 11AA specific number of NVLink 1010 and interconnect 1002 connections are shown, but the number of connections connected to each PPU and CPU 1130 can vary. Switch 1155 interfaces between interconnect 1002 and CPU 1130. PPUs, memory 1004, and NVLink 1010 can be located on a single semiconductor platform to form parallel processing module 1125. In one embodiment, switch 1155 supports two or more protocols that interface between various different connections and / or links.

[0160] In another embodiment (not shown), NVLink 1010 provides one or more high-speed communication links between each PPU and CPU 1130, and switch 1155 interfaces between interconnect 1002 and each PPU. The PPUs, memory 1004, and interconnect 1002 may be located on a single semiconductor platform to form parallel processing module 1125. In yet another embodiment (not shown), interconnect 1002 provides one or more communication links between each PPU and CPU 1130, and switch 1155 interfaces between each PPU using NVLink 1010 to provide one or more high-speed communication links between the PPUs. In another embodiment (not shown), NVLink 1010 provides one or more high-speed communication links between the PPUs and CPU 1130 through switch 1155. In yet another embodiment (not shown), interconnect 1002 provides one or more communication links directly between each PPU. One or more NVLink 1010 high-speed communication links may be implemented as a physical NVLink interconnect or as an on-chip or on-die interconnect using the same protocol as NVLink 1010.

[0161] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that emulates on-chip operation and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 1125 may be implemented as a circuit board substrate, and each of the PPU 800 and / or memory 1004 may be a packaged device. In one embodiment, the CPU 1130, switch 1155, and parallel processing module 1125 are located on a single semiconductor platform.

[0162] In one embodiment, NVLink 1010 allows direct load / store / atomic access from the CPU 1130 to the memory 1004 of each PPU 800. In one embodiment, NVLink 1010 supports coherency operations, allowing data read from memory 1004 to be stored in the cache hierarchy of the CPU 1130, reducing cache access latency for the CPU 1130. In one embodiment, NVLink 1010 includes support for Address Translation Services (ATS), allowing the PPU 800 to directly access page tables within the CPU 1130. One or more NVLinks 1010 can also be configured to operate in a low-power mode.

[0163] Figure 11B An exemplary system 1165 is shown in which various architectures and / or functionality of various previous embodiments can be implemented. The exemplary system 1165 can be configured to implement the methods disclosed herein.

[0164] As shown, a system 1165 is provided that includes at least one central processing unit 1130 connected to a communication bus 1175. The communication bus 1175 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or one or more point-to-point communication protocols. The system 1165 also includes a main memory 1140. Control logic (software) and data are stored in the main memory 1140, which can take the form of random access memory (RAM).

[0165] System 1165 also includes an input device 1160, a parallel processing system 1125, and a display device 1145, such as a conventional CRT (cathode ray tube), an LCD (liquid crystal display), an LED (light emitting diode), a plasma display, etc. User input can be received from input device 1160 (e.g., a keyboard, a mouse, a touchpad, a microphone, etc.). Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 1165. Alternatively, the modules can be located separately or in various combinations of semiconductor platforms, depending on user needs.

[0166] Furthermore, the system 1165 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through the network interface 1135 for communication purposes.

[0167] System 1165 may also include auxiliary storage (not shown). Auxiliary storage 610 includes, for example, a hard drive and / or a removable storage drive, representative of a floppy disk drive, a tape drive, an optical drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0168] Computer programs or computer control logic algorithms may be stored in the main memory 1140 and / or the secondary storage. These computer programs, when executed, enable the system 1165 to perform various functions. The memory 1140, storage, and / or any other storage are possible examples of computer-readable media.

[0169] The architecture and / or functionality of the various preceding figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated for entertainment purposes, a dedicated system, and / or any other desired system. For example, system 1165 can take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0170] An application program can be implemented by an application program executed by a host processor, such as a CPU. In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that an application program can use to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of a PPU 800, which includes two or more PPUs 800. The API provides an abstraction for programmers, allowing them to utilize specialized graphics hardware, such as a PPU 800, to generate graphics data without requiring them to utilize the specific instruction set of the PPU 800. The application program can include API calls that are routed to the device driver of the PPU. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other instances, the device driver can perform operations at least in part by initiating operations on the PPU 800 using an input / output interface between the CPU and the PPU. In one embodiment, the device driver is configured to implement a graphics processing pipeline using the hardware of the PPU 800.

[0171] Various programs may be executed within a PPU 800 comprising two or more PPUs 800 to implement various stages of processing for an application. For example, a device driver may launch a kernel on a PPU 800 to perform one stage of processing on one SM 940 (or multiple SMs 940). The device driver (or the initial kernel executed by the PPU 800) may also launch other kernels on the PPU 800 to perform other stages of processing. If the application processing includes a graphics processing pipeline, some stages of the graphics processing pipeline may be implemented on fixed unit hardware, such as a rasterizer or data assembler implemented within the PPU 800. It should be understood that the results from one kernel may be processed by one or more intermediate fixed-function hardware units before being processed by a subsequent kernel on the SM 940.

[0172] The technology disclosed herein can be incorporated into any processor that can be used to process neural networks, such as a central processing unit (CPU), a graphics processing unit (GPU), an intelligence processing unit (IPU), a neural processing unit (NPU), a tensor processing unit (TPU), a neural network processor (NNP), a data processing unit (DPU), a vision processing unit (VPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and the like. Such a processor can be integrated into a personal computer (e.g., a laptop), a data center, an Internet of Things (IoT) device, a handheld device (e.g., a smartphone), a vehicle, a robot, or any other device that performs inference, training, or any other processing of a neural network. Such a processor can be used in a virtualized system so that an operating system executing in a virtual machine on the system can utilize the processor.

[0173] As an example, a processor incorporating the techniques disclosed herein can be used to process one or more neural networks in a machine to identify, classify, manipulate, process, operate, modify, or navigate physical objects in the real world. For example, such a processor can be used in an autonomous vehicle (e.g., a car, motorcycle, helicopter, drone, airplane, ship, submarine, delivery robot, etc.) to enable the vehicle to move in the real world. Additionally, such a processor can be used in a robot in a factory to select parts and assemble the parts into assemblies.

[0174] As an example, a processor incorporating the techniques disclosed herein may be employed to process one or more neural networks to identify one or more features in an image, or to alter, generate, or compress an image. For example, such a processor may be used to enhance images rendered using rasterization, ray tracing (e.g., using NVIDIA RTX), and / or other rendering techniques. In another example, such a processor may be employed to reduce the amount of image data transmitted from a rendering device to a display device over a network (e.g., the Internet, a mobile telecommunications network, a WIFI network, and any other wired or wireless network system). Such transmission may be used to stream image data from a server or a data center in the cloud to a user device (e.g., a personal computer, a video game console, a smartphone, other mobile device, etc.) to enhance services that stream images, such as NVIDIA GeForce Now (GFN), Google Stadia, and the like.

[0175] As an example, a processor incorporating the techniques disclosed herein can be used to process one or more neural networks for any other type of application that can utilize neural networks. For example, these applications may involve translating from one spoken language to another, recognizing and negating sounds in audio, detecting anomalies or defects in the production of goods and services, monitoring living and / or non-living things, medical diagnosis, decision making, etc.

[0176] As an example, a processor incorporating the techniques disclosed herein may be used to implement a neural network, such as a large language model (LLM), to generate content (e.g., images, video, text, essays, audio, etc.), respond to user queries, solve problems in mathematics and other fields, and the like.

[0177] All patents, patent applications, and publications cited herein are incorporated by reference for all purposes as if expressly stated.

[0178] While the invention has been described in connection with what are presently considered to be the most practical and preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments, but on the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A computer system comprising: a matrix multiplication and addition (MMA) circuit configured to compute a product of the A matrix and the B matrix based on operands of the A matrix and the B matrix and scaling metadata associated with the operands; as well as a device memory coupled to the MMA circuitry and storing a plurality of scaled metadata block data structures, each scaled metadata block data structure comprising s scale factors for a corresponding region defined by p rows and q columns in the A matrix or q rows and p columns in the B matrix, wherein q is determined based on a first vector length, s, p, and a scale factor assignment associated with the first vector length, The scaling metadata includes the s scaling factors. 2 . The computer system of claim 1 , wherein the first vector length is determined according to a data path of the MMA circuit.

3. The computer system of claim 1, wherein the operand comprises a narrow operand. 4 . The computer system of claim 1 , wherein the scaling factor assignment comprises a first number of scaling factors. 5 . The computer system of claim 4 , wherein the first number is determined according to one of a first layout designating 1 as the first number, a second layout designating 2 as the first number, or a third layout designating 4 as the first number.

6. The computer system of claim 1, wherein q = d*s / (p*r), where d is the first vector length and r is the scaling factor assignment.

7. The computer system of claim 6 , wherein s=512, p=128, the first vector length is determined according to a data path of the MMA circuit, and r is determined based on the first vector length and one of a first layout designating 1 as the first number, a second layout designating 2 as the first number, or a third layout designating 4 as the first number.

8. The computer system of claim 1, wherein each of the s scaling factors is 1 byte.

9. The computer system according to claim 8, wherein: For each scaled metadata block data structure in the plurality of scaled metadata block data structures, the values ​​assigned to s, p, the first vector length, and the scaling factor are the same as those of other scaled metadata block data structures in the plurality of scaled metadata blocks.

10. The computer system of claim 1 , wherein the device memory further stores a second plurality of scaled metadata block data structures, each scaled metadata block data structure in the second plurality of scaled metadata block data structures comprising s scale factors for a corresponding region defined by p rows and q columns in the A matrix or q rows and p columns in the B matrix, and wherein the scaled metadata further comprises s scale factors from a scaled metadata block structure from the second plurality of scaled metadata block data structures.

11. The computer system of claim 1 , further comprising a global memory having stored therein instances of the plurality of scaled metadata block data structures.

12. The computer system of claim 1 , further comprising a shared memory having another instance of the plurality of scaled metadata block data structures stored therein, wherein the another instance of the plurality of scaled metadata block data structures is copied from the global memory to the shared memory, and the plurality of scaled metadata block data structures are copied from the shared memory to the device memory.

13. The computer system according to claim 1, wherein: In each of the scaling metadata block data structures, the s scaling factors of the corresponding region are arranged so that a scaling factor corresponding to a column of the A matrix is ​​interleaved with scaling factors of one or more other columns of the A matrix, or a scaling factor corresponding to a row of the B matrix is ​​interleaved with scaling factors of one or more other rows of the B matrix.

14. The computer system of claim 13 , wherein the interleaving comprises arranging the scaling factors of one or more other columns of the A matrix between the scaling factors corresponding to a first set of consecutive rows in the column of the A matrix and the scaling factors corresponding to a second set of consecutive rows in the column of the A matrix, or arranging the scaling factors of one or more other rows of the B matrix between the scaling factors corresponding to a first set of consecutive columns in the row of the B matrix and the scaling factors corresponding to a second set of consecutive columns in the row of the B matrix.

15. The computer system of claim 13, wherein the one or more other columns of the A matrix include r-1 of the other columns, or the one or more other rows of the B matrix include r-1 of the other rows, where r is the scaling factor assignment.

16. The computer system of claim 1 , wherein the plurality of scale metadata block data structures are logically arranged as an m×n array of scale factors, and at least one of the scale factors in the m×n array corresponds to a number of the operands of the A matrix or the B matrix that is different from a number of other scale factors in the m×n array.

17. A method for computing the product of an A matrix and a B matrix based on operands of the A matrix and the B matrix and scaling metadata associated with the operands, comprising: storing scaled metadata block data structures in a global memory of the computer system, each scaled metadata block data structure comprising s scale factors for a corresponding region defined by p rows and q columns in the A matrix or q rows and p columns in the B matrix, wherein q is determined based on a first vector length, s, p, and a scale factor assignment associated with the first vector length; copying a plurality of scaled metadata block data structures of a matrix multiplication and addition (MMA) circuit in the computer system from the global memory to a device memory; as well as The product is calculated in the MMA circuit, wherein the scaling metadata associated with the operands comprises the plurality of scaling metadata block data structures read from the device memory.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer system, cause the computer system to perform operations comprising: storing scaled metadata block data structures in a global memory of the computer system, each scaled metadata block data structure comprising s scale factors for a corresponding region defined by p rows and q columns in the A matrix or q rows and p columns in the B matrix, wherein q is determined based on a first vector length, s, p, and a scale factor assignment associated with the first vector length; copying a plurality of scaled metadata block data structures of a matrix multiplication and addition (MMA) circuit in the computer system from the global memory to a device memory; as well as The product is calculated in the MMA circuit, wherein the scaling metadata associated with the operands comprises the plurality of scaling metadata block data structures read from the device memory.

Images