Increasing computational efficiency by multi-level 8-bit floating point matrix multiplication with format conversion

Abstract

An example solution to multi-level 8-bit floating point (FP8) matrix multiplication for specific format conversion facilitates computational efficiency of matrix multiplication operations by a processor, including: copying data values in the FP8 format from global memory to shared memory; loading a thread block tile of the FP8 data value from the shared memory into a register set; converting each of the plurality of FP8 data values in the register set to a 16-bit floating point (FP16) data value; the FP16 data value is submitted to a tensor kernel; and performing a matrix multiply-accumulate calculation using the tensor kernel.

Description

Background Art

[0001] Matrix multiplication is used in a wide variety of computing tasks, such as image, text, and software code generation, in conjunction with machine learning (ML) models such as neural networks (NNs). Graphics processing units (GPUs) with tensor cores are examples of hardware that excel at performing matrix multiplication quickly.

[0002] However, hardware-optimized solutions are configured for specific data types (e.g., data formats) with data values ​​in a specific layout to maximize throughput. If data is submitted to the hardware-optimized solution in a different layout and / or in a different format, processing is delayed while the format is converted and / or the layout is shifted to an acceptable layout. This can occur with some degree of regularity when storage limitations or latency limitations for large matrices drive developers to use a data format with fewer bit fields than the data type accepted by the hardware-optimized matrix multiplication solution, or when upstream software writes the data in a different layout. Summary of the invention

[0003] The disclosed examples are described in detail below with reference to the accompanying drawings listed below. The following overview is provided to illustrate some of the examples disclosed herein. However, in the following sequence, the reference frames in the reference frame set are not meant to limit all examples to any specific configuration or sequence of operations.

[0004] An example solution for multi-stage 8-bit floating point (FP8) matrix multiplication with format conversion facilitates computational efficiency of matrix multiplication operations by a processor, including: copying data values ​​in FP8 format from a global memory to a shared memory; loading thread block tiles of the FP8 data values ​​from the shared memory into a register set; converting each of a plurality of FP8 data values ​​in the register set into a 16-bit floating point (FP16) data value; submitting the FP16 data values ​​to a tensor core; and performing a matrix multiply-accumulate (MMA) calculation using the tensor core. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed examples are described in detail below with reference to the following figures:

[0006] Figure 1 An example architecture is illustrated that provides multi-stage 8-bit floating point (FP8) matrix multiplication with format conversion and facilitates computational efficiency of matrix multiplication operations by a processor;

[0007] Figure 2 Provides an example graphical representation of matrix multiplication, including the Figure 1 some matrix multiply-accumulate (MMA) parameters used by the example architecture in ;

[0008] Figure 3Illustrated are example bit fields for 8-bit floating point numbers (FP8) and 16-bit floating point numbers (FP16), and the bit fields of an example architecture such as Figure 1 Example transformation scheme used in the example architecture in ;

[0009] Figure 4 The diagram shows an example architecture such as Figure 1 Example workflow view of FP8 to FP16 conversion and MMA calculations performed by ;

[0010] Figure 5 The diagram shows an example architecture such as Figure 1 Another example workflow view of an MMA computation performed by an example architecture in FIG.

[0011] Figure 6 Picture shows Figure 5 Additional details of the workflow view;

[0012] Figure 7 The diagram shows how to use the example architecture (such as Figure 1 Example timeline of events that occur when you use the sample schema in

[0013] Figure 8 shows that when using an example architecture such as Figure 1 Example data flows that occur concurrently when using the example architecture in

[0014] Fig. 9 The diagram shows an example architecture such as Figure 1 An example pipeline for concurrent execution of various events implemented using the example architecture in ;

[0015] Fig.10 Provides examples of how to use the architecture (such as Figure 1 Example graphical representation of the copying of data values ​​from global memory to shared memory that occurs when

[0016] Fig.11 Provides examples of how to use the architecture (such as Figure 1 An example graphical representation of loading a data value from shared memory into a register that occurs when

[0017] Fig.12 Provides a sample architecture such as Figure 1 ) will occur when Fig.11 Example graphical representation of the conversion of data values ​​in registers from FP8 to FP16;

[0018] Fig.13 shows how to use an example architecture such as Figure 1 A flowchart of exemplary operations that may be performed when using the example architecture in FIG.

[0019] Fig.14 Another flow chart is shown, which illustrates when using an example architecture (such as Figure 1 ) in the example architecture; and

[0020] Fig.15 A block diagram of an example computing device suitable for implementing some of the various examples disclosed herein is shown.

[0021] Corresponding reference numerals indicate corresponding parts throughout the drawings.Any of the drawings may be combined into a single embodiment. DETAILED DESCRIPTION

[0022] Various examples will be described in detail with reference to the accompanying drawings. In preferred cases, the same reference numerals will be used throughout the accompanying drawings to refer to the same or similar components. References to specific examples and implementations throughout this disclosure are for illustrative purposes only and are not intended to limit all examples unless otherwise specified.

[0023] An example solution for multi-stage 8-bit floating point (FP8) matrix multiplication with format conversion facilitates computational efficiency of matrix multiplication operations by a processor, including: copying FP8 format data values ​​from a global memory to a shared memory; loading a thread block tile of the FP8 data values ​​from the shared memory into a register set; converting each of a plurality of FP8 data values ​​in the register set into a 16-bit floating point (FP16) format data value; submitting the FP16 data values ​​to a tensor core; and performing a matrix multiply-accumulate (MMA) calculation using the tensor core.

[0024] Aspects of the present disclosure facilitate the operation of a computing device, for example, by increasing the speed of matrix multiplication, reducing the required memory, and reducing power consumption. The speed of matrix multiplication is increased by at least converting each of a plurality of FP8 data values ​​in a register set to an FP16 data value. Performing the conversion in the register before submitting to the tensor core can increase the speed compared to the conversion before storing the data value in the shared memory. By using FP8 data values ​​in global memory and shared memory, shifting the traditional memory / speed tradeoff, and reducing the size of any neural network (NN) or other machine learning (ML) component computed using MMA, a reduction in memory capacity and bandwidth is enabled. Electrical power consumption is reduced by at least using less memory and fewer processing cycles.

[0025] Figure 1An example architecture 100 is illustrated that provides multi-stage FP8 matrix multiplication with format conversion and facilitates computational efficiency of matrix multiplication operations by a processor. A computing platform 102 performs a matrix multiplication, C=A x B, to produce an output product 150, including a recommendation, such as a generated image, generated text, generated software code, or another output product that relies on the matrix multiplication. In some examples, several matrix multiplications and other operations are required between the multiplications to produce the output. In the architecture 100, the matrix multiplication is implemented as an MMA in a tensor core 140 within a graphics processing unit (GPU) 104.

[0026] Typically, GPUs are able to process multiple slices of data simultaneously, making them useful for ML, video editing, and gaming applications. GPUs use several different types of memory, including global memory, shared memory, and registers. These can be referred to as primary memory, secondary memory, and tertiary memory, respectively. Global memory tends to be the slowest, but is also usually the largest storage space. Shared memory is usually faster than global memory, but is more limited in size. Registers are faster still, although the capacity is usually strictly limited.

[0027] The tensor core 140 expects FP16 data values ​​in a specific layout. However, in the matrices to be multiplied (matrix A 111 and matrix B 112), at least one of the matrices already stores FP8 data values ​​to save memory space. In some examples, matrix A 111 is in FP16 format and matrix B 112 is in FP8 format. When the tensor core itself does not support FP8 matrix multiplication or the two matrices have different data formats, format conversion is required.

[0028] To increase the speed of matrix multiplication, MMA acceleration 132 performs FP8 to FP16 format conversion within register set 106 and arranges the layout of data values ​​extracted from matrix A 111 and matrix B 112 into the layout expected by tensor core 140. Format conversion is performed in register set 106, and the layout is rearranged and the data values ​​from matrix A 111 and matrix B 112 are submitted in the order expected by MMA (e.g., as thread warp tiles), which allows the MMA process in tensor core 140 to execute without bottlenecks. This significantly increases the speed of the matrix multiplication stage that produces output products 150.

[0029] When running ML models in a GPU or other hardware accelerator, matrix multiplication is the main type of operation. Transformer-based workloads that generate text or software code operate using a technique called autoregressive generation. During generation, each model forward step produces a new token. The model forward step has multiple matrix multiplication operations, which can be less computationally intensive individually in the token-by-token generation stage. The main speed bottleneck for these operations is loading the model weights from the global memory 108 to the shared memory 130 or register set 106.

[0030] The GPU 104 generates output products 150 by executing the task 110 using the task logic 120 (e.g., a software program that executes the task 110 to generate the output products 150). The task logic 120 includes an ML model 122, which may include a NN. Executing the task 110 requires multiplying the matrix A 111 and the matrix B 112 to calculate the matrix C 113. The task logic 120 and the data of the task 110 are stored in the global memory 108, which may include a high bandwidth memory (HBM), such as a second generation high bandwidth memory (HBM2) or a third generation high bandwidth memory (HBM3). In some examples, the global memory 108 is not located in the same integrated circuit (IC) package as the tensor core 140.

[0031] In order to save space in global memory 108, matrix A 111 or matrix B 112 or both adopt FP8 format. Because FP8 format only requires 8 bits, FP16 format requires 16 bits, and other floating point formats require 32 or 64 bits, it may be preferred to use FP8 when the required numerical precision and value range allow. Storing data with lower precision (e.g., FP8 format) also saves the cost of data loading between global memory 108 and shared memory 130. Unfortunately, some GPUs do not support FP8 in MMA. Before storing the data value in shared memory 130, converting the data value from FP8 format to FP16 format may prevent the use of some optimized asynchronous copy operations provided by common GPUs, and may also limit data loading parallelism. In some examples, shared memory 130 is located in the same IC package as tensor core 140 and is a memory type faster than global memory 108.

[0032] Figure 1 Matrix A 111 or matrix B 112 is illustrated in global memory 108, although in some examples, only one of matrix A 111 and matrix B 112 is in global memory 108. For example, matrix B 112 may be stored in global memory in FP8, while matrix A 112 may be generated by previous calculations and stored in shared memory 103.

[0033] The data values ​​of matrix A 111 and / or matrix B 112 (e.g., Figure 4 The data values ​​402a-402d) are copied from the global memory 108 to the shared memory 130 and loaded into the register set 106 in its original format. The MMA acceleration 132 manages the FP8 to FP16 format conversion while the data values ​​are still in the register set 106, and also arranges the layout of the data values ​​according to the layout expected by the tensor core 140. The data values ​​are then provided to the tensor core 140 already in FP16 format and in the appropriate layout. In some examples, the GPU 104 has multiple tensor cores 140. In some examples, the register set 106 is located in the same IC package as the tensor core 140 and is a faster memory type than the shared memory 130.

[0034] The tensor core 140 performs MMA on the submitted data values ​​to compute an MMA result 144. In some examples, the MMA result 144 is in FP32, and the tensor core 140 performs accumulation in FP32 as part of the MMA process. The MMA result 144 is returned to the register set 106 (or to a different register set in some examples) and is optionally converted back to FP16 or FP8 format. The MMA result 144 is then copied back to the shared memory 130 and then copied back to the global memory 108 as the data value of the matrix C 113.

[0035] In some examples, moving data values ​​between global memory 108 and shared memory 130 uses asynchronous copy operations. Asynchronous copy (non-synchronous copy) transfers data between memory locations under the control of hardware so that software on the GPU can proceed concurrently with other work while the copy is completed. The use of asynchronous copy operations allows for further speed improvements through various aspects of the present disclosure. The use of FP8 and FP16 further reduces the copy time between global memory 108 and shared memory 130 because fewer bits are transferred.

[0036] Figure 2A graphical representation 200 of a matrix multiplication is provided, including some MMA parameters used by the architecture 100. The illustrated matrix multiplication solves C=AxB, where C will become matrix C 113, and A and B represent data values ​​extracted from matrix A 111 and matrix B 112, respectively. Matrix A 111 is illustrated as having a plurality of thread block tiles, specifically thread block tile 201, thread block tile 202, thread block tile 203, and thread block tile 204. Matrix B 112 is also illustrated as having a plurality of thread block tiles, specifically thread block tile 211, thread block tile 212, thread block tile 213, and thread block tile 214. The relative orientation of the thread block tiles (primarily vertical for matrix A 111 and primarily horizontal for matrix B 112) reflects the indexing progression of the respective matrices in the matrix multiplication operation. The thread block tiles in the illustrated example are 128 bytes wide, although this is configurable and other sizes may be used in some examples. Also shown are the thread warps 116 of the matrix A 111. Figure 5 Describes thread block tiles and warps.

[0037] Figure 3 The bit fields of FP8 and FP16 numbers, and the conversion scheme 300 used by the architecture 100 are illustrated. The FP8 number 308 has a single sign bit, denoted as S, an exponent field, denoted as E, with four exponent bits, and a mantissa segment, denoted as M, with three mantissa bits. The FP16 number 316 has a single sign bit, an exponent field with five exponent bits, and a mantissa segment with ten mantissa bits. The conversion fills the exponent field of the FP16 number 316 with leading zeros, followed by the four exponent bits of the FP8 number 308. For the mantissa, the conversion copies the three mantissa bits of the FP8 number 308, and then fills the remaining mantissa segment of the FP16 number 316 with seven zeros.

[0038] Figure 4 The workflow 400 of FP8 to FP16 conversion and MMA calculation performed by the architecture 100 is illustrated. The global memory 40 has multiple data values ​​in FP8 format. Data value 402a, data value 402b, data value 402c, and data value 402d are shown. These data values ​​are copied from the global memory 108 to the shared memory 130. Some examples use asynchronous replication. The data values ​​are loaded into the register set 106 from the shared memory 130. The MMA acceleration 132 converts the data values ​​from the FP8 format to the FP16 format and permutes (shifts) them as needed to place them in the expected layout of the tensor core 140. Data values ​​404a and data value 404b are converted from data values ​​402a and 402c in FP16, respectively. Data values ​​402b and 404d are mapped to two additional numbers stored in another register (not shown). It should be understood that more data values ​​are used in some examples.

[0039] Data values ​​(e.g., data values ​​404a and 404b) are submitted to the tensor core 140 and are stored in the register set 106 while the MMA logic 408 performs MMA processing (e.g., MMA calculations) on them. This produces an MMA result 144 as an FP32 data value within the register set 106. The tensor core 140 can read directly from and be written to the register set 106. Data value 414a and data value 414b are shown. In some examples, the data values ​​in the register set 106 are then converted to FP16 or FP8 format and copied to the shared memory 130, and then some may be copied to the global memory 108. In some examples, asynchronous copying is used to move to the global memory 108.

[0040] Figure 5 The workflow 500 of an MMA computation performed by the architecture 100 is illustrated. The architecture 100 applies a tiling structure to efficiently implement general matrix multiplication (GEMM) by decomposing the computation into a hierarchy of thread block tiles, warp tiles, and thread tiles, and then accumulating matrix products. Stage 502 shows a block-based GEMM in global memory 108, which has matrix A 111, matrix B 112, and matrix C 113, and shows a single thread block in each matrix. In some examples, only one of matrix A 111 and matrix B 112 is in global memory 108, and the other matrix is ​​small enough to be in shared memory 130. Similarly, depending on the size of the output, matrix C 113 can be small enough to remain in shared memory 130.

[0041] The diagram illustrates data movement from global memory 108 to shared memory 130 (matrix to thread block tile), from shared memory 130 to register files in register sets 106 (thread block tile to warp tile), and from register files to tensor cores 140 for computation (warp tile to thread tile).

[0042] Each thread block computes its GEMM output portion by iteratively loading blocks of matrix data from the input matrix and computing cumulative matrix products. The thread block tile structure is further divided into warps (groups of threads that execute together). Warps provide organization for GEMM computations.

[0043] A thread block tile 504 and a subsequent level of thread warp tiles 506 are shown. Figure 6 The thread block tile 504 and the warp tile 506 are described in more detail. The data values ​​in the warp tile 506 are distributed among each thread without duplication. The various threads cooperate to compute matrix multiplications in the tensor core 140, which reduces register traffic.

[0044] Figure 6 Additional details of workflow 500 are illustrated. After the data is stored in shared memory 130, each warp computes a sequence of cumulative matrix products by iterating the dimensions of the thread block tile, loading sub-matrices (or fragments) from shared memory, and computing cumulative outer products. In some examples, the size of the fragment is small to maximize the computational intensity relative to the amount of data loaded from the shared memory. This prevents the bandwidth of shared memory 130 from becoming a bottleneck. A tile 602 has fragment 604, and B tile 606 has fragment 608 corresponding to warp tile 506. In some examples, fragment 604 is a 16×16 matrix, and fragment 608 is a 16×8 matrix. The 16×2 and 2×16 representations are for illustration purposes only.

[0045] Figure 7 A timeline 700 of events occurring in the architecture 100 is illustrated. For the thread block tile 201, the first event is a pointer calculation 710, followed by a copy operation 712 from the global memory 108. This is followed by storing 714 the data value in the shared memory 130, followed by loading 716 the data value into the register set 106, and finally an MMA calculation 718. In some examples, when an asynchronous copy is used, operations 712 and 714 are combined. A similar sequence of events also occurs for the thread block tile 202. The first event is a pointer calculation 720, followed by a copy 722 from the global memory 108. This is followed by storing 724 the data value in the shared memory 130, followed by loading 726 the data value into the register set 106, and finally an MMA calculation 728. The sequence of events continues for other thread block tiles, for example, pointer calculation 730 for the thread block tile 203.

[0046] Figure 8 The data flow that occurs concurrently with the architecture 100 is illustrated. In stage 802, multiple thread block tiles are copied from the global memory 108 to the shared memory 130 at a time. In some examples, three thread block tiles are copied simultaneously. In some examples, a different number is copied simultaneously. The number of thread block tiles that are copied concurrently (simultaneously) may depend on the amount of available memory and the size of the matrix.

[0047] In stage 804, data values ​​are loaded from shared memory 130 to register set 106. In stage 806, data values ​​are submitted to tensor core 140 for MMA calculation. As shown, stages 802, 804, and 806 occur concurrently with the data passing through these stages. As shown, the data values ​​in thread block tile 201 first pass through stage 802, stage 804, stage 806, and then the data values ​​in thread block tile 202. When the data from thread block tile 201 that has passed through stages 802 and 804 is already in stage 806, some data from thread block tiles 202, 203, and 204 may still be in stage 802.

[0048] The portion shown is shown in the main loop body 808, which repeats (after the initial pipeline fill) until the pipeline is purged when the matrix multiplication is complete. A wait period 810 is shown after the data load of thread block tile 201 has completed, while data values ​​from thread block tile 202 are being loaded. As shown by the shading, all blocks illustrated in stage 806 are from thread block tile 201, and all blocks illustrated in stage 804 are from thread block tile 201, except for the last block. The final block illustrated in stage 804 is from thread block tile 202.

[0049] Fig. 9 A pipeline 900 for concurrently executing various events implemented by architecture 100 is illustrated. Fig. 9 An alternative representation of the concurrency of operations in architecture 100 is provided. Three stages are shown. Fill pipeline stage 902, steady state 904 (roughly corresponding to main loop body 808), and flush pipeline stage 906. For simplicity, only the operations of two thread block tiles are shown.

[0050] Operation 910 copies a first thread block tile (e.g., thread block tile 201). Next are four load operations for data values ​​from the first thread block tile: load operation 912a, load operation 912b, load operation 912c, and load operation 912d. After loading, the data values ​​may be transformed. The data values ​​loaded in load operation 912a are transformed in transform operation 914a and submitted to the tensor core 140 for MMA calculation 916a.

[0051] This scheme allows different data values ​​to be in different processes of pipeline 900 concurrently. For example, a set of data values ​​undergoes load operation 912c, conversion operation 914c, and MMA calculation 916c. Another set of data values ​​undergoes load operation 912d, conversion operation 914d, and MMA calculation 916d.

[0052] Operation 920 copies a second thread block tile (e.g., thread block tile 202). A set of data values ​​from the second thread block tile is subjected to a load operation 922a, a conversion operation 924a, and an MMA calculation 926a. Another set of data values ​​is subjected to a load operation 922b, a conversion operation 924b, and an MMA calculation 926b. Another set of data values ​​is subjected to a load operation 922c, a conversion operation 924c, and an MMA calculation 926c. Another set of data values ​​is subjected to a load operation 922d, a conversion operation 924d, and an MMA calculation 926d.

[0053] The stable state 904 exists from the time a data value is sent to the tensor core 140 until no more data values ​​are sent to the tensor core 140. In some embodiments, the FP8 to FP16 format conversion occurs simultaneously with the MMA calculations 916a-d and 926a-d. In some examples, there are 7 concurrent operating stages in the pipeline 900 at the same time.

[0054] After the steady state 904 , during the flush pipeline stage 906 , the data values ​​may be copied to the shared memory 130 and may be cumulatively retained in the registers.

[0055] Fig.10 A graphical representation of copying data values ​​from global memory 108 to shared memory 130 is provided. Each location in memory representation grids 1008 and 1030 is 128 bits, which holds 16 FP8 data values. Memory representation grid 1008 reflects the data value locations in global memory 108. Memory representation grid 1030 reflects the data value locations in shared memory 130. As can be seen, there is an offset in the data value locations between memory representation grids 1008 and 1030.

[0056] At each stage, a row of data values ​​in the memory representation grids 1008 and 1030 is read. The counter for the read stage is indicated to the right of the memory representation grid 1030. Each thread needs to process consecutive values ​​when reading from or writing to the global memory 108. When reading from or writing to the shared memory 130, the threads in the warp only need to read a 4-byte data from any memory bank in one stage.

[0057] Fig.11A graphical representation of loading data values ​​from shared memory 130 into register set 106 is provided. Each location in memory representation grid 1130 is 128 bits, which holds 16 FP8 data values ​​or 8 FP16 data values. Each location in memory representation grid 1106 is 32 bits, which holds four FP8 data values ​​or two FP16 data values. Memory representation grid 1130 reflects the data value locations in shared memory 130. Memory representation grid 1106 reflects the data value locations in register set 106. The first 16 threads in each warp load 128 bits from shared memory 130 and are distributed to 32 threads. Each thread places eight FP8 data values ​​into two registers. As can be seen, the data value locations are different between memory representation grids 1130 and 1106.

[0058] Registers 1102 in register set 106 are shown as holding four FP8 data values, namely FP8 data value E0, FP8 data value E1, FP8 data value E2, and FP8 data value E3. Because each register in register set 106 is 32 bits (in some examples), when the data is in FP8 format, a single register stores four FP8 data values. In some examples, tensor core 140 has layout requirements that differ from the pattern shown in memory representation grid 1106.

[0059] Fig.12 A graphical representation of converting data values ​​in register set 106 from FP8 to FP16 is provided. Fig.11 As shown, when the data value is in FP8 format, the memory representation grid 1106 reflects the data value location in the register set 106. When the data value is in FP16 format, the memory representation grid 1206 reflects the data value location in the register set 106. FP8 data value E0 is converted to FP16 data value H0; FP8 data value E2 is converted to FP16 data value H1; FP8 data value E1 is converted to FP16 data value H2; FP8 data value E3 is converted to FP16 data value H3.

[0060] After conversion, each register includes two FP16 data values. 32 bits can be occupied as {8 bits, 8 bits, 8 bits, 8 bits} or {16 bits, 16 bits}. Register 1102 holds all four FP8 data values ​​E0-E3. After conversion (and permutation / shift), register 1202 holds FP16 data values ​​H0 and H1, and register 1204 holds FP16 data values ​​H2 and H3. In some examples, the dual MMA core compute memory represents the odd and even columns of grid 1206.

[0061] Fig.131300 is shown, which illustrates exemplary operations that may be performed by architecture 100. In some examples, the operations described with respect to flowchart 1300 are performed by Fig.15 The flowchart 1300 begins with the architecture 100 initiating a computing project in operation 1302, which may include generating an image, generating text, generating software code, or another ML task involving matrix multiplication or other project.

[0062] In operation 1304, the architecture 100 generates matrix A 111 and matrix B 112 for multiplication in global memory 108. Operation 1306 performs a matrix multiplication, specifically MMA, of matrix A 111 and matrix B using operations 1308-1330 to compute matrix C 113. Operation 1308 copies data values ​​(e.g., data values ​​402a and 402b) in FP8 format from global memory 108 to shared memory 130. In some examples, copying data values ​​in FP8 format data from global memory 108 to shared memory 130 includes performing an asynchronous copy operation.

[0063] Operation 1310 loads a thread block tile of FP8 data values ​​(e.g., three thread block tiles, such as thread block tiles 201-203) from shared memory 130 into register set 106. In some examples, each register in register set 106 comprises a 32-bit register. In some examples, loading the thread block tile of FP8 data values ​​from shared memory 130 into register set 106 comprises loading four data values ​​in FP8 format data from shared memory 130FP8 into a single register in register set 106. During steady state operation, loading the thread block tile of FP8 data values ​​from shared memory 130 into register set 106 occurs while performing MMA calculations on previously converted data values.

[0064] Operation 1312 converts each of the plurality of FP8 data values ​​in register set 106 to an FP16 data value before submitting the data value to tensor core 140. Operation 1314 also shifts (permutes) the data position of the FP16 data value to a layout accepted by tensor core 142 before submitting the data value to tensor core 140. In some examples, operations 1312 and 1314 are performed simultaneously, such that shifting the data position of the FP16 data value to a layout accepted by tensor core 140 occurs simultaneously with converting each of the plurality of FP8 data values ​​to an FP16 data value.

[0065] Operation 1316 submits the FP16 data value to the tensor core 140, and in operation 1318, the tensor core 140 performs the MMA calculation using the FP16 data value. After performing the MMA calculation, the tensor core 140 returns the MMA result 144 (e.g., the result of the MMA calculation) in operation 1320. In some examples, the MMA result 144 includes the FP32 data value. Operation 1322 copies the MMA result 144 to the register set 106.

[0066] Optional operation 1324 converts the MMA result 144 in the register set 106 from FP32 format to FP16 format or FP8 format. Operation 1326 copies the MMA result 144 from the register set 106 to the shared memory 130. Operation 1328 stores the MMA result 144 to the global memory 108. Decision operation 1330 determines whether the matrix multiplication of operation 1306 is complete. If not, the flowchart 1300 returns to operation 1308 to continue copying data values ​​from the global memory 108 to the shared memory 130, loading the data values ​​into the register set 106, converting the data value format, and submitting the data values ​​to the tensor core until the MMA calculation for the matrix in the global memory 108 is completed.

[0067] Otherwise, operation 1332 completes the computing task 110 to generate an output (output product 150), such as generating an image, generating text, or generating software code by computing using MMA.

[0068] Fig.14 1400 is shown, which illustrates exemplary operations that may be performed by architecture 100. In some examples, the operations described with respect to flowchart 1400 are performed by Fig.15 The flowchart 1400 begins at operation 1402, which includes copying a data value in a first floating point format from a global memory to a shared memory.

[0069] Operation 1404 includes loading a thread block tile of a first floating point data value from a shared memory into a register set. Operation 1406 includes converting the first floating point data value in the register set to a second floating point data value in a second floating point format. Operation 1408 includes submitting the second floating point data value to a tensor core. Operation 1410 includes performing an MMA calculation using the tensor core. Operation 1412 includes generating an output using the MMA calculation.

[0070] Additional Examples

[0071] An example system includes: a processor; and a computer-readable medium storing instructions that, when executed by the processor, operate to: copy a data value in a first floating point format from a global memory to a shared memory; load a thread block tile of the first floating point data value from the shared memory into a register set; convert the first floating point data value in the register set to a second floating point data value in a second floating point format; submit the second floating point data value to a tensor core; perform an MMA calculation using the tensor core; and generate an output using the MMA calculation.

[0072] An example method includes: asynchronously copying a data value in a first floating point format from a global memory to a shared memory; loading a thread block tile of the first floating point data value from the shared memory into a register set; converting the first floating point data value in the register set to a second floating point data value in a second floating point format; submitting the second floating point data value to a tensor core; performing an MMA calculation using the tensor core; and generating an output using the MMA calculation.

[0073] One or more example computer storage devices have computer executable instructions stored thereon that, when executed by a computer, cause the computer to perform operations including: copying a data value in a first floating point format from a global memory to a shared memory; loading a thread block tile of the first floating point data value from the shared memory into a register set while performing a matrix multiply-accumulate (MMA) calculation on the previously converted data value; converting the first floating point data value in the register set to a second floating point data value in a second floating point format; submitting the second floating point data value to a tensor core; performing the MMA calculation using the tensor core; and generating an output using the MMA calculation.

[0074] As an alternative or in addition to other examples described herein, examples include any combination of the following:

[0075] - the first floating point format includes FP8;

[0076] - the second floating point format includes FP16;

[0077] -copying the data value in the first floating point format from the global memory to the shared memory comprises performing an asynchronous copy operation;

[0078] -Global memory includes HBM;

[0079] -Global memory includes HBM2;

[0080] - Each register in the register set comprises a 32-bit register;

[0081] - loading the thread block tile of the first floating point format data values ​​from the shared memory into the register set comprises loading four data values ​​of the first floating point format from the shared memory

[0082] Load into a single register in a register set;

[0083] - loading the thread block tile of the first floating point format data values ​​from the shared memory into the register set while performing the MMA calculation on the previously converted data values;

[0084] - Before submitting the data value to the Tensor Core, convert the data of the second floating point format data value to

[0085] Position shifting to the layout accepted by the tensor core;

[0086] - concurrently with converting each first floating point format data value to a second floating point format data value, shifting data positions of the second floating point format data values ​​to a layout accepted by the tensor core;

[0087] -Tensor cores are inside the GPU;

[0088] - performing an MMA calculation using the second floating point format data value;

[0089] -After performing the MMA calculation, the MMA result of the MMA calculation is returned through the Tensor Core;

[0090] -MMA results include FP32 data values;

[0091] - Load the MMA result into the register set;

[0092] -converting the MMA result in the register set to a second floating point format data value;

[0093] -converting the MMA result in the register set to a first floating point format data value;

[0094] -Copy the MMA results to shared memory;

[0095] -Store the MMA result to global memory;

[0096] - continue to copy data values ​​from global memory to shared memory, load data values ​​into register sets, convert data value formats, and submit data values ​​to tensor cores until the MMA computation for the matrix in global memory is complete;

[0097] - Generate images using MMA calculations;

[0098] - generate text computationally using MMA; and

[0099] - Generate software code using MMA calculations.

[0100] Although aspects of the present disclosure have been described in terms of various examples and their related operations, those skilled in the art will appreciate that combinations of operations from any number of different examples are also within the scope of aspects of the present disclosure.

[0101] Example operating environment

[0102] Fig.15 1 is a block diagram of an example computing device 1500 (e.g., a computer storage device) for implementing aspects disclosed herein, and is generally designated as computing device 1500. In some examples, one or more computing devices 1500 are provided for a local computing solution. In some examples, one or more computing devices 1500 are provided as a cloud computing solution. In some examples, a combination of local and cloud computing solutions are used. Computing device 1500 is only one example of a suitable computing environment and is not intended to suggest any limitation on the scope of use or functionality of the examples disclosed herein, whether used alone or as part of a larger collection.

[0103] Computing device 1500 should also not be interpreted as having any dependency or requirement on any one or combination of the components / modules shown. The examples disclosed herein can be described in the general context of computer code or machine-usable instructions, including computer executable instructions executed by a computer or other machine (such as a personal data assistant or other handheld device), such as program components. Typically, program components (including routines, programs, objects, components, data structures, etc.) refer to codes that perform specific tasks or implement specific abstract data types. The disclosed examples can be practiced in various system configurations, including personal computers, laptops, smart phones, mobile tablets, handheld devices, consumer electronics, professional computing devices, etc. When tasks are performed by remote processing devices linked by a communication network, the disclosed examples can also be implemented in a distributed computing environment.

[0104] The computing device 1500 includes a bus 1510 that directly or indirectly couples the following devices: computer storage memory 1512, one or more processors 1514, one or more presentation components 1516, input / output (I / O) ports 1518, I / O components 1520, power supply 1522, and network components 1524. Although the computing device 1500 is depicted as appearing to be a single device, multiple computing devices 1500 can work together and share the depicted device resources. For example, the memory 1512 can be distributed across multiple devices, and the processor(s) 1514 can be housed in different devices.

[0105] Bus 1510 may represent one or more buses (such as an address bus, a data bus, or a combination thereof). Although for clarity, Fig.15The various blocks of the are shown with lines, but demarcating the various components may be implemented with alternative representations. For example, in some examples, presentation components such as display devices are I / O components, and some examples of processors have their own memory. Classifications such as "workstation," "server," "laptop," "handheld device," etc. are not distinguished, as all of these categories are in Fig.15 1512a and instructions 1512b that are executable by the processor 1514 and are configured to perform various operations disclosed herein.

[0106] In some examples, memory 1512 includes computer storage media. Memory 1512 may include any number of memories associated with or accessible by computing device 1500. Memory 1512 may be internal to computing device 1500 (e.g., Fig.15 1500), external to computing device 1500 (not shown), or both (not shown). Additionally or alternatively, memory 1512 may be distributed across multiple computing devices 1500, for example, in a virtualized environment where instruction processing is performed on multiple computer devices 1500. For purposes of this disclosure, "computer storage media," "computer storage memory," "memory," and "memory device" are synonymous with computer storage memory 1512, and none of these terms include carrier waves or propagated signals.

[0107] (Multiple) processors 1514 may include any number of processing units that read data from various entities (such as memory 1512 or I / O components 1520). Specifically, (multiple) processors 1514 are programmed to execute computer executable instructions for implementing various aspects of the present disclosure. The instructions can be executed by a processor, multiple processors within computing device 1500, or a processor external to client computing device 1500. In some examples, (multiple) processors 1514 are programmed to execute instructions, such as those shown in the flowcharts discussed below and shown in the accompanying drawings. In addition, in some examples, (multiple) processors 1514 represent implementations of simulation techniques for performing the operations described herein. For example, the operations can be performed by simulating client computing device 1500 and / or digital client computing device 1500. (Multiple) presentation components 1516 present data indications to a user or other device. Exemplary presentation components include display devices, speakers, printing components, vibration components, etc. Those skilled in the art will understand and appreciate that computer data can be presented in a variety of ways, such as visually in a graphical user interface (GUI), audibly through a speaker, wirelessly between computing devices 1500, through a wired connection, or in other ways. I / O ports 1518 allow computing device 1500 to be logically coupled to other devices, including I / O components 1520, some of which may be built-in. Example I / O components 1520 include, but are not limited to, microphones, joysticks, game controllers, satellite dishes, scanners, printers, wireless devices, and the like.

[0108] The computing device 1500 can operate in a networked environment via a network component 1524 using a logical connection to one or more remote computers. In some examples, the network component 1524 includes a network interface card and / or a computer executable instruction (e.g., a driver) for operating the network interface card. The communication between the computing device 1500 and other devices can be performed using any protocol or mechanism on any wired or wireless connection. In some examples, the network component 1524 operates to communicate data between devices wirelessly using a transmission protocol through public, private or mixed (public and private) communication data, using short-range communication technology (e.g., near field communication (NFC), Bluetooth TM brand communication, etc.), or a combination thereof. The network component 1524 communicates with a remote resource 1528 (e.g., a cloud resource) through a network 1530 via a wireless communication link 1526 and / or a wired communication link 1526a. Various different examples of communication links 1526 and 1526a include wireless connections, wired connections, and / or dedicated links, and in some examples, at least a portion is routed through the Internet.

[0109] Although described in conjunction with the example computing device 1500, the examples of the present disclosure can be implemented with many other general or special computing system environments, configurations or devices. Examples of known computing systems, environments and / or configurations suitable for various aspects of the present disclosure include, but are not limited to, smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, wearable or accessory mobile computing and / or communication devices (e.g., watches, glasses, headphones or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality devices, holographic devices, etc. Such systems or devices can accept input from users in any manner, including from input devices such as keyboards or pointing devices, via gesture input, proximity input (such as by hovering) and / or via voice input.

[0110] Examples of the present disclosure can be described in the general context of computer executable instructions (such as program modules), which are executed by one or more computers or other devices in software, firmware, hardware or a combination thereof. Computer executable instructions can be organized into one or more computer executable components or modules. Typically, program modules include but are not limited to routines, programs, objects, components and data structures that perform specific tasks or implement specific abstract data types. Various aspects of the present disclosure can be implemented with such components or modules of any number and organization. For example, various aspects of the present disclosure are not limited to specific computer executable instructions or specific components or modules shown in the figures and described herein. Other examples of the present disclosure may include different computer executable instructions or components with more or less functions than shown and described herein. In an example involving a general-purpose computer, when configured to execute the instructions described herein, various aspects of the present disclosure convert the general-purpose computer into a special-purpose computing device.

[0111] As an example and not limitation, computer-readable media include computer storage media and communication media. Computer storage media include volatile and non-volatile removable and non-removable memory for storing information such as computer-readable instructions, data structures, program modules, etc. implemented in any method or technology. Computer storage media are tangible and mutually exclusive with communication media. Computer storage media are implemented in hardware and do not include carrier waves and propagation signals. For the purposes of this disclosure, computer storage media themselves are not signals. Exemplary computer storage media include hard disks, flash drives, solid-state memories, phase change random access memories (PRAM), static random access memories (SRAM), dynamic random access memories (DRAM), other types of random access memories (RAM), read-only memories (ROM), electrically erasable programmable read-only memories (EEPROM), flash memory or other memory technologies, compact disk read-only memories (CD-ROM), digital versatile disks (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computing device. In contrast, communication media typically embodies computer readable instructions, data structures, program modules, etc. in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0112] The execution or execution order of the operations in the examples of the present disclosure shown and described herein is not necessary, and can be executed in different sequential ways in various examples. For example, it is conceivable that performing a specific operation before, concurrently with, or after another operation is within the scope of the various aspects of the present disclosure. When introducing the elements of various aspects of the present disclosure or its examples, the articles "a", "an", "the" and "said" are intended to represent the presence of one or more elements. The terms "comprising", "including" and "having" are intended to be inclusive, and represent that in addition to the listed elements, there may be other elements. The term "exemplary" is intended to represent "an example of...". The phrase "one or more of the following: A, B and C" refers to "at least one of A and / or at least one of B and / or at least one of C".

[0113] Having described in detail various aspects of the disclosure, it will be apparent that modifications and variations may be made without departing from the scope of the various aspects of the disclosure as defined in the appended claims. As various changes may be made to the above-described constructions, products, and methods without departing from the scope of the various aspects of the disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not limiting.

Claims

1. A system comprising: processor; as well as A computer readable medium storing instructions that, when executed by the processor, operate to: copying the data value in the first floating point format from the global memory to the shared memory; loading a thread block tile of the first floating point data value from the shared memory into a set of registers; converting the first floating point data value in the register set to a second floating point data value in a second floating point format; submitting the second floating point data value to the tensor core; Performing a matrix multiply-accumulate (MMA) computation using the tensor core; as well as Recommendations are generated using the MMA calculations.

2. The system according to claim 1, wherein the first floating point format comprises an 8-bit floating point (FP8) format; wherein the second floating point format comprises a 16-bit floating point (FP16) format; and The result of the MMA calculation includes a 16-bit FP16 data value or a 32-bit floating point (FP32) data value.

3. The system of any one of claims 1 to 2, wherein loading the thread block tile of the first floating point data values ​​from the shared memory into the register set comprises loading four data values ​​in the first floating point format from the shared memory into a single register in the register set.

4. The system of claim 3, wherein the instructions further operate to: The thread block tile of the first floating point data value is loaded from the shared memory into the set of registers while performing an MMA calculation on the previously converted data value.

5. The system of any one of claims 1 to 4, wherein the instructions further operate to: Prior to submitting the data value to the tensor core, concurrently with converting the first floating point data value to the second floating point data value, shifting a data position of the second floating point data value to a layout accepted by the tensor core.

6. The system of any one of claims 1 to 5, wherein the instructions further operate to: The data values ​​are copied from the global memory to the shared memory, loaded into the register set, formatted, and submitted to the tensor core until the MMA computation for the matrix in the global memory is completed.

7. The system of any one of claims 1 to 6, wherein generating the recommendation comprises: Generate an image using the MMA calculation; Generate text using the MMA calculation; or Software code was generated using the MMA calculations described.

8. A method comprising: asynchronously copying data values ​​in a first floating point format from a global memory to a shared memory; loading a thread block tile of the first floating point data value from the shared memory into a set of registers; converting the first floating point data value in the register set to a second floating point data value in a second floating point format; submitting the second floating point data value to the tensor core; Performing a matrix multiply-accumulate (MMA) computation using the tensor core; as well as Output is generated using the MMA calculation.

9. The method according to claim 8, wherein the first floating point format comprises an 8-bit floating point (FP8) format; wherein the second floating point format comprises a 16-bit floating point (FP16) format; and The result of the MMA calculation includes a 16-bit FP16 data value or a 32-bit floating point (FP32) data value.

10. The method of any one of claims 8 to 9, wherein loading the thread block tile of the first floating point data values ​​from the shared memory into the register set comprises loading four data values ​​in the first floating point format from the shared memory into a single register in the register set.

11. The method according to claim 10, further comprising: The thread block tile of the first floating point data value is loaded from the shared memory into the set of registers while performing an MMA calculation on the previously converted data value.

12. The method according to any one of claims 8 to 11, further comprising: Prior to submitting the data value to the tensor core, concurrently with converting the first floating point data value to the second floating point data value, shifting a data position of the second floating point data value to a layout accepted by the tensor core.

13. The method according to any one of claims 8 to 12, further comprising: The data values ​​are copied from the global memory to the shared memory, loaded into the register set, formatted, and submitted to the tensor core until the MMA computation for the matrix in the global memory is completed.

14. The method of any one of claims 8 to 13, wherein generating the output comprises: Generate an image using the MMA calculation; Generate text using the MMA calculation; or Software code was generated using the MMA calculations described.

15. One or more computer storage devices having stored thereon computer-executable instructions which, when executed by a computer, cause the computer to perform operations comprising: copying the data value in the first floating point format from the global memory to the shared memory; loading a thread block tile of the first floating point data value from the shared memory into a set of registers while performing a matrix multiply-accumulate (MMA) calculation on the previously converted data value; converting the first floating point data value in the register set to a second floating point data value in a second floating point format; submitting the second floating point data value to the tensor core; performing MMA calculations using the tensor core; as well as Output is generated using the MMA calculation.

16. One or more computer storage devices according to claim 15, wherein the first floating point format comprises an 8-bit floating point (FP8) format; wherein the second floating point format comprises a 16-bit floating point (FP16) format; and The result of the MMA calculation includes a 16-bit FP16 data value or a 32-bit floating point (FP32) data value.

17. One or more computer storage devices according to any one of claims 15 to 16, wherein loading the thread block tile of the first floating point data value from the shared memory into the register set includes loading four data values ​​in the first floating point format from the shared memory into a single register in the register set.

18. One or more computer storage devices according to any one of claims 15 to 17, wherein the operations further comprise: Prior to submitting the data value to the tensor core, concurrently with converting the first floating point data value to the second floating point data value, shifting a data position of the second floating point data value to a layout accepted by the tensor core.

19. One or more computer storage devices according to any one of claims 15 to 18, wherein the operations further comprise: The data values ​​are copied from the global memory to the shared memory, loaded into the register set, formatted, and submitted to the tensor core until the MMA computation for the matrix in the global memory is completed.

20. The one or more computer storage devices of any one of claims 15 to 19, wherein generating the output comprises: Generate an image using the MMA calculation; Generate text using the MMA calculation; or Software code was generated using the MMA calculations described.

Images