Method, computer-readable medium, and processor for performing matrix product and accumulation operations

Abstract

A method, computer-readable medium, and processor for performing a matrix multiply and accumulate (MMA) operation are disclosed. The processor includes a data path configured to perform the MMA operation to generate a plurality of elements of a result matrix at an output of the data path. Each element of the result matrix is generated by computing at least one dot product of a corresponding pair of vectors associated with a matrix operand specified in an instruction for the MMA operation. The dot product operation includes the steps of generating a plurality of partial products by multiplying each element of a first vector by a corresponding element of a second vector; aligning the plurality of partial products based on an exponent associated with each element of the first vector and each element of the second vector; and accumulating the plurality of aligned partial products into a result queue using at least one adder.

Description

[0001] This application is a divisional application. The Chinese national application number of the original application is 201810425869.9, the application date is May 7, 2018, and the name of the invention is “Generalized acceleration of matrix multiplication and accumulation operations”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 62 / 503,159, filed May 8, 2017, entitled “Generalized Acceleration of Matrix Multiply Accumulate Operations” (Attorney Docket No. NVIDP1157+), which is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates to implementing arithmetic operations on a processor, and more particularly to the acceleration of matrix multiply-accumulate operations. Background Art

[0005] Modern computer processors are essentially integrated circuits designed to perform logical tasks. One task that processors are particularly good at is performing arithmetic operations on numbers encoded in different formats (e.g., 8-bit integers, 32-bit integers, 32-bit floating-point values, etc.). However, most processors include logic for performing these arithmetic operations on scalar operands. For example, logic designed to perform an addition operation is designed to perform the operation using two different operands, each encoding a specific value to be added to the other operand. However, arithmetic operations are not limited to scalar values. In fact, many applications may use arithmetic operations on vector or matrix inputs. An example of an arithmetic operation performed on vectors is the dot product operation. Although computing dot products is common in these applications (e.g., physics), modern processors are generally not designed with the hardware in their circuits to efficiently perform these operations. Instead, using scalar values simplifies higher-level operations into a series of basic arithmetic operations. For example, in a dot product operation, each vector operand includes multiple elements, and the dot product operation is performed by multiplying corresponding pairs of elements of two input vectors to generate multiple partial products (i.e., intermediate results) and then summing the multiple partial products. Each basic arithmetic operation can be performed sequentially using hardware logic designed into the processor, and the intermediate results can be stored in a temporary memory and reused as an operand for another subsequent arithmetic operation.

[0006] Conventional processors include one or more cores, each of which may include an arithmetic logic unit (ALU) and / or a floating-point unit for performing basic operations on integer and / or floating-point values. Conventional floating-point units can be designed to implement a fused multiply accumulate (FMA) operation, which multiplies two scalar operands and adds the intermediate result and an optional third scalar operand to an accumulation register. The matrix multiply and accumulate (MMA) operation is an extension of the FMA operation applied to scalar values of matrix operands. In other words, the MMA operation multiplies two matrices and optionally adds the resulting intermediate matrix to a third matrix operand. Fundamentally, the MMA operation can be simplified to multiple basic dot product operations added to an accumulation register. In addition, the dot product operation can be further simplified to a series of FMA operations on pairs of scalar operands.

[0007] Conventional processors can implement matrix operations by decomposing MMA operations into a series of dot product operations and addition operations, and each dot product operation can be further decomposed into a series of FMA instructions on the corresponding elements of a pair of vectors. However, since the MMA operation must be decomposed into each basic arithmetic operation using scalar operands, this technique is not very efficient. Each basic arithmetic operation performed by the processor's logic involves moving the scalar operands between the processor's register file and the input to the data path (i.e., the logic circuit). However, the basic concept of matrix operations is that the same elements of the matrix are reused in multiple dot product operations (for example, the same row of the first matrix is used to generate multiple dot products corresponding to multiple columns of the second matrix). If each basic arithmetic operation requires data to be loaded from the register file to the input of the data path before the arithmetic operation is performed, then each data element of the input operand can be loaded from the register file to the data path many times, which is an inefficient use of the register file bandwidth. While there may be techniques to improve processor efficiency (e.g., register files with multiple banks so that operands can be efficiently stored in separate banks and multiple operands can be loaded from the register file to the input of the datapath in a single clock cycle), datapaths are typically not designed specifically for matrix operations. Therefore, there is a need to address these and / or other issues associated with the prior art. Summary of the Invention

[0008] A method, computer-readable medium, and processor for performing a matrix multiply and accumulate (MMA) operation are disclosed. The processor includes a data path configured to perform the MMA operation to generate a plurality of elements of a result matrix at an output of the data path. Each element of the result matrix is generated by computing at least one dot product of a corresponding pair of vectors associated with a matrix operand specified in an instruction for the MMA operation. The dot product operation includes the steps of generating a plurality of partial products by multiplying each element of a first vector by a corresponding element of a second vector; aligning the plurality of partial products based on an exponent associated with each element of the first vector and each element of the second vector; and accumulating the plurality of aligned partial products into a result queue using at least one adder. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A flow chart illustrating a method for performing a matrix product and accumulate operation according to one embodiment is shown;

[0010] Figure 2 shows a parallel processing unit (PPU) according to one embodiment;

[0011] Figure 3A According to one embodiment, Figure 2 PPU general processing cluster;

[0012] Figure 3B According to one embodiment, Figure 2 Partition unit of PPU;

[0013] Figure 4 According to one embodiment, Figure 3A Streaming multiprocessors;

[0014] Figure 5 According to one embodiment, a Figure 2 PPU system on chip;

[0015] Figure 6 According to one embodiment, Figure 2 A conceptual diagram of the graphics processing pipeline implemented by the PPU;

[0016] Figure 7 shows a matrix multiply-add operation according to one embodiment;

[0017] Figure 8 is a conceptual diagram of a dot product operation according to one embodiment;

[0018] Figure 9 shows a portion of a processor including a data path configured to implement matrix operations according to one embodiment;

[0019] Figure 10 shows a conventional double-precision floating-point fused multiply-accumulate data path according to one embodiment;

[0020] Figure 11 shows a half-precision matrix product and accumulate data path according to one embodiment;

[0021] Figure 12 shows a half-precision matrix product and accumulate data path according to another embodiment;

[0022] Figure 13 shows a half-precision matrix product and accumulate data path according to yet another embodiment;

[0023] Figure 14 According to one embodiment, a Figure 10 The double-precision floating-point fused multiply-accumulate datapaths share at least one pipeline stage. Figure 13 A half-precision matrix product and accumulate data path; and

[0024] Figure 15 Illustrated are exemplary systems in which the various architecture and / or functionality of various previous embodiments may be implemented. DETAILED DESCRIPTION

[0025] Many modern applications can benefit from more efficient processing of matrix operations by processors. Arithmetic operations performed on matrix operands are commonly used in a variety of algorithms, including but not limited to deep learning algorithms, linear algebra, and graphics acceleration. This higher efficiency can be achieved by utilizing parallel processing units (PPUs), as matrix operations can be simplified into multiple parallel operations on different parts of the matrix operand.

[0026] This paper explores a new paradigm for datapath design to accelerate matrix operations such as those performed by processors. The basic concept of a datapath is that the datapath performs one or more dot product operations on multiple vector operands. Matrix operations can then be accelerated by reducing them to multiple dot product operations, and some dot product operations can benefit from data sharing within the datapath, which reduces the bandwidth between the register file and the datapath's inputs.

[0027] Figure 1A flow chart of a method 100 for performing a matrix product and accumulate operation according to one embodiment is shown. It will be appreciated that the method 100 is described in the context of software executed by a processor; however, in some embodiments, the method 100 may be implemented in hardware or in some combination of hardware and software. The method 100 begins at step 102, where an instruction for a matrix product and accumulate (MMA) operation is received. In one embodiment, the instruction for the MMA operation specifies a plurality of matrix operands. A first operand specifies a multiplicand input matrix A, a second operand specifies a multiplier input matrix B, and a third operand specifies a collector matrix C for accumulating the product of the first two input matrices. Each operand specified in the instruction is a matrix having a plurality of elements in a two-dimensional array of rows and columns.

[0028] In step 104, at least two vectors of the first operands specified in the instruction and at least two vectors of the second operands specified in the instruction are loaded from the register file into a plurality of operand collectors. In one embodiment, the operand collectors are a plurality of flip-flops coupled to the inputs of a data path configured to perform an MMA operation. The plurality of flip-flops temporarily store data for the operands of the MMA instruction at the inputs of the data path so that the plurality of operands can be loaded from the register file to the inputs of the data path within a plurality of clock cycles. Typically, a register file has a limited amount of bandwidth on one or more read ports so that only a limited amount of data can be read from the register file within a given clock cycle. Therefore, before starting the execution of the MMA operation on the data path, the operand collector enables all operands required for the data path to be read from the data file within a plurality of clock cycles.

[0029] In step 106, an MMA operation is performed to generate multiple elements of a result matrix at the output of the data path. In one embodiment, each element of the result matrix is generated by calculating at least one dot product of corresponding vector pairs stored in multiple operand collectors. During each pass, different combinations of vectors stored in the operand collectors are consumed, and the data path can be designed to generate multiple elements of the result matrix in multiple passes of the data path. Alternatively, different logic sets can be used to calculate multiple dot products in parallel, and the data path can be designed to generate multiple elements of the result matrix in a single pass of the data path. Of course, in some embodiments, in order to generate more result matrix elements within a single instruction cycle, multiple logic sets can be used to calculate multiple dot products in parallel and multiple passes of the data path can be used. It should be understood that in subsequent passes or instruction cycles, multiple elements of the result matrix are generated without the need to load new operand data from the register file into the operand collectors. In addition, it should be understood that each vector of the input matrix operands (i.e., A and B) stored in the operand collectors can be consumed by multiple dot product operations that contribute to multiple elements of the result matrix.

[0030] Depending on the user's needs, more illustrative information will now be provided regarding various optional architectures and features that may or may not implement the aforementioned framework. It should be noted that the following information is provided for illustrative purposes and should not be construed as limiting in any way. Any of the following features may optionally be incorporated with or without the other features described.

[0031] Parallel processing architecture

[0032] Figure 2A parallel processing unit (PPU) 200 is shown according to one embodiment. In one embodiment, the PPU 200 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 200 is a potential hidden architecture designed to process a large number of threads in parallel. A thread (i.e., an execution thread) is an instance of a group of instructions configured to be executed by the PPU 200. In one embodiment, the PPU 200 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 200 can be used to perform general-purpose computations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be noted that such a processor is described for illustrative purposes only and that any processor may be used in addition to and / or in place of the processor.

[0033] like Figure 2 As shown, PPU 200 includes an input / output (I / O) unit 205, a host interface unit 210, a front-end unit 215, a scheduler unit 220, a work distribution unit 225, a hub 230, a crossbar switch (Xbar) 270, one or more general processing clusters (GPCs) 250, and one or more partition units 280. PPU 200 can be connected to a host processor or other peripheral devices via a system bus 202. PPU 200 can also be connected to local memory including multiple memories 204. In one embodiment, the local memory can include multiple dynamic random access memory (DRAM) devices.

[0034] The I / O unit 205 is configured to send and receive communications (i.e., commands, data, etc.) from a host processor (not shown) via the system bus 202. The I / O unit 205 can communicate with the host processor directly via the system bus 202 or through one or more intermediate devices (such as a memory bridge). In one embodiment, the I / O unit 205 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating via a PCIe bus. In alternative embodiments, the I / O unit 205 can implement other types of known interfaces for communicating with external devices.

[0035] The I / O unit 205 is coupled to the host interface unit 210, which decodes data packets received via the system bus 202. In one embodiment, the data packets represent commands configured to cause the PPU 200 to perform various operations. The host interface unit 210 sends the decoded commands to various other units of the PPU 200 in the manner specified by the commands. For example, some commands may be sent to the front end unit 215. Other commands may be sent to the hub 230 or other units of the PPU 200, such as one or more copy engines, a video encoder, a video decoder, a power management unit, etc. (not explicitly shown). In other words, the host interface unit 210 is configured to route communications between the various logical units of the PPU 200.

[0036] In one embodiment, a program executed by a host processor encodes a command stream in a buffer, which provides a workload to the PPU 200 for processing. The workload may include a large number of instructions and data to be processed by those instructions. A buffer is an area in memory that can be accessed (i.e., read / written) by both the host processor and the PPU 200. For example, the host interface unit 210 can be configured to access a buffer in system memory connected to the system bus 202 via a memory request sent by the I / O unit 205 on the system bus 202. In one embodiment, the host processor writes a command stream to the buffer and then sends a pointer to the starting point of the command stream to the PPU 200. The host interface unit 210 provides pointers to one or more command streams to the front end unit 215. The front end unit 215 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 200.

[0037] The front-end unit 215 is coupled to a scheduler unit 220, which configures the various GPCs 250 to process tasks defined by one or more streams. The scheduler unit 220 is configured to track state information related to the various tasks managed by the scheduler unit 220. The state may indicate which GPC 250 the task is assigned to, whether the task is active or inactive, the priority associated with the task, and the like. The scheduler unit 220 manages the execution of multiple tasks on one or more GPCs 250.

[0038] Scheduler unit 220 is coupled to work distribution unit 225, which is configured to dispatch tasks for execution on GPCs 250. Work distribution unit 225 can track a plurality of scheduled tasks received from scheduler unit 220. In one embodiment, work distribution unit 225 manages a pending task pool and an active task pool for each of GPCs 250. The pending task pool can include a plurality of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular GPC 250. The active task pool can include a plurality of time slots (e.g., 4 time slots) for tasks that are actively being processed by GPC 250. When a GPC 250 completes execution of a task, the task is evicted from the active task pool of GPC 250, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 250. If an active task on a GPC 250 has become idle, such as while waiting for data dependencies to be resolved, the active task may be evicted from the GPC 250 and returned to the pending task pool while another task in the pending task pool is selected and scheduled for execution on the GPC 250 .

[0039] Work distribution unit 225 communicates with one or more GPCs 250 via XBar 270. XBar 270 is an interconnect network that couples many units of PPU 200 to other units of PPU 200. For example, XBar 270 can be configured to couple work distribution unit 225 to a specific GPC 250. Although not explicitly shown, one or more other units of PPU 200 are coupled to host interface unit 210. Other units can also be connected to XBar 270 via hub 230.

[0040] Tasks are managed by the scheduler unit 220 and dispatched to the GPCs 250 by the work distribution unit 225. The GPCs 250 are configured to process the tasks and generate results. The results can be consumed by other tasks within the GPC 250, routed to different GPCs 250 via the XBar 270, or stored in the memory 204. The results can be written to the memory 204 via the partition unit 280, which implements a memory interface for writing / reading data to / from the memory 204. In one embodiment, the PPU 200 includes a number U of partition units 280, which is equal to the number of separate and distinct memories 204 coupled to the PPU 200. This will be discussed below in conjunction with Figure 3B The partition unit 280 is described in more detail.

[0041] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications executing on the host processor to schedule operations to be performed on the PPU 200. The application can generate instructions (i.e., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 200. The driver kernel outputs the tasks to one or more streams processed by the PPU 200. Each task can include one or more related thread groups, referred to herein as warps. A thread block can refer to multiple thread groups that include instructions for performing a task. Threads in the same thread group can exchange data through shared memory. In one embodiment, a thread group includes 32 related threads.

[0042] Figure 3A According to one embodiment, Figure 2 PPU 200 GPC 250. Figure 3A As shown, each GPC 250 includes multiple hardware units for processing tasks. In one embodiment, each GPC 250 includes a pipeline manager 310, a pre-raster operation (PROP) unit 315, a raster engine 325, a work distribution crossbar (WDX) 380, a memory management unit (MMU) 390, and one or more texture processing clusters (TPCs).

[0043] 320. It should be understood that Figure 3A The GPC 250 may include instead Figure 3A The units shown in or except Figure 3A Other hardware units besides those shown in .

[0044] In one embodiment, the operation of the GPC 250 is controlled by a pipeline manager 310. The pipeline manager 310 manages the configuration of one or more TPCs 320 for processing tasks assigned to the GPC 250. In one embodiment, the pipeline manager 310 can configure at least one of the one or more TPCs 320 to implement at least a portion of a graphics rendering pipeline. For example, a TPC 320 can be configured to execute vertex shading programs on a programmable streaming multiprocessor (SM) 340. The pipeline manager 310 can also be configured to route packets received from the work distribution unit 225 to appropriate logic units within the GPC 250. For example, some packets can be routed to fixed-function hardware units in the PROP 315 and / or raster engine 325, while other packets can be routed to the TPC 320 for processing by the primitive engine 335 or SM 340.

[0045] The PROP unit 315 is configured to route data generated by the raster engine 325 and the TPC 320 to a raster operation (ROP) unit in the partition unit 280, as described in more detail below. The PROP unit 315 may also be configured to perform optimizations for color blending, organize pixel data, perform address translation, and the like.

[0046] The raster engine 325 includes a plurality of fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster engine 325 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile joint engine. The setup engine receives transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are transmitted to the coarse raster engine to generate coverage information for the primitives (e.g., x, y coverage masks for tiles). The output of the coarse raster engine can be transmitted to the culling engine, where fragments associated with primitives that fail the z test are culled, and to the clipping engine, where fragments outside the viewing cone are clipped. Those fragments that are exempt from clipping and culling can be passed to the fine raster engine to generate properties for pixel fragments based on the plane equations generated by the setup engine. The output of the raster engine 325 includes fragments to be processed, such as fragment shading implemented in the TPC 320.

[0047] Each TPC 320 included in the GPC 250 includes an M-Pipe Controller (MPC) 330, a primitive engine 335, one or more SMs 340, and one or more texture units 345. The MPC 330 controls the operation of the TPC 320 and routes data packets received from the pipeline manager 310 to appropriate units in the TPC 320. For example, data packets associated with vertices may be routed to the primitive engine 335, which is configured to retrieve vertex attributes associated with the vertices from the memory 204. Conversely, data packets associated with shading programs may be sent to the SM 340.

[0048] In one embodiment, the texture unit 345 is configured to load a texture map (e.g., a 2D array of texels) from memory 204 and sample the texture map to produce sampled texture values for use by shading programs executed by the SM 340. The texture unit 345 performs texture operations, such as filtering operations using mip maps (i.e., texture maps at different levels of detail). The texture unit 345 is also used as a load / store path from the SM 340 to the MMU 390. In one embodiment, each TPC 320 includes two (2) texture units 345.

[0049] SM 340 includes a programmable streaming processor that is configured to process tasks represented by multiple threads. Each SM 340 is multi-threaded and is configured to execute multiple threads (e.g., 32 threads) from a particular thread group simultaneously. In one embodiment, SM 340 implements a SIMD (single instruction, multiple data) architecture, wherein each thread in a thread group (i.e., warp) is configured to process a different data set based on the same instruction set. All threads in the thread group execute the same instruction. In another embodiment, SM 340 implements a SIMT (single instruction, multiple thread) architecture, wherein each thread in a thread group is configured to process a different data set based on the same instruction set, but wherein individual threads in the thread group are allowed to diverge during execution. In other words, when an instruction for the thread group is dispatched for execution, some threads in the thread group may be active, thereby executing the instruction, while other threads in the thread group may be inactive, thereby performing no-operation (NOP) instead of executing the instruction. The following is combined with Figure 4 SM 340 is described in more detail.

[0050] The MMU 390 provides an interface between the GPC 250 and the partition unit 280. The MMU 390 can provide virtual-to-physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the MMU 390 provides one or more translation lookaside buffers (TLBs) to improve the translation of virtual addresses to physical addresses in the memory 204.

[0051] Figure 3B According to one embodiment, Figure 2 The partition unit 280 of the PPU 200. Figure 3B As shown, partition unit 280 includes a raster operations (ROP) unit 350, a level 2 (L2) cache 360, a memory interface 370, and an L2 crossbar (XBar) 365. Memory interface 370 is coupled to memory 204. Memory interface 370 can implement a 16-, 32-, 64-, or 128-bit data bus for high-speed data transfer. In one embodiment, PPU 200 includes U memory interfaces 370, one for each partition unit 280, where each partition unit 280 is connected to a corresponding memory 204. For example, PPU 200 can be connected to up to U memories 204, such as graphics double-data-rate, version 5, synchronous dynamic random access memory (GDDR5 SDRAM). In one embodiment, memory interface 370 implements a DRAM interface, and U is equal to 8.

[0052] In one embodiment, the PPU 200 implements a multi-level memory hierarchy. Memory 204 is located off-chip in SDRAM coupled to the PPU 200. Data from memory 204 can be retrieved and stored in an L2 cache 360 located on-chip and shared between the various GPCs 250. As shown, each partition unit 280 includes a portion of the L2 cache 360 associated with the corresponding memory 204. Lower-level caches can then be implemented in various units within the GPC 250. For example, each of the SMs 340 can implement a level 1 (L1) cache. The L1 cache is a dedicated memory dedicated to a particular SM 340. Data from the L2 cache 360 can be retrieved and stored in each of the L1 caches for processing within the functional units of the SM 340. The L2 cache 360 is coupled to a memory interface 370 and an XBar 270.

[0053] The ROP unit 350 includes a ROP manager 355, a color ROP (CROP) unit 352, and a Z ROP (ZROP) unit 354. The CROP unit 352 performs raster operations related to pixel color, such as color compression, pixel blending, and the like. The ZROP unit 354 performs depth testing in conjunction with the raster engine 325. The ZROP unit 354 receives the depth of a sample position associated with a pixel fragment from the culling engine of the raster engine 325. The ZROP unit 354 tests the depth relative to the corresponding depth of the sample position associated with the fragment in the depth buffer. If the fragment passes the depth test for the sample position, the ZROP unit 354 updates the depth buffer and sends the result of the depth test to the raster engine 325. The ROP manager 355 controls the operation of the ROP unit 350. It should be appreciated that the number of partition units 280 may be different than the number of GPCs 250, and therefore each ROP unit 350 may be coupled to each of the GPCs 250. Thus, ROP manager 355 tracks packets received from different GPCs 250 and determines to which GPC 250 the results generated by ROP unit 350 are routed. CROP unit 352 and ZROP unit 354 are coupled to L2 cache 360 via L2 XBar 365 .

[0054] Figure 4 According to one embodiment, Figure 3A The streaming multiprocessor 340. Figure 4 As shown, SM 340 includes an instruction cache 405, one or more scheduler units 410, a register file 420, one or more processing cores 450, one or more special function units (SFUs) 452, one or more load / store units (LSUs) 454, an interconnection network 480, a shared memory 470, and an L1 cache 490.

[0055] As described above, the work distribution unit 225 dispatches tasks for execution on the GPCs 250 of the PPU 200. Tasks are assigned to specific TPCs 320 within the GPCs 250, and if the task is associated with a shader program, the task may be assigned to an SM 340. The scheduler unit 410 receives tasks from the work distribution unit 225 and manages the scheduling of instructions for one or more thread groups (i.e., warps) assigned to the SMs 340. The scheduler unit 410 schedules threads for execution in parallel thread groups, where each group is called a warp. In one embodiment, each warp includes 32 threads. The scheduler unit 410 can manage multiple different warps, schedule the warps for execution, and then dispatch instructions from the multiple different warps to various functional units (i.e., cores 450, SFUs 452, and LSUs 454) during each clock cycle.

[0056] In one embodiment, each scheduler unit 410 includes one or more instruction dispatch units 415. Each dispatch unit 415 is configured to transmit instructions to one or more of the functional units. Figure 4 In the embodiment shown, the scheduler unit 410 includes two dispatch units 415 that enable dispatching two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 410 may include a single dispatch unit 415 or additional dispatch units 415.

[0057] Each SM 340 includes a register file 420 that provides a set of registers for the functional units of the SM 340. In one embodiment, the register file 420 is divided between each of the functional units so that each functional unit is allocated a dedicated portion of the register file 420. In another embodiment, the register file 420 is divided between the different warps being executed by the SM 340. The register file 420 provides temporary storage for operands connected to the datapaths of the functional units.

[0058] Each SM 340 includes L processing cores 450. In one embodiment, the SM 340 includes a large number (e.g., 128, etc.) of different processing cores 450. Each core 450 may include a fully pipelined, single-precision processing unit, including a floating-point arithmetic logic unit (ALU) and an integer ALU. The core 450 may also include a double-precision processing unit (FPU), including a floating-point ALU. In one embodiment, the FPU implements the IEEE 754-2008 standard for floating-point arithmetic. Each SM 340 also includes M SFUs 452 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.) and N LSUs 454 that implement load and store operations between the shared memory 470 or L1 cache 490 and the register file 420. In one embodiment, the SM 340 includes 128 cores 450, 32 SFUs 452, and 32 LSUs 454.

[0059] Each SM 340 includes an interconnect network 480 that connects each of the functional units to register file 420 and connects the LSU 454 to register file 420, shared memory 470, and L1 cache 490. In one embodiment, interconnect network 480 is a crossbar switch that can be configured to connect any functional unit to any register in register file 420 and to connect the LSU 454 to memory locations in the register file and shared memory 470 and L1 cache 490.

[0060] Shared memory 470 is an on-chip memory array that allows data storage and communication between the SM 340 and the primitive engine 335, as well as between threads within the SM 340. In one embodiment, shared memory 470 includes 64 KB of storage capacity. An L1 cache 490 is located in the path from the SM 340 to the partition unit 280. The L1 cache 490 can be used to cache reads and writes. In one embodiment, the L1 cache 490 includes 24 KB of storage capacity.

[0061] The PPU 200 can be configured to perform highly parallel computations faster than a conventional CPU. Parallel computations have advantages in graphics processing, data compression, biostatistics, stream processing algorithms, and more.

[0062] When the configuration is used for general parallel computing, a simpler configuration can be used. In this model, if Figure 2As shown, the fixed-function GPU is bypassed, creating a simpler programming model. In this configuration, the work distribution unit 225 assigns and distributes thread blocks directly to the TPC 320. The threads in the block execute the same program, use unique thread IDs in computations to ensure each thread produces unique results, use SMs 340 to execute the program and perform computations, use shared memory 470 to communicate between threads, and use LSUs 454 to read and write to global memory through partitioned L1 cache 490 and partition units 280.

[0063] When configured for general-purpose parallel computing, SM 340 may also write commands that scheduler unit 220 may use to start new work on TPC 320 .

[0064] In one embodiment, the PPU 200 comprises a graphics processing unit (GPU). The PPU 200 is configured to receive commands specifying a shader for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, and the like. Typically, a primitive includes data specifying a plurality of vertices of the primitive (e.g., in a model space coordinate system) and attributes associated with each vertex of the primitive. The PPU 200 can be configured to process the primitives to generate a frame buffer (i.e., pixel data for each of the pixels of the display).

[0065] An application writes model data (i.e., a collection of vertices and attributes) for a scene into memory (such as system memory or memory 204). The model data defines each of the objects that may be visible on the display. The application then makes an API call to the driver kernel, requesting the model data to be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations to process the model data. These commands may reference different shading programs to be implemented on the SMs 340 of the PPU 200, including one or more of vertex shading, hull shading, domain shading, geometry shading, and pixel shading. For example, one or more of the SMs 340 may be configured to execute a vertex shading program that processes multiple vertices defined by the model data. In one embodiment, different SMs 340 may be configured to execute different shading programs simultaneously. For example, a first subset of SMs 340 may be configured to execute a vertex shading program, while a second subset of SMs 340 may be configured to execute a pixel shading program. The first subset of SMs 340 processes the vertex data to generate processed vertex data and writes the processed vertex data to the L2 cache 360 and / or memory 204. After the processed vertex data is rasterized (i.e., converted from three-dimensional data to two-dimensional data in screen space) to generate fragment data, a second subset of SMs 340 performs pixel shading to generate processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in memory 204. Vertex shading programs and pixel shading programs can be executed simultaneously, processing different data from the same scene in a pipelined manner, until all model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then transmitted to the display controller for display on the display device.

[0066] The PPU 200 may be included in a desktop computer, a laptop computer, a tablet computer, a smartphone (e.g., a wireless, handheld device), a personal digital assistant (PDA), a digital camera, a handheld electronic device, etc. In one embodiment, the PPU 200 is embodied on a single semiconductor substrate. In another embodiment, the PPU 200 is included in a system-on-chip (SoC) along with one or more other logic units (such as a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0067] In one embodiment, the PPU 200 may be included on a graphics card that includes one or more memories 204 (such as GDDR5 SDRAM). The graphics card may be configured to interface with a PCIe slot on a desktop computer's motherboard, which includes, for example, a northbridge chipset and a southbridge chipset. In another embodiment, the PPU 200 may be an integrated graphics processing unit (iGPU) included in the motherboard's chipset (i.e., northbridge).

[0068] Figure 5 According to one embodiment, a Figure 2 The system-on-chip (SoC) 500 of the PPU 200 is shown in FIG. Figure 5 As shown, as described above, SoC 500 includes a CPU 550 and a PPU 200. SoC 500 may also include a system bus 202 to enable communication between various components of SoC 500. Memory requests generated by CPU 550 and PPU 200 may be routed through a system MMU 590 shared by multiple components of SoC 500. SoC 500 may also include a memory interface 595 coupled to one or more memories 204. Memory interface 595 may implement, for example, a DRAM interface.

[0069] Although not explicitly shown, Figure 5 In addition to the components shown, SoC 500 may include other components. For example, SoC 500 may include multiple PPUs 200 (e.g., four PPUs 200), a video encoder / decoder, and a wireless broadband transceiver, among other components. In one embodiment, SoC 500 may be included in a package-on-package (PoP) configuration along with memory 204.

[0070] Figure 6 According to one embodiment, Figure 2 2. A conceptual diagram of a graphics processing pipeline 600 implemented by the PPU 200 of FIG. The graphics processing pipeline 600 is an abstract flow diagram of the processing steps implemented to generate a 2D computer-generated image from 3D geometric data. As is well known, pipeline architectures can perform long-latency operations more efficiently by dividing the operations into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, the graphics processing pipeline 600 receives input data 601 that is transmitted from one stage of the graphics processing pipeline 600 to the next stage to generate output data 602. In one embodiment, the graphics processing pipeline 600 may represent a graphics processing pipeline composed of API-defined graphics processing pipeline. Alternatively, graphics processing pipeline 600 can be implemented within the functional and architectural context of the previous figures and / or one or more of any subsequent figures.

[0071] like Figure 6 As shown, the graphics processing pipeline 600 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 610, a vertex shading stage 620, a primitive assembly stage 630, a geometry shading stage 640, a viewport scale, cull, and clip (VSCC) stage 650, a rasterization stage 660, a fragment shading stage 670, and a raster operation stage 680. In one embodiment, input data 601 includes commands that configure a processing unit to implement the stages of the graphics processing pipeline 600 and configure geometric primitives (e.g., points, lines, triangles, quads, triangle strips, or fans, etc.) to be processed by these stages. Output data 602 may include pixel data (i.e., color data), which is copied to a frame buffer or other type of surface data structure in memory.

[0072] The data assembly stage 610 receives input data 601, which specifies vertex data for high-level surfaces, primitives, etc. The data assembly stage 610 collects the vertex data in temporary storage or queues, such as by receiving a command from the host processor that includes a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then transferred to the vertex shading stage 620 for processing.

[0073] The vertex shading stage 620 processes vertex data by executing a set of operations (i.e., a vertex shader or program) once for each vertex. A vertex may be specified, for example, as a 4-coordinate vector (i.e., a 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 5<x,y,z,w> ). The vertex shading stage 620 can manipulate various vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 620 performs operations on the vertex coordinates or other vertex attributes associated with the vertex. These operations typically include lighting operations (i.e., modifying the color attributes of the vertex) and transformation operations (i.e., modifying the coordinate space of the vertex). For example, a vertex can be specified using coordinates in object coordinate space, which are transformed by multiplying the coordinates by a matrix that converts the coordinates from the object coordinate space to world space or normalized-device-coordinate (NCD) space. The vertex shading stage 620 generates transformed vertex data that is passed to the primitive assembly stage 630.

[0074] The primitive assembly stage 630 collects the vertices output by the vertex shading stage 620 and groups the vertices into geometric primitives for processing by the geometry shading stage 640. For example, the primitive assembly stage 630 can be configured to group every three consecutive vertices into geometric primitives (i.e., triangles) for transmission to the geometry shading stage 640. In some embodiments, particular vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). The primitive assembly stage 630 transmits the geometric primitives (i.e., a collection of associated vertices) to the geometry shading stage 640.

[0075] The geometry shading stage 640 processes geometric primitives by executing a set of operations (i.e., geometry shaders or programs) on the geometric primitives. Tessellation operations can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 640 can subdivide each geometric primitive into a finer mesh of two or more geometric primitives for processing by the rest of the graphics processing pipeline 600. The geometry shading stage 640 transmits the geometric primitives to the viewport SCC stage 650.

[0076] In one embodiment, the graphics processing pipeline 600 may operate within a streaming multiprocessor and vertex shading stage 620, primitive assembly stage 630, geometry shading stage 640, fragment shading stage 670, and / or hardware / software associated therewith, to sequentially perform processing operations. Once the sequential processing operations are complete, in one embodiment, the viewport SCC stage 650 may utilize the data. In one embodiment, primitive data processed by one or more stages in the graphics processing pipeline 600 may be written to a cache (e.g., an L1 cache, a vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 650 may access the data in the cache. In one embodiment, the viewport SCC stage 650 and the rasterization stage 660 are implemented as fixed-function circuits.

[0077] The viewport SCC stage 650 performs viewport scaling, culling, and clipping of geometric primitives. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of the viewer viewing the scene and defines a viewing frustum that encloses the objects in the scene. The viewing frustum can include a viewing plane, a back plane, and four clipping planes. Any geometric primitives that are completely outside the viewing frustum can be culled (i.e., discarded) because they will not contribute to the final rendered scene. Any geometric primitives that are partially inside the viewing frustum and partially outside the viewing frustum can be clipped (i.e., converted to new geometric primitives that are enclosed within the viewing frustum). In addition, each geometric primitive can be scaled based on the depth of the viewing frustum. All potentially visible geometric primitives are then transferred to the rasterization stage 660.

[0078] The rasterization stage 660 converts 3D geometric primitives into 2D fragments (e.g., usable for display, etc.). The rasterization stage 660 can be configured to use the vertices of the geometric primitives to set a set of plane equations from which various attributes can be interpolated. The rasterization stage 660 can also calculate a coverage mask for multiple pixels, which indicates whether one or more sampling positions of the pixel intercept the geometric primitive. In one embodiment, a z test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have already been rasterized. The rasterization stage 660 generates fragment data (i.e., the interpolated vertex attributes associated with the specific sampling position of each covered pixel), which is transmitted to the fragment shading stage 670.

[0079] The fragment shading stage 670 processes the fragment data by executing a set of operations (i.e., a fragment shader or program) on each of the fragments. The fragment shading stage 670 can generate pixel data (i.e., color values) for the fragment, such as by performing lighting operations or sampling texture mapping using the fragment's interpolated texture coordinates. The fragment shading stage 670 generates pixel data that is sent to the raster operations stage 680.

[0080] In one embodiment, the fragment shading stage 670 may sample a texture map using one or more texture units 345 of the PPU 200. Texture data 603 may be read from memory 204 and may be sampled using the texture unit 345 hardware. The texture unit 345 may return the sampled values to the fragment shading stage 670 for processing by the fragment shader.

[0081] The raster operations stage 680 may perform various operations on the pixel data, such as performing alpha tests, stencil tests, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 680 has completed processing the pixel data (i.e., output data 602), the pixel data may be written to a render target, such as a frame buffer, a color buffer, etc.

[0082] It should be appreciated that one or more additional stages may be included in graphics processing pipeline 600 in addition to or in place of one or more of the above-described stages. Various implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments, one or more of the above-described stages may be excluded from the graphics processing pipeline (such as geometry shading stage 640). Other types of graphics processing pipelines are contemplated as being within the scope of the present disclosure. Furthermore, any stage of graphics processing pipeline 600 may be implemented by one or more dedicated hardware units within a graphics processor (such as PPU 200). Other stages of graphics processing pipeline 600 may be implemented by programmable hardware units (such as SM 340 of PPU 200).

[0083] Graphics processing pipeline 600 can be implemented via an application program executed by a host processor, such as CPU 550. In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that can be utilized by the application program to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of PPU 200. The API provides an abstraction for programmers, allowing them to utilize specialized graphics hardware, such as PPU 200, to generate graphics data without requiring them to utilize the specific instruction set of PPU 200. An application program can include API calls that are routed to the device driver of PPU 200. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on CPU 550. In other cases, the device driver can perform operations at least in part by initiating operations on PPU 200 using an input / output interface between CPU 550 and PPU 200. In one embodiment, the device driver is configured to implement graphics processing pipeline 600 using the hardware of PPU 200.

[0084] Various programs may be executed within the PPU 200 to implement the various stages of the graphics processing pipeline 600. For example, a device driver may launch a kernel on the PPU 200 to execute the vertex shading stage 620 on one SM 340 (or multiple SMs 340). The device driver (or the initial kernel executed by the PPU 200) may also launch other kernels on the PPU 200 to execute other stages of the graphics processing pipeline 600, such as the geometry shading stage 640 and the fragment shading stage 670. In addition, some of the stages of the graphics processing pipeline 600 may be implemented on fixed unit hardware, such as a rasterizer or data assembler implemented within the PPU 200. It should be appreciated that the results from one kernel may be processed by one or more intermediate fixed-function hardware units before being processed by subsequent kernels on the SM 340.

[0085] Matrix multiply and accumulate (MMA) operation

[0086] The MMA operation extends the concept of the FMA operation to matrix input operands. In other words, many algorithms are designed around the basic arithmetic operation of multiplying a first input matrix by a second input matrix and adding the result to a third input matrix (i.e., a collector matrix). More specifically, the MMA operation can take two input matrices (A and B) and a third collector matrix (C in ) to perform the following operations:

[0087] C out =A*B+C in (Equation 1)

[0088] Where A is an input matrix of size N×K, B is an input matrix of size K×M, and C is a collector matrix of size N×M. The collector matrix C is read from the register file, and the results of the MMA operation are accumulated and written on the data of the collector matrix C in the register file. In one embodiment, the collector matrix C and the result matrix D (C out =D) can be different operands, so that the result of the MMA operation is not written on the collector matrix C.

[0089] Figure 7 7 shows an MMA operation according to one embodiment. The MMA operation multiplies the input matrix A 710 with the input matrix B 720 and accumulates the result into the collector matrix C 730. Figure 7 As shown, the input matrix A is given as an 8×4 matrix, the input matrix B is given as a 4×8 matrix, and the collector matrix C is given as an 8×8 matrix. In other words, Figure 7 The MMA operation shown in corresponds to (1) N = 8; (2) M = 8; and (3) K = 4. However, Figure 7Nothing shown in should be construed as limiting MMA operations to these dimensions. In fact, the processor's datapath can be designed to operate on matrix operands of arbitrary size, as will be shown in more detail below, and matrix operands that do not exactly align with the fundamental size of the vector inputs in the dot product operation can be simplified to multiple intermediate operations using the datapath.

[0090] Now back Figure 7 Each element of a matrix operand can be a value encoded in a specific format. The various formats include, but are not limited to, single-precision floating-point values (e.g., 32-bit values encoded according to the IEEE 754 standard); half-precision floating-point values (e.g., 16-bit values encoded according to the IEEE 754 standard); signed / unsigned integers (e.g., 32-bit two's complement integers); signed / unsigned short integers (e.g., 16-bit two's complement integers); fixed-point formats; and other formats.

[0091] In one embodiment, the processor can be designed as a 64-bit architecture so that data words are stored in registers with a width of 64 bits. Typically, the processor will then implement a data path that operates on values encoded using a format up to 64 bits; however, some data paths can be designed to operate on values encoded using fewer bits. For example, a vector machine can be designed to pack two or four elements encoded using 32 bits or 16 bits, respectively, into each 64-bit register. The data path is then configured to execute the same instruction for multiple elements of the input vector in parallel on multiple similar vector units. However, it should be appreciated that the vector machine typically performs operations on the elements of the input vector as completely independent operations. In other words, each of the elements packed into a single 64-bit register is used for only one vector operation and is not shared between different vector units.

[0092] In one embodiment, each element of input matrix A 710 and each element of input matrix B 720 can be encoded as half-precision floating-point value.If each data word is 64-bit wide, then the four elements of input matrix can be packed in each data word.Therefore, each register in the register file that is assigned to store at least a portion of input matrix A 710 or input matrix B 720 has the ability of storing four half-precision floating-point elements of corresponding input matrix.This makes the effective storage of matrix operation object can be realized in the common register file that is associated with one or more data paths of processor.

[0093] It should be understood that the present invention is not limited to half-precision floating-point data. In some embodiments, each element of the input matrix can be encoded as a full-precision floating-point value. In other embodiments, each element of the input matrix can be encoded as a 16-bit signed integer. In other embodiments, the elements of input matrix A 710 can be encoded as half-precision floating-point values, and the elements of input matrix B 720 can be encoded as 32-bit signed integers. In such an embodiment, the elements of any input operand can be converted from one format to another format in the first stage of the data path, so that the format of each of the input operands can be mixed within a single matrix product and accumulation operation. In addition, in another embodiment, the elements of input matrix A 710 and input matrix B 720 can be encoded as half-precision floating-point values, and the elements of collector matrix C 730 can be encoded as full-precision floating-point values. The data path can even be designed to use the elements of collector matrix C 730, which have different precision from the elements of input matrix A 710 and input matrix B 720. For example, the accumulator registers in the data path may be extended to store elements of the collector matrix C 730 as full-precision floating-point values, which adds the initial values of the elements of the collector matrix C 730 to the results of the dot product operation performed on the half-precision floating-point values, which values may be equivalent to the full-precision floating-point values of the partial products if the multiplication is performed in a lossless manner.

[0094] like Figure 7 As shown, the matrix has been divided into visually 4×4 element sub-matrices. In an embodiment, where each element of input matrix A 710 and input matrix B 720 is encoded as a half-precision floating point value (e.g., 16-bit floating point), the 4×4 element sub-matrices are essentially four 4-element vectors from the matrix. In the case of input matrix A 710, the matrix is divided into an upper vector set and a lower vector set. Each vector can correspond to a row of input matrix A 710, where each row of four elements can be packed into a single 64-bit register. In the case of input matrix B 720, the matrix is divided into a left vector set and a right vector set. Each vector can correspond to a column of input matrix B 720, where each column of four elements can be packed into a single 64-bit register. In the case of collector matrix C 730, the matrix is divided into four 4×4 element sub-matrices defined as the upper left quadrant, the upper right quadrant, the lower left quadrant, and the lower right quadrant. As long as the elements are encoded as half-precision floating point values, each quadrant stores four 4-vector elements from the collector matrix C 730. Each quadrant may correspond to multiple vectors (i.e., portions of rows or portions of columns) of the collector matrix C 730. Each quadrant also corresponds to multiple dot product operations performed using corresponding pairs of vectors from the input matrix.

[0095] For example, Figure 7 As shown, the collector matrix C 0,0The first element is the first vector of the input matrix A710 0,0 ,A 0,1 ,A 0,2 ,A 0,3 > and the first vector of the input matrix B 720 0,0 ,B 1,0 ,B 2,0 ,B 3,0 > is generated by the result of the dot product operation between the two vectors. The first vector of the input matrix A 710 represents the first row of the input matrix A 710. The first vector of the input matrix B 720 represents the first column of the input matrix B 720. Therefore, the dot product between these two vectors is given as:

[0096] C 0,0 =A 0,0 B 0,0 +A 0,1 B 1,0 +A 0,2 B 2,0 +A 0,3 B 3,0 +C 0,0 (Equation 2) where the dot product operation is essentially four multiplication operations performed on corresponding elements of the two vectors followed by four addition operations that sum the four partial products resulting from the multiplication operations with the initial values of the elements of the collector matrix. Each of the other elements of the collector matrix C 730 is then calculated in a similar manner using different combinations of vectors of the input matrix. For example, another element of the collector matrix C 730 (element C 3,2 ) is the fourth vector of the input matrix A710 3,0 ,A 3,1 ,A 3,2 ,A 3,3 > and the third vector of the input matrix B 720 0,2 ,B 1,2 ,B 2,2 ,B 3,2 > is generated by the result of the dot product operation between Figure 7 ​​​​As shown in the MMA operation of FIG710 , each vector of input matrix A 710 is consumed by eight dot product operations, which are configured to generate corresponding rows of elements of collector matrix C 730. Similarly, each vector of input matrix B 720 is consumed by eight dot product operations, which are configured to generate corresponding columns of elements of collector matrix C 730. Although each of the 64 dot product operations used to generate elements of collector matrix C 730 is uniquely defined using a different pair of vectors from the input matrices, each vector of the first input operand and each vector of the second input operand is consumed by multiple dot product operations and contributes to multiple individual elements of the result matrix.

[0097] It should be appreciated that the MMA operation described above can be accelerated by loading vector sets from two input matrices into the input of the datapath, as long as the datapath can be configured to consume vector sets in an efficient manner in order to simplify the bandwidth between the register file and the input of the datapath. For example, in one embodiment, the first two rows of the upper left quadrant of collector matrix C 730 can be calculated by a datapath configured to receive as input the first two vectors from the upper vector set of input matrix A 710 and the first four vectors from the left vector set of input matrix B 720, as well as the first two vectors (i.e., rows) from the upper left quadrant of collector matrix C 730. Such a datapath would require eight 64-bit words as input: two 64-bit words to store the two vectors of input matrix A 710, four 64-bit words to store the four vectors of input matrix B 720, and two 64-bit words to store the two vectors of collector matrix C 730. Furthermore, if the elements of collector matrix C 730 are encoded as full precision floating point values (eg, 32-bit floating point), the size of the input to the data path of the two vectors of collector matrix C 730 will be doubled to four 64-bit words.

[0098] The datapath can then be configured to perform eight dot product operations in parallel, in a single pass, in multiple passes in series, or in a combination of serial and parallel operations. For example, the datapath can be designed to perform one 4-vector dot product operation per pass, taking one vector from input matrix A 710 and one vector from input matrix B 720, and generating a single element of collector matrix C 730. The datapath is then operated on eight passes using different combinations of the six vectors from the two input matrices to generate eight different elements of collector matrix C 730. Alternatively, the datapath can be designed to perform four 4-vector dot product operations per pass, taking one vector from input matrix A 710 and four vectors from input matrix B 720, and generating four elements of collector matrix C 730 in parallel. The datapath is then operated on two passes using different vectors from input matrix A 710 and the same four vectors from input matrix B 720 to generate eight elements of collector matrix C 730 during each pass. It should be appreciated that before the data path performs multiple dot product operations using different combinations of inputs in each dot product operation, the inputs to the data path can be loaded once from the register file. This significantly reduces the bandwidth between the register file and the data path. For example, only 6 vectors of the two input matrices A and B need to be loaded from the register file into the input of the data path in order to perform eight dot product operations, while performing all eight dot product operations separately using a data path that is capable of performing a single dot product operation and has an input capacity of only two vectors will require 16 vectors to be loaded from the register file into the input of the data path because the vectors are reused in multiple dot product operations.

[0099] It should be appreciated that even if the datapath is configured to generate dot products of a different length than the size of each of the vectors (i.e., the dimension K of the input matrix is not equal to the number of partial products generated within the datapath for a single dot product operation), the datapath uses accumulator registers (e.g., collector matrix C 730) so that each vector can be divided into multiple sub-vectors and then loaded into the input of the datapath over multiple execution cycles (wherein after each cycle, the output of collector matrix C 730 is reloaded into the input of the datapath for the next cycle). Therefore, the dimension K of the input matrices A 710 and B 720 is not limited to a specific implementation of the dot product operation performed by the datapath. For example, if the datapath only generates 2-vector dot products (i.e., corresponding to a dot product of a pair of two-element vectors), each row of the input matrix A 710 can be divided into a first vector in the first half of the row and a second vector in the second half of the row, and each column of the input matrix B 720 can be divided into a first vector in the first half of the column and a second vector in the second half of the column. Then, the elements of collector matrix C 730 are generated over multiple instruction cycles, wherein the first half of the vectors of input matrix A 710 and the upper half of the vectors of input matrix B 720 are loaded into the input of the datapath during the first instruction cycle, and the second half of the vectors of input matrix A 710 and the lower half of the vectors of input matrix B 720, as well as the intermediate results stored in collector matrix C 730 during the first instruction cycle, are loaded into the input of the datapath during the second instruction cycle. By dividing each of the vectors of the input matrix into multiple parts, each part having a number of elements equal to the size of the dot product operation implemented by the datapath, the MMA operation can be simplified in this manner for input matrices of dimension K of arbitrary size. Even if dimension K is not divisible by the size of the dot product operation, the vector can be padded with zeros to obtain the correct result.

[0100] Figure 8 is a conceptual diagram of a dot product operation according to one embodiment. The dot product operation basically adds multiple partial products. The dot product operation can specify three operands, vector A, vector B, and scalar collector C. Vector A and vector B have the same length (i.e., the number of elements). Figure 8 As shown, vectors A and B are given a length of 2; however, it should be appreciated that the dot product operation can have any length greater than or equal to 2.

[0101] The dot product operation multiplies pairs of elements from the input vectors A and B. Figure 8As shown, in multiplier 822, the first element A0812 from input vector A is multiplied by the corresponding element B0814 from input vector B to generate a partial product A0B0826. In multiplier 824, the second element A1816 from input vector A is multiplied by the corresponding element B1818 from input vector B to generate a partial product A1B1828. The partial product A0B0826, the partial product A1B1828, and the scalar collector value C are then added together using a three-element adder 830. in 820 is summed to generate the result value C out 832. Result value C out 832 can be stored in the collector for scalar values C in 820 and can be reused to accumulate multiple dot product operations for longer vectors.

[0102] Furthermore, the dot product operation may be extended by adding additional multipliers 822, 824, etc. in parallel to compute additional partial products, which are then summed with either a larger element-wise adder or a tree of smaller adders that generate intermediate sums, which are then summed by additional multi-element adders.

[0103] While the dot product operation can be implemented in a traditional FMA datapath, where each partial product is computed during a single pass through the datapath and accumulated into an accumulation register, it is more efficient to compute multiple partial products of the dot product operation in parallel and sum the results in a single multi-stage pipeline. Furthermore, while multiple cores can be used simultaneously in a SIMD / SIMT machine to compute partial products in parallel, an additional step of summing all partial products is still required, which is not trivial to accomplish efficiently in such machines.

[0104] Figure 9 A portion of a processor 900 including an HMMA data path 930 configured to implement matrix operations is shown according to one embodiment. The processor 900 may refer to a central processing unit (CPU), a graphics processing unit (GPU) or other parallel processing unit, a reduced instruction set computer (RISC) type processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), etc. Nothing in this disclosure should be construed to limit the processor 900 to a parallel processing unit such as the PPU 200.

[0105] like Figure 9As shown, the processor 900 includes a multi-bank register file implemented as a plurality of register banks 910. Each register bank 910 can store a plurality of data words in a plurality of registers. Each register bank 910 can have independent and different read and write ports so that one register in the register bank 910 can be read and another register can be written in any given clock cycle. Therefore, a data word can be read from each register bank 910 simultaneously during a single clock cycle and loaded into the operand collector 920. The register file is configured to store operands specified in instructions for MMA operations. In one embodiment, each operand specified in the instruction is a matrix having a plurality of elements in rows and columns of a two-dimensional array, and each register can store one or more elements of a specific operand. Of course, in one embodiment, the register file can include only a single bank, meaning that only one register can be read from the register file and loaded into the operand collector 920 per clock cycle.

[0106] The processor 900 also includes a plurality of operand collectors coupled to the inputs of one or more data paths. In one embodiment, the operand collector 920 includes a plurality of asynchronous flip-flops (flip-flops), which enable data to be loaded into the operand collector 920 during any specific clock cycle, and then read data from the operand collector 920 in any subsequent clock cycle. In other words, based on the signal at the input of the flip-flop, the flip-flop will not be set or reset in each clock cycle. On the contrary, the control logic determines when the flip-flop is set or reset and when the data stored in the flip-flop is transferred to the output of the flip-flop based on the input signal. This enables the plurality of operand collectors 920 to load operands from the register file within a plurality of clock cycles, and then during a single clock cycle, a plurality of operands are provided to the input of the data path in parallel. It should be appreciated that the operand collector 920 can be implemented in a variety of different ways, including various types of latches and / or flip-flops, and various embodiments can use different basic technologies to implement the operand collector 920. However, the function of the operand collector 920 is to temporarily store operands required to perform operations on the datapath, where the operands can be loaded from the register file 910 in one or more clock cycles, depending on which register bank 910 the operands are stored in and how many read ports are available in those register banks 910.

[0107] A crossbar 915 or other type of switchable interconnect can be coupled to the read ports of the register banks 910 and the inputs of the operand collectors. The crossbar 915 can be configured to route signals from the read port associated with any register bank 910 to a specific operand collector 920. For example, the read port of register bank 1 910(1) can include 64 interconnects carrying signals corresponding to the 64 bits contained in a single register of the register file. These 64 interconnects can be connected to one of a plurality of different operand collectors 920, each operand collector 920 including 64 flip-flops for storing the 64 bits encoded by the signals transmitted via the read port. If a data path requires three operand collectors 920 coupled to the inputs of the data path, each operand collector 920 includes 64 flip-flops to store the 64 bits of the corresponding operands of the data path, and then the crossbar 915 can be configured to route the 64 signals on the 64 interconnects of the read port to any one of the three operand collectors 920.

[0108] The operand collector 920 may be coupled to the input of one or more data paths. Figure 9 As shown, the operand collector 920 can be coupled to a half-precision matrix multiply-accumulate (HMMA) data path 930 and a double-precision (64-bit) floating-point (FP64) data path 940. The FP64 data path 940 can be a conventional double-precision floating-point FMA data path that enables addition, subtraction, multiplication, division, and other operations to be performed on double-precision floating-point operands. In one embodiment, the FP64 data path 940 can include logic for performing FMA operations on three scalar double-precision floating-point operands (e.g., A, B, and C), as is known in the art.

[0109] The output of the FP64 datapath 940 is coupled to a result queue 950. The result queue 950 stores the results generated by the FP64 datapath 940. In one embodiment, the result queue 950 includes multiple flip-flops for storing multiple bits of the results generated by the FP64 datapath 940. For example, the result queue 950 may include 64 flip-flops for storing double-precision floating-point results from the FP64 datapath 940. The result queue 950 enables temporary storage of the results while waiting for the availability of a write port to write the value back to the register file. It should be appreciated that the FP64 datapath 940 may be included in each of multiple similar cores that share the multi-bank register file 910. During a particular clock cycle, only one core can write a value back to each register bank. Therefore, if two or more cores generate results in a given clock cycle and both results need to be written back to the same register bank 910, one result may be written to the register bank 910 during the first clock cycle, and the other result may be written to the register bank 910 during the second clock cycle.

[0110] It should be appreciated that the result queue 950 can be attached to an accumulator register contained within the datapath that does not need to be written back to the register file between executing multiple instructions. For example, an FMA instruction can include operands A, B, and C during the first instruction, and then include only operands A and B during one or more subsequent instructions, chaining multiple instructions together using the internal accumulator register, reusing the accumulated value C as the third operand for each subsequent instruction. In some embodiments, if the results generated by the FP64 datapath 940 are always written to the register file as soon as they become available, the result queue 950 can be omitted. However, such an architecture requires more advanced control over the memory allocation of the multi-bank register file to avoid any conflicts with the write ports, as well as knowledge of the pipeline lengths of two or more cores sharing the register file to appropriately schedule which core will need to access a given write port within a particular clock cycle. In processors with a large number of cores, it may be easier to use the result queue 950 to write values back to the register file as needed, causing any core to stall to complete subsequent instructions until the result has been written back to the register file.

[0111] In one embodiment, the HMMA datapath 930 and the FP64 datapath 940 share the same operand collector 920 available. The HMMA datapath 930 and the FP64 datapath 940 can be included in a common core of the processor 900, which includes multiple cores, each core including an FP64 datapath 940 and an HMMA datapath 930 and possibly an integer arithmetic logic unit (ALU). In one embodiment, the HMMA datapath 930 is configured to perform a matrix multiply and accumulate (MMA) operation. The instruction for the MMA operation specifies multiple matrix operands, which are configured to perform an operation equivalent to the function specified by Equation 1 above.

[0112] In one embodiment, a plurality of operand collectors 920 include at least two vectors for the first operand specified in the instruction (i.e., input matrix A 710) and at least two vectors for the second operand specified in the instruction (i.e., input matrix B 720). Each of the at least two vectors has at least two elements in the row or column of the matrix operand. For example, in one embodiment, the HMMA data path 930 is configured to receive two vectors from the first operand and four vectors from the second operand as inputs to the data path. Therefore, the number of operand collectors 920 should be enough to store at least six vectors (e.g., at least six 64-bit operand collectors) of the two input matrix operands. According to the design of the HMMA data path 930, other embodiments may require more or less operand collectors 920.

[0113] In one embodiment, the HMMA data path 930 is further configured to receive at least two vectors of a third operand (i.e., collector matrix C 730) specified in the instruction. The collector matrix C 730 is added to the result of multiplying the first operand and the second operand specified in the instruction. The number of combined elements in the vector from the third operand must match the product of the number of vectors of the first operand and the number of vectors of the second operand. For example, if the multiple operand collectors 920 store two vectors (e.g., rows) of the input matrix A 710 and four vectors (e.g., columns) of the input matrix B 720, the number of elements in at least two vectors of the collector matrix C 730 must be equal to eight. In addition, the index of the elements of the third operand must match the index of the vectors of the first operand and the second operand. For example, if the two vectors of the first operand correspond to the first and second rows of input matrix A 710, and the four vectors of the second operand correspond to the first through fourth rows of input matrix B 720, then the indices of the elements of the third operand must match the <row, column> index vectors associated with the dot product of any vector of input matrix A 710 and any vector of input matrix B 720, where the indices of the elements of the third operand are all two-dimensional.

[0114] Similarly, the HMMA data path 930 generates multiple elements of the result matrix at the output of the HMMA data path 930. Each element of the multiple elements of the result matrix is generated by calculating at least one dot product of the corresponding vector pair selected from the matrix operation object. The dot product operation may include the step of accumulating multiple partial products into the result queue 950. Each partial product in the multiple partial products is generated by multiplying each element of the first vector with the corresponding element of the second vector. An example of a dot product operation is given in the above equation 2. It should be appreciated that in one embodiment, multiple partial products are calculated in parallel in the HMMA data path 930, and the adder tree inside the HMMA data path 930 is used to accumulate the multiple partial products before outputting to the result queue 950. In another embodiment, the partial products are calculated serially over multiple passes within the HMMA data path 930, and each partial product is accumulated into the accumulation register inside the HMMA data path 930. When all partial products and addend values from collector matrix C 730 have been accumulated into the internal accumulation register in multiple passes, the final result is output to the result queue 950 .

[0115] In one embodiment, processor 900 is implemented as a PPU 200. In such an embodiment, each core 450 in an SM 340 includes an HMMA datapath 930 and an FP64 datapath 940, and optionally an integer ALU. Register file 420 may implement one or more register banks 910. A crossbar 915 and an operand collector 920 may be implemented between register file 420 and one or more cores 450. In addition, a result queue 950 may be implemented between one or more cores 450 and an interconnect network 480, which enables results stored in result queue 950 to be written back to register file 420. Thus, processor 900 is a PPU 200 that includes multiple SMs 340, each of which includes a register file 420 and multiple cores 450, each of which includes an instance of HMMA datapath 930.

[0116] The PPU 200 implements a SIMT architecture that enables multiple threads to be executed in parallel on multiple cores 450 in multiple SMs 340. In one embodiment, an MMA operation is configured to execute multiple threads in parallel on multiple cores 450. Each thread is configured to generate a portion of the elements in a result matrix (e.g., collector matrix C 730) on a particular core 450 using a different combination of vectors of operands for the MMA operation specified in the instruction.

[0117] For example, Figure 7 As shown, the MMA operation can be performed on 8×4 input matrix A710 and 4×8 input matrix B720 simultaneously on 8 threads. The first thread is assigned to the first two vectors of input matrix A710 (e.g., 0,0 ,A 0,1 ,A 0,2 ,A 0,3 > and 1,0 ,A 1,1 ,A 1,2 ,A 1,3 >) and the first four vectors of input matrix B 720 (e.g., 0,0 ,B 1,0 ,B 2,0 ,B 3,0 >, 0,1 ,B 1,1 ,B 2,1 ,B 3,1 >, 0,2 ,B 1,2 ,B 2,2 ,B 3,2 > and 0,3 ,B 1,3 ,B​​​​​​2,3 ,B 3,3 >). The first thread generates the eight elements contained in the two vectors of the result matrix (e.g., <C 0,0 ,C 0,1 ,C 0,2 ,C 0,3 > and <C 1,0 ,C 1,1 ,C 1,2 ,C 1,3 >). Similarly, the second thread is assigned to the first two vectors of the input matrix A710 (e.g., 0,0 ,A 0,1 ,A 0,2 ,A 0,3 > and 1,0 ,A 1,1 ,A 1,2 ,A 1,3 >) and the next four vectors of input matrix B 720 (e.g., 0,4 ,B 1,4 ,B 2,4 ,B 3,4 >, 0,5 ,B 1,5 ,B 2,5 ,B 3,5 >, 0,6 ,B 1,6 ,B 2,6 ,B 3,6 >, and 0,7 ,B 1,7 ,B 2,7 ,B 3,7 >). The second thread generates eight elements contained in two different vectors of the result matrix (e.g., <C 0,4 ,C 0,5 ,C 0,6 ,C 0,7 > and <C 1,4 ,C 1,5 ,C 1,6 ,C 1,7 >). The third thread is assigned to the next two vectors of the input matrix A710 (e.g., 2,0 ,A 2,1 ,A 2,2 ,A 2,3 > and 3,0 ,A 3,1 ,A 3,2 ,A 3,3 >) and the first four vectors of input matrix B 720 (e.g., 0,0 ,B 1,0 ,B 2,0 ,B 3,0 ​​​​​​​​​>,

[0118] 0,1 ,B 1,1 ,B 2,1 ,B 3,1 >, 0,2 ,B 1,2 ,B 2,2 ,B 3,2 > and 0,3 ,B 1,3 ,B 2,3 ,B 3,3 >). The third thread generates the eight elements contained in the two vectors of the result matrix (e.g., <C 2,0 ,C 2,1 ,C 2,2 ,C 2,3 > and <C 3,0 ,C 3,1 ,C 3,2 ,C 3,3 The other five threads perform similar operations with additional vector combinations from input matrix A 710 and input matrix B 720.

[0119] It should be understood that each thread is assigned a core 450, and the vectors assigned to that thread are loaded into the operand collector 920 of the core 450. The elements of the result matrix are then generated by performing MMA operations on the HMMA datapath within the core 450. In one embodiment, each core is coupled to a dedicated set of operand collectors 920 that is coupled only to that core 450. In another embodiment, multiple cores 450 share operand collectors 920. For example, two cores 450 with two HMMA datapaths 930 can share a set of operand collectors 920, where common vectors assigned to two threads scheduled on the two cores 450 are shared by the two cores 450. In this way, common vectors assigned to two or more threads are not loaded into two separate sets of operand collectors 920. For example, the first two threads described above are each assigned the first two vectors of input matrix A 710, while being assigned a different set of vectors of input matrix B 720. Thus, operand collectors 920 for storing vectors of the input matrix A 710 may be shared between the two cores 450 by coupling these operand collectors 920 to the inputs of the two HMMA datapaths 930 .

[0120] ​​​It should be understood that any number of threads can be combined to increase the size of the MMA operation. In other words, by adding more threads to process additional calculations in parallel, the dimensions M, N, and K of the MMA operation can be increased without increasing execution time. Alternatively, increasing the size of the MMA operation can also be accomplished on a fixed number of cores 450 by executing multiple instructions on each core 450 over multiple instruction cycles. For example, a first thread can be executed on a specific core 450 during a first instruction cycle, and a second thread can be executed on a specific core 450 during a second instruction cycle. Since the vectors of the input matrix A 710 are shared between the first thread and the second thread, and therefore the vectors of the input matrix A 710 do not need to be reloaded from the register bank 910 to the operand collector 920 between the execution of two instructions over two instruction cycles, it may be beneficial to execute the MMA operation over multiple instruction cycles.

[0121] Figure 10 A conventional double-precision floating-point FMA datapath 1000 is shown according to one embodiment. The conventional double-precision floating-point FMA datapath 1000 illustrates one possible implementation of the FP64 datapath 940 of the processor 900. The datapath 1000 implements an FMA operation that takes three operands (A, B, and C) as inputs, multiplies operand A by operand B, and adds the product to operand C. Each of the three operands is a double-precision floating-point value, encoded using 64 bits: 1 sign bit, 11 exponent bits, and 52 mantissa bits.

[0122] like Figure 10 As shown, data path 1000 includes a multiplier 1010 that multiplies the mantissa bits from the A operand 1002 with the mantissa bits from the B operand 1004. In one embodiment, multiplier 1010 can be, for example, a Wallace tree that multiplies each bit of one mantissa by each bit of the other mantissa and combines the partial products with adder trees in multiple reduction layers to generate two n-bit integers that are added together to produce the binary result of the multiplication. If the multiplier is designed for 64-bit floating point numbers, thereby multiplying two 52-bit mantissas plus a hidden bit (for normalization values), then multiplier 1010 can be a 53×53 Wallace tree that produces two 106-bit values at output, which are added together in a 3:2 carry sum adder (CSA) 1040.

[0123] In parallel, the exponent bits from A operand 1012 are added to the exponent bits from B operand 1014 in adder 1020. Adder 1020 can be a full adder rather than a CSA adder because the exponent bits are only 11 bits wide, and the full addition can be propagated through adder 1020 in a similar amount of time as propagating the result of multiplier 1010 through the simplified layers of the Wallace tree to generate two integers. The result produced by adder 1020 provides an exponent associated with the result of the product of the mantissa bits. The exponent associated with the result is then used to shift the mantissa bits from C operand 1006 based on the exponent bits of C operand 1016. It should be appreciated that the exponent must be the same when performing the addition of mantissa bits, and that incrementing or decrementing the exponent of a floating-point value is equivalent to shifting the mantissa left or right. The shifted mantissa bits of C operand 1006 are then added to the two integers generated by multiplier 1010 in 3:2 CSA 1040. The 3:2CSA 1040 generates a carry value and a sum value representing the result of the addition, wherein each bit of the sum value represents the result of adding three corresponding bits (one bit from each of the three inputs), and wherein each bit of the carry value represents a carry bit that indicates whether the addition of the three corresponding bits results in a carry (i.e., a bit that needs to be added to the next most significant bit in the sum value). The 3:2CSA 1040 enables all carry bits and sum bits to be calculated immediately without having to propagate the carry bit to each subsequent three-bit addition operation.

[0124] The carry value and the sum value from 3:2 CSA 1040 are then added in completion adder 1050. The result produced by completion adder 1050 represents the sum of the three mantissa values from the three operands. However, this sum is not normalized, whereas the floating-point value is normalized. As a result, the result produced by completion adder 1050 is shifted by a number of bits by normalization logic 1060 so that the most significant bit of the result is 1, and the exponent produced by adder 1020 is incremented or decremented by normalization logic 1060 based on the number of bits by which the result is shifted. Finally, the normalized result is rounded by rounding logic 1070. The result produced by completion adder 1050 is much larger than 52 bits. Since the result cannot be losslessly encoded within the 52 mantissa bits of a double-precision floating-point value, the result is rounded so that the mantissa bits of the result are only 52 bits wide.

[0125] It should be understood that Figure 10 Although the sign logic is omitted from the illustration, the sign logic and the operation of the conventional floating-point datapath are well understood by those skilled in the art and should be considered within the scope of datapath 1000. The sign bit, normalized exponent bits, and rounded mantissa bits are output by datapath 1000 and stored in C operand 1008 as a double-precision floating-point result.

[0126] Figure 11 An HMMA datapath 1100 according to one embodiment is shown. HMMA datapath 1100 includes a pair of half-precision floating-point FMA units 1110. Similar to datapath 1000, each of units 1110 implements an FMA operation: taking three operands (A, B, and C) as input, multiplies operand A by operand B, and adds the product to operand C. However, unlike datapath 1000, each of the three operands is a half-precision floating-point value encoded with 16 bits: 1 sign bit, 5 exponent bits, and 10 mantissa bits. Units 1110 are similar in implementation to datapath 1000, except that the components of unit 1110 are significantly smaller than similar components of datapath 1000 because the number of bits in each operand is reduced from 64 bits to 16 bits. Consequently, the multipliers of datapath 1100 can be implemented as 11×11 Wallace trees instead of 53×53 Wallace trees. Similarly, the size of the 3:2 CSA adder, completion adder, normalization logic, and rounding logic is reduced by approximately a factor of 4. Otherwise, the functional description of the datapath 1000 applies equally well to the half-precision floating-point FMA unit 1110, only on operands represented using fewer bits of precision.

[0127] It will be appreciated that each unit 1110 is used to multiply two half-precision floating point values from two input operands and add the product to an addend from a third input operand. Thus, each unit 1110 can be used in parallel to compute partial products of a dot product operation. In one embodiment, the first unit 1110(0) is provided with two input vectors and and the second unit 1110(1) is provided with two input vectors and , wherein each input vector comprises two elements. For example, the first unit 1110 (0) is provided with input vectors and The size of the dot product operation corresponds to the number of units 1110 implemented in parallel. However, summing the partial products produced by each of the units 1110 requires additional combinational logic.

[0128] It should be appreciated that if the HMMA datapath 1100 is implemented as a vector machine, the combinatorial logic can be omitted, and each unit 1110 can perform an FMA operation on a scalar half-precision floating-point value to generate two FMA results at the respective outputs of each unit 1110. In fact, in some embodiments, the HMMA datapath 1100 can be configured to do exactly this. However, additional combinatorial logic is required to implement the dot product operation and generate a single result using the multipliers in both units. Therefore, the HMMA datapath 1100 can be configured in two modes of operation: a first mode in which each unit 1110 performs an FMA operation on the vector inputs in parallel, and a second mode in which each unit 1110 generates a partial product that is passed to the combinatorial logic. The combinatorial logic then adds the partial product to the addend from the third input operand.

[0129] In one embodiment, the combinatorial logic includes exponent comparison logic 1120, product alignment logic 1130, a carry adder tree including a 4:2 CSA 1142 and a 3:2 CSA 1144, a completion adder 1150, normalization logic 1160, and rounding logic 1170. The product alignment logic 1130 receives two integers from each unit 1110 output by the multiplier for the unit 1110. The product alignment logic 1130 is controlled by the exponent comparison logic 1120, which receives the exponent associated with the partial product. In one embodiment, each of the units 1110 includes logic, such as an adder 1020, that combines the two input operands (e.g., A of unit i) with the carry adder tree. i 、B i ) are added. The output of the logic equivalent to adder 1020 is then routed from unit 1110 to exponent comparison logic 1120. Exponent comparison logic 1120 then compares the exponent values associated with the partial products produced by the multipliers in each unit 1110 and uses the difference in the exponents to generate a control signal that causes product alignment logic 1130 to shift one of the partial products produced by the multipliers in each unit 1110. Again, the partial products represent the mantissas of the floating-point values, and therefore, the bits of the partial products must first be aligned so that the exponents match before the addition operation is performed.

[0130] The shifted partial products are then passed to the 4:2 CSA 1142, which adds the four integer values and generates a carry value and a sum value. The output of the 4:2 CSA is passed as two inputs to the 3:2 CSA 1144, which adds the carry value and the sum value to the addend from the third operand C. It should be appreciated that the addend can be in half-precision floating point format or single-precision floating point format, which is encoded in 32 bits: 1 sign bit, 8 exponent bits, and 23 mantissa bits. Remember that the result of multiplying two 11-bit values (10 mantissa bits plus a leading hidden bit) is a 22-bit value. Therefore, even though the partial products are generated based on half-precision floating point values, the width of the partial products is almost the same as the width of the mantissa of the single-precision floating point addend from the third operand. Of course, the addend can also be similar to the input vector and The half-precision floating-point value of the elements.

[0131] The result output by 3:2 CSA 1144 is passed to completion adder 1150, which is similar to completion adder 1050, except that it is smaller in width. The result produced by completion adder 1150 is then passed to normalization logic 1160 and rounding logic 1170 to shift and truncate the result. Normalization logic 1160 receives the value of the common exponent used for the two partial products after the shift and shifts the result by incrementing or decrementing the exponent value and shifting the bits of the result left or right until the MSB of the result is 1. Rounding logic 1170 then truncates the result to fit within the width of at least one floating-point format's mantissa bits. The sign bit, normalized exponent bits, and rounded mantissa bits are output by datapath 1100 and stored in C operand 1108 as a half-precision floating-point value or a single-precision floating-point value.

[0132] Returning to the top of the data path 1100, it is apparent that the selection logic 1105 is coupled to the inputs of the three operands of each unit 1110. As described above, the two-element vector and a two-element vector The addition of the scalar operand C may be used to perform a dot product operation using the datapath 1100. Although a datapath that can be configured to perform dot product operations is generally more useful than a datapath that can only be configured to perform FMA operations, additional functionality is added by including selection logic 1105 that makes FMA operations more efficient when the datapath 1100 is coupled to the additional operand collector 920.

[0133] For example, the operand collector 920 coupled to the data path 1100 may include a plurality of operand collectors 920 sufficient to store at least two input vectors associated with the input matrix A 710 . and at least two input vectors associated with input matrix B 720 The one or more vectors associated with the multiple elements of the collector operand C 730 are added. The selection logic 1105 is then used to select elements from the different vectors stored in the operand collector 920 to perform multiple dot product operations in multiple passes of the data path 1100, all of which are associated with a single instruction cycle of the data path 1100. The selection logic 1105 may include a plurality of multiplexers and control logic for switching the multiplexers between two or more inputs of each multiplexer.

[0134] For example, during the first pass, the first input vector is selected and the first input vector Each input vector has two elements, and the first element / addend from the collector matrix C is selected to generate the first dot product result. During the second pass, the first input vector is selected and the second input vector and the second element / addend from the collector matrix C to generate the second dot product result. During the third pass, the second input vector is selected and the first input vector and the third element / addend from the collector matrix C to generate the third dot product result. Finally, during the fourth pass, the second input vector is selected and the second input vector and the fourth element / addend from collector matrix C to generate a fourth dot product result. These results can be stored in a result queue 950 having a width of 64 or 128 bits, depending on whether the dot product results stored in collector matrix C are encoded as half-precision floating point values or single-precision floating point values.

[0135] In another embodiment, the 4:2 CSA 1142 and the 3:2 CSA 1144 can be combined into a 5:2 CSA. Although the actual difference between the two embodiments is minimal, since the 4:2 CSA is typically implemented as a tree of 3:2 CSAs and the 5:2 CSA is also typically implemented as a tree of 3:2 CSAs, there is a slight difference in the order that controls how the five parameters are added.

[0136] In yet another embodiment, the product alignment logic 1130 can be configured to truncate the partial products as they are shifted, thereby reducing the size of the CSA configured to sum the aligned partial products. To shift the partial products without truncation, the product alignment logic 1130 needs to output additional width of the partial products to the 4:2 CSA 1142. To avoid increasing the width of the partial products, the product alignment logic 1130 can shift the partial products to a wider bit width and then truncate to the MSB before transferring them to the 4:2 CSA 1142. This results in a reduction in the size of the required CSA for summing the partial products and the addends.

[0137] In yet another embodiment, the data path 1100 can be scaled to generate dot products for vectors with more than two elements. Typically, for a pair of p-element vectors, the data path 1100 can include p units 1110 for computing p partial products and additional combinational logic for combining all the partial products. For example, Figure 11 The logic shown in can be doubled to generate the two parts of the dot product, and then an additional simplified layer of combinatorial logic can be included to combine the sum of two partial products with the sum of two other partial products.

[0138] It should be appreciated that the addend of a dot product operation with two input vectors may be provided as input to only one of the units 1110 in the datapath 1100. All other units 1110 of the datapath 1100 should receive a constant value of zero as the addend operand of the unit 1110 so that the addend is added to the dot product result only once.

[0139] It should be appreciated that when datapath 1100 is configured to use additional combinational logic to generate dot product results, the CSA, completion adder, normalization logic, and rounding logic of each unit 1110 are not utilized. However, when datapath 1100 is configured as a vector machine to generate vectors of FMA results, this logic is used. It should be appreciated that when datapath 1100 is configured to generate dot product results, 4:2 CSA adder 1142, 3:2 CSA adder 1144, completion adder 1150, normalization logic 1160, and rounding logic 1170 are very similar to the unused logic in each unit, albeit with different precision. If possible, it is useful to utilize logic within unit 1110 to perform a portion of the same operation as the additional combinational logic.

[0140] Figure 121 shows an HMMA data path 1200 according to another embodiment. The HMMA data path 1200 includes a "small" half-precision floating-point FMA unit 1210 and a "large" half-precision floating-point FMA unit 1220. The small unit 1210 is similar to each of the units 1110 and implements an FMA operation: taking three operands (A, B, and C) as inputs, multiplying operand A by operand B and adding the product to operand C. The large unit 1220 is similar to the small unit 1210 in that the large unit 1220 implements an FMA operation: taking three operands (A, B, and C) as inputs, multiplying operand A by operand B and adding the product to operand C. However, the large unit 1220 internally includes slightly different logic to implement both half-precision floating-point FMA operations and combinational logic to implement dot product operations in conjunction with the small unit 1210.

[0141] like Figure 12 As shown, the partial products generated by small unit 1210 are output to a first partial product parser 1232. In one embodiment, partial product parser 1232 is a completion adder that combines the two integers generated by the multiplier into a final value representing the product of the two mantissas of the first partial product. Similarly, the partial products generated by large unit 1220 are output to a second partial product parser 1234, which is similar to first partial product parser 1232. The output of first partial product parser 1232 and the first of the two integers of the second partial product generated by the multiplier in large unit 1220 are coupled to a first switch, and the output of second partial product parser 1234 and the second of the two integers of the second partial product generated by the multiplier in large unit 1220 are coupled to a second switch. The first and second switches control whether large unit 1220 is configured in a first mode to generate the result of a scalar FMA operation or in a second mode to generate the result of a vector dot product operation.

[0142] The outputs of the first and second switches are coupled to product alignment logic 1240, which is configured to shift the partial products passed as inputs via the first and second switches when the large unit 1220 is configured in the second mode. Product alignment logic 1240 is controlled by exponent comparison logic 1245, which operates similarly to exponent comparison logic 1120. If the large unit 1220 is configured in the first mode, product alignment logic 1240 does not shift either of the two integers passed to product alignment logic 1240 via the first and second switches. When the small unit 1210 and the large unit 1220 are operating as vector machines to generate scalar FMA results, no shifting is performed because the exponents associated with the two partial products are unrelated.

[0143] The aligned partial products are then passed to the 3:2 CSA 1250, which adds the two partial products with the addend from the third input operand. The 3:2 CSA 1250 in the large unit 1220 can be significantly wider (i.e., have greater precision) than the corresponding CSA in the small unit 1210. This is necessary to handle the additional bits of precision required for lossless operation of the partial product parsers 1232, 1234 and the product alignment logic 1240.

[0144] In one embodiment, the data path 1200 also includes selection logic 1205 for selecting between multiple combinations of at least two vectors from input matrix A 710 and at least two vectors from input matrix B 720 and different elements / addends from collector matrix C 730 to generate multiple dot product results in the result queue 950.

[0145] Figure 13 FIG. 13 shows an HMMA data path 1300 according to yet another embodiment. The HMMA data path 1300 includes four multipliers 1310 for generating two four-element vectors and partial products of ; four negation logic 1320 blocks for merging sign bits of operands; five shift logic 1330 blocks for shifting the partial products and addends to align all values based on the exponent of the partial products; a CSA tree including multiple simplified layers of 3:2 CSA 1342 and 4:2 CSA 1344; completion adder 1350; normalization logic 1360; and rounding logic 1370. The data path 1300 includes multiple pipeline stages: a first pipeline stage 1301, which includes conversion / encoding logic 1315; a second pipeline stage 1302, which includes a multiplier 1310; a third pipeline stage 1303, which includes negation logic 1320, shift logic 1330 and a CSA tree; a fourth pipeline stage 1304, which includes a completion adder; a fifth pipeline stage 1305, which includes normalization logic 1360; and a sixth pipeline stage 1306, which includes rounding logic 1370.

[0146] In the first pipeline stage 1301, the conversion / encoding logic 1315 receives the elements of the two input vectors and the elements / addends of the collector matrix C 730 and performs one or more pre-processing operations on these elements. The pre-processing may involve converting the elements from one format to a half-precision floating point value format. For example, the input vectors may be provided in 16-bit floating point, 8-bit signed / unsigned integer, 16-bit signed / unsigned integer, 32-bit fixed point format, etc. and The conversion / encoding logic 1315 is configured to convert all input values to a half-precision floating point value format for compatibility with the rest of the data path 1300 .

[0147] In one embodiment, the conversion / encoding logic 1315 may also include a modified Booth encoder. The modified Booth encoder encodes a value for each three bits of the multiplicand (e.g., The selector signal is generated by the bit of the element of the vector. The selector signal is then passed to the multiplier 1310, which is designed to implement a modified Booth algorithm to generate partial products. The modified Booth algorithm can speed up the multiplier 1310 by reducing the number of simplification layers (adders) in the multiplier 1310. It should be appreciated that in some embodiments, the data paths 1100 and 1200 can also be modified to merge the conversion / encoding logic 1315 and the multiplier designed to implement the modified Booth algorithm.

[0148] In the second pipeline stage 1302, each of the multipliers 1310 takes two input vectors and Receive corresponding pairs of corresponding elements. For example, the first multiplier 1310 receives elements A0 and B0; the second multiplier 1310 receives elements A1 and B1; the third multiplier 1310 receives elements A2 and B2; and the fourth multiplier 1310 receives elements A3 and B3. Each of the multipliers 1310 generates two integers representing the partial products formed by multiplying the elements input to the multiplier 1310.

[0149] In the third pipeline stage 1303, negation logic block 1320 combines the sign bits of the elements input to the corresponding multiplier 1310 and, if the combined sign bit is negative, negates the partial product via a two's complement operation applied to the pair of integers. For example, negation logic block 1320 may perform an exclusive OR (XOR) on the sign bits of the two elements input to the corresponding multiplier 1310. If the partial product is negative, the result of the XOR operation is 1. If the partial product is positive, the result of the XOR operation is 0. If the result of the XOR operation is 1, the two integers from multiplier 1310 are negated by performing a two's complement operation on each value (i.e., switching the state of each bit in the value and then adding 1 to the result). It should be appreciated that in some embodiments, negation logic block 1320 may be implemented in a similar manner in datapaths 1100 and 1200 to handle the sign bits of various operands.

[0150] The shift logic 1330 block shifts the partial products based on the exponents associated with all four partial products. Although not explicitly shown, an adder (such as adder 1020) is used to calculate the exponent associated with each partial product by summing the exponent bits included in the elements associated with the corresponding multiplier 1310. The maximum exponent associated with all four partial products and the addend is provided to each of the shift logic 1330 blocks. Each shift logic 1330 block then determines the number of bits by which to shift the partial product corresponding to that shift logic 1330 block in order to align the partial product with the maximum exponent of all partial products. One of the shift logic 1330 blocks also shifts the mantissa of the addend. It should be appreciated that the maximum possible shift distance in bits will increase the required bit width of the partial product integers passed to the CSA tree to avoid loss of precision. In one embodiment, the shift logic 1330 block is configured to truncate aligned partial products to reduce the precision of the adders in the CSA tree.

[0151] The CSA tree consists of multiple simplification levels where three or four inputs are added to generate two outputs, a carry value and a sum value. Figure 13 As shown, a 4-element dot product operation requires three simplification levels, including three 3:2 CSAs 1342 at the first simplification level; two 3:2 CSAs 1342 at the second simplification level; and one 4:2 CSA 1344 at the third simplification level. The output of the 4:2 CSA 1344 at the third simplification level generates a carry value and a sum value for the dot product of the two vectors, which are added to the addend. In the fourth pipeline stage 1304, the carry value and the sum value are passed to the completion adder 1350, which sums the carry value and the sum value to produce the mantissa value of the dot product. In the fifth pipeline stage 1305, the result is passed to the normalization logic 1360, which normalizes the mantissa value of the dot product and adjusts the maximum exponent value of the dot product based on the alignment. In the sixth pipeline stage 1306 , the normalized mantissa and exponent values are passed to rounding logic 1370 , which rounds the result to the width of the format of the elements of the collector matrix C730 (eg, half precision or single precision).

[0152] Although not explicitly shown, selection logic similar to selection logic 1105 and 1205 can be coupled to conversion / encoding logic 1315 so that multiple vectors stored in operand collector 920 can be used to generate dot product results of four-element vectors on two or more passes through data path 1300 during a single instruction cycle.

[0153] In another embodiment, Figure 13The logic shown may be replicated one or more times to generate multiple dot product results in parallel using shared elements from operand collector 920. For example, data path 1300 may include four four-element dot product logic units that are connected to Figure 13 All four dot product logic cells share the same vector, but loaded with different vector to generate four dot product results in parallel for different elements of the collector matrix C 730. In addition, during a given instruction cycle, multiple passes of the data path 1300 may be used to generate different operands stored in the operand collector 920. The dot product of the vectors.

[0154] In yet another embodiment, Figure 13 Eight four-element dot product logic units matching the logic shown in can be included in the data path, enabling the generation of the corresponding two Vector and four The eight dot product values of the vector. It should be understood that Figure 13 Any number of copies of the logic shown in can be implemented in a single datapath 1300 to generate multiple separate and distinct dot product values in parallel. Each copy of the logic can then be paired with any two four-element vectors stored in the operand collector 920 to generate a dot product value for the two vectors. Each copy of the logic can also be coupled to selection logic that enables different vector pairs stored in the operand collector to be consumed by copies of the logic in multiple passes through the datapath during a single instruction cycle.

[0155] Figure 14 According to one embodiment, a Figure 10 The double-precision floating-point FMA data paths share at least one pipeline stage. Figure 131300. It will be appreciated that when analyzing the architecture of datapaths 1000, 1100, 1200, and 1300, the fourth pipeline stage 1304, the fifth pipeline stage 1305, and the sixth pipeline stage 1306 appear relatively familiar, as described above. Essentially, these pipeline stages all include a completion adder, normalization logic, and rounding logic to convert the carry and sum values generated for the dot product to fit the specific format of the elements of collector matrix C 730. The only difference between the logic in each of datapaths 1000, 1100, 1200, and 1300 is the precision of the logic. However, the completion adder 1050 of double-precision floating-point FMA datapath 1000 will be larger than the completion adders of HMMA datapaths 1100, 1200, and 1300. Thus, the datapath 1300 may be simplified in any architecture where a core includes both the HMMA datapath 1300 and the double-precision floating-point FMA datapath 1000 coupled to the same operand collector 920 and result queue 950 .

[0156] In one embodiment, core 450 includes both HMMA datapath 1300 and double-precision floating-point FMA datapath 1000. However, HMMA datapath 1300 is modified to omit fourth pipeline stage 1304, fifth pipeline stage 1305, and sixth pipeline stage 1306. Instead, the output of third pipeline stage 1303 (i.e., the carry value and sum value representing the dot product value output by the CSA tree) is routed to a pair of switches included in FMA datapath 1000. This pair of switches enables FMA datapath 1000 to sum either the dot product value from HMMA datapath 1300 or the FMA result from FMA datapath 1000 using a completion adder. Thus, HMMA datapath 1300 shares the pipeline stages of FMA datapath 1000, including the completion adder, normalization logic, and rounding logic. It should be appreciated that, although not explicitly shown, the maximum exponent value associated with the dot product may also be sent to a switch in the FMA datapath 1000 so that the normalization logic can convert between the exponent value produced by the adder 1020 and the maximum exponent value associated with the dot product generated by the HMMA datapath 1300.

[0157] Sharing logic between the two datapaths, as well as sharing operand collector 920 and result queue 950, can significantly reduce the die footprint of core 450 on an integrated circuit. Consequently, more cores 450 can be designed on a single integrated circuit die.

[0158] As previously mentioned, various datapaths can be designed to implement MMA operations more efficiently than in current datapath designs, such as scalar FMA datapaths and even vector machines configured to compute partial products in parallel. A key aspect of this design is that more than one pair of vectors can be loaded from a register file and coupled to the input of the datapath, thereby generating multiple dot product values within a single instruction cycle. As used herein, an instruction cycle refers to all operations related to loading an operand collector with multiple operands from a register file, then performing an MMA operation on the datapath to generate multiple dot product values corresponding to different elements of the result matrix, and then writing the multiple dot product values to the register file. Each instruction cycle can include multiple passes through the datapath to generate results for the combination of different vector pairs across the multiple passes. In addition, each pass can be pipelined so that the second pass begins before the first pass is completed. Instructions for MMA operations can be implemented over multiple instruction cycles, with different portions of the input matrix operands being loaded into the operand collector of the datapath during each instruction cycle. Thus, instructions for MMA operations may include matrix operands of arbitrary size that are processed over multiple instruction cycles and / or multiple cores, wherein the multiple cores apply a different vector from the matrix operand to each datapath during each instruction cycle until all vectors from the matrix operand have been processed.

[0159] Known applications of MMA operations include image processing (e.g., performing affine transformations on images), machine learning (e.g., using matrix operations when performing linear algebra, optimizing functions, or calculating statistics), and others. Matrix algebra is a fundamental field that can be applied to a wide variety of applications. Therefore, improving the processing efficiency of MMA operations by designing processors that can perform these operations faster is highly beneficial for the speed and efficiency of computing processing.

[0160] More specifically, MMA operations performed using the disclosed datapath exhibit better numerical behavior and / or provide greater efficiency for the processor implementing the datapath. For example, parallel accumulation of partial products using a single adder eliminates multiple rounding operations compared to using a serial adder that performs rounding each time as part of an accumulation operation. For vectors of any length, the worst-case error bound can be pushed to one (or half) unit of machine precision, while the serial multiply-add-add-add datapath implemented in a conventional dot product datapath exhibits a worst-case error bound that is proportional to the vector length.

[0161] Furthermore, the disclosed datapath exhibits lower latency than prior art datapaths. Fewer pipeline stages require the implementation of a smaller number of flip-flops, which improves the power consumption of the datapath. Because the datapath reuses operand vectors, a smaller register file is required to implement the same MMA operation that could be implemented with a traditional datapath serial operation. Furthermore, the internal shifter and adder lengths can be simplified to match the desired error bounds, further reducing the number of flip-flops in the datapath. Furthermore, energy can be saved by simply updating some matrix operands at the operand collector at the input of the datapath, chaining dot product operations to generate larger dot product results in the result queue, without having to force the intermediate results to be written back to the register file and then reloading the intermediate results from the register file back into the operand collector at the input of the datapath.

[0162] Figure 15 An exemplary system 1500 is shown in which various architectures and / or functions of various previous embodiments can be implemented. As shown, a system 1500 is provided that includes at least one central processor 1501 connected to a communication bus 1502. The communication bus 1502 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol. The system 1500 also includes a main memory 1504. Control logic (software) and data are stored in the main memory 1504, which can take the form of random access memory (RAM).

[0163] System 1400 also includes an input device 1512, a graphics processor 1506, and a display 1508, such as a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display, etc. User input can be received from input device 1512, such as a keyboard, mouse, touchpad, microphone, etc. In one embodiment, graphics processor 1506 may include multiple shader modules, rasterization modules, etc. Each of the aforementioned modules can even be located on a single semiconductor platform to form a graphics processing unit (GPU).

[0164] In this specification, a single semiconductor platform may refer to a single semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity, which simulates on-chip computing and is a significant improvement over conventional central processing unit (CPU) and bus implementations. Of course, various modules may also be located individually or in various combinations of semiconductor platforms, depending on user needs.

[0165] System 1500 may also include secondary storage 1510. Secondary storage 1510 includes, for example, a hard drive and / or a removable storage drive, such as a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0166] Computer programs or computer control logic algorithms may be stored in the main memory 1504 and / or the secondary memory 1510. These computer programs, when executed, enable the system 1500 to perform various functions. The memory 1504, memory 1510, and / or any other memory are possible examples of computer-readable media.

[0167] In one embodiment, the architecture and / or functionality of the various previous figures may be implemented within the context of a central processing unit 1501, a graphics processor 1506, an integrated circuit (not shown) having at least a portion of the capabilities of both the central processing unit 1501 and the graphics processor 1506, a chipset (i.e., a group of integrated circuits designed to operate and be sold as a unit for performing related functions, etc.), and / or any other integrated circuit for this purpose.

[0168] Still, the architecture and / or functionality of the various previous figures can be implemented in a general-purpose computer system, a circuit board system, a game console system dedicated to entertainment purposes, a dedicated system, and / or any other desired system. For example, the system 1500 can take the form of a desktop computer, a laptop computer, a server, a workstation, a game console, an embedded system, and / or any other type of logic. Furthermore, the system 1500 can take the form of various other devices, including but not limited to a personal digital assistant (PDA) device, a mobile phone device, a television, and the like.

[0169] Furthermore, although not shown, system 1500 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable television network, etc.) for communication purposes.

[0170] Although various embodiments have been described above, it should be understood that they are presented by way of example only, and not limitation. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A multi-threaded processor, comprising: Decoder to decode floating-point matrix multiply and accumulate (MMA) instructions; a buffer for storing 16-bit floating-point matrix data specified by an operand of the MMA instruction; A scheduler, configured to schedule the MMA instructions; a fused multiply-accumulate (FMA) unit to perform dot products of corresponding pairs of the 16-bit floating-point matrix data; logic to shift a plurality of partial products of the dot product; an arithmetic logic unit (ALU) for adding the plurality of partial products to be accumulated to a register; as well as A memory is used to store the result of the MMA instruction. 2 . The multi-threaded processor of claim 1 , wherein the logic for shifting the plurality of partial products of the dot product aligns the plurality of partial products based on an exponent associated with each element in the corresponding pair.

3. The multi-threaded processor of claim 1 , further comprising an adder tree, wherein the adder tree comprises at least 3:2 carry sum adders (CSAs).

4. The multi-threaded processor of claim 1 , further comprising a dispatch unit to transmit the MMA instruction to the FMA unit.

5. The multi-threaded processor of claim 1, further comprising a register file to provide the registers. The multi-threaded processor of claim 1 , further comprising a special function unit (SFU).

7. The multi-threaded processor of claim 1 further comprising an interconnect for connecting the ALU and the registers.

8. A system having a multi-threaded processor, comprising the multi-threaded processor according to claim 1, wherein the system further comprises: a system bus for connecting the multi-threaded processor to one or more peripheral devices; and One or more dynamic random access memory (DRAM) devices.

9. A single instruction multiple data (SIMD) multi-threaded processor, comprising: A plurality of cores are configured to execute floating-point matrix multiply-accumulate (MMA) instructions, wherein each of the plurality of cores comprises: Front-end, used to obtain MMA instructions; an instruction cache for storing the MMA instructions; L1 cache, used to store data; L2 cache, used to store data; a plurality of ports for reading from and writing to the memory; one or more load / store units to read from and write to the memory; an interconnection for coupling the memory and the plurality of cores; A decoder for decoding the MMA instruction; a buffer for storing 16-bit floating-point matrix data specified by an operand of the MMA instruction; A scheduler, configured to schedule the MMA instructions; a fused multiply-accumulate (FMA) unit to perform dot products on corresponding pairs of the 16-bit floating-point matrix data; logic to shift a plurality of partial products of the dot product; an arithmetic logic unit (ALU) for adding the plurality of partial products to be accumulated into a register; and The memory is used to store the result of the MMA instruction.

10. The SIMD multi-threaded processor of claim 9, wherein the L1 cache comprises at least 24 kilobytes (KB) of storage.

11. The SIMD multi-threaded processor of claim 9, wherein the memory comprises at least 64 kilobytes (KB) of storage.

12. The SIMD multi-threaded processor of claim 9, wherein the interconnect connects the one or more load / store units to the registers.

13. The SIMD multi-threaded processor of claim 9, wherein the scheduler dispatches the MMA instruction to one or more cores of the plurality of cores.

14. The SIMD multi-threaded processor of claim 9, wherein the logic to shift the plurality of partial products of the dot product aligns the plurality of partial products based on an exponent associated with each element of the corresponding pair.

15. A computer-implemented method comprising: The decoder decodes the floating-point matrix multiply and accumulate MMA instruction; Storing, by a buffer, 16-bit floating-point matrix data specified by an operand of the MMA instruction; The MMA instruction is scheduled by a scheduler; performing dot products on corresponding pairs of the 16-bit floating-point matrix data using a fused multiply-accumulate (FMA) unit; logically shifting a plurality of partial products of the dot product; adding the plurality of partial products to be accumulated to a register by an arithmetic logic unit (ALU); and The result of the MMA instruction is stored by the memory.

16. The computer-implemented method of claim 15, wherein the ALU comprises at least one adder.

17. The computer-implemented method of claim 15, wherein shifting the plurality of partial products further comprises aligning the plurality of partial products based on an exponent associated with each element of the corresponding pair.

18. The computer-implemented method of claim 15, wherein the scheduler comprises a dispatch unit to dispatch the MMA instructions.

19. The computer-implemented method of claim 15, further comprising accumulating the plurality of partial products into the register using an interconnect.

20. The computer implemented method of claim 15, wherein a register file provides the registers.

Images