Apparatuses, methods, and systems for instructions of a matrix operations accelerator
Patent Information
- Application Number
- KR1020240002721
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-12-28
- Filing Date
- 2024-01-08
- Publication Date
- 2026-09-18
- Estimated Expiration
- 2040-09-23
Smart Images
Figure 112024002359739-PAT00026_ABST
Abstract
Description
Technology Field
[0001] The present disclosure generally relates to computer processor architectures, and more specifically to devices, systems, and methods for executing instructions for performing matrix operations using matrix operation accelerator circuits. Background Technology
[0002] A processor, or a set of processors, executes instructions from an instruction set, for example, an instruction set architecture (ISA). An instruction set is part of a computer architecture related to programming and generally includes native data types, instructions, register architecture, addressing modes, memory architecture, interrupt and exception handling, and external input and output (I / O). It should be noted that in this specification, the term "instruction" may refer to a macro instruction, for example, an instruction provided to a processor for execution, or a micro instruction, for example, an instruction originating from a processor's decoder that decodes macro instructions. Brief explanation of the drawing
[0003] The present disclosure is not limited to but is illustrated by example in the drawings of the accompanying drawings in which similar reference numbers indicate similar elements. FIG. 1a illustrates an example of tiles configured according to embodiments of the present disclosure. FIG. 1b illustrates an example of tiles configured according to embodiments of the present disclosure. FIG. 2 illustrates several examples of matrix storage according to embodiments of the present disclosure. FIG. 3 illustrates an embodiment of a system utilizing a matrix (tile) operation accelerator according to embodiments of the present disclosure. FIGS. 4 and 5 illustrate different embodiments of a method in which memory is shared using a matrix operation accelerator. FIG. 6 illustrates an example of a matrix multiplication accumulation operation ("TMMA") using tiles. FIG. 7 illustrates an example of a subset of iterative executions of chained and fused multiplication accumulator instructions. FIG. 8 illustrates an example of a subset of iterative executions of chained and unified multiplication accumulator instructions. FIG. 9 illustrates an example of a subset of iterative executions of chained and unified multiplication accumulator instructions. FIG. 10 illustrates an example of a subset of iterative executions of chained and unified multiplication accumulator instructions. FIG. 11 illustrates SIMD implementations of powers of 2, wherein the accumulators use input sizes larger than the inputs to the multipliers, depending on the embodiment. FIG. 12 illustrates an example of a system utilizing a matrix operation circuit. FIG. 13 illustrates an example of a processor core pipeline that supports matrix operations using tiles. FIG. 14 illustrates an example of a processor core pipeline that supports matrix operations using tiles. Figure 15 illustrates an example of a matrix represented in row major format and column major format. Figure 16 illustrates an example of the use of matrices (tiles). FIG. 17 illustrates an example of how matrices (tiles) are used. FIG. 18 illustrates support for the configuration of the use of tiles according to an embodiment. FIG. 19 illustrates an example of the description of matrices (tiles) to be supported. FIGS. 20a through 20d illustrate examples of register(s). FIG. 21 illustrates an embodiment of a system utilizing a matrix (tile) operation accelerator according to embodiments of the present disclosure. FIG. 22 illustrates a matrix operation accelerator circuit comprising a two-dimensional grid of processing element circuits according to embodiments of the present disclosure. FIG. 23 illustrates a dispatch circuit of a matrix operation accelerator circuit according to embodiments of the present disclosure. FIG. 24 illustrates a scheduling circuit of a dispatch circuit of a matrix operation accelerator circuit according to embodiments of the present disclosure. FIG. 25 illustrates a scheduling circuit of a dispatch circuit of a matrix operation accelerator circuit that is capable of switching from a baseline scheduling mode to a scheduling mode that reuses an input matrix, according to embodiments of the present disclosure. FIG. 26 illustrates a dispatch circuit of a matrix operation accelerator circuit for multiple paths according to embodiments of the present disclosure. FIG. 27 illustrates a scheduling circuit of a dispatch circuit of a matrix operation accelerator circuit for multiple passes according to embodiments of the present disclosure. FIG. 28 illustrates pseudocode for a matrix operation circuit according to embodiments of the present disclosure. FIG. 29 illustrates a method for processing matrix operation instructions according to embodiments of the present disclosure. FIG. 30a is a block diagram illustrating a generic vector-friendly instruction format and its Class A instruction templates according to embodiments of the present disclosure. FIG. 30b is a block diagram illustrating a general vector-friendly instruction format and its Class B instruction templates according to embodiments of the present disclosure. FIG. 31a is a block diagram illustrating fields for general vector-friendly instruction formats in FIG. 30a and FIG. 30b according to embodiments of the present disclosure. FIG. 31b is a block diagram illustrating fields of a specific vector-friendly instruction format in FIG. 31a that constitute a full opcode field according to one embodiment of the present disclosure. FIG. 31c is a block diagram illustrating fields of a specific vector-friendly instruction format in FIG. 31a that constitute a register index field according to one embodiment of the present disclosure. FIG. 31d is a block diagram illustrating fields of a specific vector-friendly instruction format in FIG. 31a that constitute an augmentation operation field (3050) according to one embodiment of the present disclosure. FIG. 32 is a block diagram of a register architecture according to one embodiment of the present disclosure. FIG. 33a is a block diagram illustrating both an exemplary sequential pipeline and an exemplary register renaming, non-sequential publish / execute pipeline according to embodiments of the present disclosure. FIG. 33b is a block diagram illustrating both an exemplary embodiment of a sequential architecture core and an exemplary register renaming, non-sequential publish / execute architecture core to be included in a processor according to embodiments of the present disclosure. FIG. 34a is a block diagram showing a single processor core with its connection to an on-die interconnect network and its local subset of a Level 2 (L2) cache, according to embodiments of the present disclosure. FIG. 34b is an enlarged view of a portion of the processor core of FIG. 34a according to embodiments of the present disclosure. FIG. 35 is a block diagram of a processor that may have two or more cores, may have an integrated memory controller, and may have integrated graphics, according to embodiments of the present disclosure. FIG. 36 is a block diagram of a system according to one embodiment of the present disclosure. FIG. 37 is a block diagram of a more specific exemplary system according to an embodiment of the present disclosure. FIG. 38 is a block diagram of a second, more specific exemplary system according to an embodiment of the present disclosure. FIG. 39 is a block diagram of a System-on-a-Chip (SoC) according to an embodiment of the present disclosure. FIG. 40 is a block diagram illustrating the use of a software instruction converter to convert binary instructions in a source instruction set into binary instructions in a target instruction set according to embodiments of the present disclosure. Specific details for implementing the invention
[0004] In the following description, many specific details are provided. However, it is understood that embodiments may be practiced without such details. In other cases, known circuits, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.
[0005] References to “one embodiment,” “an embodiment,” “an exemplary embodiment,” etc., in this specification indicate that while the described embodiment may include a specific feature, structure, or characteristic, not all embodiments necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in connection with one embodiment, it is disclosed that it is within the knowledge of a person skilled in the art that such feature, structure, or characteristic is influenced in connection with other embodiments, whether or not explicitly described.
[0006] Matrices are becoming increasingly important in many computing tasks, such as machine learning and other bulk data processing. Deep learning is a class of machine learning algorithms. Deep learning architectures, such as deep neural networks, can be applied to fields including computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, bioinformatics, and drug design.
[0007] Inference and training, the two tools used for deep learning, can utilize low-precision arithmetic. Maximizing the throughput of deep learning algorithms and computations can help meet the demands of deep learning processors, for example, those performing deep learning in data centers.
[0008] Matrix-matrix multiplication (also known as GEMM or General Matrix Multiplication) is a computationally intensive operation for certain processors. Specialized hardware for matrix multiplication (e.g., GEMM) is a good option for improving peak computation (and energy efficiency) for certain applications, such as deep learning. Some of these applications, including deep learning, can operate on input data components with relatively few bits without loss of accuracy, as long as the output components have sufficient bits (e.g., more than the inputs).
[0009] On certain processors, handling matrices is a difficult and / or instruction-intensive task. For example, the rows of a matrix can be placed into multiple packed data (e.g., SIMD or vector) registers and then operated on individually. For instance, the addition of two 8x2 (e.g., row x column) matrices may require loading or gathering four packed data registers, depending on the data sizes. Then, a first addition of the packed data registers corresponding to the first row of each matrix is performed, followed by a second addition of the packed data registers corresponding to the second row of each matrix. The resulting packed data registers are then returned to memory and scattered. While this scenario may be acceptable for small matrices, it is often unacceptable for larger matrices.
[0010] argument
[0011] Mechanisms for supporting matrix operations in computer hardware, such as central processing units (CPUs), graphics processing units (GPUs), and accelerators, are described herein. Matrix operations utilize two-dimensional (2-D) data structures that represent one or more packed regions of memory, such as registers. Throughout this description, these 2-D data structures are referred to as tiles. Note that a matrix may be smaller than a tile (smaller than all of the tiles) or may utilize multiple tiles (larger than any single tile). Throughout this description, the matrix (tile) language is used to denote operations performed using tiles that affect the matrix. Whether the matrix is larger than any single tile is typically irrelevant.
[0012] Each tile may be subjected to different operations as described herein, including but not limited to matrix (tile) multiplication, tile addition, tile subtraction, tile diagonal, tile zero, tile transformation, tile dot product, tile broadcast, tile row broadcast, tile column broadcast, tile multiplication, tile multiplication and accumulation, tile shift, etc. Additionally, support for operators such as the use of scale and / or bias may be used in conjunction with these operations or in the future to support non-numerical applications such as OpenCL "local memory", data compression / decompression, etc. Also described herein are instructions for performing matrix operations (e.g., TILEPARTIALDOTPRODUCT).
[0013] Parts of storage (e.g., memory (non-volatile and volatile), registers, cache, etc.) are arranged as tiles with different horizontal and vertical dimensions. For example, a tile may have 4 horizontal dimensions (e.g., 4 rows of a matrix) and 8 vertical dimensions (e.g., 8 columns of a matrix). Typically, the horizontal dimensions correspond to component sizes (e.g., 2-, 4-, 8-, 16-, 32-, 64-, 128-bit, etc.). Multiple data types (single-precision floating-point, double-precision floating-point, integer, etc.) may be supported.
[0014] Exemplary use of constructed tiles
[0015] In some embodiments, tile parameters may be configured. For example, a given tile may be configured to provide tile options. Exemplary tile options include, but are not limited to, the number of rows of the tile, the number of columns of the tile, whether the tile is VALID, and whether the tile consists of a PAIR of tiles of the same size.
[0016] FIG. 1a illustrates an example of configured tiles. As illustrated, 4 kB of application memory (102) stores four 1 kB titles thereon: tile t0 (104), tile t1 (106), tile t2 (108), and tile t3 (110). In this example, the four tiles are not configured in pairs, and each has components arranged in rows and columns. Tile t0 (104) and tile t1 (106) have K rows and N columns of 4-byte components (e.g., single-precision data), where K is 8 and N = 32. Tile t2 (108) and tile t3 (110) have K rows and N / 2 columns of 8-byte components (e.g., double-precision data). Since double-precision operands are twice the width of single-precision operands, this configuration matches the palette used to provide tile options and supplies at least four names with a total storage of at least 4 kB. During operation, tiles can be loaded from memory and stored in memory using load and store operations. Depending on the instruction encoding scheme used, the amount of available application memory, as well as the size, number, and configuration of available tiles, vary.
[0017] FIG. 1b illustrates an example of the configured tiles. As illustrated, 4 kB of application memory (122) stores two pairs of 1 kB titles, the first pair being tile t4L (124) and tile t4R (126), and the second pair being tile t5L (128) and tile t5R (130). As illustrated, the tile pairs are divided into left tiles and right tiles. In other embodiments, the tile pairs are divided into even tiles and odd tiles. In this example, each of the four tiles has components arranged in rows and columns. Tile t4L (124) and tile t4R (126) have K rows and N columns of 4-byte components (e.g., single-precision floating-point data), where K is equal to 8 and N is equal to 32. Tile t5L (128) and tile t5R (130) have K rows and N / 2 columns of 8-byte components (e.g., double-precision floating-point data). Since double-precision operands are twice the width of single-precision operands, this configuration matches the palette used to provide tile options and supplies at least two names with a total storage of at least 4 kB. The four tiles in FIG. 1a use four names, each naming a 1 kB tile, whereas the two tile pairs in FIG. 1b may use two names to designate the paired tiles. In some embodiments, tile instructions accept the names of the paired tiles as operands. During operation, tiles may be loaded from memory and stored in memory using load and store operations. Depending on the instruction encoding scheme used, the size, number, and configuration of available tiles, as well as the amount of available application memory, vary.
[0018] In some embodiments, tile parameters may be defined. For example, a "palette" is used to provide tile options. Exemplary options include, but are not limited to, the number of tile names, the number of bytes in rows of storage, the number of rows and columns in a tile, etc. For example, the maximum "height" (number of rows) of a tile may be defined as follows:
[0019] Tile Max Rows = Architected Storage / (The Number of Palette Names * The Number of Bytes per row).
[0020] In this way, applications can be written so that the use of fixed names can utilize different storage sizes across implementations.
[0021] The configuration of tiles is performed using the tile configuration ("TILECONFIG") command, where specific tile usage is defined in the selected palette. This declaration includes the number of tile names to be used, the requested number of rows and columns per name (tile), and, in some embodiments, the requested data type for each tile. In some embodiments, consistency checks are performed during the execution of the TILECONFIG command to determine whether it matches the qualifications of the palette entry.
[0022] Exemplary tile storage types
[0023] Figure 2 illustrates several examples of matrix storage. In (A), tiles are stored in memory. As illustrated, each "row" consists of four packed data components. To obtain the next "row," a stride value is used. Note that rows can be stored contiguously in memory. Stride memory accesses allow access from one row to the next when the tile storage does not map the row width of the underlying memory array.
[0024] Tile loads from memory and stores into memory are typically stride accesses to packed rows of data from application memory. Exemplary TILELOAD and TILESTORE instructions, or other instruction references to application memory as TILE operands in load-op instructions, are restartable in some embodiments to handle up to 2* row page faults, unmasked floating-point exceptions, and / or per-instruction interrupts.
[0025] In (B), the matrix is stored in a tile composed of multiple registers, such as packed data registers (single instruction, multiple data (SIMD), or vector registers). In this example, the tile is overlaid on three physical registers. Typically, consecutive registers are used, but this is not strictly necessary.
[0026] In (C), the matrix is stored in a tile in non-register storage accessible to the fused multiply accumulate (FMA) circuit used in tile instructions. This storage may be inside or adjacent to the FMA. Additionally, in some embodiments discussed below, the storage may be for a single data component rather than an entire row or tile.
[0027] Supported parameters for the TMMA architecture are reported via CPUID. In some embodiments, the list of information includes the maximum height and maximum SIMD dimensions. Configuring the TMMA architecture requires specifying the dimensions for each tile, the component size for each tile, and the palette identifier. This configuration is performed by executing the TILECONFIG command.
[0028] The successful execution of the TILECONFIG instruction enables subsequent TILE operators. The TILERELEASEALL instruction clears the tile configuration and disables TILE operations (until the next TILECONFIG instruction is executed). In some embodiments, XSAVE, XSTORE, etc., are used in context switching for tiles. In some embodiments, 2 XCR0 bits are used in XSAVE, one for TILECONFIG metadata and one bit corresponding to the actual tile payload data.
[0029] TILECONFIG not only configures tile usage but also sets a state variable indicating that the program is in the area of code where the tiles are configured. The implementation can enumerate one by one the constraints for other instructions that can be used with the tile area, such as not using an existing register set.
[0030] Exiting a tile area is typically done using the TILERELEASEALL command. This takes no parameters, rapidly invalidates all tiles (indicating that data no longer requires arbitrary saving or restoration), and clears the internal state corresponding to what is in the tile area.
[0031] In some embodiments, tile operations will zero any rows and any columns that exceed the dimensions specified by the tile configuration. For example, tile operations will zero data that exceeds the number of columns configured (factored by the size of the components) as each row is written. For example, in the case of a tile composed of 64-byte rows and 10 rows and 12 columns, the operation to write FP32 components will write each of the first 10 rows as 12*4 bytes containing output / result data, and zero the remaining 4*4 bytes in each row. Tile operations also completely zero any rows after the first 10 configured rows. When using a 1K tile with 64-byte rows, there will be 16 rows, and therefore in this example, the last 6 rows will also be zeroed.
[0032] In some embodiments, the context restore command (e.g., XRSTOR) enforces that when loading data, data exceeding the configured rows for the tile will be kept zero. If there is no valid configuration, all rows become zero. XRSTOR of tile data may load garbage into columns exceeding the configured ones. Since there is no component width associated with the tile configuration, it should not be possible for XRSTOR to clear beyond the number of configured columns.
[0033] Context saving (e.g., XSAVE) exposes this when writing the entire TILE storage area into memory. If XRSTOR loads garbage data into the far right part of a tile, that data will be saved by XSAVE. XSAVE will write zeros for rows exceeding a specified number for each tile.
[0034] In some embodiments, tile instructions are restartable. Operations accessing memory allow restarts after page faults. Compute instructions handling floating-point operations also allow unmasked floating-point exceptions, and the masking of exceptions is controlled by control and / or status registers.
[0035] To support restart instructions after these events, the instructions store information in the start registers described in detail below.
[0036] Matrix (Tile) Operation System
[0037] Exemplary hardware support
[0038] FIG. 3 illustrates an embodiment of a system utilizing a matrix (tile) operation accelerator. In this example, a host processor / processing system (301) passes commands (311) (e.g., matrix manipulation operations such as arithmetic or matrix manipulation operations, or load and store operations) to a matrix operation accelerator (307). However, this is illustrated in this manner merely for the purpose of discussion. As described below, this accelerator (307) may be part of a processing core. Typically, commands (311), which are tile manipulation operator instructions, will refer to tiles in a register-register ("reg-reg") or register-memory ("reg-mem") format. Other commands, such as TILESTORE, TILELOAD, TILECONFIG, etc., do not perform data operations on the tiles. The commands may be decoded instructions (e.g., micro-ops) or macro-instructions for the matrix operation accelerator (307) to handle.
[0039] In this example, the coherent memory interface (303) is coupled to the host processor / processing system (301) and the matrix operation accelerator (307) to enable them to share memory. FIGS. 4 and 5 illustrate different embodiments of how memory is shared using the matrix operation accelerator. As illustrated in FIG. 4, the host processor (401) and the matrix operation accelerator circuit (405) share the same memory (403). FIG. 5 illustrates an embodiment in which the host processor (501) and the matrix operation accelerator (505) do not share memory but can access each other's memory. For example, the processor (501) can access the tile memory (507) and utilize its host memory (503) as a regular one. Similarly, the matrix operation accelerator (505) can access the host memory (503), but more typically uses its own memory (507). Note that these memories may be of different types.
[0040] In some embodiments, tiles are supported by using overlays on physical registers. For example, tiles may utilize 16 1,024-bit registers, 32 512-bit registers, etc., depending on the implementation. In some embodiments, matrix operations utilize two-dimensional (2-D) data structures representing one or more packed regions of memory, such as registers. Throughout this description, these 2-D data structures are referred to as tiles or tile registers.
[0041] In some embodiments, the matrix operation accelerator (307) includes a plurality of FMAs (309) coupled to data buffers (305) (in some implementations, one or more of these data buffers (305) are stored in the FMAs of the grid as illustrated). The data buffers (305) buffer tiles loaded from memory and / or tiles stored in memory (e.g., using tile load or tile store instructions). The data buffers may be, for example, a plurality of registers. Typically, these FMAs are arranged as a grid (309) of chained FMAs capable of reading and writing tiles. In this example, the matrix operation accelerator (307) will perform matrix multiplication operations using tiles T0, T1, and T2. At least one of the tiles is housed in the FMA grid (309). In some embodiments, all tiles in the operation are stored in the FMA grid (309). In other embodiments, only a subset is stored in the FMA grid (309). As described, T1 is housed, and T0 and T2 are not. Note that A, B, and C refer to matrices of these tiles that may or may not occupy the entire space of the tile.
[0042] FIG. 6 illustrates an example of a matrix multiplication accumulation operation ("TMMA") using tiles.
[0043] The number of rows in the matrix (TILE A(601)) corresponds to the number of serial (chained) FMAs containing computational latency in certain embodiments. The implementation freely recycles on a grid of smaller height, but the computation remains the same.
[0044] The source / destination vector comes from N rows of tiles (TILE C (605)), and the grid (611) of FMAs performs N vector-matrix operations, resulting in a complete instruction to perform matrix multiplication of the tiles. Tile B (603) is another vector source and supplies "broadcast" terms to the FMAs at each stage.
[0045] During the operation, in some embodiments, the components of matrix B (stored in tile B (603)) are spread across a rectangular grid of FMAs. Matrix A (stored in tile A (601)) has row components that are transformed to correspond to the column dimensions of the rectangular grid of FMAs. In each FMA in the grid, the components of A and B are multiplied and added to the incoming summand (from the top in the drawing), and the outgoing sum is passed to the next row (or final output) of FMAs.
[0046] The latency of a single step is proportional to K (row height of matrix B), and dependent TMMAs typically have enough source-destination rows (either on a single tile or across tiles) to hide the corresponding latency. The implementation can also partition the SIMD (packed data component) dimension M (row height of matrix A) across time steps, but this simply changes the constant to which K is multiplied. When the program specifies a K smaller than the maximum enumerated by each TMMA, the implementation is free to implement this as "masking" or "early outs."
[0047] The total latency of the TMMA is proportional to N*K. The repetition rate is proportional to N. The number of MACs per TMMA instruction is N*K*M.
[0048] FIG. 7 illustrates an example of a subset of iterative executions of a chained and fused multiplication accumulator instruction. In particular, it illustrates an execution circuit for iterative execution of a single packed data component location at the destination. In this example, the chained and fused multiplication accumulator is operated on signed sources, where the accumulator is 2x input data size.
[0049] Each of the first encoded source (source 1 (701)) and the second encoded source (source 2 (703)) has four packed data components. Each of these packed data components stores encoded data, such as floating-point data. The third encoded source (source 3 (709)) has two packed data components, each of which stores encoded data. The sizes of the first and second encoded sources (701 and 703) are half the size of the third encoded source (initial value or previous result) (709). For example, the first and second encoded sources (701 and 703) may have 32-bit packed data components (e.g., single-precision floating-point), while the third encoded source (709) may have 64-bit packed data components (e.g., double-precision floating-point).
[0050] In this example, only the two top-level packed data component locations of the first and second code sources (701 and 703) and the top-level packed data component location of the third code source (709) are shown. Of course, other packed data component locations will also be processed.
[0051] As illustrated, packed data components are processed in pairs. For example, data from the top-most packed data component locations of the first and second encoded sources (701 and 703) is multiplied using a multiplier circuit (705), and data from the second-most packed data component locations of the first and second encoded sources (701 and 703) is multiplied using a multiplier circuit (707). In some embodiments, these multiplier circuits (705 and 707) are reused for other packed data component locations. In other embodiments, additional multiplier circuits are used to process the packed data components in parallel. In some contexts, parallel execution is performed using lanes that are the size of the third encoded source (709). The results of each multiplication are added using an adder circuit (711).
[0052] The result of the addition of the results of the multiplications is added to the data from the top packed data component location of the encoded source 3 (709) (using a different adder (713) or the same adder (711).
[0053] Finally, the result of the second addition is stored in the code destination (715) at the packed data component location corresponding to the packed data component location used from the code third source (709), or, if there is a repetition, passed on to the next repetition. In some embodiments, a write mask is applied to this storage such that storage occurs when the corresponding write mask (bit) is set, and storage does not occur when it is not set.
[0054] FIG. 8 illustrates an example of a subset of iterative executions of a chained and unified multiplication accumulator instruction. In particular, it illustrates an execution circuit for iterative execution of a single packed data component location at the destination. In this example, the chained and unified multiplication accumulator is operated on signed sources, where the accumulator is 2x the input data size.
[0055] Each of the first coded source (source 1 (801)) and the second coded source (source 2 (803)) has four packed data components. Each of these packed data components stores coded data, such as integer data. The third coded source (source 3 (809)) has two packed data components, each of which stores coded data. The sizes of the first and second coded sources (801 and 803) are half the size of the third coded source (809). For example, the first and second coded sources (801 and 803) may have 32-bit packed data components (e.g., single-precision floating-point), and the third coded source (809) may have 64-bit packed data components (e.g., double-precision floating-point).
[0056] In this example, only the two top-level packed data component locations of the first and second code sources (801 and 803) and the top-level packed data component location of the third code source (809) are shown. Of course, other packed data component locations will also be processed.
[0057] As illustrated, packed data components are processed in pairs. For example, data from the top-most packed data component locations of the first and second encoded sources (801 and 803) is multiplied using a multiplier circuit (805), and data from the second-most packed data component locations of the first and second encoded sources (801 and 803) is multiplied using a multiplier circuit (807). In some embodiments, these multiplier circuits (805 and 807) are reused for other packed data component locations. In other embodiments, additional multiplier circuits are used to process the packed data components in parallel. In some contexts, parallel execution is performed using lanes that are the size of the encoded third source (initial value or previous iteration result) (809). The results of each multiplication are added to the encoded third source (809) using an addition / saturation circuit (813).
[0058] The addition / saturation (accumulator) circuit (813) preserves the sign of the operand when the addition results in a value that is too large. In particular, saturation evaluation occurs for the infinite precision result between the multi-way-add and the write to the destination or the next iteration. When the accumulator (813) is floating-point and the input terms are integers, the sum of the floating-point accumulator input values and products is converted into infinite precision values (fixed-point numbers of hundreds of bits), the addition of the multiplication results and the third input is performed, and a single round to the actual accumulator type is performed.
[0059] Unsigned saturation means that output values are limited to the maximum number of unsigned values (all 1s) for the corresponding component width. Signed saturation means that values are limited to a range between a minimum negative and a maximum positive for the corresponding component width (for example, for bytes, the range is -128 (= -2^7) to 127 (= 2^7-1)).
[0060] The result of the addition and saturation check is stored in the code result (815) at the packed data component location corresponding to the packed data component location used from the code third source (809), or is passed on to the next iteration if there is a repetition. In some embodiments, a write mask is applied to this storage such that storage occurs when the corresponding write mask (bit) is set, and storage does not occur when it is not set.
[0061] FIG. 9 illustrates an example of a subset of iterative executions of a chained and unified multiplication accumulator instruction. In particular, it illustrates an execution circuit for iterative execution of a single packed data component location at a destination. In this example, the chained and unified multiplication accumulator is operated on a coded source and an uncoded source, where the accumulator is 4x the input data size.
[0062] Each of the first encoded source (source 1 (901)) and the second uncoded source (source 2 (903)) has four packed data components. Each of these packed data components has data such as floating-point or integer data. The third encoded source (initial value or result (915)) has packed data components, the packed data components of which store encoded data. The sizes of the first and second sources (901 and 903) are one-fourth the size of the third encoded source (915). For example, the first and second sources (901 and 903) may have 16-bit packed data components (e.g., words), and the third encoded source (915) may have 64-bit packed data components (e.g., double-precision floating-point or 64-bit integers).
[0063] In this example, the four top-level packed data component locations of the first and second sources (901 and 903) and the top-level packed data component location of the third code source (915) are shown. Of course, other packed data component locations will also be processed if they exist.
[0064] As illustrated, packed data components are processed in four groups. For example, data from the top packed data component locations of the first and second sources (901 and 903) is multiplied using a multiplier circuit (905), data from the second top packed data component locations of the first and second sources (901 and 903) is multiplied using a multiplier circuit (907), data from the third top packed data component locations of the first and second sources (901 and 903) is multiplied using a multiplier circuit (909), and data from the lowest packed data component locations of the first and second sources (901 and 903) is multiplied using a multiplier circuit (911). In some embodiments, the code-packed data components of the first source (901) are code-extended, and the code-unpacked data components of the second source (903) are zero-extended before the multiplications.
[0065] In some embodiments, these multiplier circuits (905-911) are reused for other packed data component locations. In other embodiments, additional multiplier circuits are used so that packed data components are processed in parallel. In some circumstances, parallel execution is performed using lanes that are the size of the coded third source (915). The results of each multiplication are added using an adder circuit (913).
[0066] The result of the addition of the results of the multiplications is added to the data from the top packed data component location of the encoded source 3 (915) (using a different adder (917) or the same adder (913).
[0067] Finally, the result of the second addition (919) is stored in the code destination at the packed data component location corresponding to the packed data component location used from the code third source (915), or passed on to the next iteration. In some embodiments, a write mask is applied to this storage such that storage occurs when the corresponding write mask (bit) is set, and storage does not occur when it is not set.
[0068] FIG. 10 illustrates an example of a subset of iterative executions of a chained and unified multiplication accumulator instruction. In particular, it illustrates an execution circuit for iterative execution of a single packed data component location at a destination. In this example, the chained and unified multiplication accumulator is operated on a coded source and an uncoded source, where the accumulator is 4x the input data size.
[0069] Each of the first encoded source (1001) and the second uncoded source (1003) has four packed data components. Each of these packed data components stores data such as floating-point or integer data. The third encoded source (1015) (initial or previous result) has packed data components, the packed data components of which store encoded data. The sizes of the first and second sources are one-fourth of the third encoded source (1015) (initial or previous result). For example, the first and second sources may have 16-bit packed data components (e.g., words), and the third encoded source (1015) may have 64-bit packed data components (e.g., double-precision floating-point or 64-bit integers).
[0070] In this example, four top-level packed data component locations of the first encoded source (1001) and the second uncoded source (1003) and the top-level packed data component location of the third encoded source (1015) are shown. Of course, other packed data component locations will also be processed if they exist.
[0071] As illustrated, packed data components are processed in four groups. For example, data from the top packed data component locations of the first encoded source (1001) and the second uncoded source (1003) is multiplied using a multiplier circuit (1005), data from the second top packed data component locations of the first encoded source (1001) and the second uncoded source (1003) is multiplied using a multiplier circuit (1007), data from the third top packed data component locations of the first encoded source (1001) and the second uncoded source (1003) is multiplied using a multiplier circuit (1009), and data from the lowest packed data component locations of the first encoded source (1001) and the second uncoded source (1003) is multiplied using a multiplier circuit (1011). In some embodiments, the code-packed data components of the first code-packed source (1001) are code-extended, and the code-packed data components of the second code-unpacked source (1003) are zero-extended before multiplications.
[0072] In some embodiments, these multiplier circuits (1005-1011) are reused for other packed data component locations. In other embodiments, additional multiplier circuits are used so that packed data components are processed in parallel. In some circumstances, parallel execution is performed using lanes that are the size of the third encoded source (1015) (initial or previous result). The sum of the results of the multiplications is added to the data from the top packed data component location of the third encoded source (1015) (initial or previous result) using an add / saturation circuit (1013).
[0073] The addition / saturation (accumulator) circuit (1013) preserves the sign of the operand when the addition results in a value that is too large or too small for sign saturation. In particular, saturation evaluation occurs for the infinite precision result between multi-way addition and writing to the destination. When the accumulator (1013) is floating-point and the input terms are integers, the sum of the floating-point accumulator input values and products is converted into infinite precision values (fixed-point numbers of hundreds of bits), the addition of the multiplication results and the third input is performed, and a single round to the actual accumulator type is performed.
[0074] The result (1019) of the addition and saturation check is stored in a coded destination at a packed data component location corresponding to a packed data component location used from a third coded source (1015) (initial or previous result) or passed to the next iteration. In some embodiments, a write mask is applied to this storage such that storage occurs when a corresponding write mask (bit) is set, and storage does not occur when it is not set.
[0075] FIG. 11 illustrates SIMD implementations of powers of 2 sizes, wherein the accumulators use input sizes larger than the inputs to the multipliers, depending on the embodiment. Note that the source and accumulator values (for the multipliers) may be signed or unsigned values. For an accumulator with 2X input sizes (i.e., the accumulator input value is twice the size of the packed data component sizes of the sources), Table (1101) illustrates different configurations. For byte-sized sources, the accumulator uses word or half-precision floating-point (HPFP) values of size 16-bit. For word-sized sources, the accumulator uses 32-bit integer or single-precision floating-point (SPFP) values of size 32-bit. For SPFP or 32-bit integer-sized sources, the accumulator uses 64-integer or double-precision floating-point (DPFP) values of size 64-bit.
[0076] For accumulators having 4X input sizes (i.e., the accumulator input value is four times the size of the packed data component sizes of the sources), table (1103) illustrates different configurations. For byte-sized sources, the accumulator uses 32-bit integers or SPFP (single-precision floating-point) values of size 32-bit. For word-sized sources, the accumulator uses 64-bit integers or DPFP (double-precision floating-point) values of size 64-bit in some embodiments.
[0077] For an accumulator having 8X input sizes (i.e., the accumulator input value is 8 times the size of the packed data component sizes of the sources), table (1105) illustrates a configuration. For sources of byte size, the accumulator uses a 64-bit integer.
[0078] As previously mentioned, a matrix operation circuit may be included in the core or as an external accelerator. FIG. 12 illustrates an example of a system utilizing a matrix operation circuit. In this example, multiple entities are combined by a ring interconnect (1245).
[0079] Multiple cores, Core 0 (1201), Core 1 (1203), Core 2 (1205), and Core N (1207) provide non-tile-based instruction support. In some embodiments, a matrix operation circuit (1251) is provided in Core (1203), and in other embodiments, the matrix operation circuits (1211 and 1213) are accessible on the ring interconnect (1245).
[0080] Additionally, one or more memory controllers (1223 to 1225) are provided to communicate with the memory (1233 and 1231) on behalf of the cores and / or matrix operation circuits.
[0081] FIG. 13 illustrates an embodiment of a processor core pipeline that supports matrix operations using tiles. The branch prediction and decoding circuit (1303) performs branch prediction of instructions, decoding of instructions, and / or both from instructions stored in the instruction storage (1301). For example, instructions described in detail herein may be stored in the instruction storage. In some implementations, a separate circuit is used for branch prediction, and in some embodiments, at least some instructions are decoded using microcode (1305) into one or more micro-operations, microcode entry points, microinstructions, other instructions, or other control signals. The branch prediction and decoding circuit (1303) may be implemented using various and different mechanisms. Examples of suitable mechanisms include, but are not limited to, lookup tables, hardware implementations, programmable logic arrays (PLAs), microcode read-only memory (ROMs), etc.
[0082] The branch prediction and decoding circuit (1303) is coupled to an assignment / renaming circuit (1307) which is coupled to the scheduler circuit (1309) in some embodiments. In some embodiments, these circuits provide register renaming, register assignment, and / or scheduling functions by performing one or more of 1) renaming logical operand values to physical operand values (e.g., a register alias table in some embodiments), 2) assigning status bits and flags to the decoded instruction, and 3) scheduling the decoded instruction from the instruction pool for execution on the execution circuit (e.g., using a reservation station in some embodiments).
[0083] The scheduler circuit (1309) represents any number of different schedulers, including reservation stations, a central instruction window, etc. The scheduler circuit (1309) is coupled to or includes physical register file(s) (1315). Each physical register file(s) (1315) represents one or more physical register files, and different physical register files among these store one or more different data types, such as scalar integers, scalar floating-point numbers, packed integers, packed floating-point numbers, vector integers, vector floating-point numbers, states (e.g., instruction pointers, which are the addresses of the next instruction to be executed), tiles, etc. In one embodiment, the physical register file(s) (1315) includes a vector register circuit, a write mask register circuit, and a scalar register circuit. These register circuits may provide architectural vector registers, vector mask registers, and general-purpose registers. Physical register file(s) (1315) are overlapped by a retirement circuit (1317) to exemplify various ways in which register renaming and out-of-order execution can be implemented (e.g., by using reorder buffer(s) and retirement register file(s); by using future file(s), history buffer(s), and retirement register file(s); by using register maps and a pool of registers, etc.). The retirement circuit (1317) and the physical register file(s) (1315) are coupled to an execution circuit (1311).
[0084] Although register renaming has been described in the context of out-of-order execution, it should be understood that register renaming can be used in sequential architectures. While the illustrated embodiment of the processor also includes separate instruction and data cache units and a shared L2 cache unit, alternative embodiments may have a single internal cache for both instructions and data, such as, for example, a Level 1 (L1) internal cache or multiple levels of internal cache. In some embodiments, the system may include a combination of an internal cache and an external cache located outside the core and / or processor. Alternatively, all caches may be located outside the core and / or processor.
[0085] The execution circuit (1311) is a set of one or more execution circuits, including a scalar circuit (1321), a vector / SIMD circuit (1323), and a matrix operation circuit (1327), as well as a memory access circuit (1325) that accesses the cache (1313). The execution circuits perform various operations (e.g., shift, addition, subtraction, multiplication) on various types of data (e.g., scalar floating-point, packed integer, packed floating-point, vector integer, vector floating-point). Some embodiments may include multiple execution units dedicated to specific functions or sets of functions, while other embodiments may include a single execution unit or multiple execution units in which all functions are performed. The scalar circuit (1321) performs scalar operations, the vector / SIMD circuit (1323) performs vector / SIMD operations, and the matrix operation circuit (1327) performs matrix (tile) operations as described in detail herein.
[0086] For example, an exemplary register renaming, out-of-order write / execute core architecture may implement the pipeline as follows: 1) an instruction fetch circuit performs fetch and length decoding stages; 2) a branch and decoding circuit (1303) performs a decoding stage; 3) an allocation / rename (1307) circuit performs an allocation stage and a renaming stage; 4) a scheduler circuit (1309) performs a scheduling stage; 5) physical register file(s) (coupled to or included in the scheduler circuit (1309) and the allocation / rename circuit (1307)) and memory unit perform a register read / memory read stage; an execution circuit (1311) performs an execution stage; 6) memory unit and physical register file(s) unit(s) perform a writeback / memory write stage; 7) various units may accompany the exception handling stage; and 8) The retirement unit and the physical register file(s) unit(s) perform the commit stage.
[0087] The core may support one or more instruction sets (e.g., an x86 instruction set with some extensions with new versions added), including the instruction(s) described herein; a MIPS instruction set from MIPS Technologies of Sunnyvale, California; and an ARM instruction set from ARM Holdings of Sunnyvale, California (with optional additional extensions such as NEON). In one embodiment, the core (1390) includes logic that supports packed data instruction set extensions (e.g., AVX1, AVX2), thereby allowing operations used by many multimedia applications to be performed using packed data.
[0088] It should be understood that a core can support multithreading (executing two or more parallel sets of operations or threads) and can do so in various ways, including time-sliced multithreading, concurrent multithreading (in which case a single physical core provides a logical core to each thread that the physical core is simultaneously multithreading), or a combination thereof (e.g., time-sliced fetching and decoding and subsequent concurrent multithreading as in Intel® Hyperthreading technology).
[0089] FIG. 14 illustrates an embodiment of a processor core pipeline that supports matrix operations using tiles. The branch prediction and decoding circuit (1403) performs branch prediction of instructions stored in the instruction storage (1401), decoding of instructions, and / or both from instructions. For example, instructions described in detail herein may be stored in the instruction storage. In some implementations, a separate circuit is used for branch prediction, and in some embodiments, at least some instructions are decoded using microcode (1405) into one or more micro-operations, microcode entry points, microinstructions, other instructions, or other control signals. The branch prediction and decoding circuit (1403) may be implemented using various and different mechanisms. Examples of suitable mechanisms include, but are not limited to, lookup tables, hardware implementations, programmable logic arrays (PLAs), microcode read-only memory (ROMs), etc.
[0090] The branch prediction and decoding circuit (1403) is coupled to an assignment / renaming circuit (1407) which is coupled to the scheduler circuit (1409) in some embodiments. In some embodiments, these circuits provide register renaming, register assignment, and / or scheduling functions by performing one or more of 1) renaming logical operand values to physical operand values (e.g., a register alias table in some embodiments), 2) assigning status bits and flags to the decoded instruction, and 3) scheduling the decoded instruction from the instruction pool for execution on the execution circuit (e.g., using a reservation station in some embodiments).
[0091] Scheduler circuits (1409) represent any number of different schedulers, including reservation stations, a central instruction window, etc. Scheduler unit(s) Scheduler circuits (1409) are coupled to or include physical register file(s) (1415). Each physical register file(s) (1415) represents one or more physical register files, and different physical register files among these store one or more different data types, such as scalar integers, scalar floating-point numbers, packed integers, packed floating-point numbers, vector integers, vector floating-point numbers, states (e.g., instruction pointers, which are the addresses of the next instruction to be executed), tiles, etc. In one embodiment, physical register file(s) (1415) include vector register circuits, write mask register circuits, and scalar register circuits. These register circuits may provide architectural vector registers, vector mask registers, and general-purpose registers. The physical register file(s) (1415) is overlapped by a retirement circuit (1417) to exemplify various ways in which register renaming and out-of-order execution can be implemented (e.g., by using reorder buffer(s) and retirement register file(s); by using future file(s), history buffer(s), and retirement register file(s); by using register maps and a pool of registers, etc.). The retirement circuit (1417) and the physical register file(s) (1415) are coupled to an execution circuit (1411).
[0092] Although register renaming has been described in the context of out-of-order execution, it should be understood that register renaming can be used in sequential architectures. While the illustrated embodiment of the processor also includes separate instruction and data cache units and a shared L2 cache unit, alternative embodiments may have a single internal cache for both instructions and data, such as, for example, a Level 1 (L1) internal cache or multiple levels of internal cache. In some embodiments, the system may include a combination of an internal cache and an external cache located outside the core and / or processor. Alternatively, all caches may be located outside the core and / or processor.
[0093] An execution circuit (1411), a set of one or more execution circuits (1427), and a set of one or more memory access circuits (1425) for accessing a cache (1413). The execution circuits (1427) perform matrix (tile) operations as described in detail herein.
[0094] For example, an exemplary register renaming, out-of-order write / execute core architecture may implement the pipeline as follows: 1) an instruction fetch circuit performs fetch and length decoding stages; 2) a branch and decoding circuit (1403) performs a decoding stage; 3) an allocation / rename (1407) circuit performs an allocation stage and a renaming stage; 4) a scheduler circuit (1409) performs a scheduling stage; 5) physical register file(s) (combined with or included in the scheduler circuit (1409) and the allocation / rename circuit (1407)) and memory unit perform a register read / memory read stage; an execution circuit (1411) performs an execution stage; 6) memory unit and physical register file(s) unit(s) perform a writeback / memory write stage; 7) various units may accompany the exception handling stage; and 8) The retirement unit and the physical register file(s) unit(s) perform the commit stage.
[0095] The core may support one or more instruction sets (e.g., an x86 instruction set with some extensions with new versions added), including the instruction(s) described herein; a MIPS instruction set from MIPS Technologies of Sunnyvale, California; and an ARM instruction set from ARM Holdings of Sunnyvale, California (with optional additional extensions such as NEON). In one embodiment, the core (1490) includes logic that supports packed data instruction set extensions (e.g., AVX1, AVX2), thereby allowing operations used by many multimedia applications to be performed using packed data.
[0096] It should be understood that a core can support multithreading (executing two or more parallel sets of operations or threads) and can do so in various ways, including time-sliced multithreading, concurrent multithreading (in which case a single physical core provides a logical core to each thread that the physical core is simultaneously multithreading), or a combination thereof (e.g., time-sliced fetching and decoding and subsequent concurrent multithreading as in Intel® Hyperthreading technology).
[0097] layout
[0098] Throughout this description, data is represented using a row major data layout. Column major users must transform the terms according to their orientation. Figure 15 illustrates examples of matrices represented in row major format and column major format. As illustrated, matrix A is a 2x3 matrix. When this matrix is stored in row major format, the data elements in the rows are continuous. When this matrix is stored in column major format, the data elements in the columns are continuous. is a well-known property of matrices, where the superscript T stands for transform. Reading column major data as row major data results in the matrix appearing like a transform matrix.
[0099] In some embodiments, row major semantics are used in hardware, and column major data swaps the order of operands, and the result is a matrix transform, but for subsequent column major reads from memory, it is an accurate non-transform matrix.
[0100] For example, if there are two column major matrices to be multiplied:
[0101]
[0102]
[0103] The input matrices will be stored in linear memory (column major) as follows:
[0104]
[0105] If these matrices are read as row majors with 2x3 and 3x2 dimensions, they will appear as follows:
[0106]
[0107] If you swap the order and perform matrix multiplication:
[0108]
[0109] The transform matrix is output and can then be stored in row-major order:
[0110]
[0111] And it is used in subsequent column major calculations, and this is an exact non-transform matrix.
[0112]
[0113] Exemplary use
[0114] FIG. 16 illustrates an example of the use of matrices (tiles). In this example, matrix C (1601) contains 2 tiles, matrix A (1603) contains 1 tile, and matrix B (1605) contains 2 tiles. This figure illustrates an example of an inner loop of an algorithm for calculating matrix multiplication. In this example, two result tiles, tmm0 and tmm1, from matrix C (1601) are used to accumulate intermediate results. One tile (tmm2) from matrix A (1603) is reused twice as it is multiplied by two tiles from matrix B (1605). Pointers for loading a new A matrix (tile) and two new B matrices (tiles) from the direction indicated by the arrow. An outer loop, not illustrated, adjusts the pointers for the C tiles.
[0115] The exemplary code described above includes the use of a tile configuration command, configures tile usage, executes a loop to load tiles, processes tiles, stores tiles in memory, and executes to release tile usage.
[0116] FIG. 17 illustrates an example of the use of matrices (tiles). In (1701), tile use is configured. For example, the TILECONFIG command is executed to configure tile use, including setting the number of rows and columns per tile. Typically, at least one matrix (tile) is loaded from memory in (1703). At least one matrix (tile) operation is performed using the matrices (tiles) in (1705). In (1707), at least one matrix (tile) is output stored in memory, and a context switch may occur in (1709).
[0117] Exemplary composition
[0118] Tile configuration hardware support
[0119] As previously discussed, tile usage typically needs to be configured prior to use. For example, the full use of all rows and columns may not be required. Not only does not configuring these rows and columns save power in some embodiments, but this configuration can also be used to determine whether an operation will result in an error. For example, matrix multiplication of the form (N x M) * (L x N) will typically not work if M and L are not identical.
[0120] Before using matrices with tiles, in some embodiments, tile support will be configured. For example, how many rows and columns per tile, the tiles to be used, etc., are configured. The TILECONFIG instruction is an improvement to the computer itself, as it provides support for configuring the computer to use a matrix accelerator (as part of a processor core or as an external device). In particular, the execution of the TILECONFIG instruction causes the configuration to be retrieved from memory and applied to the matrix (tile) settings within the matrix accelerator.
[0121] Tile usage configuration
[0122] FIG. 18 illustrates support for configuring the use of tiles according to an embodiment. Memory (1801) includes a tile description (1803) of matrices (tiles) to be supported.
[0123] Instruction execution resources (1811) of the processor / core (1805) store aspects of the tile description (1803) in tile configurations (1817). Tile configurations (1817) include a palette table (1813) to detail which tiles are configured for the palette (the number of rows and columns in each tile) and marking that matrix support is in use. In particular, instruction execution resources (1811) are configured to use tiles as specified by the tile configurations (1817). Instruction execution resources (1811) may also include machine-specific registers or configuration registers to indicate tile usage. Additional values, such as in use and start values, are also set. Tile configurations (1817) utilize register(s) (1819) to store tile usage and configuration information.
[0124] FIG. 19 illustrates an example of a description of matrices (tiles) to be supported. This is a description stored at the execution of the STTILECFG instruction. In this example, each field is a byte. In byte[0], a palette ID (1901) is stored. The palette ID is used to index a palette table (1813) that stores the number of bytes in a tile for each palette ID, and the bytes per row of tiles associated with this ID, as defined by the configuration.
[0125] Byte 1 stores the value to be stored in the "startRow" register (1903), and Byte 2 stores the value to be stored in the "startP" register (1905). To support restart instructions after these events, the instructions store information in these registers. To support restart instructions after interruption events such as those described in detail above, the instructions store information in these registers. The startRow value indicates the row to be used for the restart. The startP value indicates the position within the row for store operations when pairs are used, and in some embodiments, indicates the lower half of the row (in the lower tile of the pair) or the upper half of the row (in the upper tile of the pair). Generally, the position within the row (column) is not required.
[0126] Except for TILECONFIG and STTILECFG, successfully executed matrix (tile) commands will set both startRow and startP to zero.
[0127] It is the responsibility of the software to set the startRow and startP values to zero at any time when the interrupted matrix (tile) instruction has not been restarted. For example, unmasked floating-point exception handlers may decide to terminate operations in the software and change the program counter value to another instruction, usually the next instruction. In this case, the software exception handler must set the startRow and startP values to zero in the exception presented to it by the operating system before resuming the program. The operating system will subsequently reload those values using a restore instruction.
[0128] Byte 3 stores the indication of pairs of tiles (1907) (1b per tile).
[0129] Bytes 16-17 store the number of rows (1913) and columns (1915) for tile 0, and bytes 18-19 store the number of rows and columns for tile 1, etc. In other words, each 2-byte group specifies multiple rows and columns for a tile. If a 2-byte group is not used to specify tile parameters, it must have a value of zero. Specifying tile parameters for more tiles than the implementation limit or palette limit results in a failure. Unconfigured tiles are set to an initial state with 0 rows and 0 columns.
[0130] Finally, the configuration in memory typically ends with an ending delineation consisting of all zeros for several consecutive bytes.
[0131] Exemplary Tile and Tile Configuration Storage
[0132] FIGS. 20a through 20d illustrate examples of register(s) (1819). FIG. 20a illustrates a plurality of registers (1819). As illustrated, each tile (TMM0 2001... TMMN 2003) has a separate register in which each register stores the row and column sizes for that specific tile. StartP (2011) and StartRow (2013) are stored in separate registers. One or more status registers (2015) are set to indicate that the tiles are configured for use (e.g., TILES_CONFIGURED = 1).
[0133] FIG. 20b illustrates a plurality of registers (1819). As illustrated, each tile has separate registers for its rows and columns. For example, TMM0 row configuration (2021), TMM0 column configuration (2023), StartP (2011), and StartRow (2013) are stored in separate registers. One or more status registers (2015) are set to indicate that the tiles are configured for use (e.g., TILES_CONFIGURED = 1).
[0134] FIG. 20c illustrates a single register (1819). As illustrated, this register stores tile configurations (rows and columns per tile) (2031), StartP (2011), and StartRow (2013) as packed data registers stored in the single register. One or more status registers (2015) are set to indicate that the tiles are configured for use (e.g., TILES_CONFIGURED = 1).
[0135] FIG. 20d illustrates a plurality of registers (1819). As illustrated, a single register stores tile configurations (rows and columns per tile) (2031). StartP and StartRow are stored in separate registers (2011 and 2013). One or more status registers (2015) are set to indicate that the tiles are configured for use (e.g., TILES_CONFIGURED = 1).
[0136] Other combinations are considered, such as combining start registers into a single register in which they are individually plotted.
[0137] Scheduling for Systolic Arrays
[0138] As mentioned above, hardware for General Matrix Multiplication (also known as GEMM) is a good option for improving peak computation (and energy efficiency) for certain applications such as deep learning. The massive computational demands of applications based on deep neural networks (DNNs) can lead to the use of hardware (e.g., accelerators) utilizing numerous (e.g., hundreds) processing elements, e.g., fused multiply add (FMA) circuits. However, accelerators (e.g., DNNs) can be optimized for very regular data flow patterns of dense matrix multiplications. In certain embodiments, accelerators use systolic array implementations to maximize performance and area / power efficiency. Systolic arrays may include dense two-dimensional arrays optimized for very regular data flows.
[0139] In certain hardware, the decoding and execution of instructions cause configurable systolic array hardware (e.g., matrix operation accelerator circuit) to calculate C=A*B+C, where A, B, and C are each 2D matrices, and for each element of C, the hardware calculates the dot product of one row of the input matrix A and one column of the matrix B.
[0140] In certain embodiments, the matrix operation accelerator circuit takes as inputs matrix A (of dimensions M x K, where M and K are integers), matrix B (of dimensions K x N, where M and K are integers), and matrix C (of dimensions M x N, where M and N are integers), and then performs operations (e.g., single multiplication accumulation) on each component to produce a result stored in a matrix (e.g., returned to matrix C of dimensions M x N, where M and N are integers). In one embodiment, M, K, and N are less than or equal to 16. In certain embodiments, the matrix operation accelerator circuit performs the following operations (e.g., for floating-point numbers):
[0141]
[0142] FIG. 21 illustrates an embodiment of a system utilizing a matrix (tile) operation accelerator (2107) according to embodiments of the present disclosure. In certain embodiments, a host processor / processing system (2101) (e.g., a hardware processor core, e.g., the processor core (3390) of FIG. 33b)) transmits commands (e.g., matrix manipulation operations such as arithmetic or matrix manipulation operations, load, and / or store operations) to the matrix operation accelerator (2107). However, this is illustrated in this manner merely for the purpose of discussion. As described in detail in this specification, the accelerator (2107) may be part of a processing core. Commands that are tile manipulation operator instructions will refer to tiles in a register-register ("reg-reg") or register-memory ("reg-mem") format. Other commands such as TILESTORE, TILELOAD, TILECONFIG, etc., do not perform data operations on tiles in certain embodiments. The commands may be decoded instructions (e.g., micro-operations) or macro-instructions for the accelerator (2107) to handle. In one embodiment, the hardware processor core sends micro-ops to the matrix (tile) operation accelerator (2107) in response to the matrix operation instruction being executed by the hardware processor core.
[0143] In one embodiment, a reservation station (RS) circuit (2111) sends commands (e.g., micro-ops) to a matrix operation accelerator (2107). In certain embodiments, the matrix operation accelerator (2107) is a tile matrix unit (TMU). In certain embodiments, the matrix operation accelerator (2107) includes a matrix accelerator controller circuit (2113). In one embodiment, the matrix accelerator controller (e.g., circuit (2113)) controls the operations and flow of data in, outside of, and / or within the matrix operation accelerator (2107). A matrix operation accelerator (2107) (e.g., a matrix accelerator controller circuit (2113)) may include a dispatch circuit (2115) that controls the dispatch of received requests (e.g., commands) from, for example, a host processor / processing system (2101) to one or more components of the matrix operation accelerator (2107). An example of a dispatch circuit is discussed below with reference to FIG. 23.
[0144] The described matrix operation accelerator (2107) includes data buffers (e.g., registers) (2105). In certain embodiments, the data buffers (e.g., registers) (2105) can be configured to store their respective matrices in a first plurality of registers (e.g., tiles) representing, for example, a first two-dimensional matrix (e.g., a tile labeled T0 storing matrix A in storage (2105)), a second two-dimensional matrix (e.g., a tile labeled T1 storing matrix B in storage (2105)), a third two-dimensional matrix (e.g., a tile labeled T3 storing matrix C in storage (2105)), etc. The system (e.g., host processor / processing system (2101)) may include a memory interface (2103) (e.g., data cache unit) for transmitting and receiving data (e.g., coherent) between the host processor / processing system (2101) (e.g., as an OoO (Out of Order) core) and the matrix operation accelerator (2107) (e.g., in contrast to commands).
[0145] In certain embodiments, the matrix operation accelerator (2107) utilizes a grid of processing elements (2109) (e.g., fused multiply add (FMA) circuits) to perform operations. In one embodiment, a dispatch circuit (2115) controls the transmission of data (e.g., one or more values from a tile) from data buffers (2105) (e.g., registers forming a tile) to a grid of processing elements (2109) (e.g., a grid of FMA circuits). In certain embodiments, the grid of processing elements (2109) is a two-dimensional grid of processing elements, e.g., a two-dimensional grid of processing element circuits (2200) of FIG. 22.
[0146] FIG. 22 illustrates an embodiment of a matrix operation accelerator circuit (2200) comprising a two-dimensional grid of processing element circuits (2206-1 to 2206-4). In certain embodiments, a data storage (2205) (e.g., a register file) comprises a plurality of registers, each having a set of registers (e.g., tiles) representing a first input two-dimensional matrix (A), a second input two-dimensional matrix (B), and a third input two-dimensional matrix (C), and a result storage. In one embodiment, the output two-dimensional matrix result is stored in registers forming the third input two-dimensional matrix (C), for example, overwriting the values for the input two-dimensional matrix (C) after they are utilized by the matrix operation accelerator circuit (2200). The described matrix operation accelerator circuit (2200) includes a plurality of routing / buffer circuits (2402-1 to 2402-4) for routing and storing input values (e.g., from matrix A and matrix B) to processing elements according to the operation to be performed.
[0147] It should be noted that the drawings in this specification may not depict all data communication combinations (e.g., connections). A person skilled in the art will understand that this is intended to avoid obscuring specific details in the drawings. It should be noted that double-headed arrows in the drawings may not require bidirectional communication and, for example, may indicate unidirectional communication (e.g., to or from the relevant component or device). Any or all combinations of communication paths may be utilized in the specific embodiments of this specification. A single line may contain multiple paths, for example, multiple channels. For example, line (2210) may contain multiple paths (e.g., "X", where X is any positive integer), for example, one path for values from matrix A and one path for values from matrix B.
[0148] Upon request to perform an operation (e.g., by decoding and executing a command to cause the operation), the matrix operation accelerator circuit (2200) sends values from matrix A and matrix B to their respective routing circuits in certain embodiments. For example, the operation may be to multiply matrix A by matrix B and then add the respective results to the corresponding values from matrix C. In one embodiment, the first routing circuit (2204-1) receives a first value A[0][0] (value from row index 0 and column index 0) from matrix A[row][column] and broadcasts the value to each processing element (2206-1) in the corresponding row for the first input of each of the processing elements (2206-1), and receives a set of values from the first row of matrix B and sends these values to each of the second inputs of each of the processing elements (2206-1) (e.g., so that processing element (2212) receives a value from B[0][0], processing element (2214) receives a value from B[0][1], etc.). In one embodiment, a processing element (2212) provides the result of a multiplication A[0][0]*B[0][0] on its output (2216), and a processing element (2214) provides the result of a multiplication A[0][0]*B[0][1] on its output (2218). The outputs (e.g., outputs (2216 and 2218)) are sent to a routing / buffer circuit 2 (2204-2).
[0149] In one embodiment, the second routing / buffer circuit (2204-2) receives a second value A[0][1] (value from row index 0 and column index 1) from matrix A[row][column] and broadcasts the value to each processing element (2206-2) in the corresponding row for the first input of each of the processing elements (2206-2), receives a set of values from the second row of matrix B and sends these values to each of the second inputs of each of the processing elements (2206-2) (e.g., so that processing element (2222) receives a value from B[1][0], processing element (2224) receives a value from B[1][1], etc.), and sends each output from the outputs of the upper row of the processing elements (2206-1) to do so. In one embodiment, the processing element (2222) provides the result of a multiplication A[0][1]*B[1][0] added to the output (2216)(A[0][0]*B[0][0]) on the output (2226), and the processing element (2224) provides the result of a multiplication A[0][1]*B[1][1] added to the output (2218)(A[0][0]*B[0][1]) on the output (2228). In certain embodiments, this single multiplication accumulating operation is continued by each row of the processing elements (2206-3 to 2206-4) to generate the output (2230) and output (2232) from the processing elements (2206-4). Note that 4 is an exemplary number of rows of processing elements (e.g., the rows and columns of matrices A, B, and C, respectively), but there can be any multiple rows.As the end of the rows of processing elements for the matrix operation accelerator circuit (2200) is reached, the bias adder circuit (2208) adds the respective components of matrix C (e.g., the respective components from the first row of matrix C) from the input (2234) (e.g., a set of parallel input ports) and stores the result in the respective components of matrix C (e.g., the respective component positions of the first row of matrix C) through the output (2236) (e.g., a set of parallel output ports). For example, the first component from the first row of matrix C is added to the result from the output (2230) and the result is stored back in the first component position of the first row of matrix C, and the second component from the first row of matrix C is added to the result from the output (2232) and the result is stored back in the second component position of the first row of matrix C. This can be repeated for each row of matrix A to generate the total multiplication of matrix A * matrix B (e.g., and add the bias from matrix C to each result).
[0150] Accordingly, certain embodiments of the circuit use a two-dimensional (2D) array of processing elements (PEs) (e.g., FMA units), along with some input and output buffers and local control logic circuits. In one embodiment, each PE obtains some of its inputs from a data store, such as a register file, obtains other inputs from other PEs, and the final row of PEs returns its output to the data store. Thus, the PEs form a pipeline in these embodiments. The user may generally intend to perform a sequence of operations on a large set of data components (e.g., more data components than PEs). Thus, components can be input at the top of the array to start the pipeline and allow data to flow downward through the pipeline in small increments (providing additional inputs at various stages of the pipeline where appropriate).
[0151] In one embodiment, each instance of a processing element is a fused multiply accumulate (FMA) circuit comprising a multiplier circuit (e.g., which takes a first input a, a second input b, and produces a resulting output) and an adder circuit (which adds the resulting output from the multiplier circuit as a first input with a third input c to produce a result). Control of the dispatch of operations and / or data processing element circuits (e.g., FMA circuits) to a two-dimensional grid can be performed by dispatch circuitry.
[0152] FIG. 23 illustrates a dispatch circuit (2300) of a matrix operation accelerator circuit according to embodiments of the present disclosure. In one embodiment, the dispatch circuit (2300) is part of the dispatch circuit (2115) in FIG. 21. In certain embodiments, the dispatch circuit (2300) selects one of a number of pending operations (micro-ops (micro-operations)) by a multiplexer (2302) and sends the selected operation on an output (2304) to an operation circuit (e.g., the FMA grid (2109) in FIG. 21 or the PE circuits in FIG. 22). In FIG. 23, the pending operations are received on an input (2306) to become a queue (2308). The input (2306) may be from a reservation station (e.g., the RS circuit (2111) in FIG. 21).
[0153] In one embodiment, the dispatch circuit receives operation requests (e.g., one for each instruction or a set of micro-operations for a single instruction) from the out-of-order (program) reservation station circuit and executes operations sequentially, for example, under the assumption that two consecutive operations (e.g., instructions) can be executed sequentially in the matrix operation accelerator circuit (e.g., pipeline) without any dependency restrictions, where the out-of-order circuit manages the dependencies of the matrix operations by tiles A and B, while the dependencies between consecutive matrix operations are handled by the dispatch queue. In one embodiment, the RS circuit dispatches the next operation (e.g., μop (micro-operation)) to the matrix operation accelerator circuit after the dependencies are resolved. It should be understood that the queue (2308) described includes eight slots, but a single or any multiple slots may be used.
[0154] In certain embodiments, entries in the queue (2308) store information for identifying operations to be performed by the input tile(s) and / or output tile(s) and / or grid (e.g., the FMA grid (2109) of FIG. 21) utilized in the data buffers (e.g., data buffers (2105) or data buffers (2205)). In one embodiment, each entry in the queue (2308) stores the value of the valid bit (v) (or an indicator for that value), the operation (e.g., μop) opcode, tile dimensions (e.g., M, K, N), controls (e.g., zero controls), and tile ID values (e.g., values for each of the A tile, B tile, and C tile positions in the data buffers). The main head may be a pointer to the start of the current operation (e.g., the first micro-operation of a plurality of micro-operations of a single instruction) dispatched to the matrix operation accelerator circuit. The main tail may be a pointer to the end of the current operation (e.g., the final micro-operation of multiple micro-operations of the corresponding single instruction) dispatched to the matrix operation accelerator circuit. The valid bit (v) may be used to mark the operation (e.g., micro-operation) as completed when, for example, the requested operation for the corresponding operation is completed by the matrix operation accelerator circuit.
[0155] In certain embodiments, a scheduling circuit (2310) is included to select an operation (e.g., a head of a plurality of micro-operations) and send it to an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22) to start operations for the selected operation.
[0156] FIG. 24 illustrates a scheduling circuit (2400) of a dispatch circuit of a matrix operation accelerator circuit according to embodiments of the present disclosure. As noted above, an operation (e.g., a micro-operation) is It may be. The scheduling circuit may wait for: M (e.g., the number of cycles in which buffers of the first array (e.g., the first row of the PEs (2212, 2214, …, 2206-1) of FIG. 22) or the first column (e.g., buffer (2105) of FIG. 21 or buffer (2205) of FIG. 22) are occupied (e.g., and the bus is used to transmit A tile to the matrix operation accelerator circuit), K (e.g., the number of cycles in which resources (e.g., the bus) that load B tile to the matrix operation accelerator circuit are occupied), and the maximum of a pre-selected minimum (e.g., 4 cycles) gap between dispatches. In FIG. 24, the output (2402) is asserted (e.g., as a value of 1 instead of 0) to cause a dispatch to the matrix operation accelerator circuit for the next operation (e.g., a micro-operation). In certain embodiments, the scheduling circuit (2400) has a K counter (2404) that assigns a value of 1 for K while counting down the number of cycles during which resources (e.g., a bus) that load B tiles into the matrix operation accelerator circuit are occupied, and then assigns 0 (e.g., a predetermined value for counter K) when completed; an M counter (2406) that assigns a value of 1 for M while counting down the number of cycles during which resources (e.g., a bus) that load A tiles into the matrix operation accelerator circuit are occupied, and then assigns 0 (e.g., a predetermined value for counter M) when completed; and dispatching operations (e.g., micro-operations) for an output (2402) (e.g., dispatch is 1 assigned to output (2402)) to assign a value of 1 while counting down the number of minimum cycles, and then assign 0 when terminated. It includes a minimum cycle counter (2408) that counts down the minimum cycles between (e.g., 4 cycles, but any single or multiple cycles may be selected).In one embodiment, the OR logic gate (2412) asserts 1 when any of its inputs is 1, and the NOT logic gate (2414) inverts the output of the OR logic gate (2412). For example, once each counter counts down to its triggering threshold (e.g., 0), it will cause an assertion of 0 on its respective line to the OR logic gate (2412) (e.g., until a dispatch occurs for (2402)), and thus the NOT logic gate (2414) will output 1 when all counters (2404, 2406, and 2408) assert 0. After each dispatch, the counter(s) can be reset to their original (e.g., non-zero) values.
[0157] Optionally, the override dispatch control (2410) may be utilized to assign 0, for example, in a mode where the dispatch of the operation is not overridden and to assign 1 in a mode where the dispatch of the operation is overridden (and thus cause an output of 0 from the output (2402)).
[0158] In one embodiment, a trigger for selecting the next operation (e.g., micro-operation) for execution by a matrix operation accelerator circuit (e.g., an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22)) is given by MAX{4, M, K}, where M is the number of rows of matrix A, K is the number of columns of matrix A (or rows of matrix B), and 4 is a lower bound on the number of cycles for executing (e.g., micro-operation). It should be noted that in certain embodiments, the scheduling circuit waits for at least K cycles (e.g., when there is only a single bus for B tiles to be loaded into the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22)), (e.g., enabling the fetching of only a single B row per cycle). In certain of these embodiments, K sets the number of accumulators, which is the number of rows of matrix B, so that the B bus will be occupied for the next K cycles, and no new micro-operations can be dispatched during this time. In certain of these embodiments, M sets the number of cycles in which the rows of matrix A use the A bus, so that, for example, the next micro-operation can be dispatched only after the K counter as well as the M counter have been zeroed out. Additionally, a minimum number of cycles (e.g., 4) may be required between consecutive dispatches, for example, where matrix C must be read in cycle 1, accumulated with the multiplication result (e.g., A x B) in cycle 2, written back to the data buffers in cycle 3, and one more cycle is required before it can be read again in cycle 4. This implies that the steps described above are in a single cycle. In other embodiments, the steps may each take more than one cycle.
[0159] In certain embodiments, the scheduling circuit (2400) clears valid bits in the corresponding queue (e.g., queue (2308) of FIG. 23) and / or updates the head pointer in the corresponding queue (e.g., queue (2308) of FIG. 23) when an operation (e.g., a micro-operation) is selected and / or dispatched to the operation circuit (the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22) for execution.
[0160] FIG. 25 illustrates a scheduling circuit (2500) of a dispatch circuit of a matrix operation accelerator circuit that is switchable from a baseline scheduling mode to a scheduling mode that reuses an input matrix, according to embodiments of the present disclosure. In these embodiments, the input matrix refers to an input “B” matrix (e.g., tile), but this is merely illustrative and other matrices (e.g., tiles) may be similarly scheduled.
[0161] The scheduling circuit (2500) includes an input (2502) that receives a value identifying the next operation (e.g., a micro-operation) to be performed on the matrix operation accelerator circuit, and the next operation to be selected and / or dispatched to the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22) for execution. In one embodiment, the input (2502) receives an output (2304) from FIG. 23.
[0162] In certain embodiments, the input (2502) identifies the next operation (e.g., and corresponding input / output tiles) to be dispatched by the scheduling circuit (2500). In certain of these embodiments, the input (2502) includes an identification (ID) value representing a tile (e.g., a set of registers forming a tile in the data buffers (e.g., registers) (2105) in FIG. 21). The scheduling circuit (2500) includes a storage (2504) that identifies the current tile(s) being processed by the matrix operation accelerator circuit, and in particular, it can identify the current tile(s) whose data is being loaded into (or whose data is being loaded into / out of) the operation circuit (e.g., the FMA grid (2109) in FIG. 21 or the buffer circuits in FIG. 22). In one embodiment, the scheduling circuit (2500) includes a valid tile (e.g., tile B) storage (2506) that indicates whether an identification value in storage (2504) is a valid value or an invalid value (e.g., the hardware is loading another tile and therefore another identification value).
[0163] The scheduling circuit (2500) includes a comparison circuit (2508) that compares (i) a current tile (e.g., tile B) loaded into an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22) with (ii) a next tile (e.g., next tile B) to be loaded into an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuits of FIG. 22). In one embodiment, this comparison is performed by comparing an identification value provided by the scheduling circuit for each tile, e.g., an identification value provided on the entry of the tile into the data buffers of the matrix operation accelerator (e.g., the data buffers (2105) of FIG. 21), e.g., where the identification value is provided to each tile before the values for the tile are input into the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuits of FIG. 22).
[0164] In certain embodiments, the comparison circuit (2508) compares (i) an input tile (e.g., its ID value in the data buffers (2105) of FIG. 21) from the input (2502) for the next operation (e.g., "new tile B") with (ii) a current tile (e.g., tile B) currently loaded into the operation circuit (e.g., FMA grid (2109) of FIG. 21 or currently loaded into the buffer circuit(s) of FIG. 22), for example, with the ID value for the data buffers (2105) of FIG. 21 for the tile currently loaded into the FMA grid (2109) of FIG. 21. In one embodiment, the comparison circuit (2508) compares the ID value of the current tile in use with the ID value of the next tile to be used in the operation circuit, and if they are the same, assigns a value (e.g., 1) that they are the same. In one embodiment, the comparison circuit (2508) compares the ID value of the current tile in use from the storage (2504) with the ID value of the next tile to be used in the calculation circuit, and if they are the same, the storage (2506) also assigns a value (e.g., 1) indicating that they are the same, and if they are different, the comparison circuit assigns a value (e.g., 0) indicating that the identification value in the storage (2504) is also valid.
[0165] In certain embodiments, when (i) the current tile (e.g., tile B) loaded into the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22) is the same as (ii) the next tile (e.g., next tile B) to be loaded into the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuits of FIG. 22), the value of the same tile (e.g., same tile B) is asserted (e.g., 1), causing a switching to a scheduling mode (e.g., via output (2510)) that reuses the input matrix, for example, without reloading it from data buffers separated from the operation circuit (e.g., the data buffers (2105) of FIG. 21), for example, reuses tile B already stored in the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22).
[0166] In certain embodiments, the comparison circuit (2508) outputs a first value (e.g., 1) on the output (2510) when (i) the current tile (e.g., tile B) loaded into the operation circuit (e.g., the FMA grid (2109) in FIG. 21 or the buffer circuit(s) in FIG. 22) is the same as (ii) the next tile (e.g., next tile B) to be loaded into the operation circuit. The output (2510) may then be input to a NOT logic gate (2512), and the output and the output from the K counter (2516) may be input to an AND logic gate (2514) (e.g., having an output of 1 only when all inputs are 1) and into an OR logic gate (2522).
[0167] In certain embodiments, the scheduling circuit (2500) has a K counter (2516) that assigns a value of 1 for K while counting down the number of cycles during which resources (e.g., a bus) that load B tiles into the matrix operation accelerator circuit are occupied, and then assigns 0 (e.g., a predetermined value for counter K) when completed; an M counter (2518) that assigns a value of 1 for M while counting down the number of cycles during which resources (e.g., a bus) that load A tiles into the matrix operation accelerator circuit are occupied, and then assigns 0 (e.g., a predetermined value for counter M) when completed; and dispatching operations (e.g., micro-operations) for an output (2526) (e.g., dispatch is 1 assigned to output (2526)) to assign a value of 1 while counting down the number of minimum cycles, and then assign 0 when terminated. It includes a minimum cycle counter (2520) that counts down the minimum cycles between (e.g., 4 cycles, but any single or multiple cycles may be selected). In one embodiment, an OR logic gate (2522) assigns 1 when any of its inputs is 1, and a NOT logic gate (2524) inverts the output of the OR logic gate (2522). For example, once each counter counts down to its triggering threshold (e.g., 0) and (thus, the AND logic gate (2514) outputs 0 when the K counter (2516) asserts 0), this will cause an assertion of 0 on its respective line to the OR logic gate (2522) (e.g., until a dispatch occurs for (2526), and thus the NOT logic gate (2514) will output 1 when (i) all counters (2516, 2518, and 2520) assert 0 or (ii) the output (2510) is 1 for matching with the counters (2518 and 2520) asserting 0.After each dispatch, the counter(s) may be reset to their original (e.g., non-zero) values (e.g., as set by the dimensions of tiles A and B). In one embodiment, when the K counter (2516), M counter (2518) and minimum cycle counter (2520) assign 0 on their respective lines to the OR logic gate (2528), it outputs 0 to the NOT logic gate (2530), which then outputs 1 as a reset (in other cases, instead of 0), causing the valid tile (tile B) storage (2506) to indicate that the identification value in storage (2504) is an invalid value (e.g., the hardware loads another tile into the operation circuit).
[0168] Optionally, the override dispatch control (2532) may be utilized to assign 0, for example, in a mode where the dispatch of the operation is not overridden and to assign 1 in a mode where the dispatch of the operation is overridden (and thus cause an output of 0 from the output (2526)).
[0169] As noted above, the scheduling circuit may select between (i) a first (e.g., throughput-efficient) scheduling mode that reuses an input matrix, such as reusing a tile already stored in the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22) without reloading it from separate data buffers (e.g., the data buffers (2105) of FIG. 21) from the operation circuit, and (ii) a second scheduling mode that does not reuse a tile already stored in the operation circuit.
[0170] In one embodiment, a trigger for selecting the next operation (e.g., micro-operation) for execution by a matrix operation accelerator circuit (e.g., operation circuit (e.g., FMA grid (2109) of FIG. 21 or PE circuits of FIG. 22)) is given by (i) MAX{4,M} in a first (e.g., throughput-efficient) scheduling mode and (ii) MAX{4,M,K} in a second scheduling mode, where M is the number of rows of matrix A, K is the number of columns of matrix A, and 4 is a lower bound on the number of cycles for execution (e.g., micro-operation). It should be noted that in certain embodiments, the scheduling circuit waits for at least K cycles (e.g., when there is only a single bus for B tiles to be loaded into the operation circuit (e.g., FMA grid (2109) of FIG. 21 or PE circuits of FIG. 22)), (e.g. enabling the fetching of only a single B row per cycle). In certain of these embodiments, K sets the number of accumulators, which is the number of rows of matrix B, so that the B bus will be occupied for the next K cycles, and no new micro-operations can be dispatched during this time. In certain of these embodiments, M sets the number of cycles in which the rows of matrix A use the A bus, so that, for example, the next micro-operation can be dispatched only after the K counter as well as the M counter have been zeroed out. Additionally, a minimum number of cycles (e.g., 4) may be required between consecutive dispatches, for example, where matrix C must be read in cycle 1, accumulated with the multiplication result (e.g., A x B) in cycle 2, written back to the data buffers in cycle 3, and one more cycle is required before it can be read again in cycle 4. This implies that the steps above are in a single cycle. In other embodiments, the steps may take more than one cycle for each s.In certain embodiments, tile B is. It is matrix B in. For example, Scheduling based on waiting for the maximum of (M or minimum cycles) if, and waiting for the maximum of (M, K, or minimum cycles) otherwise.
[0171] In certain embodiments, when performing matrix operations on M, K, and N, the transfer of matrix B is for K cycles (e.g., up to 16) (e.g., B-transmission periods), and during the B-transmission periods, the following operations using the same matrix B (e.g., It can start execution when its A-buffer is free. For example, when the B-transmission period of M, K, and N ends, the operation returns to normal and the B matrix is read and transferred from the data buffers (e.g., the data buffers (2105) of FIG. 21) (e.g., the tile matrix buffer (TMB)) to the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22) (e.g., the tile matrix multiplier (TMM)). In one embodiment, if usage is continuous, the circuit (e.g., the scheduling circuit) (e.g., operations from the OoO RS circuit) will prevent reclamation and reallocation of the buffer (e.g., the B-buffer), and no caching is required. Thus, a computer utilizing a first (e.g., throughput-efficient) scheduling mode not only has the ability to dispatch operations earlier, but also the data buffers (e.g., the data of FIG. 21 Power consumption is reduced by eliminating the reading of data (e.g., via the B-bus) from buffers (2105)).
[0172] In certain embodiments, when data from data buffers (e.g., data buffers (2105) of FIG. 21) is stored in an operation circuit (e.g., within the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22), the data buffers are reclaimed, e.g., deallocated from use, and then reallocated to store a matrix (e.g., a new tile B) for a different operation (e.g., a different instruction). In certain embodiments, when in a first (e.g., throughput-efficient) scheduling mode that reuses an input matrix, e.g., the reuse of a tile already stored in the operation circuit (e.g., within the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22), the reclamation of what.
[0173] In one embodiment, when in a first (e.g., throughput-efficient) scheduling mode in which the input matrix is reused, the scheduling circuit prevents the retrieval of a buffer (e.g., a buffer for matrix B in the operation circuit) between the execution of a previous operation (e.g., a single instruction) and a next operation (e.g., a single instruction), and / or when in a second scheduling mode in which the input matrix is not reused, the scheduling circuit allows the retrieval of a buffer (e.g., a buffer for matrix B in the operation circuit) between the execution of a previous operation (e.g., a single instruction) and a next operation (e.g., a single instruction). In one embodiment, when in a second scheduling mode in which the input matrix is not reused, the scheduling circuit initiates the retrieval of a buffer (e.g., a buffer for matrix B in the operation circuit) between the execution of a previous operation (e.g., an instruction) and a next operation (e.g., an instruction). A matrix operation accelerator circuit (e.g., a controller circuit) may assign identification values to each of a plurality of registers representing an input 2D matrix, and checking whether the second buffer of the 2D grid of single multiplication accumulator circuits stores the immediately preceding input 2D matrix identical to the second input 2D matrix may involve comparing the respective identification values for the corresponding input 2D matrices from the previous operation (e.g., an instruction) and the next operation (e.g., an instruction). The matrix operation accelerator circuit (e.g., a controller circuit) may assign the same ID value to matrices having the same value. The matrix operation accelerator circuit (e.g., a controller circuit) may assign different ID values to matrices having different values. For example, a first operation (e.g., an instruction) may have a first value assigned to the input 2D matrix, and a second operation (e.g., an instruction) may have the same value assigned when it is the same input 2D matrix and a different value assigned when it is a different input 2D matrix from that of the first operation.
[0174] In certain embodiments, the scheduling circuit (2500) clears valid bits in the corresponding queue (e.g., queue (2308) of FIG. 23) and / or updates the head pointer in the corresponding queue (e.g., queue (2308) of FIG. 23) when an operation (e.g., micro-operation) is selected and / or dispatched to the operation circuit (FMA grid (2109) of FIG. 21 or PE circuits of FIG. 22) for execution. In one embodiment, the reuse of B-buffer data is permitted only when the operation of loading B values into the B-buffer is still being executed when "possible reuse" is possible, and a second operation arrives, and then the second operation is checked for continuity with the first operation, and the second operation is to use the reclaimed value only when continuity with the first operation is found, for example, if the first operation using the B values to be reused is not completed, its B-buffer cannot be reclaimed and thus can be used by the second operation (for example, as long as continuity exists).
[0175] In certain embodiments, a first (e.g., throughput-efficient) scheduling mode that reuses input matrices is used to accelerate the performance of (e.g., floating-point) FP matrix operation accelerator circuits. It should be understood that the scheduling disclosed herein is not limited to matrix operation accelerator circuit designs but can be extended to other modified latency microarchitectures having inherited limitations. Furthermore, it can be extended to cases where the FMA circuit assumes 1-cycle latency.
[0176] In certain embodiments, the matrix may have a dimension greater than the available hardware (e.g., arithmetic circuits (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22)). Certain embodiments herein utilize a plurality (e.g., three) of counters for scheduling (e.g., dispatching operations (e.g., micro-operations)), namely unit limits and B KxN K handling the mechanism to check for matrix reuse even , K odd and M c. Additionally, certain embodiments of this specification implement an auxiliary queue for handling operations (e.g., micro-operations) when the dimension (e.g., K) is greater than the available hardware (e.g., K > 8), for example, when these operations (e.g., micro-operations) are dispatched with high priority for a second round in an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22).
[0177] In one embodiment, the scheduling circuit performs a first row during M cycles. Maintain, and matrix Each of the M components of the first column is broadcast in different cycles. After one cycle or any number of cycles (e.g., 2 cycles), the next FMA array (e.g., row or column) of the FMA grid is occupied for M cycles to compute the second partial FMA operation in certain embodiments. In certain embodiments, this sequential pipeline mode is executed until K partial FMA operations are computed through K FMA arrays. In one embodiment, FMA grid utilization is limited by three main limits: (1) the number of rows at the destination, (2) B-bus bandwidth (e.g., 512 bits per cycle), and (3) the number of FMA arrays (e.g., rows or columns). (1) As an example for the case where M < a given value (e.g., 16), it is possible to send the next operation (e.g., micro-operation) to execution after M cycles because the first FMA0 array is available and free after M cycles. For example, in the case of M=12, the first FMA0 is used during the first 12 cycles. However, the first result of the destination WB is calculated over 16 cycles. This means that the next operation (e.g., micro-operation) can be dispatched before the first operation (e.g., micro-operation) is completed in this example. As an example for (2), the FMA grid is a single in one cycle It is limited to receiving matrix rows. In one embodiment, the adders of the FMA circuits have a given number (e.g., 2) cycle latency, which means that it is necessary to send up to a B-bus threshold (e.g., 512 bits) once every 2 cycles. Thus, certain embodiments execute two operations (e.g., micro-operations) simultaneously and maintain B-bus busyness every cycle. Furthermore, certain embodiments B for reuse through successive instructions KxNThe matrix rows are kept in a buffer. As an example for (3), the FMA grid architecture performs partial product addition of a given number (e.g., 16) (e.g., While it supports up to ), the FMA grid contains fewer than the given number (e.g., 8) of FMA arrays (e.g., rows or columns). When K≤8, the result is ready after 16 cycles, whereas when K>8, an alternative solution is required.
[0178] FIG. 26 illustrates a dispatch circuit (2600) of a matrix operation accelerator circuit for multiple paths according to embodiments of the present disclosure. The dispatch circuit (2600) includes two queues, namely a main queue (2610) and a second pass queue (2612). In one embodiment, the main queue (2610) holds important information for executing operations in an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22).
[0179] In one embodiment, the second pass queue (2612) holds operations (e.g., micro-operations) eligible for the second pass in the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22) handling (3) above (e.g., the K > 8). In one embodiment, all operations (e.g., micro-operations) inserted into the second pass queue (2612) are suspended for 16 cycles before being dispatched again for the second pass. In embodiments where the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22) is limited to a narrower number (e.g., 8) FMA circuits, the second pass is required. The second pass may be restarted after at least 16 cycles (e.g., 16 cycles = 8 FMA units * 2 cycle ADD latency). Note that this partition is made possible by the following, which also describes terms calculated in the first and second paths:
[0180]
[0181] In one embodiment, when dispatching an operation (e.g., a micro-operation) from the main queue (2610) (e.g., as an instance of the dispatch circuit (2300)), its accumulation count is observed (e.g., checking whether K > 8). In certain embodiments, if K <= threshold value (e.g., 8), the operation (e.g., a micro-operation) is sent directly to the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22) without copying it to the second pass queue (2612), and / or if K > threshold value (e.g., 8), the operation (e.g., a micro-operation) continues to be sent to the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22), for example, while it is also copied into the second pass queue (2612). In one embodiment, in the second pass queue (2612), K is updated to be the remainder of the accumulations to be executed (e.g., K-8). In one embodiment, every operation (e.g., micro-operation) inserted into the second pass queue (2612) triggers a cycle counter (e.g., 16). In one embodiment, once the counter completes its count (e.g., 16 cycles), the corresponding operation (e.g., micro-operation) is eligible to be sent to the operation circuit for the second execution unit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22). In one embodiment, because the operations (e.g., micro-operations) in the second pass queue (2612) are older, they are given priority for dispatch if they are ready. The following may be used for queue selection: Select_queue = (Valid second-pass-uop) AND (16-cycles) ? second-PASS-QUEUE : MAIN-QUEUE.In one embodiment, the scheduling circuit (2602) of the dispatch circuit (2600) utilizes a multiplexer (2604) to select an entry from the second pass queue (2612). For example, the multiplexer (2606) selects either the main queue (2610) or the second queue (2612) to provide an operation to be sent to the FMA grid through the output (2608).
[0182] FIG. 27 illustrates a scheduling circuit (2700) of a dispatch circuit of a matrix operation accelerator circuit for multiple passes according to embodiments of the present disclosure.
[0183] The input matrix in these embodiments refers to the input "B" matrix (e.g., tile), but this is merely illustrative, and other matrices (e.g., tiles) may be similarly scheduled.
[0184] The scheduling circuit (2700) includes an input (2702) that receives a value identifying the next operation (e.g., micro-operation) to be performed on the matrix operation accelerator circuit, and the next operation to be selected and / or dispatched to the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the PE circuits of FIG. 22) for execution. In one embodiment, the input (2702) receives an output (2304) from FIG. 23.
[0185] In certain embodiments, the input (2702) identifies the next operation (e.g., and corresponding input / output tiles) to be dispatched by the scheduling circuit (2700). In certain of these embodiments, the input (2702) includes an identification (ID) value representing a tile (e.g., a set of registers forming a tile in the data buffers (e.g., registers) (2105) in FIG. 21). The scheduling circuit (2700) includes a storage (2704) that identifies the current tile(s) being processed by the matrix operation accelerator circuit, and in particular, it can identify the current tile(s) whose data is being loaded into (or whose data is being loaded into / out of) the operation circuit (e.g., the FMA grid (2109) in FIG. 21 or the buffer circuits in FIG. 22). In one embodiment, the scheduling circuit (2700) includes a valid tile (e.g., tile B) storage (2706) that indicates whether an identification value in storage (2704) is a valid value or an invalid value (e.g., the hardware is loading another tile and therefore another identification value).
[0186] The scheduling circuit (2700) includes a comparison circuit (2708) that compares (i) a current tile (e.g., tile B) loaded into an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22) with (ii) a next tile (e.g., next tile B) to be loaded into an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuits of FIG. 22). In one embodiment, this comparison is performed by comparing an identification value provided by the scheduling circuit for each tile, e.g., an identification value provided on the entry of the tile into the data buffers of the matrix operation accelerator (e.g., the data buffers (2105) of FIG. 21), e.g., where the identification value is provided to each tile before the values for the tile are input into the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuits of FIG. 22).
[0187] In certain embodiments, the comparison circuit (2708) compares (i) an input tile (e.g., its ID value in the data buffers (2105) of FIG. 21) from the input (2702) for the next operation (e.g., "new tile B") with (ii) a current tile (e.g., tile B) currently loaded into the operation circuit (e.g., FMA grid (2109) of FIG. 21 or currently loaded into the buffer circuit(s) of FIG. 22), for example, with the ID value for the data buffers (2105) of FIG. 21 for the tile currently loaded into the FMA grid (2109) of FIG. 21. In one embodiment, the comparison circuit (2708) compares the ID value of the current tile in use with the ID value of the next tile to be used in the operation circuit, and if they are the same, assigns a value (e.g., 1) that they are the same. In one embodiment, the comparison circuit (2708) compares the ID value of the current tile in use from the storage (2704) with the ID value of the next tile to be used in the calculation circuit, and if they are the same, the storage (2706) also assigns a value (e.g., 1) indicating that they are the same, and if they are different, the comparison circuit assigns a value (e.g., 0) indicating that the identification value in the storage (2704) is also valid.
[0188] In certain embodiments, when (i) the current tile (e.g., tile B) loaded into the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22) is the same as (ii) the next tile (e.g., next tile B) to be loaded into the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuits of FIG. 22), the value of the same tile (e.g., same tile B) is asserted (e.g., 1), causing a switching to a scheduling mode (e.g., via output (2710)) that reuses the input matrix, for example, reuses the tile B already stored in the operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22), without reloading it from the data buffers separated from the operation circuit (e.g., the data buffers (2105) of FIG. 21).
[0189] In certain embodiments, the comparison circuit (2708) outputs a first value (e.g., 1) on the output (2710) when (i) the current tile (e.g., tile B) loaded into the operation circuit (e.g., the FMA grid (2109) in FIG. 21 or the buffer circuit(s) in FIG. 22) is the same as (ii) the next tile (e.g., next tile B) to be loaded into the operation circuit. The output (2710) may then be input into the OR logic gate (2726) along with an input from the NOT logic gate (2724). The output from the OR logic gate (2726) is sent to the first input of the AND logic gate (2730). The other input of the AND logic gate (2730) is coupled to the NOT logic gate (2728) coupled to the M counter (2718). The output (2732) of the AND logic gate (2730) causes the dispatch of the next operation (e.g., a micro-operation) when it is 1 in certain embodiments. Compared to FIG. 25, FIG. 27 includes a counter (2716A) for an even K and a counter (2716B) for an odd K, and both counters are coupled to their respective inputs of the AND logic gate (2720).In one embodiment, the K even counter (2716A) assigns a value of 1 for the even Ks while counting down the number of cycles occupied by resources (e.g., a bus) loading B tiles into the matrix operation accelerator circuit, and then assigns 0 (e.g., a predetermined value for the even counter K) upon completion, and the K odd counter (2716B) assigns a value of 1 for the odd Ks while counting down the number of cycles occupied by resources (e.g., a bus) loading B tiles into the matrix operation accelerator circuit, and then assigns 0 (e.g., a predetermined value for the odd counter K) upon completion, and the M counter (2718) assigns a value of 1 for M while counting down the number of cycles occupied by resources (e.g., a bus) loading A tiles into the matrix operation accelerator circuit, and then assigns 0 (e.g., a predetermined value for the counter M) upon completion Assert the value. After each dispatch, the counter(s) can be reset to their original (e.g., non-zero) value.
[0190] In one embodiment, FIG. 27 specifies the trigger for the next operation to be sent to an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuits of FIG. 22) as follows: even cycles are tracked by K even counters (2716A), odd cycles are tracked by K odd counters (2716B), any dispatched operation (e.g., micro-operation) updates its corresponding K even or K odd counters, and any dispatched operation (e.g., micro-operation) also updates, for example, M counters, and B KxN Matrix reuse is checked.
[0191] In one embodiment, if (((K-even == 0) OR (K-odd == 0) OR (SAME-B == 1)) AND (M <= 1)) THEN, the next operation (e.g., a micro-operation) is dispatched.
[0192] In one embodiment, the dispatch state indicates that one of the B bus cycles (e.g., even cycles or odd cycles) is available (e.g., its corresponding counter is 0) and the first FMA0 is available (e.g., M counter is 0 or 1), and will be sent to an operation circuit (e.g., the FMA grid (2109) of FIG. 21 or the buffer circuit(s) of FIG. 22) for execution of the next operation (e.g., micro-operation) from the main queue (2610) or the second pass queue (2612).
[0193] In certain embodiments, when a trigger occurs when M-counter==0, A and B must be sent to the TMM in the same cycle and / or when M-counter==1, the B bus must be sent to the TMM for one cycle prior to the A bus. It should be noted that if certain embodiments assume multiple (e.g., 2) cycle latencies for the execution of the FMA circuit (e.g., PE), scheduling for the case where the FMA circuit (e.g., PE) has a single cycle latency is possible, for example, by tile reuse.
[0194] The following are examples for specifying scheduling. In one embodiment, a tile (e.g., tile B) reuse mechanism is utilized in conjunction with the following.
[0195] Example 1.1 Three numbers a, b, and c are given. The product of a and b plus c, i.e., Defines the FMA (Fused Multiplication Addition) operation as.
[0196] Example 1.2 Three matrices and This is given. Matrix multiplication MM(M, K, N) is a new matrix Defined as an operation that causes, where each component .
[0197] Specific embodiments in this specification are Imposes an architecture limit of a threshold value (e.g., 16).
[0198] Example 1.3 An MQ (Matrix Queue) is an ordered set of matrix multiplication operations, i.e., MQ = {M M1, M M2, ...}. This set indicates that matrix multiplication operations are ready in RS.
[0199] Example 1.4 A BMQ (Big Matrix Queue) is an ordered set of matrix multiplication operations, i.e., BMQ = {M M1, M M2, ...}. This queue is utilized for matrix multiplication operations K > 8 that require an extra round in the matrix multiplication unit according to limit 3. Each element in this queue has an internal counter and is ready in 16 cycles after its insertion. This structure implements two additional methods, update and ready, to update and check whether the top of the queue MM is ready.
[0200] In one embodiment, each of the matrix multiplication (MM) operation parameters (M, K, N) and operands (A, B, C) is defined as an MM component attribute, e.g., MM.K. Given a matrix multiplication MM and its operands A, B, C, the scheduler can synchronize the initial cycle operations of these operands through functions Dispatch_A, Dispatch_B, and Dispatch_C that receive an initial cycle as an input parameter. Then, once a start cycle for the operation is defined, the scheduler can cause the continuation of the execution of the remainder of the MM operations according to the operation mode. Specifically, in the first (e.g., FMA_0) array, starting from the input cycle of Dispatch_A, components a_(i,0) (where i=0...M) are broadcast in each resulting cycle, and then in the second (e.g., FMA_1) array, components a_(i,1) (where i=0...M) are broadcast in cycle+2, and so on. In one embodiment, the MM.C operand is inserted into the FMA_0 array in cycle 2 after the insertion of A through Dispatch_C, because this is the time to complete the FMA multiplication operation. In one embodiment, Dispatch_B inserts the entire B_i row into FMA_i every 2 cycles, where i=0...K. Note that according to some embodiments, the loading of a single matrix row per cycle is a system limit. In practice, based on these restrictions, the scheduler can divide the time line into even and odd cycles and insert MM operations into these classified cycles. Then, one of these cycles racks time for insertion through counters M,K_even,K_odd, which are updated every cycle. In the case where MM.K > 8, in certain embodiments, the MM(M,K,N) operation is split into two operations, namely MM(M,8,N) and MM(M,K-8,N).In one embodiment, the MM(M,8,N) operation includes the first eight FMA operations of MM(M,K,N) and is sent straight to execution when inserted, where MM(M,K-8,N) is incubated (e.g., in the main queue (2610) in FIG. 26) and waits for at least a threshold number (e.g., 16) cycles for the operation (until MM(M,8,N) finishes calculating its first line result), after which the result of MM(M,8,N) is inserted as the C input of the matrix MM(M,K-8,N) when executed.
[0201] FIG. 28 illustrates a pseudocode (2800) for a matrix operation circuit according to embodiments of the present disclosure.
[0202] FIG. 29 illustrates a method (2900) for processing matrix operation instructions according to embodiments of the present disclosure. A processor (e.g., or processor core) may perform the method (2900) in response to receiving a request to execute instructions, for example, from software. The processor (e.g., or processor core) may offload specific operations to a matrix operation accelerator circuit. The described method (2900) comprises: fetching an instruction having a first field identifying a first input 2D matrix, a second field identifying a second input 2D matrix, and a field identifying a result storage (2902); decoding the instruction into a decoded instruction by a decoder of a hardware processor core - the hardware processor core is coupled to a matrix operation accelerator circuit comprising a 2D grid of single multiplication accumulator circuits, and the matrix operation accelerator circuit is coupled to a first plurality of registers representing a first input 2D matrix and a second plurality of registers representing a second input 2D matrix - (2904); retrieving data associated with the first field (e.g., tile A), the second field (e.g., tile B), (and optionally, a third field (e.g., tile C)) (2906); (optional) scheduling the decoded instruction for execution (2908); and executing the decoded instruction to single multiplication accumulator from the first plurality of registers into the first input 2D matrix Load the first buffer of the 2D grid of the circuits, check whether the second buffer of the 2D grid of the single-multiplication accumulator circuits stores the immediately preceding 2D input matrix that is identical to the second input 2D matrix, and if the second buffer of the 2D grid of the single-multiplication accumulator circuits stores the immediately preceding 2D input matrix from the execution of the previous instruction that is identical to the second input 2D matrix: prevent retrieval of the second buffer between the execution of the previous instruction and the decoded single instruction, andThe method comprises the step of processing a matrix operation (e.g., FMA) instruction by performing an operation on a first input 2D matrix from a first buffer and an immediately preceding input 2D matrix from a second buffer to produce a result, and storing the result in a result storage, and in the case where the second buffer of the 2D grid of single multiplication accumulator circuits does not store an immediately preceding input 2D matrix from the execution of a previous instruction that is identical to the second input 2D matrix: loading the second input 2D matrix into the second buffer of the 2D grid of single multiplication accumulator circuits, performing an operation on the first input 2D matrix from the first buffer and the second input 2D matrix from the second buffer to produce a result, and storing the result in a result storage (2910), and committing the result of the executed instruction (2912).
[0203] Exemplary architectures, systems, etc., in which the above matters can be used are described in detail below.
[0204] At least some embodiments of the disclosed technologies may be described with consideration of the following examples:
[0205] Example 1. As a device:
[0206] Matrix operation accelerator circuit including a two-dimensional grid of single multiplication accumulator circuits;
[0207] A first plurality of registers representing a first input two-dimensional matrix coupled to a matrix operation accelerator circuit;
[0208] A second plurality of registers representing a second input two-dimensional matrix coupled to a matrix operation accelerator circuit;
[0209] A decoder of a hardware processor core coupled to a matrix operation accelerator circuit that decodes a single instruction into a single instruction that is decoded—the single instruction includes a field identifying the result storage—; and
[0210] Execution circuit of a hardware processor core - The execution circuit of a hardware processor core executes a decoded single instruction to: load a first buffer of a 2D grid of fused multiply accumulate circuits with a first input 2D matrix from a first plurality of registers; check whether a second buffer of a 2D grid of fused multiply accumulate circuits stores the immediately preceding input 2D matrix that is identical to the second input 2D matrix; and if the second buffer of a 2D grid of fused multiply accumulate circuits stores the immediately preceding input 2D matrix from the execution of the previous instruction that is identical to the second input 2D matrix: prevent reclamation of the second buffer between the execution of the previous instruction and the decoded single instruction; perform operations on the first input 2D matrix from the first buffer and the immediately preceding input 2D matrix from the second buffer to produce a result; and store the result in result storage; and if the second buffer of a 2D grid of fused multiply accumulate circuits stores the previous If the input 2D matrix immediately preceding the execution of the instruction is not stored: the second input 2D matrix is loaded into the second buffer of the 2D grid of the single multiplication accumulator circuits, an operation is performed on the first input 2D matrix from the first buffer and the second input 2D matrix from the second buffer to produce a result, and the result is stored in the result storage - includes.
[0211] Example 2. As the device of Example 1, if the second buffer of the 2D grid of single multiplication accumulator circuits does not store the input 2D matrix immediately preceding the execution of the previous instruction, which is identical to the second input 2D matrix, the matrix operation accelerator circuit starts retrieving the second buffer between the execution of the previous instruction and the decoded single instruction.
[0212] Example 3. As a device in Example 1, the matrix operation accelerator circuit assigns identification values to each of a plurality of registers representing an input two-dimensional matrix, and checking whether the second buffer of the two-dimensional grid of single multiplication accumulator circuits stores the immediately preceding input two-dimensional matrix identical to the second input two-dimensional matrix includes comparing the respective identification values for the corresponding input two-dimensional matrices from the previous instruction and the single instruction.
[0213] Example 4. As the device of Example 3, the matrix operation accelerator circuit assigns different identification values to a second plurality of registers that store the input 2D matrix of the following instruction.
[0214] Example 5. As the device of Example 1, if the second buffer of the two-dimensional grid of the single multiplication accumulator circuits stores the input two-dimensional matrix immediately preceding the execution of the previous instruction, which is identical to the second input two-dimensional matrix, the matrix operation accelerator circuit does not load the second buffer of the two-dimensional grid of the single multiplication accumulator circuits during the execution of the decoded single instruction.
[0215] Example 6. As the device of Example 1, the result storage is a third plurality of registers representing at least one output two-dimensional matrix formed by the execution of a single decoded instruction.
[0216] Example 7. As the device of Example 1, the operation comprises transmitting downstream the respective outputs of each of the first proper subset of single multiplicative accumulator circuits of the two-dimensional grid to the respective inputs of each of the second proper subset of single multiplicative accumulator circuits of the two-dimensional grid to form a result.
[0217] Example 8. As the device of Example 7, the first binary subset of single multiplicative accumulator circuits is one of the rows or columns of the two-dimensional grid of single multiplicative accumulator circuits, and the second binary subset of single multiplicative accumulator circuits is another of the rows or columns of the two-dimensional grid of single multiplicative accumulator circuits.
[0218] Example 9. As a method:
[0219] A step of decoding a single instruction into a decoded single instruction using a decoder of a hardware processor core - the hardware processor core is coupled to a matrix operation accelerator circuit comprising a two-dimensional grid of single multiplication accumulator circuits, the matrix operation accelerator circuit is coupled to a first plurality of registers representing a first input two-dimensional matrix and a second plurality of registers representing a second input two-dimensional matrix, and the single instruction includes a field identifying a result storage -; and
[0220] Execute a single decoded instruction using the execution circuitry of the hardware processor core:
[0221] Loading a first buffer of a two-dimensional grid of fused multiply accumulate circuits with a first input two-dimensional matrix from a first plurality of registers, and
[0222] Check whether the second buffer of the 2D grid of the single multiplication accumulator circuits stores the immediately preceding 2D input matrix that is identical to the second 2D input matrix, and
[0223] If the second buffer of the 2D grid of the single multiplication accumulator circuits stores the input 2D matrix immediately preceding the execution of the previous instruction, which is identical to the second input 2D matrix:
[0224] Prevents the reclamation of the second buffer between the execution of the previous instruction and a single decoded instruction, and
[0225] Performing operations on the first input 2D matrix from the first buffer and the immediately preceding input 2D matrix from the second buffer to produce a result, and
[0226] Save the results to the results storage, and
[0227] If the second buffer of the 2D grid of the single multiplication accumulator circuits does not store the 2D input matrix immediately preceding the execution of the previous instruction, which is identical to the second input 2D matrix:
[0228] Load the second input 2D matrix into the second buffer of the 2D grid of the single multiplication accumulator circuits, and
[0229] Performing operations on a first input 2D matrix from a first buffer and a second input 2D matrix from a second buffer to produce a result, and
[0230] Includes the step of storing the result in the result storage.
[0231] Example 10. As a method of Example 9, if the second buffer of the two-dimensional grid of single multiplication accumulator circuits does not store the input two-dimensional matrix immediately preceding the execution of the previous instruction that is identical to the second input two-dimensional matrix, the method further includes the step of starting the retrieval of the second buffer between the execution of the previous instruction and the execution of the decoded single instruction.
[0232] Example 11. As a method of Example 9, the step of assigning identification values to each of a plurality of registers representing an input two-dimensional matrix is further included, and checking whether the second buffer of the two-dimensional grid of single multiplication accumulator circuits stores the immediately preceding input two-dimensional matrix that is identical to the second input two-dimensional matrix includes comparing the respective identification values for the corresponding input two-dimensional matrices from the previous instruction and the single instruction.
[0233] Example 12. As a method of Example 11, the method further includes the step of assigning different identification values to a second plurality of registers that store the input 2D matrix of the next instruction.
[0234] Example 13. As a method of Example 9, if the second buffer of the two-dimensional grid of the single multiplication accumulator circuits stores the input two-dimensional matrix immediately preceding the execution of the previous instruction, which is identical to the second input two-dimensional matrix, the method further includes the step of not loading the second buffer of the two-dimensional grid of the single multiplication accumulator circuits during the execution of the decoded single instruction.
[0235] Example 14. As a method of Example 9, the result storage is a third plurality of registers representing at least one output two-dimensional matrix formed by the execution of a single decoded instruction.
[0236] Example 15. As a method of Example 9, the operation comprises transmitting downstream the respective outputs of each of the first binary subsets of the single multiplication accumulator circuits of the two-dimensional grid to the respective inputs of each of the second binary subsets of the single multiplication accumulator circuits of the two-dimensional grid to form a result.
[0237] Example 16. As in the method of Example 15, the first binary subset of single multiplicative accumulators is one of the rows or columns of the two-dimensional grid of single multiplicative accumulators, and the second binary subset of single multiplicative accumulators is another of the rows or columns of the two-dimensional grid of single multiplicative accumulators.
[0238] Example 17. As a non-transient machine-readable medium storing code that causes the machine to perform a method when executed by a machine, the method is:
[0239] A step of decoding a single instruction into a decoded single instruction using a decoder of a hardware processor core - the hardware processor core is coupled to a matrix operation accelerator circuit comprising a two-dimensional grid of single multiplication accumulator circuits, the matrix operation accelerator circuit is coupled to a first plurality of registers representing a first input two-dimensional matrix and a second plurality of registers representing a second input two-dimensional matrix, and the single instruction includes a field identifying a result storage -; and
[0240] Execute a single decoded instruction using the execution circuitry of the hardware processor core:
[0241] Loading a first buffer of a two-dimensional grid of fused multiply accumulate circuits with a first input two-dimensional matrix from a first plurality of registers, and
[0242] Check whether the second buffer of the 2D grid of the single multiplication accumulator circuits stores the immediately preceding 2D input matrix that is identical to the second 2D input matrix, and
[0243] If the second buffer of the 2D grid of the single multiplication accumulator circuits stores the input 2D matrix immediately preceding the execution of the previous instruction, which is identical to the second input 2D matrix:
[0244] Prevents the reclamation of the second buffer between the execution of the previous instruction and a single decoded instruction, and
[0245] Performing operations on the first input 2D matrix from the first buffer and the immediately preceding input 2D matrix from the second buffer to produce a result, and
[0246] Save the results to the results storage, and
[0247] If the second buffer of the 2D grid of the single multiplication accumulator circuits does not store the 2D input matrix immediately preceding the execution of the previous instruction, which is identical to the second input 2D matrix:
[0248] Load the second input 2D matrix into the second buffer of the 2D grid of the single multiplication accumulator circuits, and
[0249] Performing operations on a first input 2D matrix from a first buffer and a second input 2D matrix from a second buffer to produce a result, and
[0250] Includes the step of storing the result in the result storage.
[0251] Example 18. As a non-transient machine-readable medium of Example 17, if the second buffer of a two-dimensional grid of single multiplication accumulator circuits does not store the input two-dimensional matrix immediately preceding the execution of a previous instruction that is identical to the second input two-dimensional matrix, the method further comprises the step of initiating the retrieval of the second buffer between the execution of the previous instruction and the decoded single instruction.
[0252] Example 19. As a non-transient machine-readable medium of Example 17, the method further comprises the step of assigning identification values to each of a plurality of registers representing an input two-dimensional matrix, and checking whether the second buffer of the two-dimensional grid of single multiplication accumulator circuits stores the immediately preceding input two-dimensional matrix identical to the second input two-dimensional matrix comprises comparing the respective identification values for the corresponding input two-dimensional matrices from the previous instruction and the single instruction.
[0253] Example 20. As a non-transient machine-readable medium of Example 19, the method further comprises the step of assigning different identification values to a second plurality of registers that store an input two-dimensional matrix of the following instruction.
[0254] Example 21. As a non-transient machine-readable medium of Example 17, where a second buffer of a two-dimensional grid of single multiplication accumulator circuits stores an input two-dimensional matrix immediately preceding the execution of a previous instruction that is identical to the second input two-dimensional matrix, the method further comprises the step of not loading the second buffer of a two-dimensional grid of single multiplication accumulator circuits during the execution of a decoded single instruction.
[0255] Example 22. As a non-transient machine-readable medium of Example 17, the result storage is a third plurality of registers representing at least one output two-dimensional matrix formed by the execution of a single decoded instruction.
[0256] Example 23. As a non-transient machine-readable medium of Example 17, the operation comprises transmitting downstream the respective outputs of each of the first binary subsets of single-multiplication accumulator circuits of a two-dimensional grid to the respective inputs of each of the second binary subsets of single-multiplication accumulator circuits of a two-dimensional grid to form a result.
[0257] Example 24. As a non-transient machine-readable medium of Example 23, a first binary subset of single multiplicative accumulator circuits is one of the rows or columns of a two-dimensional grid of single multiplicative accumulator circuits and a second binary subset of single multiplicative accumulator circuits is another of one of the rows or columns of a two-dimensional grid of single multiplicative accumulator circuits.
[0258] In another embodiment, the device includes a data storage device that stores code that causes the hardware processor to perform any method disclosed herein when executed by the hardware processor. The device may be as described in the detailed description. The method may be as described in the detailed description.
[0259] Detailed exemplary systems, processors, and emulation
[0260] Examples of hardware, software, etc. for executing the aforementioned instructions are described in detail in this specification. For example, the following describes in detail modes of instruction execution including various pipeline stages such as fetch, decode, schedule, execute, retire, etc.
[0261] Instruction sets.
[0262] An instruction set may include one or more instruction formats. A given instruction format may include, among other things, various fields (e.g., number of bits, bit location) and / or other data field(s) (e.g., mask) for specifying the operation to be performed (e.g., opcode) and the operand(s) on which the operation will be performed. Some instruction formats are further subdivided through the definition of instruction templates (or subformats). For example, an instruction template of a given instruction format may be defined to have different subsets of the fields of the instruction format (the included fields are typically in the same order, but at least some have different bit locations because fewer fields are included) and / or may be defined to have a given field that is interpreted differently. Thus, each instruction of an ISA is represented using a given instruction format (and, where defined, in a given template among the instruction templates of that instruction format) and includes fields for specifying the operation and operands. For example, an exemplary ADD instruction has an instruction format including a specific op-code, an op-code field specifying that op-code, and operand fields for selecting operands (source 1 / destination and source 2); an occurrence of this ADD instruction in an instruction stream will have specific contents in the operand fields selecting the specific operands. A SIMD extension set referred to as AVX (Advanced Vector Extensions) (AVX1 and AVX2) and utilizing the VEX (Vector Extensions) coding scheme has been announced and / or published (see, for example, Intel® 64 and IA-32 Architectures Software Developer's Manual, November 2018; and Intel® Architecture Instruction Set Extensions Programming Reference, October 2018).
[0263] Exemplary command formats
[0264] Embodiments of the instruction(s) described herein may be implemented in different formats. Additionally, exemplary systems, architectures, and pipelines are described in detail below. Embodiments of the instruction(s) may be executed on such systems, architectures, and pipelines, but are not limited to those described in detail.
[0265] General vector-friendly command format
[0266] A vector-friendly instruction format is an instruction format suitable for vector instructions (e.g., specific fields specific to vector operations exist). While embodiments are described in which both vector and scalar operations are supported through the vector-friendly instruction format, alternative embodiments use only the vector operations of the vector-friendly instruction format.
[0267] FIGS. 30a and FIGS. 30b are block diagrams illustrating a generic vector-friendly instruction format and its instruction templates according to embodiments of the present disclosure. FIG. 30a is a block diagram illustrating a generic vector-friendly instruction format and its Class A instruction templates according to embodiments of the present disclosure; FIG. 30b is a block diagram illustrating a generic vector-friendly instruction format and its Class B instruction templates according to embodiments of the present disclosure. Specifically, Class A and Class B instruction templates are defined for a generic vector-friendly instruction format (3000), both of which include no memory access (3005) instruction templates and memory access (3020) instruction templates. In the context of a vector-friendly instruction format, the term generic refers to an instruction format that is not bound to any specific set of instructions.
[0268] Although embodiments of the present disclosure will be described that support a vector-friendly instruction format hereinafter: a 64-byte vector operand length (or size) with 32-bit (4-byte) or 64-bit (8-byte) data component widths (sizes) (and thus, a 64-byte vector consists of 16 double-word size elements or alternatively 8 quad-word size elements); a 64-byte vector operand length (or size) with 16-bit (2-byte) or 8-bit (1-byte) data component widths (or sizes); a 32-byte vector operand length (or size) with 32-bit (4-byte), 64-bit (8-byte), 16-bit (2-byte), or 8-bit (1-byte) data component widths (or sizes); and a 16-byte vector operand length (or size) with 32-bit (4-byte), 64-bit (8-byte), 16-bit (2-byte), or 8-bit (1-byte) data component widths (or sizes); Alternative embodiments may support larger, smaller, and / or different vector operand sizes (e.g., 256-byte vector operands) with larger, smaller, or different data component widths (e.g., 128-bit (16-byte) data component widths).
[0269] Class A instruction templates in FIG. 30a include: 1) a no-memory-access, full-round control type operation (3010) instruction template and a no-memory-access, data-conversion type operation (3015) instruction template shown within no-memory-access (3005) instruction templates; and 2) a memory-access, transient (3025) instruction template and a memory-access, non-transient (3030) instruction template shown within memory-access (3020) instruction templates. Class B instruction templates in FIG. 30b include: 1) a no-memory-access, write-mask-control, partial-round control type operation (3012) instruction template and a no-memory-access, write-mask-control, vsize type operation (3017) instruction template shown within no-memory-access (3005) instruction templates; and 2) a memory-access, write-mask-control (3027) instruction template shown within memory-access (3020) instruction templates.
[0270] The general vector-friendly command format (3000) includes the following fields listed below in the order exemplified in FIG. 30a-30b.
[0271] Format field (3040) – A specific value in this field (instruction format identifier value) uniquely identifies the vector-friendly instruction format and, thus, the occurrences of instructions in the vector-friendly instruction format within the instruction streams. As such, this field is optional in that it is not required for instruction sets that have only the general vector-friendly instruction format.
[0272] Base operation field (3042) - The content of this distinguishes different base operations.
[0273] Register index field (3044) – The contents thereof specify the locations of source and destination operands, whether they are in registers or in memory, either directly or through address generation. These contain a number of bits sufficient to select N registers from a PxQ (e.g., 32x512, 16x128, 32x1024, 64x1024) register file. In one embodiment, N may be up to three source and one destination register, but alternative embodiments may support more or fewer source and destination registers (e.g., up to two sources may be supported if one of these sources also serves as a destination, up to three sources may be supported if one of these sources also serves as a destination, and up to two sources and one destination may be supported).
[0274] Modifier field (3046) – The content of this distinguishes occurrences of instructions in a general vector instruction format that specify memory access from those that do not; namely, between no memory access (3005) instruction templates and memory access (3020) instruction templates. While memory access operations read and / or write to a memory hierarchy (in some cases, using values in registers to specify source and / or destination addresses), no memory access operations do not (e.g., the source and destination are registers). In one embodiment, this field also selects among three different ways of performing memory address calculations, but alternative embodiments may support more, fewer, or different ways of performing memory address calculations.
[0275] Augmentation operation field (3050) – The content of this distinguishes which of various different operations should be executed in addition to the base operation. This field is context-specific. In one embodiment of the present disclosure, this field is divided into a class field (3068), an alpha field (3052), and a beta field (3054). The augmentation operation field (3050) allows common groups of operations to be performed in a single instruction rather than two, three, or four instructions.
[0276] Scaling field (3060) - The contents of this allow scaling of the contents of the index field for generating memory addresses (e.g., for generating addresses using 2 scale * index + base).
[0277] Displacement field (3062A) - The contents of this are used as part of memory address generation (e.g., for address generation using 2 scale*index + base + displacement).
[0278] Displacement factor field (3062B) (Note that juxtaposition of displacement field (3062A) directly across displacement factor field (3062B) indicates that either one or the other is used) — the contents of which are used as part of an address generation; this specifies a displacement factor to be scaled by the size (N) of the memory access — where N is the number of bytes in the memory access (e.g., for an address generation using 2scale*index + base + scaled displacement). Redundant low-order bits are ignored, and thus the contents of the displacement factor field are multiplied by the total size (N) of the memory operands to generate the final displacement to be used to calculate the effective address. The value of N is determined by the processor hardware at runtime based on the full opcode field (3074) and the data manipulation field (3054C) (described later in this specification). The displacement field (3062A) and the displacement factor field (3062B) are optional in that they are not used for memory access-free (3005) instruction templates and / or different embodiments may implement only one of the two or neither.
[0279] Data component width field (3064) – The content of this distinguishes which of the multiple data component widths will be used (for all instructions in some embodiments; for only some of the instructions in other embodiments). This field is optional in that it is not required when only one data component width is supported and / or when data component widths are supported using some aspects of the opcodes.
[0280] Write mask field (3070) – The content thereof controls whether the data component location in the destination vector operand reflects the result of the base operation and the augmentation operation on a per-data component location basis. Class A instruction templates support merging-writemasking, whereas Class B instruction templates support both merging-writemasking and zeroing-writemasking. When merging, the vector masks allow any set of components in the destination to be protected from updates during the execution of any operation (specified by the base operation and the augmentation operation); in another embodiment, the old value of each component in the destination is preserved when the corresponding mask bit has 0. In contrast, when zeroing, the vector masks allow any set of components in the destination to be zeroed during the execution of any operation (specified by the base operation and the augmentation operation); in one embodiment, the components in the destination are set to 0 when the corresponding mask bit has a value of 0. A subset of these functionalities is the ability to control the vector length of the operation being performed (i.e., the span of the components is modified from the first to the last); however, the elements being modified do not need to be consecutive. Thus, the write mask field (3070) allows partial vector operations including load, store, arithmetic, logic, etc. Although embodiments of the disclosure are described for selecting one of a plurality of write mask registers containing the write mask to be used for the contents of the write mask field (3070) (and thus indirectly identifying the masking to be performed for the contents of the write mask field (3070)), alternative embodiments allow for directly specifying the masking to be performed for the contents of the mask write field (3070) instead or additionally.
[0281] Immediate field (3072) - The content of this allows for the specification of the immediate. This field is optional in that it does not exist in implementations of general vector-friendly formats that do not support the immediate, and does not exist in instructions that do not use the immediate.
[0282] Class field (3068) – The content of this distinguishes between different classes of instructions. Referring to FIG. 30a and FIG. 30d, the content of this field selects between Class A instructions and Class B instructions. In FIG. 30a and FIG. 30b, rounded corner squares are used to indicate that a specific value exists in the field (e.g., Class A (3068A) and Class B (3068B) for the class field (3068) in FIG. 30a and FIG. 30b, respectively).
[0283] Command templates of Class A
[0284] In the case of Class A no-memory-access (3005) instruction templates, the alpha field (3052) is interpreted as the RS field (3052A), and its content distinguishes which of the different augmentation operation types should be performed (e.g., round (3052A.1) and data conversion (3052A.2) are respectively specified for no-memory-access, round type operation (3010) and no-memory-access, data conversion type operation (3015) instruction templates), while the beta field (3054) distinguishes which of the specific types of operations should be performed. In no-memory-access (3005) instruction templates, the scaling field (3060), the displacement field (3062A), and the displacement factor field (3062B) do not exist.
[0285] No-memory-access instruction templates - Full-round controlled operations
[0286] In the memory access-free full round control operation (3010) instruction template, the beta field (3054) is interpreted as the round control field (3054A), and its contents provide static rounding. In the embodiments described in this disclosure, the round control field (3054A) includes the suppress all floating point exceptions (SAE) field (3056) and the round operation control field (3058), but alternative embodiments may encode both of these concepts into the same field, or have only one of these concepts / fields (e.g., may have only the round operation control field (3058)).
[0287] SAE field (3056) - The content of this distinguishes whether to disable exception event reporting; when the content of the SAE field (3056) indicates that suppression is enabled, the given command does not report any kind of floating-point exception flag and does not raise any floating-point exception handler.
[0288] Round operation control field (3058) – the contents thereof distinguish which of the groups of rounding operations to perform (e.g., Round-up, Round-down, Round-towards-zero, and Round-to-nearest). Thus, the round operation control field (3058) allows for a change in the rounding mode on an instruction-by-instruction basis. In one embodiment of the present disclosure, the contents of the round operation control field (3050) override the value of the register.
[0289] No-memory-access instruction templates - Data conversion operations
[0290] In the memory access-free data conversion type operation (3015) instruction template, the beta field (3054) is interpreted as the data conversion field (3054B), and its contents distinguish which of the multiple data conversions should be executed (e.g., no data conversion, swizzle, broadcast).
[0291] In the case of a memory access (3020) instruction template of class A, the alpha field (3052) is interpreted as an eviction hint field (3052B), and its content distinguishes which of the eviction hints should be used (in FIG. 30a, the memory access, transient (3025) instruction template and the memory access, non-transient (3030) instruction template are respectively specified as transient (3052B.1) and non-transient (3052B.2), while the beta field (3054) is interpreted as a data manipulation field (3054C), and its content distinguishes which of the multiple data manipulation operations (also known as primitives) should be performed (e.g., no manipulation; broadcast; up-conversion of source; and down-conversion of destination). Memory access (3020) instruction templates include a scaling field (3060) and optionally a displacement field (3062A) or a displacement factor field (3062B).
[0292] Vector memory instructions perform vector loads from memory and vector stores into memory with the support of transformations. As with regular vector instructions, vector memory instructions transfer data from / to memory on a per-data basis, and the components actually transferred are determined by the contents of a vector mask selected as a write mask.
[0293] Memory access instruction templates - temporary
[0294] Transient data is data that is likely to be reused quickly enough to benefit from caching. However, this is a hint, and different processors can implement it in different ways, including completely ignoring the hint.
[0295] Memory access instruction templates - non-transient
[0296] Non-transient data is data that is unlikely to be reused quickly enough to benefit from caching within the first-level cache, and should be given priority for evicting. However, this is a hint, and different processors may implement it in different ways, including completely ignoring the hint.
[0297] Command templates of Class B
[0298] In the case of command templates of class B, the alpha field (3052) is interpreted as a write mask control (Z) field (3052C), the contents of which distinguish whether the write masking controlled by the write mask field (3070) should be merged or zeroed.
[0299] In the case of Class B no-memory-access (3005) instruction templates, part of the beta field (3054) is interpreted as the RL field (3057A), the contents of which distinguish which of the different augmentation operation types should be performed (e.g., for the no-memory-access, write-mask-control, partial-round-control type operation (3012) instruction template and the no-memory-access, write-mask-control, VSIZE type operation (3017) instruction template, the round (3057A.1) and vector length (VSIZE) (3057A.2) are respectively specified), while the remainder of the beta field (3054) distinguishes which of the specific types of operations should be performed. In the no-memory-access (3005) instruction templates, the scaling field (3060), the displacement field (3062A), and the displacement factor field (3062B) do not exist.
[0300] In the no memory access, write mask control, partial round control type operation (3010) instruction template, the remainder of the beta field (3054) is interpreted as the round operation field (3059A), and exception event reporting is disabled (the given instruction does not report any kind of floating-point exception flag and does not raise any floating-point exception handler).
[0301] Round operation control field (3059A) – Like the round operation control field (3058), its contents distinguish which of the groups of round operations to perform (e.g., round-up, round-down, round to zero, and round to nearest value). Thus, the round operation control field (3059A) allows for a change in the rounding mode on a per-instruction basis. In one embodiment of the present disclosure, the processor includes a control register for specifying rounding modes, the contents of the round operation control field (3050) override the value of that register.
[0302] In the no memory access, write mask control, VSIZE type operation (3017) instruction template, the remainder of the beta field (3054) is interpreted as a vector length field (3059B), the contents of which distinguish which of the multiple data vector lengths to be performed (e.g., 128, 256, or 512 bytes).
[0303] In the case of a memory access (3020) instruction template of class B, a portion of the beta field (3054) is interpreted as a broadcast field (3057B), the contents of which distinguish whether a broadcast type data manipulation operation should be performed, while the remainder of the beta field (3054) is interpreted as a vector length field (3059B). Memory access (3020) instruction templates include a scaling field (3060) and optionally a displacement field (3062A) or a displacement factor field (3062B).
[0304] In relation to a general vector-friendly instruction format (3000), a full opcode field (3074) is illustrated, comprising a format field (3040), a base operation field (3042), and a data component width field (3064). While an embodiment is illustrated in which the full opcode field (3074) includes all of these fields, in embodiments that do not support all of them, the full opcode field (3074) includes fewer fields than all of these fields. The full opcode field (3074) provides an opcode.
[0305] The augmentation operation field (3050), data component width field (3064), and write mask field (3070) allow these features to be specified in a general vector-friendly instruction format on a per-instruction basis.
[0306] The combination of the entry mask field and the data component width field generates typed instructions in that they allow the mask to be applied based on different data component widths.
[0307] Various instruction templates found within Class A and Class B are advantageous in different situations. In some embodiments of the present disclosure, different processors or different cores within a processor may support only Class A, only Class B, or both classes. For example, a high-performance general-purpose out-of-order core intended for general-purpose computing may support only Class B, a core intended primarily for graphics and / or scientific (throughput) computing may support only Class A, and a core intended for both may support both (of course, a core having some mix of instructions and templates from both classes but not all instructions and templates from both classes is within the scope of the present disclosure). Additionally, a single processor may include multiple cores, all of which support the same class or different cores support different classes. For example, in a processor having separate graphics and general-purpose cores, one of the graphics cores intended primarily for graphics and / or scientific computing may support only Class A, whereas one or more of the general-purpose cores may be high-performance general-purpose cores intended for general-purpose computing with out-of-order execution and register renaming that support only Class B. Another processor that does not have separate graphics cores may include one or more general-purpose sequential or out-of-order cores that support both Class A and Class B. Of course, features from one class may also be implemented in another class in different embodiments of the present disclosure.Programs written in a high-level language will be in various and different executable forms, including: 1) a form having only the instructions of the class(s) supported by the target processor for execution; or 2) a form having alternative routines written using different combinations of instructions of all classes and control flow code that selects routines to be executed based on the instructions supported by the processor currently executing the code (e.g., will be just-in-time compiled or statically compiled).
[0308] Exemplary specific vector-friendly command format
[0309] FIG. 31 is a block diagram illustrating an exemplary specific vector-friendly instruction format according to embodiments of the present disclosure. FIG. 31 illustrates a specific specific vector-friendly instruction format (3100) in that it specifies the location, size, interpretation, and order of fields, as well as the values for some of these fields. The specific vector-friendly instruction format (3100) can be used to extend an x86 instruction set, and thus some of the fields are similar or identical to those used in the existing x86 instruction set and its extensions (e.g., AVX). This format is maintained consistently with the prefix encoding field, real opcode byte field, MOD R / M field, SIB field, displacement field, and immediacy field of the existing x86 instruction set having extensions. Fields from FIG. 30 to which the fields from FIG. 31 are mapped are illustrated.
[0310] Embodiments of the present disclosure are described with reference to a specific vector-friendly instruction format (3100) in the context of a general vector-friendly instruction format (3000) for illustrative purposes, but it should be understood that the present invention is not limited to the specific vector-friendly instruction format (3100) except as described in the claims. For example, while the general vector-friendly instruction format (3000) considers various possible sizes for various fields, the specific vector-friendly instruction format (3100) is illustrated as having fields of a specific size. As a specific example, the data component width field (3064) is illustrated as a 1-bit field in the specific vector-friendly instruction format (3100), but the present disclosure is not limited to this [i.e., the general vector-friendly instruction format (3000) considers other sizes of the data component width field (3064)].
[0311] The general vector-friendly command format (3000) includes the following fields listed below in the order shown in FIG. 31a.
[0312] EVEX prefix (bytes 0-3)(3102) - encoded in the form of 4 bytes.
[0313] Format field (3040) (EVEX byte 0, bits [7:0]) - The first byte (EVEX byte 0) is the format field (3040), which includes 0x62 (a unique value used to distinguish vector-friendly instruction formats in one embodiment of the present disclosure).
[0314] The second to fourth bytes (EVEX bytes 1-3) include a plurality of bit fields that provide specific capabilities.
[0315] The REX field (3105) (EVEX byte 1, bits [7-5]) consists of the EVEX.R bit field (EVEX byte 1, bit - R), the EVEX.X bit field (EVEX byte 1, bit - X), and the EVEX.B bit field (EVEX byte 1, bit - B). The EVEX.R, EVEX.X, and EVEX.B bit fields provide the same function as the corresponding VEX bit fields and are encoded using one's complement form, that is, ZMM0 is encoded as 1111B and ZMM15 is encoded as 0000B. The other fields of the instructions are encoded by encoding the lower 3 bits of the register indices (rrr, xxx, and bbb) as known in the art, so that Rrrr, Xxxx, and Bbbb can be formed by adding EVEX.R, EVEX.X, and EVEX.B.
[0316] REX' field (3110) - This is the first part of the REX' field (3110) and is the EVEX.R' bit field (EVEX byte 1, bit [4]-R') used to encode either the upper 16 or the lower 16 of the extended 32 register set. In one embodiment of the present disclosure, this bit is stored in a bit-inverted format to distinguish it from the BOUND instruction (in the well-known x86 32-bit mode), along with others as shown below, and its actual opcode byte is 62, but does not accept the value of 11 in the MOD field in the MOD R / M field (described below); alternative embodiments of the present disclosure do not store this and the other bits shown below in an inverted format. A value of 1 is used to encode the lower 16 registers. In other words, R'Rrrr is formed by combining EVEX.R', EVEX.R, and other RRRs from other fields.
[0317] Opcode map field (3115) (EVEX byte 1, bit [3:0] - mmmm) - the contents of this encode an implied leading opcode byte (0F, 0F 38, or 0F 3).
[0318] The data component width field (3064) (EVEX byte 2, bit [7] - W) is represented by the notation EVEX.W. EVEX.W is used to define the granularity (size) of the data type (32-bit data component or 64-bit data component).
[0319] EVEX.vvvv(3120)(EVEX byte 2, bits [6:3]-vvvv) - The role of EVEX.vvvv may include: 1) EVEX.vvvv encodes a first source register operand valid for instructions having two or more source operands specified in inverted (one's complement) form; 2) EVEX.vvvv encodes a destination register operand specified in one's complement form for specific vector shifts; or 3) EVEX.vvvv encodes no operands, and the field is reserved and must contain 1111b. Thus, the EVEX.vvvv field (3120) encodes the four lower bits of the first source register specifier stored in inverted (one's complement) form. Depending on the instruction, additional different EVEX bit fields are used to extend the specifier size to 32 registers.
[0320] EVEX.U (3068) class field (EVEX byte 2, bit -U) - If EVEX.U=0, this indicates class A or EVEX.U0; if EVEX.U=1, this indicates class B or EVEX.U1.
[0321] The prefix encoding field (3125) (EVEX byte 2, bits [1:0] - pp) provides additional bits for the base operation field. In addition to providing support for legacy SSE instructions in the EVEX prefix format, this also has the benefit of simplifying SIMD prefixes (whereas a byte is not required to represent a SIMD prefix, an EVEX prefix requires only 2 bits). In one embodiment, to support legacy SSE instructions using SIMD prefixes (66H, F2H, F3H) in both the legacy format and the EVEX prefix format, these legacy SIMD prefixes are encoded in the SIMD prefix encoding field; and expanded into legacy SIMD prefixes before being provided to the decoder's PLA at runtime (so that the PLA can execute both the legacy and the EVEX format of these legacy instructions without modification). While newer instructions can directly use the contents of EVEX prefix encoding fields as opcode extensions, certain embodiments are extended in a similar manner for consistency, but allow different meanings to be specified by these legacy SIMD prefixes. An alternative embodiment can redesign the PLA to support 2-bit SIMD prefix encodings and thus does not require extension.
[0322] Alpha field (3052) (EVEX byte 3, bit [7]-EH; also known as EVEX.EH, EVEX.rs, EVEX.RL, EVEX.write mask control, and EVEX.N; also exemplified by α) - as previously described, this field is context-specific.
[0323] Beta field (3054)(EVEX byte 3, bit [6:4]-SSS, EVEX.s 2-0 , EVEX.r 2-0, also known as EVEX.rr1, EVEX.LL0, EVEX.LLB; also exemplified by βββ) - As previously explained, this field is context-specific.
[0324] REX' field (3110) - This is the remainder of the REX' field and is the EVEX.V' bit field (EVEX byte 3, bit [3]-V') which can be used to encode either the upper 16 or lower 16 of the extended 32 register set. This bit is stored in a bit-inverted format. A value of 1 is used to encode the lower 16 registers. In other words, V'VVVV is formed by combining EVEX.V' and EVEX.vvvv.
[0325] Write mask field (3070) (EVEX byte 3, bits [2:0] - kkk) - the contents thereof specify the index of a register in the write mask registers as previously described. In one embodiment of the present disclosure, a specific value EVEX.kkk=000 has special behavior implying that no write mask is used for a specific instruction (this may be implemented in various ways, including using a write mask hardwired to all or hardware that bypasses the masking hardware).
[0326] The actual opcode field (3130) (byte 4) is also known as the opcode byte. Part of the opcode is specified in this field.
[0327] The MOD R / M field (3140) (byte 5) includes the MOD field (3142), the Reg field (3144), and the R / M field (3146). As previously described, the contents of the MOD field (3142) distinguish between memory access operations and no memory access operations. The role of the Reg field (3144) can be summarized into two situations: encoding either a destination register operand or a source register operand, or being treated as an opicode extension and not used to encode any instruction operand. The role of the R / M field (3146) may include: encoding an instruction operand that references a memory address, or encoding a destination register operand or a source register operand.
[0328] SIB(Scale, Index, Base) byte (byte 6) - As previously mentioned, the contents of the scaling field (3050) are used to generate memory addresses. SIB.xxx (3154) and SIB.bbb (3156) - The contents of these fields were previously mentioned regarding the register indices Xxxx and Bbbb.
[0329] Displacement field (3062A) (bytes 7-10) - When the MOD field (3142) contains 10, byte 7-10 is the displacement field (3062A), which acts as the same as the legacy 32-bit displacement (disp32) and acts as the byte granularity.
[0330] Displacement Factor Field (3062B) (Byte 7) - When the MOD field (3142) contains 01, Byte 7 is the displacement factor field (3062B). The location of this field is identical to the location of the legacy x86 instruction set 8-bit displacement (disp8), which acts as byte granularity. Because disp8 is sign-extended, it can only address between -128 and 127-byte offsets; regarding 64-byte cache lines, disp8 uses 8 bits that can only be set to four actually useful values: -128, -64, 0, and 64; because a larger range is often needed, disp32 is used; however, disp32 requires 4 bytes. In contrast to disp8 and disp32, the displacement factor field (3062B) is a reinterpretation of disp8; When using the displacement factor field (3062B), the actual displacement is determined by multiplying the contents of the displacement factor field by the size (N) of the memory operand access. This type of displacement is referred to as disp8*N. This reduces the average instruction length (a single byte is used for the displacement, but it has a much larger range). This compressed displacement is based on the assumption that the effective displacement is a multiple of the granularity of the memory access, and therefore the redundant lower bits of the address offset do not need to be encoded. In other words, the displacement factor field (3062B) replaces the legacy x86 instruction set 8-bit displacement. Thus, the displacement factor field (3062B) is encoded in the same way as the x86 instruction set 8-bit displacement, except that disp8 is overloaded as disp8*N (so nothing changes in the ModRM / SIB encoding rules).In other words, there are no changes in the encoding rules or encoding lengths, but only in the interpretation of the displacement value by the hardware (which needs to scale the displacement by the size of the memory operand to obtain the byte-wise address offset). The immediate field (3072) operates as previously described.
[0331] Full OP code field
[0332] FIG. 31b is a block diagram illustrating fields of a specific vector-friendly instruction format (3100) constituting a full opcode field (3074) according to one embodiment of the present disclosure. Specifically, the full opcode field (3074) includes a format field (3040), a base operation field (3042), and a data component width (W) field (3064). The base operation field (3042) includes a prefix encoding field (3125), an opcode map field (3115), and an actual opcode field (3130).
[0333] Register index field
[0334] FIG. 31c is a block diagram illustrating fields of a specific vector-friendly instruction format (3100) that constitute a register index field (3044) according to one embodiment of the present disclosure. Specifically, the register index field (3044) includes a REX field (3105), a REX' field (3110), a MODR / M.reg field (3144), a MODR / Mr / m field (3146), a VVVV field (3120), an xxx field (3154), and a bbb field (3156).
[0335] Augmentation operation field
[0336] FIG. 31d is a block diagram illustrating fields of a specific vector-friendly instruction format (3100) that constitute an augmentation operation field (3050) according to one embodiment of the present disclosure. When the class (U) field (3068) contains 0, it signifies EVEX.U0 (Class A 3068A); when it contains 1, it signifies EVEX.U1 (Class B 3068B). When U=0 and the MOD field (3142) contains 11 (indicating no memory access operation), the alpha field (3052) (EVEX byte 3, bit [7] - EH) is interpreted as the rs field (3052A). When the rs field (3052A) contains 1 (round (3052A.1)), the beta field (3054) (EVEX byte 3, bits [6:4] - SSS) is interpreted as the round control field (3054A). The round control field (3054A) contains a 1-bit SAE field (3056) and a 2-bit round operation field (3058). When the rs field (3052A) contains 0 (data conversion (3052A.2)), the beta field (3054) (EVEX byte 3, bits [6:4] - SSS) is interpreted as a 3-bit data conversion field (3054B). When U=0 and the MOD field (3142) contains 00, 01, or 10 (indicating a memory access operation), the alpha field (3052) (EVEX byte 3, bit [7] - EH) is interpreted as an ejection hint (EH) field (3052B), and the beta field (3054) (EVEX byte 3, bits [6:4] - SSS) is interpreted as a 3-bit data manipulation field (3054C).
[0337] When U=1, the alpha field (3052) (EVEX byte 3, bit [7] - EH) is interpreted as the write mask control (Z) field (3052C). When U=1 and the MOD field (3142) contains 11 (indicating no memory access operation), part of the beta field (3054) (EVEX byte 3, bit [4] - S0) is interpreted as the RL field (3057A); when this contains 1 (round 3057A.1), the remainder of the beta field (3054) (EVEX byte 3, bits [6-5] - S 2-1 While ) is interpreted as a round operation field (3059A), when the RL (3057A) field contains 0 (VSIZE 3057.A2), the remainder of the beta field (3054) (EVEX byte 3, bits [6-5]- S 2-1 ) is a vector length field (3059B)(EVEX byte 3, bits [6-5]- L 1-0 It is interpreted as ). If U=1 and the MOD field (3142) contains 00, 01, or 10 (indicating a memory access operation), the beta field (3054) (EVEX byte 3, bits [6:4] - SSS) is interpreted as the vector length field (3059B) (EVEX byte 3, bits [6-5] - L 1-0 ) and is interpreted as a broadcast field (3057B) (EVEX byte 3, bit [4] - B).
[0338] Exemplary register architecture
[0339] FIG. 32 is a block diagram of a register architecture (3200) according to one embodiment of the present disclosure. In the illustrated embodiment, there are 32 vector registers (3210) having a width of 512 bits; these registers are referenced as zmm0 through zmm31. The lower 256 bits of the lower 16 zmm registers are overlaid on registers ymm0-16. The lower 128 bits of the lower 16 zmm registers (the lower 128 bits of the ymm registers) are overlaid on registers xmm0-15. A specific vector-friendly instruction format (3100) is operated on these overlaid register files as illustrated in the table below.
[0340]
[0341] In other words, the vector length field (3059B) selects between a maximum length and one or more other shorter lengths, wherein each such shorter length is half the length of the preceding length; instruction templates that do not have the vector length field (3059B) perform operations on the maximum vector length. Additionally, in one embodiment, Class B instruction templates of a specific vector-friendly instruction format (3100) perform operations on packed or scalar single / double-precision floating-point data and packed or scalar integer data. Scalar operations are operations performed on the lowest data component locations within the zmm / ymm / xmm registers; and the upper data component locations are left as they were before the instruction or are zeroed out, depending on the embodiment. Write mask registers (3215) - In the illustrated embodiment, there are eight write mask registers (k0 through k7), each of which has a size of 64 bits. In an alternative embodiment, the write mask registers (3215) have a size of 16 bits. As previously described, in one embodiment of the present disclosure, the vector mask register k0 cannot be used as a write mask; and when an encoding that typically represents k0 is used for the write mask, this selects a hardwired write mask of 0xFFFF to effectively disable write masking for the corresponding instruction.
[0342] General-purpose registers (3225) - In the illustrated embodiment, there are 16 64-bit general-purpose registers used with existing x86 addressing modes to address memory operands. These registers are referred to by the names RAX, RBX, RCX, RDX, RBP, RSI, RDI, RSP, and R8 through R15.
[0343] An MMX-packed integer flat register file (3250) is aliased on a scalar floating-point stack register file (x87 stack) (3245) - in the illustrated embodiment, the x87 stack is an 8-component stack used to execute scalar floating-point operations on 32 / 64 / 80-bit floating-point data using x87 instruction set extensions; the MMX registers are used not only to execute operations on 64-bit packed integer data but also to hold operands for some operations executed between the MMX registers and the XMM registers.
[0344] Alternative embodiments of the present disclosure may use wider or narrower registers. Additionally, alternative embodiments of the present disclosure may use more, fewer, or different register files and registers.
[0345] Exemplary core architecture, processor, and computer architecture
[0346] Processor cores may be implemented in different ways, for different purposes, and on different processors. For example, implementations of these cores may include 1) general-purpose sequential cores intended for general-purpose computing; 2) high-performance general-purpose non-sequential cores intended for general-purpose computing; and 3) special-purpose cores intended primarily for graphics and / or scientific (throughput) computing. Implementations of different processors may include: 1) a CPU comprising one or more general-purpose sequential cores intended for general-purpose computing and / or one or more general-purpose non-sequential cores intended for general-purpose computing; and 2) a coprocessor comprising one or more special-purpose cores intended primarily for graphics and / or scientific (throughput). Such different processors lead to different computer system architectures, which include: 1) a coprocessor on a chip separate from the CPU; 2) a separate die or more of coprocessors in the same package as the CPU; 3) a coprocessor on the same die as the CPU (in this case, such processors are sometimes referred to as special-purpose logic, such as integrated graphics and / or scientific (throughput) logic or special-purpose cores); 4) may include a system on a chip (SoC) that may include the CPU described on the same die (sometimes referred to as application core(s) or application processor(s), the coprocessor described above, and additional functionality. Exemplary core architectures are described below, followed by exemplary processors and computer architectures.
[0347] Exemplary Core Architectures
[0348] Sequential and non-sequential core block diagrams
[0349] FIG. 33a is a block diagram illustrating both an exemplary sequential pipeline and an exemplary register renaming and out-of-order publish / execute pipeline according to embodiments of the present disclosure. FIG. 33b is a block diagram illustrating both an exemplary embodiment of a sequential architecture core to be included in a processor according to embodiments of the present disclosure and an exemplary register renaming and out-of-order publish / execute architecture core. The solid boxes in FIG. 33a and FIG. 33b illustrate the sequential pipeline and sequential core, while optional additions of dashed boxes illustrate the register renaming, out-of-order publish / execute pipeline and core. The out-of-order aspect will be described with consideration that the sequential aspect is a subset of the out-of-order aspect.
[0350] In FIG. 33a, the processor pipeline (3300) includes a fetch stage (3302), a length decoding stage (3304), a decoding stage (3306), an allocation stage (3308), a renaming stage (3310), a scheduling (also known as dispatch or issue) stage (3312), a register read / memory read stage (3314), an execution stage (3316), a writeback / memory write stage (3318), an exception handling stage (3322), and a commit stage (3324).
[0351] FIG. 33b illustrates a processor core (3390) comprising a front-end unit (3330) coupled to an execution engine unit (3350), both of which are coupled to a memory unit (3370). The core (3390) may be a RISC (reduced instruction set computing) core, a CISC (complex instruction set computing) core, a VLIW (very long instruction word) core, or a hybrid or alternative core type. As another option, the core (3390) may be a special-purpose core, such as, for example, a network or communication core, a compression engine, a coprocessor core, a GPGPU (general purpose computing graphics processing unit) core, a graphics core, or something similar.
[0352] The front-end unit (3330) includes a branch prediction unit (3332) coupled to an instruction cache unit (3334) coupled to an instruction TLB (translation lookaside buffer) (3336) coupled to an instruction fetch unit (3338) coupled to a decoding unit (3340). The decoding unit (3340) (e.g., a decoding circuit) can decode instructions (e.g., macro instructions) and can generate, as output, one or more micro-operations, microcode entry points, micro-instructions, other instructions, or other control signals, which are decoded from the original instructions, or otherwise reflect the original instructions, or are derived from the original instructions. The decoding unit (3340) can be implemented using various different mechanisms. Examples of suitable mechanisms include, but are not limited to, lookup tables, hardware implementations, PLAs (programmable logic arrays), microcode ROMs (read-only memory), etc. In one embodiment, the core (3390) includes a microcode ROM or other medium that stores microcode for specific macro instructions (e.g., in the decode unit (3340) or otherwise within the front-end unit (3330). The decoding unit (3340) is coupled to the rename / assignor unit (3352) in the execution engine unit (3350).
[0353] The execution engine unit (3350) includes a set of one or more scheduler unit(s) (3356) and a rename / assignor unit (3352) coupled to a retirement unit (3354). The scheduler unit(s) (3356) represent any number of different schedulers, including reservation stations, a central instruction window, etc. The scheduler unit(s) (3356) are coupled to physical register file(s) unit(s) (3358). Each of the physical register file(s) (3358) represents one or more physical register files, and the different physical register files among them store one or more different data types, such as scalar integers, scalar floating-point numbers, packed integers, packed floating-point numbers, vector integers, vector floating-point numbers, states (e.g., instruction pointers, which are the addresses of the next instruction to be executed), tiles, etc. In one embodiment, the physical register file(s) unit (3358) includes a vector register unit, a write mask register unit, and a scalar register unit. These register units may provide architectural vector registers, vector mask registers, and general-purpose registers. The physical register file(s) unit(s) (3358) is overlapped by a retirement circuit (3354) to exemplify various ways in which register renaming and out-of-order execution can be implemented (e.g., by using reorder buffer(s) and retirement register file(s); by using future file(s), history buffer(s), and retirement register file(s); by using register maps and a pool of registers, etc.). The retirement unit (3354) and the physical register file(s) unit(s) (3358) are coupled to an execution cluster(s) (3360). The execution cluster(s) (3360) includes a set of one or more execution units (3362) (e.g., execution circuits) and a set of one or more memory access units (3364).The execution units (3362) can perform various operations (e.g., shift, addition, subtraction, multiplication) on various types of data (e.g., scalar floating-point, packed integer, packed floating-point, vector integer, vector floating-point). Some embodiments may include multiple execution units dedicated to specific functions or sets of functions, but other embodiments may include a single execution unit or multiple execution units in which all perform all functions. The scheduler unit(s) (3356), physical register file(s) unit(s) (3358), and execution cluster(s) (3360) are depicted as possibly being multiple, because certain embodiments may generate individual pipelines for certain types of data / operations (e.g., scalar integer pipeline, scalar floating-point / packed integer / packed floating-point / vector integer / vector floating-point pipeline, and / or memory access pipelines, each having its own scheduler unit, physical register file(s) unit, and / or execution cluster—in the case of individual memory access pipelines, certain embodiments are implemented in which only the execution cluster of such pipeline has memory access unit(s) (3364). It should also be understood that when separate pipelines are used, one or more of these pipelines may be non-sequential issue / execute and the others may be sequential.
[0354] A set of memory access units (3364) is coupled to a memory unit (3370), which includes a data TLB unit (3372) coupled to a data cache unit (3374) coupled to a level 2 (L2) cache unit (3376). In one exemplary embodiment, the memory access units (3364) may include a load unit, a store address unit, and a store data unit, each of which is coupled to a data TLB unit (3372) in the memory unit (3370). An instruction cache unit (3334) is also coupled to a level 2 (L2) cache unit (3376) in the memory unit (3370). The L2 cache unit (3376) is coupled to one or more other levels of cache and finally to main memory.
[0355] For example, an exemplary register renaming, out-of-order issue / execute core architecture may implement the pipeline (3300) as follows: 1) an instruction fetch (3338) performs fetch and length decoding stages (3302 and 3304); 2) a decoding unit (3340) performs a decoding stage (3306); 3) a rename / allocator unit (3352) performs an allocation stage (3308) and a renaming stage (3310); 4) a scheduler unit(s) (3356) performs a scheduling stage (3312); 5) a physical register file(s) unit(s) (3358) and a memory unit (3370) perform a register read / memory read stage (3314); an execution cluster (3360) performs an execution stage (3316); 6) The memory unit (3370) and the physical register file(s) unit(s) (3358) perform the write-back / memory write stage (3318); 7) Various units may be involved in the exception handling stage (3322); and 8) The retirement unit (3354) and the physical register file(s) unit(s) (3358) perform the commit stage (3324).
[0356] The core (3390) may support one or more instruction sets (e.g., an x86 instruction set with some extensions with new versions added), including the instruction(s) described herein; a MIPS instruction set from MIPS Technologies of Sunnyvale, California; and an ARM instruction set from ARM Holdings of Sunnyvale, California (with optional additional extensions such as NEON). In one embodiment, the core (3390) includes logic that supports packed data instruction set extensions (e.g., AVX1, AVX2), thereby allowing operations used by many multimedia applications to be performed using packed data.
[0357] It should be understood that a core can support multithreading (executing two or more parallel sets of operations or threads) and can do so in various ways, including time-sliced multithreading, concurrent multithreading (in which case a single physical core provides a logical core to each thread that the physical core is simultaneously multithreading), or a combination thereof (e.g., time-sliced fetching and decoding and subsequent concurrent multithreading as in Intel® Hyper-Threading technology).
[0358] Although register renaming has been described in the context of out-of-order execution, it should be understood that register renaming can be used in sequential architectures. While the illustrated embodiment of the processor also includes separate instruction and data cache units (3334 / 3374) and a shared L2 cache unit (3376), alternative embodiments may have a single internal cache for both instructions and data, such as a Level 1 (L1) internal cache or multiple levels of internal cache. In some embodiments, the system may include a combination of an internal cache and an external cache located outside the core and / or processor. Alternatively, all caches may be located outside the core and / or processor.
[0359] Specific exemplary sequential core architecture
[0360] FIGS. 34a and 34b illustrate block diagrams of a more specific and exemplary sequential core architecture, in which the core may be one of several logic blocks within the chip (including other cores of the same type and / or different types). The logic blocks communicate, depending on the application, through a high-bandwidth interconnect network (e.g., a ring network) having some fixed-function logic, memory I / O interfaces, and other necessary I / O logic.
[0361] FIG. 34a is a block diagram illustrating, according to embodiments of the present disclosure, a single processor core with its connection to an on-die interconnect network (3402) and its local subset (3404) of a Level 2 (L2) cache. In one embodiment, an instruction decoder unit (3400) supports an x86 instruction set having a packed data instruction set extension. An L1 cache (3406) allows low-latency accesses to scalar and vector units of cache memory. (To simplify the design) In one embodiment, the scalar unit (3408) and the vector unit (3410) use separate sets of registers (each using scalar registers (3412) and vector registers (3414)), and data transferred between them is written to memory and then read back from a level 1 (L1) cache (3406), but alternative embodiments of the present disclosure may use different approaches (e.g., using a single set of registers, or including a communication path that allows data to be transferred between two register files without being written and read back).
[0362] A local subset (3404) of the L2 cache is part of the global L2 cache, which is divided into distinct local subsets, one per processor core. Each processor core has a direct access path to its own local subset (3404) of the L2 cache. Data read by a processor core is stored in that L2 cache subset (3404) and can be accessed quickly in parallel with other processor cores accessing their own local L2 cache subsets. Data written by a processor core is stored in its own L2 cache subset (3404) and is flushed from other subsets if necessary. A ring network ensures coherence for shared data. The ring network is bidirectional, allowing agents such as processor cores, L2 caches, and other logic blocks to communicate with each other within the chip. Each ring data path is 10¹² bits wide per direction.
[0363] FIG. 34b is an enlarged view of a portion of the processor core of FIG. 34a according to embodiments of the present disclosure. FIG. 34b includes more details regarding the vector unit (3410) and vector registers (3414), as well as the portion of the L1 data cache (3406A) of the L1 cache (3406). Specifically, the vector unit (3410) is a 16-width vector processing unit (VPU) (see 16-width ALU (3428)), which executes one or more of integer, single-precision floating-point and double-precision floating-point instructions.
[0364] The VPU supports swizzling of register inputs by a swizzle unit (3420), numerical conversion by numerical conversion units (3422A-B), and duplication using a duplication unit (3424) for memory inputs. Write mask registers (3426) allow predicating resulting vector writes.
[0365] FIG. 35 is a block diagram of a processor (3500) that may have two or more cores, may have an integrated memory controller, and may have integrated graphics, according to embodiments of the present disclosure. The solid boxes of FIG. 35 illustrate a processor (3500) having a single core (3502A), a system agent (3510), and a set of one or more bus controller units (3516), while optional additions of dashed boxes illustrate an alternative processor (3500) having multiple cores (3502A-N), a set of one or more integrated memory controller units (3514) within the system agent unit (3510), and special purpose logic (3508).
[0366] Accordingly, different implementations of the processor (3500) may include: 1) a CPU having special purpose logic (3508) which is integrated graphics and / or scientific (throughput) logic (which may include one or more cores), and cores (3502A-N) which are one or more general-purpose cores (e.g., general-purpose sequential cores, general-purpose non-sequential cores, or a combination of the two); 2) a coprocessor having cores (3502A-N) which are a large number of special purpose cores intended primarily for graphics and / or scientific (throughput); and 3) a coprocessor having cores (3502A-N) which are a large number of general-purpose sequential cores. Accordingly, the processor (3500) may be a general-purpose processor, for example, a network or communication processor, a compression engine, a graphics processor, a GPGPU (general purpose graphics processing unit), a high-throughput MIC (many integrated core) coprocessor (containing 30 or more cores), an embedded processor, or a similar coprocessor or special-purpose processor. The processor may be implemented on one or more chips. The processor (3500) may be implemented on one or more substrates using any of a number of process technologies, such as BiCMOS, CMOS, or NMOS, for example, and / or may be part of such.
[0367] The memory hierarchy includes one or more levels of cache within the cores, a set of one or more shared cache units (3506), and external memory (not shown) coupled to a set of integrated memory controller units (3514). The set of shared cache units (3506) may include one or more intermediate level caches, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, a last level cache (LLC), and / or a combination thereof. In one embodiment, a ring-based interconnect unit (3512) interconnects integrated graphics logic (3508), a set of shared cache units (3506), and a system agent unit (3510) / integrated memory controller unit(s) (3514), but alternative embodiments may use any number of known techniques to interconnect these units. In one embodiment, coherence is maintained between one or more cache units (3506) and cores (3502A-N).
[0368] In some embodiments, one or more of the cores (3502A-N) are multithreadable. The system agent (3510) includes components that coordinate and operate the cores (3502A-N). The system agent unit (3510) may include, for example, a power control unit (PCU) and a display unit. The PCU may include or contain logic and components necessary to control the power state of the cores (3502A-N) and the integrated graphics logic (3508). The display unit is intended to drive one or more externally connected displays.
[0369] The cores (3502A-N) may be homogeneous or heterogeneous in terms of the architectural instruction set; that is, two or more of the cores (3502A-N) may execute the same instruction set, while other cores may execute only a subset of that instruction set or a different instruction set.
[0370] Exemplary computer architectures
[0371] FIGS. 36 through 39 are block diagrams of exemplary computer architectures. Other system designs and configurations known in the art are also suitable for laptops, desktops, handheld PCs, personal digital assistants, engineering workstations, servers, network devices, network hubs, switches, embedded processors, digital signal processors (DSPs), graphics devices, video game devices, set-top boxes, microcontrollers, mobile phones, portable media players, handheld devices, and various other electronic devices. In general, a wide variety of systems or electronic devices capable of integrating processors and / or other execution logic as disclosed herein are generally suitable.
[0372] Now, referring to FIG. 36, a block diagram of a system (3600) according to one embodiment of the present disclosure is shown. The system (3600) may include one or more processors (3610, 3615) coupled to a controller hub (3620). In one embodiment, the controller hub (3620) includes a graphics memory controller hub (GMCH) (3690) and an input / output hub (IOH) (3650) (which may be on separate chips); the GMCH (3690) includes memory and graphics controllers to which a memory (3640) and a coprocessor (3645) are coupled; and the IOH (3650) couples input / output (I / O) devices (3660) to the GMCH (3690). Alternatively, one or both of the memory and graphics controllers are integrated within the processor (as described herein), and the memory (3640) and coprocessor (3645) are directly coupled to the processor (3610) and the controller hub (3620) within a single chip with the IOH (3650). The memory (3640) may include, for example, matrix acceleration code (3640A) that stores code that causes the processor to perform any method of the present disclosure when executed.
[0373] The optional nature of the additional processors (3615) is indicated by dashed lines in FIG. 36. Each processor (3610, 3615) may include one or more of the processing cores described herein and may be a partial version of the processor (3500).
[0374] The memory (3640) may be, for example, DRAM (dynamic random access memory), PCM (phase change memory), or a combination of both. In at least one embodiment, the controller hub (3620) communicates with the processor(s) (3610, 3615) via a multidrop bus such as the frontside bus (FSB), a point-to-point interface such as QuickPath Interconnect (QPI), or a similar connection (3695).
[0375] In one embodiment, the coprocessor (3645) is a special-purpose processor, such as, for example, a high-throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, an embedded processor, etc. In one embodiment, the controller hub (3620) may include an integrated graphics accelerator.
[0376] There may be various differences between physical resources (3610, 3615) when considering the range of merit criteria including architectural, microarchitectural, thermal, power consumption characteristics and similar ones.
[0377] In one embodiment, the processor (3610) executes instructions that control general type data processing operations. Coprocessor instructions may be embedded within the instructions. The processor (3610) recognizes these coprocessor instructions as being of a type to be executed by the attached coprocessor (3645). Accordingly, the processor (3610) issues these coprocessor instructions (or control signals indicating coprocessor instructions) to the coprocessor (3645) via a coprocessor bus or other interconnect. The coprocessor(s) (3645) accepts and executes the received coprocessor instructions.
[0378] Now, referring to FIG. 37, a block diagram of a first more specific exemplary system (3700) according to an embodiment of the present disclosure is shown. As shown in FIG. 37, the multiprocessor system (3700) is a point-to-point interconnect system and includes a first processor (3770) and a second processor (3780) coupled through a point-to-point interconnect (3750). Each of the processors (3770 and 3780) may be a partial version of the processor (3500). In one embodiment of the present disclosure, the processors (3770 and 3780) are each processors (3610 and 3615), while the coprocessor (3738) is the coprocessor (3645). In another embodiment, the processors (3770 and 3780) are each processor (3610) and coprocessor (3645).
[0379] Processors (3770 and 3780) are each shown to include integrated memory controller (IMC) units (3772 and 3782). Processor (3770) also includes point-to-point (PP) interfaces (3776, 3778) as part of its bus controller units; similarly, the second processor (3780) includes PP interfaces (3786, 3788). Processors (3770, 3780) can exchange information through the point-to-point (PP) interface (3750) using the PP interface circuits (3778, 3788). As illustrated in FIG. 37, the IMCs (3772 and 3782) each combine processors with their respective memories, namely memory (3732) and memory (3734), which may be parts of the main memory locally belonging to their respective processors.
[0380] Processors (3770, 3780) can each exchange information with the chipset (3790) via individual PP interfaces (3752, 3754) using point-to-point interface circuits (3776, 3794, 3786, 3798). The chipset (3790) can optionally exchange information with the coprocessor (3738) via the high-performance interface (3739). In one embodiment, the coprocessor (3738) is a special-purpose processor, such as, for example, a high-throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, an embedded processor, etc.
[0381] A shared cache (not shown) may be included in one processor or be external to both processors but still connected to the processors via a PP interconnect, so that when the processors are placed in low-power mode, local cache information of either or both processors can be stored in the shared cache.
[0382] The chipset (3790) can be coupled to the first bus (3716) via an interface (3796). In one embodiment, the first bus (3716) may be a PCI (Peripheral Component Interconnect) bus, or a bus such as a PCI Express bus or another third-generation I / O interconnect bus, but the scope of the present disclosure is not limited thereto.
[0383] As illustrated in FIG. 37, various I / O devices (3714) may be coupled to the first bus (3716) together with a bus bridge (3718) that couples the first bus (3716) to the second bus (3720). In one embodiment, one or more additional processor(s) (3715), such as a coprocessor, a high-throughput MIC processor, GPGPUs, accelerators (e.g., graphics accelerators or digital signal processing (DSP) units), field programmable gate arrays, or any other processor, are coupled to the first bus (3716). In one embodiment, the second bus (3720) may be a low pin count (LPC) bus. In one embodiment, various devices including, for example, a keyboard and / or mouse (3722), communication devices (3727), and a storage unit (3728), such as a disk drive or other mass storage device that may contain instructions / code and data (3730), may be coupled to the second bus (3720). Additionally, audio I / O (3724) may be coupled to the second bus (3720). Note that other architectures are also possible. For example, instead of the point-to-point architecture of FIG. 37, the system may implement a multidrop bus or other such architecture.
[0384] Now, referring to FIG. 38, a block diagram of a second, more specific exemplary system (3800) according to an embodiment of the present disclosure is shown. Similar elements in FIG. 37 and FIG. 38 have similar reference numerals, and certain aspects of FIG. 37 have been omitted from FIG. 38 to avoid obscuring other aspects of FIG. 38.
[0385] FIG. 38 illustrates that processors (3770, 3780) may each include integrated memory and I / O control logic ("CL") (3772 and 3782). Thus, CL (3772, 3782) includes integrated memory controller units and I / O control logic. FIG. 38 illustrates that not only are memories (3732, 3734) coupled to CL (3772, 3782), but I / O devices (3814) are also coupled to control logic (3772, 3782). Legacy I / O devices (3815) are coupled to the chipset (3790).
[0386] Now, referring to FIG. 39, a block diagram of an SoC (3900) according to an embodiment of the present disclosure is illustrated. Similar elements in FIG. 35 have similar reference numbers. Also, dashed boxes are optional features for more advanced SoCs. In FIG. 39, interconnect unit(s) (3902) are coupled to: an application processor (3910) comprising a set of one or more cores (202A-N) and shared cache unit(s) (3506); a system agent unit (3510); a bus controller unit(s) (3516); an integrated memory controller unit(s) (3514); a set or one or more coprocessors (3920) that may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit (3930); a direct memory access (DMA) unit (3932); and a display unit (3940) for coupling to one or more external displays. In one embodiment, the coprocessor(s) (3920) include a special purpose processor, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, an embedded processor, or something similar.
[0387] Embodiments disclosed herein (e.g., mechanisms) may be implemented in hardware, software, firmware, or a combination of such implementation approaches. Embodiments of the present disclosure may be implemented as computer programs or program code executed on programmable systems comprising at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.
[0388] Program code, such as the code (3730) exemplified in FIG. 37, may be applied to input instructions to perform the functions described herein and to generate output information. The output information may be applied to one or more output devices in a known manner. For this application purpose, a processing system comprises, for example; any system having a processor such as a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.
[0389] The program code may be implemented in a high-level procedural or object-oriented programming language to communicate with the processing system. The program code may also be implemented in assembly language or machine language if desired. In fact, the mechanisms described herein are not limited in scope to any specific programming language. In any case, the language may be a compiled or interpreted language.
[0390] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium representing various logic within a processor, which, when read by a machine, cause the machine to produce logic for performing the techniques described herein. These representations, known as "IP cores," may be stored on a tangible machine-readable medium and supplied to various customers or manufacturing facilities to be loaded into manufacturing machines that actually produce the logic or processor.
[0391] Such machine-readable storage media may include, but are not limited to, non-transient types of arrays of articles manufactured or formed by a machine or device, comprising storage media, such as hard disks, any other type of disk including floppy disks, optical disks, compact disk read-only memory (CD-ROM), CD-RW (compact disk rewritable), and magneto-optical disks, semiconductor devices, such as read-only memory (ROM), random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), erasable programmable read-only memory (EPROM), flash memory, electrically erasable programmable read-only memory (EEPROM), phase change memory (PCM), magnetic or optical cards, or any other type of medium suitable for storing electronic instructions.
[0392] Accordingly, embodiments of the present disclosure also include a non-transient and tangible machine-readable medium containing design data, such as a Hardware Description Language (HDL), which includes instructions or defines structures, circuits, devices, processors, and / or system features disclosed herein. These embodiments may also be referred to as program products.
[0393] Emulation (including binary conversion, code morphing, etc.)
[0394] In some cases, an instruction translator may be used to translate instructions from a source instruction set to a target instruction set. For example, the instruction translator may translate, morph, emulate, or otherwise convert instructions into one or more other instructions to be processed by the core (e.g., using static binary conversion, dynamic binary conversion including dynamic compilation). The instruction translator may be implemented in software, hardware, firmware, or a combination thereof. The instruction translator may be on-processor, off-processor, or part-on and part-off processor.
[0395] FIG. 40 is a block diagram illustrating the use of a software instruction converter to convert binary instructions in a source instruction set into binary instructions in a target instruction set according to embodiments of the present disclosure. In the illustrated embodiment, the instruction converter is a software instruction converter, but alternatively, the instruction converter may be implemented in software, firmware, hardware, or various combinations thereof. FIG. 40 illustrates that a program in a high-level language (4002) may be compiled using an x86 compiler (4004) to generate x86 binary code (4006) that can be executed natively by a processor (4016) having at least one x86 instruction set core. A processor (4016) having at least one x86 instruction set core represents any processor capable of performing substantially the same function as an Intel® processor having at least one x86 instruction set core by (1) a substantial portion of the instruction set of an Intel® x86 instruction set core or (2) compatiblely executing or otherwise processing object code versions of applications or other software intended to be executed on an Intel® processor having at least one x86 instruction set core in order to achieve substantially the same result as an Intel® processor having at least one x86 instruction set core. An x86 compiler (4004) represents a compiler capable of generating x86 binary code (4006) (e.g., object code) that can be executed on a processor having at least one x86 instruction set core (4016) with or without additional link processing.Similarly, FIG. 40 illustrates that a program in a high-level language (4002) can be compiled using an alternative instruction set compiler (4008) to generate alternative instruction set binary code (4010) that can be executed natively by a processor (4014) that does not have at least one x86 instruction set core (e.g., a processor having cores that execute the MIPS instruction set of MIPS Technologies, Sunnyvale, California, USA and / or the ARM instruction set of ARM Holdings, Sunnyvale, California, USA). The instruction converter (4012) is used to convert the x86 binary code (4006) into code that can be executed natively by a processor (4014) that does not have an x86 instruction set core. This converted code is unlikely to be identical to the alternative instruction set binary code (4010)—because it is difficult to make an instruction converter capable of doing this—; The converted code will achieve general operations and will consist of instructions from an alternative instruction set. Thus, the instruction converter (4012) represents software, firmware, hardware, or a combination thereof that enables an x86 instruction set processor or a processor without a core or other electronic device to execute x86 binary code (4006) through emulation, simulation, or any other process.
Claims
Claim 1 As a device, a first one or more vector registers for storing a first plurality of source matrix data elements; a second one or more vector registers for storing a second plurality of source matrix data elements, wherein each of the first and second plurality of source matrix data elements has a first data element width; a third one or more vector registers for storing a plurality of accumulated matrix data elements, wherein each of the plurality of accumulated data elements has a second data element width that is at least twice the first data element width; a matrix processing circuit capable of operating on a plurality of processing lanes, wherein the matrix processing circuit executes a single matrix instruction to perform a corresponding plurality of multiplications on the plurality of processing lanes; An apparatus comprising: an operand routing circuit that broadcasts a first source matrix data element of a first plurality of source matrix data elements to a plurality of processing lanes of the plurality of processing lanes according to the single matrix instruction; wherein, in each processing lane of the plurality of processing lanes, the matrix processing circuit performs a corresponding multiplication of a different data element of the first source matrix data element and a second plurality of source matrix data elements to generate a corresponding product, and the corresponding product is added to one or more other products of the corresponding accumulating matrix data element and the respective data elements of the first and second plurality of source matrix data elements to generate a corresponding result matrix data element having the width of the second data element. Claim 2 A device according to claim 1, wherein each processing lane has a processing lane width equal to the width of the second data element. Claim 3 A device according to claim 1 or 2, wherein the first and second plurality of source matrix data elements comprise 16-bit floating-point data elements and the accumulation matrix data elements comprise 32-bit floating-point data elements. Claim 4 An apparatus according to claim 1 or 2, further comprising a local buffer storage that stores at least a subset of the first plurality of source matrix data elements and the second plurality of source matrix data elements during the execution of the single matrix instruction. Claim 5 A device according to claim 4, wherein one or more matrix data elements within the subset of the first plurality of source matrix data elements or the second plurality of source matrix data elements are reused in a plurality of operations by the matrix processing circuit. Claim 6 In paragraph 4, the local buffer storage is a device that stores a subset of the first plurality of source matrix data elements and a subset of the second plurality of source matrix data elements. Claim 7 In claim 5, the matrix processing circuit performs matrix operations with a subset of the first plurality of source matrix data elements and a subset of the second plurality of source matrix data elements, stores the corresponding result data elements in the local buffer storage, and combines the corresponding result data elements with the subsequent result data elements of subsequent matrix operations. Claim 8 In paragraph 4, the matrix processing circuit is a device that prevents the reclamation of at least a portion of the local buffer storage. Claim 9 A device according to claim 1 or 2, wherein the matrix processing circuit comprises a two-dimensional grid of processing elements arranged in the plurality of processing lanes. Claim 10 A device according to claim 1 or 2, further comprising an integrated circuit (IC), wherein the IC comprises the first one or more vector registers, the second one or more vector registers, the third one or more vector registers, the matrix processing circuit, and the operand routing circuit.
Citation Information
Patent Citations
A data processing apparatus and method for moving databetween registers and memory
KR1020070001902A
Data processing apparatus and method for performing vector scan operation
KR1020150105209A
Systems and methods for implementing chained tile operations
KR1020190114745A