Multimodal systolic array for matrix multiplication

Abstract

A system and method for matrix multiplication using a systolic array configurable between multiple operating modes. The systolic processor may receive a data type indicator for matrix multiplication. For a first data type, the systolic processor may load right-side data from a right-side matrix register into data processing cells of the systolic array between rows 0 and M-1 and pass each row of left-side data through a corresponding row of the systolic array between rows 0 and M-1. For a second data type, the systolic processor may split each element of the left-side data and right-side data into half of a first element and half of a second element and move half of each element through a corresponding row of the systolic array between rows 0 and 2M-1.

Description

Technical Field

[0001] Cross - reference to Related Applications This application is a continuation of U.S. Application No. 18 / 168,972, filed on February 14, 2023, and claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 477,836, filed on December 30, 2022, the disclosure of which is incorporated herein by reference.

Background Art

[0002] Accelerators for neural networks, such as deep neural networks (DNNs), utilize systolic arrays for high - density computing. A systolic array is an array of processing elements such as processors, microprocessors, or dedicated circuits configured to process some data. Adjacent processing elements of a systolic array may be connected via one or more interconnects, such as wires, or other physical connections on, for example, a printed circuit board.

[0003] Existing systolic arrays utilize one or more matrix multiplication units (MXUs) within a processor to perform matrix multiplication operations. To achieve high performance in matrix multiplication by the processor, a data format called "brain floating point" or abbreviated as "bfloat16" or "bf16" for short is used in the multiply - and - accumulate operations within the MXU. Bf16 is a 16 - bit floating - point format that includes one sign bit, eight exponent bits, and seven mantissa bits. It has been found that adapting the systolic array architecture to support the bf16 data type saves on - chip memory and improves processing speed. However, these adaptations also limit the types of data formats that can be processed using the systolic array.

Summary of the Invention

[0004] This disclosure provides a novel systolic array architecture capable of processing data of multiple data types, that is, it can perform matrix multiplication operations with high efficiency on both bf16 and other data types such as 4-bit integer values, 8-bit integer values, and 8-bit floating-point values.

[0005] In one aspect of the present disclosure, a system for performing matrix multiplication of input data including left-side data and right-side data includes a right-side matrix register having size M × N, a systolic array of data processing cells configurable between a first size M × N and a second size 2M × N, and a systolic processor, the systolic processor receiving a data type indicator indicating the data type of the input data, and (i) in response that the data type indicator indicates a first data type, loading the right-side data from the right-side matrix register into a data processing cell between row 0 and M-1, and each row of the left-side data via the corresponding row of the systolic array between row 0 and M-1 (ii) In response to the data type indicator indicating a second data type, each element of the left-side data and the right-side data is divided into half of the first element and half of the second element, each of which is loaded into the data processing cell between row 0 and M-1 of the right-side matrix register, and each of the second elements of the right-side matrix register is loaded into the data processing cell between row M and 2M-1, and for each row of the left-side data, half of the first element of each row of the left-side data is passed through the corresponding row of the data processing cell between row 0 and M-1, and half of the second element of each row of the left-side data is passed through the corresponding row of the data processing cell between row M and 2M-1.

[0006] According to aspects of this disclosure, the first data type includes at least one of an 8-bit integer, an 8-bit floating-point number, or a 16-bit floating-point number, and the second data type may include a 4-bit integer.

[0007] According to aspects of the present disclosure, a systolic processor may be configured to pass a vector of left-side data elements having shape 1*M in each matrix multiplication cycle in response to a data type indicator indicating a 16-bit floating-point or 8-bit floating-point number; to pass a vector of left-side data elements having shape 2*M in each matrix multiplication cycle in response to a data type indicator indicating an 8-bit integer; and to pass a vector of left-side data elements having shape 2*2M in each matrix multiplication cycle in response to a data type indicator indicating a 4-bit integer.

[0008] According to aspects of the present disclosure, the system further includes one or more 16-bit floating-point multiply-accumulate chains, two additional 8-bit integer multiply-accumulate chains for each 16-bit floating-point multiply-accumulate chain, and two additional 4-bit integer multiply-accumulate chains for each 16-bit floating-point multiply-accumulate chain. The systolic processor is configured to process 8-bit floating-point and 16-bit floating-point data using the 16-bit floating-point multiply-accumulate chains, to process 8-bit integer data using two 8-bit integer multiply-accumulate chains, and to process 4-bit integer data using two 8-bit integer multiply-accumulate chains and two 4-bit integer multiply-accumulate chains.

[0009] According to aspects of the present disclosure, the systolic processor is configured to generate 2 × M 24-bit results per cycle for 8-bit and 4-bit integer inputs, and to generate 1 × M 32-bit results per cycle for 8-bit and 16-bit floating-point inputs.

[0010] According to aspects of the present disclosure, the system further includes a holding register having size M × N and configured to provide right-side data to a right-side matrix register. The holding register is configured to include at least one data type not supported by the systolic array. The systolic processor is configured to convert the right-side data of the unsupported data type to right-side data of a supported data type once the right-side data is provided from the holding register to the right-side matrix register.

[0011] According to aspects of this disclosure, the unsupported data type is an 8-bit floating-point data type, and the systolic processor is configured to convert the right-side data of the unsupported data type contained in the holding register to a 16-bit floating-point data type when the right-side data is provided from the holding register to the right-side matrix register.

[0012] According to aspects of the present disclosure, a systolic array data processing cell includes a floating-point dot product accumulator configured to process left-side and right-side data having a floating-point data type, and a plurality of integer dot product accumulators configured to process left-side and right-side data having an integer data type.

[0013] According to aspects of the present disclosure, the systolic processor is configured to, in response to a data type indicator indicating a floating-point data type, pass data from one vector from the left-side data to a floating-point dot product accumulator with each matrix multiplication cycle, and in response to a data type indicator indicating an integer data type, pass data from two vectors from the left-side data to a plurality of integer dot product accumulators with each matrix multiplication cycle. Each integer dot product accumulator is configured to receive data from each vector of the left-side data.

[0014] According to aspects of the present disclosure, each integer dot product accumulator further comprises a separate first data path and a second data path, each data path comprising a respective partial product generating layer, a respective carry-storage adder tree layer, and a respective decrement tree layer.

[0015] According to aspects of the present disclosure, the systolic processor is configured to, in response to a data type indicator indicating an 8-bit integer data type, pass data from the left-side data only to the first data path of each integer dot product accumulator in each matrix multiplication cycle, and in response to a data type indicator indicating a 4-bit integer data type, pass data from the left-side data to both the first and second data paths of each integer dot product accumulator in each matrix multiplication cycle.

[0016] According to aspects of this disclosure, for each element of the left-hand data, the systolic processor is configured to pass half of the first element to a corresponding first data path of an integer dot product accumulator, and the second element to a corresponding second data path of an integer dot product accumulator.

[0017] Other aspects of this disclosure relate to accelerator hardware units, including systems such as those described in any of the embodiments herein. The accelerator hardware unit may be either a graphics processing unit or a tensor processing unit. In some examples, the accelerator hardware unit includes a plurality of matrix multiplication units, at least one of which includes a system as described in any of the embodiments herein.

[0018] However, a further aspect of the present disclosure is a method for performing matrix multiplication in a systric array of data processing cells configurable between a first size of M × N and a second size of 2M × N, comprising receiving a data type indicator by one or more processors indicating the data type of input data for matrix multiplication, the input data comprising left-side data and right-side data, the method comprising (i) in response that the data type of the data type indicator indicates a first data type, loading the right-side data from a right-side matrix register having size M × N into a data processing cell of the systric array between row 0 and row M-1 by one or more processors; for each row of left-side data, passing each row of left-side data through the corresponding row of the data processing cell of the systric array between row 0 and row M-1 to derive the matrix multiplication result of the left-side data and the right-side data; and (ii) in response that the data type of the data type indicator indicates a second data type The present invention relates to a method for deriving a matrix multiplication result between left-side data and right-side data, comprising: one or more processors dividing each element of left-side data and right-side data into half of the first element and half of the second element; one or more processors loading half of each first element from the right-side matrix register into a data processing cell of a systric array between row 0 and row M-1; one or more processors loading half of each second element from the right-side matrix register into a data processing cell of a systric array between row M and row 2M-1; for each row of left-side data, one or more processors passing half of the first element of each row of left-side data through the corresponding row of a data processing cell of a systric array between row 0 and row M-1; and one or more processors passing half of the second element of each row of left-side data through the corresponding row of a data processing cell of a systric array between row M and row 2M-1.

[0019] According to aspects of this disclosure, the first data type may include at least one of a 16-bit floating-point number, an 8-bit floating-point number, or an 8-bit integer, and the second data type may include a 4-bit integer. In some examples, passing left-side data in response to the data type indicator indicating the first data type may include passing one or more vectors of elements of the left-side data to rows 0 to M-1 of the systolic array only in each matrix multiplication cycle, and passing left-side data in response to the data type indicator indicating the second data type may include passing one or more vectors of elements of the left-side data to all rows 0 to 2M-1 of the systolic array in each matrix multiplication cycle.

[0020] According to aspects of the present disclosure, passing left-side data in response to a data type indicator indicating a second data type may include passing two vectors of elements of the left-side data to a plurality of integer dot product accumulators contained in each cell of a systolic array for each matrix multiplication cycle. Each cell may contain two integer dot product accumulators, and the corresponding row of each pair may correspond to a separate data path of the pair in one of the corresponding integer dot product accumulators.

[0021] According to aspects of the present disclosure, the data type indicator may further distinguish between integer data types and floating-point data types. For example, passing left-side data in response to the data type indicator indicating a floating-point data type may include passing one vector of elements of the left-side data to the systolic array in each matrix multiplication cycle, and passing left-side data in response to the data type indicator indicating an integer data type may include passing two vectors of elements of the left-side data to the systolic array in each matrix multiplication cycle.

[0022] According to an aspect of the present disclosure, the method may further include storing right-side data in a holding register having a size M×N, where the right-side data is a data type not supported by the systolic array. The method further includes storing the right-side data from the holding register into a right-side matrix register. Loading includes converting, by one or more processors, the right-side data into a data type supported by the systolic array. The data type not supported by the systolic array is 8-bit floating point, and the data type supported by the systolic array is 16-bit floating point.

[0023] According to an aspect of the present disclosure, the left-side and right-side data may be received as a 128×128 matrix, where M = 128 and N = 128. The method may further include generating a 1×128 32-bit floating point result for each cycle of the systolic array for a data input including 8-bit or 16-bit floating point operands, and generating a 2×128 24-bit result for each cycle of the systolic array for a data input including 4-bit or 8-bit integer operands.

Brief Description of the Drawings

[0024] [Figure 1] A block diagram of a systolic array according to an aspect of the present disclosure. [Figure 2A] A block diagram of an exemplary cell included in the systolic array of FIG. 1. [Figure 2B] A block diagram of an exemplary cell included in the systolic array of FIG. 1. [Figure 3] A block diagram of an exemplary accumulator included in the cells of FIGS. 2A and 2B. [Figure 4] A chart showing an exemplary data flow within the systolic arrays of FIGS. 1-3. [Figure 5] A chart showing an exemplary data flow within the systolic arrays of FIGS. 1-3. [Figure 6]A diagram showing an exemplary data flow within the systolic array of FIGS. 1-3. [Figure 7] A diagram showing an exemplary data flow within the systolic array of FIGS. 1-3. [Figure 8] A - C are diagrams showing the shapes of exemplary matrix multiplication and addition operations executed by a systolic array according to aspects of the present disclosure. [Figure 9] A block diagram of an exemplary electronic device according to aspects of the present disclosure. [Figure 10] A flowchart of an exemplary routine for loading and passing data to a systolic array according to aspects of the present disclosure. [Figure 11] A block diagram regarding an exemplary arrangement of a systolic processor according to aspects of the present disclosure. **Modes for Carrying Out the Invention**

[0025] Overview According to an embodiment, a system and method for matrix multiplication uses a systolic array configurable among a plurality of operating modes. The systolic processor is configured to receive a data type indicator for matrix multiplication. In the case of a first data type, the systolic processor loads right - hand side data from a right - hand matrix register into data processing cells of the systolic array between row 0 and row M - 1, and is configured to pass each row of the left - hand side data through corresponding rows of the systolic array between row 0 and M - 1. In the case of a second data type, the systolic processor divides each element of the left - hand side data and the right - hand side data into halves of a first element and a second element respectively, and is configured to move the halves of each element through corresponding rows of the systolic array between row 0 and 2M - 1.

[0026] A systolic array is configured to perform matrix multiplication operations on operands of two or more different data type formats, such as 4-bit integer operands, 8-bit integer operands, 8-bit floating-point operands, or 16-bit floating-point operands. A systolic array uses at least some of the same input and output buses for multiple formats. Furthermore, the cells of a systolic array include processing elements to support 16-bit floating-point × 16-bit floating-point product, 8-bit integer × 8-bit integer product, and 4-bit integer × 4-bit integer product, respectively. For example, each cell of a systolic array may include a single 16-bit floating-point dot product accumulator (DPA) that supports 16-bit floating-point numbers through 16-bit floating-point matrix multiplication, and a pair of 8-bit integer DPAs that support both 8-bit integer × 8-bit integer and 4-bit integer × 4-bit integer matrix multiplication operations.

[0027] A systolic processor, which may include several processing elements for operating a systolic array, can receive data type indications for both the right-side data loaded into the array and the left-side data flowing into the array, and based on the received data type indications, can determine how to input and process the data into the systolic array. The indications may be received in the form of flags, indicating whether the data type is integer or floating-point, the size of the data type such as 4-bit, 8-bit, or 16-bit, and, in some situations, other characteristics of the data type that may be necessary to perform appropriate matrix multiplication operations.

[0028] For a 16-bit floating-point operand stored in the form of an M×N element matrix, the right-side data can be loaded into a systolic array between rows 0 and M-1 of the array, and each element of the left-side data can be passed through each row of the systolic array between rows 0 and M-1 of the array. In such a case, a 16-bit floating-bit input can be effectively interpreted as 1×M elements per matrix multiplication cycle of the systolic array, and the output of the array can result in 1×M 32-bit floating-point numbers (also called "f32") per cycle.

[0029] For 8-bit floating-point operands stored in the form of an M×N matrix, the 8-bit operands can be converted to 16-bit floating-point operands so that they flow into and out of the systolic array in the same way as 16-bit floating-point operands.

[0030] Similar to 16-bit floating-point elements, for 8-bit integer operands stored in the form of an M×N matrix, each element of the right-side data can be loaded into a systolic array between rows 0 and M-1 of the array, and each element of the left-side data can be passed through each row of the systolic array between rows 0 and M-1 of the array. However, unlike 16-bit floating-point elements, each cell of the systolic array has twice the throughput of an 8-bit integer operation, so each cell of the systolic array can receive and process two rows of 8-bit integer elements in parallel per cycle. In such a case, an 8-bit integer input can be effectively interpreted as 2×M elements per matrix multiplication cycle of the systolic array, and the output of the array can result in 24 bits, 2×M per cycle.

[0031] Finally, in the case of a 4-bit integer operand stored in the form of an M×N element matrix, each element of the stored matrix may contain two distinct 4-bit operands instead of a single 8-bit operand. Consequently, when the elements are moved to the systolic array, each element may be interpreted as half of two elements, thereby each half of an element being a distinct 4-bit integer operand. Since the stored M×N element matrix is ​​interpreted as having twice as many elements per row, half of each element of the right-side data may be loaded into each systolic row between row 0 and 2M-1 of the array, and half of each element of the left-side data may be passed through each row of the systolic array between row 0 and 2M-1 of the array. In such a case, a 4-bit integer input may effectively be interpreted as 2×2M elements per matrix multiplication cycle of the systolic array. Furthermore, a 4-bit integer input may effectively be interpreted as 2×2M elements per matrix multiplication cycle, resulting in an array output of 2×M 24 bits per cycle.

[0032] The disclosed systolic array architecture offers the advantage of being able to handle multiple data types, including both integer and floating-point values ​​of different sizes, while maintaining good throughput. Instead of providing entirely different hardware for each data type, the same input and output buses can be used for various data types to feed the various accumulators included in the systolic array. Ultimately, these advantages are beneficial in maintaining a highly efficient processor that can handle many different data types without sacrificing excessive space for additional hardware to support those various data types.

[0033] Exemplary System Figure 1 is a diagram of an exemplary systolic array 100 according to an aspect of the present disclosure. The systolic array 100 is shown having a plurality of cells 110 arranged in an [I,J] array having I rows and J columns.

[0034] Each cell 110 may be loaded with right-side data from the right-side matrix register 120. The right-side matrix register 120 may be formed as an M × N matrix having M rows and N columns of data elements. Generally, each data element corresponds to a distinct operand of a matrix multiplication operation, but in at least some circumstances of this disclosure, each data element may contain multiple operands.

[0035] The first column [0] of cell 110 can also receive left-side data from the left-side vector register 130. The left-side matrix register 130 can also be formed as an M × N matrix having M rows and N columns of data elements. Similar to the right-side data, each data element typically corresponds to a separate operand of a matrix multiplication operation, although in at least some circumstances of this disclosure, each data element may contain multiple operands.

[0036] In some examples, the systolic array may be stationary, meaning that the entire right-side matrix register 120 is preloaded and remains in the array during matrix multiplication. In other examples, the vector of right-side data may be loaded cycle by cycle, and the systolic array will be transient. The choice between a stationary or transient array may be influenced by timing considerations within the systolic array, such as the direction of control flow and pipelining.

[0037] Each cell 110 of the systolic array 100 may be responsible for receiving a portion of the right-side data, receiving a portion of the left-side data, receiving the output from the previous cell in the same row, calculating the product of the received portions of the right-side and left-side data, adding the calculated product to the output from the previous cell, and passing the sum to the next cell in the same row. For example, in the case of a given cell [i,j] of the systolic array, the right-side data loaded into the received cell may correspond to column [n] of the right-side matrix register 120, the left-side data passed through the cell may correspond to row [m] of the left-side vector register 130, the output may be received from cell [i-1,j], the product of the received portions of [m] and [n] may be calculated and added to the output from cell [i-1,j], and the sum may be transferred to the next cell [i+1,j]. Finally, the calculations performed by each cell 110 may result in a matrix multiplication result 140.

[0038] Figures 2A and 2B are block diagrams of exemplary individual cells 200, such as cell 110 of the systolic array 100 in Figure 1. Cell 200 includes a processing element capable of receiving right-side data 210 and left-side data 220, as well as receiving values ​​calculated from previous cells along the row of the systolic array. In the example in Figures 2A and 2B, cell 200 is shown to include one dot product accumulator (DPA) for floating-point values ​​(Float DPA230) and two dot product accumulators (DPA) for integer values ​​(Int DPA_0 240 and Int DPA_1 250). Float DPA230 is used to process the input floating-point operands, and Int DPA240, 250 is used to process the input integer operands.

[0039] The example in Figure 2A illustrates the data flow for an input floating-point operand. In the example in Figure 2A, the floating-point operand can be a 16-bit operand or an 8-bit operand. In other examples, the floating-point operand can be of a different size. Cell 200 receives right-side data 212 from column [n] of the right-side matrix 210. Furthermore, in each cycle of the systolic array, cell 200 receives left-side data 222 from the previous cell of the systolic array, corresponding to row [m] of the vector in the left-side matrix 220. The cell then receives result 252 of the previous cell [m,n-1], which is the result of a calculation performed on row [m] of the left-side data and column [n-1] of the right-side data, added along row [m] to any previous result. The calculation is performed in the floating-point DPA 230, where the dot product of the left-side data 222 and the right-side data 212 is calculated and then added to result 252 of the previous cell. In the next cycle, the left-side data 222 is then passed along the row to the next cell [m,n+1], along with the calculation result 262 of the calculation.

[0040] The example in Figure 2B illustrates the data flow for an input integer operand. In the example in Figure 2B, the integer operand can be an 8-bit operand or a 4-bit operand. In other examples, the integer operand can be of different sizes. Cell 200 receives right-side data 214, 216 from column [n] of the right-side matrix 210. Furthermore, in each cycle of the systolic array, cell 200 receives two rows of left-side data 224, 226 corresponding to rows [m] and [m+1] of two vectors in the left-side matrix 220 from the previous cell of the systolic array. The cell further receives results 254, 256 from the previous cell [m, n-1]. Result 254 is the result of a calculation performed on row [m] of the left-side data and column [n-1] of the right-side data, added along row [m] to any previous result. Result 256 is the result of calculations performed in row [m+1] of the left-side data and column [n-1] of the right-side data, added to any previous result along row [m+1]. The calculations for each row [m] and [m+1] are performed with their respective integers DPA 240, 250, and the dot product of each left-side data 224, 226 and each right-side data 214, 216 is calculated and then added to the results 254, 256 of the respective previous cell [m,n-1] or [m+1,n-1]. In the next cycle, each part of the left-side data 224, 226 is then passed along that row to its next cell [m,n+1] or [m+1,n+1], along with each calculation result 264, 266 of the calculations for that row.

[0041] Figure 2B shows that the 8-bit integer arithmetic in cell 200 has twice the throughput compared to 16-bit floating-point arithmetic. This double throughput is efficiently achieved because the cost of passing 16 bits of data for a 16-bit floating-point operand is comparable to the cost of passing 16 bits of data for two 8-bit integer operands. Furthermore, while two operands may require twice the dot multiplication operation compared to one operand, cell 200 has two accumulators for 8-bit integer operands, resulting in no loss of efficiency during the computation phase.

[0042] The cells in Figures 2A and 2B can also handle 4-bit integers with high efficiency. This is achieved by including two separate data paths for the inputs in the integer DPA. Figure 3 is a block diagram of an exemplary integer DPA 300, which includes a first data path 310 used for processing both 4-bit and 8-bit integer operands, and a second data path 320 used only for processing 4-bit integer operands. Each data path 310, 320 includes a respective partial product generating layer 312, 322, a respective carry-save adder tree layer 314, 324, and a respective decreasing tree layer including one or more decreasing trees 316, 326. The components used in these layers 312, 314, 316, 322, 324, 326 can be any conventional components known in the art for performing dot product accumulation of integer operands such as 8-bit integers or 4-bit integers.

[0043] The DPA300 may further include a right-hand matrix register and a circuit shown in Figure 3 as a "type-dependent split and expand" operation 330 to control whether the input data 340 received from the previous cell is maintained as a single input or split into two inputs. For example, if the data type is a first data type such as an 8-bit integer, the data may be maintained as a single input, and if the data type is a second data type such as a 4-bit integer, it may be split into two inputs.

[0044] The DPA 300 may further include other decreasing trees 350 for combining the outputs of each of the two data paths. In scenarios where the input data 340 is not split, calculations in the further decreasing trees 350 may be skipped. The DPA may also further include a carry-propagating adder 360 for adding the result from the previous cell to the result of the current cell. The DPA result 370 may be output from the carry-propagating adder 360 and provided to the next cell along the corresponding row. In some examples, the further decreasing trees 350 and carry-propagating adder 360 may be shared with the cell's floating-point DPA instead of providing separate decreasing trees and carry-propagating adders for both the floating-point and integer paths within the DPA. Alternatively, separate decreasing trees and carry-propagating adders may be provided.

[0045] Figures 4 to 7 are block diagrams illustrating exemplary data flows for different data types that may be received by the systolic array of the present disclosure. The example in Figure 4 shows the data flow for a 16-bit floating-point operand, the example in Figure 5 shows the data flow for an 8-bit floating-point operand, the example in Figure 6 shows the data flow for an 8-bit integer operand, and the example in Figure 7 shows the data flow for a 4-bit integer operand.

[0046] In the example in Figure 4, left-side data 410 from a matrix having shape M×N is provided to the systolic array 420. The left-side data 410 includes a vector 412 of M 16-bit elements having shape 1×M, and a data type indicator 414 indicating the data type of the element vector. In the example in Figure 4, the elements are shown to be of type "bf16", which is a 16-bit floating-point value. In response, the systolic processor 422 of the systolic array 420 controls the flow of M elements to rows 0 to M-1 of the systolic array, so that each element occupies its own row. Other vectors 412 can be passed from the left-side data 410 to the systolic array 420 in each cycle of the matrix multiplication operation.

[0047] In the example in Figure 5, left-side data 510 from a matrix having shape M×N is provided to the systolic array 520. The left-side data 510 includes a vector 512 of M 8-bit elements having shape 1×M, and a data type indicator 514 indicating that the data type of the element vector is "fp8", which is a type of 8-bit floating-point value. In response, the systolic processor 522 of the systolic array 520 can control the conversion of the 8-bit elements to "bf16" 16-bit floating-point values, and then control the flow of the M converted elements to rows 0 to M-1 of the systolic array 520, so that each element occupies its own row. In each cycle of the matrix multiplication operation, other vectors 512 may be converted and passed from the left-side data 510 to the systolic array 520.

[0048] In the example in Figure 6, left-side data 610 from a matrix having shape M×N is provided to the systolic array 620. The left-side data 610 includes two vectors 612, 614, each having M 8-bit elements with shape 1×M, and a data type indicator 616 indicating that the data type of the element vectors is "int8", which is a type of 8-bit integer value. In response, the systolic processor 622 of the systolic array 620 can control the flow of elements from each vector to each row between rows 0 to M-1 of the systolic array 620, so that each element occupies each row and each row receives two elements. In each cycle of the matrix multiplication operation, the other two vectors 612, 614 may be transformed and passed from the left-side data 610 to the systolic array 620.

[0049] As illustrated in the example in Figure 6, the ability of a systolic array to receive and process two vectors of int8 data within a single cycle allows matrix multiplication operations on an entire matrix of int8 data to be completed twice as fast compared to a systolic array that receives only one vector at a time. Therefore, the efficiency of the systolic array can be maintained because the speed at which elements can be processed in the array can correspond to the speed at which elements can be streamed to the array.

[0050] Finally, in the example of Figure 7, left-side data 710 from a matrix having shape M×N is provided to the systolic array 720. The left-side data 710 contains two vectors 712, 714, each having M 8-bit elements, with each element containing two operands, so that each vector of operands gives a shape of 1×2M. The left-side data 710 also contains a data type indicator 716 indicating that the data type of the element vector is "int4", which is a kind of 4-bit integer value. In response, the systolic processor 722 of the systolic array 720 can control the division of each element from an 8-bit element into two 4-bit elements, so that the first half of the 8-bit element with bits 0-3 constitutes the first half of the element corresponding to the first 4-bit operand, and the second half of the 8-bit element with bits 4-7 constitutes the second half of the element corresponding to the second 4-bit operand. The systolic processor 722 then controls the flow of the half of the elements to each row of the systolic array. For half of the first element directed to row [m] of the array, the corresponding half of the second element may be directed to the corresponding row [M+m], which may be physically located in the same place as row [m] in the same DPA of the systolic array cell. In this way, for every 1 × M vectors 712, 714 of the elements of the left-side data 710, the operands are provided to 2M rows of the systolic array. As with the 8-bit integer operands in the example of Figure 6, for the 4-bit integer operands, 2 vectors may flow into the systolic array for each cycle of the matrix multiplication operation.

[0051] As can be seen from the example in Figure 7, the ability to receive two vectors of int4 data within a single cycle, and furthermore, the ability to process two rows of int4 data within a single cell of a systolic array, makes it possible to complete matrix multiplication operations on an entire matrix of int4 data four times faster compared to a systolic array that receives only one vector at a time and processes only one element per cell. Therefore, the efficiency of the systolic array can be maintained because the speed for processing elements in the array can correspond to the speed at which elements can be streamed into the array.

[0052] Although not shown in Figures 4-7, the process of loading right-side data into a systolic array can be comparable to the process of streaming left-side data. For example, in the case of 16-bit floating-point values, 8-bit floating-point values, and 8-bit integer values, the right-side data can be loaded into rows 0-M-1 of the systolic array. Furthermore, in the case of 8-bit floating-point values, the right-side data can also be converted to the 16-bit floating-point value "bf16" format. In a further example, in the case of 4-bit integer values, each element of the right-side data can be split into two halves and loaded into rows 0-2M-1 of the systolic array, so that half of a pair of elements is loaded into rows 0 and M, or row 1 and M+1, or row 2 and M+2, etc., which are physically located in the same row.

[0053] Figures 8A to 8C are diagrams that provide further visualization of matrix multiplication operations performed in each given cycle of the exemplary systolic arrays in Figures 4 to 7. The operations shown in Figure 8A correspond to 16-bit floating-point operations, which are performed by the exemplary systolic arrays in Figures 4 and 5. The operations shown in Figure 8B correspond to 8-bit integer operations, which are performed by the exemplary systolic array in Figure 6. The operations shown in Figure 8C correspond to 4-bit integer operations, which are performed by the exemplary systolic array in Figure 7.

[0054] In the example in Figure 8A, a 1×M vector of bf16 elements from left-side data 812 is multiplied by an M×N matrix of bf16 elements from right-side data 814. The bf16 elements can start as 16-bit floating-point values, or as 8-bit floating-point values ​​that can be converted to 16-bit floating-point values ​​using data type conversion instructions such as roundTiesToEven. Converting 8-bit values ​​to 16-bit values ​​can be done by further utilizing a hold register of size M×N to store the right-side data and load it into the right-side matrix register. In such an example, the 8-bit floating-point values ​​may be a data type that is not supported for the right-side matrix register, but may be a data type that is supported for the hold register. Therefore, moving the 8-bit floating-point values ​​from the vector register to the hold register can make it easier to convert the 8-bit values ​​to 16-bit values ​​when the data is moved from the hold register to the right-side matrix register.

[0055] In either case, where the data type indicator shows an 8-bit or 16-bit floating-point value, the result of the matrix multiplication operation is a 1×M vector of 32-bit floating-point operands, which can be added to a 1×M vector derived from calculations in the previous cell of the systolic array 816, so that the array itself is a 1×M vector of 32-bit floating-point operands, reaching the final output result 818 of the cycle. The output may be in 32-bit floating-point format to facilitate cumulative operations.

[0056] In the example in Figure 8B, two 1×M vectors of int8 elements from left-side data 822 are multiplied by an M×N matrix of int8 elements from right-side data 824. The multiplication operation of the two vectors can occur in parallel in each corresponding cell using the two integer DPAs of the corresponding cells. The result of the parallel matrix multiplication is a 2×M vector of 24-bit integer operands, which can be added to the 2×M vector derived from the calculation in the previous cell of the systolic array 826, so that the array itself is a 2×M vector of 24-bit integer operands, and the final output result 828 of the cycle.

[0057] In the example in Figure 8C, two 1×M vectors of 8-bit integer elements from the left-side data are split into two vectors of 1×2M 4-bit integer operands 832, and an M×N matrix of 8-bit integer elements from the right-side data is split into a 2M×N matrix of 4-bit integer operands 834, and loaded into the systolic array. The multiplication operation of the two vectors can occur in parallel in each corresponding cell using the two integer DPAs of the corresponding cells. Furthermore, in each integer DPA, rows located in the same physical location can receive the respective parts of the left-side data. The result of the parallel matrix multiplication operation is a 2×2M vector of 24-bit integer operands, which can be added to the 2×2M vector derived from the calculation in the previous cell of the systolic array 836, and the array arrives at the final output result 838 of the cycle, which itself is a 2×2M vector of 24-bit integer operands.

[0058] Figure 9 shows a block diagram of an exemplary electronic device 900 for implementing a systolic array according to one of the exemplary embodiments of the present disclosure. The electronic device 900 may include one or more processors 910, such as one or more CPUs, system memory 920, a bus 930, network interface(s) 940, and other components (not shown), such as storage(s), output device interfaces(s), and input device interfaces(s). The bus 930 may be used for communication between the processors 910, system memory 920, network interface(s) 940, and other components. Any or all components of the electronic device 900 may be used in conjunction with the subject matter of the present disclosure.

[0059] Depending on the desired configuration, the processor 910 may be any type including, but not limited to, one or more central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs), or any combination thereof. The processor 910 may include a systolic array. The processor 910 may include another level of caching, such as a level 1 cache 911 and a level 2 cache 912, a processor core 913, and registers 914. The processor core 913 may include one or more arithmetic logic units (ALUs), one or more floating-point units (FPUs), one or more DSP cores, or any combination thereof. A memory controller 915 may also be used with the processor 910, or in some embodiments, the memory controller 915 may be an internal part of the processor 910.

[0060] Depending on the desired configuration, the physical memory 920 may be of any type, including but not limited to volatile memory such as RAM, non-volatile memory such as ROM, flash memory, or any combination thereof. The physical memory 920 may include program data 924 which may include an operating system 921, one or more applications 922, and service data 925. The non-temporary computer-readable medium program data 924 may include storing instructions that, when executed by one or more processing devices, carry out a process for calculating the result of a multiply-accumulate operation 923. In some examples, one or more applications 922 may be arranged to operate with the program data 924 and service data 925 in the operating system 921.

[0061] The electronic device 900 may have additional features or functions, as well as additional interfaces to facilitate communication between the basic configuration 901 and any necessary devices and interfaces.

[0062] Physical memory 920 may be an example of a computer storage medium. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, or may be accessed by electronic devices 900. Any such computer storage medium may be part of device 900.

[0063] Network interfaces 940 may connect electronic device 900 to a network (not shown) and / or other electronic devices (not shown). In this way, electronic device 900 may be part of a network of the electronic device, such as a local area network ("LAN"), a wide area network ("WAN"), an intranet, or any other network, such as the Internet. In some examples, electronic device 900 may include a network connection interface for forming a network connection to a network, and a local communication connection interface for forming a tethering connection with other devices. The connection may be wired or wireless. Electronic device 900 may bridge the network connection and the tethering connection to connect other devices to the network via network interfaces 940.

[0064] The systolic array may include multiple MAC units 950 to perform the multiply-accumulate operations required for matrix multiplication. The MAC units 950 and the systolic array on which the MAC units operate may be used in an accelerator that can be used in a DNN embodiment.

[0065] The electronic device 900 may be implemented as part of a small form factor portable (or mobile) electronic device, such as a speaker, headphones, earphones, mobile phone, smartphone, smartwatch, personal data assistant (PDA), personal media player device, tablet computer (tablet), wireless webwatch device, personal headset device, wearable device, application-specific device, or as part of a hybrid device that includes any of the above functions. The electronic device 900 may also be implemented as a personal computer, including both laptop and non-laptop computer configurations. Furthermore, the electronic device 900 may be implemented as a server, accelerator, or large-scale system.

[0066] Exemplary Method Figure 10 is a flowchart illustrating an exemplary routine 1000 for controlling the flow of data through the systolic array of this disclosure. Routine 1000 may be executed by one or more systolic processors included in the system, such as the systolic processors shown in Figures 4 to 7.

[0067] The operation can begin in block 1010, in which the systolic processor receives a data type indicator that indicates the data type of the input data. The input data may include the left-side and right-side data of a matrix multiplication in a systolic array. In some examples, the data type indicator may be a flag. In some examples, the flag may include one or more bits appended to the bits of the element.

[0068] In block 1020, the systolic processor can determine the data type of input data based on a data type indicator. For example, the data type could be any one of the following: 16-bit floating-point, 8-bit floating-point, 8-bit integer, 4-bit integer, etc. In the example in Figure 10, a classification between two data types is shown, where the first data type is processed according to the first set of operations, and the second data type is processed according to the second set of operations.

[0069] If the data type corresponds to a first data type that is either a 16-bit floating-point number, an 8-bit floating-point number, or an 8-bit integer, the operation may continue to block 1030, where the right-side data is loaded between row 0 and M-1 of the systolic array, and then in block 1040, each row of the left-side data is passed through the corresponding row between row 0 and M-1 of the systolic array. The left-side data may be passed as one vector per cycle.

[0070] Alternatively, if the data type corresponds to a second data type corresponding to a 4-bit integer format, the operation may continue to block 1050, where each element of the left-side data and the right-side data is divided into half of the first element and half of the second element, and then to block 1060 and 1 0 Following 70, the right-side data is loaded between row 0 and 2M-1 of the systolic array, with half of each first element placed between row 0 and M-1, and half of each second element placed in the corresponding row between row M and 2M-1. The left-side data is passed through rows 0 to 2M-1 of the systolic array, with half of each first element passed between row 0 and M-1, and half of each second element passed in the corresponding row between row M and 2M-1.

[0071] In further examples, a systolic processor may further distinguish additional data types and further control different data flows for different data types. For example, a data type indicator may distinguish between 16-bit floating-point values ​​and 8-bit floating-point values, thereby allowing an 8-bit floating-point indicator to signal the systolic processor to convert the input left-side and right-side operands from 8-bit to 16-bit floating-point format. Figure 11 shows an example of a configuration for a systolic processor to convert an 8-bit floating-point value to a 16-bit floating-point value. The arrangement 1100 in Figure 11 shows four registers 1110, 1120, 1130, and 1140 used to prepare the right-side and left-side operands that are loaded into and passed through a systolic array 1150, respectively. The left-side data is first stored in the first matrix staging register (MSR_A) 1110, which can support operands in 8-bit floating-point format. Next, the left-side data is moved from the first matrix staging register 1110 to the left matrix register 1120 for loading into the systolic array. Since the systolic array does not support matrix multiplication of 8-bit floating-point data, the 8-bit floating-point operands may be converted from 8-bit floating-point format to 16-bit floating-point format during the transfer from the first matrix staging register 1110 to the left matrix register 1120. Similar operations may be performed on the right-side data using the second matrix staging register 1130 and the right matrix register 1140, respectively. Furthermore, while Figure 11 shows that the first matrix staging register 1110 is connected to the left matrix register 1120 and the second matrix staging register 1130 is connected to the right matrix register 1140, it should be understood that each of the matrix staging registers 1110 and 1130 can be connected interchangeably to either the left matrix register 1120 or the right matrix register 1140, thereby allowing each matrix staging register 1110 and 1130 to be used to store either left-side data or right-side data.In addition to the data format conversion benefits provided by the four registers, it should be noted that the four registers can provide additional functionality to the systolic processor. For example, data transferred to the matrix staging register can be transposed at the time of transfer, meaning that both the left-side and right-side data can be set up for matrix multiplication operations in transposed form within the systolic processor, without the need to rely on a separate transposition unit located away from the systolic processor.

[0072] Furthermore, or alternatively, the data type indicator can distinguish between floating-point and integer values, thereby directing incoming floating-point and integer values ​​to separate DPAs for processing within the cells of the systolic array. In some examples, moving integer values ​​to DPAs may involve each cell receiving data from two vectors from left-side data during each cycle of a matrix multiplication operation, and then moving the received data to different integer DPAs within the cell for initial parallel processing and subsequent integrating processing.

[0073] In one exemplary embodiment of the present disclosure, a systolic array may be used as a matrix multiplication unit (MXU) within a tensor processing unit (TPU). For example, the TPU may include one or more core processors, each of which may be connected to one or more MXUs. The MXU may be a systolic array processor with inner product steps. The MXU may be further configurable between shapes of 128 × 128 and 256 × 128 depending on the data type received. The MXU may support generating 2 × 128 24-bit results per cycle for both 8-bit and 4-bit integer operands, and providing 1 × 128 32-bit floating-point results per cycle for both 16-bit and 8-bit floating-point operands.

[0074] The exemplary MXUs, and more generally systolic arrays, of this disclosure provide increased versatility and efficiency of matrix multiplication operations by supporting several types of input data using common input and output lines. This helps reduce the overall footprint of TPUs and other chips incorporating MXUs and systolic arrays without sacrificing processing efficiency.

[0075] While the technology of this specification has been described with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the technology. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other configurations can be devised, without departing from the spirit and scope of the technology as defined by the appended claims.

[0076] Most of the alternatives described herein are not mutually exclusive, but can be implemented in various combinations to achieve their own advantages. Since these and other variations and combinations of the above features can be used without departing from the subject matter defined by the claims, the above description of embodiments should be interpreted as illustrative rather than as an limitation of the subject matter defined by the claims. For example, the prior actions do not need to be performed in the exact order described above. Rather, the various steps can be performed in different orders, such as in reverse order or simultaneously. Unless otherwise specified, steps can be omitted. Furthermore, the provision of examples described herein, as well as sections expressed as "such as," "including," etc., should not be interpreted as limiting the subject matter of the claims to specific examples. Rather, the examples are intended to illustrate only one of many possible embodiments. Furthermore, the same reference numeral in different drawings may identify identical or similar elements.

Claims

1. A system for performing matrix multiplication of input data including left-side data and right-side data, A right-hand matrix register having size M × N, A systolic array of data processing cells that can be configured between a first size M x N and a second size 2M x N, A systolic processor is included, and the systolic processor is A data type indicator indicating the data type of the input data is received, (i) In response to the data type indicator indicating a first data type, The right-side data from the right-side matrix register is loaded into the data processing cell between row 0 and M-1. Each row of the left-side data is passed through the corresponding row of the systolic array between row 0 and M-1. (ii) In response to the data type indicator indicating a second data type, Each element of the left-side data and the right-side data is divided into half of the first element and half of the second element, Half of each first element of the aforementioned right-hand matrix register is loaded into the data processing cell between row 0 and M-1. Half of each second element of the aforementioned right-hand matrix register is loaded into the data processing cell between row M and 2M-1, For each row of the left-side data, Half of the first element of each row of the left-side data is passed through the corresponding row of the data processing cell between row 0 and M-1. A system configured to pass half of the second element of each row of the left-side data through the corresponding row of the data processing cell between row M and 2M-1.

2. The system according to claim 1, wherein the first data type includes at least one of an 8-bit integer, an 8-bit floating-point number, or a 16-bit floating-point number, and the second data type includes a 4-bit integer.

3. The aforementioned systolic processor is In response to the data type indicator indicating a 16-bit floating-point or 8-bit floating-point number, a vector of the elements of the left-side data having shape 1*M is passed in each matrix multiplication cycle. In response to the data type indicator indicating an 8-bit integer, a vector of the elements of the left-side data having shape 2*M is passed in each matrix multiplication cycle. The system according to claim 2, configured to pass a vector of the elements of the left-side data having shape 2*2M in response to the data type indicator indicating a 4-bit integer, for each matrix multiplication cycle.

4. One or more 16-bit floating-point multiply-accumulate operation chains, For each 16-bit floating-point multiply-accumulate chain, there are two 8-bit integer multiply-accumulate chains, It further includes two 4-bit integer multiply-accumulate chains for each 16-bit floating-point multiply-accumulate chain, The aforementioned systolic processor is The 16-bit floating-point multiply-accumulate chain is used to process 8-bit floating-point and 16-bit floating-point data. The two aforementioned 8-bit integer multiply-accumulate chains are used to process 8-bit integer data. The system according to claim 2, configured to process 4-bit integer data using the two 8-bit integer multiply-accumulate chains and the two 4-bit integer multiply-accumulate chains.

5. The system according to claim 4, wherein the systolic processor is configured to generate 2 × M 24-bit results per cycle for 8-bit and 4-bit integer inputs, and to generate 1 × M 32-bit results per cycle for 8-bit and 16-bit floating-point inputs.

6. The system according to claim 1, further comprising a holding register having size M × N and configured to provide the right-side data to the right-side matrix register, wherein the holding register is configured to include at least one data type not supported by the systolic array, and the systolic processor is configured to convert the right-side data of the unsupported data type to the right-side data of a supported data type when the right-side data is provided from the holding register to the right-side matrix register.

7. The system according to claim 6, wherein the unsupported data type is an 8-bit floating-point data type, and the systolic processor is configured to convert the right-side data of the unsupported data type contained in the holding register to a 16-bit floating-point data type when the right-side data is provided from the holding register to the right-side matrix register.

8. The data processing cells of the aforementioned systolic array are A floating-point dot product accumulator configured to process the left-side data and right-side data having a floating-point data type, The system according to claim 1, comprising a plurality of integer dot product accumulators configured to process the left-side data and the right-side data having integer data types.

9. The aforementioned systolic processor is In response to the data type indicator indicating the floating-point data type, for each matrix multiplication cycle, the data of one vector from the left-side data is passed to the floating-point dot product accumulator. The system according to claim 8, wherein, in response to the data type indicator indicating the integer data type, data from two vectors from the left-side data is passed to the plurality of integer dot product accumulators for each matrix multiplication cycle, and each integer dot product accumulator is configured to receive data from each of the vectors from the left-side data.

10. The system according to claim 8, wherein each integer dot product accumulator further comprises a separate first data path and a second data path, each data path comprising a respective partial product generating layer, a respective carry-storage adder tree layer, and a respective decrement tree layer.

11. The aforementioned systolic processor is In response to the data type indicator indicating an 8-bit integer data type, in each matrix multiplication cycle, the data from the left-side data is passed only to the first data path of each integer dot product accumulator. The system according to claim 10, configured to pass data from the left-side data to both the first data path and the second data path of each integer dot product accumulator in each matrix multiplication cycle, in response to the data type indicator indicating a 4-bit integer data type.

12. For each element of the left-side data, the systolic processor: Half of the first element is passed to one of the first data paths of the integer dot product accumulators, The system according to claim 11, configured to pass the second element to a second corresponding data path of the integer dot product accumulator.

13. An accelerator hardware unit comprising the system according to claim 1, which is one of a graphics processing unit or a tensor processing unit.

14. An accelerator hardware unit comprising a plurality of matrix multiplication units, wherein at least one of the matrix multiplication units comprises the system described in claim 1.

15. A method for performing matrix multiplication on a systolic array of data processing cells that can be configured between a first size of M × N and a second size of 2M × N, The method includes receiving a data type indicator indicating the data type of the input data for the matrix multiplication by one or more processors, wherein the input data includes left-side data and right-side data, and the method is (i) In response to the data type of the data type indicator indicating a first data type, The one or more processors load the right-side data from the right-side matrix register having size M × N into the data processing cell of the systolic array between row 0 and row M-1. For each row of the left-side data, the one or more processors pass each row of the left-side data through the corresponding row of the data processing cell in the systolic array between row 0 and row M-1 so that the matrix multiplication result of the left-side data and the right-side data is derived. (ii) In response to the data type of the data type indicator indicating a second data type, The one or more processors divide each element of the left-side data and the right-side data into half of the first element and half of the second element, The one or more processors load half of each first element from the right-hand matrix register into the data processing cell of the systolic array between row 0 and row M-1, The one or more processors load half of each second element from the right-hand matrix register into the data processing cell of the systolic array between row M and row 2M-1, For each row of the left-side data, The one or more processors pass half of the first element of each row of the left-side data through the corresponding row of the data processing cell in the systolic array between row 0 and row M-1, The process includes, by one or more processors, passing half of the second element of each row of the left-side data through the corresponding row of the data processing cell in the systolic array between row M and row 2M-1, A method for deriving the result of matrix multiplication of the left-side data and the right-side data.

16. The first data type includes at least one of a 16-bit floating-point number, an 8-bit floating-point number, or an 8-bit integer, and the second data type includes a 4-bit integer. In response to the data type indicator indicating the first data type, passing the left-side data includes passing one or more vectors of the elements of the left-side data to rows 0 to M-1 of the systolic array in each matrix multiplication cycle. The method according to claim 15, wherein passing the left-side data in response to the data type indicator indicating the second data type includes passing one or more vectors of elements of the left-side data to all rows 0 to 2M-1 of the systolic array in each matrix multiplication cycle.

17. The method according to claim 16, in response to the data type indicator indicating the second data type, passing the left-side data involves passing two vectors of elements of the left-side data to a plurality of integer dot product accumulators contained in each cell of the systolic array with each matrix multiplication cycle, each cell containing two integer dot product accumulators, and the corresponding row of each pair corresponds to a separate data path of the pair in one of the corresponding integer dot product accumulators.

18. The aforementioned data type indicator further distinguishes between integer data types and floating-point data types. In response to the data type indicator indicating the floating-point data type, passing the left-side data includes passing a vector of elements of the left-side data to the systolic array in each matrix multiplication cycle. The method according to claim 15, wherein passing the left-side data in response to the data type indicator indicating the integer data type includes passing two vectors of elements of the left-side data to the systolic array in each matrix multiplication cycle.

19. The method further includes storing the right-side data in a holding register having size M × N, wherein the right-side data is of a data type not supported by the systolic array, and the method further includes The method according to claim 15, comprising storing the right-side data from the holding register into the right-side matrix register, wherein the loading comprises converting the right-side data to a data type supported by the systolic array by one or more processors, wherein the data type not supported by the systolic array is an 8-bit floating-point number, and the data type supported by the systolic array is a 16-bit floating-point number.

20. The left-side data and right-side data are received as 128 x 128 matrices, where M = 128 and N = 128, and the method is as follows: For data inputs including 8-bit or 16-bit floating-point operands, a 1 x 128 32-bit floating-point result is generated for each cycle of the systolic array, The method according to claim 15, further comprising generating a 2 x 128 24-bit result for each cycle of the systolic array in the case of a data input including a 4-bit or 8-bit integer operand.

Images