Multimodal systolic arrays for matrix multiplication.

Abstract

A system and method for matrix multiplication using a systolic array configurable between multiple operating modes. The systolic processor may receive a data type indicator for matrix multiplication. For a first data type, the systolic processor may load right-side data from a right-side matrix register into data processing cells of the systolic array between rows 0 and M-1 and pass each row of left-side data through a corresponding row of the systolic array between rows 0 and M-1. For a second data type, the systolic processor may split each element of the left-side data and right-side data into half of a first element and half of a second element and move half of each element through a corresponding row of the systolic array between rows 0 and 2M-1.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Application No. 18 / 168,972, filed February 14, 2023, and claims the benefit of the filing date of U.S. Provisional Application No. 63 / 477,836, filed December 30, 2022, the disclosure of which is incorporated herein by reference. [Background technology]

[0002] Accelerators for neural networks, such as deep neural networks (DNNs), utilize systolic arrays for high-density computations. A systolic array is an array of processing elements, such as processors, microprocessors, or dedicated circuits configured to process a portion of data. Adjacent processing elements of a systolic array may be connected via one or more interconnects, such as wires or other physical connections, for example on a printed circuit board.

[0003] Existing systolic arrays utilize one or more matrix multiplication units (MXUs) within a processor to perform matrix multiplication operations. To enable processors to achieve high performance in matrix multiplication, a data format called "bfloat16," or "bf16" for short, for brain floating point, is used for multiply-and-accumulate operations within the MXU. bf16 is a 16-bit floating-point format containing one sign bit, eight exponent bits, and seven mantissa bits. Adapting systolic array architectures to support the bf16 data type has been shown to save on-chip memory and improve processing speed. However, these adaptations also limit the types of data formats that can be processed using the systolic array. Summary of the Invention

[0004] The present disclosure provides a new systolic array architecture that can process data of multiple data types, i.e., it can perform matrix multiplication operations with high efficiency for both bf16 and other data types such as 4-bit integer values, 8-bit integer values, and 8-bit floating-point values.

[0005] In one aspect of the disclosure, a system for performing matrix multiplication of input data including left-side data and right-side data includes a right-side matrix register having a size M×N, a systolic array of data processing cells configurable between a first size M×N and a second size 2M×N, and a systolic processor, wherein the systolic processor receives a data type indicator indicating a data type of the input data, and (i) in response to the data type indicator indicating the first data type, loads the right-side data from the right-side matrix register into the data processing cells between rows 0 and M−1, and transmits each row of the left-side data through a corresponding row of the systolic array between rows 0 and M−1. and (ii) in response to the data type indicator indicating a second data type, splitting each element of the left-side data and the right-side data into half of a first element and half of a second element, loading half of each first element of the right-side matrix register into data processing cells between rows 0 and M-1 and half of each second element of the right-side matrix register into data processing cells between rows M and 2M-1, and for each row of left-side data, passing half of the first elements of each row of left-side data through a corresponding row of data processing cells between rows 0 and M-1 and passing half of the second elements of each row of left-side data through a corresponding row of data processing cells between rows M and 2M-1.

[0006] According to aspects of the present disclosure, the first data type may include at least one of an 8-bit integer, an 8-bit floating point, or a 16-bit floating point, and the second data type may include a 4-bit integer.

[0007] According to aspects of the present disclosure, the systolic processor may be configured to pass a vector of left-side data elements having shape 1*M in each matrix multiplication cycle in response to the data type indicator indicating 16-bit floating point or 8-bit floating point, to pass a vector of left-side data elements having shape 2*M in each matrix multiplication cycle in response to the data type indicator indicating 8-bit integer, and to pass a vector of left-side data elements having shape 2*2M in each matrix multiplication cycle in response to the data type indicator indicating 4-bit integer.

[0008] According to aspects of the present disclosure, the system further includes one or more 16-bit floating-point multiply-add chains, two additional 8-bit integer multiply-add chains for each 16-bit floating-point multiply-add chain, and two additional 4-bit integer multiply-add chains for each 16-bit floating-point multiply-add chain. The systolic processor is configured to process 8-bit floating-point and 16-bit floating-point data using the 16-bit floating-point multiply-add chain, process 8-bit integer data using the two 8-bit integer multiply-add chains, and process 4-bit integer data using the two 8-bit integer multiply-add chains and two 4-bit integer multiply-add chains.

[0009] According to aspects of the present disclosure, a systolic processor is configured to generate 2×M 24-bit results per cycle for 8-bit and 4-bit integer inputs, and 1×M 32-bit results per cycle for 8-bit and 16-bit floating-point inputs.

[0010] According to an aspect of the present disclosure, the system further includes a holding register having a size M×N and configured to provide right side data to the right matrix register, the holding register configured to include at least one data type not supported by the systolic array, and the systolic processor configured to convert the right side data of the unsupported data type to right side data of a supported data type when the right side data is provided from the holding register to the right matrix register.

[0011] According to an aspect of the present disclosure, the unsupported data type is an 8-bit floating-point data type, and the systolic processor is configured to convert the right-side data of the unsupported data type contained in the holding register to a 16-bit floating-point data type when the right-side data is provided from the holding register to the right-side matrix register.

[0012] According to aspects of the present disclosure, a data processing cell of a systolic array includes a floating-point dot product accumulator configured to process left-side and right-side data having a floating-point data type, and a plurality of integer dot product accumulators configured to process left-side and right-side data having an integer data type.

[0013] According to aspects of the present disclosure, the systolic processor is configured to, in response to the data type indicator indicating a floating-point data type, pass data for one vector from the left-side data to a floating-point dot-product accumulator for each matrix multiplication cycle, and, in response to the data type indicator indicating an integer data type, pass data from two vectors from the left-side data to a plurality of integer dot-product accumulators for each matrix multiplication cycle, each integer dot-product accumulator configured to receive data from a respective vector of the left-side data.

[0014] According to aspects of the present disclosure, each integer dot product accumulator further includes separate first and second data paths, each data path including a respective partial product generation layer, a respective carry-save adder tree layer, and a respective decrease tree layer.

[0015] According to an aspect of the present disclosure, the systolic processor is configured to pass data from the left side data to only the first data path of each integer dot product accumulator in each matrix multiplication cycle in response to the data type indicator indicating an 8-bit integer data type, and to pass data from the left side data to both the first data path and the second data path of each integer dot product accumulator in each matrix multiplication cycle in response to the data type indicator indicating a 4-bit integer data type.

[0016] According to aspects of the present disclosure, for each element of the left-side data, the systolic processor is configured to pass half of the first element to a first data path of a corresponding one of the integer dot product accumulators and pass the second element to a second data path of a second corresponding one of the integer dot product accumulators.

[0017] Another aspect of the present disclosure is directed to an accelerator hardware unit including a system as described in any of the embodiments herein. The accelerator hardware unit may be one of a graphics processing unit or a tensor processing unit. In some examples, the accelerator hardware unit includes multiple matrix multiplication units, at least one of which includes the system as described in any of the embodiments herein.

[0018] However, a further aspect of the present disclosure is a method for performing matrix multiplication in a systolic array of data processing cells configurable between a first size of M×N and a second size of 2M×N, comprising receiving, by one or more processors, a data type indicator indicating a data type of input data for the matrix multiplication, the input data including left-side data and right-side data, the method comprising: (i) in response to the data type of the data type indicator indicating a first data type, loading, by the one or more processors, right-side data from a right-side matrix register having size M×N into data processing cells of the systolic array between rows 0 and M−1; and for each row of left-side data, passing, by the one or more processors, the respective row of left-side data through a corresponding row of data processing cells of the systolic array between rows 0 and M−1 to derive a matrix multiplication result of the left-side data and the right-side data; and (ii) in response to the data type of the data type indicator indicating a second data type, loading, by the one or more processors, half of each first element from the right-side matrix register into data processing cells of the systolic array between rows 0 and M-1; loading, by the one or more processors, half of each second element from the right-side matrix register into data processing cells of the systolic array between rows M and 2M-1; and for each row of left-side data, passing, by the one or more processors, half of the first elements of each row of left-side data through corresponding rows of data processing cells of the systolic array between rows 0 and M-1; and passing, by the one or more processors, half of the second elements of each row of left-side data through corresponding rows of data processing cells of the systolic array between rows M and 2M-1, thereby deriving a result of matrix multiplication of the left-side data and the right-side data.

[0019] According to aspects of the present disclosure, the first data type may include at least one of a 16-bit floating point, an 8-bit floating point, or an 8-bit integer, and the second data type may include a 4-bit integer. In some examples, passing the left-side data in response to the data type indicator indicating the first data type may include passing one or more vectors of elements of the left-side data to only rows 0 through M-1 of the systolic array in each matrix multiplication cycle, and passing the left-side data in response to the data type indicator indicating the second data type may include passing one or more vectors of elements of the left-side data to all rows 0 through 2M-1 of the systolic array in each matrix multiplication cycle.

[0020] According to aspects of the present disclosure, passing the left-side data in response to the data type indicator indicating the second data type may include passing two vectors of elements of the left-side data to a plurality of integer dot product accumulators included in each cell of the systolic array for each matrix multiplication cycle, where each cell may include two integer dot product accumulators, and corresponding rows of each pair may correspond to separate data paths of the pair in a corresponding one of the plurality of integer dot product accumulators.

[0021] According to aspects of the present disclosure, the data type indicator may further distinguish between integer and floating-point data types. For example, in response to the data type indicator indicating a floating-point data type, passing the left-side data may include passing one vector of elements of the left-side data to the systolic array on each matrix multiplication cycle, and in response to the data type indicator indicating an integer data type, passing the left-side data may include passing two vectors of elements of the left-side data to the systolic array on each matrix multiplication cycle.

[0022] According to aspects of the present disclosure, the method may further include storing the right side data in a holding register having a size M×N, where the right side data is of a data type not supported by the systolic array, and the method further includes storing the right side data from the holding register to a right matrix register, and the loading includes converting, by the one or more processors, the right side data to a data type supported by the systolic array, where the data type not supported by the systolic array is an 8-bit floating point and the data type supported by the systolic array is a 16-bit floating point.

[0023] According to aspects of the present disclosure, the left side and right side data may be received as 128x128 matrices, where M=128 and N=128, and the method may further include, for data inputs including 8-bit or 16-bit floating-point operands, generating 1x128 32-bit floating-point results for each cycle of the systolic array, and for data inputs including 4-bit or 8-bit integer operands, generating 2x128 24-bit results for each cycle of the systolic array. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a block diagram of a systolic array according to aspects of the present disclosure. [Figure 2A] FIG. 2 is a block diagram of an exemplary cell included in the systolic array of FIG. [Figure 2B] FIG. 2 is a block diagram of an exemplary cell included in the systolic array of FIG. [Figure 3] FIG. 3 is a block diagram of an exemplary accumulator included in the cell of FIGS. 2A and 2B. [Figure 4] 4 is a diagram illustrating an exemplary data flow within the systolic array of FIGS. [Figure 5] 4 is a diagram illustrating an exemplary data flow within the systolic array of FIGS. [Figure 6]4 is a diagram illustrating an exemplary data flow within the systolic array of FIGS. [Figure 7] 4 is a diagram illustrating an exemplary data flow within the systolic array of FIGS. [Figure 8] 1A-C are diagrams illustrating exemplary matrix multiply-add operation shapes performed by a systolic array according to aspects of the present disclosure. [Figure 9] FIG. 1 is a block diagram of an exemplary electronic device according to aspects of the present disclosure. [Figure 10] 4 is a flow diagram of an exemplary routine for loading and passing data to a systolic array, in accordance with aspects of the present disclosure. [Figure 11] FIG. 1 is a block diagram of an exemplary arrangement of a systolic processor according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0025] overview According to an embodiment, a system and method for matrix multiplication uses a systolic array configurable between multiple operating modes. The systolic processor is configured to receive a data type indicator for matrix multiplication. For a first data type, the systolic processor is configured to load right-side data from the right-side matrix register into data processing cells of the systolic array between rows 0 and M-1 and pass each row of left-side data through a corresponding row of the systolic array between rows 0 and M-1. For a second data type, the systolic processor is configured to split each element of the left-side data and right-side data into a respective first half and second half and move each half of the element through a corresponding row of the systolic array between rows 0 and 2M-1.

[0026] The systolic array is configured to perform matrix multiplication operations on operands of two or more different data type formats, such as 4-bit integer operands, 8-bit integer operands, 8-bit floating-point operands, or 16-bit floating-point operands. The systolic array uses at least some of the same input buses and the same output buses for the multiple formats. Furthermore, cells of the systolic array include processing elements for supporting 16-bit floating-point by 16-bit floating-point multiplication, 8-bit integer by 8-bit integer multiplication, and 4-bit integer by 4-bit integer multiplication. For example, each cell of the systolic array may include a single 16-bit floating-point dot product accumulator (DPA) that supports 16-bit floating-point by 16-bit floating-point matrix multiplication operations, and a pair of 8-bit integer DPAs that support both 8-bit integer by 8-bit integer and 4-bit integer by 4-bit integer matrix multiplication operations.

[0027] A systolic processor, which may include several processing elements for operating the systolic array, may receive an indication of the data type of both the right-side data loaded into the array and the left-side data flowing into the array, and may determine how to input and process the data into the systolic array based on the received indication of the data type. The indication may be received in the form of a flag indicating whether the data type is integer or floating-point, the size of the data type, such as 4 bits, 8 bits, or 16 bits, and other characteristics of the data type that may be necessary in some circumstances to perform a proper matrix multiplication operation.

[0028] For 16-bit floating-point operands stored in the form of an M×N element matrix, the right-side data may be loaded into the systolic array between rows 0 and M−1 of the array, and each element of the left-side data may be passed through each row of the systolic array between rows 0 and M−1 of the array. In such a case, the 16-bit floating-point input may be effectively interpreted as 1×M elements per matrix multiplication cycle of the systolic array, and the output of the array may result in 1×M 32-bit floating-point (also known as “f32”) values ​​per cycle.

[0029] For 8-bit floating-point operands stored in the form of M×N element matrices, the 8-bit operands can be converted to 16-bit floating-point operands so that they flow in and out of the systolic array in the same way as 16-bit floating-point operands.

[0030] Similar to 16-bit floating-point elements, for 8-bit integer operands stored in the form of an M × N element matrix, each element of the right-side data may be loaded into the systolic array between rows 0 and M-1 of the array, and each element of the left-side data may be passed through each row of the systolic array between rows 0 and M-1 of the array. However, unlike 16-bit floating-point elements, each cell of the systolic array is capable of twice the throughput of 8-bit integer operations, so each cell of the systolic array can receive and process two rows of 8-bit integer elements in parallel per cycle. In such a case, the 8-bit integer input may be effectively interpreted as 2 × M elements per matrix multiplication cycle of the systolic array, resulting in 2 × M 24-bit outputs per cycle.

[0031] Finally, for 4-bit integer operands stored in the form of an M × N element matrix, each element of the stored matrix, instead of containing a single 8-bit operand, may contain two separate 4-bit operands. As a result, when elements are moved into the systolic array, each element may be interpreted as half of two elements, whereby each half of an element is a separate 4-bit integer operand. Because the stored M × N element matrix is ​​interpreted as having twice as many elements per row, half of each element of the right-side data may be loaded into each systolic row between rows 0 and 2M-1 of the array, and half of each element of the left-side data may be passed through each row of the systolic array between rows 0 and 2M-1 of the array. In such a case, the 4-bit integer input may be effectively interpreted as 2 × 2M elements per matrix multiplication cycle of the systolic array. Furthermore, the 4-bit integer input may be effectively interpreted as 2 × 2M elements per matrix multiplication cycle, resulting in an array output of 2 × M 24 bits per cycle.

[0032] The disclosed systolic array architecture provides the advantage of being able to process multiple data types, including both integer and floating-point values ​​of different sizes, while maintaining good throughput. Instead of providing completely different hardware for each data type, the same input and output buses can be used for the various data types to feed the various accumulators included in the systolic array. Ultimately, these advantages are beneficial in maintaining a highly efficient processor that can process many different data types without sacrificing excessive space for additional hardware to support the various data types.

[0033] Exemplary System 1 is a diagram of an exemplary systolic array 100 according to an embodiment of the present disclosure. Systolic array 100 is shown as having a plurality of cells 110 arranged in an [I,J] array having I rows and J columns.

[0034] Each of the cells 110 may load right-side data from a right-side matrix register 120. The right-side matrix register 120 may be formed as an M×N matrix having M rows and N columns of data elements. Generally, each data element corresponds to a separate operand of a matrix multiplication operation, although in at least some circumstances of the present disclosure, each data element may include multiple operands.

[0035] The first column [0] of cell 110 can also receive left-side data from the left-side vector register 130. The left-side matrix register 130 can also be formed as an M×N matrix having M rows and N columns of data elements. As with the right-side data, each data element typically corresponds to a separate operand of a matrix multiplication operation, although in at least some circumstances of the present disclosure, each data element can include multiple operands.

[0036] In some examples, the systolic array may be stationary, meaning that the entire right-side matrix register 120 is preloaded and kept intact within the array during the matrix multiplication. In other examples, a vector of right-side data may be loaded every cycle, making the systolic array non-stationary. The choice of stationary or non-stationary array may be influenced by timing considerations within the systolic array, such as the direction of control flow and pipelining.

[0037] Each cell 110 of the systolic array 100 may be responsible for receiving a portion of the right-side data, a portion of the left-side data, receiving an output from a previous cell in the same row, calculating the product of the received right-side and left-side data, adding the calculated product to the output from the previous cell, and passing the sum to the next cell in the same row. For example, for a given cell [i,j] of the systolic array, the received right-side data loaded into the cell may correspond to column [n] of the right-side matrix register 120, the left-side data passed through the cell may correspond to row [m] of the left-side vector register 130, an output may be received from cell [i-1,j], the product of the received portions of [m] and [n] may be calculated and added to the output from cell [i-1,j], and the sum may be forwarded to the next cell [i+1,j]. Ultimately, the calculations performed by each of the cells 110 may culminate in a matrix multiplication result 140.

[0038] 2A and 2B are block diagrams of an exemplary individual cell 200, such as cell 110 of systolic array 100 of FIG. 1. Cell 200 includes a processing element that can receive right-side data 210 and left-side data 220, as well as a calculated value from a previous cell along a row of the systolic array. In the example of FIGS. 2A and 2B, cell 200 is shown to include one dot product accumulator (DPA) for floating-point values ​​(Float DPA 230) and two dot product accumulators (DPA) for integer values ​​(Int DPA_0 240 and Int DPA_1 250). Float DPA 230 is used to process incoming floating-point operands, and Int DPAs 240, 250 are used to process incoming integer operands.

[0039] The example of FIG. 2A illustrates the data flow for an incoming floating-point operand. In the example of FIG. 2A, the floating-point operand may be a 16-bit operand or an 8-bit operand. In other examples, the floating-point operand may be a different size. Cell 200 receives right-side data 212 from column [n] of right-side matrix 210. Additionally, on each cycle of the systolic array, cell 200 receives left-side data 222 corresponding to row [m] of the vector in left-side matrix 220 from the previous cell of the systolic array. The cell also receives result 252 of the previous cell [m,n-1], which is the result of a calculation performed on row [m] of the left-side data and column [n-1] of the right-side data, added to any previous results along row [m]. The calculation is performed in floating-point DPA 230, where a dot product of left-side data 222 and right-side data 212 is calculated and then added to the result 252 of the previous cell. On the next cycle, the left side data 222 is then passed along the row to the next cell [m,n+1] along with the calculation result 262 of the calculation.

[0040] The example of Figure 2B illustrates the data flow for incoming integer operands. In the example of Figure 2B, the integer operands may be 8-bit operands or 4-bit operands. In other examples, the integer operands may be different sizes. Cell 200 receives right-side data 214, 216 from column [n] of right-side matrix 210. Additionally, with each cycle of the systolic array, cell 200 receives two rows of left-side data 224, 226 corresponding to rows [m] and [m+1] of two vectors in left-side matrix 220 from the previous cell of the systolic array. Cell 200 also receives result 254, 256 of the previous cell [m,n-1]. Result 254 is the result of the calculation performed on row [m] of the left-side data and column [n-1] of the right-side data, added to any previous results along row [m]. Result 256 is the result of the calculation performed on row [m+1] of the left-side data and column [n-1] of the right-side data added to any previous results along row [m+1]. The calculations for each row [m] and [m+1] are performed by respective integer DPAs 240, 250, and the dot product of each left-side data 224, 226 with each right-side data 214, 216 is calculated and then added to the result 254, 256 of each previous cell [m,n-1] or [m+1,n-1]. On the next cycle, each portion of the left-side data 224, 226 is then passed to its respective next cell [m,n+1] or [m+1,n+1] along its respective row, along with its respective calculation result 264, 266 of the calculation for that row.

[0041] 2B, it can be seen that cell 200's 8-bit integer operations have twice the throughput compared to 16-bit floating-point operations. This doubled throughput is achieved efficiently because the cost of passing 16 bits of data for a 16-bit floating-point operand is comparable to the cost of passing 16 bits of data for two 8-bit integer operands. Furthermore, although two operands may require twice as many dot-product-accumulate operations to be performed compared to one operand, cell 200 has two accumulators for the 8-bit integer operands, without any efficiency loss in the calculation stage.

[0042] The cells of Figures 2A and 2B can also handle 4-bit integers with high efficiency. This is achieved by including two separate data paths for the inputs within the integer DPA. Figure 3 is a block diagram of an exemplary integer DPA 300 including a first data path 310 utilized for processing both 4-bit and 8-bit integer operands, and a second data path 320 utilized only for processing 4-bit integer operands. Each data path 310, 320 includes a respective partial product generation layer 312, 322, a respective carry-save adder tree layer 314, 324, and a respective reduction tree layer including one or more reduction trees 316, 326. The components used in these layers 312, 314, 316, 322, 324, 326 can be any conventional components known in the art for performing dot-product accumulation of integer operands, such as 8-bit integers or 4-bit integers.

[0043] 3 as a "type-dependent split and expand" operation 330 to control whether input data 340 received from a right-hand column register and a previous cell is maintained as one input or split into two inputs. In one example, if the data type is a first data type, such as an 8-bit integer, the data may be maintained as one input, and if the data type is a second data type, such as a 4-bit integer, the data may be split into two inputs.

[0044] The DPA 300 may further include other reduce tree(s) 350 for combining the outputs of each of the two data paths. In scenarios where the input data 340 is not split, the operations in the further reduce tree(s) 350 may be skipped. The DPA may also further include a carry propagate adder 360 for adding the result from the previous cell to the result of the current cell. The DPA result 370 may be output from the carry propagate adder 360 and provided to the next cell along the corresponding row. In some examples, the further reduce tree(s) 350 and the carry propagate adder 360 may be shared with the floating-point DPA of the cell, instead of providing separate reduce trees and carry propagate adders for both the floating-point and integer paths within the DPA. Alternatively, separate reduce trees and carry propagate adders may be provided.

[0045] Figures 4-7 are block diagrams illustrating example data flows for different data types that may be received by the systolic array of the present disclosure. The example of Figure 4 illustrates the data flow for a 16-bit floating-point operand, the example of Figure 5 illustrates the data flow for an 8-bit floating-point operand, the example of Figure 6 illustrates the data flow for an 8-bit integer operand, and the example of Figure 7 illustrates the data flow for a 4-bit integer operand.

[0046] In the example of FIG. 4, left-side data 410 from a matrix having a shape M×N is provided to a systolic array 420. The left-side data 410 includes a vector 412 of M 16-bit elements having a shape 1×M and a data type indicator 414 indicating the data type of the vector of elements. In the example of FIG. 4, the elements are shown to be of type "bf16," which is a 16-bit floating-point value. In response, a systolic processor 422 of the systolic array 420 controls the flow of the M elements into rows 0 through M−1 of the systolic array, such that each element occupies a respective row. Another vector 412 may be passed from the left-side data 410 to the systolic array 420 during each cycle of the matrix multiplication operation.

[0047] In the example of FIG. 5, left-side data 510 from a matrix having a shape M×N is provided to systolic array 520. Left-side data 510 includes a vector 512 of M 8-bit elements having a shape of 1×M and a data type indicator 514 indicating that the data type of the vector of elements is “fp8,” a type of 8-bit floating-point value. In response, systolic processor 522 of systolic array 520 can control the conversion of the 8-bit elements to “bf16” 16-bit floating-point values ​​and then control the flow of the M converted elements into rows 0 through M−1 of systolic array 520, whereby each element occupies a respective row. During each cycle of the matrix multiplication operation, another vector 512 can be converted and passed from left-side data 510 to systolic array 520.

[0048] In the example of FIG. 6, left-side data 610 from a matrix having a shape M×N is provided to a systolic array 620. The left-side data 610 includes two vectors 612, 614, each having M 8-bit elements and a shape of 1×M, and a data type indicator 616 indicating that the data type of the vector of elements is “int8,” a type of 8-bit integer value. In response, a systolic processor 622 of the systolic array 620 can control the flow of elements from each vector to each row between rows 0 through M−1 of the systolic array 620, such that each element occupies a row and each row receives two elements. In each cycle of the matrix multiplication operation, the other two vectors 612, 614 can be transformed and passed from the left-side data 610 to the systolic array 620.

[0049] As can be seen from the example in Figure 6, the ability of a systolic array to receive and process two vectors of int8 data in a single cycle allows a matrix multiplication operation on an entire matrix of int8 data to be completed twice as fast as a systolic array that receives only one vector at a time. Thus, the speed for processing elements in the array can correspond to the rate at which elements can be streamed into the array, thereby maintaining the efficiency of the systolic array.

[0050] Finally, in the example of FIG. 7, left-side data 710 from a matrix having a shape M×N is provided to systolic array 720. Left-side data 710 includes two vectors 712, 714, each having M 8-bit elements, with each element containing two operands, thus giving each vector of operands a shape of 1×2M. Left-side data 710 also includes a data type indicator 716, indicating that the data type of the vector of elements is "int4," a type of 4-bit integer value. In response, systolic processor 722 of systolic array 720 can control the division of each element from an 8-bit element into two 4-bit elements, such that the first half of the 8-bit element from bits 0 through 3 constitutes half of the first element corresponding to the first 4-bit operand, and the second half of the 8-bit element from bits 4 through 7 constitutes half of the second element corresponding to the second 4-bit operand. Systolic processor 722 then controls the flow of half of the elements into respective rows of the systolic array. For half of the first elements directed to row [m] of the array, half of the corresponding second elements may be directed to the corresponding row [M+m], which may be physically co-located with row [m] within the same DPA of the systolic array cell. In this way, for each 1×M vector 712, 714 of elements in the left-side data 710, operands are provided to 2M rows of the systolic array. As with the 8-bit integer operands in the example of FIG. 6, a 4-bit integer operand may flow into the systolic array two vectors per cycle of the matrix multiplication operation.

[0051] 7, the ability to receive two vectors of int4 data in a single cycle, and further the ability to process two rows of int4 data within a single cell of the systolic array, allows a matrix multiplication operation on an entire matrix of int4 data to be completed four times faster than a systolic array that receives only one vector at a time and processes only one element per cell. Thus, the efficiency of the systolic array can be maintained because the rate at which elements in the array can be processed can correspond to the rate at which elements can be streamed into the array.

[0052] Although not shown in FIGS. 4-7, the process of loading right-side data into the systolic array may be comparable to the process of streaming left-side data. For example, for 16-bit floating-point values, 8-bit floating-point values, and 8-bit integer values, the right-side data may be loaded into rows 0 through M-1 of the systolic array. Furthermore, for 8-bit floating-point values, the right-side data may also be converted to a 16-bit floating-point "bf16" format. As a further example, for 4-bit integer values, each element of the right-side data may be split into two half elements and loaded into rows 0 through 2M-1 of the systolic array, whereby the half elements of the pair are loaded into physically co-located rows, such as rows 0 and M, or rows 1 and M+1, or rows 2 and M+2.

[0053] Figures 8A-8C are diagrams providing further visualization of the matrix multiplication operations performed in each given cycle of the exemplary systolic arrays of Figures 4-7. The operations shown in Figure 8A correspond to 16-bit floating-point operations, which are performed by the exemplary systolic arrays of Figures 4 and 5. The operations shown in Figure 8B correspond to 8-bit integer operations, which are performed by the exemplary systolic array of Figure 6. The operations shown in Figure 8C correspond to 4-bit integer operations, which are performed by the exemplary systolic array of Figure 7.

[0054] In the example of FIG. 8A , a 1×M vector of bf16 elements from the left-side data 812 is multiplied with an M×N matrix of bf16 elements from the right-side data 814. The bf16 elements may begin as 16-bit floating-point values ​​or as 8-bit floating-point values ​​that are converted to 16-bit floating-point values ​​using a data type conversion instruction such as roundTiesToEven. Converting 8-bit values ​​to 16-bit values ​​may further utilize holding registers having size M×N to store the right-side data and load it into the right-side matrix register. In such an example, 8-bit floating-point may be an unsupported data type for the right-side matrix register, but may be a supported data type for the holding register. Thus, moving 8-bit floating-point values ​​from vector registers to holding registers may facilitate converting the 8-bit values ​​to 16-bit values ​​as the data is moved from the holding register to the right-side matrix register.

[0055] Whether the data type indicator indicates an 8-bit or 16-bit floating-point value, the result of the matrix multiplication operation is a 1×M vector of 32-bit floating-point operands, which may be added to the 1×M vector derived from the calculation in the previous cell of the systolic array 816 to arrive at the cycle's final output result 818, which is itself a 1×M vector of 32-bit floating-point operands. The output may be in 32-bit floating-point format to facilitate accumulation operations.

[0056] 8B, two 1×M vectors of int8 elements from the left-side data 822 are multiplied with an M×N matrix of int8 elements from the right-side data 824. The multiplication operation of the two vectors may occur in parallel in each corresponding cell using the corresponding cell's two integer DPAs. The result of the parallel matrix multiplication operation is a 2×M vector of 24-bit integer operands, which may be added to the 2×M vector derived from the calculation in the previous cell of the systolic array 826, arriving at the cycle's final output result 828, which is itself a 2×M vector of 24-bit integer operands.

[0057] In the example of FIG. 8C , two 1×M vectors of 8-bit integer elements from the left-side data are split into two vectors of 1×2M 4-bit integer operands 832, and an M×N matrix of 8-bit integer elements from the right-side data is split into a 2M×N matrix of 4-bit integer operands 834 and loaded into the systolic array. The multiplication operation of the two vectors may occur in parallel in each corresponding cell using the corresponding cell's two integer DPAs. Furthermore, in each integer DPA, physically co-located rows may receive respective portions of the left-side data. The result of the parallel matrix multiplication operation is a 2×2M vector of 24-bit integer operands, which may be added to the 2×2M vector derived from the calculation in the previous cell of the systolic array 836, resulting in the cycle's final output result 838, which is itself a 2×2M vector of 24-bit integer operands.

[0058] 9 shows a block diagram of an exemplary electronic device 900 for implementing a systolic array according to any of the exemplary embodiments of the present disclosure. The electronic device 900 may include one or more processors 910, such as one or more CPUs, a system memory 920, a bus 930, network interface(s) 940, and other components (not shown), such as storage(s), output device interface(s), and input device interface(s). The bus 930 may be used to communicate between the processor 910, the system memory 920, the network interface(s) 940, and other components. Any or all of the components of the electronic device 900 may be used in conjunction with the subject matter of the present disclosure.

[0059] Depending on the desired configuration, the processor 910 may be of any type, including, but not limited to, one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs), or any combination thereof. The processor 910 may include a systolic array. The processor 910 may include another level of caching, such as a level 1 cache 911 and a level 2 cache 912, a processor core 913, and registers 914. The processor core 913 may include one or more arithmetic logic units (ALUs), one or more floating-point units (FPUs), one or more DSP cores, or any combination thereof. A memory controller 915 may also be used with the processor 910, or in some implementations, the memory controller 915 may be an internal part of the processor 910.

[0060] Depending on the desired configuration, physical memory 920 may be of any type, including, but not limited to, volatile memory such as RAM, non-volatile memory such as ROM, flash memory, etc., or any combination thereof. Physical memory 920 may include an operating system 921, one or more applications 922, and program data 924, which may include service data 925. Non-transitory computer-readable medium program data 924 may include stored instructions that, when executed by one or more processing devices, implement a process for calculating a result of a multiply-accumulate operation 923. In some examples, one or more applications 922 may be arranged to operate with the operating system 921, the program data 924, and the service data 925.

[0061] The electronic device 900 may have additional features or functionality and additional interfaces that facilitate communication between the basic configuration 901 and any necessary devices and interfaces.

[0062] Physical memory 920 may be an example of computer storage media, such as, but not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, or accessible by electronic device 900. Any such computer storage media may be part of device 900.

[0063] Network interface(s) 940 may couple electronic device 900 to a network (not shown) and / or to other electronic devices (not shown). In this manner, electronic device 900 can be part of a network of electronic devices, such as a local area network (“LAN”), a wide area network (“WAN”), an intranet, or a network of networks, e.g., the Internet. In some examples, electronic device 900 may include a network connection interface for forming a network connection to a network and a local communication connection interface for forming a tethering connection with other devices. The connection may be wired or wireless. Electronic device 900 may bridge the network connection and the tethering connection to connect other devices to the network via network interface(s) 940.

[0064] The systolic array may include multiple MAC units 950 to perform the multiply-and-accumulate operations required for matrix multiplication. The MAC units 950, and the systolic arrays on which they operate, may be used in accelerators that may be used in DNN implementations.

[0065] The electronic device 900 may be implemented as part of a small form factor portable (or mobile) electronic device, such as a speaker, headphones, earphones, a mobile phone, a smartphone, a smart watch, a personal data assistant (PDA), a personal media player device, a tablet computer (tablet), a wireless web watch device, a personal headset device, a wearable device, an application-specific device, or a hybrid device including any of the above functionality. The electronic device 900 may also be implemented as a personal computer, including both laptop and non-laptop computer configurations. The electronic device 900 may also be implemented as a server, accelerator, or larger system.

[0066] Exemplary Methods 10 is a flow diagram illustrating an exemplary routine 1000 for controlling the flow of data through a systolic array of the present disclosure. Routine 1000 may be executed by one or more systolic processors included in a system, such as the systolic processors shown in FIGS. 4-7.

[0067] The operations may begin at block 1010, where a systolic processor receives a data type indicator indicating a data type of input data. The input data may include left side data and right side data of a matrix multiplication in a systolic array. In some examples, the data type indicator may be a flag. In some examples, the flag may include one or more bits appended to the bits of the element.

[0068] At block 1020, the systolic processor may determine the data type of the input data based on the data type indicator. For example, the data type may be any one of 16-bit floating point, 8-bit floating point, 8-bit integer, 4-bit integer, etc. In the example of Figure 10, a classification between two data types is shown, where the first data type is processed according to a first set of operations and the second data type is processed according to a second set of operations.

[0069] If the data type corresponds to a first data type corresponding to either a 16-bit floating point, an 8-bit floating point, or an 8-bit integer format, operations may continue to block 1030, where the right-side data is loaded between rows 0 and M-1 of the systolic array, and then in block 1040, each row of the left-side data is passed through a corresponding row between rows 0 and M-1 of the systolic array. The left-side data may be passed one vector per cycle.

[0070] Alternatively, if the data type corresponds to a second data type corresponding to a 4-bit integer format, operations may continue to block 1050, where each element of the left-side data and right-side data is split into half of a first element and half of a second element, and then may continue to blocks 1060 and 170, where the right-side data is loaded between rows 0 and 2 of the systolic array, with half of each first element being placed between rows 0 and M-1 and half of each second element being placed in a corresponding row between rows M and 2-1, and the left-side data is passed through rows 0 through 2 of the systolic array, with half of each first element being passed between rows 0 and M-1 and half of each second element being passed to a corresponding row between rows M and 2-1.

[0071] In a further example, the systolic processor may further distinguish between additional data types and further control different data flows for different data types. For example, the data type indicator may distinguish between 16-bit and 8-bit floating-point values, whereby the 8-bit floating-point indicator may signal the systolic processor to convert the incoming left-side and right-side operands from 8-bit to 16-bit floating-point format. An example of a configuration for a systolic processor to convert 8-bit floating-point values ​​to 16-bit floating-point values ​​is shown in FIG. 11. The arrangement 1100 in FIG. 11 shows four registers 1110, 1120, 1130, and 1140 used to prepare the right-side and left-side operands to be loaded into and passed through the systolic array 1150, respectively. The left-side data is initially stored in a first matrix staging register (MSR_A) 1110, which can support operands in 8-bit floating-point format. The left-side data is then moved from the first matrix staging register 1110 to the left-side matrix register 1120 for loading into the systolic array. Because the systolic array does not support matrix multiplication operations on 8-bit floating-point data, the 8-bit floating-point operands may be converted from 8-bit floating-point format to 16-bit floating-point format during transfer from the first matrix staging register 1110 to the left-side matrix register 1120. Similar operations may be performed on the right-side data using the second matrix staging register 1130 and the right-side matrix register 1140, respectively. Furthermore, although FIG. 11 shows the first matrix staging register 1110 connected to the left matrix register 1120 and the second matrix staging register 1130 connected to the right matrix register 1140, it should be understood that each of the matrix staging registers 1110, 1130 may be interchangeably connected to either the left matrix register 1120 or the right matrix register 1140, thereby allowing each matrix staging register 1110, 1130 to be used to store either left side data or right side data.Note that in addition to the data format conversion benefits provided by the four registers, the four registers may provide additional functionality to the systolic processor. For example, the data transferred to the matrix staging registers may be transposed as it is transferred, meaning that both the left-side and right-side data can be set up for a matrix multiplication operation in transposed form within the systolic processor without having to rely on a separate transpose unit separate from the systolic processor.

[0072] Additionally or alternatively, the data type indicator can distinguish between floating-point and integer values, so that incoming floating-point and integer values ​​can be directed to separate DPAs for processing within a cell of the systolic array. In some examples, moving integer values ​​to the DPAs can include each cell receiving data from two vectors from the left-side data during each cycle of the matrix multiplication operation, and moving the received data to different integer DPAs within the cell for initial parallel processing and subsequent joint processing.

[0073] In one exemplary embodiment of the present disclosure, a systolic array may be used as a matrix multiplication unit (MXU) within a tensor processing unit (TPU). For example, the TPU may include one or more core processors, each connected to one or more MXUs. The MXU may be a systolic array processor for dot product steps. The MXU may be further configurable between 128x128 and 256x128 geometries depending on the data type received. The MXU may support generating 2x128 24-bit results per cycle for both 8-bit and 4-bit integer operands and providing 1x128 32-bit floating-point results per cycle for both 16-bit and 8-bit floating-point operands.

[0074] The exemplary MXUs of this disclosure, and systolic arrays more generally, provide increased versatility and efficiency of matrix multiplication operations by supporting several types of input data using common input and output lines, which helps reduce the overall footprint of TPUs and other chips incorporating MXUs and systolic arrays without sacrificing processing efficiency.

[0075] Although the technology herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the technology. It is therefore to be understood that many modifications can be made to the exemplary embodiments and that other arrangements can be devised without departing from the spirit and scope of the technology as defined by the appended claims.

[0076] Most of the foregoing alternatives are not mutually exclusive but may be implemented in various combinations to realize unique advantages. Because these and other variations and combinations of the above features can be utilized without departing from the subject matter defined by the claims, the foregoing description of embodiments should be construed as illustrative, and not limiting, of the subject matter defined by the claims. By way of example, the preceding actions need not be performed in the exact order described above. Rather, various steps may be processed in a different order, such as in reverse order or simultaneously. Unless otherwise specified, steps may be omitted. Furthermore, the provision of examples described herein, as well as clauses using terms such as "such as," "including," etc., should not be construed as limiting the subject matter of the claims to any particular example. Rather, an example is intended to illustrate only one of many possible embodiments. Furthermore, the same reference numbers in different drawings may identify the same or similar elements.

Claims

1. 1. A system for performing matrix multiplication of input data including left-side data and right-side data, comprising: a right-hand column register having size M×N; a systolic array of data processing cells configurable between a first size M×N and a second size 2M×N; a systolic processor, the systolic processor comprising: receiving a data type indicator indicative of a data type of the input data; (i) in response to the data type indicator indicating a first data type: loading the right-side data from the right-side matrix register into the data processing cells between rows 0 and M-1; passing each row of said left-side data through a corresponding row of said systolic array between rows 0 and M-1; (ii) in response to the data type indicator indicating a second data type: Dividing each element of the left-side data and the right-side data into a respective first half and a second half of the element; loading half of each first element of said right-side matrix register into said data processing cells between rows 0 and M-1; loading half of each second element of said right-side matrix register into said data processing cells between rows M and 2M−1; For each row of the left side data, passing half of the first elements of each row of the left-side data through a corresponding row of the data processing cells between rows 0 and M-1; a system configured to pass half of the second elements of each row of the left-side data through a corresponding row of the data processing cells between rows M and 2M-1.

2. 2. The system of claim 1, wherein the first data type comprises at least one of an 8-bit integer, an 8-bit floating point, or a 16-bit floating point, and the second data type comprises a 4-bit integer.

3. The systolic processor responsive to the data type indicator indicating 16-bit floating point or 8-bit floating point, passing a vector of elements of the left-side data having shape 1*M in each matrix multiplication cycle; responsive to the data type indicator indicating an 8-bit integer, passing a vector of elements of the left-side data having shape 2*M in each matrix multiplication cycle; 3. The system of claim 2, configured to pass a vector of elements of the left-side data having shape 2*2M per matrix multiplication cycle in response to the data type indicator indicating a 4-bit integer.

4. one or more 16-bit floating-point multiply-accumulate chains; two additional 8-bit integer multiply-add chains for each 16-bit floating-point multiply-add chain; two additional 4-bit integer multiply-add chains for each 16-bit floating-point multiply-add chain; The systolic processor using the 16-bit floating-point multiply-accumulate chain to process 8-bit floating-point and 16-bit floating-point data; processing 8-bit integer data using the two 8-bit integer multiply-accumulate chains; 3. The system of claim 2, configured to process 4-bit integer data using the two 8-bit integer multiply-accumulate chains and the two 4-bit integer multiply-accumulate chains.

5. 5. The system of claim 4, wherein the systolic processor is configured to generate 2×M 24-bit results per cycle for 8-bit and 4-bit integer inputs and 1×M 32-bit results per cycle for 8-bit and 16-bit floating-point inputs.

6. 2. The system of claim 1, further comprising a holding register having a size M×N and configured to provide the right side data to the right row and column registers, the holding register configured to contain at least one data type not supported by the systolic array, and the systolic processor configured to convert the right side data of the unsupported data type to right side data of a supported data type as the right side data is provided from the holding register to the right row and column registers.

7. 7. The system of claim 6, wherein the unsupported data type is an 8-bit floating-point data type, and the systolic processor is configured to convert the right-side data of the unsupported data type contained in the holding register to the 16-bit floating-point data type when the right-side data is provided from the holding register to the right-side matrix register.

8. The data processing cells of the systolic array include: a floating-point dot product accumulator configured to process the left-side and right-side data having a floating-point data type; a plurality of integer dot product accumulators configured to process the left-side and right-side data having an integer data type.

9. The systolic processor responsive to the data type indicator indicating the floating-point data type, passing one vector of data from the left side data to the floating-point dot product accumulator for each matrix multiplication cycle; 9. The system of claim 8, configured to, in response to the data type indicator indicating the integer data type, pass data from two vectors from the left side data to the plurality of integer dot product accumulators for each matrix multiplication cycle, each integer dot product accumulator receiving data from a respective vector of the left side data.

10. 9. The system of claim 8, wherein each integer dot product accumulator further includes separate first and second data paths, each data path including a respective partial product generation layer, a respective carry-save adder tree layer, and a respective decrease tree layer.

11. The systolic processor in response to the data type indicator indicating an 8-bit integer data type, passing data from the left side data to only the first data path of each integer dot product accumulator in each matrix multiplication cycle; 11. The system of claim 10, configured to pass data from the left side data to both the first data path and the second data path of each integer dot product accumulator in each matrix multiplication cycle in response to the data type indicator indicating a 4-bit integer data type.

12. For each element of the left-hand data, the systolic processor: passing half of the first elements to the first data path of a corresponding one of the integer dot product accumulators; 12. The system of claim 11 configured to pass the second element to the second data path of the second corresponding one of the integer dot product accumulators.

13. 10. An accelerator hardware unit comprising the system of claim 1, wherein the accelerator hardware unit is one of a graphics processing unit or a tensor processing unit.

14. An accelerator hardware unit comprising a plurality of matrix multiplication units, at least one of the matrix multiplication units comprising the system of claim 1.

15. 1. A method for performing matrix multiplication in a systolic array of data processing cells configurable between a first size of M×N and a second size of 2M×N, comprising: receiving, by one or more processors, a data type indicator indicating a data type of input data for the matrix multiplication, the input data including left side data and right side data; (i) in response to the data type of the data type indicator indicating the first data type, loading, by the one or more processors, the right-side data from a right-side matrix register having size M×N into the data processing cells of the systolic array between row 0 and row M−1; passing, for each row of the left-side data, by the one or more processors, the respective row of the left-side data through a corresponding row of the data processing cells of the systolic array between row 0 and row M−1 to derive, for each row of the left-side data, a matrix multiplication result of the left-side data and the right-side data; (ii) in response to the data type of the data type indicator indicating a second data type; dividing, by the one or more processors, each element of the left-side data and the right-side data into a respective first half and a second half; loading, by the one or more processors, half of each first element from the right-side matrix register into the data processing cells of the systolic array between row 0 and row M-1; loading, by the one or more processors, half of each second element from the right-side matrix register into the data processing cells of the systolic array between row M and row 2M−1; For each row of the left side data, passing, by the one or more processors, half of the first elements of each row of the left-side data through corresponding rows of the data processing cells of the systolic array between row 0 and row M-1; passing, by the one or more processors, half of the second elements of each row of the left-side data through corresponding rows of the data processing cells of the systolic array between row M and row 2M−1; deriving the matrix multiplication result of the left side data and the right side data.

16. the first data type comprises at least one of a 16-bit floating point, an 8-bit floating point, or an 8-bit integer, and the second data type comprises a 4-bit integer; responsive to the data type indicator indicating the first data type, passing the left-side data includes passing one or more vectors of elements of the left-side data to only rows 0 through M-1 of the systolic array in each matrix multiplication cycle; 16. The method of claim 15, wherein passing the left-side data in response to the data type indicator indicating the second data type comprises passing one or more vectors of elements of the left-side data to all rows from rows 0 to 2-1 of the systolic array in each matrix multiplication cycle.

17. 17. The method of claim 16, wherein passing the left-side data in response to the data type indicator indicating the second data type comprises passing two vectors of elements of the left-side data to a plurality of integer dot product accumulators included in each cell of the systolic array for each matrix multiplication cycle, each cell including two integer dot product accumulators, and corresponding rows of each pair corresponding to separate data paths of the pair in a corresponding one of the plurality of integer dot product accumulators.

18. the data type indicator further distinguishes between integer and floating point data types; responsive to the data type indicator indicating the floating-point data type, passing the left-side data includes passing one vector of elements of the left-side data to the systolic array on each matrix multiplication cycle; 16. The method of claim 15, wherein, in response to the data type indicator indicating the integer data type, passing the left-side data comprises passing two vectors of elements of the left-side data to the systolic array in each matrix multiplication cycle.

19. storing the right-side data in a holding register having a size M×N, the right-side data being of a data type not supported by the systolic array; 16. The method of claim 15, further comprising storing the right-side data from the holding register to the right-side matrix register, and wherein the loading comprises converting, by the one or more processors, the right-side data to a data type supported by the systolic array, wherein the data type not supported by the systolic array is 8-bit floating point and the data type supported by the systolic array is 16-bit floating point.

20. The left side and right side data are received as 128x128 matrices, where M=128 and N=128, and the method comprises: for a data input including 8-bit or 16-bit floating point operands, producing 1 x 128 32-bit floating point results for each cycle of said systolic array; 16. The method of claim 15, further comprising: for data inputs comprising 4-bit or 8-bit integer operands, generating a 2x128 24-bit result for each cycle of the systolic array.

Images