Multi-modal systolic array for matrix multiplication

Abstract

A system and method for matrix multiplication using a systolic array configurable between a plurality of operating modes. A pulsation processor may receive a data type indicator for matrix multiplication. For a first data type, the systolic processor may load right-hand side data from a right-hand matrix register into data processing units of the systolic array between rows 0 and M-1, and pass respective rows of left-hand side data through corresponding rows of the systolic array between rows 0 and M-1. For the second data type, the pulsation processor may segment each element of the left-hand-side data and the right-hand-side data into a respective first element half and a respective second element half, and move each element half through a corresponding row of the pulsation array between rows 0 and 2M-1.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation of U.S. application No. 18 / 168,972 filed on February 14, 2023, which claims the benefit of the filing date of U.S. provisional patent application No. 63 / 477,836 filed on December 30, 2022, and the disclosures of that U.S. application and that U.S. provisional patent application are hereby incorporated herein by reference. Background Art

[0002] Accelerators for neural networks, such as deep neural networks (DNNs), have utilized systolic arrays for high-density computation. A systolic array is an array of processing elements, such as processors, microprocessors, or specialized circuitry, configured to process some data. Adjacent processing elements of a systolic array can be connected by one or more interconnects, such as wires or other physical connections, on, for example, a printed circuit board.

[0003] Existing systolic arrays utilize one or more matrix multiplication units (MXUs) within a processor to perform matrix multiplication operations. To enable the processor to achieve high performance in matrix multiplication, a floating-point data format called "bfloat16" or simply "bf16" is used for multiply-accumulate operations within the MXU. bf16 is a 16-bit floating-point format that includes a sign bit, eight exponent bits, and seven mantissa bits. Adapting the systolic array architecture to support the bf16 data type has been found to save on-chip memory and increase processing speed. However, these adaptations also limit the types of data formats that can be processed using systolic arrays. Summary of the Invention

[0004] The present disclosure provides a new systolic array architecture that allows processing of data of multiple data types, which means that matrix multiplication operations can be performed with high efficiency for both bf16 and other data types such as 4-bit integer values, 8-bit integer values, and 8-bit floating point values.

[0005] In one aspect of the present disclosure, a system for performing matrix multiplication of input data including left-hand-side data and right-hand-side data includes: a right-hand matrix register having a size of M×N; a systolic array of data processing units configurable between a first size of M×N and a second size of 2M×N; and a systolic processor configured to: receive a data type indicator indicating a data type of the input data; (i) in response to the data type indicator indicating a first data type: load the right-hand-side data from the right-hand matrix register into the data processing units between rows 0 and M−1; and pass each corresponding row of the left-hand-side data through a corresponding row of the systolic array between rows 0 and M−1; (ii) in response to the data type indicator indicating a first data type: In response to the data type indicator indicating a second data type: splitting each element of the left-hand data and the right-hand data into corresponding first element halves and second element halves; loading each first element half from the right-hand matrix register into the data processing unit between rows 0 and M-1; and loading each second element half from the right-hand matrix register into the data processing unit between rows M and 2M-1; and for each corresponding row of the left-hand data: passing the first element half of the corresponding row of the left-hand data through the corresponding row of the data processing unit between rows 0 and M-1; and passing the second element half of the corresponding row of the left-hand data through the corresponding row of the data processing unit between rows M and 2M-1.

[0006] According to an aspect of the present disclosure, the first data type includes at least one of an 8-bit integer, an 8-bit floating point, or a 16-bit floating point, and the second data type may include a 4-bit integer.

[0007] According to an aspect of the present disclosure, the systolic processor may be configured to: in response to the data type indicator indicating 16-bit floating point or 8-bit floating point, pass a vector of elements of left-hand-side data having a shape of 1*M during each matrix multiplication cycle; in response to the data type indicator indicating 8-bit integer, pass a vector of elements of left-hand-side data having a shape of 2*M during each matrix multiplication cycle; and in response to the data type indicator indicating 4-bit integer, pass a vector of elements of left-hand-side data having a shape of 2*2M during each matrix multiplication cycle.

[0008] According to one aspect of the present disclosure, the system further includes one or more 16-bit floating-point multiply-add chains, two additional 8-bit integer multiply-add chains for each 16-bit floating-point multiply-add chain, and two additional 4-bit multiply-add chains for each 16-bit floating-point multiply-add chain. The systolic processor is configured to process 8-bit floating-point and 16-bit floating-point data using the 16-bit floating-point multiply-add chain, process 8-bit integer data using the two 8-bit integer multiply-add chains, and process 4-bit integer data using the two 8-bit integer multiply-add chains and the two 4-bit integer multiply-add chains.

[0009] According to one aspect of the disclosure, a systolic processor is configured to produce 2×M 24-bit results per cycle for 8-bit and 4-bit integer inputs and 1×M 32-bit results per cycle for 8-bit and 16-bit floating-point inputs.

[0010] According to one aspect of the present disclosure, the system further includes a holding register having a size of M×N and configured to provide right-hand-side data to the right-hand matrix register. The holding register is configured to contain at least one data type not supported by the systolic array. The systolic processor is configured to convert the right-hand-side data of the unsupported data type to right-hand-side data of the supported data type when the right-hand-side data is provided from the holding register to the right-hand matrix register.

[0011] According to an aspect of the present disclosure, the unsupported data type is an 8-bit floating point data type, and the systolic processor is configured to convert right-hand side data of the unsupported data type contained in the holding register to a 16-bit floating point data type when the right-hand side data is provided from the holding register to the right-hand matrix register.

[0012] According to one aspect of the present disclosure, a data processing unit of a systolic array includes a floating-point accumulator configured to process left-hand-side data and right-hand-side data having a floating-point data type, and a plurality of integer accumulators configured to process left-hand-side data and right-hand-side data having an integer data type.

[0013] According to one aspect of the present disclosure, a systolic processor is configured to: in response to a data type indicator indicating a floating-point data type, pass data from one vector of left-hand-side data to a floating-point accumulation / adder during each matrix multiplication cycle; and in response to the data type indicator indicating an integer data type, pass data from two vectors of left-hand-side data to a plurality of integer accumulation / adders during each matrix multiplication cycle. Each integer accumulation / adder is configured to receive data from a corresponding vector of left-hand-side data.

[0014] According to an aspect of the present disclosure, each integer point accumulation adder further includes independent first and second data paths, and each data path includes a corresponding partial product generation layer, a corresponding carry-save adder tree layer, and a corresponding reduction tree layer.

[0015] According to an aspect of the present disclosure, the systolic processor is configured to, in response to the data type indicator indicating an 8-bit integer data type, pass data from the left-hand side data to only the first data path of each integer-point accumulator-adder during each matrix multiplication cycle, and, in response to the data type indicator indicating a 4-bit integer data type, pass data from the left-hand side data to both the first data path and the second data path of each integer-point accumulator-adder during each matrix multiplication cycle.

[0016] According to one aspect of the present disclosure, for each element of the left-hand side data, the systolic processor is configured to pass a first element half to a first data path of a corresponding one of the integer point accumulators, and pass a second element half to a second data path of a second corresponding one of the integer point accumulators.

[0017] Another aspect of the present disclosure relates to an accelerator hardware unit comprising a system as described in any of the embodiments herein. The accelerator hardware unit may be one of a graphics processing unit or a tensor processing unit. In some examples, the accelerator hardware unit comprises a plurality of matrix multiplication units, and at least one of the matrix multiplication units comprises a system as described in any of the embodiments herein.

[0018] Yet further aspects of the present disclosure relate to a method for performing matrix multiplication in a systolic array of data processing units configurable between a first size of M×N and a second size of 2M×N, the method comprising: receiving, by one or more processors, a data type indicator indicating a data type of input data for matrix multiplication, the input data comprising left-hand side data and right-hand side data; (i) in response to the data type indicator indicating a first data type: loading, by the one or more processors, the right-hand side data from a right-hand matrix register having a size of M×N into a data processing unit of the systolic array between row 0 and row M−1; and for each respective row of the left-hand side data, passing, by the one or more processors, the respective row of the left-hand side data through a corresponding row of the data processing unit of the systolic array between row 0 and row M−1 to obtain a matrix multiplication result of the left-hand side data and the right-hand side data; and (ii) in response to the data type indicator indicating a first data type: Indicating a second data type: splitting, by the one or more processors, each element of the left-hand data and the right-hand data into a corresponding first element half and a second element half; loading, by the one or more processors, each first element half from the right-hand matrix register into a data processing element of the systolic array between row 0 and row M-1; and loading, by the one or more processors, each second element half from the right-hand matrix register into a data processing element of the systolic array between row M and row 2M-1; for each corresponding row of the left-hand data: passing, by the one or more processors, the first element half of the corresponding row of the left-hand data through a corresponding row of the data processing elements of the systolic array between row M and row 2M-1, to obtain a matrix multiplication result of the left-hand data and the right-hand data.

[0019] According to an aspect of the present disclosure, the first data type may include at least one of a 16-bit floating point, an 8-bit floating point, or an 8-bit integer, and the second data type may include a 4-bit integer. In some examples, in response to the data type indicator indicating the first data type, passing the left-hand-side data may involve passing the one or more vectors of elements of the left-hand-side data to only rows 0 through M-1 of the systolic array during each matrix multiplication cycle, and in response to the data type indicator indicating the second data type, passing the left-hand-side data may involve passing the one or more vectors of elements of the left-hand-side data to all rows between 0 and 2M-1 of the systolic array during each matrix multiplication cycle.

[0020] According to an aspect of the present disclosure, in response to the data type indicator indicating the second data type, passing the left-hand side data may involve passing two vectors of elements of the left-hand side data to a plurality of integer point accumulators included in each cell of the systolic array during each matrix multiplication cycle. Each cell may include two integer point accumulators, and each pair of corresponding rows may correspond to a pair of independent data paths within a corresponding one of the plurality of integer point accumulators.

[0021] According to an aspect of the present disclosure, the data type indicator may further distinguish between integer data types and floating-point data types. For example, passing the left-hand-side data in response to the data type indicator indicating the floating-point data type may involve passing one vector of elements of the left-hand-side data to the systolic array during each matrix multiplication cycle, and passing the left-hand-side data in response to the data type indicator indicating the integer data type may involve passing two vectors of elements of the left-hand-side data to the systolic array during each matrix multiplication cycle.

[0022] According to an aspect of the present disclosure, the method may further include: storing right-hand-side data in a holding register having a size of M×N, the right-hand-side data being a data type not supported by the systolic array; and loading the right-hand-side data from the holding register into a right-hand matrix register, whereby the loading involves converting, by one or more processors, the right-hand-side data into a data type supported by the systolic array, and the data type not supported by the systolic array is 8-bit floating point, while the data type supported by the systolic array is 16-bit floating point.

[0023] According to an aspect of the present disclosure, the left-hand side data and the right-hand side data may be received as 128×128 matrices, where M=128 and N=128, and the method may further include generating 1×128 32-bit floating-point results per cycle of the systolic array for data inputs including 8-bit or 16-bit floating-point operands, and generating 2×128 24-bit results per cycle of the systolic array for data inputs including 4-bit or 8-bit integer operands. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a block diagram of a systolic array according to an aspect of the present disclosure.

[0025] Figure 2A and Figure 2B is included in Figure 1 Block diagram of an example cell in a systolic array.

[0026] Figure 3 is included in Figure 2A and Figure 2B Block diagram of an example accumulator in the unit.

[0027] Figures 4 to 7 It shows Figures 1 to 3 Diagram of example data flow within a systolic array.

[0028] Figures 8A to 8C is a diagram illustrating the shape of example matrix multiplication and addition operations performed by a systolic array according to an aspect of the present disclosure.

[0029] Figure 9 is a block diagram of an example electronic device according to an aspect of the present disclosure.

[0030] Figure 10 is a flow chart of an example routine for loading and transferring data in a systolic array according to an aspect of the present disclosure.

[0031] Figure 11 is a block diagram of an example arrangement for a systolic processor according to an aspect of the present disclosure. DETAILED DESCRIPTION Overview

[0032] According to an implementation, a system and method for matrix multiplication uses a systolic array configurable between multiple operating modes. A systolic processor is configured to receive a data type indicator for matrix multiplication. For a first data type, the systolic processor is configured to load right-hand-side data from a right-hand matrix register into a data processing element of the systolic array between rows 0 and M-1, and pass corresponding rows of left-hand-side data through corresponding rows of the systolic array between rows 0 and M-1. For a second data type, the systolic processor is configured to split each element of the left-hand-side data and the right-hand-side data into corresponding first and second element halves, and shift each element half through corresponding rows of the systolic array between rows 0 and 2M-1.

[0033] The systolic array is arranged to perform matrix multiplication operations on operands of two or more different data type formats, such as 4-bit integer operands, 8-bit integer operands, 8-bit floating-point operands, or 16-bit floating-point operands. The systolic array uses at least some of the same input buses and the same output buses for the multiple formats. Additionally, the cells of the systolic array include processing elements for supporting each of 16-bit floating-point by 16-bit floating-point multiplication, 8-bit integer by 8-bit integer multiplication, and 4-bit integer by 4-bit integer multiplication. For example, each cell of the systolic array may include a single 16-bit floating-point accumulator-adder (DPA) for supporting a 16-bit floating-point by 16-bit floating-point matrix multiplication operation, and a pair of 8-bit integer DPAs for supporting both an 8-bit integer by 8-bit integer matrix multiplication operation and a 4-bit integer by 4-bit integer matrix multiplication operation.

[0034] A systolic processor, which may include several processing elements for operating a systolic array, may receive an indication of the data type of both right-hand side data loaded into the array and left-hand side data flowing into the array, and determine how to input the data into the systolic array and process the data based on the received indication of the data type. The indication may be received in the form of a flag and may indicate whether the data type is integer or floating point, the size of the data type (such as 4 bits, 8 bits, or 16 bits), and other characteristics of the data type that may be required to perform appropriate matrix multiplication operations in some cases.

[0035] In the case of storing 16-bit floating-point operands in the form of an M×N element matrix, the right-hand side data is loaded into the systolic array between rows 0 and M-1 of the array, and each element of the left-hand side data can be passed through the corresponding row of the systolic array between rows 0 and M-1 of the array. In such a case, the 16-bit floating-point bit input can be effectively interpreted as 1×M elements of the systolic array for each matrix multiplication cycle, and the output of the array can be 1×M 32-bit floating-point (also called "f32") results per cycle.

[0036] In the case of 8-bit floating-point operands stored in the form of an M×N element matrix, the 8-bit operands may be converted to 16-bit floating-point operands so that they flow into and out of the systolic array in the same manner as 16-bit floating-point operands.

[0037] In the case of 8-bit integer operands stored in the form of an M×N element matrix, similar to 16-bit floating-point elements, each element of the right-hand side data can be loaded into the systolic array between rows 0 and M-1 of the array, and each element of the left-hand side data can be passed through the corresponding row of the systolic array between rows 0 and M-1 of the array. However, unlike 16-bit floating-point elements, since each element of the systolic array supports twice the throughput for 8-bit integer operations, each element of the systolic array can receive and process two rows of 8-bit integer elements in parallel per cycle. In such a case, the 8-bit integer input can be effectively interpreted as 2×M elements of the systolic array per matrix multiplication cycle, and the output of the array can be 2×M 24-bit results per cycle.

[0038] Finally, in the case of 4-bit integer operands stored as an M×N matrix of elements, each element of the stored matrix can contain two independent 4-bit operands instead of a single 8-bit operand. As a result, when the elements are moved into the systolic array, each element can be interpreted as two element halves, whereby each element half is an independent 4-bit integer operand. Since the stored M×N matrix of elements is interpreted as having twice as many elements per row, each element half of the right-hand side data can be loaded into the corresponding row of the systolic array between rows 0 and 2M-1 of the array, and each element half of the left-hand side data can be passed through the corresponding row of the systolic array between rows 0 and 2M-1 of the array. In this case, the 4-bit integer input can be effectively interpreted as 2×2M elements of the systolic array per matrix multiplication cycle. Furthermore, the 4-bit integer input can be effectively interpreted as 2×2M elements per matrix multiplication cycle, and the output of the array can be 2×M 24-bit results per cycle.

[0039] The disclosed systolic array architecture offers the advantage of being able to handle multiple data types, including both integer and floating-point values ​​of varying sizes, while maintaining good throughput. Rather than providing completely different hardware for each data type, the same input and output buses can be used for different data types to feed different accumulators included in the systolic array. Ultimately, these advantages facilitate maintaining a highly efficient processor capable of handling many different data types without sacrificing excessive space for added hardware to support the different data types. Example System

[0040] Figure 1 is a diagram of an example systolic array 100 according to an aspect of the present disclosure. The systolic array 100 is shown having a plurality of cells 110 arranged in an [I, J] array having I rows and J columns.

[0041] Each of the cells 110 may be loaded with right-hand side data from the right-hand matrix register 120. The right-hand matrix register 120 may be shaped as an M×N matrix having M rows and N columns of data elements. Typically, each data element corresponds to a separate operand for a matrix multiplication operation, although in at least some cases of the present disclosure, each data element may contain multiple operands.

[0042] The first column [0] of unit 110 may also receive left-hand side data from left-hand vector register 130. Left-hand matrix register 130 may also be shaped as an M×N matrix having M rows and N columns of data elements. As with the right-hand side data, each data element corresponds to a separate operand for a matrix multiplication operation, although in at least some cases of the present disclosure, each data element may contain multiple operands.

[0043] In some examples, the systolic array can be fixed, meaning that the entire right-hand matrix register 120 is preloaded and remains fixed in the array during matrix multiplication. Alternatively, in other examples, a vector of right-hand side data can be loaded every cycle, making the systolic array non-fixed. The decision between a fixed array or a non-fixed array can be influenced by timing considerations within the systolic array, such as control flow direction and pipelining.

[0044] Each cell 110 of the systolic array 100 may be responsible for receiving a portion of the right-hand side data, receiving a portion of the left-hand side data, receiving an output from a previous cell in the same row, computing the product of the right-hand side data and the received portion of the left-hand side data, adding the computed product to the output from the previous cell, and passing the sum to the next cell in the same row. For example, in the case of a given cell [i, j] of the systolic array, the right-hand side data loaded into the received cell may correspond to column [n] of the right-hand matrix register 120, the left-hand side data passed through the cell may correspond to row [m] of the left-hand vector register 130, an output may be received from cell [i-1, j], the product of the received portions of [m] and [n] may be computed and added to the output from cell [i-1, j], and the sum may be forwarded to the next cell [i+1, j]. Ultimately, the computations performed by each cell in the cell 110 may result in a matrix multiplication result 140.

[0045] Figure 2A and Figure 2B Such as Figure 1 1. Block diagram of an example single cell 200 of cell 110 of systolic array 100. Cell 200 includes a processing element capable of receiving right-hand side data 210 and left-hand side data 220, as well as receiving calculated values ​​from previous cells along a row of the systolic array. Figure 2A and Figure 2BIn the example of FIG, unit 200 is shown as including one dot accumulator / adder (DPA) for floating-point values ​​(floating-point DPA 230), and two dot accumulator / adders (DPA) for integer values ​​(Int DPA_0 240 and Int DPA_1 250). Floating-point DPA 230 is used to process incoming floating-point operands, and Int DPAs 240, 250 are used to process incoming integer operands.

[0046] Figure 2A The example shows the data flow of incoming floating-point operands. Figure 2A In the example of FIG. 2 , the floating-point operands may be 16-bit operands or 8-bit operands. In other examples, the floating-point operands may be of different sizes. Unit 200 receives right-hand-side data 212 from column [n] of right-hand-side matrix 210. Additionally, in each cycle of the systolic array, unit 200 receives left-hand-side data 222 corresponding to row [m] of a vector of left-hand-side matrix 220 from a previous unit of the systolic array. The unit further receives results 252 from the previous unit [m, n-1], which are the results of the calculations performed on row [m] of the left-hand-side data and column [n-1] of the right-hand-side data, plus any previous results along row [m]. A calculation is performed in floating-point DPA 230, where the dot product of left-hand-side data 222 and right-hand-side data 212 is calculated and then added to the result 252 of the previous unit. In the next cycle, left-hand-side data 222 is then passed to the next unit along the row [m, n+1] along with the calculated result 262 of the calculation.

[0047] Figure 2B The example shows the data flow of the incoming integer operands. Figure 2BIn the example of FIG. 2 , the integer operands may be 8-bit operands or 4-bit operands. In other examples, the integer operands may be of different sizes. Unit 200 receives right-hand-side data 214, 216 from column [n] of right-hand-side matrix 210. Additionally, during each cycle of the systolic array, unit 200 receives two rows of left-hand-side data 224, 226 corresponding to rows [m] and [m+1] of two vectors of left-hand-side matrix 220 from a previous unit of the systolic array. The unit further receives results 254, 256 from the previous unit [m, n-1]. Result 254 is the result of the calculation performed on row [m] of the left-hand-side data and column [n-1] of the right-hand-side data, plus any previous results along row [m]. Result 256 is the result of the calculation performed on row [m+1] of the left-hand-side data and column [n-1] of the right-hand-side data, plus any previous results along row [m+1]. The calculation for each row [m] and [m+1] is performed in the respective integer DPA 240, 250, where the dot product of the respective left-hand side data 224, 226 and the respective right-hand side data 214, 216 is calculated and then added to the result 254, 256 of the respective previous cell [m, n-1] or [m+1, n-1]. In the next cycle, each portion of the left-hand side data 224, 226 is then passed to its respective next cell [m, n+1] or [m+1, n+1] along its respective row along with the respective calculated result 264, 266 of the calculation for this row.

[0048] from Figure 2B As can be seen, the 8-bit integer operations of unit 200 have double the throughput compared to 16-bit floating-point operations. Since the cost of transferring 16-bit data for a 16-bit floating-point operand is comparable to the cost of transferring 16-bit data for two 8-bit integer operands, double the throughput can be achieved efficiently. Additionally, although double the number of accumulated addition operations may need to be performed for two operands compared to one operand, unit 200 is equipped with twice the accumulators for 8-bit integer operands, so there is no loss in efficiency at the calculation stage.

[0049] Figure 2A and Figure 2B The unit can also handle 4-bit integers with high efficiency. This is achieved by including two independent data paths for input within the integer DPA. Figure 3is a block diagram of an example integer DPA 300 that includes a first datapath 310 for processing both 4-bit integer operands and 8-bit integer operands and a second datapath 320 for processing only 4-bit integer operands. Each datapath 310, 320 includes a respective partial product generation layer 312, 322, a respective carry-save adder tree layer 314, 324, and a respective reduction tree layer 316, 326 including one or more reduction trees. The components used in these layers 312, 314, 316, 322, 324, 326 can be any conventional components known in the art for performing dot-point accumulation on integer operands, such as 8-bit integers or 4-bit integers.

[0050] DPA 300 may further include Figure 3 3. Circuitry is shown in FIG. 3 as a "type-dependent split and expand" operation 330 for controlling whether input data 340 received from the right-hand matrix register and from the previous unit is maintained as one input or split into two inputs. In one example, when the data type is a first data type such as an 8-bit integer, the data may be maintained as one input, and when the data type is a second data type such as a 4-bit integer, the data may be split into two inputs.

[0051] The DPA 300 may further include another reduction tree 350 for combining the outputs of each of the two data paths. For cases where the input data 340 is not split, operations at the further reduction tree 350 may be skipped. The DPA may further include a carry-propagation adder 360 for adding the results from the previous cell to those of the current cell. A DPA result 370 may be output from the carry-propagation adder 360 and provided to the next cell along the corresponding row. In some examples, the further reduction tree 350 and carry-propagation adder 360 may be shared with the cell's floating-point DPA, rather than providing separate reduction trees and carry-propagation adders for floating-point and integer paths within the DPA. Alternatively, separate reduction trees and carry-propagation adders may be provided.

[0052] Figures 4 to 7 is a block diagram illustrating example data flows of different data types that may be received by a systolic array of the present disclosure. Figure 4 The example shows the data flow of 16-bit floating-point operands, Figure 5 The example shows the data flow of 8-bit floating-point operands, Figure 6 The example shows the data flow of 8-bit integer operands, Figure 7 The example shows the data flow for 4-bit integer operands.

[0053] exist Figure 4In the example of , left-hand side data 410 from a matrix having a shape of M×N is provided to a systolic array 420. The left-hand side data 410 includes a vector 412 of M 16-bit elements having a shape of 1×M, and a data type indicator 414 indicating the data type of the vector of elements. Figure 4 In the example shown in FIG. 4 , the element is indicated as "bf16," which is a type of 16-bit floating point value. In response, the systolic processor 422 of the systolic array 420 controls the flow of M elements into rows 0 through M-1 of the systolic array, such that each element occupies a corresponding row. During each cycle of the matrix multiplication operation, another vector 412 may be passed from the left-hand side data 410 to the systolic array 420.

[0054] exist Figure 5 In the example of FIG. 5 , left-hand data 510 from a matrix having a shape of M×N is provided to a systolic array 520. The left-hand data 510 includes a vector 512 of M 8-bit elements having a shape of 1×M and a data type indicator 514 indicating that the data type of the vector of elements is "fp8," which is a type of 8-bit floating-point value. In response, a systolic processor 522 of the systolic array 520 can control the conversion of the 8-bit elements to "bf16" 16-bit floating-point values ​​and then control the flow of the M converted elements to rows 0 through M-1 of the systolic array 520, whereby each element occupies a corresponding row. During each cycle of the matrix multiplication operation, another vector 512 can be converted and passed from the left-hand data 510 to the systolic array 520.

[0055] exist Figure 6 In the example of FIG. 5 , left-hand data 610 from a matrix having a shape of M×N is provided to a systolic array 620. The left-hand data 610 includes two vectors 612 and 614, each of which has M 8-bit elements with a shape of 1×M and a data type indicator 616 indicating that the data type of the vector of elements is "int8," which is a type of 8-bit integer value. In response, a systolic processor 622 of the systolic array 620 can control the flow of elements from each of the vectors to a corresponding row between rows 0 and M-1 of the systolic array 620, whereby each element occupies a corresponding row and each row receives two elements. The other two vectors 612 and 614 can be converted and passed from the left-hand data 610 to the systolic array 620 during each cycle of the matrix multiplication operation.

[0056] As from Figure 6As can be seen in the example, the ability of a systolic array to receive and process two vectors of int8 data in a single cycle allows a matrix multiplication operation on an entire matrix of int8 data to be completed twice as fast as a systolic array that receives only one vector at a time. Thus, the efficiency of systolic arrays can be maintained because the speed at which elements within the array are processed can correspond to the speed at which elements are streamed into the array.

[0057] Finally, in Figure 7 In the example of , left-hand data 710 from a matrix having a shape of M×N is provided to a systolic array 720. The left-hand data 710 includes two vectors 712 and 714, each of which has M 8-bit elements, and each element contains two operands, thus giving each vector of operands a shape of 1×2M. The left-hand data 710 also includes a data type indicator 716 indicating that the data type of the vector of elements is "int4," which is a type of 4-bit integer value. In response, the systolic processor 722 of the systolic array 720 can control the splitting of each element from an 8-bit element into two 4-bit elements, whereby the first half of the 8-bit elements from bits 0 to 3 constitutes the first element half corresponding to the first 4-bit operand, and the second half of the 8-bit elements from bits 4 to 7 constitutes the second element half corresponding to the second 4-bit operand. The systolic processor 722 then controls the flow of the element halves to the corresponding rows of the systolic array. For a first element half pointing to row [m] of the array, the corresponding second element half may point to the corresponding row [M+m], which may be physically co-located with row [m] in the same DPA of the systolic array element. In this manner, for each 1×M vector 712, 714 of elements of the left-hand data 710, operands are provided to 2M rows of the systolic array. As Figure 6 In the example of 8-bit integer operands, 4-bit integer operands can flow into the systolic array for two vectors per cycle of the matrix multiplication operation.

[0058] As from Figure 7 As can be seen in the example, the ability of a systolic array to receive and process two vectors of int4 data in a single cycle, and further process two rows of int4 data within a single cell of the systolic array, allows a matrix multiplication operation for an entire matrix of int4 data to be completed four times faster than in a systolic array that receives only one vector at a time and processes only one element per cell. Thus, the efficiency of the systolic array can be maintained because the speed at which elements within the array are processed can correspond to the speed at which elements are streamed into the array.

[0059] Although not in Figures 4 to 7Although not shown in FIG, the process of loading the right-hand side data into the systolic array can be comparable to the streaming process for the left-hand side data. For example, in the case of 16-bit floating-point values, 8-bit floating-point values, and 8-bit integer values, the right-hand side data can be loaded into rows 0 through M-1 of the systolic array. Further, in the case of 8-bit floating-point values, the right-hand side data can also be converted to the "bf16" 16-bit floating-point value format. As another example, in the case of 4-bit integer values, each element of the right-hand side data can be split into two element halves and loaded into rows 0 through 2M-1 of the systolic array, whereby the element half pairs are loaded into physically co-located rows, such as rows 0 and M, or rows 1 and M+1, or rows 2 and M+2, etc.

[0060] Figures 8A to 8C is provided in Figures 4 to 7 FIG. 1 is a diagram further visualizing the matrix multiplication operations performed in a given cycle of each of the example systolic arrays. Figure 8A The operation shown in Figure 4 and Figure 5 The example systolic arrays correspond to 16-bit floating-point operations performed. Figure 8B The operation shown in Figure 6 The example systolic arrays perform 8-bit integer operations. Figure 8C The operation shown in Figure 7 The example systolic arrays correspond to 4-bit integer operations performed.

[0061] exist Figure 8A In the example of FIG, a 1×M vector of bf16 elements from the left-hand side data 812 is multiplied by an M×N matrix of bf16 elements from the right-hand side data 814. The bf16 elements may begin as 16-bit floating point values, or as 8-bit floating point values ​​that are converted to 16-bit floating point values, such as by using a data type conversion instruction such as roundTiesToEven. Converting the 8-bit values ​​to 16-bit values ​​can further utilize a holding register of size M×N to store the right-hand side data for loading into the right-hand matrix register. In such an example, the 8-bit floating point may be an unsupported data type for the right-hand matrix register, but may be a supported data type for the holding register. Therefore, moving the 8-bit floating point value from the vector register to the holding register may facilitate the conversion of the 8-bit value to the 16-bit value when the data is moved from the holding register to the right-hand matrix register.

[0062] In either case where the data type indicator indicates an 8-bit or 16-bit floating point value, the result of the matrix multiplication operation is a 1×M vector of 32-bit floating point operands, which can be added to the 1×M vector 816 resulting from the calculation in the previous element of the systolic array to obtain the final output result 818 for the cycle, which itself is a 1×M vector of 32-bit floating point operands. The output can be in 32-bit floating point format to facilitate accumulation operations.

[0063] exist Figure 8B In the example of FIG, two 1×M vectors of int8 elements from the left-hand side data 822 are multiplied by an M×N matrix of int8 elements from the right-hand side data 824. The multiplication operation for the two vectors can occur in parallel at each corresponding unit using the two integer DPAs of the corresponding unit. The result of the parallel matrix multiplication operation is a 2×M vector of 24-bit integer operands, which can be added to the 2×M vector 826 resulting from the calculation in the previous unit of the systolic array to obtain the final output result 828 of the array for that cycle, which itself is a 2×M vector of 24-bit integer operands.

[0064] exist Figure 8C In the example of FIG, two 1×M vectors of 8-bit integer elements from the left-hand data are split into two vectors of 1×2M 4-bit integer operands 832, and an M×N matrix of 8-bit integer elements from the right-hand data is split into a 2M×N matrix of 4-bit integer operands 834 and loaded into the systolic array. The multiplication operation for the two vectors can occur in parallel at each corresponding element using the two integer DPAs of the corresponding elements. Additionally, at each integer DPA, physically co-located rows can receive corresponding portions of the left-hand data. The result of the parallel matrix multiplication operation is a 2×2M vector of 24-bit integer operands, which can be added to the 2×2M vector 836 derived from the calculation in the previous element of the systolic array to obtain the final output result 838 of the array for that cycle, which itself is a 2×2M vector of 24-bit integer operands.

[0065] Figure 9 A block diagram of an example electronic device 900 for implementing a systolic array according to any of the example embodiments of the present disclosure is depicted. The electronic device 900 may include one or more processors 910, such as one or more CPUs, a system memory 920, a bus 930, a networking interface 940, and other components (not shown), such as storage, output device interfaces, and input device interfaces. The bus 930 may be used to communicate between the processor 910, the system memory 920, the networking interface 940, and the other components. Any or all components of the electronic device 900 may be used in conjunction with the subject matter of the present disclosure.

[0066] Depending on the desired configuration, the processor 910 can be of any type, including but not limited to one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs) and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs), or any combination thereof. The processor 910 may include a systolic array. The processor 910 may include one or more levels of cache, such as a level 1 cache 911 and a level 2 cache 912, a processor core 913, and registers 914. The processor core 913 may include one or more arithmetic logic units (ALUs), one or more floating point units (FPUs), one or more DSP cores, or any combination thereof. A memory controller 915 may also be used with the processor 910, or in some implementations, the memory controller 915 may be an internal part of the processor 910.

[0067] Depending on the desired configuration, physical memory 920 can be of any type, including but not limited to volatile memory such as RAM, non-volatile memory such as ROM, flash memory, etc., or any combination thereof. Physical memory 920 can include an operating system 921, one or more applications 922, and program data 924, which can include service data 925. Non-transitory computer-readable medium program data 924 can include stored instructions that, when executed by one or more processing devices, implement process 923 for calculating the result of a multiplication and accumulation operation. In some examples, one or more applications 922 can be arranged to operate on operating system 921 using program data 924 and service data 925.

[0068] The electronic device 900 may have additional features or functionality, as well as additional interfaces, to facilitate communication between the basic configuration 901 and any desired devices and interfaces.

[0069] Physical memory 920 may be an example of computer storage media. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, or any other media that can be used to store the desired information and can be accessed by electronic device 900. Any such computer storage media may be part of device 900.

[0070] The network interface 940 can couple the electronic device 900 to a network (not shown) and / or another electronic device (not shown). In this way, the electronic device 900 can be part of a network of electronic devices, such as a local area network ("LAN"), a wide area network ("WAN"), an intranet, or a network within a network such as the Internet. In some examples, the electronic device 900 can include a network connection interface for forming a network connection to the network and a local communication connection interface for forming a tethering connection with another device. The connection can be wired or wireless. The electronic device 900 can bridge the network connection and the tethering connection to connect other devices to the network via the network interface 940.

[0071] The systolic array may include multiple MAC units 950 to perform the multiplication and accumulation operations required for matrix multiplication. The MAC units 950 and the systolic array in which the MAC units operate may be used in an accelerator that can be used for DNN implementation.

[0072] The electronic device 900 can be implemented as part of a small form factor portable (or mobile) electronic device, such as a speaker, earphones, earbuds, a cellular phone, a smart phone, a smart watch, a personal data assistant (PDA), a personal media player device, a tablet computer (tablet), a wireless network watch device, a personal head-mounted device, a wearable device, a dedicated device, or a hybrid device including any of the above functions. The electronic device 900 can also be implemented as a personal computer including both laptop and non-laptop computer configurations. The electronic device 900 can also be implemented as a server, an accelerator, or a large-scale system. Example Method

[0073] Figure 10 is a flow chart illustrating an example routine 1000 for controlling data flow through a systolic array of the present disclosure. The routine 1000 may be executed by a program such as Figures 4 to 7 The systolic processors shown include one or more systolic processors executing in the system.

[0074] Operations may begin at block 1010, where a systolic processor receives a data type indicator indicating a data type of input data. The input data may include left-hand side data and right-hand side data for matrix multiplication within a systolic array. In some examples, the data type indicator may be a flag. In some examples, the flag may include one or more bits appended to the bits of an element.

[0075] At block 1020, the systolic processor may determine the data type of the input data based on the data type indicator. For example, the data type may be any one of 16-bit floating point, 8-bit floating point, 8-bit integer, 4-bit integer, etc. Figure 10 In the example of , classification between two data types is shown, where the first data type is handled according to a first set of operations and the second data type is handled according to a second set of operations.

[0076] If the data type corresponds to a first data type corresponding to any one of 16-bit floating point, 8-bit floating point, or 8-bit integer formats, operations may continue at block 1030, where the right-hand side data is loaded between rows 0 and M-1 of the systolic array, and then continue at block 1040, where each corresponding row of the left-hand side data is passed through the corresponding row between rows 0 and M-1 of the systolic array. The left-hand side data may be passed one vector per cycle.

[0077] Alternatively, if the data type corresponds to a second data type corresponding to a 4-bit integer format, operations may continue at block 1050 where each element of the left-hand side data and the right-hand side data is split into respective first element halves and second element halves, and then continue at blocks 1060 and 170 where the right-hand side data is loaded between rows 0 and 2M−1 of the systolic array, where each first element half travels between rows 0 and M−1 and each second element half travels to a corresponding row between rows M and 2M−1, and the left-hand side data is passed through rows 0 to 2M−1 of the systolic array, where each first element half is passed between rows 0 and M−1 and each second element half is passed to a corresponding row between rows M and 2M−1.

[0078] In further examples, the systolic processor can further distinguish between additional data types and further control different data flows for different data types. For example, the data type indicator can distinguish between 16-bit floating point values ​​and 8-bit floating point values, whereby the 8-bit floating point indicator can signal the systolic processor to convert the incoming left-hand operand and right-hand operand from 8-bit floating point format to 16-bit floating point format. Figure 11 An example arrangement for a systolic processor to convert 8-bit floating point values ​​to 16-bit floating point values ​​is shown in FIG. Figure 11The arrangement 1100 shows that four registers 1110, 1120, 1130, 1140 are used to prepare right-hand side operands and left-hand side operands to be loaded into and passed through the systolic array 1150, respectively. The left-hand side data is first stored in a first matrix staging register (MSR_A) 1110, which is capable of supporting operands in 8-bit floating point format. Next, the left-hand side data is moved from the first matrix staging register 1110 to the left-hand side matrix register 1120 to be loaded into the systolic array. Since the systolic array does not support matrix multiplication operations for 8-bit floating point data, the 8-bit floating point operands can be converted from 8-bit floating point format to 16-bit floating point format during the transfer from the first matrix staging register 1110 to the left-hand matrix register 1120. Similar operations can be performed on the right-hand side data using the second matrix staging register 1130 and the right-hand matrix register 1140, respectively. Additionally, although Figure 11 The first matrix scratch register 1110 is shown connected to the left-hand matrix register 1120, and the second matrix scratch register 1130 is shown connected to the right-hand matrix register 1140. However, it should be understood that each of the matrix scratch registers 1110 and 1130 can be interchangeably connected to either the left-hand matrix register 1120 or the right-hand matrix register 1140, such that each matrix scratch register 1110 and 1130 can be used to store either left-hand data or right-hand data. In addition to the data format conversion benefits provided by the four registers, it should be noted that the four registers can provide additional functionality to the systolic processor. For example, data transferred to the matrix scratch registers can be transposed upon transfer, meaning that both left-hand data and right-hand data can be provided in transposed form within the systolic processor for matrix multiplication operations without having to rely on a separate transposition unit remote from the systolic processor.

[0079] Additionally or alternatively, the data type indicator can distinguish between floating-point values ​​and integer values, whereby incoming floating-point values ​​and integers can be directed to separate DPAs for processing within the cells of the systolic array. In some examples, moving integer values ​​to the DPAs can involve each cell receiving data from two vectors of left-hand data during each cycle of a matrix multiplication operation and moving the received data to different integer DPAs within the cell for initial parallel processing and subsequent combined processing.

[0080] In an example embodiment of the present disclosure, a systolic array may be used as a matrix multiplication unit (MXU) within a tensor processing unit (TPU). For example, a TPU may include one or more core processors, and each core processor may be connected to one or more MXUs. The MXU may be an inner product strided systolic array processor. Depending on the type of data received, the MXU may be further configurable between 128×128 and 256×128 shapes. The MXU may support 2×128 24-bit results per cycle for both 8-bit integer operands and 4-bit integer operands, and 1×128 32-bit floating-point results per cycle for 16-bit floating-point operands and 8-bit floating-point operands.

[0081] The exemplary MXUs of the present disclosure, and systolic arrays more generally, provide increased versatility and efficiency for matrix multiplication operations by supporting several types of input data using common input and output lines. This helps reduce the overall footprint of TPUs and other chips incorporating MXUs and systolic arrays without sacrificing processing efficiency.

[0082] Although the technology herein has been described with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the technology. Therefore, it should be understood that various modifications may be made to the illustrative embodiments, and that other arrangements may be devised without departing from the spirit and scope of the technology as defined by the appended claims.

[0083] Most of the aforementioned alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be made in an illustrative manner rather than in a manner that limits the subject matter defined by the claims. As an example, the foregoing operations do not have to be performed in the exact order described above. Instead, the various steps can be handled in a different order, such as in the opposite order, or simultaneously. Unless otherwise stated, steps may also be omitted. In addition, the examples described herein and the provision of clauses expressed as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; on the contrary, these examples are intended to illustrate only one of many possible embodiments. Further, the same figure numbers in different figures may identify the same or similar elements.

Claims

1. A system for performing matrix multiplication of input data comprising left-hand side data and right-hand side data, comprising: a right-hand matrix register having a size of M×N; a systolic array of data processing units configurable between a first size M×N and a second size 2M×N; as well as A systolic processor configured to: receiving a data type indicator, the data type indicator indicating a data type of the input data; (i) in response to the data type indicator indicating a first data type: loading the right-hand side data from the right-hand matrix register into the data processing unit between rows 0 and M-1; as well as passing each respective row of the left-hand side data through a corresponding row of the systolic array between rows 0 and M-1; (ii) in response to the data type indicator indicating a second data type: splitting each element of the left-hand data and the right-hand data into a corresponding first element half and a corresponding second element half; loading each first element half from said right hand matrix register into said data processing unit between rows 0 and M-1; as well as loading each second element half from said right hand matrix register into said data processing unit between rows M and 2M-1; as well as For each corresponding row of the left-hand side data: passing the first element half of the respective row of the left-hand side data through the corresponding row of the data processing units between rows 0 and M-1; and The second element half of the respective row of the left-hand side data is passed through the corresponding row of the data processing units between rows M and 2M-1.

2. The system of claim 1, wherein: The first data type comprises at least one of an 8-bit integer, an 8-bit floating point, or a 16-bit floating point, and wherein the second data type comprises a 4-bit integer.

3. The system of claim 2, wherein: The systolic processor is configured to: In response to the data type indicator indicating 16-bit floating point or 8-bit floating point, delivering a vector of elements of the left-hand side data having a shape of 1*M in each matrix multiplication cycle; as well as In response to the data type indicator indicating an 8-bit integer, passing a vector of elements of the left-hand side data having a shape of 2*M in each matrix multiplication cycle; as well as In response to the data type indicator indicating a 4-bit integer, a vector of elements of the left-hand side data having a shape of 2*2M is delivered in each matrix multiplication cycle.

4. The system of claim 2, further comprising: One or more 16-bit floating-point multiply-add chains; Two additional 8-bit integer multiply-add chains for each 16-bit floating-point multiply-add chain; as well as Two additional 4-bit multiply-add chains for each 16-bit floating-point multiply-add chain, Wherein, the systolic processor is configured to: Using the 16-bit floating point multiply-add chain to process 8-bit floating point and 16-bit floating point data; Processing 8-bit integer data using the two 8-bit integer multiply-add chains; and The two 8-bit integer multiply-add chains and the two 4-bit integer multiply-add chains are used to process 4-bit integer data.

5. The system of claim 4, wherein: The systolic processor is configured to produce 2×M 24-bit results per cycle for 8-bit and 4-bit integer inputs and 1×M 32-bit results per cycle for 8-bit and 16-bit floating-point inputs.

6. The system of claim 1 , further comprising a holding register having a size of M×N and configured to provide the right-hand side data to the right-hand matrix register, wherein The holding registers are configured to contain at least one data type not supported by the systolic array, and wherein the systolic processor is configured to convert right-hand side data of the unsupported data type to right-hand side data of a supported data type when the right-hand side data is provided from the holding registers to the right-hand matrix register.

7. The system of claim 6, wherein: The unsupported data type is an 8-bit floating point data type, and wherein the systolic processor is configured to convert right-hand side data of the unsupported data type contained in the holding register to the 16-bit floating point data type when the right-hand side data is provided from the holding register to the right-hand matrix register.

8. The system of claim 1, wherein: The data processing unit of the systolic array comprises: a floating point accumulation adder configured to process the left-hand side data and the right-hand side data having a floating point data type; and A plurality of integer dot accumulators and adders are configured to process the left-hand side data and the right-hand side data having integer data types.

9. The system of claim 8, wherein: The systolic processor is configured to: In response to the data type indicator indicating the floating point data type, passing data from one vector of the left-hand side data to the floating point accumulation adder during each matrix multiplication cycle, and In response to the data type indicator indicating the integer data type, data from two vectors of the left-hand side data are passed to the plurality of integer-point accumulators during each matrix multiplication cycle, wherein each integer-point accumulator receives data from a corresponding vector of the left-hand side data.

10. The system of claim 8, wherein: Each integer point accumulation adder further includes an independent first data path and an independent second data path, each data path including a corresponding partial product generation layer, a corresponding carry-save adder tree layer, and a corresponding reduction tree layer.

11. The system of claim 10, wherein: The systolic processor is configured to: responsive to the data type indicator indicating an 8-bit integer data type, passing data from the left-hand side data to only the first data path of each integer dot accumulator during each matrix multiplication cycle, and In response to the data type indicator indicating a 4-bit integer data type, data from the left-hand side data is passed to both the first data path and the second data path of each integer-point accumulator during each matrix multiplication cycle.

12. The system of claim 11, wherein: For each element of the left-hand side data, the systolic processor is configured to: passing the first element half to the first data path of a corresponding one of the integer dot accumulators, and The second element is passed to the second data path of a second corresponding one of the integer point accumulators.

13. An accelerator hardware unit comprising the system of claim 1, wherein: The accelerator hardware unit is one of a graphics processing unit or a tensor processing unit.

14. The accelerator hardware unit of claim 13, comprising a plurality of matrix multiplication units, wherein: At least one of the matrix multiplication units comprises the system of claim 1 .

15. A method for performing matrix multiplication in a systolic array of data processing units configurable between a first size of M x N and a second size of 2M x N, the method comprising: receiving, by one or more processors, a data type indicator indicating a data type of input data for the matrix multiplication, wherein the input data includes left-hand side data and right-hand side data; (i) in response to the data type of the data type indicator indicating the first data type: loading, by the one or more processors, the right-hand side data from a right-hand matrix register having size M×N into the data processing unit of the systolic array between row 0 and row M−1; and for each corresponding row of the left-hand side data, passing, by the one or more processors, the corresponding row of the left-hand side data through a corresponding row of the data processing elements of the systolic array between row 0 and row M-1 to obtain a matrix multiplication result of the left-hand side data and the right-hand side data; (ii) in response to the data type of the data type indicator indicating the second data type: splitting, by the one or more processors, each element of the left-hand data and the right-hand data into a respective first element half and a respective second element half; loading, by the one or more processors, each first element half from the right hand matrix register into the data processing unit of the systolic array between row 0 and row M-1; and loading, by the one or more processors, each second element half from the right hand matrix register into the data processing unit of the systolic array between row M and row 2M-1; For each corresponding row of the left-hand side data: passing, by the one or more processors, the first element half of the respective row of the left-hand side data through corresponding rows of the data processing elements of the systolic array between row 0 and row M-1; and passing, by the one or more processors, the second element half of the respective row of the left-hand side data through the corresponding row of the data processing elements of the systolic array between row M and row 2M-1, To obtain the matrix multiplication result of the left-hand side data and the right-hand side data.

16. The method of claim 15, wherein: The first data type comprises at least one of a 16-bit floating point, an 8-bit floating point, or an 8-bit integer, and wherein the second data type comprises a 4-bit integer, and wherein: Passing the left-hand side data in response to the data type indicator indicating the first data type comprises passing one or more vectors of elements of the left-hand side data to only rows 0 through M-1 of the systolic array during each matrix multiplication cycle; and Passing the left-hand side data in response to the data type indicator indicating the second data type includes passing one or more vectors of elements of the left-hand side data to all rows between 0 and 2M-1 of the systolic array during each matrix multiplication cycle.

17. The method of claim 16, wherein: Passing the left-hand side data in response to the data type indicator indicating the second data type includes passing two vectors of elements of the left-hand side data to a plurality of integer point accumulators included in each cell of the systolic array during each matrix multiplication cycle, wherein each cell includes two integer point accumulators, and wherein each pair of corresponding rows corresponds to a pair of independent data paths within a corresponding one of the plurality of integer point accumulators.

18. The method of claim 15, wherein: The data type indicator further distinguishes between integer data types and floating point data types, and wherein: Passing the left-hand side data in response to the data type indicator indicating the floating point data type includes passing one vector of elements of the left-hand side data to the systolic array during each matrix multiplication cycle, and Passing the left-hand side data in response to the data type indicator indicating the integer data type includes passing two vectors of elements of the left-hand side data to the systolic array during each matrix multiplication cycle.

19. The method of claim 15, further comprising: storing the right-hand side data in a holding register having a size of M×N, the right-hand side data being a data type not supported by the systolic array; The right-hand side data is loaded from the holding register into the right-hand matrix register, wherein the loading comprises converting, by the one or more processors, the right-hand side data to a data type supported by the systolic array, wherein the data type not supported by the systolic array is 8-bit floating point, and wherein the data type supported by the systolic array is 16-bit floating point.

20. The method of claim 15, wherein: The left-hand side data and the right-hand side data are received as 128×128 matrices, where M=128 and N=128, and wherein the method further comprises: For data inputs comprising 8-bit or 16-bit floating-point operands, generating 1×128 32-bit floating-point results for each cycle of the systolic array, and For data inputs comprising 4-bit or 8-bit integer operands, 2 x 128 24-bit results are generated for each cycle of the systolic array.

Images