Systolic array cell having multiple accumulators
The integration of multiple accumulators and a controller circuit in systolic array cells addresses inefficiencies in matrix multiplication, especially for sparse matrices, enhancing computation efficiency and reducing resource usage.
Patent Information
- Application Number
- JP2022570187
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-11-30
- Filing Date
- 2021-11-30
- Publication Date
- 2024-11-11
- Estimated Expiration
- 2041-11-30
AI Technical Summary
Existing systolic arrays struggle to efficiently perform matrix multiplication, particularly with sparse matrices, requiring significant hardware adjustments and resource inefficiencies.
Incorporation of multiple accumulators and a controller circuit within systolic array cells to dynamically handle different sparsity patterns, allowing flexible and efficient matrix computations without hardware adjustments.
Enhances matrix computation efficiency, particularly for sparse matrices, reducing computational resources and accelerating training and computation in machine learning applications.
Smart Images

Figure 0007580491000001 
Figure 0007580491000002 
Figure 0007580491000003
Abstract
Description
[Technical field]
[0001] REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority under 35 U.S.C. § 119 to U.S. Provisional Application No. 63 / 119,556, filed November 30, 2020, which is incorporated by reference herein in its entirety.
[0002] Technical Field This specification relates to a systolic array of hardware processing units. [Background technology]
[0003] background A systolic array is a network of processing units that computes data and passes it through a network. Data in a systolic array flows in a pipelined manner between the processing units, and each processing unit may independently compute a partial result based on data received from its upstream neighboring processing unit. The processing units, which may also be referred to as cells, may be wired together to pass data from upstream processing units to downstream processing units. Systolic arrays are used in machine learning applications, for example, to perform matrix multiplication. Summary of the Invention
[0004] overview In general, one innovative aspect of the subject matter described herein may be embodied in a matrix computation unit including a plurality of cells arranged in a systolic array, each cell including a multiplier circuit configured to multiply elements or sub-matrices of an input matrix, an adder circuit configured to sum an input accumulation value and a product output by the multiplier circuit, a plurality of accumulators connected to an output of the adder circuit, and a controller circuit configured to select a given accumulator from the plurality of accumulators for receiving the sum output by the adder circuit.
[0005] These and other implementations may each optionally include one or more of the following features: In some aspects, the controller circuit is configured to select a given accumulator for each of a plurality of products determined by the multiplication circuit based on selector data received by the cell.
[0006] In some aspects, each cell includes a first input register configured to receive a first submatrix and a second input register configured to receive a second submatrix, and the products determined by the multiplication circuitry include products of the first and second submatrix. Each cell may further include one or more selector registers configured to receive selector data. The controller circuitry may be configured to select a given accumulator for each of the multiple products determined by the multiplication circuitry based on the selector data.
[0007] In some aspects, the selector data may include data defining a sparsity pattern for a first submatrix indicating locations of non-zero elements in the first submatrix, and the selector data may include data defining a sparsity pattern for a second submatrix indicating locations of non-zero elements in the second submatrix.
[0008] In some aspects, the selector data may indicate a first sub-multiplication to which the first sub-matrix belongs. The selector data may indicate a second sub-multiplication to which the second sub-matrix belongs. When the first sub-multiplication matches the second sub-multiplication, the controller circuit may be configured to select a given accumulator corresponding to the first sub-multiplication and the second sub-multiplication. When the first sub-multiplication does not match the second sub-multiplication, the controller may be configured to disable write inputs to all of the multiple accumulators.
[0009] In some aspects, each accumulator accumulates values output by a summation circuit for a given set of input matrices.
[0010] In general, another innovative aspect of the subject matter described herein may be embodied in a data processing cell that may include a multiplier circuit configured to determine products of sub-matrices of an input matrix, an adder circuit configured to determine a sum of an input accumulated value and a product output by the multiplier circuit, a plurality of accumulators connected to an output of the adder circuit, and a controller circuit configured to select a given accumulator from the plurality of accumulators to receive the sum output by the adder circuit.
[0011] These and other implementations may each optionally include one or more of the following features: In some aspects, the controller circuitry is configured to select a given accumulator for each of a plurality of products determined by the multiplication circuitry based on selector data received by the data processing cell.
[0012] In some aspects, the data processing cell includes a first input register configured to receive a first submatrix and a second input register configured to receive a second submatrix. The products determined by the multiplication circuit include products of the first and second submatrix. The data processing cell may include one or more selector registers configured to receive selector data. The controller circuit may be configured to select a given accumulator for each of the multiple products determined by the multiplication circuit based on the selector data.
[0013] In some aspects, the selector data includes data defining a sparsity pattern for a first submatrix that indicates locations of non-zero elements in the first submatrix. The selector data may include data defining a sparsity pattern for a second submatrix that indicates locations of non-zero elements in the second submatrix.
[0014] In some aspects, the selector data indicates a first sub-multiplication to which the first sub-matrix belongs. The selector data may indicate a second sub-multiplication to which the second sub-matrix belongs. When the first sub-multiplication matches the second sub-multiplication, the controller may be configured to select a given accumulator corresponding to the first sub-multiplication and the second sub-multiplication. When the first sub-multiplication does not match the second sub-multiplication, the controller may be configured to disable write inputs to all of the multiple accumulators.
[0015] In some aspects, each accumulator of the plurality of accumulators accumulates values output by the summation circuit for a given set of input matrices.
[0016] These and other implementations may each optionally include one or more of the following features: In some aspects, a method for multiplying matrices includes a first input register of a cell receiving a first input submatrix, a second input register of the cell receiving a second input submatrix, a controller of the cell selecting a given accumulator from a plurality of accumulators of the cell to receive a sum of (i) a product of the first input submatrix and the second input submatrix and (ii) a current accumulated value of the given accumulator, a multiplication circuit of the cell generating a product of the first input matrix and the second input matrix, an adder circuit of the cell generating an updated accumulated value by adding the product of the first input matrix and the second input matrix to the current accumulated value, and storing the updated accumulated value in the given accumulator.
[0017] These and other implementations may each optionally include one or more of the following features. In some aspects, the product determined by the multiplication circuit includes a product of a first submatrix and a second submatrix. Some aspects include one or more selector registers of the cell receiving selector data. Selecting the given accumulator may include selecting the given accumulator based on the selector data.
[0018] In some aspects, the selector data includes data defining a sparsity pattern for a first input submatrix indicating locations of non-zero elements in the first submatrix, and the selector data includes data defining a sparsity pattern for a second input submatrix indicating locations of non-zero elements in the second submatrix.
[0019] In some aspects, the selector data indicates a first sub-multiplication to which the first input sub-matrix belongs. The selector data may indicate a second sub-multiplication to which the second input sub-matrix belongs. If the first sub-multiplication matches the second sub-multiplication, the controller may select a given accumulator corresponding to the first sub-multiplication and the second sub-multiplication. If the first sub-multiplication does not match the second sub-multiplication, the controller disables the write input to all of the multiple accumulators.
[0020] In some aspects, each accumulator of the plurality of accumulators accumulates values output by the summation circuit for a given set of input matrices.
[0021] The subject matter described herein may be implemented in certain embodiments to achieve one or more of the following advantages: The systolic array cells described herein may include multiple accumulators and a controller circuit that enables the cells to perform a variety of different matrix multiplication calculations. This provides additional flexibility within the systolic array and increases the efficiency of matrix calculations while using less hardware. For example, the use of the controller circuit and multiple accumulators may enable operations performed on sparse matrices to be performed faster and more efficiently than performing those operations directly on dense matrices. The controller circuit and multiple accumulators also enable the cells to perform matrix calculations on different sparsity patterns, for example, "1 out of n" patterns with submatrices and tile sharing.
[0022] Various features and advantages of the foregoing subject matter are described below with reference to the drawings. Further features and advantages will be apparent from the subject matter described herein and from the claims. [Brief description of the drawings]
[0023] [Figure 1] 1 illustrates an exemplary processing system including a matrix calculation unit. [Diagram 2] 1 illustrates an exemplary architecture including a matrix computation unit. [Diagram 3] 1 illustrates an exemplary architecture of a cell in a systolic array. [Figure 4] FIG. 1 is a flow diagram of an example process for performing matrix multiplication. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description Generally, this document describes a systolic array of cells that includes multiple accumulators. The cells may include computation units, such as multiplication and / or addition circuits, for performing computations. For example, the systolic array may perform matrix-matrix multiplication on input matrices, with each cell determining a partial matrix product of a portion of each input matrix. The systolic array of cells may be part of a matrix computation unit of a processing system, such as a dedicated machine learning processor used to train machine learning models and / or perform machine learning computations, a graphics processing unit (GPU), or another suitable processing system that performs matrix multiplication.
[0025] The systolic array may implement an output stationary matrix multiplication technique in which each cell computes a partial sum of products of some of the elements of the input matrices. In the output stationary technique, the elements of the input matrices may be shifted in opposite or orthogonal directions across the rows or across the columns of the systolic array. Each time a cell receives two sub-matrices, it computes the products of the sub-matrices and accumulates all the partial sums of the products computed by that cell for its portions of the two input sub-matrices.
[0026] A systolic array cell may include a controller, e.g., a control circuit, and multiple accumulators that enable the systolic array to support various matrix operations, such as operations on different matrices with different sparsity patterns. The sparsity pattern indicates the number of non-zero elements in a matrix and may be expressed as an "x out of y" sparsity pattern, where x is the maximum number of non-zero elements and y is the total number of elements. For example, a "1 out of 4" sparsity pattern may indicate that a matrix contains four elements, at most one of the elements being non-zero. The controller may control in which accumulator a product is accumulated based on selector data received by the cell. For example, the selector data may include sparsity data of a submatrix and data identifying a non-zero element in the submatrix. Based on this data, the controller may enable one of the accumulators to accumulate a product of a non-zero element with another matrix element.
[0027] The systolic array is adapted to more efficiently handle sparse matrices when training machine learning models and performing machine learning calculations, resulting in faster training and calculations using fewer computational resources than performing the same or similar calculations directly on dense matrices. The inclusion of multiple accumulators and control circuits provides the flexibility to dynamically and efficiently handle matrices with different sparsity patterns without having to adjust the hardware of the systolic array. Instead, the control circuitry and control inputs can be used to select the appropriate accumulator for each calculation based on the sparsity pattern of the input matrix, which provides the dynamic flexibility to more efficiently handle different sparsity patterns.
[0028] 1 illustrates an exemplary processing system 100 that includes a matrix computation unit 112. System 100 is one example of a system that may implement matrix computation unit 112 with a systolic array of cells having multiple accumulators.
[0029] System 100 includes a processor 102 that may include one or more computational cores 103. Each computational core 103 may include a matrix computation unit 112 that may be used to perform matrix-matrix multiplication using a systolic array of cells having multiple accumulators. System 100 may be in the form of a dedicated hardware chip.
[0030] In some implementations, the computation core 103 or another component thereof may send matrices to the matrix computation unit 112 along with control information. The control information may define the operations to be performed by the matrix computation unit 112. The control information may also define or otherwise control the data flow through the systolic array of the matrix computation unit 112. For example, the control information may define whether individual elements or sub-matrices of each input matrix should be shifted through the systolic array. In the case of sub-matrices, the control information may define the dimensions of the sub-matrix, e.g., 2×2, 2×4, the sparsity pattern of the sub-matrix, if appropriate, and / or the non-zero elements of each sub-matrix. A sub-matrix having a single element, e.g., a 1×1 sub-matrix, that is part of a larger input matrix may also be referred to as a matrix element. Information defining the sparsity pattern and non-zero elements for each sub-matrix may be shifted through the systolic array, e.g., along with the sub-matrix, as described in more detail below.
[0031] Each matrix computation unit 112 may be used to perform matrix multiplication calculations during training or use of a machine learning model. For example, matrix multiplication is a common calculation performed during training and use of a deep learning model, such as a deep neural network model. The systolic array of the matrix computation unit 112 is adapted to more efficiently handle sparse matrices when training machine learning models and performing machine learning calculations, resulting in faster training and calculations using fewer computational resources than performing the same or similar calculations directly on dense matrices. When aggregated across many matrix calculations of a deep learning model, this results in substantial performance improvements.
[0032] FIG. 2 illustrates an exemplary architecture including a matrix computation unit. The matrix computation unit is a two-dimensional systolic array 206. The two-dimensional systolic array 206 may be a square array. The array 206 includes a plurality of cells 204. In some implementations, a first dimension 220 of the systolic array 206 corresponds to a column of cells, and a second dimension 222 of the systolic array 206 corresponds to a row of cells. The systolic array 206 may have more rows than columns, more columns than rows, or an equal number of columns and rows. Thus, the systolic array 206 may have a shape other than a square. The matrix computation unit 112 of FIG. 1 may be implemented as a systolic array 206.
[0033] The systolic array 206 may be used for matrix multiplication or other calculations, such as convolution, correlation, or data sorting. For example, the systolic array 206 may be used for neural network calculations.
[0034] The systolic array 206 includes a value loader 202 and a value loader 208. The value loader 202 may send a submatrix to the rows of the array 206, and the value loader 208 may send a submatrix to the columns of the array. However, in some other implementations, the value loaders 202 and 208 may send a submatrix to both sides of the columns of the systolic array 206. In another example, the value loader 202 may send a submatrix across the rows of the systolic array 206, and the value loader may send a submatrix across the columns of the systolic array 206, or vice versa. In an example of a neural network, the value loader 202 may send activation inputs to the rows (or columns) of the array 206, and the value loader 208 may send weight inputs to the rows (or columns) of the array 206 from the opposite (or orthogonal) side of the value loader 202. In yet another example, value loader 202 may send activation inputs diagonally across array 206, and value loader 208 may send weight inputs diagonally across array 206, e.g., in the opposite direction to that of value loader 202 or in a direction perpendicular to the direction of value loader 202.
[0035] The value loaders 202 may receive sub-matrices from a unified buffer or other suitable source. Each value loader 202 may send a corresponding sub-matrix to a separate left-most cell of the array 206. The left-most cell may be a cell along the left-most column of the array 206. For example, value loader 202A may send a sub-matrix to cell 214. Value loader 202A may also send the sub-matrix to an adjacent value loader, which may be used in another left-most cell of the array 206. This allows a sub-matrix to be shifted for use in another particular cell of the array 206.
[0036] The value loaders 208 may also receive sub-matrices from a unified buffer or other suitable source. Each value loader 208 may send a corresponding sub-matrix to a separate topmost cell of the array 206. The topmost cell may be a cell along the topmost row of the array 206. For example, value loader 208A may send a sub-matrix to cell 214. Value loader 208A may also send that sub-matrix to an adjacent value loader, which may be used in another topmost cell of the array 206. This allows a sub-matrix to be shifted for use in another particular cell of the array 206.
[0037] In some implementations, the host interface shifts sub-matrices (e.g., activation inputs) along one dimension, e.g., to the right, across array 206 while shifting sub-matrices (e.g., weight inputs) along an orthogonal dimension, e.g., down, across array 206. For example, over one clock cycle, the sub-matrix (e.g., activation inputs) in cell 214 may shift into a register in cell 215 to the right of cell 214. Similarly, the sub-matrix (e.g., weight inputs) in cell 214 may shift into a register in cell 218 below cell 215. In other examples, the weight inputs may be shifted in a direction opposite to that of the activation inputs (e.g., from right to left).
[0038] Value loaders 202 and 208 may also send selector data with each submatrix they send to array 206. When used in sparse matrix applications, the selector data may include sparsity data that defines the sparsity pattern of the submatrix. In such applications, only one of the elements of the submatrix may have a nonzero value. The sparsity pattern may indicate the location of one element in the submatrix that may have a nonzero value. This data may be included with the selector data because an element that may have a nonzero value in the submatrix may nevertheless have a value of zero.
[0039] For example, to multiply two matrices, one representing activation inputs and one representing weights, using the output clamping technique, each cell accumulates the sum of the products of the matrix elements shifted into the cell. At each clock cycle, each cell may process a given weight input and a given activation input to multiply the two inputs. The cell may add each product to an accumulated value maintained by the cell's accumulator. For example, cell 215 may perform a first product of two matrix elements, e.g., a first activation input and a first weight input, and store the product in the accumulator. Cell 215 may shift an activation input to cell 216 and a weight input to cell 218. Similarly, cell 215 may receive a second activation input from cell 214 and a second weight input from value loader 208B. Cell 215 may multiply the second activation input with the second weight input. Cell 215 may add this to the previous accumulated value to generate an updated accumulated value.
[0040] For sparsity, tile sharing, and other applications, the cell may accumulate values in each of the cell's multiple accumulators. For each pair of submatrices received by the cell, the cell may multiply the two submatrices and store the product in one of the accumulators. The controller of each cell may select the appropriate accumulator based on selector data shifted into the cell along with the submatrices, as described in more detail below.
[0041] After all of the matrix elements have passed through the rows of the systolic array, each cell may shift out its accumulated value as a partial result of the matrix multiplication. These accumulated values may then be used for further calculations during training or use of the machine learning model. Exemplary individual cells are further described below with reference to FIG.
[0042] The cells may pass, e.g., shift, their outputs along their columns, e.g., toward the bottom of the columns in the array 206. In some implementations, at the bottom of each column, the array 206 may include an accumulator unit 210 that stores and accumulates each output from each column. The accumulator unit 210 may accumulate each output of its column to generate a final accumulated value. The final accumulated value may be forwarded to a vector computation unit or other suitable component.
[0043] A cell 204 of the systolic array 206 may be wired to adjacent cells. For example, cell 215 may be wired to cell 214 and cell 216 using a set of wires. In some implementations, when shifting output data out of a cell to the accumulator unit 210, the cell may output a numeric value in a single clock cycle. To do so, the cell may have an output wire for each bit of the computer numeric format used to represent the output value. For example, if the output value is represented using a 32-bit floating-point format, e.g., float32 or FP32, the cell may have 32 output wires to shift out the entire output value in a single clock cycle.
[0044] In some cases, the input to the computation unit and / or accumulator of the cell has a lower precision than the internal precision of the computation unit and / or accumulator. For example, the floating-point values of the input matrix may be 16 bits, for example in bfloat16 or BF16 format. However, the multiplier circuit, adder circuit, and / or accumulator may operate on numbers with higher precision, for example FP32 numbers. In this example, the output of the accumulator of the upstream cell may be an FP32 number. Thus, to output an FP32 number in one clock cycle, the upstream cell may have 32 output wires to the downstream cell. The cell 204 may operate on other numeric formats with other levels of precision.
[0045] 3 illustrates an example architecture 300 of a cell in a systolic array. For example, cell 204 of systolic array 206 of FIG. 2 may be implemented using architecture 300. The cell may be used to perform matrix-matrix multiplication of two input matrices. Although the cell is described with respect to performing matrix-matrix multiplication, the cell may be used to perform other calculations, such as convolution, correlation, or data sorting.
[0046] The cell may include input registers including input register 302 and input register 304. Input register 302 includes A register 303 and A selector register 304. A register 302 receives a submatrix of the input matrix from a right neighboring cell (e.g., an adjacent cell located to the right of a given cell) depending on the location of the cell in the systolic array, or from another component (e.g., value loader 208) if used in systolic array 206 of FIG. 2. A selector register 304 is a selector register that receives selector data for each submatrix received from a right neighboring cell or value loader 208 depending on the location of the cell in the systolic array. In a neural network implementation, A register 303 may receive a submatrix of the weight input matrix. The submatrix and selector data are received via bus 330, which may include one or more wires.
[0047] The input registers 306 include a B register 307 and a B selector register 308. The B register 307 receives a submatrix of the input matrix from a left neighbor (e.g., a neighboring cell located to the left of a given cell) or from another component (e.g., value loader 202 when used in systolic array 206 of FIG. 2) depending on the cell's location in the systolic array. The B-selector register 308 is a selector register that receives selector data for each received submatrix from a left neighbor or value loader 202 depending on the cell's location in the systolic array. In a neural network implementation, the B register 307 may receive a submatrix of the activation input matrix. The submatrix and selector data are received via a bus 332, which may include one or more wires. During training and use of a machine learning model such as a neural network, the activation inputs may be multiplied by corresponding weights, which may be in the form of a matrix.
[0048] Cell 300 includes a multiplication circuit 312, an addition circuit 314, a controller 310, N accumulators 316-1 to 316-N (N is an integer equal to or greater than 2), and a multiplexer 330, each of which may be implemented in hardware circuits. Multiplexer 330 is optional and may be omitted depending on the application for the systolic array that includes cell 300.
[0049] In general, multiplication circuit 312 may multiply the sub-matrices stored in registers 303 and 306. Addition circuit 314 may sum the product with a current accumulated value of one of accumulators 316 and send the sum to that one accumulator 316 for storage.
[0050] The controller 310 may select an accumulator 316 to add the product to based on the selector data in the A selector register 304 and / or the selector data in the B selector register 308. Examples of how the selector data is used to select an accumulator based on the selector data are provided below. In either case, the controller 310 may set a write enable for the selected accumulator 316 to enable writing from the adder circuit 314. For example, the controller 310 sets a write enable for the selected accumulator 316 to enable writing from the adder circuit 314 during the clock cycle corresponding to the add operation.
[0051] In some implementations, the cell 300 may include a single selector register or more than two selector registers. For example, one or more selector registers may receive selector data for use by the controller 310.
[0052] Similarly, to enable the adder circuit to add the product to the current accumulated value of the selected accumulator, the controller 310 may set the selector value of the multiplexer 330 such that the multiplexer passes the current value of the selected accumulator 316 as an input to the adder circuit 314.
[0053] After multiplications are completed for all elements of the input matrix, each accumulator 316 may shift its accumulated value out of cell 300. In some implementations, as shown in FIG. 3, each accumulator 316 has a respective bus 334-1 through 334-N for shifting its accumulated value out of cell 300. In some implementations, multiplexer 330 or another multiplexer may be used to shift each output from cell 300 onto a bus, e.g., one at a time.
[0054] The cell also includes buses for shifting matrix elements in and out from other cells. For example, the cell includes bus 332 for receiving matrix elements from a left neighboring cell and bus 338 for shifting matrix elements to a right neighboring cell. Similarly, the cell includes bus 330 for receiving matrix elements from a top neighboring cell and bus 340 for shifting matrix elements to a bottom neighboring cell. The cell also includes buses 334-1 to 334-N for receiving accumulated values from a top neighboring cell and buses 342-1 to 342-N for shifting accumulated values to a bottom neighboring cell. Each bus may be implemented as a set of wires.
[0055] Systolic arrays including cell 300 may be used in a variety of matrix computation applications. In these applications, denser matrices may be processed using multiple passes over variations of the same input matrix. For example, a matrix with a "2 out of 4" sparsity pattern may be split into the sum of two matrices with a "1 out of 4" sparsity pattern and their subparts processed separately by cells of the systolic array. In another example, a matrix with a "2 out of 4" sparsity pattern may be split into two matrices with a "1 out of 3" sparsity pattern with appropriate shifts and additions of the results to produce a combined result. In another example, the size of one or both matrices may be increased to increase their sparsity to fit a pattern, and the other matrix may be adjusted to produce the same result as in the case of the unwidened input.
[0056] An example of an application is elementary sparsity. In this application, the matrix is divided into k×1 or 1×k blocks with at most one nonzero element in each block, i.e., a “1 out of k” sparsity pattern. In this example, if only one matrix is sparse and the other is dense, only one of the A selector register 304 or the B selector register 308 must be used. This reduces the amount of data that needs to be sent to the systolic array, reduces the number of control operations performed by the systolic array, and may result in faster, more efficient computations. An example is multiplying a matrix A of k×1 blocks with “1 out of k” sparsity by a dense matrix B (1×1 blocks with trivial “1 out of 1” sparsity). In this example, the output may also be constructed from k×1 blocks, one block per array cell, and one element of that block per accumulator 316. That is, if the block is a 3×1 block, three accumulators 316 may be used, one for each of the three elements. The position of the non-zero element in A may be encoded using the selector data shifted into the A selector register 304, which may directly encode which accumulator the multiplication result should be added to.
[0057] In this example, each time a new 1×k block is shifted into the A register 307 and a new 1×1 block is shifted into the B register 303, the controller 310 may use the selector data to identify a non-zero value and select its corresponding accumulator 316. The controller 310 may then set the write enable of the selected accumulator 316 and the selector value of the multiplexer 303 such that the adder circuit 314 adds the product to the current accumulated value of the selected accumulator 316 and the sum is stored in the selected accumulator 316. The 1×k blocks may be shifted along rows from the value loader 213 and the 1×1 blocks may be shifted along rows from the value loader 202.
[0058] Another illustrative application is intrablock sparsity, where a single A or B input element represents a small submatrix with at most one nonzero element. In that case, the selector data in the A selector register 304 and the B selector register 308 would indicate which elements are nonzero. For example, each element could be a 2×2 submatrix. The product of two submatrices could be computed with at most one scalar product, either another submatrix of the same form or all zeros. Each cell 300 then represents an output submatrix with one element in each of its accumulators 316. In particular, if A represents a submatrix with value x at location (ar,ac) and B represents a submatrix with value y at location (br,bc), the result is a submatrix with zero if ac≠br, and value x*y at location (ar,bc) otherwise. This may be used by the controller 310 to set the multiplexer selector values and accumulator write enables to add this resulting sub-matrix to the cell's current value.
[0059] By accommodating different sparsity patterns, the systolic array may perform matrix calculations more efficiently, for example, this may ensure that calculations are performed only on non-zero values (or at least reduce the number of calculations involving zero values) without having to adjust the matrices input to the systolic array.
[0060] Another exemplary application is tile sharing, where multiple, smaller multiplications are performed within the same, larger array. For example, each matrix element of the A and B matrices may be assigned a specific sub-multiplication, with each sub-multiplication going into a different accumulator 316. The selector data in the A selector register 304 and the B selector register 308 are used to tag each element of A and B with the sub-multiplication to which it belongs. If the A and B elements stored in the registers 303 and 307, respectively, do not belong to the same sub-multiplication, the write enable of the accumulator 316 may be disabled by the controller 310. In the absence of multiple accumulators in the same cell, such tile sharing would not be possible without using multiple cells to perform each sub-multiplication. Thus, the use of multiple accumulators in the same cell, and control circuitry for enabling / disabling the accumulators, can reduce the amount of computational resources (e.g., number of cells) required to perform the same operation, resulting in significant speed and other performance advantages over a single accumulator cell.
[0061] For example, for each pair of elements shifted into registers 303 and 307, the controller 310 may determine which sub-multiplication the two elements belong to. If the elements belong to the same sub-multiplication, the controller 310 may set the write enable of the accumulator 316 so that the accumulator 316 corresponding to the sub-multiplication is enabled and the write enable of the other accumulator is disabled. The controller 310 may also set the selector value of the multiplexer so that the summing circuit 314 adds the product to the current accumulated value of the corresponding accumulator 316. If the two elements belong to different sub-multiplications, the controller 310 may disable the write enable to all of the accumulators 316. Additional logic allows the same matrix elements to be shared between the sub-multiplications.
[0062] The controller 310 may be configurable to handle various applications, for example, based on control signals received from the cores or other components. The controller 310 may also perform matrix calculations for dense matrices using a single accumulator, for example, by not using the selector data in the A selector register 304 or the B selector register 308, and sending the sum of the product and the current accumulator value of the single accumulator back to the single accumulator. The use of the controller 310 in combination with multiple accumulators 316 provides the flexibility to handle various applications in the most efficient manner for each application, without requiring hardware modifications.
[0063] 5 is a flow diagram of an example process 500 for performing matrix multiplication. The process 500 may be performed by each of one or more cells of a systolic array of a multiplication unit. The process 500 may be performed multiple times by each cell, and the results calculated by each cell may be used to determine a final matrix multiplication result.
[0064] A first input register of the cell receives the first input submatrix (502). For example, the A register 303 of the cell 300 may receive the first input submatrix. The first input submatrix may represent weight inputs. Along with the first input submatrix, a first selector register, for example, the A selector register 304, may receive first selector data. The first selector data may define, for example, the sparsity of the first input submatrix and the positions of the non-zero elements in the first input submatrix. In another example, the first selector data may indicate a first sub-multiplication to which the first input submatrix belongs.
[0065] A second input register of the cell receives the second input submatrix (504). For example, the B register 307 of the cell 300 may receive the second input submatrix. The second input submatrix may represent activation inputs. Along with the second input submatrix, a second selector register, for example, the B selector register 308, may receive second selector data. The second selector data may define, for example, the sparsity of the second input submatrix and the positions of the non-zero elements in the second input submatrix. In another example, the second selector data may indicate a second sub-multiplication to which the second input submatrix belongs.
[0066] The controller of the cell selects one or more accumulators from the multiple accumulators of the cell (506). The controller may select the one or more accumulators based on the first selector value and / or the second selector value. For example, if the selector data defines sparsity and locations of non-zero elements for one of the input submatrices, the controller may select an accumulator that corresponds to the non-zero elements. The controller may enable a write input to the selected accumulator. The controller may use multiple accumulators to share the same multiplier, e.g., multiplication circuit, among multiple adders, e.g., multiple addition circuits.
[0067] If the first selector data indicates a first sub-multiplication to which the first input sub-matrix belongs and the second selector data indicates a second sub-multiplication to which the second input sub-matrix belongs, the controller may determine whether the first sub-multiplication matches the second sub-multiplication. If so, the controller may select an accumulator corresponding to the matching sub-multiplication and enable a write input to the selected accumulator. If not, the cell may not perform a multiplication and the controller may disable the write input to all of the accumulators.
[0068] Multiplication circuitry of the cell multiplies the first input submatrix with the second input submatrix (508). For example, the multiplication circuitry may perform matrix-matrix multiplication by multiplying corresponding elements of the first input submatrix by corresponding elements of the second input submatrix, one at a time.
[0069] The summing circuit of the cell sums the product with the current accumulated value of the selected accumulator (510). For example, the controller may set a selector value for a multiplexer disposed between the output of the accumulator and the input to the summing circuit such that the output of the selected accumulator is passed to the input of the summing circuit. The sum may be sent to the selected accumulator for storage.
[0070] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. The subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing device. Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.
[0071] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and devices may be realized by, special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).
[0072] A processor suitable for executing a computer program may be based, for example, on a general-purpose or special-purpose microprocessor or both, or on any kind of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. In general, a computer will also include one or more mass storage devices for storing data, such as, for example, a magnetic disk, a magneto-optical disk, or an optical disk, or be operatively coupled to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).
[0073] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices, including semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.
[0074] Although the present specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination, and may even be initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.
[0075] Similarly, although operations are shown in a particular order in the figures, it should not be understood that such operations need to be performed in the particular order shown, or sequential order, to achieve desirable results, or that all of the shown operations need to be performed. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, it should be understood that the separation of various system modules and components in the above-described embodiments does not require such separation in all embodiments, and that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.
[0076] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.
Claims
1. 1. A data processing cell comprising: a multiplication circuit configured to multiply a submatrix of an input matrix; a summing circuit configured to sum an input accumulation value and the product output by the multiplier circuit; a plurality of accumulators connected to the outputs of the summing circuits; a controller circuit configured to select a given accumulator from the plurality of accumulators to receive the sum output by the summing circuit; 1. A data processing cell, comprising: a data processing unit for processing a data set including: a first accumulator having a first accumulator and a second accumulator having a second accumulator; a second accumulator having a second accumulator;
2. 2. The data processing cell of claim 1, wherein the controller circuit is configured to select the given accumulator for each of a plurality of products determined by the multiplication circuit based on selector data received by the data processing cell.
3. 3. A data processing cell as claimed in claim 1 or 2, further comprising a first input register configured to receive a first submatrix and a second input register configured to receive a second submatrix, and the product determined by the multiplication circuit comprises a product of the first submatrix and the second submatrix.
4. 4. The data processing cell of claim 3, further comprising one or more selector registers configured to receive selector data, and wherein the controller circuit selects the given accumulator for each of a plurality of products determined by the multiplication circuit based on the selector data.
5. the selector data includes data defining a sparsity pattern of the first submatrix indicating positions of non-zero elements within the first submatrix; and / or 5. The data processing cell of claim 4, wherein the selector data includes data defining a sparsity pattern of the second submatrix indicating positions of non-zero elements within the second submatrix.
6. A data processing cell, comprising: a multiplication circuit configured to multiply a submatrix of an input matrix; a summing circuit configured to sum an input accumulation value and the product output by the multiplier circuit; a plurality of accumulators connected to the outputs of the summing circuits; a controller circuit configured to select a given accumulator from the plurality of accumulators to receive the sum output by the summing circuit; a first input register configured to receive a first submatrix and a second input register configured to receive a second submatrix, wherein the products determined by the multiplication circuit include products of the first submatrix and the second submatrix; one or more selector registers configured to receive selector data, the controller circuit selecting the given accumulator for each of a plurality of products determined by the multiplication circuit based on the selector data; the selector data indicates a first sub-multiplication to which the first sub-matrix belongs; the selector data indicates a second sub-multiplication to which the second sub-matrix belongs; If the first sub-multiplication matches the second sub-multiplication, the controller circuit is configured to select the given accumulator corresponding to the first sub-multiplication and the second sub-multiplication; The data processing cell, wherein the controller circuit is configured to disable write inputs to all of the plurality of accumulators if the first sub-multiplication does not match the second sub-multiplication.
7. 7. A data processing cell as claimed in claim 1, wherein each accumulator of said plurality of accumulators accumulates values output by said summation circuit for a given set of input matrices.
8. A matrix calculation unit comprising a plurality of the data processing cells according to any one of claims 1 to 7.
9. 1. A method for matrix multiplication comprising the steps of: a first input register of the cell receiving a first input sub-matrix; a second input register of the cell receiving a second input sub-matrix; a controller of the cell selecting a given accumulator from a plurality of accumulators of the cell to receive a sum of (i) a product of the first input submatrix and the second input submatrix and (ii) a current accumulated value of the given accumulator; the selection of the given accumulator is based on a sparsity pattern of the input matrix, the sparsity pattern indicating the number of non-zero elements in the input matrix with x being a maximum number of non-zero elements out of y being a total number of elements in the input matrix; The method comprises: a multiplication circuit of the cell generating a product of the first input submatrix and the second input submatrix; a summing circuit of the cell generating an updated accumulated value by adding the product of the first input submatrix and the second input submatrix to the current accumulated value; storing the updated accumulated value in the given accumulator.
10. 10. The method of claim 9, wherein the products determined by the multiplication circuit include products of the first input submatrix and the second input submatrix.
11. 11. The method of claim 9 or 10, wherein one or more selector registers of the cell include receiving selector data, and selecting the given accumulator includes selecting the given accumulator based on the selector data.
12. the selector data includes data defining a sparsity pattern of the first input submatrix indicating positions of non-zero elements in the first input submatrix; and / or 12. The method of claim 11, wherein the selector data includes data defining a sparsity pattern of the second input submatrix indicating locations of non-zero elements in the second input submatrix.
13. A method for matrix multiplication, comprising the steps of: a first input register of the cell receiving a first input sub-matrix; a second input register of the cell receiving a second input sub-matrix; a controller of the cell selecting a given accumulator from a plurality of accumulators of the cell to receive a sum of (i) a product of the first input submatrix and the second input submatrix and (ii) a current accumulated value of the given accumulator; The method comprises: one or more selector registers of the cells receiving selector data; Selecting the given accumulator includes selecting the given accumulator based on the selector data; the selector data indicates a first sub-multiplication to which the first input sub-matrix belongs; the selector data indicates a second sub-multiplication to which the second input sub-matrix belongs; If the first sub-multiplication matches the second sub-multiplication, the controller selects the given accumulator corresponding to the first sub-multiplication and the second sub-multiplication; If the first sub-multiplication does not match the second sub-multiplication, the controller invalidates write inputs to all of the plurality of accumulators.
14. 14. A method according to claim 9 or 13, wherein each accumulator of the plurality of accumulators accumulates values output by the summation circuit for a given set of input matrices.
Citation Information
Patent Citations
Program, parallel operation method, and information processor
JP2014199545A
Exploiting input data sparsity in neural network computation units
JP2020500365A
Systolic arithmetic on sparse data
JP2021082289A
Sparse convolutional neural network accelerator
US20180046900A1
Matrix multiplication at memory bandwidth
US20190004794A1
