Sparse matrix expansion in situ

Abstract

To provide a method and system for improving throughput for loading an input matrix into a matrix multiplication unit by increasing a rate at which values from the input matrix are loaded into an array of cells in the matrix multiplication unit.SOLUTION: The method includes receiving a plurality of non-zero input values from a first input matrix, receiving, for each non-zero input value in the plurality of input values, index metadata indicating which of the M * N cells in the array the non-zero input value is to be loaded into, and transmitting the non-zero input value and the index metadata to the M * N cells, and at a particular cell of the M * N cells in the array: The method includes receiving a particular non-zero input value and corresponding indexing metadata, and determining from the corresponding indexing metadata for the particular non-zero input value whether to store the particular non-zero input value in the cell or shift the particular non-zero input value to another cell.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Application No. 17 / 368,374, filed July 6, 2021, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] background This specification relates generally to processing matrices using circuits. Summary of the Invention [Means for solving the problem]

[0003] overview According to one innovative aspect of the subject matter described herein, a computing system can improve throughput for loading an input matrix into a matrix multiplication unit by increasing the rate at which values ​​from the input matrix are loaded into an array of cells in the matrix multiplication unit. For example, the matrix processor can be part of a dedicated hardware circuit that trains the neural network, computes the inference of the neural network, or both.

[0004] One way to improve throughput is to increase the rate at which matrices are loaded, and in particular, to increase the rate at which sparse matrices are loaded into an array of cells. A sparse matrix is ​​a matrix that has many elements that have "null" values, i.e., values ​​that are zero. For example, a matrix can be considered sparse if more than half of the values ​​in the matrix are null.

[0005] In general, one innovative aspect of the present subject matter described herein can be realized in a method including actions performed by a hardware device including circuitry for an array having M×N cells. The actions include receiving, in the hardware circuitry, a plurality of nonzero input values ​​from a first input matrix to be multiplied with a second input matrix using the array, the first input matrix including a plurality of nonzero input values ​​and a plurality of zero input values. The actions further include receiving, in the hardware circuitry, index metadata indicating, for each nonzero input value in the plurality of input values, which cell among the M×N cells in the array should be loaded with the nonzero input value; transmitting, using the hardware circuitry, the nonzero input value and the index metadata to the M×N cells; receiving, in a particular cell among the M×N cells in the array, the particular nonzero input value and corresponding index metadata for the particular nonzero input value; and determining, from the corresponding index metadata for the particular nonzero input value, whether to store the particular nonzero input value in the cell for use in performing the multiplication or shift the particular nonzero input value to another cell. M and N are positive integers greater than 1.

[0006] Implementations may include one or more of the following features: The method may further include, for a particular cell of the M×N cells in the array, automatically loading a zero input value into the cell before transmitting the non-zero input value and index metadata to the M×N cell. The step of transmitting the non-zero weight input value and index metadata to the M×N cell may include transmitting a non-zero weight input value along a first dimension of the array having M×N cells. The method may include transmitting zero weight input values. The first dimension of the array may include a vertical dimension of the array, and transmitting the non-zero weight inputs and index metadata to the M×N cells along the first dimension of the array may include transmitting the non-zero weight inputs and index metadata starting from a top edge of the array. The first dimension of the array may include a horizontal dimension of the array, and transmitting the non-zero weight inputs and index metadata to the M×N cells along the first dimension of the array may include transmitting the non-zero weight inputs and index metadata starting from a left edge of the array. The index metadata may include an absolute fixed-length row index for each non-zero input value. The index metadata may include a run-length encoded row index for each non-zero input value. The index metadata may include a bitmap of locations of the non-zero input values ​​in the M×N cells of the array. The hardware device may further include a comparator for each cell in the M×N cells of the array, the comparator configured to compare (i) index metadata of each non-zero input value with (ii) an index of the cell indicating its location in the M×N cells along a first dimension of the array. The multiple input values ​​may be in a compressed sparse column (CSC) matrix format. The hardware device may further include a first-in, first-out (FIFO) register for each cell in the M×N cells of the array, and loading each non-zero input value into the cell may include enqueuing each non-zero input value in a FIFO register associated with the cell. The method may further include, for each cell in the array that stores a particular non-zero input value, receiving a second input value from a second input matrix using a hardware circuit and determining a respective multiplication product based on the particular non-zero input value and the second input value using the hardware circuit. The non-zero input value may move by one cell per clock cycle.The method may further include determining that the first input matrix includes columns having only zero entry values, and in response, for each zero entry value, appending a sign bit to an index defined by the corresponding index metadata.

[0007] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods and encoded on computer storage devices. One or more computer systems can be configured by software, firmware, hardware, or combinations thereof installed on the systems, which, when operated, cause the systems to perform the actions. One or more computer programs can be configured to have instructions, such that, when executed by a data processing device, they cause the devices to perform the actions. [Effects of the Invention]

[0008] Particular embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages. This specification describes a matrix multiplication unit that can load an input matrix to be multiplied with another matrix (or vector) in significantly fewer clock cycles than conventional matrix multiplication units. In particular, a matrix multiplication unit with a comparator for each cell of the array can load only non-zero input values ​​(i.e., elements with non-zero values ​​from the input matrix) into each cell of the array, rather than all resulting input values, including zero input values. A dedicated hardware circuit can then efficiently load input values ​​into the matrix multiplication unit's array by unfolding the matrix within the matrix multiplication unit, rather than in a separate processing unit outside the matrix multiplication unit, or at the boundary of the matrix multiplication unit, thereby enabling the hardware circuit to perform matrix operations more efficiently. This reduces the number of clock cycles required to load input values ​​into the matrix multiplication unit, for example, in terms of the total number of clock cycles required to load input values ​​into each cell of the array through input value load wiring that couples an external memory unit to each cell. This can save the time required, and can also allow for better utilization of the memory capacity, bandwidth, or both of the hardware circuitry, which further improves performance.

[0009] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an exemplary architecture including a matrix calculation unit. [Figure 2] FIG. 2 illustrates an exemplary architecture of a cell within a two-dimensional array of matrix computation units. [Figure 3] FIG. 2 illustrates an exemplary illustration of different index metadata. [Figure 4] 10 is a flow diagram illustrating an example of a process for loading a matrix into a matrix computation unit. [Figure 5] FIG. 10 illustrates an exemplary illustration of pre-processing of matrices to be loaded into a matrix calculation unit. [Figure 6A] FIG. 10 illustrates an exemplary illustration of loading a matrix into a matrix calculation unit. [Figure 6B] FIG. 10 illustrates an exemplary illustration of loading a matrix into a matrix calculation unit. DETAILED DESCRIPTION OF THE INVENTION

[0011] Like reference numbers and designations in the various drawings refer to like elements. Detailed Description Generally, data can be represented in the form of matrices, and computing systems can manipulate the data by performing matrix operations. A matrix can be a one-dimensional vector or a multidimensional matrix. A matrix can be represented by a data structure such as a database table or a variable. A sparse matrix is ​​a matrix that has many elements with "null" values, i.e., values ​​that are zero. For example, a matrix can be considered sparse if more than half of the values ​​in the matrix are null.

[0012] 1 shows an exemplary architecture 100 including a matrix computation unit. The matrix computation unit is a two-dimensional array 106. Array 106 includes a plurality of cells, e.g., cells 114, 116, and 118. In some embodiments, the first dimension of array 106 is the vertical dimension corresponding to rows of cells, and the second dimension of array 106 is the horizontal dimension corresponding to columns of cells. Array 106 can have more rows than columns, more columns than rows, or an equal number of rows and columns.

[0013] In some embodiments, the matrix computation unit may include additional circuitry configured to process data, for example, within each cell of the two-dimensional array of cells 106.

[0014] As one example, the matrix computation unit may be part of a dedicated hardware circuit that trains the neural network, computes the inference of the neural network, or both, and the matrix computation unit may be a matrix multiplication unit that performs matrix multiplication operations between two matrices. That is, the matrix computation unit may be used to perform mathematical operations, such as multiplication and addition, in the neural network by using multiplication and addition circuits within the cells of array 106 and by cooperating with additional components of the hardware circuitry that may include, for example, scalar and vector processing units as well as associated scalar and vector memories.

[0015] As another example, the matrix computation unit may be a cross-lane unit configured to move data between different lanes of multiple lanes. As used herein, a lane generally corresponds to an area, section, or portion of an exemplary hardware circuit, which may include the computing / data processing resources of the hardware circuit. For example, the cross-lane unit may be a transpose unit, a reduction unit, or a permutation unit. The transpose unit may perform a matrix transpose operation. That is, the transpose unit may take numbers in one dimension (e.g., along a given row) and transpose them so that numbers across the lane are transposed with numbers in the other dimension (e.g., along a given column). The reduction or permutation unit may address cross-lane communication issues by supporting various operations such as permutation, lane rotation, rotate-permutation, lane reduction, permutation lane reduction, and segmented permutation lane reduction.

[0016] As another example, the matrix computation unit may be an arithmetic unit that performs operations such as square root, logarithm, and reciprocal through table-driven function approximation. For example, the arithmetic unit may perform multi-entry, multi-port table lookups, e.g., a 128-entry, 128-port table lookup per clock cycle. Additionally, in this example, the arithmetic unit may bridge to larger tables by using zero-hot encoding and may bridge to additional ports by regenerating the input values ​​loaded into the arithmetic unit. Zero-hot encoding is an extension to one-hot encoding using an initial state vector of zero, e.g., [000; 001; 010; 100].

[0017] As yet another example, a matrix computation unit may be a tensor unit that can compress or inject padding data into tensors, thereby providing additional storage savings while still supporting efficient matrix computations. A tensor typically refers to a multidimensional array of numbers or other values, e.g., strings, with a particular order corresponding to the dimensions of the array. For example, a scalar value is a zeroth-order tensor, a vector of numbers is a first-order tensor, and a matrix is ​​a second-order tensor.

[0018] Further details regarding the functionality of these exemplary hardware components, such as scalar or vector processing units and extended vector units, can be found in U.S. Patent No. 10,621,269 B2, entitled "PERFORMING MATRIX MULTIPLICATION IN HARDWARE," filed May 17, 2018, and granted April 14, 2020, which is incorporated herein by reference.

[0019] Typically, before starting a series of matrix operations, such as matrix-matrix multiplication or vector-matrix multiplication, at least a portion of the matrix needs to be pre-loaded into the matrix computation unit. In the above example, the matrix computation unit can improve throughput across the hardware circuit by increasing the rate at which input values ​​are loaded into the matrix computation unit, thereby enabling the hardware circuit to more efficiently perform matrix computations, e.g., computations for processing neural networks.

[0020] Loading a matrix into the matrix computation unit typically involves shifting input values ​​from the matrix across array 106 along one dimension, e.g., from left to right or from top to bottom, e.g., in accordance with host interface control signals. For example, over one clock cycle, the input value in cell 114 may be shifted into the register of cell 116, which is to the right of cell 114. Similarly, the input value in cell 114 may be shifted into the register of cell 118, which is below cell 114.

[0021] As shown, each cell in the two-dimensional array of cells 106 is associated with a comparator unit, e.g., comparator unit 115, 117, or 119. Each cell can be connected to an associated comparator unit, e.g., using wiring. A comparator unit includes hardware circuitry for comparing two input numerical values, e.g., digital or binary numbers, to determine whether one input numerical value is equal to, less than, or greater than the other input numerical value. For example, a comparator unit, e.g., comparator unit 115, can be a multi-bit binary comparator configured to determine equality between two multi-bit binary numbers, e.g., 4-bit, 7-bit, or 8-bit binary numbers. As described further below, the matrix calculation unit utilizes comparator units to improve the efficiency of loading data into the array of cells 106, e.g., before a matrix operation begins.

[0022] In some implementations, the register in each cell of the two-dimensional array of cells 106 may be a first-in, first-out (FIFO) register, and loading each non-zero input value into the cell includes enqueuing the respective non-zero input value in the FIFO register associated with the cell. Similarly, unloading each non-zero input value from the cell includes dequeuing the respective non-zero input value from the FIFO register associated with the cell.

[0023] In some embodiments, each cell of the two-dimensional array of cells 106 can include multiple registers, each operable to store a different input value, e.g., a weight input or an activation input value. The registers in the same cell can be of the same type, e.g., FIFO registers, or of different types.

[0024] The matrix may be delivered to the matrix computation unit via a source bus by a bus connected to the matrix computation unit, and then loaded into a register so that the matrix operation can begin. For example, the matrix may be delivered from a memory unit or memory buffer of the system, which may be located remotely from the circuit.

[0025] In some embodiments, matrices are typically received in a compressed format, such as a compressed sparse row (CSR) format or a compressed sparse column (CSC) format, which use a linear (i.e., one-dimensional) array to store information about the matrix. When represented in such a compressed format, the data describing the matrix typically includes, for each non-zero element (or entry) in the matrix (hereinafter referred to as a "non-zero input value"), associated index metadata that indicates the corresponding location of the non-zero input value in the original matrix. This index metadata further indicates which cell in a two-dimensional array of cells should be loaded with the non-zero input value.

[0026] Let A be an m-by-n matrix with non-zero entries. Let Av, Ac, and Ar be three one-dimensional arrays of length e, e, and (m+1), respectively. In CSR format, A is<Av、Ac、Ar> where: The value array Av holds the values ​​of all non-zero input values ​​in A in row-major order, The column array Ac holds the columns of all non-zero input values ​​in A in row-major order; The row index array Ar holds the index in Av of the first non-zero input value of each row in A, and the last element in the Ar array is the total number of elements in the Av array.

[0027] For example, the following 4x4 matrix

[0028]

number

[0029] In CSR,<Av=[1,0.5,0.5,0.5,0.5,1]、Ac=[3,0,3,0,1,2]、Ar=[0,1,3,5,6]> can be encoded as

[0030] In this example and others described below, rows and columns are indexed, with the index starting at 0 and increasing left to right for columns and top to bottom for rows. Thus, matrix A above has row and column indices that go from 0 to 3.

[0031] Matrices can also be encoded in Compressed Sparse Column (CSC) format, often considered the CSR "transpose." Let A be an m-by-n sparse matrix with non-zero entries. Let A'v, A'c, and A'r be three one-dimensional arrays of lengths e, e, and (m+1), respectively. In CSC format, A is<A’v,A’r,A’c> where: The value array A'v holds the values ​​of all non-zero input values ​​in A in column-major order, row array A'r holds the rows of all non-zero input values ​​in A in column-major order; The column index array A'c holds the index in A'v of the first non-zero input value of each column in A, and the last element in the A'c array is the total number of elements in the A'v array.

[0032] The same 4x4 matrix shown above

[0033]

number

[0034] At CSC,<A′v=[0.5,0.5,0.5,1,1,0.5]、A′r=[1,2,2,3,0,1]、A′c=[0,2,3,4,6]> can be encoded as

[0035] Thus, the row and column index information included in the CSR or CSC representation of a matrix corresponds to the associated metadata of the matrix that indicates, for each non-zero input value in the plurality of input values, which cell in the two-dimensional array of cells should be loaded with the non-zero input value.

[0036] The index metadata associated with a matrix can be pre-generated and received by the system along with the matrix, or can be generated on-the-fly from a received matrix by a different component of the system, e.g., a metadata generation unit. Similarly, the index metadata can be delivered to the matrix computation unit from a memory unit or memory buffer via an additional bus (or wiring) and can be shifted along one dimension, e.g., from left to right or from top to bottom, along with the input values ​​from the matrix across array 106. For example, over one clock cycle, index metadata describing the input value at cell 114 can be shifted to comparator unit 117 associated with cell 116, which is to the right of cell 114. Similarly, the index metadata describing the input value at cell 116 can be shifted to comparator unit 117 associated with cell 116, which is to the right of cell 114. The index metadata describing an input value of 4 can be shifted to a comparator unit 119 associated with cell 118 below cell 114. Exemplary index metadata and loading matrices into a matrix computation unit using the index metadata are further described below.

[0037] 1, the matrix computation unit is a matrix multiplication unit, where the input value fetcher interface 108 sends the input values ​​of a first input matrix and the associated index metadata for the first input matrix to the columns of the array 106, and the value loader 112 sends the input values ​​of a second input matrix and the associated index metadata for the second input matrix to the rows of the array 106. However, in some other embodiments, the input values ​​of the first input matrix are forwarded to the rows and the input values ​​of the second input matrix are forwarded to the columns of the array 106.

[0038] For example, the input values ​​of the first input matrix can correspond to weight input values ​​and the input values ​​of the second input matrix can correspond to input activation values, where the weight input values ​​and input activation values ​​are associated with one layer of multiple layers of a neural network that can be used to compute inferences.

[0039] For example, given an input, a neural network can compute an inference for the input. The neural network computes this inference by processing the input through each of the neural network's layers. In particular, the neural network's layers are arranged in a sequence, each with its own set of weights. Each layer receives an input and processes the input according to the layer's set of weights to generate an output. Thus, to compute an inference from a received input, the neural network receives the input and processes it through each of the neural network layers in the sequence to generate an inference. Here, the output from one neural network layer is provided as an input to the next neural network layer. A data input to a neural network layer, e.g., either the input to the neural network or the output to the neural network layer of a layer below this layer in the sequence, can be referred to as the activation input to the layer.

[0040] 1 can receive input values ​​and associated index metadata from a memory unit, e.g., a dynamic memory. The input value fetcher interface 108 can send the corresponding input values ​​to separate top cells of the array 106, e.g., by using data wires. The input value fetcher interface 108 can also send associated index metadata to separate top cells of the array 106, e.g., by using additional wires or by reusing the same data wires. The top cells can be cells along the top row of the array 106. For example, the input value fetcher interface 208 can send input values ​​to cells 114 and 116.

[0041] 1 may receive input values ​​and associated index metadata from a memory buffer, such as a unified buffer. Each value loader 112 may send a corresponding input value to a separate left-most cell of the array 106. The left-most cell may be a cell along the left-most column of the array 106. For example, the value loader 112 corresponding to cell 114 may send an input value to cell 114.

[0042] However, when the size of the matrix is ​​too large, the amount of time it takes to input the entire matrix (i.e., all input values ​​of the matrix) into the matrix computation unit may become long. Therefore, in order to efficiently input the matrix into the matrix computation unit, i.e., to reduce the number of cycles that the input port on the matrix computation unit is operating to input the matrix, The matrix computation unit utilizes a comparator unit to determine, for example, whether index metadata associated with an input value to be shifted into a particular cell in the two-dimensional array 106 of cells of the matrix computation unit matches the location index of the particular cell in the two-dimensional array. In response to a positive determination, the matrix computation unit then stores the input value in a register of the particular cell for later use in performing the matrix operation. Alternatively, in response to a negative determination, the system shifts the input value to a neighboring cell of the particular cell rather than storing it.

[0043] In this way, the number of clock cycles required to populate the entire matrix is ​​reduced from a number roughly equal to the dimension of the (square) matrix to a number roughly equal to the number of non-zero values ​​in the column (or row) of the matrix having the most non-zero values. If the matrix is ​​sparse, the number of clock cycles required can be reduced by an order of magnitude or more. This may also make it easier to populate dense matrices, i.e., matrices with more non-zero input values ​​than zero input values. To use the same mechanism for populating dense matrices, a counter can be placed at the top edge to add an index to each input value as it passes over the top edge.

[0044] Figure 2 shows an example architecture 200 of cells within a two-dimensional array of matrix computation units. In the example of Figure 2, the matrix computation unit is shown as a matrix multiplication unit having a two-dimensional array wired to perform a matrix multiplication operation, e.g., multiplication of a 128-element vector by a 128x128 matrix.

[0045] The cell may include a power-up register 206 that stores power-up inputs. The power-up register may receive power-up inputs from a left neighbor, i.e., a neighbor located to the left of a given cell, or from a memory buffer, depending on the cell's location in the array. The cell may include two weight path registers 212A-212B, each of which may receive a weight input and transfer the weight input to the weight register 202. The weight register 202 may then store the weight input based on a control signal. For example, a weight input may be shifted into the weight path register from an upper neighbor or from a weight fetcher interface, depending on the cell's location in the array. The cell may also include a sum in register 204. The sum in register 204 may receive power from an upper neighbor, i.e., a neighbor located to the left of a given cell, or from a memory buffer. The weight register 202, the sum input register 204, the activation register 206, and the weight path registers 212A-212B may be registers configured to store values ​​of a particular size, such as floating point values ​​of a particular format.

[0046] A multiplication circuit 208 may be used to multiply the weight input from the weight register 202 with the activation input from the activation register 206. The multiplication circuit 208 may output the product to an addition circuit 210. In some embodiments, the input and output values ​​of the multiplication circuit 208 may be of different sizes and / or formats.

[0047] The adder circuit 210 can sum the product and the accumulated value from the sum input register 204 to generate a new accumulated value. The adder circuit 210 can then send the new accumulated value to another sum input register located in a neighboring cell below. The new accumulated value can be used as an operand for addition in the neighboring cell below. The adder circuit 210 can also accept a value from the sum input register 204 and send the value from the sum input register 204 to the neighboring cell below without summing it with the product from the multiplier circuit 208. In some embodiments, the input values ​​of the adder circuit 210 can be of different sizes and / or formats. In some embodiments, some input and output values ​​of the adder circuit 210 can be of different sizes and / or formats.

[0048] A cell can also shift its weight input and activation input to an adjacent cell for processing. For example, weight path register 212B can send its weight input to another weight register in an adjacent cell below. Activation register 206 can send its activation input to another activation register in an adjacent cell to the right. Thus, both the weight input and the activation input can be reused by other cells in the array in subsequent clock cycles.

[0049] Note that the cell also includes a comparator unit 214. The comparator unit can determine equality between two input values. For example, the comparator unit 214 can compare an index 216 defined by index metadata associated with the weight input transferred from the weight path register 212A against the location index of the cell within the two-dimensional array. As another example, the comparator unit 214 can compare an index 218 defined by index metadata associated with the weight input transferred from the weight path register 212B against the location index of the cell within the two-dimensional array. The comparator unit can send a control signal to the weight register 202 based on the result of the comparison, e.g., by using hardwires. In particular, the control signal generated by the comparator unit 214 can adjust the handling of the weight input by the cell, i.e., whether the cell should store the weight input in the weight register 202 for operation by the multiplier circuit 208 (if equal) or shift the weight input to an adjacent cell (if unequal), at each clock cycle.

[0050] Index metadata can generally be created and maintained in any of a variety of ways, some examples of which are described below.

[0051] FIG. 3 shows an exemplary illustration of different index metadata. 3 has multiple non-zero input values ​​and multiple zero input values, each of which resides in a separate location in the original matrix, as shown at 302 and 304, respectively.

[0052] In one example, the index metadata can include a bitmap index, as shown at 306. For example, the index metadata can include a bitmap of the locations of each non-zero input value, referencing the original matrix.

[0053] In another example, the index metadata can include an absolute fixed-length row index, as shown at 308. For example, the absolute fixed-length index can be an absolute fixed-length column or row index for each non-zero input value of the matrix, i.e., can reference the original matrix.

[0054] In another example, the index metadata can include a run-length encoded index, as shown at 310. For example, the run-length encoded index can be a run-length encoded column or row index for each non-zero input value of a matrix, i.e., can reference the original matrix. Run-length encoding is a technique that allows for store index metadata using variable length data fields, thereby resulting in increased storage savings.

[0055] In these examples, the index metadata typically indicates or otherwise specifies which cell in a two-dimensional array of cells each non-zero input value should be loaded into. In addition, the index metadata may further indicate whether the input values ​​contained in the matrix are non-zero values. For example, the index metadata may include an additional sign bit that is appended to the original unsigned index of each zero-filled value, so that zero-filled values ​​are not loaded into the array because a negative index never matches the location index of any cell in the array.

[0056] By utilizing a comparator unit to load a matrix into a two-dimensional array of a matrix computation unit while determining whether there is a match between the index metadata associated with each non-zero input value of the matrix and the location index of a particular cell in the two-dimensional array, the matrix computation unit can reduce the amount of time it takes to load because it is no longer necessary to load the entire matrix, including any zero input values, into the matrix computation unit. Instead, only a relatively small set of the non-zero input values ​​in the matrix need be loaded. This matrix loading process is described in more detail below.

[0057] 4 is a flow diagram 400 illustrating an example of a process for loading a matrix into a matrix computation unit. For convenience, the method 400 will be described with reference to a system having one or more circuits. For example, with reference to FIG. 1, the system may include a matrix computation unit configured as a two-dimensional array including a plurality of cells physically or logically arranged in M ​​rows and N columns, where M and N are positive integers greater than 1.

[0058] The system receives, in a hardware circuit, a plurality of non-zero input values ​​from a first input matrix to be multiplied with a second input matrix using an array (402). The first input matrix can be a matrix including a plurality of non-zero input values ​​and a plurality of zero input values. The second input matrix, which can also include a plurality of non-zero input values ​​and a plurality of zero input values, can have the same or different dimensions as the first matrix.

[0059] The system receives, in the hardware circuit, index metadata indicating, for each non-zero input value in the plurality of input values, which of the M×N cells in the array should be loaded with the non-zero input value (404). The index metadata can be expressed in the form of absolute fixed-length row indices, run-length encoded row indices, or bitmap indices, to name just a few. In other words, the system can receive absolute fixed-length row indices, run-length encoded row indices, or bitmap indices, each of which specifies location information for the non-zero input values ​​with reference to the original matrix. In some implementations, the system can receive the index metadata along with the first input matrix, for example, from a memory unit or memory buffer of the system. For example, the first input matrix can be received in a compressed sparse row (CSR) format or a compressed sparse column (CSC) format, where row and column index information for the non-zero input values ​​in the first input matrix corresponds to the associated index metadata.

[0060] In other embodiments, the system may generate index metadata on-the-fly from the received first input matrix, i.e., the system may pre-process the first input matrix, for example by using a metadata generation unit of the system, before loading it into a matrix computation unit and using it to perform mathematical operations such as multiplication and addition.

[0061] FIG. 5 shows an exemplary illustration of pre-processing of matrices to be loaded into the matrix calculation unit.

[0062] As shown, the first matrix is ​​a matrix 510 of size 6 x 6. Matrix 510 has a plurality of non-zero input values, e.g., input value 522, and a plurality of zero input values, e.g., input value 526.

[0063] Preprocessing the matrix involves first eliminating all zero input values, e.g., input value 522, from input matrix 510 and then compressing the remaining input values, e.g., input value 522, of input matrix 510 along one dimension, e.g., the vertical dimension. To compress along the vertical dimension, the system can assign a corresponding row index label to each non-zero input value and then compress the non-zero input values ​​along the vertical dimension, i.e., in each column. For example, in the first (leftmost) column, the system can assign a “0” label 532 to input value 522 and a “3” label 542 to input value 532 to indicate that input values ​​522 and 532 are located in the first (top) and fourth columns of the matrix, respectively, and then compress the label data into a list below the first column. In this way, the system obtains a representation of original matrix 510 in a compressed sparse column (CSC) format, as shown in exemplary diagram 550.

[0064] The system uses hardware circuitry to send the non-zero input values ​​and index metadata to an array of MxN cells (406).

[0065] Typically, the system transmits non-zero input values ​​and index metadata along one dimension of the array, for example, from left to right or from top to bottom. For example, when loading a matrix encoded in a CSC representation, the system may transmit corresponding input values ​​to separate top-most cells of the array, i.e., cells along the top row of the array. As another example, when loading a matrix encoded in a CSR representation, the system may transmit corresponding input values ​​to separate left-most cells of the array, i.e., cells along the left-most column of the array.

[0066] 6A-6B show an exemplary illustration of loading a matrix into a matrix computation unit having a 6×6 array of cells. As described above with reference to FIG. 5, non-zero input values ​​of the matrix and associated row index metadata can be sent to cells in separate columns of the array before the matrix operation begins. Furthermore, the input of non-zero input values ​​within each column can be independent of other columns; that is, no inter-column communication is required when sending input values ​​down a column to cells. Thus, the input values ​​across separate columns do not need to start simultaneously; they can be staggered in the order of arrival of the input values ​​at the top edge of the matrix computation unit, thereby facilitating flexible timing, e.g., systolic array timing.

[0067] Illustratively, at 610, the system sends data including a first set of two non-zero input values ​​and their associated row indexes "0" and "3" to a first column of the array. The system sends data including a second set of two non-zero input values ​​and their associated row indexes "1" and "4" to a second column of the array. The system sends data including a third set of two non-zero input values ​​and their associated row index "4" to a third column of the array. The system does not send data to a fourth column of the array. The system sends data including fifth and sixth sets, each including a single non-zero input value and its associated row index "2," to the fifth and sixth columns of the array, respectively. After every clock cycle, the non-zero input values ​​and their associated row index metadata can shift by one cell and in one dimension, e.g., from top to bottom. While a system in which input values ​​are sent to columns of the array has been described, in some embodiments, the input values ​​are sent to rows of the array.

[0068] Then, for each of some or all of the M×N cells in the array (hereinafter referred to as a “particular cell”), the system utilizes a comparator unit associated with the particular cell to determine whether a particular non-zero input value should be stored in the particular cell, or whether a particular 2. Determine whether a cell should be shifted to its neighboring cell. In some embodiments, the system may do this for each of all M×N cells in the array. However, in other embodiments, the system may do this for only some of the M×N cells in the array, such as for each of all but the bottom cell in the array, assuming that any input values ​​that have not yet reached their destination will be loaded into cells along the bottom row.

[0069] More specifically, the system receives (408) a particular non-zero input value and corresponding index metadata for the particular non-zero input value in a particular cell of the M×N cells in the array.

[0070] As explained above, a particular cell may include a register, e.g., a FIFO register, that stores an input value. The register may receive the input value from an adjacent cell above or from an input value fetcher interface, depending on the cell's position in the array. Alternatively, the register may receive the input value from a left adjacent cell, i.e., the adjacent cell located to the left of the particular cell, or from a value loader, again depending on the cell's position in the array.

[0071] For a particular cell of the M×N cells in the array, the system determines from the corresponding index metadata of the particular non-zero input value whether to store the particular non-zero input value in the cell for use in performing the multiplication or shift the particular non-zero input value to another cell (410).

[0072] For example, in the first clock cycle ("clock cycle 0") 620, the array has a "3" in the top-left cell. The "3" represents a non-zero input value labeled with row index "3" stored in the cell. The array also has non-zero input values ​​labeled "4," "4," "2," and "2" stored in the second, third, fifth, and sixth cells along the first (top) row of the array, respectively. The system determines from the corresponding index metadata associated with the non-zero input values ​​that none of the non-zero input values ​​have reached their destination cells. For example, in the top-left cell, the system utilizes a comparator in the cell to compare the row index "3" associated with the non-zero input value against the row index "0" of the top-left cell and determine that the indexes are not equal to one another.

[0073] In the next clock cycle ("clock cycle 1") 630, the non-zero input values ​​labeled with row indices "3", "4", "4", "2", and "2" are shifted down to the respective cells below the cell in the first row, another non-zero input value from the matrix labeled with index "0" is loaded into the top left cell, and a non-zero input value from the matrix labeled with index "1" is loaded into the top cell in the second column.

[0074] At the top left cell 632, the system determines that the non-zero input value labeled with row index "0" has reached the destination cell, i.e., after using a comparator unit at cell 632 to compare the row index, i.e., "0" indicating the first row, with the location index of the top left cell 632, indicating that cell 632 is in the first row of the array, and determining that there is a match.

[0075] In the third clock cycle ("clock cycle 2") 640, the remaining non-zero input values ​​that are not already in place are shifted again. For each remaining non-zero input value that is not already in place, associated metadata indicating which cell in the array the non-zero input value should be loaded into is also shifted along with the non-zero input value. In the first column, the row index "3" is labeled In the second column, non-zero input values ​​labeled with row index "1" are shifted down one cell from where they were stored in the previous cycle to the third cell, while non-zero input values ​​labeled with row index "0" remain where they were stored in the previous cycle. In the second column, non-zero input values ​​labeled with row index "1" and "4" are each shifted down one cell from where they were stored in the previous cycle. Similarly, in the third, fifth, and sixth columns, non-zero input values ​​labeled with row index "4," "2," and "2" are each shifted down to cells along the third row.

[0076] In cell 642, the system determines that a non-zero input value labeled "1" has reached the destination cell after using a comparator unit in the cell to compare the corresponding row index, i.e., "1" indicating the second row, with the location index of cell 642 indicating that cell 642 is in the second row of the array and determining that there is a match. Similarly, in cells 644 and 646, the system determines that input values ​​labeled with row index "2" have reached their respective destination cells.

[0077] In the fourth clock cycle ("clock cycle 3") 650, non-zero input values ​​that are not already in place are shifted again. In the first column, non-zero input values ​​labeled with row index "3" are again shifted down from where they were stored in the previous cycle to the fourth cell. In the second column, non-zero input values ​​labeled with row index "4" are shifted down from where they were stored in the previous cycle to the fourth cell, while non-zero input values ​​labeled with row index "1" remain where they were stored in the previous cycle. In the third column, non-zero input values ​​labeled with row index "4" are shifted down one cell to the fourth cell. In the fifth and sixth columns, non-zero input values ​​labeled with row index "2" remain where they were stored in the previous cycle.

[0078] At cell 652, the system determines that the input value labeled with row index "3" has reached the destination cell after using a comparator unit in the cell to compare the corresponding row index, i.e., "3," indicating the fourth row, with the location index of cell 652, indicating that cell 652 is in the fourth row of the array, and determining that there is a match.

[0079] In the fifth clock cycle ("clock cycle 4") 660, non-zero input values ​​not already in place are shifted again. In the second and third columns, the input values ​​labeled with row index "4" are again shifted down one cell from where they were stored in the previous cycle.

[0080] In cells 662 and 664, the system uses, for example, a comparator unit in cell 662 to compare the corresponding row index, i.e., “4” indicating the fifth row, with the location index of cell 662 indicating that cell 662 is in the fifth row of the array, and after determining that there is a match, determines that the input value labeled with row index “4” has reached the destination cell.

[0081] In particular, after five clock cycles, all non-zero input values ​​from the matrix have now reached their destination cells in the exemplary 6×6 array of cells, as shown in the exemplary diagram 660 of FIG. 6B.

[0082] In some embodiments, each cell in the array may be initialized to have a default value of zero (or null) before the first input matrix is ​​loaded into the array. Once the non-zero input values ​​are in place, i.e., when the array begins operation on the first input matrix, each of the cells that do not have a non-zero input value can operate as if it were loaded with a zero input value, e.g., based on a control signal stored within the cell that specifies that the index associated with any of the non-zero input values ​​does not explicitly match a location index.

[0083] In some embodiments, during or after loading a first input matrix into the matrix calculation unit, the system can load a second input matrix into the matrix calculation unit in a similar or different manner. Once it is determined, e.g., via a control signal, that a particular cell of the matrix calculation unit has both first and second input values ​​from the first and second input matrices in place, the system can proceed to perform a calculation using the first and second input values ​​stored in the cell, e.g., by using a multiplication or addition circuit within the cell. When performing the calculation, the system can use only non-zero input values ​​from the first input matrix.

[0084] Although the examples in Figures 6A-6B show loading the same matrix into the matrix computation unit over multiple consecutive clock cycles, this is not required. When loading a sequence of multiple one-hot matrices (i.e., one-hot matrices encoded in CSR or CSC format) into the matrix computation unit, a different matrix may be used in every clock cycle. A one-hot matrix refers to a matrix in which each column (or row) has only one non-zero input value, e.g., 1, and the other input values ​​are zero. This not only extends the agility of the matrix computation unit when configured, for example, as a matrix multiplication unit, but also enables permutation and table lookup functions when configured, for example, as a permutation unit or arithmetic unit.

[0085] Embodiments of the subject matter and functional operations described herein, including the structures disclosed herein and their structural equivalents, can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or a combination of one or more of them. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively, or in addition, the program instructions can be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a receiver apparatus suitable for execution by a data processing apparatus.

[0086] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0087] program, software, software application, app, module, software A computer program, which may also be referred to or described as a software module, script, or code, can be written in any type of programming language, including compiled or interpreted, or declarative or procedural, and can be arranged in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can correspond to a file in a file system, but this is not required. A program can be stored as part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple organized files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program can be arranged to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0088] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0089] A computer suitable for running a computer program can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0090] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0091] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer that has a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. The computer may also provide interaction with the user through a variety of methods. For example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including audio, speech, or tactile input. Additionally, the computer may interact with the user by sending documents to a device used by the user and receiving documents from the device, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, e.g., a smartphone, executing a messaging application, and receiving a response message from the user.

[0092] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., a data server; a middleware component, e.g., an application server; a front-end component, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described herein; or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0093] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, the server sends data, e.g., HTML pages, to a user device acting as a client, for the purpose of displaying the data to and receiving user input from the user interacting with the device. Data generated at the user device, e.g., results of user interaction, can be received from the device at the server.

[0094] Embodiment 1 is a method performed by a hardware device including circuitry for an array having M×N cells. The method comprises: The method includes receiving, in a hardware circuit, a plurality of non-zero input values ​​from a first input matrix to be multiplied with a second input matrix using an array, the first input matrix including a plurality of non-zero input values ​​and a plurality of zero input values. receiving, in the hardware circuit, index metadata indicating, for each non-zero input value in the plurality of input values, which of the M×N cells in the array should be loaded with the non-zero input value; transmitting, using a hardware circuit, non-zero input values ​​and index metadata to the M×N cells; In a particular cell of the M×N cells in the array, receiving a particular non-zero input value and corresponding index metadata for the particular non-zero input value; determining from the corresponding index metadata of the particular non-zero input value whether to store the particular non-zero input value in a cell for use in performing the multiplication or shift the particular non-zero input value to another cell; Further includes:

[0095] M and N are positive integers greater than 1. Embodiment 2 is a method described in embodiment 1, further comprising a step of automatically loading a zero input value into a particular cell among the M×N cells in the array before transmitting the non-zero input value and index metadata to the M×N cells.

[0096] Embodiment 3 is a method described in any one of embodiments 1 to 2, wherein the step of sending non-zero weighted input values ​​and index metadata to the M×N cells includes a step of sending the non-zero weighted input values ​​along a first dimension of an array having M×N cells.

[0097] Embodiment 4 is the method of any one of embodiments 1 to 3, wherein the first dimension of the array comprises the vertical dimension of the array.

[0098] Sending the non-zero weight inputs and index metadata to the M×N cells along a first dimension of the array includes sending the non-zero weight inputs and index metadata starting from the top edge of the array.

[0099] Embodiment 5 is the method of any one of embodiments 1 to 3, wherein the first dimension of the array comprises the horizontal dimension of the array.

[0100] Sending the non-zero weight inputs and index metadata to the M×N cells along the first dimension of the array includes sending the non-zero weight inputs and index metadata starting from the left edge of the array.

[0101] A sixth embodiment is the method according to any one of the first to fifth embodiments, wherein the index metadata includes an absolute fixed-length row index for each non-zero input value.

[0102] A seventh embodiment is the method according to any one of the first to fifth embodiments, wherein the index metadata includes a run-length encoded row index for each non-zero input value.

[0103] An eighth embodiment is the method described in any one of the first to fifth embodiments, wherein the index metadata includes a bitmap of locations of non-zero input values ​​in the M×N cells of the array.

[0104]

[0023] Example 9 is the method of any one of Examples 1 to 8, wherein the hardware device further includes a comparator for each cell in the MxN cells of the array, the comparator configured to compare (i) the index metadata of each non-zero input value with (ii) an index of the cell indicating its location in the MxN cells along a first dimension of the array.

[0105] A tenth embodiment is the method according to any one of the first to ninth embodiments, wherein the plurality of input values ​​are in a compressed sparse column (CSC) matrix format.

[0106]

[0023] In an eleventh embodiment, the method of any one of the first to tenth embodiments is characterized in that the hardware device further comprises a first-in, first-out (FIFO) register for each cell in the MxN cells of the array, and loading each non-zero input value into the cell comprises enqueuing each non-zero input value in a FIFO register associated with the cell.

[0107] Embodiment 12 is the method according to any one of embodiments 1 to 10, further comprising: for each cell in the array that stores a particular non-zero input value: receiving, using a hardware circuit, second input values ​​from a second input matrix; determining, using hardware circuitry, each multiplication product based on a particular non-zero input value and a second input value; Further includes:

[0108] Embodiment 13 is the method according to any one of embodiments 1 to 12, wherein the non-zero input value moves by one cell per clock cycle.

[0109] Embodiment 14 is the method described in any one of embodiments 1 to 13, determining that a first input matrix includes columns having only zero input values; in response, for each zero input value, appending a sign bit to an index defined by the corresponding index metadata; Further includes:

[0110] Embodiment 15 is a system comprising one or more computers and one or more storage devices having stored thereon instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform a method as recited in any one of embodiments 1 to 14.

[0111] Embodiment 16 is a computer storage medium having encoded thereon a computer program, the program including instructions that, when executed by a data processing apparatus, are operable to cause the data processing apparatus to perform a method according to any one of embodiments 1 to 14.

[0112] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as operating in a particular combination and even initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from that combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0113] Similarly, although operations are shown in the figures in a particular order, this should not be interpreted as requiring such operations to be performed in the particular order shown, or sequentially, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0114] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method performed by a hardware device including circuitry for an array having M×N cells, comprising: receiving, in the hardware circuit, a plurality of non-zero input values ​​from a first input matrix to be multiplied with a second input matrix using the array, the first input matrix including the plurality of non-zero input values ​​and a plurality of zero input values; The method comprises: receiving, in the hardware circuit, index metadata indicating, for each non-zero input value in the plurality of input values, which of the M×N cells in the array should be loaded with the non-zero input value; transmitting the non-zero input values ​​and the index metadata to the M×N cells using the hardware circuitry; In a particular cell of the M×N cells in the array, receiving a particular non-zero input value and corresponding index metadata for said particular non-zero input value; determining from the corresponding index metadata for the particular non-zero input value whether to store the particular non-zero input value in the cell for use in performing a multiplication or to shift the particular non-zero input value to another cell; further comprising A method wherein M and N are positive integers greater than 1.

2. 2. The method of claim 1, further comprising: at the particular cell of the M×N cells in the array, automatically loading the cell with a zero input value before transmitting the non-zero input value and the index metadata to the M×N cell.

3. 3. The method of claim 1, wherein transmitting the non-zero weight input values ​​and the index metadata to the M×N cells comprises transmitting the non-zero weight input values ​​along a first dimension of the array having the M×N cells.

4. the first dimension of the array comprises a vertical dimension of the array; 4. The method of claim 3, wherein transmitting the non-zero weight inputs and the index metadata to the M×N cells along the first dimension of the array comprises transmitting the non-zero weight inputs and the index metadata starting from a top edge of the array.

5. the first dimension of the array comprises a horizontal dimension of the array; 4. The method of claim 3, wherein transmitting the non-zero weight inputs and the index metadata to the M×N cells along the first dimension of the array comprises transmitting the non-zero weight inputs and the index metadata starting from a left edge of the array.

6. The method of any one of claims 1 to 5, wherein the index metadata comprises an absolute fixed length row index for each non-zero input value.

7. The method of any one of claims 1 to 5, wherein the index metadata comprises, for each non-zero input value, a run-length encoded row index.

8. The method of any one of claims 1 to 5, wherein the index metadata comprises a bitmap of the locations of the non-zero input values ​​in the MxN cells of the array. 。

9. 9. The method of claim 1, wherein the hardware device further comprises a comparator for each cell in the M×N cells of the array, the comparator configured to compare (i) the index metadata for each non-zero input value with (ii) an index of the cell indicating its location in the M×N cells along the first dimension of the array.

10. The method of any one of claims 1 to 9, wherein the plurality of input values ​​are in a compressed sparse column (CSC) matrix format.

11. 11. The method of claim 1, wherein the hardware device further comprises a first-in, first-out (FIFO) register for each cell in the M×N cells of the array, and wherein loading each of the non-zero input values ​​into the cell comprises enqueuing each of the non-zero input values ​​in the FIFO register associated with the cell.

12. For each cell in the array that stores a particular non-zero input value, receiving, using the hardware circuitry, second input values ​​from the second input matrix; using the hardware circuitry to determine respective multiplication products based on the particular non-zero input value and the second input value; The method of any one of claims 1 to 11, further comprising:

13. The method of any one of claims 1 to 12, wherein the non-zero input value moves by one cell every clock cycle.

14. determining that the first input matrix includes columns having only zero input values; in response, for each zero input value, appending a sign bit to an index defined by said corresponding index metadata; The method of any one of claims 1 to 13, further comprising:

15. 10. A system comprising one or more computers and a computer-readable medium coupled to the one or more computers and having instructions stored thereon, the instructions, when executed by the one or more computers, causing the one or more computers to perform the respective operations of any one of the methods set forth in any of the preceding claims.

16. A computer-readable medium having stored thereon instructions that, when executed by one or more computers, cause the one or more computers to perform the respective operations of any one of the methods set forth in any of the preceding claims.

Images