Sparse matrix multiplication in hardware

Abstract

The present disclosure relates to sparse matrix multiplication in hardware. Methods, systems, and apparatuses, including computer-readable storage media, are provided for sparse matrix multiplication. A system for matrix multiplication includes an array of sparse tiles. Each sparse tile can be configured to receive an input submatrix and an input subvector, where the input submatrix has a number of non-zero values that is equal to or less than a predetermined maximum non-zero threshold. The sparse tile can compute, through a plurality of multiplier circuits, one or more products of a vector value multiplied by respective non-zero values of the input submatrix. The sparse tile can generate a tile output vector that is an output of the sparse tile and uses the one or more products, the tile output vector being a product of applying the tile input vector to the tile input matrix.

Description

TECHNICAL FIELD

[0001] This disclosure relates to sparse matrix multiplication in hardware. BACKGROUND

[0002] A sparse matrix is a matrix with a high proportion of zero values to non-zero values as elements of the matrix. Different sparse matrices can have different degrees of sparsity based on the proportion of zero values to non-zero values. A matrix with a high proportion of zero values to non-zero values is said to have a higher degree of sparsity than a matrix with a lower proportion.

[0003] A neural network is a machine learning model that includes one or more layers of non-linear operations to predict an output for a received input. In addition to an input layer and an output layer, some neural networks include one or more hidden layers. The output of each hidden layer can be input to another hidden layer or the output layer of the neural network. Each layer of the neural network can generate a respective output from a received input according to values of one or more model parameters for the layer. The model parameters can be weights or biases determined through a training algorithm to cause the neural network to generate accurate outputs. The values of the model parameters for a layer of the neural network can be represented as elements of a matrix or tensor. SUMMARY

[0004] Aspects of the disclosure relate to sparse matrix dense vector multiplication in hardware.

[0005] One aspect of the disclosure provides a system comprising: a sparse slice including a plurality of multiplier circuits, wherein the sparse slice is configured to: receive a slice input matrix including a number of non-zero values equal to or less than a predetermined maximum non-zero threshold; receive a slice input vector including a plurality of vector values; receive, for each of the multiplier circuits, a respective non-zero value of the slice input matrix; generate, by the plurality of multiplier circuits, one or more products of a vector value multiplied by the respective non-zero value of the slice input matrix; and generate, as an output of the sparse slice and using the one or more products, a slice output vector that is a product of the slice input vector applied to the slice input matrix.

[0006] Another aspect of the present disclosure provides one or more non-transitory computer-readable storage media storing instructions that, when executed by a system comprising a plurality of sparse tiles, cause the system to perform operations comprising: receiving, by a sparse tile comprising a plurality of multiplier circuits, a tile input matrix and a tile input vector, wherein the tile input matrix comprises a number of non-zero values equal to or less than a predetermined maximum non-zero threshold, and the tile input vector comprises a plurality of vector values; receiving, for each of the multiplier circuits, a respective non-zero value of the tile input matrix; generating, by the plurality of multiplier circuits of the sparse tile, one or more products of a respective vector value multiplied by the respective non-zero value; and generating, as an output of the sparse tile and using the one or more products, a tile output vector that is a product of applying the tile input vector to the tile input matrix.

[0007] Another aspect of the present disclosure provides a method comprising: receiving, by a sparse tile comprising a plurality of multiplier circuits, a tile input matrix and a tile input vector, wherein the tile input matrix comprises a number of non-zero values equal to or less than a predetermined maximum non-zero threshold, and the tile input vector comprises a plurality of vector values; receiving, for each of the multiplier circuits, a respective non-zero value of the tile input matrix; generating, by the plurality of multiplier circuits of the sparse tile, one or more products of a respective vector value multiplied by the respective non-zero value; and generating, as an output of the sparse tile and using the one or more products, a tile output vector that is a product of applying the tile input vector to the tile input matrix.

[0008] The foregoing and other aspects can each be optionally included alone or in combination with any one or more of the other foregoing and / or following aspects. An implementation can include one or more, several or all of the following features.

[0009] A length of the tile output vector is greater than one.

[0010] The sparse tile is one of a plurality of sparse tiles configured to: receive a plurality of tile input matrices that are submatrices of a system input matrix; receive a plurality of tile input vectors that are subvectors of a system input vector; and generate, by the plurality of sparse tiles, a system output vector that represents a product of applying the system input vector to the system input matrix.

[0011] The plurality of sparse tiles are arranged as a systolic array, the systolic array including one or more groups of sparse tiles along a column dimension of the systolic array; and wherein to generate the system output vector, the one or more processors are further configured to: for each group along the column dimension of the systolic array, sum each sparse tile in the group with a respective tile output vector to generate a respective column output vector; and concatenate the respective column output vectors of each group to generate the system output vector.

[0012] Each multiplier circuit is coupled to a respective register that includes the respective non-zero value from the respective tile input matrix of the sparse tile.

[0013] A number of multiplier circuits in the plurality of multiplier circuits is equal to the predetermined maximum non-zero threshold.

[0014] The sparse tile further includes a crossbar circuit, and wherein the sparse tile is further configured to: receive, by the crossbar circuit, the plurality of vector values of the tile input vector; and send, as input to each of the plurality of multiplier circuits and by the crossbar circuit, a vector value of the plurality of vector values.

[0015] The sparse tile is further configured to load non-zero values of a same column in the tile input matrix in registers of adjacent multiplier circuits in the plurality of multiplier circuits.

[0016] The sparse tile is further configured to receive one or more control values that specify at least locations of non-zero values along each column of the tile input matrix; and wherein the crossbar circuit for the sparse tile is further configured to: receive the one or more control values; and send, to adjacent multiplier circuits, a vector value to be multiplied with a non-zero value along a same column of the tile input matrix according to the one or more control values.

[0017] The sparse tile further includes a plurality of adder circuits, wherein the sparse tile further includes one or more segment tags, wherein each segment tag is configured to gate input to a respective adder circuit of the plurality of adder circuits based on a value of a respective control value loaded in the segment tag; and wherein the sparse tile is further configured to: load at least a portion of the one or more control values in the one or more segment tags, wherein an adder circuit for a non-zero value in a first column of the tile input matrix is gated by receiving input from an adjacent adder circuit, the input including a non-zero value in a second column of the tile input matrix different from the first column; and generate, by the plurality of adder circuits, one or more sums of the one or more products, wherein each sum of the one or more sums is a respective segment sum of one or more non-zero values of a column of the tile input matrix multiplied by one or more respective values of the tile input vector.

[0018] The plurality of adder circuits form a parallel segmented sum circuit, and wherein each segment sum of the one or more segment sums is a sum of outputs from adjacent adder circuits not gated by a segment tag.

[0019] The crossbar switch circuit is a first crossbar switch circuit; and wherein the sparse tile further includes a second crossbar switch circuit configured to: receive the one or more segment sums, and arrange the one or more segment sums to generate a respective tile output vector of the sparse tile according to the one or more control values.

[0020] The second crossbar switch circuit can form a Benes network, and wherein the tile input matrix is a square matrix.

[0021] Another aspect of the present disclosure provides a system comprising one or more processors; and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving an input matrix including zero values and non-zero values; and partitioning the input matrix into a plurality of submatrices, wherein a number of non-zero values of each submatrix is equal to or less than a predetermined maximum non-zero threshold, and wherein a dimension of each submatrix is equal to or less than a predetermined dimension threshold.

[0022] The operations can further include, for each submatrix, generating one or more respective control values specifying locations of non-zero values along each column of the submatrix.

[0023] The operations can further include sending each sub-matrix and the one or more respective control values for each sub-matrix to a plurality of sparse tiles configured to process each sub-matrix and respective control values for the sub-matrix. Each sparse tile of the plurality of sparse tiles is configured to receive a sub-matrix, the one or more respective control values for the sub-matrix, and at least a portion of an input vector and generate a respective output tile vector representing a product of the sub-matrix and the portion of the input vector. The system of claim 21, wherein partitioning the input matrix includes partitioning the matrix into a number of sub-matrices equal to a number of sparse tiles in the plurality of sparse tiles.

[0024] Other aspects of the disclosure include corresponding systems, apparatuses, and computer programs stored on one or more non-transitory computer-readable storage media. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a block diagram of an example system including an array of sparse tiles according to aspects of the disclosure.

[0026] Figure 2 is a block diagram of an example sparse tile according to aspects of the disclosure.

[0027] Figure 3A is an illustration of an example tile input matrix for a sparse tile.

[0028] Figure 3B is an illustration of a vector of non-zero values of an example tile input matrix.

[0029] Figure 3C is an illustration of a vector of control values for an example tile input matrix.

[0030] Figure 4 illustrates an example matrix multiplication by a sparse tile receiving a tile input matrix, a tile input vector, and control values for the tile input matrix.

[0031] Figure 5 is a flowchart of an example process for multiplying a sub-matrix of a sparse matrix on a sparse tile with a system input vector according to aspects of the disclosure.

[0032] Figure 6 is a flowchart of an example process of generating a system output vector representing a product of a system input vector and a system input matrix from a plurality of sparse tiles.

[0033] Figure 7A is an illustration of an example system input matrix and system input vector.

[0034] Figure 7Bis an illustration of a partitioned system input matrix and a system input vector.

[0035] Figure 7C is an illustration of a partitioned system input matrix and a system output vector representing a product of the system input matrix and the system input vector.

[0036] Figure 8A is a flowchart of an example process for configuring sparse tiles using one or more control values of a tiled input matrix according to aspects of the present disclosure.

[0037] Figure 8B is a flowchart of an example process for performing matrix-vector multiplication using sparse tiles configured according to a process of Figure 8A

[0038] Figure 9 is a flowchart of an example process for generating submatrices from a system input matrix according to aspects of the present disclosure.

[0039] Figure 10 is a block diagram of an example computing environment implementing a sparse matrix multiplication system according to aspects of the present disclosure. DETAILED DESCRIPTION

[0040] SUMMARY

[0041] Aspects of the present disclosure relate to a system comprising one or more integrated circuits configured for sparse matrix dense vector multiplication. A system of multiple sparse tiles can receive, for each sparse tile, a submatrix and a subvector of an input system matrix and vector. Each sparse tile can be at least a portion of an integrated circuit and implement a number of arithmetic units, such as multiplier circuits and adder circuits. Each sparse tile is configured to receive a submatrix having a number of non-zero values equal to or less than a maximum non-zero threshold, which can be predetermined when the system is implemented, for example, as a chip comprising the sparse tiles and optionally one or more other components as described herein.

[0042] Each sparse tile can additionally receive metadata, such as control values described herein, which the sparse tiles can use to direct different inputs to each multiplier circuit or adder circuit and gate when an output from one unit is passed to another. By configuring the sparse tiles according to the received metadata, these sparse tiles can efficiently process input submatrices of arbitrary size up to a predetermined dimension threshold and output a product as a vector of length corresponding to the input.

[0043] ​As part of receiving the sub-matrix, each sparse tile can receive one or more control values that represent the location of non-zero values at each column in the sub-matrix. With the one or more control values, the sparse tile can be configured to adjust how individual values of the sub-matrix and sub-vector are multiplied, added, and arranged for the sparse tile to generate a tile output vector that represents the product of the sub-matrix and sub-vector. The system can be further configured to generate a system output vector from the tile output vectors of each tile that represents the product of applying the system input vector to the system input matrix (e.g., by multiplying the system input vector with the system input matrix).

[0044] Aspects of the present disclosure also relate to systems for preprocessing sparse matrices for processing by an array of sparse tiles. A system of one or more processors can be configured to partition an input matrix into a plurality of sub-matrices, where each sub-matrix has dimensions that are equal to or less than a predetermined dimension threshold of a maximum matrix input size of a specified sparse tile. As part of the partitioning, the system can identify whether any sub-matrices include a number of non-zero values that is greater than a predetermined maximum non-zero threshold, and in response, repartition the sub-matrices along the same row or column as the identified sub-matrices. The system can repeat this process until sub-matrices are generated for each sparse tile such that the dimensions of the sub-matrices are within the dimension threshold and the number of non-zero values in the sub-matrices is less than or equal to the predetermined non-zero threshold.

[0045] Systems implemented in accordance with aspects of the present disclosure can more efficiently perform workloads involving repeated multiplication of sparse matrices with vectors. For example, a device implementing a system on a chip (SoC) in accordance with aspects of the present disclosure can generate a product of a sparse matrix multiplied with a vector in fewer processing cycles than conventional approaches, at least because the device omits redundant “multiply-by-zero” computations of zero elements in the matrix that do not contribute to the final product. While other approaches can experience increased performance loss for larger matrices with higher sparsity ratios, systems implemented as described herein can even more efficiently use these matrices to compute products, at least because a higher proportion of redundant computations are omitted as the sparsity ratio of the input matrix increases.

[0046] Some workloads, such as performing or training neural networks, rely heavily on performing matrix multiplication. Hardware accelerators or other devices can efficiently perform certain operations, such as matrix multiplication, but often have limiting conditions on processing. For example, a device can require a predefined sparsity ratio between zero values and non-zero values of a matrix, or have strict limits on the size of inputs to process. While matrix multiplication is a ubiquitous type of computation for many neural network workloads, accelerators with limiting conditions on processing are limited in the number of types of workloads that the accelerator can support, such as workloads involving matrix multiplication of matrices with different sparsity ratios.

[0047] Aspects of the disclosure provide a system for sparse matrix multiplication that can flexibly handle matrices of different sizes and sparsity ratios. Sparse tiles of the system configured for sparse matrix multiplication can discard zero values of an input matrix while still tracking the locations of non-zero values in a given input submatrix across multiple columns. The sparse tiles can be configured with only as many registers and multiplier circuits as a predetermined maximum non-zero threshold, and only store non-zero values of the input submatrix in memory. As a result, the sparse tiles can store and process input data more efficiently and with fewer resources than storing a full-sized matrix.

[0048] Further, the sparse tiles can be configured to arrange non-zero values and their products when multiplied with values of an input vector along adjacent multiplier and adder circuits, respectively. The sparse tiles can be configured to efficiently generate partial sums for each column of the input matrix by combining outputs of these adjacent circuits, and rearrange the partial sums to generate a tile output vector. The sparse tiles can preserve the order of non-zero values to accurately add and multiply these values with corresponding values of the input vector, and accurately generate the tile output vector as a product of the input submatrix and the input vector. The sparse tiles can perform multiplication on the input matrix without a pre-processing operation to change the shape of the input matrix.

[0049] The system can also include one or more processors configured for pre-processing the sparse matrix for multiplication by the input vector by the plurality of sparse tiles. The system can support different configurations of the sparse tiles, including generating different numbers of submatrices depending on the number of available sparse tiles, and generating submatrices for different maximum non-zero thresholds.

[0050] Aspects of the disclosure as described herein can be implemented by a system including one or more processors and a plurality of sparse tiles. The system can be implemented, for example, as a chip installed on a server computing device in a data center of a computing platform.

[0051] Example System

[0052] Figure 1 is a block diagram of an example system 100 including an array 101 of sparse tiles 101A-101P. The array 101 can be at least a portion of a sparse matrix multiplication system, such as the sparse matrix multiplication system 100, configured to perform matrix multiplication on a matrix input.

[0053] As described herein with reference to Figure 2Described in more detail, a sparse tile is a collection of circuits configured to perform arithmetic operations, arrange inputs and outputs between circuits performing arithmetic operations, and / or gate inputs and outputs between other circuits. For example, a sparse tile can be configured to perform matrix multiplication between a rectangular tile or portion of a sparse matrix and a linear tile or slice of a vector. As described in more detail herein, a sparse tile can receive any of a variety of different tiles or portions of a sparse matrix, with any number of non-zero values in the tile or portion up to a predetermined maximum threshold.

[0054] The collection of circuits can be configured as described herein to control: the state of each gate (i.e., open or closed); the operands that are computed; and / or the arrangement of inputs to and outputs from each circuit. The configuration can depend at least in part on the dimensions of the tiled input matrix that the sparse tile is configured to receive as input, and / or on the arrangement and number of non-zero values in the tiled input matrix. Each sparse tile can be a separate circuit component of a system configured to multiply a sparse matrix by a vector.

[0055] Each sparse tile 101A-101P is configured to communicate with two or more other sparse tiles in the array 101. A sparse tile can communicate with its immediate neighbors, for example, the neighbors that immediately precede or follow the sparse tile along a dimension defined by the rectangular arrangement of sparse tiles 101A-101P. Connections between sparse tiles can be implemented, for example, through a bus or one or more circuit interconnections that physically connect the sparse tiles to their neighbors. Each sparse tile 101A-101P can be implemented as one or more circuits configured to receive at least a portion of a matrix and at least a portion of a vector and generate an output vector representing a product of multiplying the input matrix and vector. In some examples, the sparse tiles 101A-101P are organized as a systolic array, although in various implementations the array 101 is generally configured or arranged according to a rectangular arrangement of sparse tiles 101A-101P.

[0056] The array 101 can be at least a portion of a system-on-a-chip that implements several components and integrated circuits on a circuit board or other material. The array 101 can be installed as part of a computing device and configured to interact with other components of the device, including memory, processors, network components, and / or peripheral devices. For example, the array 101 can receive a system input vector 105 and a system input matrix 110 from one or more memory devices implemented as part of the system 100. The system 100 can generate as output a system output vector 115 representing a product of multiplying the system input vector 105 and the system input matrix 110.

[0057] In some examples, the system output vector 115 can be fed as input to other devices or components implementing system 100. For example, if the system output vector 115 is a product of model parameter values ​​multiplied by a vector input of the neural network, the system output vector 115 can be fed as input to one or more processors configured to compute activation functions for the output vector 115.

[0058] System 100 may receive system input vector 105 and system input matrix 110 from preprocessing engine 150. Preprocessing engine 150 may be implemented on one or more computing devices, which may or may not be the devices implementing system 100. As an example, system input matrix 110 may include values ​​representing at least a portion of the model parameter values ​​used for the neural network. System input matrix 110 may be part of a more complex data structure, such as a multidimensional array or tensor. System 100 may be configured to receive each matrix, such as a three-dimensional tensor or matrix, corresponding to at least a portion of the larger data structure, and generate a corresponding output when multiplied by system input vector 105. System input vector 105 may be, for example, the input to a trained neural network whose model parameter values ​​are at least partially represented by the values ​​of system input matrix 110.

[0059] The preprocessing engine 150 can be configured to process the system input matrix 110 to generate one or more control values ​​111 and matrix partition data 112. The preprocessing engine 150 can receive the system input matrix 110 in various formats used for storing sparse matrices, such as its complete row and column form (with all non-zero and zero values). As other examples, the preprocessing engine 150 can receive the system input matrix 110 according to a compressed sparse column format, a coordinate list format, or any of various other formats used for storing sparse matrices.

[0060] In some examples where the preprocessing engine 150 receives the system input matrix 110 in its complete form, the preprocessing engine 150 is configured to convert the system input matrix 110 into a format predetermined to be suitable for storage in memory. For example, before conversion to the predetermined format, the preprocessing engine 150 may remove zero values ​​from the matrix and generate control values ​​that are used to track the position of non-zero values ​​relative to their original positions in the matrix.

[0061] As referenced in this article Figure 2The control values described in more detail are used to configure each sparse tile to receive a subvector of the system input vector 105 and multiply it with a submatrix of the system input matrix 110. The partition data 112 is data that specifies how the system input matrix 110 should be partitioned into submatrices. Each submatrix is received as input at a respective sparse tile, and the partition data 112 specifies a partitioning of as many submatrices as there are sparse tiles in the array 101.

[0062] The preprocessing engine 150 can be configured to generate the vector partition data 106. As referenced with respect to FIG. 7A through Figure 7C Described in more detail, sparse tiles along the same row or column can receive the same subvector of input for multiplication with a respective submatrix.

[0063] Although the system input matrix 110 and the system input vector 105 are shown as being fed along the left and right sides of the system 100, the exact positioning and orientation of the bus that feeds input data to the system 100 can vary according to different implementations. For example, based on the location of other components on the same chip as the system 100, the bus or circuit interconnect used to feed input to the system 100 and receive output from the system 100 can be oriented or positioned differently to account for the location of those other components.

[0064] Figure 2 is a block diagram of an example sparse tile 200 according to aspects of the present disclosure. For example, the sparse tiles 101A-101P of the system 100 can each be implemented as described herein with respect to the sparse tile 200.

[0065] The sparse tile 200 is configured to receive a tile input vector 205 and a tile input matrix 210. The tile input vector 205 includes one or more vector values and can have a maximum dimension of 1 x R. The tile input matrix includes one or more zero values and one or more non-zero values and has a maximum dimension of R x C. R (rows) and C (columns) are dimension thresholds that are predetermined and correspond to the maximum input size of vectors / matrices that the sparse tile can receive as input. In different implementations, the sparse tile can be configured for different dimensions R and C. R and C can be equal to or different from each other, and different sparse tiles can be implemented for different dimensions, for example in response to the nature of the data of different workloads that the sparse tile is configured to process. The sparse tiles of an array of sparse tiles can be configured to receive input within the same maximum dimension threshold.

[0066] Sparse tile 200 can be configured to receive a tile input vector 205 and a tile input matrix 210 within a predetermined address range in memory associated with sparse tile 200. For example, sparse tile 200 is configured to automatically retrieve tile input vector 205 from a first address range in coupled memory and tile input matrix 210 from the same or a different address range in memory. A device or component of a device implementing system 100 having sparse tile 200 can be configured to send a tile input matrix and a tile input vector to a location in memory corresponding to each of the one or more sparse tiles implemented by the system. For example, after generating a processing system input matrix and / or a system input vector, pre-processing engine 150 can be configured to cause the individual tile input matrix and tile input vector to be stored at an address range corresponding to each sparse tile, including sparse tile 200.

[0067] In addition to a maximum dimension threshold, sparse tile 200 is configured to also receive a tile input matrix having a non-zero count equal to or less than a predetermined maximum non-zero threshold. Like the dimension threshold, the maximum non-zero threshold can be set to different values for sparse tiles 200 and their corresponding arrays depending on different implementations. For example, if a system is configured to handle workloads whose data typically includes processing matrices having a high sparsity ratio, the system of sparse tiles can be configured with a relatively high maximum non-zero threshold. As described herein, the number of multiplier and adder circuits of a sparse tile corresponds to its maximum non-zero threshold, so a sparse tile having a relatively low maximum non-zero threshold can be built with fewer circuits relative to a sparse tile having a higher maximum non-zero threshold.

[0068] Sparse tile 200 can include a crossbar switch 215 and multiplier circuits 220. Crossbar switch 215 is configured to receive a maximum of R vector values and distribute those values across N multiplier circuits 220, where N is equal to the maximum non-zero threshold of sparse tile 200. Multiplier circuits 220A-220C, 220N are shown, but it should be understood that in different implementations, sparse tile 200 can include more or fewer multiplier circuits.

[0069] The crossbar switch 215 can be implemented as any circuit or circuits configured to receive an input and pass that input to one or more destinations, which can themselves be other circuits such as the multiplier circuits 220. The multiplier circuits 220 can be implemented according to any of a variety of different techniques for performing hardware multiplication between two operands. The first operand for the multiplier circuits can be the vector value received by the crossbar switch 215. The second operand for the multiplier circuits can be the non-zero values from the sliced input matrix 210. Each non-zero value is loaded into a respective register 221A-221C, 221N of a respective multiplier circuit 220A-220C, 220N. There are as many multiplier circuits as there are non-zero thresholds to provide available multiplier circuits for each sliced input matrix. Each multiplier circuit multiplies the non-zero value stored in its respective register with the vector value received by the crossbar switch 215.

[0070] The sparse slice 200 can also include adder circuits 225. Each adder circuit 225A-225C, 225N is configured to receive input from a corresponding multiplier circuit. In between each adder circuit is a segment marker. The adder circuits 225A-225C, 225N and segment markers 226A-226C, 226N-1 are shown in Figure 2 However, as with the multiplier circuits 220, it is understood that the number of adder circuits and segment markers can vary according to different implementations.

[0071] The adder circuits can be implemented using any technique for hardware addition of two operands. The first operand from the adder circuits can be the product received from the multiplier circuits. For example, the multiplier circuit 220A passes the product of multiplying the vector value and the non-zero value to the adder circuit 225A. The segment marker is a circuit or other hardware component configured to gate input between adjacent adder circuits depending on a gate input value.

[0072] The control values 230 can be received as input along with the sliced input matrix 210 and the sliced input vector 205 and used to configure one or more of the crossbar switch 215, the segment markers 226, and / or the crossbar switch 235. The control values 230 can be a sequence of values each corresponding to a respective non-zero value in the sliced input matrix 210. The control values 230 can include a first type of value, e.g., 1, which can correspond to a non-zero value that is the first non-zero value in its respective column in the sliced input matrix. The control values 230 can include a second type of value, such as 0, which can correspond to a zero value that precedes one or more other non-zero values in the same column of the sliced input matrix. As referenced herein with respect to FIG. 2, the control values 230 can be received as input to the crossbar switch 215, the segment markers 226, and / or the crossbar switch 235. Figures 3A-3CAs described, the control values 230 can also include one or more vectors of values that can be used to configure the sparse tile to process the tiled input matrix 210 and the tiled input vector 205 as described herein. Figure 4 and its corresponding description herein illustrate an example matrix multiplication using sparse tiling.

[0073] In some implementations, the crossbar switches 215, 235 can be implemented to utilize different maximum dimension thresholds for the sparse tiling 200. For example, when the dimensions R and C are equal or approximately equal, then the crossbar switch 235 can be implemented according to any technique for crossbar re-arranging square or approximately square inputs, for example as a Banyan network network).

[0074] Figures 3A-3C An example tiled input matrix 300A, a vector of non-zero values in the tiled input matrix 300B, and a vector of control values corresponding to the tiled input matrix 700A are illustrated. Figure 3C Additional control vectors 305C and 310C are also illustrated.

[0075] Figure 3A is an illustration of an example tiled input matrix 300A for sparse tiling. For clarity, non-zero values are illustrated as shaded cells. For clarity, indices are provided along the columns and rows of the tiled input matrix 300A and along the vectors 300B and 300C. For example, in the matrix 300A, the value at the 2nd row, 4th column (2,4) is 1.

[0076] Figure 3B is an illustration of a vector 300B of non-zero values for the example tiled input matrix 300A. The non-zero values in the vector 300B correspond to the order of occurrence of non-zero values in the tiled input matrix 300A when read from left to right, although the exact order of reading can differ according to different implementations, for example from right to left.

[0077] Figure 3C is an illustration of vectors 300C, 305C, and 310C of control values for the example tiled input matrix 300A. In some examples, the vectors 300C, 305C, and 310C can be part of the same vector according to a predetermined order that the sparse tile can be configured to use as described herein when receiving the vectors of control values and configuring different components of the sparse tile according to the vectors 300C, 305C, and 310C.

[0078] Vector 300C corresponds to control values used to configure segment markers for the sparse tile. The control value (in this example, a bit) with value 1 in vector 300C corresponds to the start of a new column in matrix 300A. The value at index 0 can be automatically set to 1 as the start of vector 300C of control values. In some implementations, the starting control value can be omitted by the processing sparse tile and assumed to be constant. Hardware implementations can simplify their circuitry by taking advantage of the fact that this value is known to be 1. The value at index 1 in vector 300C is also set to 1 to correspond to the value at index 1 in vector 300B, which is the first non-zero value in the next column of matrix 300A. The value at index 2 in vector 300C is set to 0 because it corresponds to a non-zero value in vector 300B that is not the first non-zero value in the next column.

[0079] As another example, the bit at index 3 in vector 300C is set to 1 because the corresponding non-zero value (value 1 in index 3 of vector 300B) is the first non-zero value in the next column (specifically, the 2nd column) of matrix 300A. The sequence in vector 300C follows this described pattern until all non-zero values in all columns have been represented.

[0080] Vector 305C corresponds to control values used to configure the input crossbar for the sparse tile. Vector 305C specifies the "y" coordinate of the non-zero value within submatrix 300A for each non-zero value in the input matrix 300A of the sparse tile. In this example, the "y" dimension moves vertically up and down along sparse matrix 300A, but in other examples the "y" dimension can be defined differently, e.g., horizontally. For example, the value "3" at element zero in vector 305C corresponds to the "y" coordinate of the non-zero value "1" in the first column of matrix 300A. As another example, the value "4" at element six in vector 305C corresponds to the value "1" at the bottom of the fourth column of matrix 300A.

[0081] Vector 310C corresponds to control values used to configure how the sums generated by the multiplier circuits of the sparse tile are arranged by the output crossbar to generate the tile output vector.

[0082] According to the mathematical definition of matrix multiplication, each value at position (x, y) in the sparse matrix is multiplied by the value at position y in the input vector. The result of the multiplication is added to the output at position x. The input crossbar uses vector 305C to arrange non-zero values of the same column to adjacent multiplier circuits in the sparse tile. The output crossbar uses vector 310C to arrange the computed sums in the correct order in the tile output vector, thereby representing the product of multiplying the tile input matrix and the tile input vector by the sparse tile.

[0083] Returning to Figure 2The cross switch 215 can be configured to receive control value 230 and arrange each value of the sliced ​​input vector 205 to be received as input to one or more multiplier circuits matching the corresponding sliced ​​input matrix column when the sliced ​​input vector is multiplied by the sliced ​​input matrix.

[0084] The cross switch 235 can be configured to receive one or more sums from an adder circuit and rearrange the received sums to obtain the correct output slice vector corresponding to the multiplication of the input slice matrix with the input slice vector. Like the cross switch 215, any of a variety of different techniques can be applied to implement the cross switch 235 as one or more circuits.

[0085] As described herein, segment markers are configured to gate inputs between adjacent adder circuits depending on the gate input value of the segment marker. For example, when a segment marker receives a control value of 1, it prevents the output from the first adder circuit adjacent to that segment marker from being passed as input to the second adder circuit adjacent to that segment marker. When a segment marker receives a control value of 0, it passes the output from the first adder circuit to the second adder circuit (or, in some implementations, the reverse). This configuration of segment markers corresponds to adding only the sums corresponding to the non-zero values ​​in the same column, separated from the sums of the non-zero values ​​in different columns. (See references herein.) Figure 2 As described, the one or more of them can be passed to the cross switch 235 and rearranged to generate the sliced ​​output vector 240.

[0086] The cross switch 235 can be configured to determine whether to discard or receive an input and rearrange that input to match its correct position in the sliced ​​output vector 240. For example... Figure 2 As shown, each adder circuit can pass its output to the cross switch 235 (indicated by the arrows pointing to the cross switch 235). If the next segment marker after the adder circuit is not selected, the cross switch 235 can discard the output to that adder circuit because the running sum between adjacent adder circuits has not yet finished. If the next segment marker is selected (or if there is no segment marker, in the case of the last adder circuit 225N), the column running sum has been completed, and the cross switch 235 receives the running sum as input to be part of the output slice vector 240. By tracking which sums to ignore and which sums to include as part of the slice output vector 240, the cross switch 235 can precisely track column sums to generate output vectors of varying lengths up to the maximum dimension C.

[0087] Figure 4The illustration shows an example computation of receiving a sliced ​​input matrix 410, a sliced ​​input vector 405, and sparse slices 400 for control values ​​430 of the sliced ​​input matrix 410.

[0088] Consider example values ​​for piecewise input vector 405 and piecewise input matrix 410:

[0089] [1 3 2](vector 405)[0 3 0 2 0 4 1 0 0](matrix 410).

[0090] In this example, the corresponding control vectors 430-432 for matrix 410 are:

[0091] [1 0 1 1](Control vector 430)

[0092] [1 2 0 1](Control vector 431)

[0093] [0 0 1 2](Control vector 432)

[0094] For ease of description, multiplier circuits 40A-40D are abbreviated as multiplier AD, segment markers 43A-43C are abbreviated as segment marker AC, and adder circuits 42A-42D are abbreviated as adder AD.

[0095] Based on the non-zero values ​​of matrix 410, multiplier A is loaded with the value 2, multiplier B with the value 1, multiplier C with the value 3, and multiplier D with the value 4. Note that for this example, the sparse partitioning consists of only four multipliers AD and four adders AD.

[0096] The cross switch 415 receives vector 405 with vector values ​​1, 3, and 2. Dashed lines 45A, solid lines 45B, and dashed lines 45C are shown to illustrate the path of data from the fragmented input vector 405 to the fragmented output vector 440. The cross switch 415 receives control vector 431, where each value corresponds to the “y” coordinate of the corresponding non-zero value in the fragmented input matrix 410. Values ​​in control vector 431 include 1, 2, 0, and 1. Note that the “y” coordinate values ​​range from 0 to 2 because the sparse input matrix 410 has a dimension of 3×3. Control vector 431 specifies to each multiplier in multiplier AD which value from the fragmented input vector 405 should be sent to which multiplier.

[0097] For example, the first value in control vector 431 is 1, which corresponds to the "y" coordinate of the first non-zero value in the sliced input matrix 410. Because the "y" coordinate of 1 is the second coordinate (after zero), the crossbar switch 415 routes the second value of the sliced input vector 405 to the first multiplier, here multiplier A. The second value of the control vector 431 is 2, which corresponds to the "y" coordinate of the second non-zero value in the sliced input matrix 410. The crossbar switch 405 can be configured to then route the third value of the sliced input vector 405 to multiplier B. As another example, the third value in control vector 431 is 0, which corresponds to the next non-zero value with a "y" coordinate of 0. The crossbar switch 415 routes the first value of the sliced input vector 405 to multiplier C.

[0098] For the first column of matrix 410, the crossbar switch 415 routes the value 3 to multiplier A and the value 2 to multiplier B. For the second column of matrix 410, the crossbar switch 415 routes the value 1 to multiplier C. For the third and last column of matrix 410, the crossbar switch 415 routes the value 3 to multiplier D. The products of the multipliers A-D are as follows: 6 (3 x 2) for multiplier A, 2 (2 x 1) for multiplier B, 3 (1 x 3) for multiplier C, and 12 (3 x 4) for multiplier D.

[0099] Next, the adders A-D receive the products calculated by the multipliers A-D. Adder A receives the product from multiplier A, which is 6. The first control value (1) is discarded. Adder A has no adder before it, so it passes the sum to segment tag A. Segment tag A is not gated on because the second value of control values 430 is zero. Adder B receives the current sum from adder A (6) and adds it to the product of multiplier B (2). Segment tag B is gated on, so the output of adder B (8) is passed to crossbar switch 420 (indicated by dashed line 45A). Adder C adds the product of multiplier C (3) to nothing because segment tag B gated on the output from adder B. Segment tag C is gated on, so the output of adder C (3) is passed to crossbar switch 420 (indicated by dashed line 45C). Finally, multiplier D receives the product of multiplier D (12), and because it is the last adder in the sparse slice, it automatically passes its output (12) to crossbar switch 420 (indicated by solid line 45B).

[0100] The crossbar switch 420 re-arranges the received sums 8, 3, and 12 according to the correct order for outputting the output slice vector 440. The crossbar switch 420 receives a control vector 432 with values 0, 0, 2, and 1. As discussed herein with reference to FIG. 4, the crossbar switch 420 can be configured to route the output of adder A (8) to the first position of the output slice vector 440, the output of adder C (3) to the second position of the output slice vector 440, and the output of adder D (12) to the third position of the output slice vector 440. Figures 3A-3CAs described, the values of the control vector output by crossbar 420 correspond to the "x" coordinate position of the non-zero value. Like the "y" coordinate, in this example, the values range from 0 to 2. The first two values of control vector 432 are 0. Crossbar 420 thus routes the received first sum to the first element of output vector 440. The next value in crossbar 432 following the zero is 2. Crossbar 432 is configured to route the received second sum from adder AD to the second element of output vector 440 (as shown by line 45C) and the third sum to the third element (as shown by line 45B). In some examples, crossbar 420 is configured to skip consecutive repeating control values in vector 432, e.g., as shown by vector 432 having the first two zeros. In some examples, rather than skipping consecutive repeating control values, crossbar 420 is configured to perform an inclusive OR operation on the received input sums and output the result of the inclusive OR operation to the position of output vector 440 corresponding to the consecutive repeating control values.

[0101] For example, line 46 shows a potential input source for adder A to output crossbar 420. Because segment marker 43A has a zero value, the output of adder A to crossbar 420 is suppressed, e.g., masked or set to zero. The output of adder A is instead passed through segment marker 43A to adder B. In some examples, when output crossbar 420 receives control values 432, output crossbar 420 performs an inclusive OR operation on the received first sum (through line 46, with a value of 0) and the received second sum (through line 45A, with a value of 8 from adder B). Crossbar 420 can be configured to perform the inclusive OR operation so as to output a non-zero operand. Crossbar 420 will output 8 after performing the inclusive OR operation on the received sums (e.g., 0 OR 8) and pass the result to the first position in output vector 440. In some examples, output crossbar 420 can receive individual outputs from at least some of adders A-D and calculate the sum of these individual outputs.

[0102] The adder circuit can be implemented according to any of a variety of different circuit configurations for adding the numbers defining a contiguous range of segments, each segment corresponding to a value within a respective column of the sliced input matrix processed by the corresponding sparse slice. For example, the adder circuit of sparse slice 200 or 400 can be implemented as one or more sequential segmented sum circuits (e.g., as illustrated by sparse slice 200 or 400) for performing a sequential segmented sum of the products by multiplying the non-zero values in each column with the corresponding values in the sliced input vector. In some implementations, the adder circuit can be configured to perform the sum tree as a parallel segmented sum circuit. Separate adder circuits can be configured to add their inputs in parallel and pass their sum to an output crossbar and / or to an adjacent adder circuit according to any intermediate segment marker gate value as described herein.

[0103] Parallel segmented sum circuits can reduce the latency of the circuit relative to sequential segmented sum circuits, especially when the number of terms being summed is large. The control values for gating the segment markers allow for parallel segmented summing at least because the range of values to be summed together can be tracked according to the gate value of the segment marker, which gates the input between adder circuits corresponding to values in different segments and allows the input between adder circuits corresponding to values to be added in the same segment.

[0104] In some implementations, the output crossbar 420 is configured to receive the individual sums computed by each adder and add and route these sums to corresponding elements of the output slice vector 440 using the control vector 432. If two or more sums are routed to the same element, e.g., the control vector 432 includes a copy of the same value as the first zero in vector 432, then the output crossbar is configured to add each received sum to be routed to the same element in the output slice vector 440 rather than receiving a single sum as illustrated by line 45A in Figure 4

[0105] Example Method

[0106] Figure 5 is a flowchart of an example process 500 for multiplying a submatrix of a sparse matrix on a sparse slice with a system input vector according to aspects of the present disclosure. For ease of description, the submatrix of the sparse matrix is referred to as a sliced input matrix. Reference is made to Figure 9 An example process for partitioning an input sparse matrix into a plurality of submatrices is described herein. For example, a sparse slice such as sparse slice 200 of Figure 2 Sparse slice 200 of FIG. 4A performs process 500.

[0107] ​According to block 510, the sparse tile receives a tile input matrix. The tile input matrix is within a predetermined dimension threshold and has a non-zero value count equal to or less than a predetermined maximum non-zero threshold.

[0108] According to block 520, the sparse tile receives a tile input vector comprising a plurality of vector values. The tile input vector is a subvector of the system input vector, which can be generated as part of pre-processing the input of the system by a pre-processing engine, as described herein with reference to Figure 1 and Figure 9 .

[0109] According to block 530, the sparse tile generates one or more products of respective vector values multiplied by respective non-zero values. As described herein with reference to Figure 2 and Figure 4 , the sparse tile can be configured with control values corresponding to the locations of non-zero values in the tile input matrix. Based on the control values, the sparse tile can be configured to route incoming vector values of the subvector to corresponding adjacent multiplier circuitry, stored with non-zero values along the same column of the input tile matrix. The sparse tiles described herein Figure 8A illustrate an example process for configuring a sparse tile using control values of a tile input matrix.

[0110] According to block 540, the sparse tile generates one or more sums of the one or more products. As described herein with reference to Figure 2 , the sparse tile comprises a plurality of adder circuits configured to receive inputs from corresponding multiplier circuitry. The adder circuits are further configured to add these inputs along adjacent adder circuits until a segment marker set to a gate circuit adjacent to the marker is reached. The sum of the adder circuit inputs up to the segment marker can be passed to a crossbar switch, such as Figure 2 crossbar switch 235.

[0111] The sparse tile generates a tile output vector from the one or more sums as a product of the tile input vector multiplied by the tile input matrix. The sparse tile can rearrange the received sums depending on how the operands of the products generated by the multiplier circuitry are ordered by the first crossbar switch, for example by a second crossbar switch.

[0112] Figure 6 is a flowchart of an example process 600 of generating a system output vector representing a product of a system input vector and a system input matrix from a plurality of sparse tiles. A system of sparse tiles, such as the sparse matrix multiplication system 100 of Figure 1 , can perform the process 600.

[0113] According to block 610 and for each group of sparse tiles along the column dimension of the array of sparse tiles, the tile output vectors of each sparse tile in the group are added together to generate a column output vector for the group. The following Figure 7C An example grouping of sparse tiles according to the sparse tiles of a respective tile input matrix is shown. The dimension along which the groups of sparse tiles are formed can vary according to different implementations. For example, depending on the direction in which the system input matrix and system input vector are fed into the array of sparse tiles, the groups can be along the rows, rather than the columns, of the array.

[0114] According to block 620, the system concatenates each column output vector to generate a system output vector. The system output vector is the product of multiplying the system input matrix by the system input vector. In some implementations and as referenced herein with respect to Figure 9 In more detail, the system can receive a system input matrix in which the columns of the matrix are permuted, for example to make the occurrences of non-zero values more evenly distributed across the submatrices assigned to the sparse tiles. In those implementations, the system can be configured to rearrange the elements of the concatenated system output vector to reverse the original permutation. The system can receive data defining the reordering as part of receiving the system input matrix, the control values, and the data defining the partitioning for the system input matrix.

[0115] Figures 7A-7C and the corresponding description set forth an example multiplication between the example sparse matrix 700 and the vector 750. For purposes of illustration, the multiplication is described as being performed on a system having a 4x4 array of 16 sparse tiles.

[0116] Figure 7A is an illustration of an example system input matrix 700 and system input vector 750. In this illustration, the system input matrix 700 is shown with integer values, but it should be understood that the elements can be other values, such as floating point values. In addition, in Figures 7A-7C In the illustration of the system input matrix 700 in

[0117] Figure 7B is an illustration of a partitioning of the system input matrix 700 and system input vector 750. In this example, the system input matrix 700 is partitioned into sixteen submatrices 700A-700P, one for each sparse tile of the 4x4 array. The system input vector 750 is partitioned into four subvectors 750A-750D, one for each column of the 4x4 array of sparse tiles.

[0118] One example mapping of the subvectors and submatrices to the sixteen sparse tiles, referred to as sparse tiles A-P, is shown in Table 1 below:

[0119] Sparse tiles Input sub-vectors Input sub-matrices A 750A 700A B 750A 700B C 750A 700C D 750A 700D E 750B 700E F 750B 700F G 750B 700G H 750B 700H I 750C 700I J 750C 700K K 750C 700K L 750C 700L M 750D 700M N 750D 700N O 750D 700O P 750D 700P

[0120] Table 1

[0121] Figure 7C is an illustration of a partitioned system input matrix 700 and a system output vector 770 representing a product of the system input matrix multiplied by a system input vector 750. Figure 7C Partitioned matrices grouped along columns 705A-705D are shown. As described herein with reference to Figure 6 The system can sum the slice output vectors of each column of the array of sparse slices and generate a column output vector. In Figure 7C , column output vectors 710A-710D correspond to columns 705A-705D. The system can concatenate column output vectors 710A-710D to generate a system output vector as a product of the system input matrix multiplied by the system input vector.

[0122] Figures 8A-8B is a flowchart of example processes 600A-600B for configuring sparse slices using one or more control values of an input matrix and performing matrix multiplication, according to aspects of the present disclosure.

[0123] Figure 8A is a flowchart of example process 800A for configuring sparse slices using one or more control values of a slice input matrix, according to aspects of the present disclosure. Sparse slices, such as sparse slices 200 of Figure 2 , can perform the flow of 800A.

[0124] According to block 810, the sparse slice receives one or more control values specifying locations of non-zero values along each column of the slice input matrix.

[0125] According to block 820, the sparse slice loads non-zero values of the slice input matrix into registers of the multiplier circuit. As described herein with reference to Figure 2 , the sparse slice can implement a number of multiplier circuits equal to a predetermined maximum non-zero threshold of the system. The sparse slice can load non-zero values in an order that they occur when reading the slice input matrix along a predetermined read direction, such as from left to right.

[0126] According to block 830, the sparse slice loads the one or more control values into crossbar switches of the sparse slice. A first crossbar switch, such as crossbar switch 815 of Figure 2 , can be configured to receive the control values and arrange each value of the slice input vector to be received as input to one or more multiplier circuits that match the corresponding slice input matrix array that is multiplied by the vector value as part of a matrix-vector multiplication. A second crossbar switch, such as crossbar switch 825 of Figure 2The crossbar switch 235) can be configured to receive one or more sums from the adder circuits and rearrange the received sums to obtain a correct output slice vector that corresponds to multiplying the input slice matrix with the input slice vector.

[0127] According to block 840, the sparse slice loads the one or more control values in one or more segment markers that are configured to gate inputs to the adder circuits based on values of the control values.

[0128] Figure 8B is an example process for performing matrix-vector multiplication using a sparse slice configured according to the process 800A of Figure 8A is a flowchart of an example process for generating submatrices from a system input matrix according to aspects of the present disclosure. One or more processors in one or more locations can perform the process 900. For example, a pre-processing engine (such as the pre-processing engine 150 of

[0129] According to block 850, the sparse slice receives and loads non-zero values of the input slice matrix.

[0130] According to block 860, the sparse slice receives and sends vector values of the slice input vector along with non-zero values of the same column of the slice input matrix to the multiplier circuits.

[0131] According to block 870, the sparse slice generates one or more partial sums from adjacent adder circuits that are not gated by the segment markers. As described herein with reference to Figure 2 and Figure 8A The sparse slice can aggregate sums between adder circuits that are not gated by the segment markers and use the control values to configure the segment markers to gate adder circuits representing calculations from different columns of the slice input matrix. When the sparse slice reaches a segment marker with an active gate bit, the sparse slice passes the partial sums to a second crossbar switch that is configured to rearrange the partial sums and generate a slice output vector.

[0132] According to block 880, the sparse slice generates a slice output vector from the one or more partial sums. As described herein with reference to Figure 2 and Figure 8A The crossbar switch can be configured using the control values to receive one or more partial sums from the adder circuits of the sparse slice. The crossbar switch can be further configured to rearrange the partial sums to generate a correct slice output vector that represents a product of multiplying the slice input matrix with the slice input vector.

[0133] Figure 9 is a flowchart of an example process 900 for generating submatrices from a system input matrix according to aspects of the present disclosure. One or more processors in one or more locations can perform the process 900. For example, a pre-processing engine (such as the pre-processing engine 150 of Figure 1 is a flowchart of an example process for performing matrix-vector multiplication using a sparse slice configured according to the process 800A of is a flowchart of an example process for performing matrix-vector multiplication using a sparse slice configured according to the process 800A of

[0134] According to block 910, the pre-processing engine receives a system input matrix. The system input matrix can be, for example, a matrix as illustrated by system input matrix 700. Figure 7A

[0135] According to block 920, the pre-processing engine partitions the system input matrix into a plurality of candidate sub-matrices. As part of the partitioning, the pre-processing engine can receive a parameter specifying a predetermined dimension threshold, and one or more parameter values indicating a number of sparse tiles implemented by the system receiving the sub-matrices. For example, the pre-processing engine can be configured to generate 16 candidate sub-matrices within an 8 row by 8 column dimension threshold (to obtain a 4x4 array of sparse tiles). In some examples, the pre-processing engine can receive updated parameter values, for example to pre-process input across different systems having different dimension thresholds and / or configurations of sparse tiles.

[0136] In some implementations, prior to the pre-processing engine partitioning the system input matrix, the pre-processing engine permutes columns in the system input matrix to evenly distribute non-zero values among the candidate sub-matrices. For example, if non-zero values occur more than a predetermined tolerance more frequently on one side of the input matrix than on the other, the pre-processing engine can be configured to change the order of the columns of the input matrix so that the occurrence of non-zero values is more spread out, and thus more evenly distributed among the sparse tiles after partitioning.

[0137] If the pre-processing engine performs this ordering, the pre-processing engine passes data representing the ordering as additional input to the system, for example as part of the partitioned data sent to the system having the sparse tiles. The system can be configured to reorder elements of the system output vector according to the ordering in order to match the output vector to the output of multiplying the system input matrix and the system input vector prior to the columns of the system input matrix being permuted.

[0138] Permuting the columns of the system input matrix can improve the overall rate at which the system input matrix is processed by the system of sparse tiles. For example, by permuting the columns, each sparse tile can be used more efficiently, particularly as in some examples, some sparse tiles can receive a tile input matrix having only zero values, while other tiles can receive a tile input matrix value having only non-zero values, or can receive up to a non-zero threshold of non-zero values.

[0139] According to block 930, the pre-processing engine determines whether there is a candidate sub-matrix having a count of non-zero values greater than a predetermined non-zero threshold. According to block 940, if the pre-processing engine determines that there is a candidate sub-matrix having a count of non-zero values greater than a predetermined non-zero threshold, the pre-processing engine re-partitions the sub-matrices along the same row or column as the candidate sub-matrix.

[0140] As Figure 7B ​As shown, the submatrices can be organized along columns and rows based on their location in the input matrix. For example, if the pre-processing engine determines that submatrix 700J includes a non-zero value count above the non-zero threshold, the pre-processing engine can repartition the submatrix along the rows of submatrix 700J (including submatrices 7001, 700K, and 700L) and / or along the columns of submatrix 700J (including submatrices 700B, 700F, and 700N). In performing the repartitioning, the pre-processing engine uses the predetermined dimension threshold and performs the repartitioning such that the number of candidate submatrices remains the same. For example, the pre-processing can split the candidate submatrices and redistribute the rows / columns of the submatrices along the determined rows / columns of the candidate submatrices.

[0141] According to blocks 930 and 940, after repartitioning the submatrices, the pre-processing engine again determines whether there are candidate submatrices with a non-zero value count greater than the non-zero threshold. The pre-processing engine can repeat the determination and repartitioning according to blocks 930 and 940 until it determines that there are no candidate submatrices with a non-zero value count above the maximum non-zero threshold and proceeds to block 950.

[0142] As shown in block 950, the pre-processing engine partitions the system input vector. Depending on how the bus feeding the array of sparse tiles is arranged, each subvector of the system input vector is input to each row or column of the sparse tiles. The pre-processing engine partitions the vector such that the vector dimensions can be multiplied with a matrix in the receiving sparse tile, for example, that has valid dimensions for mathematically efficient matrix multiplication.

[0143] For example as Figures 7B-7C shown in FIG. 7 and Table 1, vector 750 is partitioned into subvectors 750A-750D, which are each passed as input to one or more sparse tiles. Also in Figure 7B each dimension of each subvector 750A-750D has the correct dimension for multiplication with a corresponding submatrix 700A-700P. For example, subvector 750B is 1 x 2 (row x column) and each of submatrices 700E-700H has 2 rows, allowing for efficient matrix multiplication between subvector 750B and submatrices 700E-700H.

[0144] As shown in block 960, the pre-processing engine generates control values for each candidate submatrix. As described with reference to Figures 3A-3C the pre-processing engine can generate a vector of control values indicating the starting non-zero value of each column of the matrix. The pre-processing engine repeats this generation for each candidate submatrix, generating a corresponding control value for each submatrix.

[0145] According to block 970, the pre-processing engine outputs the control values and the candidate submatrices. The pre-processing engine can output the control values and the candidate submatrices, for example, to a memory device, such as memory 110 of FIG. 1.Figure 1 The illustrated system 100 outputs control values and data specifying partitions of a system input matrix.

[0146] Example Computing Environment

[0147] Figure 10 is a block diagram of an example computing environment implementing a sparse matrix multiplication system 100 and a preconditioning engine 150 according to aspects of the present disclosure. The preconditioning engine 150 can be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 1015. The user computing device 1012 and the server computing device 1015 can be communicatively coupled to one or more storage devices 1030 through a network 1060. The storage devices 1030 can be a combination of volatile and non-volatile memory and can be at the same or different physical locations as the computing devices 1012, 1015. For example, the storage devices 1030 can include any type of non-transitory computer-readable medium capable of storing information, such as a hard-disk drive, a solid-state drive, a tape drive, an optical storage device, a memory card, ROM, RAM, DVD, CD-ROM, writeable and read-only memories.

[0148] The server computing device 1015 can include one or more processors 1013 and memory 1014. The memory 1014 can store information accessible by the processor(s) 1013, including instructions 1021 that can be executed by the processor(s) 1013. The memory 1014 can also include data 1023 which can be retrieved, manipulated or stored by the processor(s) 1013. The memory 1014 can be one or more non-transitory computer-readable media, such as volatile and non-volatile memory. The processor(s) 1013 can include one or more central processing units (CPUs), graphical processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0149] The server computing device 1015 can implement the sparse matrix multiplication system 100 in hardware, for example as a system on a chip. The system 100 can be implemented as part of a physical chip that is plugged into or installed into the server computing device 1015. The system 100 is configured to communicate with other components of the server computing device 1015.

[0150] Instructions 1021 can include one or more instructions that, when executed by processor 1013, cause the one or more processors to perform actions defined by the instructions. Instructions 1021 can be stored in object code format for direct processing by processor 1013, or in other format including as interpreted scripts or independent source code modules that are compiled just in time or precompiled in advance. Instructions 1021 can include instructions to implement sparse shards 400 consistent with aspects of the present disclosure. Preprocessor engine 105 can be executed using processor 1013 and / or using other processors remote from server computing device 1015.

[0151] Processor 1013 can retrieve, store, or modify data 1023 in accordance with instructions 1021. Data 1023 can be stored in computer registers, in a relational or non-relational database as a table with a plurality of different fields and records, or as a JSON, YAML, proto, or XML document. Data 1023 can also be formatted as computer-readable formatted such as, but not limited to, binary values, ASCII, or Unicode. Moreover, data 1023 can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories such as other network locations, or information that is used by a function to calculate relevant data.

[0152] User computing device 1012 can also be configured similar to server computing device 1015, configured with one or more processors 1016, memory 1017, instructions 1018, and data 1019. User computing device 1012 can also include user output 1026, and user input 1024. User input 1024 can include any suitable mechanism or technology for receiving input from a user, such as a keyboard, mouse, mechanical actuator, soft actuator, touch screen, microphone, and sensor.

[0153] Server computing device 1015 can be configured to transmit data to user computing device 1012, and user computing device 1012 can be configured to display at least a portion of the received data on a display implemented as part of user output 1026. User output 1026 can also be used to display an interface between user computing device 1012 and server computing device 1015. User output 1026 can alternatively or additionally include one or more speakers, transducers, or other audio output, haptic interface, or other tactile feedback that provides non-visual and non-audible information to a platform user of user computing device 1012.

[0154] While Figure 10Processors 1013, 1016 and memories 1014, 1017 are illustrated as being within computing devices 1015, 1012. However, the components including processors 1013, 1016 and memories 1014, 1017 described herein may include multiple processors and memories that may be in different physical locations and operate outside the same computing device. For example, some of instructions 1021, 1018 and data 1023, 1019 may be stored on a removable SD card, while others may be stored within a read-only computer chip. Some or all of the instructions and data may be stored physically away from processors 1013, 1016 but still accessible by them. Similarly, processors 1013, 1016 may include a collection of processors capable of performing concurrent and / or sequential operations. Computing devices 1015, 1012 may each include one or more internal clocks that provide timing information for time measurement of operations and programs run by computing devices 1015, 1012.

[0155] Server computing device 1015 can be configured to receive requests for processing data from user computing device 1012. For example, environment 1000 can be part of a computing platform configured to provide various services to users through various user interfaces and / or APIs that expose platform services. As part of executing services, server computing device 1015 can use system 100 to process incoming data. For example, if the service is training a machine learning model, server computing device 1015 can be configured to use system 100 to perform multiplication operations as part of training the machine learning model.

[0156] Devices 1012 and 1015 can communicate directly and indirectly through network 1060. Devices 1015 and 1012 can be configured to accept listening sockets for initiating connections to send and receive information. Network 1060 itself can include various configurations and protocols, including the Internet, World Wide Web, Intranet, Virtual Private Network, Wide Area Network, Local Area Network, and private networks using one or more company-proprietary communication protocols. Network 1060 can support various short-range and long-range connections. Short-range and long-range connections can be made at different bandwidths, such as 2.402 GHz to 2.480 GHz (typically...). (Standard-related), 2.4GHz and 5GHz (usually associated with) (related to communication protocols); or using various communication standards, such as those used for wireless broadband communication. Standard. Alternatively, network 1060 may also support wired connections between devices 1012 and 1015, including via various types of Ethernet connections.

[0157] AlthoughFigure 10 While a single server computing device 1015 and user computing device 1012 are shown in the middle, it should be understood that aspects of the present disclosure can be implemented according to a variety of different configurations and numbers of computing devices, including in examples for sequential or parallel processing, or through a distributed network of multiple devices. In some implementations, aspects of the present disclosure can be executed on a single device and any combination thereof. Moreover, while the pre-processing engine and sparse matrix multiplication system 100 are shown as implemented on the same server computing device 1015, in some implementations, the pre-processing engine 150 is implemented on one or more server computing devices separate from the server computing device 1015 and / or user computing device 1012.

[0158] Aspects of the present disclosure can be implemented in digital electronic circuitry, in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of the same. Aspects of the present disclosure can be implemented using one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable storage medium for execution by, or to control the operation of, computer software, firmware, or hardware. The computer-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more of them. The computer-readable storage medium can be a non-transitory computer-readable storage medium with a short shelf life, such as one or more instructions capable of being executed by a computing device and stored on a tangible storage device.

[0159] In this specification, the phrase "configured to" is used in the context of different aspects of a computer system, hardware, or computer program, engine, or module. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed on the system that, when run, causes the system to perform the one or more operations. When some hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, when operating, receive input and produce output corresponding to the one or more operations from the input. When a computer program, engine, or module is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform the one or more operations.

[0160] While operations are shown in a particular order in the drawings and recited in claims, it is understood that the operations can be performed in different orders than those shown, and that some operations can be omitted, performed more than once, and / or in parallel and / or concurrently with other operations. Moreover, the separation of different system components in the manner described does not require such separation in all implementations. The described components, modules, programs, and engines can be integrated together in a single system, or spread over several systems.

[0161] Unless otherwise indicated, the alternative examples described above are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. As such, the foregoing description of the examples shall be understood to be illustrative only rather than limiting of the subject matter defined by the claims. As such, other variations and combinations of the features discussed above are possible in light of this disclosure, and brary of possible implementations, and the examples provided should not be construed as limiting the subject matter defined by the claims to only the one described example. Furthermore, the described examples should not be construed as limiting, but rather as a description of the embodiments. Accordingly, the examples are not intended to limit the scope of the claims, but rather to illustrate various combinations in which features can be employed.

Claims

1. A system for sparse matrix multiplication, the system comprising: a sparse tile comprising a plurality of multiplier circuits, wherein the sparse tile is configured to: receive a tile input matrix comprising a number of non-zero values equal to or less than a predetermined maximum non-zero threshold and having dimensions equal to or less than a predetermined dimension threshold, the non-zero threshold corresponding to a number of multiplier circuits of the sparse tile, the dimension threshold corresponding to a maximum matrix input size that the sparse tile can process; receive a tile input vector comprising a plurality of vector values; for each of the plurality of multiplier circuits, receive a respective non-zero value of the tile input matrix; generate, by the plurality of multiplier circuits, one or more products of a vector value multiplied by the respective non-zero value of the tile input matrix; and generate, as an output of the sparse tile and using the one or more products, a tile output vector, the tile output vector being a product of applying the tile input vector to the tile input matrix.

2. The system of claim 1, wherein, a length of the tile output vector is greater than one.

3. The system of claim 1, wherein, the system comprises a plurality of sparse tiles and the sparse tile is one of the plurality of sparse tiles, the plurality of sparse tiles being configured to: receive a plurality of tile input matrices as sub-matrices of a system input matrix; receive a plurality of tile input vectors as sub-vectors of a system input vector; and generate, by the plurality of sparse tiles, a system output vector representing a product of applying the system input vector to the system input matrix.

4. The system of claim 3, the plurality of sparse tiles are arranged as a systolic array, the systolic array comprising one or more groups of sparse tiles along a column dimension of the systolic array; and wherein wherein, to generate the system output vector, the plurality of sparse tiles are further configured to: for each group along the column dimension of the systolic array, sum each sparse tile in the group with a respective tile output vector to generate a respective column output vector; and concatenate the respective column output vectors of each group to generate the system output vector. each multiplier circuit is coupled to a respective register comprising the respective non-zero value from the respective tile input matrix of the sparse tile.

5. The system of claim 1, wherein, a number of multiplier circuits in the plurality of multiplier circuits is equal to the predetermined maximum non-zero threshold.

6. The system of claim 5, wherein, the sparse tile further comprises a crossbar circuit, and wherein the crossbar circuit is configured to:

7. The system of claim 5, wherein, receive the plurality of vector values of the tile input vector; and send a vector value of the plurality of vector values as input to each of the plurality of multiplier circuits. the sparse tile is further configured to:

8. The system of claim 7, wherein, load non-zero values of a same column in the tile input matrix in registers of adjacent multiplier circuits in the plurality of multiplier circuits.

9. The system of claim 8, ​ wherein, The sparse slice is further configured to receive one or more control values specifying at least positions of non-zero values along each column of the slice input matrix; and wherein the crossbar switch circuit of the sparse slice is further configured to: receive the one or more control values; and send, to an adjacent multiplier circuit, a vector value to be multiplied with a non-zero value along a same column of the slice input matrix according to the one or more control values.

10. The system of claim 9, wherein, the sparse slice further comprising a plurality of adder circuits, wherein the sparse slice further comprises one or more segment markers, wherein each segment marker is configured to gate input to a respective adder circuit of the plurality of adder circuits based on a value of a respective control value loaded in the segment marker; and wherein the sparse slice is further configured to: load at least a portion of the one or more control values in the one or more segment markers, wherein an adder circuit for a non-zero value in a first column of the slice input matrix is gated by receiving input from an adjacent adder circuit, the input comprising a non-zero value in a second column of the slice input matrix different from the first column; and generate, by the plurality of adder circuits, one or more sums of the one or more products, wherein each sum of the one or more sums is a respective segment sum of one or more non-zero values of a column of the slice input matrix multiplied by one or more respective values of the slice input vector.

11. The system of claim 10, wherein the plurality of adder circuits form a parallel segmented sum circuit, and wherein each segment sum of the one or more segment sums is a sum of outputs to adjacent adder circuits not gated by a segment marker.

12. The system of claim 10, wherein the crossbar switch circuit is a first crossbar switch circuit; and wherein the sparse slice further comprises a second crossbar switch circuit configured to: receive the one or more segment sums, and arrange the one or more segment sums to generate a respective slice output vector of the sparse slice according to the one or more control values.

13. The system of claim 12, wherein, The second crossbar circuit forms a Benes network, and wherein the tile input matrix is a square matrix.

14. A method for sparse matrix multiplication, comprising: receiving, by a sparse slice comprising a plurality of multiplier circuits, a slice input matrix and a slice input vector, wherein the slice input matrix comprises a number of non-zero values equal to or less than a predetermined maximum non-zero threshold and has a dimension equal to or less than a predetermined dimension threshold, the non-zero threshold corresponding to a number of multiplier circuits of the sparse slice, the dimension threshold corresponding to a maximum matrix input size that the sparse slice is capable of processing, the slice input vector comprising a plurality of vector values; for each multiplier circuit of the plurality of multiplier circuits, receiving a respective non-zero value of the slice input matrix; generating, by the plurality of multiplier circuits of the sparse slice, one or more products of a respective vector value multiplied by the respective non-zero value of the slice input matrix; and generating a tile output vector as output of the sparse tile and using the one or more products, the tile output vector being a product of applying the tile input vector to the tile input matrix.

15. The method of claim 14, wherein, generating the tile output vector includes generating the tile output vector having a length greater than one.

16. The method of claim 14, wherein, the sparse tile is one of a plurality of sparse tiles in a system, and the plurality of sparse tiles are configured to: receive a plurality of tile input matrices as sub-matrices of a system input matrix; receive a plurality of tile input vectors as sub-vectors of a system input vector; and generate, by the plurality of sparse tiles, a system output vector, the system output vector representing a product of applying the system input vector to the system input matrix.

17. The method of claim 16, wherein the plurality of sparse tiles are arranged as a systolic array, the systolic array including one or more groups of sparse tiles along a column dimension of the systolic array; and wherein generating the system output vector includes: for each group along the column dimension of the systolic array, adding together each sparse tile in the group with a respective tile output vector to generate a respective column output vector; and concatenating the respective column output vectors of each group to generate the system output vector.

18. The method of claim 14, wherein, each multiplier circuit is coupled to a respective register, the respective register including the respective non-zero value from the respective tile input matrix of the sparse tile.

19. The method of claim 18, wherein, a number of multiplier circuits in the plurality of multiplier circuits is equal to the predetermined maximum non-zero threshold.

20. One or more non-transitory computer-readable storage media storing instructions that, when executed by a system comprising a plurality of sparse tiles, cause the system to perform operations comprising: receiving, by a sparse tile of the plurality of sparse tiles comprising a plurality of multiplier circuits, a tile input matrix and a tile input vector, wherein the tile input matrix includes a number of non-zero values equal to or less than a predetermined maximum non-zero threshold and has a dimension equal to or less than a predetermined dimension threshold, the non-zero threshold corresponding to a number of multiplier circuits of the sparse tile, the dimension threshold corresponding to a maximum matrix input size that the sparse tile is capable of processing, the tile input vector including a plurality of vector values; for each of the plurality of multiplier circuits, receiving a respective non-zero value of the tile input matrix; generating, by the plurality of multiplier circuits of the sparse tile, one or more products of a respective vector value multiplied by the respective non-zero value; and generating a tile output vector as output of the sparse tile and using the one or more products, the tile output vector being a product of applying the tile input vector to the tile input matrix.

Images