Sparse Matrix Multiplication in Hardware

Abstract

To provide a sparse shard system for sparse matrix multiplication capable of efficiently processing an input submatrix of an arbitrary size up to a predetermined dimension threshold and outputting a product to an input as a vector of a corresponding length, a method, and a device including computer readable storage media.SOLUTION: A sparse shard matrix multiplication system includes an array 101 of sparse shards 101A to 101P. Each sparse shard receives an input sub-matrix and an input sub-vector. The input sub-matrix has a non-zero value whose number is less than or equal to a predetermined maximum non-zero threshold. The sparse shard may calculate, by a plurality of multiplication circuits, one or more products obtained by multiplying a vector value by each non-zero value of the input submatrix. The sparse shard may generate a shard output vector that is a product obtained by applying a shard input vector to a shard input matrix by using one or more products as an output to the sparse shard.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] background A sparse matrix is ​​a matrix that has a higher percentage of zero values ​​than non-zero values ​​as its elements. Different sparse matrices can have different degrees of sparsity based on the percentage of zero values ​​to non-zero values. A matrix that has a higher percentage of zero values ​​than non-zero values ​​is said to be more sparse than a matrix that has a lower percentage of zero values ​​than non-zero values.

[0002] A neural network is a machine learning model that includes one or more nonlinear computational layers for predicting an output for a received input. In addition to an input layer and an output layer, some neural networks include one or more hidden layers. The output of each hidden layer can be input to another hidden layer or the output layer of the neural network. Each layer of a neural network can generate a respective output from a received input according to the values ​​of one or more model parameters of that layer. The model parameters may be weights or biases that are determined through a training algorithm to ensure that the neural network generates an accurate output. The model parameter values ​​of a layer of a neural network can be represented as elements of a matrix or tensor. Summary of the Invention [Means for solving the problem]

[0003] Quick Overview Aspects of the present disclosure relate to sparse matrix dense vector multiplication in hardware.

[0004] One aspect of the present disclosure provides a system including a sparse shard including a plurality of multiplier circuits, the sparse shard configured to receive a shard input matrix including a number of non-zero values ​​less than or equal to a predetermined maximum non-zero threshold, receive a shard input vector including a plurality of vector values, receive, for each of the multiplier circuits, a respective non-zero value of the shard input matrix, generate, by the plurality of multiplier circuits, one or more products of vector values ​​multiplied by the respective non-zero values ​​of the shard input matrix, and generate, as output to the sparse shard, a shard output vector that is the product of the shard input vector applied to the shard input matrix using the one or more products.

[0005] Another aspect of the present disclosure provides one or more non-transitory computer-readable storage media having stored thereon instructions that, when executed by a system including a plurality of sparse shards, cause the system to perform operations including: receiving, by a sparse shard including a plurality of multiplier circuits, a shard input matrix including a number of non-zero values ​​less than or equal to a predetermined maximum non-zero threshold, and a shard input vector including a plurality of vector values; receiving, for each of the multiplier circuits, each of the non-zero values ​​of the shard input matrix; generating, by the plurality of multiplier circuits of the sparse shard, one or more products of each vector value multiplied by each of the non-zero values; and generating, as output to the sparse shard, a shard output vector that is a product of the shard input vector applied to the shard input matrix using the one or more products.

[0006] Another aspect of the present disclosure provides a method, the method comprising: a sparse shard including a plurality of multiplier circuits receiving a shard input matrix including a number of non-zero values ​​less than or equal to a predetermined maximum non-zero threshold; and a shard input vector including a plurality of vector values; each of the multiplier circuits receiving each of the non-zero values ​​of the shard input matrix; and the plurality of multiplier circuits of the sparse shard multiplying each of the vector values ​​by the non-zero values ​​of the shard input matrix. generating one or more products by multiplying each of the non-zero values, and the sparse shard generating, as an output to the sparse shard, a shard output vector that is a product of applying the shard input vector to the shard input matrix using the one or more products.

[0007] Each of the above and other aspects may optionally include one or more of the following features, alone or in combination: One implementation may include all of the following features in combination.

[0008] The length of the shard output vector is greater than 1. The sparse shard is one of a plurality of sparse shards configured to receive a plurality of shard input matrices that are submatrices of a system input matrix, and a plurality of shard input vectors that are subvectors of a system input vector, and to generate, by the plurality of sparse shards, a system output vector that represents the product of the system input vectors applied to the system input matrix.

[0009] The plurality of sparse shards are arranged as a systolic array, the systolic array including one or more groups of sparse shards along a column dimension of the systolic array, and to generate the system output vector, the one or more processors are further configured to: for each group along the column dimension of the systolic array, sum the shard output vectors of each sparse shard in the group to generate a respective column output vector, and concatenate the respective column output vectors of each group to generate the system output vector.

[0010] Each multiplier circuit is coupled to a respective register containing the respective non-zero values ​​from each of the shard input matrices for the sparse shard.

[0011] The number of multiplier circuits in the plurality of multiplier circuits is equal to the predetermined maximum non-zero threshold value.

[0012] The sparse shard further includes a crossbar circuit, and the sparse shard is further configured to receive the vector values ​​of the shard input vector by the crossbar circuit and to send one vector value of the vector values ​​by the crossbar circuit as an input to each of the multiplication circuits.

[0013] The sparse shard is further configured to load non-zero values ​​in the same column of the shard input matrix into registers of adjacent multiplier circuits of the plurality of multiplier circuits.

[0014] The sparse shard is further configured to receive one or more control values ​​that specify at least the location of non-zero values ​​along each column of the shard input matrix, and the crossbar circuit of the sparse shard is further configured to receive the one or more control values ​​and send vector values ​​to adjacent multiplication circuits to be multiplied by non-zero values ​​along the same column of the shard input matrix in accordance with the one or more control values.

[0015] The sparse shard further includes a plurality of summation circuits, the sparse shard further includes one or more segment markers, each segment marker configured to gate an input to a respective one of the plurality of summation circuits based on a value of a respective control value loaded into the segment marker, the sparse shard is further configured to load at least a portion of the one or more control values ​​into the one or more segment markers, and the summation circuit for a non-zero value in a first column of the shard input matrix gates an input to an adjacent summation circuit that includes a non-zero value in a second column of the shard input matrix that is different from the first column. The sparse shard is gated to receive circuit input, and the sparse shard is further configured to generate, by the plurality of summation circuits, one or more sums of the one or more products, each of the one or more sums being a respective segmented sum of one or more non-zero values ​​in a column of the shard input matrix multiplied by one or more respective values ​​in the shard input vector.

[0016] The plurality of summing circuits form parallel segmented summing circuits, each of one or more of the segmented summing circuits being a sum of the outputs to adjacent summing circuits that are not gated by a segment marker.

[0017] The crossbar circuit is a first crossbar circuit, and the sparse shard further includes a second crossbar circuit configured to receive one or more of the segmented sums and arrange the one or more segmented sums according to the one or more control values ​​to generate each of the shard output vectors for the sparse shard.

[0018] The second crossbar circuit may form a Benesh network, and the shard input matrix is ​​a square matrix.

[0019] Another aspect of the present disclosure provides a system including one or more processors and one or more memory devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including receiving an input matrix including zero values ​​and non-zero values ​​and partitioning the input matrix into a plurality of sub-matrices, the number of non-zero values ​​in each sub-matrix being less than or equal to a predetermined maximum non-zero threshold, and the dimension of each sub-matrix being less than or equal to a predetermined dimension threshold.

[0020] The operations may further include generating, for each sub-matrix, one or more respective control values ​​that specify the location of non-zero values ​​along each column of the sub-matrix.

[0021] The operations may further include sending each submatrix and the one or more respective control values ​​for each submatrix to a plurality of sparse shards configured to process each submatrix and the one or more respective control values ​​for the submatrix. Each of the plurality of sparse shards is configured to receive a submatrix, the one or more respective control values ​​for the submatrix, and at least a portion of an input vector, and to generate a respective output shard vector representing a product of the submatrix and the portion of the input vector. The system of claim 21, wherein partitioning the input matrix includes partitioning the matrix into a number of submatrices equal to the number of sparse shards in the plurality of sparse shards.

[0022] Other aspects of the present disclosure include corresponding systems, devices, and computer programs stored on one or more non-transitory computer-readable storage media. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a block diagram of an example of a system including an array of sparse shards, according to aspects of the present disclosure. [Figure 2] FIG. 1 is a block diagram of an example of a sparse shard, according to aspects of the present disclosure. [Figure 3A] FIG. 10 illustrates an example of a shard input matrix for a sparse shard. [Figure 3B] FIG. 10 illustrates a vector of non-zero values ​​for an example shard input matrix. [Figure 3C] FIG. 10 illustrates a vector of control values ​​for an example shard input matrix. [Figure 4] FIG. 10 illustrates an example of matrix multiplication with sparse shards that receives a shard input matrix, a shard input vector, and a control value for the shard input matrix. [Figure 5] FIG. 10 is a flow diagram of an example process for multiplying a submatrix of a sparse matrix on a sparse shard by a system input vector according to aspects of the present disclosure. [Figure 6]FIG. 10 is a flow diagram of an example process for generating a system output vector that represents the product of system input vectors from multiple sparse shards and a system input matrix. [Figure 7A] FIG. 2 is a diagram illustrating an example of a system input matrix and a system input vector. [Figure 7B] FIG. 2 illustrates a system input matrix and a partition of a system input vector. [Figure 7C] FIG. 1 illustrates a partitioned system input matrix and a system output vector representing the product of the system input matrix multiplied by the system input vector. [Figure 8A] FIG. 10 is a flow diagram of an example process for configuring sparse shards using one or more control values ​​of a shard input matrix according to aspects of the disclosure. [Figure 8B] FIG. 8B is a flow diagram of an example process for performing matrix-vector multiplication using sparse shards configured according to the process of FIG. 8A. [Figure 9] FIG. 10 is a flow diagram of an example process for generating submatrices from a system input matrix according to aspects of the present disclosure. [Figure 10] FIG. 1 is a block diagram of an example computing environment for implementing a sparse matrix multiplication system according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0024] Detailed Description overview Aspects of the present disclosure relate to a system including one or more integrated circuits configured for sparse matrix-dense vector multiplication. A system of multiple sparse shards can receive submatrices and subvectors of input system matrices and vectors, with each sparse shard being capable of implementing multiple arithmetic units, such as multiplication and addition circuits. Each sparse shard can be at least a portion of an integrated circuit and can implement multiple arithmetic units, such as multiplication and addition circuits. Each sparse shard is configured to receive a submatrix having a number of non-zero values ​​less than or equal to a maximum non-zero threshold, which can be predetermined, for example, when the system is implemented as a chip including the sparse shards and, optionally, one or more other components as described herein.

[0025] Each sparse shard can further receive metadata, such as the control values ​​described herein, that the sparse shard can use to direct different inputs to each multiplier or adder circuit and gate when the output from one unit is passed to another. By configuring the sparse shard according to the received metadata, the sparse shard can efficiently process input submatrices of any size up to a predetermined dimensionality threshold and output products as vectors of the corresponding length to the input.

[0026] As part of receiving the submatrix, each sparse shard can receive one or more control values ​​that represent the positions of non-zero values ​​in each column of the submatrix. Using the one or more control values, the sparse shard can be configured to adjust how the individual values ​​of the submatrix and subvector are to be multiplied, added, and arranged such that the sparse shard generates a shard output vector that represents the product of the submatrix and the subvector. The system can further be configured to generate a system output vector from the shard output vector of each shard that represents the product of applying the system input vector to the system input matrix, for example, by multiplying the system input vector by the system input matrix.

[0027] Aspects of the present disclosure also relate to a system for preprocessing a sparse matrix for processing by an array of sparse shards. A system of one or more processors preprocesses an input matrix into a plurality of shards. The system may be configured to partition the sparse shard into a number of submatrices, each with a dimension less than or equal to a predetermined dimensionality threshold that specifies the maximum matrix input size for the sparse shard. As part of the partitioning, the system may identify whether any of the submatrices includes a number of non-zero values ​​greater than a predetermined maximum non-zero threshold and, accordingly, repartition the submatrix along the same row or column as the identified submatrix. The system may repeat this process until it has generated a submatrix for each sparse shard such that the dimension of the submatrix is ​​within the dimensionality threshold and the number of non-zero values ​​in the submatrix is ​​less than or equal to the predetermined non-zero threshold.

[0028] Systems implemented according to aspects of the present disclosure can more efficiently perform workloads involving repeated multiplication of sparse matrices and vectors. For example, a device implementing a system-on-a-chip (SoC) according to aspects of the present disclosure can generate a product of a sparse matrix multiplied by a vector in fewer processing cycles than conventional approaches, at least because redundant "multiply-by-zero" calculations of zero elements in the matrix that do not contribute to the final product are omitted. While other approaches may experience increased performance loss for large matrices with high sparse ratios, systems implemented as described herein can more efficiently calculate products using these matrices, at least because a greater proportion of redundant calculations are omitted as the sparse ratio of the input matrices increases.

[0029] Some workloads, such as running or training neural networks, rely heavily on performing matrix multiplication. While hardware accelerators or other devices can efficiently perform certain operations such as matrix multiplication, they often have limited processing requirements. For example, the device may require a predefined sparsity ratio between zero and non-zero values ​​in the matrix, or may be severely constrained in the size of the inputs it processes. While matrix multiplication is a ubiquitous type of computation for many neural network workloads, accelerators with limited processing requirements are limited in the number of workload types they can support, such as workloads involving matrix multiplication of matrices with varying sparsity ratios.

[0030] Aspects of the present disclosure provide a system for sparse matrix multiplication that can flexibly process matrices of different sizes and sparsity ratios. A sparse shard in a system of sparse shards configured for sparse matrix multiplication can discard zero values ​​in a given input submatrix while keeping track of the positions of non-zero values ​​in the input submatrix across multiple columns. The sparse shard is configured with only as many registers and multiplication circuits as a predetermined maximum non-zero threshold, and can store only the non-zero values ​​in the input submatrix in memory. As a result, the sparse shard can store and process input data more efficiently and with fewer resources than storing a full-size matrix.

[0031] Furthermore, the sparse shard may be configured to arrange the non-zero values ​​and their products multiplied by values ​​of the input vector along adjacent multiplication and addition circuits, respectively. The sparse shard may be configured to efficiently generate segmented sums for each column of the input matrix by combining the outputs of these adjacent circuits and rearrange the segmented sums to generate the shard output vector. The sparse shard can preserve the order of the non-zero values ​​and accurately add and multiply the non-zero values ​​by corresponding values ​​of the input vector to generate the shard output vector exactly as the product of the input submatrix and the input vector. The sparse shard can perform multiplication on an input matrix without requiring preprocessing operations to change the shape of the input matrix.

[0032] The system may also include one or more processors configured to preprocess the sparse matrix for multiplication with the input vector by a plurality of sparse shards. The system may generate a different number of submatrices depending on the number of available sparse shards. , and different configurations of sparse shards can be supported, such as generating submatrices for different maximum non-zero thresholds.

[0033] Aspects of the disclosure described herein may be implemented by a system including one or more processors and multiple sparse shards, for example, as a chip installed in a computing device, such as a server computing device in a data center of a computing platform.

[0034] System Example 1 is a block diagram of an example system 100 including an array 101 of sparse shards 101A-P. Array 101 may be at least part of a sparse matrix multiplication system, such as sparse matrix multiplication system 100, configured to perform matrix multiplication on matrix inputs.

[0035] As described in more detail herein with reference to Figure 2, a sparse shard is a collection of circuits configured to perform arithmetic operations, to arrange inputs and outputs between circuits that perform arithmetic operations, and / or to gate inputs and outputs between other circuits. For example, a sparse shard may be configured to perform matrix multiplication between a rectangular shard or portion of a sparse matrix and a linear shard or piece of a vector. As described in more detail herein, a sparse shard can receive various different shards or portions of a sparse matrix having any number of non-zero values ​​up to a predetermined maximum threshold.

[0036] The collection of circuits may be configured as described herein to control the state (i.e., open or closed) of each gate, the operands computed, and / or the ordering of the inputs and outputs to and from each circuit. This configuration may depend, at least in part, on the dimensions of the shard input matrix that the sparse shard is configured to receive as input and / or the ordering and number of non-zero values ​​in the shard input matrix. Each sparse shard may be an individual circuit component for a system configured to multiply a sparse matrix by a vector.

[0037] Each sparse shard 101A-P is configured to communicate with two or more other sparse shards in the array 101. A sparse shard can communicate with its immediate neighbor, e.g., the sparse shard immediately preceding or succeeding it along a dimension defined by the rectangular arrangement of the sparse shards 101A-P. The connections between sparse shards can be realized, for example, via buses or one or more circuit interconnects that physically connect the sparse shard to its neighboring sparse shards. Each sparse shard 101A-P can be realized as one or more circuits configured to receive at least a portion of a matrix and at least a portion of a vector and to generate an output vector representing the product of the input matrix and the vector. In some examples, the sparse shards 101A-P are organized as a systolic array, although in various implementations, the array 101 is generally configured or arranged according to a rectangular arrangement of the sparse shards 101A-P.

[0038] Array 101 may be at least part of a system-on-chip that implements multiple components and integrated circuits on a circuit board or other material. Array 101 may be installed as part of a computing device and configured to interact with other components of the device, such as memory, a processor, network components, and / or peripherals. For example, array 101 may be installed as part of system 100. The system 100 may receive a system input vector 105 and a system input matrix 110 from one or more memory devices implemented using a memory controller. The system 100 may generate, as an output, a system output vector 115 that represents the product of the system input vector 105 and the system input matrix 110.

[0039] In some examples, the system output vector 115 may be provided as an input to other devices or components of devices that implement the system 100. For example, if the system output vector 115 is a product of model parameter values ​​multiplied by a vector input for a neural network, the system output vector 115 may be provided as an input to one or more processors configured to calculate an activation function for the output vector 115.

[0040] System 100 can receive system input vector 105 and system input matrix 110 from preprocessing engine 150. Preprocessing engine 150 can be implemented on one or more computing devices, which may or may not be the device implementing system 100. System input matrix 110, by way of example, can include values ​​representing at least a portion of model parameter values ​​for a neural network. System input matrix 110 can also be part of a more complex data structure, such as a multidimensional array or tensor. System 100 can be configured to receive matrices corresponding to at least a portion of a larger data structure, such as a three-dimensional tensor or matrix, and generate corresponding outputs by multiplying the matrices by system input vector 105. System input vector 105 can be, for example, an input to a trained neural network whose model parameter values ​​are represented at least in part by the values ​​of system input matrix 110.

[0041] Pre-processing engine 150 may be configured to process system input matrix 110 to generate one or more control values ​​111 and matrix partition data 112. Pre-processing engine 150 may receive system input matrix 110 in a variety of different formats for storing sparse matrices, such as in its complete row-column form (with all non-zero and zero values). As another example, pre-processing engine 150 may receive system input matrix 110 according to a compressed sparse column format, a coordinate list format, or any of a variety of other formats for storing sparse matrices.

[0042] In some instances where pre-processing engine 150 receives system input matrix 110 in its entirety, pre-processing engine 150 is configured to convert system input matrix 110 into a predetermined format suitable for storage in memory. For example, pre-processing engine 150 may remove zero values ​​from the matrix and generate control values ​​to track the position of non-zero values ​​relative to their original position in the matrix before converting to this predetermined format.

[0043] The control values ​​are used to configure each sparse shard to receive a subvector of the system input vector 105 and multiply it by a submatrix of the system input matrix 110, as described in more detail herein with reference to Figure 2. Partition data 112 is data that specifies how the system input matrix 110 should be partitioned into submatrices. Each submatrix is ​​received as an input at a respective sparse shard, and the partition data 112 specifies the partitioning for as many submatrices as there are sparse shards in the array 101.

[0044] The pre-processing engine 150 may be configured to generate vector partition data 106. As described in more detail with reference to FIGS. 5A-5C, Sparse shards along can take the same subvector as input to multiply with their respective submatrices.

[0045] Although system input matrix 110 and system input vector 105 are shown as being provided along the left and right sides of system 100, the exact positioning and orientation of buses providing input data to system 100 may vary between implementations. For example, based on the location of other components on the same chip as system 100, the buses or circuit interconnects for providing inputs to and receiving outputs from system 100 may be oriented or positioned differently to account for the locations of those other components.

[0046] 2 is a block diagram of an example sparse shard 200 according to aspects of the present disclosure. For example, each of the sparse shards 101A-P of the system 100 may be implemented as described herein with reference to the sparse shard 200.

[0047] The sparse shard 200 is configured to receive a shard input vector 205 and a shard input matrix 210. The shard input vector 205 may include one or more vector values ​​and have a maximum dimension of 1×R. The shard input matrix includes one or more zero values ​​and one or more non-zero values ​​and has a maximum dimension of R×C. R (row) and C (column) are predetermined dimensionality thresholds corresponding to the maximum vector / matrix input size that the sparse shard can receive as input. In different implementations, the sparse shards may be configured with different dimensions R and C. R and C may be equal or different from each other, and different sparse shards may be implemented for different dimensions depending on, for example, the nature of the data for different workloads that the sparse shards are configured to process. The sparse shards of an array of sparse shards may be configured to receive inputs within the same maximum dimensionality threshold.

[0048] A sparse shard 200 may be configured to receive a shard input vector 205 and a shard input matrix 210 within a predetermined address range in a memory associated with the sparse shard 200. For example, the sparse shard 200 is configured to automatically retrieve the shard input vector 205 from a first address range in the combined memory and retrieve the shard input matrix 210 from the same or a different address range in the memory. A device or component of a device implementing the system 100 with the sparse shard 200 may be configured to send the shard input matrix and vector to a location in memory corresponding to each of the one or more sparse shards implemented by the system. For example, the pre-processing engine 150 may be configured to process and generate the system input matrix and / or system input vector, and then store the individual shard input matrix and shard input vector in an address range corresponding to each sparse shard, including the sparse shard 200.

[0049] In addition to the maximum dimension threshold, the sparse shard 200 is configured to accept shard input matrices with a non-zero count less than or equal to a predetermined maximum non-zero threshold. Like the dimension threshold, the maximum non-zero threshold may be set to different values ​​depending on the implementation of the sparse shard 200 and its corresponding array. For example, if the workload data the system is configured to process generally includes processing matrices with a high sparse ratio, the sparse shard system may be configured with a relatively high maximum non-zero threshold. As described herein, the number of multiplication and addition circuits in a sparse shard corresponds to its maximum non-zero threshold; therefore, a sparse shard with a relatively low maximum non-zero threshold can be built with fewer circuits than a sparse shard with a relatively high maximum non-zero threshold.

[0050] A sparse shard 200 may include a crossbar 215 and multiplier circuits 220. The crossbar 215 is configured to receive up to R vector values ​​and distribute the values ​​to N multiplier circuits 220, where N is equal to the maximum non-zero threshold for the sparse shard 200. While N multiplier circuits 220A-C are shown, it is understood that in different implementations the sparse shard 200 may include more or fewer multiplier circuits.

[0051] The crossbar 215 may be implemented as any one or more circuits configured to receive an input and pass the input to one or more destinations, which may be other circuits, such as the multiplier circuit 220. The multiplier circuit 220 may be implemented according to any of a variety of different techniques for performing hardware multiplication between two operands. The first operand of the multiplier circuit may be a vector value received by the crossbar 215. The second operand of the multiplier circuit may be a non-zero value from the shard input matrix 210. Each non-zero value is loaded into a respective register 221A-C,N of each multiplier circuit 220A-C,N. Having as many multiplier circuits as the maximum non-zero threshold provides an available multiplier circuit for each shard input matrix within the non-zero threshold. Each multiplier circuit multiplies the vector value received by the crossbar 215 by the non-zero value stored in its respective register.

[0052] The sparse shard 200 may also include summing circuits 225. Each summing circuit 225A-C,N is configured to receive an input from a corresponding multiplier circuit. Between each summing circuit is a segment marker. While summing circuits 225A-C,N and segment markers 226A-C,N-1 are shown in FIG. 2, it is understood that, like multiplier circuit 220, the number of summing circuits and segment markers may vary depending on the implementation.

[0053] The adder circuit can be implemented using any technique for hardware addition of two operands. The first operand from the adder circuit can be the product received from the multiplier circuit. For example, multiplier circuit 220A multiplies a vector value by a non-zero value and passes the product to adder circuit 225A. A segment marker is a circuit or other hardware component configured to gate inputs between adjacent adders depending on the value of a gate input.

[0054] Control values ​​230 may be received as input along with shard input matrix 210 and shard input vector 205 and used to configure one or more of crossbar 215, segment marker 226, and / or crossbar 235. Control values ​​230 may be a sequence of values ​​each corresponding to a respective non-zero value in shard input matrix 210. Control values ​​230 may include a first type of value, such as 1, which may correspond to a non-zero value that is the first non-zero value in a respective column of the shard input matrix. Control values ​​230 may include a second type of value, such as 0, which may correspond to a non-zero value that is located after one or more other non-zero values ​​in the same column of the shard input matrix. Control values ​​230 may also include one or more vectors of values ​​that may be used as described herein to configure sparse shards to process shard input matrix 210 and shard input vector 205, as described herein with reference to FIGS. 3A-3C. FIG. 4 and its corresponding description herein illustrate one example of matrix multiplication using sparse shards.

[0055] In some implementations, crossbars 215, 235 may be implemented to utilize different maximum dimensionality thresholds for sparse shards 200. For example, if dimensions R and C are equal or nearly equal, crossbar 235 may be implemented according to any technique for crossbar rearrangement for square or nearly square inputs, for example as a Benesh network.

[0056] 3A-3C show an example of a shard input matrix 300A, a vector 300B of non-zero values ​​in the shard input matrix, and a vector 300C of control values ​​corresponding to the shard input matrix 700A. Figure 3C also shows additional control vectors 305C and 310C.

[0057] 3A illustrates an example of a sparse shard shard input matrix 300A. For clarity, non-zero values ​​are shown as shaded cells. For clarity, indices are provided along the columns and rows of shard input matrix 300A and along vectors 300B and C. For example, in matrix 300A, the value at row 2, column 4 (2,4) is 1.

[0058] 3B illustrates a vector 300B of non-zero values ​​for an example shard input matrix 300A. The non-zero values ​​in vector 300B correspond to the order in which the non-zero values ​​appear in shard input matrix 300A when read from left to right, although the exact reading order may vary between implementations and may be, for example, from right to left.

[0059] 3C illustrates vectors of control values ​​300C, 305C, and 310C for one example of shard input matrix 300A. In some examples, vectors 300C, 305C, and 310C may be part of the same vector according to a predetermined order, and a sparse shard may be configured using this predetermined order to receive the vectors of control values ​​and configure different components of the sparse shard according to vectors 300C, 305C, and 310C, as described herein.

[0060] Vector 300C corresponds to the control values ​​used to configure the segment markers for the sparse shard. A control value (bit in this example) of 1 in vector 300C corresponds to the start of a new column in matrix 300A. The value at index 0 may be automatically set to 1 as the start of vector 300C of control values. In some implementations, the start control value may be omitted and assumed to be a constant by the processing sparse shard. A hardware implementation can simplify its circuitry by taking advantage of the fact that this value is known to be 1. The value at index 1 in vector 300C is also set to 1, as it corresponds to the value at index 1 in vector 300B, which is the first non-zero value in the next column of matrix 300A. The value at index 2 in vector 300C is set to 0, as it corresponds to a non-zero value in vector 300B that is not the first non-zero value in the next column.

[0061] As another example, the bit at index 3 in vector 300C is set to 1 because the corresponding non-zero value (value 1 at index 3 in vector 300B) is the first non-zero value in the next column (specifically, column 2) of matrix 300A. The sequence in vector 300C follows this described pattern until all non-zero values ​​in all columns have been represented.

[0062] Vector 305C corresponds to control values ​​for configuring the input crossbar for the sparse shard. For each non-zero value in shard input matrix 300A, vector 305C specifies the "y" coordinate of the non-zero value in submatrix 300A. In this example, the "y" dimension runs vertically up and down sparse matrix 300A, but in other examples the "y" dimension may be defined differently, e.g., horizontally. For example, the value "3" of element zero in vector 305C corresponds to the "y" coordinate of the non-zero value "1" in the first column of matrix 300A. As another example, the value "4" of element six in vector 305C corresponds to the value "1" at the bottom of the fourth column of matrix 300A.

[0063] Vector 310C configures how the sums produced by the adder circuits of the sparse shards are arranged by the output crossbar to produce the shard output vector. corresponds to the control value for

[0064] According to the mathematical definition of matrix multiplication, each value at location (x,y) in the sparse matrix is ​​multiplied by the value of the input vector at location y. The result of this multiplication is added to the output at location x. The input crossbar uses vector 305C to order non-zero values ​​in the same column to adjacent multiplier circuits in the sparse shard. The output crossbar uses vector 310C to order the calculated sums in the shard output vector, which represents the product of the multiplication of the shard input matrix and the shard input vector by the shard shard.

[0065] Returning to FIG. 2, the crossbar 215 may be configured to receive the control value 230 and arrange each value for the shard input vector 205 to be received as an input to one or more multiplier circuits that match the corresponding columns of the shard input matrix when the shard input vector is multiplied by the shard input matrix.

[0066] Crossbar 235 may be configured to receive one or more sums from the summation circuit and rearrange the received sums to obtain the correct output shard vector corresponding to the multiplication of the input shard matrix and the input shard vector. As with crossbar 215, any of a variety of different techniques can be applied to implement crossbar 235 as one or more circuits.

[0067] As described herein, the segment marker is configured to gate inputs between adjacent summing circuits depending on the gate input value of the segment marker. For example, if the segment marker receives a control value of 1, the segment marker can prevent the output from a first summing circuit adjacent to the segment marker from being passed as an input to a second summing circuit adjacent to the segment marker. If the segment marker receives a control value of 0, the segment marker passes the output from the first summing circuit to the second summing circuit (or vice versa, depending on the implementation). This segment marker configuration is equivalent to summing only sums corresponding to non-zero values ​​in the same column separately from sums of non-zero values ​​in different columns. As described herein with reference to FIG. 2, the one or more sums can be passed to the crossbar 235 and rearranged to generate the shard output vector 240.

[0068] The crossbar 235 can be configured to determine whether to discard or accept an input and rearrange the input to match its correct position in the shard output vector 240. As shown in FIG. 2, each summation circuit can pass its output to the crossbar 235 (indicated by an arrow to the crossbar 235). If the next segment marker after a summation circuit is not gated, the running sum between adjacent summation circuits has not finished, and the crossbar 235 can discard the output to that summation circuit. If the next segment marker is gated (or there is no segment marker, in the case of the last summation circuit 225N), the running sum for that column has finished, and the crossbar 235 receives the running sum as an input to become part of the output shard vector 240. By knowing which sums to ignore and which sums to include as part of the shard output vector 240, the crossbar 235 can accurately track column sums to generate output vectors of various lengths, up to a maximum dimension C.

[0069] FIG. 4 illustrates an example of a computation of a sparse shard 400 that receives a shard input matrix 410, a shard input vector 405, and a control value 430 for the shard input matrix 410.

[0070] Consider an example of values ​​for shard input vector 405 and shard input matrix 410: .

[0071]

[0132] (Vector 405) [0 3 0 2 0 4 1 0 0] (matrix 410) The corresponding control vectors 430-432 for matrix 410 in this example are: [1 0 1 1] (control vector 430) [1 2 0 1] (control vector 431) [0 0 1 2] (control vector 432) is.

[0072] For ease of explanation, the multiplication circuits 40A to 40D are abbreviated as multipliers A to D, the segment markers 43A to 43C are abbreviated as segment markers A to C, and the addition circuits 42A to 42D are abbreviated as adders A to D.

[0073] Based on the non-zero values ​​in matrix 410, multiplier A is loaded with a value of 2, multiplier B is loaded with a value of 1, multiplier C is loaded with a value of 3, and multiplier D is loaded with a value of 4. Note that in this example, the sparse shard includes only four multipliers A-D and four adders A-D.

[0074] Crossbar 415 receives vector 405 with vector values ​​1, 3, and 2. Dashed lines 45A, solid lines 45B, and dotted lines 45C are shown to indicate the path of data from shard input vector 405 to shard output vector 440. Crossbar 415 receives control vector 431, each value of which corresponds to the "y" coordinate of a respective non-zero value in shard input matrix 410. Values ​​of control vector 431 include 1, 2, 0, and 1. Note that because sparse input matrix 410 has dimensions 3x3, the values ​​of the "y" coordinate range from 0 to 2. Control vector 431 specifies, for each of multipliers A-D, which value of shard input vector 405 should be sent to which multiplier.

[0075] For example, the first value of control vector 431 is 1, corresponding to the “y” coordinate of the first non-zero value in shard input matrix 410. Because “y” coordinate 1 is the second coordinate (after zero), crossbar 415 routes the second value of shard input vector 405 to the first multiplier (here, multiplier A). The second value of control vector 431 is 2, corresponding to the “y” coordinate of the second non-zero value in shard input matrix 410. Crossbar 405 may then be configured to route the third value of shard input vector 405 to multiplier B. As another example, the third value of control vector 431 is 0, corresponding to the next non-zero value having a “y” coordinate of 0. Crossbar 415 routes the first value of shard input vector 405 to multiplier C.

[0076] For the first column of matrix 410, crossbar 415 directs the value 3 to multiplier A and the value 2 to multiplier B. For the second column of matrix 410, crossbar 415 directs the value 1 to multiplier C. For the third and final column of matrix 410, crossbar 415 directs the value 3 to multiplier D. The products of multipliers A to D are 6 (3×2) for multiplier A, 2 (2×1) for multiplier B, 3 (1×3) for multiplier C, and 12 (3×4) for multiplier D.

[0077] Next, adders A-D receive the products calculated by multipliers A-D. Adder A receives the product from multiplier A, which is 6. The first control value (1) is discarded. Because there is no adder before adder A, adder A passes the sum to segment marker A. The second value of control value 430 is zero, so segment marker A is not gated. Adder B receives the current sum (6) from adder A and adds it to the product of multiplier B (2). Because segment marker B is gated, the output of adder B (8) is passed to crossbar 420 (destroyed). (Indicated by line 45A). Segment marker B gates the output from adder B, so adder C adds a zero to the product of multiplier C (3). Because segment marker C is gated, the output of adder C (3) is passed to crossbar 420 (indicated by dotted line 45C). Finally, multiplier D receives the product of multiplier D (12) and, because it is the last adder in the sparse shard, automatically passes its output (12) to crossbar 420 (indicated by solid line 45B).

[0078] Crossbar 420 rearranges the received sums, 8, 3, and 12, according to the correct order for outputting output shard vector 440. Crossbar 420 receives control vector 432 with values ​​0, 0, 2, and 1. As described herein with reference to FIGS. 3A-3C, the control vector values ​​for output crossbar 420 correspond to non-zero "x" coordinate positions. Like the "y" coordinate, this value ranges from 0 to 2 in this example. The first two values ​​of control vector 432 are 0. Therefore, crossbar 420 routes the first received sum to the first element of output shard vector 440. The value after 0 for crossbar 432 is 2. Crossbar 432 is configured to route the second sum received from adders A-D to the second element of output vector 440 (indicated by line 45C) and route the third sum to the element (indicated by line 45B). In some examples, crossbar 420 is configured to skip consecutive duplicate control values ​​in vector 432, such as shown in vector 432 where the first two are zero. In some examples, rather than skipping consecutive duplicate control values, crossbar 420 is configured to perform an inclusive OR operation on the received input sums and output the results of this inclusive OR operation at positions in output vector 440 that correspond to the consecutive duplicate control values.

[0079] For example, line 46 indicates a potential input source of adder A to output crossbar 420. Because segment marker 43A has a value of zero, the output of adder A to crossbar 420 is suppressed, e.g., masked or set to zero. Instead, the output of adder A is passed through segment marker 43A to adder B. In some examples, when output crossbar 420 receives control value 432, output crossbar 420 performs an inclusive OR operation on the first sum (having a value of zero) received (through line 46) and the second sum (having a value of eight from adder B) received (through line 45A). Crossbar 420 may be configured to perform the inclusive OR operation to output non-zero operands. Crossbar 420 performs the inclusive OR operation (e.g., 0 OR 8), and then outputs 8 and places the result in the first position in output vector 440. In some examples, output crossbar 420 can receive individual outputs from at least some of adders A-D and sum them.

[0080] The summation circuits may be implemented according to any of a variety of different circuit configurations for adding consecutive ranges of values ​​defining segments, each segment corresponding to the values ​​in a respective column of the shard input matrix processed by the corresponding sparse shard. For example, the summation circuits of sparse shard 200 or 400 may be implemented as one or more sequential segmented summation circuits (e.g., as shown by sparse shard 200 or 400) for performing sequential segmented summation of products by multiplying the non-zero values ​​in each column by the respective values ​​of the shard input vector. In some implementations, the summation circuits may be configured to implement the summation tree as parallel segmented summation circuits. Individual summation circuits may be configured to sum corresponding inputs in parallel and pass the sum to an output crossbar and / or adjacent summation circuit according to the gate values ​​of any intervening segment markers, as described herein.

[0081] A parallel segmented sum circuit can reduce the latency of the circuit compared to a sequential segmented sum circuit, especially when the number of terms being summed is large. The control values ​​for gating enable parallel segmented summation because at least the range of values ​​to be summed can be tracked according to the gate values ​​of the segment markers, which gate inputs between summation circuits corresponding to values ​​in different segments and allow inputs between summation circuits corresponding to values ​​being added in the same segment.

[0082] In some implementations, output crossbar 420 is configured to receive the individual sums calculated by each adder, add the sums, and use control vector 432 to direct the sums to the corresponding element of output shard vector 440. If two or more sums are directed to the same element, for example, if control vector 432 contains duplicates of the same value, such as the first zero of vector 432, output crossbar 420 is configured to add each received sum to direct them to the same element in output shard vector 440, rather than receiving a single sum as shown by line 45A in FIG.

[0083] Example of how to FIG. 5 is a flow diagram of an example process 500 for multiplying a submatrix of a sparse matrix on a sparse shard by a system input vector according to aspects of the present disclosure. For ease of explanation, the submatrix of the sparse matrix is ​​referred to herein as the shard input matrix. FIG. 9 illustrates an example process for partitioning an input sparse matrix into multiple submatrices. For example, a sparse shard, such as sparse shard 200 of FIG. 2, performs process 500.

[0084] A sparse shard receives a shard input matrix, according to block 510. The shard input matrix is ​​within a predetermined dimensionality threshold and has a non-zero value count less than or equal to a predetermined maximum non-zero threshold.

[0085] In accordance with block 520, the sparse shard receives a shard input vector that includes multiple vector values. The shard input vector is a sub-vector of the system input vector, which may be generated by a pre-processing engine as part of the pre-processing input to the system, as described herein with reference to FIGS. 1 and 9.

[0086] In accordance with block 530, the sparse shard generates one or more products of each vector value multiplied by each non-zero value. As described herein with reference to FIGS. 2 and 4, the sparse shard may be configured with control values ​​corresponding to the positions of non-zero values ​​in the shard input matrix. Based on the control values, the sparse shard may be configured to direct the received vector values ​​of the partial vector to corresponding adjacent multiplier circuits where non-zero values ​​along the same column of the input shard matrix are stored. FIG. 8A described herein illustrates an example of a process for configuring a sparse shard using control values ​​for the shard input matrix.

[0087] In accordance with block 540, the sparse shard generates one or more sums of the one or more products. As described herein with reference to FIG. 2, the sparse shard includes a plurality of summation circuits configured to receive inputs from corresponding multiplication circuits. The summation circuits are further configured to sum these inputs along adjacent summation circuits until a segment marker is reached that is configured to gate the circuits adjacent to the segment marker. The sum of the summation circuit inputs up to the segment marker may be passed to a crossbar, such as crossbar 235 of FIG. 2.

[0088] From this one or more sums, the sparse shard generates a shard output vector that is the product of the shard input vector multiplied by the shard input matrix. The sparse shard does not care how the operands of the product generated by the multiplication circuit are ordered by the first crossbar. Depending on which sums were received, the received sums may be rearranged, for example through a second crossbar.

[0089] 6 is a flow diagram of an example process 600 for generating a system output vector that represents the product of system input vectors and a system input matrix from multiple sparse shards. A system of sparse shards, such as the sparse matrix multiplication system 100 of FIG. 1, can perform the process 600.

[0090] In accordance with block 610, for each group of sparse shards along the column dimension of the array of sparse shards, the shard output vectors of each sparse shard in the group are summed to generate a column output vector for that group. Figure 7C, described below, shows an example of grouping sparse shards according to their respective shard input matrices. The dimension along which the groups of sparse shards are formed may vary depending on the implementation. For example, depending on the direction in which the system input matrix and system input vectors are provided to the array of sparse shards, the groups may be along the rows of the array as opposed to the columns.

[0091] In accordance with block 620, the system concatenates each column output vector to generate a system output vector. The system output vector is the product of the system input matrix multiplied by the system input vector. In some implementations, as described in more detail herein with reference to FIG. 9, the system may receive a system input matrix in which the columns of the matrix are permuted, e.g., to more evenly distribute the occurrences of non-zero values ​​among the submatrices assigned to the sparse shards. In these implementations, the system may be configured to rearrange the elements of the concatenated system output vector to reverse the original permutation. The system may receive data defining the permutation as part of receiving the system input matrix, the control values, and the data defining the partitions of the system input matrix.

[0092] 7A-7C and the corresponding description illustrate an example multiplication between an example sparse matrix 700 and a vector 750. For purposes of explanation, this multiplication is described as being performed for a system having 16 sparse shards in a 4x4 array.

[0093] 7A illustrates an example system input matrix 700 and a system input vector 750. In this figure, system input matrix 700 is shown with integer values, although it is understood that the elements may be other values, such as floating-point values. Also, in FIGS. 7A-7C, non-zero elements of the various matrices comprising system input matrix 700 are shown as shaded cells.

[0094] 7B is a diagram illustrating the partitioning of system input matrix 700 and system input vector 750. In this example, system input matrix 700 is partitioned into 16 sub-matrices 700A-P, one sub-matrix for each sparse shard in the 4x4 array. System input vector 750 is partitioned into four sub-vectors 750A-D, one sub-vector for each column in the sparse shard in the 4x4 array.

[0095] An example of mapping the subvectors and submatrices to 16 sparse shards (referred to as sparse shards A to P) is shown in Table 1 below.

[0096] [Table 1]

[0097] FIG. 7C illustrates a partitioned system input matrix 700 and a system output vector 770 that represents the product of the system input matrix multiplied by a system input vector 750. FIG. 7C shows the partitioned matrix grouped along columns 705A-D. As described herein with reference to FIG. 6, the system can add the shard output vectors for each column of an array of sparse shards to generate a column output vector. In FIG. 7C, column output vectors 710A-D correspond to columns 705A-D. The system can concatenate the column output vectors 710A-D to generate the system output vector as the product of the system input matrix multiplied by the system input vector.

[0098] 8A-8B are flow diagrams of example processes 600A-B for configuring sparse shards and performing matrix multiplication using one or more control values ​​of input matrices according to aspects of the disclosure.

[0099] FIG. 8A illustrates a method for sharding a shard input matrix using one or more control values ​​according to aspects of the disclosure. 8 is a flow diagram of an example of a process 800A for configuring a sparse shard. A sparse shard, such as sparse shard 200 of FIG. 2, can perform process 800A.

[0100] According to block 810, the sparse shard receives one or more control values ​​that specify the location of non-zero values ​​along each column of the shard input matrix.

[0101] In accordance with block 820, the sparse shard loads the non-zero values ​​of the shard input matrix into registers of the multiplier circuits. As described herein with reference to FIG. 2, the sparse shard may implement a number of multiplier circuits equal to a predetermined maximum non-zero threshold for the system. The sparse shard may load the non-zero values ​​in the order in which they appear when reading the shard input matrix along a predetermined read direction, such as from left to right.

[0102] In accordance with block 830, the sparse shard loads the one or more control values ​​into the sparse shard's crossbar. A first crossbar, such as crossbar 815 of FIG. 2, may be configured to receive the control values ​​and arrange each value of the shard input vector to be received as an input to one or more multiplier circuits that match the columns of the corresponding shard input matrix by which the vector value is multiplied as part of the matrix-vector multiplication. A second crossbar, such as crossbar 235 of FIG. 2, may be configured to receive one or more sums from the adder circuits and rearrange the received sums to obtain the correct output shard vector corresponding to the multiplication of the input shard matrix and the input shard vector.

[0103] According to block 840, the sparse shard loads the one or more control values ​​into one or more segment markers configured to gate inputs to a summing circuit based on the value of the control values.

[0104] FIG. 8B is a flow diagram of an example process for performing matrix-vector multiplication using sparse shards configured according to process 800A of FIG. 8A.

[0105] According to block 850, the sparse shard receives and loads the non-zero values ​​of the input shard matrix.

[0106] According to block 860, the sparse shard receives the vector values ​​of the shard input vector and sends these vector values ​​to a multiplication circuit along with the non-zero values ​​along the same column of the shard input matrix.

[0107] In accordance with block 870, the sparse shard generates one or more segmented sums from adjacent summation circuits that are not gated by the segment marker. As described herein with reference to Figures 2 and 8A, the sparse shard aggregates sums between summation circuits that are not gated by the segment marker and can configure the segment marker with a control value to gate summation circuits that represent calculations from different columns of the shard input matrix. When the sparse shard reaches a segment marker with an active gating bit, the sparse shard passes the segmented sums to a second crossbar configured to rearrange the segmented sums to generate the shard output vector.

[0108] The sparse shard generates a shard output vector from the one or more segmented sums, according to block 880. As described herein with reference to Figures 2 and 8A, the crossbar may be configured with a control value to receive one or more segmented sums from the summation circuit of the sparse shard. The crossbar may further generate a shard output vector from the one or more segmented sums. to generate the correct shard output vector representing the product of the shard input matrix multiplied by the shard input vector.

[0109] 9 is a flow diagram of an example process 900 for generating submatrices from a system input matrix according to aspects of the present disclosure. One or more processors in one or more locations may perform process 900. For example, a pre-processing engine such as pre-processing engine 150 of FIG. 1 may perform process 900.

[0110] According to block 910, the pre-processing engine receives a system input matrix, which may be, for example, a matrix such as that shown as system input matrix 700 in Figure 7A.

[0111] In accordance with block 920, the pre-processing engine partitions the system input matrix into a plurality of candidate sub-matrices. As part of the partitioning, the pre-processing engine may receive parameters specifying a predetermined dimensionality threshold and one or more parameter values ​​indicating the number of sparse shards implemented by the system receiving the sub-matrices. For example, the pre-processing engine may be configured to generate 16 candidate sub-matrices (for a 4x4 array of sparse shards) within a dimensionality threshold of 8 rows by 8 columns. In some examples, the pre-processing engine may receive updated parameter values, e.g., for pre-processing inputs across different systems having different dimensionality thresholds and / or sparse shard configurations.

[0112] In some implementations, before the pre-processing engine partitions the system input matrix, the pre-processing engine permutes the columns of the system input matrix to distribute non-zero values ​​evenly among the candidate sub-matrices. For example, if non-zero values ​​occur more frequently on one side of the input matrix than on the other side by more than a predetermined tolerance, the pre-processing engine may be configured to reorder the columns of the input matrix so that the occurrence of non-zero values ​​is more spread out, resulting in a more even distribution among the sparse shards after partitioning.

[0113] If the pre-processing engine performs this ordering, it passes data representing the ordering as an additional input to the system, e.g., as part of the partition data sent to a system having an array of sparse shards. The system may be configured to reorder the elements of the system output vector according to the ordering to make the output vector match the output of multiplying the system input vector by the system input matrix before the columns of the system input matrix were permuted.

[0114] Permuting columns of the system input matrix can improve the overall speed at which the system input matrix is ​​processed by a system of sparse shards. For example, permuting columns can allow more efficient use of each sparse shard, especially in some instances where some sparse shards may receive shard input matrices with only zero values ​​and other shards may receive shard input matrices with only non-zero values, or shard input matrices with a number of non-zero values ​​up to a non-zero threshold.

[0115] The pre-processing engine determines whether there is a candidate sub-matrix with a non-zero value count greater than a predetermined non-zero threshold, per block 930. If the pre-processing engine determines there is a candidate sub-matrix with a non-zero value count greater than a predetermined non-zero threshold, then, per block 940, the pre-processing engine repartitions the sub-matrix along the same row or column as the candidate sub-matrix.

[0116] 7B, the submatrices may be organized along columns and rows based on the location of their values ​​in the input matrix. For example, if submatrix 700J has values ​​higher than a non-zero threshold, If the pre-processing engine determines that submatrix 700J contains a nonzero count, the pre-processing engine may repartition the submatrix along its rows (including submatrices 700I, 700K, and 700L) and / or along its columns (including submatrices 700B, 700F, and 700N). When performing the repartitioning, the pre-processing engine uses a predetermined dimensionality threshold and performs the repartitioning so that the number of candidate submatrices remains the same. For example, the pre-processing may partition the candidate submatrices and redistribute the rows / columns of the submatrix along the rows / columns of the determined candidate submatrices.

[0117] After repartitioning the submatrices according to blocks 930 and 940, the pre-processing engine again determines whether there are any candidate submatrices with non-zero value counts greater than the non-zero threshold. The pre-processing engine may repeat the determination and repartitioning according to blocks 930 and 940 until it determines that there are no candidate submatrices with non-zero value counts greater than the maximum non-zero threshold, and proceed to block 950.

[0118] As shown in block 950, the pre-processing engine partitions the system input vector. Each sub-vector of the system input vector is input to a respective row or column of the sparse shard, depending on how the buses feeding the array of sparse shards are arranged. The pre-processing engine partitions the vector so that the vector dimensions can be multiplied by the matrices of the received sparse shards, e.g., by valid dimensions for mathematically valid matrix multiplication.

[0119] For example, as shown in Figures 7B-7C and Table 1, vector 750 is partitioned into sub-vectors 750A-D, each of which is passed as an input to one or more sparse shards. Also, in Figure 7B, each dimension of each sub-vector 750A-D has the correct dimension for multiplication with the corresponding sub-matrix 700A-P. For example, sub-vector 750B is 1 x 2 (row x column), and sub-matrices 700E-H each have two rows, allowing for valid matrix multiplication between sub-vector 750B and sub-matrices 700E-H.

[0120] As shown in block 960, the pre-processing engine generates control values ​​for each candidate submatrix. As described with reference to Figures 3A-3C, the pre-processing engine may generate a vector of control values ​​indicating a starting non-zero value for each column of the matrix. The pre-processing engine repeats this generation for each candidate submatrix to generate corresponding control values ​​for each submatrix.

[0121] The pre-processing engine outputs the control values ​​and candidate sub-matrices according to block 970. The pre-processing engine may output the control values ​​and data specifying the partition of the system input matrix to, for example, a system 100 such as that shown in FIG.

[0122] Example Computing Environment 10 is a block diagram of an example computing environment for implementing a sparse matrix multiplication system 100 and a pre-processing engine 150 according to aspects of the present disclosure. The pre-processing engine 150 may be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 1015. The user computing device 1012 and the server computing device 1015 may be communicatively coupled to one or more storage devices 1030 via a network 1060. The storage device(s) 1030 may be a combination of volatile and non-volatile memory and may be in the same or a different physical location as the computing devices 1012, 1015. For example, the storage device(s) 1030 may include hard drives, solid-state drives, tape drives, optical storage, memory cards, ROM, RAM, DVDs, CD-ROMs, writable memory, and read-only memory. The information may include any type of non-transitory computer-readable medium capable of storing information, such as read-only memory.

[0123] The server computing device 1015 may include one or more processors 1013 and memory 1014. The memory 1014 may store information accessible to the processor(s) 1013, including instructions 1021 that may be executed by the processor(s) 1013. The memory 1014 may also include data 1023 that the processor(s) 1013 may retrieve, manipulate, or store. The memory 1014 may be a type of non-transitory computer-readable medium that may store information accessible to the processor(s) 1013, such as volatile and non-volatile memory. The processor(s) 1013 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0124] The server computing device 1015 may implement the sparse matrix multiplication system 100 in hardware, for example, as a system on a chip. The system 100 may be implemented as part of a physical chip slotted into or installed in the server computing device 1015. The system 100 is configured to communicate with other components of the server computing device 1015.

[0125] The instructions 1021 may include one or more instructions that, when executed by the processor(s) 1013, cause the one or more processors to perform the operations defined by the instructions. The instructions 1021 may be stored in object code format for direct processing by the processor(s) 1013 or in other formats, including interpretable scripts or collections of independent source code modules that are interpreted on-demand or pre-compiled. The instructions 1021 may include instructions for implementing a sparse shard 400 consistent with aspects of the present disclosure. The preprocessor engine 105 may be executed using the processor(s) 1013 and / or using other processors located remotely from the server computing device 1015.

[0126] The data 1023 may be retrieved, stored, or modified by the processor(s) 1013 according to the instructions 1021. The data 1023 may be stored in computer registers, in a relational or non-relational database as a table with multiple different fields and records, or as a JSON, YAML, proto, or XML document. The data 1023 may also be formatted into a computer-readable format such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, the data 1023 may include information sufficient to identify related information or used by a function to compute the related data, such as numbers, descriptive text, proprietary code, pointers, references to data stored in other memory, including other network locations, etc.

[0127] The user computing device 1012 may be configured similarly to the server computing device 1015, having one or more processors 1016, memory 1017, instructions 1018, and data 1019. The user computing device 1012 may also include a user output 1026 and a user input 1024. The user input 1024 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touchscreen, a microphone, and a sensor.

[0128] The server computing device 1015 may be configured to transmit data to the user computing device 1012, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 1026. The user output 1026 may also be used to display an interface between the user computing device 1012 and the server computing device 1015. The user output 1026 may alternatively or additionally include one or more speakers, transducers or other audio outputs, a haptic interface or other tactile feedback that provides non-visual and non-audible information to a platform user of the user computing device 1012.

[0129] While Figure 10 depicts the processors 1013, 1016 and memories 1014, 1017 as being internal to the computing devices 1015, 1012, the components described herein, including the processors 1013, 1016 and memories 1014, 1017, may include multiple processors and memories that can operate in different physical locations and are not internal to the same computing device. For example, some of the instructions 1021, 1018 and data 1023, 1019 may be stored on a removable SD card, while other portions may be stored in a read-only computer chip. Some or all of the instructions and data may be stored in a location physically separate from but accessible to the processors 1013, 1016. Similarly, the processors 1013, 1016 may include a collection of processors capable of performing simultaneous and / or sequential operations. Each of the computing devices 1015, 1012 may include one or more internal clocks that provide timing information that may be used to time operations and programs executed by the computing devices 1015, 1012.

[0130] The server computing device 1015 may be configured to receive requests to process data from the user computing devices 1012. For example, the environment 1000 may be part of a computing platform configured to provide various services to users via various user interfaces and / or APIs that expose platform services. As part of running the services, the server computing device 1015 may process the received data using the system 100. For example, if the service is training a machine learning model, the server computing device 1015 may be configured to use the system 100 to perform a multiplication operation as part of training the machine learning model.

[0131] The devices 1012, 1015 may be capable of direct and indirect communication over the network 1060. The devices 1015, 1012 may set up listening sockets that can accept initiating connections to send and receive information. The network 1060 itself may include a variety of configurations and protocols, such as the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, and a private network using one or more company-proprietary communication protocols. The network 1060 may support a variety of short-range and long-range connections. These short-range and long-range connections may occur over different bandwidths, such as 2.402 GHz to 2.480 GHz (commonly associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (commonly associated with the Wi-Fi® communication protocol), or various communication standards, such as the LTE® standard for wireless broadband communication. The network 1060 may also or alternatively support wired connections between the devices 1012, 1015, such as via various types of Ethernet® connections.

[0132] 10 illustrates one server computing device 1015 and one user computing device 1012, it is understood that aspects of the disclosure can be implemented according to a variety of different configurations and quantities of computing devices, such as in a paradigm for sequential or parallel processing, or via a distributed network of multiple devices. In some implementations, aspects of the disclosure can be performed on one device, or any combination thereof. Furthermore, while the pre-processing engine and the sparse matrix multiplication system 100 are shown as being implemented on the same server computing device 1015, in some implementations, the pre-processing engine 150 is implemented on one or more server computing devices and / or user computing devices 1012 that are separate from the server computing device 1015.

[0133] Aspects of the present disclosure can be implemented as digital circuitry, one or more computer programs in a computer-readable storage medium, or a combination of one or more of the above. The computer-readable storage medium may be non-transitory, for example, as one or more instructions executable by a computing device and stored on a tangible storage device.

[0134] As used herein, the phrase "configured to" is used in different contexts relating to a computer system, hardware, or part of a computer program, engine, or module. When we say that a system is configured to perform one or more operations, we mean that the system has installed thereon appropriate software, firmware, and / or hardware that, when operated, causes the system to perform the one or more operations. When we say that hardware is configured to perform one or more operations, we mean that the hardware includes one or more circuits that, when operated, receive input and generate output corresponding to the one or more operations according to the input. When we say that a computer program, engine, or module is configured to perform one or more operations, we mean that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform the one or more operations.

[0135] While the operations illustrated in the figures and recited in the claims are shown in a particular order, it is understood that these operations may be performed in an order different from that shown, and that some operations may be omitted, performed multiple times, and / or performed in parallel and / or concurrently with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring separation of the components. The described components, modules, programs, and engines may be integrated as a single system or may be part of multiple systems.

[0136] Unless otherwise specified, the above-described alternatives are not mutually exclusive and may be implemented in various combinations to achieve specific advantages. Because these and other variations and combinations of the above features can be utilized without departing from the subject matter defined by the claims, the above description of the embodiments should be interpreted as illustrative, and not limiting, of the subject matter defined by the claims. Additionally, the provision of embodiments described herein, as well as clauses such as "such as" and "including," should not be interpreted as limiting the subject matter of the claims to any particular embodiment; rather, the embodiment is intended to illustrate only one of many possible implementations. Furthermore, the same reference numbers in various figures may identify the same or similar elements.

Claims

1. A sparse shard including a plurality of multiplication circuits, the sparse shard comprising: configured to receive a shard input matrix containing a number of non-zero values ​​equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits, the multiplier circuits receiving each non-zero value in the shard input matrix, the sparse shard further comprising: receives a shard input vector containing multiple vector values; generating, by the plurality of multiplication circuits, products of vector values ​​multiplied by the respective non-zero values ​​of the sharded input matrix; The system is configured to use one or more of the products to generate a sharded output vector that is the product of applying the sharded input vector with the sharded input matrix.

2. The system described in claim 1, wherein the shard input matrix has dimensions less than or equal to a predetermined dimensionality threshold corresponding to a maximum matrix input size.

3. The sparse shard is one of a plurality of sparse shards, and the plurality of sparse shards comprises: receiving a plurality of shard input matrices that are submatrices of the system input matrix; receiving a plurality of shard input vectors that are subvectors of the system input vector; 3. A system according to claim 1 or 2, configured to generate a system output vector representing the product of said system input vector applied to said system input matrix.

4. The system described in claim 3, wherein when generating the system output vector, the multiple sparse shards are further configured to concatenate their respective shard output vectors to generate the system output vector.

5. A system as described in any one of claims 1 to 4, wherein the multiplication circuit is coupled to a register containing each of the non-zero values.

6. A system described in any one of claims 1 to 5, wherein the predetermined non-zero threshold is a maximum non-zero threshold.

7. The sparse shard further includes a crossbar circuit, the crossbar circuit comprising: receiving the plurality of vector values ​​of the shard input vector; 7. The system of claim 1, configured to provide one vector value of the plurality of vector values ​​as an input to each of the plurality of multiplication circuits in accordance with one or more control values.

8. 8. The system of claim 1, wherein the sparse shard is further configured to load non-zero values ​​in a same column of the shard input matrix into registers of adjacent multiplier circuits of the plurality of multiplier circuits.

9. 10. The system of claim 8, wherein the sparse shard is further configured to receive one or more control values ​​that specify the location of non-zero values ​​along columns of the shard input matrix.

10. The system of any one of claims 1 to 9, wherein the sparse shard further comprises a plurality of summation circuits configured to generate one or more sums of one or more of the products of vector values.

11. The system of claim 10 , wherein the plurality of adder circuits form a parallel segmented sum circuit.

12. The sparse shard further includes a crossbar circuit, the crossbar circuit comprising: receiving the one or more sums; 12. The system of claim 10 or 11, configured to arrange the one or more sums according to one or more control values ​​to generate the sharded output vector.

13. The system of claim 12 , wherein the crossbar circuitry forms a Benesh network.

14. A method comprising: a sparse shard including a plurality of multiplier circuits receiving a shard input matrix including a number of non-zero values ​​less than or equal to a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits, the multiplier circuits receiving each non-zero value in the shard input matrix; receiving, by the sparse shard, a shard input vector including a plurality of vector values; the plurality of multiplication circuits generating products of vector values ​​multiplied by the respective non-zero values ​​of the sharded input matrix; the sparse shard using one or more of the products to generate a shard output vector that is the product of applying the shard input vector with the shard input matrix.

15. The method of claim 14, wherein the shard input matrix has dimensions less than or equal to a predetermined dimensionality threshold corresponding to a maximum matrix input size. a plurality of sparse shards, of which the sparse shard is one, receiving a plurality of shard input matrices that are submatrices of a system input matrix; the plurality of sparse shards receiving a plurality of shard input vectors that are sub-vectors of a system input vector; The method of claim 14 or 15, further comprising: the plurality of sparse shards generating a system output vector representing the product of applying the system input vector to the system input matrix.

17. The method of claim 16, wherein generating the system output vector further comprises the multiple sparse shards concatenating their respective shard output vectors to generate the system output vector.

18. A method according to any one of claims 14 to 17, wherein the multiplication circuit is coupled to a register containing the respective non-zero value.

19. A method according to any one of claims 14 to 18, wherein the predetermined non-zero threshold is a maximum non-zero threshold.

20. One or more computer programs that, when executed by a system having a sparse shard including a plurality of multiplication circuits, cause the system to perform a method according to any one of claims 14 to 19.

Images