Sparse matrix multiplication in hardware
Patent Information
- Application Number
- JP2023206881
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-05-25
- Filing Date
- 2023-12-07
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-12-21
AI Technical Summary
Existing hardware solutions for neural network computations face inefficiencies in performing sparse matrix dense vector multiplication due to the high sparsity of matrices, leading to increased processing times and resource utilization for redundant zero calculations.
A system comprising sparse shards with multiplier circuits and adder circuits, configured to process submatrices with non-zero values within a predetermined threshold, efficiently performs sparse matrix dense vector multiplication by discarding zero values and tracking non-zero values across columns, using crossbar and summing circuits to generate accurate shard output vectors.
The system reduces processing cycles and resource usage by eliminating redundant calculations, enabling efficient handling of matrices with varying sparsity ratios without preprocessing, thus enhancing the performance of neural network workloads.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Background technology]
[0001] background A sparse matrix is a matrix that has a higher percentage of zero values than nonzero values as elements of the matrix. Different sparse matrices can have different degrees of sparseness based on the percentage of zero values to nonzero values. A matrix that has a higher percentage of zero values than nonzero values is said to be more sparse than a matrix that has a lower percentage of zero values than nonzero values.
[0002] A neural network is a machine learning model that includes one or more nonlinear operational layers for predicting an output for a received input. In addition to an input layer and an output layer, some neural networks include one or more hidden layers. The output of each hidden layer may be input to another hidden layer or to an output layer of the neural network. Each layer of the neural network may generate a respective output from a received input according to the value of one or more model parameters of that layer. The model parameters may be weights or biases that are determined through a training algorithm so that the neural network generates an accurate output. The model parameter values of a layer of a neural network may be represented as elements of a matrix or a tensor. Summary of the Invention [Means for solving the problem]
[0003] Quick Overview Aspects of the present disclosure relate to sparse matrix dense vector multiplication in hardware.
[0004] One aspect of the disclosure provides a system including a sparse shard including a plurality of multiplier circuits, the sparse shard configured to receive a shard input matrix including a number of nonzero values less than or equal to a predetermined maximum nonzero threshold, receive a shard input vector including a plurality of vector values, for each of the multiplier circuits, receive respective nonzero values of the shard input matrix, generate one or more products by the plurality of multiplier circuits of vector values multiplied by the respective nonzero values of the shard input matrix, and generate as output to the sparse shard a shard output vector that is a product of applying the shard input vector to the shard input matrix using the one or more products.
[0005] Another aspect of the disclosure provides one or more non-transitory computer-readable storage media having instructions stored thereon that, when executed by a system including a plurality of sparse shards, cause the system to perform operations including: receiving, by a sparse shard including a plurality of multiplier circuits, a shard input matrix including a number of nonzero values less than or equal to a predetermined maximum nonzero threshold, and a shard input vector including a plurality of vector values; receiving, for each of the multiplier circuits, each of the nonzero values of the shard input matrix; generating, by the plurality of multiplier circuits of the sparse shard, one or more products of multiplying each vector value by the respective nonzero value; and generating, as an output to the sparse shard, a shard output vector that is a product of applying the shard input vector to the shard input matrix using the one or more products.
[0006] Another aspect of the disclosure provides a method, comprising: a sparse shard including a plurality of multiplier circuits receiving a shard input matrix including a number of non-zero values less than or equal to a predetermined maximum non-zero threshold, and a shard input vector including a plurality of vector values; receiving, for each of the multiplier circuits, each of the non-zero values of the shard input matrix; and the plurality of multiplier circuits of the sparse shard multiplying each of the vector values by the non-zero values of the shard input matrix. generating one or more products by multiplying each of the non-zero values, and the sparse shard generating, as output to the sparse shard, a shard output vector that is a product of the shard input vector and the shard input matrix using the one or more products.
[0007] Each of the above and other aspects may optionally include one or more of the following features alone or in combination. One implementation may include all of the following features in combination.
[0008] The length of the shard output vector is greater than 1. The sparse shard is one of a plurality of sparse shards configured to receive a plurality of shard input matrices that are sub-matrices of a system input matrix, and a plurality of shard input vectors that are sub-vectors of a system input vector, and to generate, by the plurality of sparse shards, a system output vector that represents the product of the system input vectors and the system input matrix.
[0009] The multiple sparse shards are arranged as a systolic array, the systolic array including one or more groups of sparse shards along a column dimension of the systolic array, and to generate the system output vector, the one or more processors are further configured to: for each group along the column dimension of the systolic array, sum the respective shard output vectors of each sparse shard in the group to generate a respective column output vector, and concatenate the respective column output vectors of each group to generate the system output vector.
[0010] Each multiplier circuit is coupled to a respective register that contains the respective non-zero values from each of the shard input matrices for the sparse shard.
[0011] The number of multiplier circuits in the plurality of multiplier circuits is equal to the predetermined maximum non-zero threshold value.
[0012] The sparse shard further includes a crossbar circuit, and the sparse shard is further configured to receive the vector values of the shard input vector by the crossbar circuit and to send one of the vector values by the crossbar circuit as an input to each of the multiplication circuits.
[0013] The sparse shard is further configured to load non-zero values in a same column of the shard input matrix into registers of adjacent multiplier circuits of the plurality of multiplier circuits.
[0014] The sparse shard is further configured to receive one or more control values that specify at least the location of non-zero values along each column of the shard input matrix, and the crossbar circuit of the sparse shard is further configured to receive the one or more control values and send vector values to adjacent multiplication circuits that are multiplied by non-zero values along the same column of the shard input matrix in accordance with the one or more control values.
[0015] The sparse shard further includes a plurality of summation circuits, the sparse shard further includes one or more segment markers, each segment marker configured to gate an input to a respective one of the plurality of summation circuits based on a value of a respective control value loaded into the segment marker, the sparse shard is further configured to load at least a portion of the one or more control values into the one or more segment markers, and a summation circuit for a non-zero value in a first column of the shard input matrix gates an input to an adjacent summation circuit that includes a non-zero value in a second column of the shard input matrix that is different from the first column. the sparse shard is further configured to generate, by the plurality of summation circuits, one or more sums of the one or more products, each of the one or more sums being a respective segmented sum of one or more non-zero values of a column of the shard input matrix multiplied by one or more respective values of the shard input vector.
[0016] The plurality of summing circuits form parallel segmented summing circuits, each of one or more of the segmented summing circuits being a sum of the outputs to adjacent summing circuits that are not gated by a segment marker.
[0017] The crossbar circuit is a first crossbar circuit and the sparse shard further includes a second crossbar circuit configured to receive one or more of the segmented sums and arrange the one or more segmented sums according to the one or more control values to generate each of the shard output vectors for the sparse shard.
[0018] The second crossbar circuit may form a Benesh network, and the shard input matrix is a square matrix.
[0019] Another aspect of the present disclosure provides a system including one or more processors and one or more memory devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including receiving an input matrix including zero values and non-zero values and partitioning the input matrix into a number of sub-matrices, the number of non-zero values in each sub-matrix being less than or equal to a predetermined maximum non-zero threshold and a dimension of each sub-matrix being less than or equal to a predetermined dimension threshold.
[0020] The operations may further include generating, for each sub-matrix, one or more respective control values that specify locations of non-zero values along each column of the sub-matrix.
[0021] The operations may further include sending each submatrix and the one or more respective control values for each submatrix to a plurality of sparse shards configured to process each submatrix and the one or more respective control values for the submatrix. Each of the plurality of sparse shards is configured to receive a submatrix, the one or more respective control values for the submatrix, and at least a portion of an input vector, and to generate a respective output shard vector representing a product of the submatrix and the portion of the input vector. The system of claim 21, wherein partitioning the input matrix includes partitioning the matrix into a number of submatrices equal to a number of sparse shards in the plurality of sparse shards.
[0022] Other aspects of the present disclosure include corresponding systems, apparatus, and computer programs stored on one or more non-transitory computer-readable storage media. [Brief description of the drawings]
[0023] [Figure 1] FIG. 1 is a block diagram of an example of a system including an array of sparse shards according to an aspect of the disclosure. [Diagram 2] FIG. 1 is a block diagram of an example of a sparse shard according to an aspect of the disclosure. [Figure 3A] FIG. 13 illustrates an example of a shard input matrix for a sparse shard. [Figure 3B] FIG. 13 illustrates a vector of non-zero values for an example shard input matrix. [Figure 3C] FIG. 13 illustrates a vector of control values for an example shard input matrix. [Figure 4] FIG. 13 illustrates an example of matrix multiplication with sparse shards receiving a shard input matrix, a shard input vector, and a control value for the shard input matrix. [Diagram 5] FIG. 13 is a flow diagram of an example of a process for multiplying a submatrix of a sparse matrix on a sparse shard by a system input vector according to an aspect of the disclosure. [Figure 6]FIG. 13 is a flow diagram of an example of a process for generating a system output vector that represents the product of system input vectors from multiple sparse shards and a system input matrix. [Figure 7A] FIG. 2 illustrates an example of a system input matrix and a system input vector. [Figure 7B] FIG. 2 illustrates a system input matrix and a partition of a system input vector. [Figure 7C] FIG. 2 illustrates a partitioned system input matrix and a system output vector that represents the product of the system input matrix multiplied by the system input vector. [Figure 8A] FIG. 11 is a flow diagram of an example of a process for configuring sparse shards using one or more control values of a shard input matrix according to an aspect of the disclosure. [Figure 8B] FIG. 8B is a flow diagram of an example of a process for performing matrix-vector multiplication using sparse shards configured according to the process of FIG. 8A. [Figure 9] FIG. 2 is a flow diagram of an example process for generating a submatrix from a system input matrix according to an aspect of the present disclosure. [Figure 10] FIG. 1 is a block diagram of an example computing environment for implementing a sparse matrix multiplication system according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] Detailed Description overview An aspect of the present disclosure relates to a system including one or more integrated circuits configured for sparse matrix dense vector multiplication. A system of multiple sparse shards can receive sub-matrices and sub-vectors of input system matrices and vectors, per sparse shard. Each sparse shard can be at least a portion of an integrated circuit and can implement multiple arithmetic units, such as multiplication circuits and addition circuits. Each sparse shard is configured to receive a sub-matrix having a number of non-zero values less than or equal to a maximum non-zero threshold, which can be predetermined when the system is implemented, for example, as a chip including the sparse shards and optionally one or more other components as described herein.
[0025] Each sparse shard may further receive metadata, such as control values described herein, that the sparse shard may use to direct different inputs to each multiplier or adder circuit and gate when the output from one unit is passed to another unit. By configuring the sparse shard according to the received metadata, the sparse shard may efficiently process input submatrices of any size up to a predetermined dimensionality threshold and output products as vectors of the corresponding length to the input.
[0026] As part of receiving the submatrix, each sparse shard may receive one or more control values that represent the positions of non-zero values in each column of the submatrix. Using the one or more control values, the sparse shard may be configured to adjust how the individual values of the submatrix and subvector should be multiplied, added, and arranged such that the sparse shard generates a shard output vector that represents the product of the submatrix and the subvector. The system may further be configured to generate a system output vector from the shard output vector of each shard, for example by multiplying the system input vector by the system input matrix, that represents the product of the system input vector applied to the system input matrix.
[0027] Aspects of the present disclosure also relate to a system for pre-processing a sparse matrix for processing by an array of sparse shards. A system of one or more processors performs multiple operations on an input matrix. The system may be configured to partition the sparse shard into a number of sub-matrices, each with a dimension less than or equal to a predetermined dimensionality threshold that specifies a maximum matrix input size for the sparse shard. As part of the partitioning, the system may identify whether any of the sub-matrices include a number of non-zero values greater than a predetermined maximum non-zero threshold, and accordingly repartition the sub-matrix along the same row or column as the identified sub-matrix. The system may repeat this process until it generates a sub-matrix for each sparse shard, such that the dimension of the sub-matrix is within the dimensionality threshold and the number of non-zero values in the sub-matrix is less than or equal to the predetermined non-zero threshold.
[0028] Systems implemented according to aspects of the present disclosure can more efficiently perform workloads involving repeated multiplication of sparse matrices with vectors. For example, a device implementing a system-on-a-chip (SoC) according to aspects of the present disclosure can generate a product of a sparse matrix multiplied by a vector in fewer processing cycles than conventional approaches, at least because redundant "multiplication by zero" calculations of zero elements of the matrix that do not contribute to the final product are omitted. While other approaches may experience increased performance loss for large matrices with high sparse ratios, systems implemented as described herein can more efficiently compute products with these matrices, at least because a higher proportion of redundant calculations are omitted as the sparse ratio of the input matrices increases.
[0029] Some workloads, such as running or training neural networks, are heavily dependent on performing matrix multiplication. Although hardware accelerators or other devices can efficiently perform certain operations, such as matrix multiplication, they often have limited processing conditions. For example, the device may require a predefined sparsity ratio between zero and nonzero values in the matrix, or may be severely constrained in the size of the inputs it processes. While matrix multiplication is a ubiquitous type of computation for many neural network workloads, accelerators with limited processing conditions are limited in the number of types of workloads they can support, such as workloads involving matrix multiplication of matrices with various sparsity ratios.
[0030] Aspects of the present disclosure provide a system for sparse matrix multiplication that can flexibly handle matrices of different sizes and sparse ratios. A sparse shard of a system of sparse shards configured for sparse matrix multiplication can discard zero values of an input matrix while keeping track of the location of non-zero values of a given input submatrix across multiple columns. A sparse shard is composed of only as many registers and multiplication circuits as a predetermined maximum non-zero threshold, and can store only non-zero values of an input submatrix in memory. As a result, a sparse shard can store and process input data more efficiently and with fewer resources compared to storing a full-sized matrix.
[0031] Furthermore, the sparse shard may be configured to arrange the non-zero values and their products multiplied by values of the input vector along adjacent multiplication and addition circuits, respectively. The sparse shard may be configured to efficiently generate segmented sums for each column of the input matrix by combining the outputs of these adjacent circuits, and rearrange the segmented sums to generate the shard output vector. The sparse shard may preserve the order of the non-zero values to precisely add and multiply the non-zero values by corresponding values of the input vector, generating the shard output vector exactly as the product of the input submatrix and the input vector. The sparse shard may perform multiplications on the input matrix without requiring pre-processing operations to change the shape of the input matrix.
[0032] The system may also include one or more processors configured to preprocess the sparse matrix for multiplication with the input vector by the multiple sparse shards. The system may generate a different number of sub-matrices depending on the number of available sparse shards. , and different configurations of sparse shards can be supported, such as generating submatrices for different maximum non-zero thresholds.
[0033] Aspects of the disclosure described herein may be implemented by a system including one or more processors and a plurality of sparse shards, for example, as a chip installed in a computing device, such as a server computing device in a data center of a computing platform.
[0034] Example System 1 is a block diagram of an example system 100 that includes an array 101 of sparse shards 101A-P. The array 101 may be at least part of a sparse matrix multiplication system, such as sparse matrix multiplication system 100, configured to perform matrix multiplication on matrix inputs.
[0035] As described in more detail herein with reference to Figure 2, a sparse shard is a collection of circuits configured to perform arithmetic operations, to arrange inputs and outputs between circuits performing arithmetic operations, and / or to gate inputs and outputs between other circuits. For example, a sparse shard may be configured to perform matrix multiplication between a rectangular shard or portion of a sparse matrix and a linear shard or piece of a vector. As described in more detail herein, a sparse shard can receive a variety of different shards or portions of a sparse matrix having any number of non-zero values up to a predetermined maximum threshold.
[0036] The collection of circuits may be configured as described herein to control the state (i.e., open or closed) of each gate, the operands computed, and / or the sequence of inputs to and outputs from each circuit. This configuration may depend, at least in part, on the dimensions of the shard input matrix that the sparse shard is configured to receive as input, and / or on the sequence and number of non-zero values in the shard input matrix. Each sparse shard may be an individual circuit component for a system configured to multiply a sparse matrix by a vector.
[0037] Each sparse shard 101A-P is configured to communicate with two or more other sparse shards in the array 101. A sparse shard can communicate with its immediate neighbors, e.g., the sparse shard immediately preceding or following it along a dimension defined by the rectangular arrangement of the sparse shards 101A-P. The connections between the sparse shards can be realized, for example, via a bus or one or more circuit interconnects that physically connect the sparse shard to its neighbors. Each sparse shard 101A-P can be realized as one or more circuits configured to receive at least a portion of a matrix and at least a portion of a vector and generate an output vector that represents the product of the input matrix and the vector. In some examples, the sparse shards 101A-P are organized as a systolic array, although in various implementations, the array 101 is generally configured or arranged according to a rectangular arrangement of the sparse shards 101A-P.
[0038] Array 101 may be at least a part of a system-on-chip that implements multiple components and integrated circuits on a circuit board or other material. Array 101 may be installed as part of a computing device and configured to interact with other components of the device, such as memory, a processor, network components, and / or peripherals. For example, array 101 may be installed as part of system 100. The system 100 may receive a system input vector 105 and a system input matrix 110 from one or more memory devices implemented using a memory array. The system 100 may generate as an output a system output vector 115 that represents the product of the system input vector 105 and the system input matrix 110.
[0039] In some examples, the system output vector 115 may be provided as an input to other devices or components of devices that implement the system 100. For example, if the system output vector 115 is a product of model parameter values multiplied by certain vector inputs for a neural network, the system output vector 115 may be provided as an input to one or more processors configured to calculate an activation function for the output vector 115.
[0040] The system 100 may receive the system input vector 105 and the system input matrix 110 from a pre-processing engine 150. The pre-processing engine 150 may be implemented on one or more computing devices that may or may not be the device implementing the system 100. The system input matrix 110 may include values that represent at least a portion of the model parameter values for the neural network, as an example. The system input matrix 110 may be part of a more complex data structure, such as a multi-dimensional array or tensor. The system 100 may be configured to receive each matrix that corresponds to at least a portion of a larger data structure, such as a three-dimensional tensor or matrix, and generate a corresponding output of the matrix multiplied by the system input vector 105. The system input vector 105 may be, for example, an input to a trained neural network whose model parameter values are at least partially represented by the values of the system input matrix 110.
[0041] The pre-processing engine 150 may be configured to process the system input matrix 110 to generate one or more control values 111 and matrix partition data 112. The pre-processing engine 150 may receive the system input matrix 110 in a variety of different formats for storing sparse matrices, such as in its complete row-column form (with all non-zero and zero values). As another example, the pre-processing engine 150 may receive the system input matrix 110 according to a compressed sparse column format, a coordinate list format, or any of a variety of other formats for storing sparse matrices.
[0042] In some instances where pre-processing engine 150 receives system input matrix 110 in its entirety, pre-processing engine 150 is configured to convert system input matrix 110 into a predetermined format suitable for storage in memory. For example, pre-processing engine 150 may remove zero values from the matrix and generate control values to track the positions of the non-zero values relative to their original positions in the matrix before converting to this predetermined format.
[0043] The control values are used to configure each sparse shard to receive subvectors of the system input vector 105 and multiply them by submatrices of the system input matrix 110, as described in more detail herein with reference to Figure 2. Partition data 112 is data that specifies how the system input matrix 110 should be partitioned into submatrices. Each submatrix is received as an input at a respective sparse shard, and the partition data 112 specifies the partitioning for as many submatrices as there are sparse shards in the array 101.
[0044] The pre-processing engine 150 may be configured to generate the vector partition data 106. As described in more detail with reference to FIGS. 5A-5C, Sparse shards along can take the same subvector as input to multiply with their respective submatrices.
[0045] Although system input matrix 110 and system input vectors 105 are shown as being provided along the left and right sides of system 100, the exact positioning and orientation of buses providing input data to system 100 may vary between implementations. For example, based on the location of other components on the same chip as system 100, the buses or circuit interconnects for providing inputs to and receiving outputs from system 100 may be oriented or positioned differently to account for the locations of those other components.
[0046] 2 is a block diagram of an example sparse shard 200 according to aspects of the disclosure. For example, each of the sparse shards 101A-P of the system 100 may be implemented as described herein with reference to the sparse shard 200.
[0047] The sparse shard 200 is configured to receive a shard input vector 205 and a shard input matrix 210. The shard input vector 205 may include one or more vector values and have a maximum dimension of 1×R. The shard input matrix includes one or more zero values and one or more non-zero values and has a maximum dimension of R×C. R (row) and C (column) are dimensionality thresholds that correspond to a predetermined maximum input size of a vector / matrix that the sparse shard can receive as input. In different implementations, the sparse shards may be configured for different dimensions R and C. R and C may be equal or different from each other, and different sparse shards may be implemented for different dimensions, for example, depending on the nature of data for different workloads that the sparse shards are configured to process. The sparse shards of an array of sparse shards may be configured to receive inputs within the same maximum dimensionality threshold.
[0048] A sparse shard 200 may be configured to receive the shard input vector 205 and the shard input matrix 210 within a predetermined address range in a memory associated with the sparse shard 200. For example, the sparse shard 200 is configured to automatically retrieve the shard input vector 205 from a first address range in the combined memory and retrieve the shard input matrix 210 from the same or a different address range in the memory. A device or a component of a device implementing the system 100 having the sparse shard 200 may be configured to send the shard input matrix and vector to a location in the memory corresponding to each of the one or more sparse shards implemented by the system. For example, the pre-processing engine 150 may be configured to process and generate the system input matrix and / or system input vector, and then store the individual shard input matrices and shard input vectors in an address range corresponding to each sparse shard, including the sparse shard 200.
[0049] In addition to the maximum dimension threshold, the sparse shard 200 is configured to receive shard input matrices that have a nonzero count less than or equal to a predetermined maximum nonzero threshold. As with the dimension threshold, the maximum nonzero threshold may be set to different values depending on the implementation of the sparse shard 200 and its corresponding array. For example, if the data of the workload that the system is configured to process generally includes processing matrices with a high sparse ratio, the sparse shard system may be configured with a relatively high maximum nonzero threshold. As described herein, the number of multiplication and addition circuits of a sparse shard corresponds to its maximum nonzero threshold, and thus a sparse shard with a relatively low maximum nonzero threshold can be constructed with less circuitry than a sparse shard with a relatively high maximum nonzero threshold.
[0050] The sparse shard 200 may include a crossbar 215 and a multiplier circuit 220. The crossbar 215 is configured to receive up to R vector values and distribute the values to N multiplier circuits 220, where N is equal to the maximum non-zero threshold for the sparse shard 200. Although multiplier circuits 220A-C, N are shown, it is understood that in different implementations the sparse shard 200 may include more or fewer multiplier circuits.
[0051] The crossbar 215 may be implemented as any one or more circuits configured to receive an input and pass the input to one or more destinations, which may be other circuits, such as the multiplier circuit 220. The multiplier circuit 220 may be implemented according to any of a variety of different techniques for performing hardware multiplication between two operands. A first operand of the multiplier circuit may be a vector value received by the crossbar 215. A second operand of the multiplier circuit may be a non-zero value from the shard input matrix 210. Each non-zero value is loaded into a respective register 221A-C,N of a respective multiplier circuit 220A-C,N. Having as many multiplier circuits as the maximum non-zero threshold provides an available multiplier circuit for each shard input matrix within the non-zero threshold. Each multiplier circuit multiplies the vector value received by the crossbar 215 by the non-zero value stored in its respective register.
[0052] The sparse shard 200 may also include summing circuits 225. Each summing circuit 225A-C,N is configured to receive an input from a corresponding multiplier circuit. Between each summing circuit is a segment marker. Although summing circuits 225A-C,N and segment markers 226A-C,N-1 are shown in FIG. 2, it will be understood that, as with multiplier circuits 220, the number of summing circuits and segment markers may vary depending on the implementation.
[0053] The adder circuit can be implemented using any technique for hardware addition of two operands. The first operand from the adder circuit can be the product received from the multiplier circuit. For example, the multiplier circuit 220A passes the product of a vector value multiplied by a non-zero value to the adder circuit 225A. The segment marker is a circuit or other hardware component configured to gate inputs between adjacent adders depending on a gate input value.
[0054] The control values 230 may be received as input along with the shard input matrix 210 and the shard input vectors 205 and used to configure one or more of the crossbars 215, the segment markers 226, and / or the crossbars 235. The control values 230 may be a sequence of values that respectively correspond to respective non-zero values of the shard input matrix 210. The control values 230 may include a first type of value, such as 1, which may correspond to a non-zero value that is the first non-zero value in a respective column of the shard input matrix. The control values 230 may include a second type of value, such as 0, which may correspond to a non-zero value that is located after one or more other non-zero values in the same column of the shard input matrix. The control values 230 may also include one or more vectors of values that may be used as described herein to configure sparse shards to process the shard input matrix 210 and the shard input vectors 205, as described herein with reference to Figures 3A-3C. Figure 4 and its corresponding description herein illustrates one example of matrix multiplication using sparse shards.
[0055] In some implementations, the crossbars 215, 235 may be implemented to take advantage of different maximum dimensionality thresholds of the sparse shards 200. For example, if the dimensions R and C are equal or nearly equal, the crossbar 235 may be implemented according to any technique for crossbar rearrangement for square or nearly square inputs, for example as a Benesh network.
[0056] 3A-3C show an example of a shard input matrix 300A, a vector of non-zero values in the shard input matrix 300B, and a vector of control values 300C that correspond to the shard input matrix 700A. Figure 3C also shows additional control vectors 305C and 310C.
[0057] 3A illustrates an example of a shard input matrix 300A for a sparse shard. For clarity, non-zero values are shown as shaded cells. For clarity, indices are given along the columns and rows of shard input matrix 300A and along vectors 300B and C. For example, in matrix 300A, the value at row 2, column 4 (2,4) is 1.
[0058] 3B illustrates a vector 300B of non-zero values for an example shard input matrix 300A. The non-zero values in vector 300B correspond to the order in which the non-zero values appear in shard input matrix 300A when read from left to right, although the exact reading order may vary from implementation to implementation and may be, for example, from right to left.
[0059] 3C illustrates vectors of control values 300C, 305C, and 310C for one example of shard input matrix 300A. In some examples, vectors 300C, 305C, and 310C may be part of the same vector according to a predetermined order, and a sparse shard may be configured to receive the vector of control values and configure different components of the sparse shard according to vectors 300C, 305C, and 310C, as described herein.
[0060] Vector 300C corresponds to the control values used to configure the segment markers for the sparse shard. A control value (bit in this example) of 1 in vector 300C corresponds to the start of a new column of matrix 300A. The value of index 0 may be automatically set to 1 as the start of vector 300C of control values. In some implementations, the start control value may be omitted and assumed to be a constant by the processing sparse shard. A hardware implementation may simplify its circuitry by taking advantage of the fact that this value is known to be 1. The value of index 1 in vector 300C is also set to 1, as it corresponds to the value of index 1 in vector 300B, which is the first non-zero value in the next column of matrix 300A. The value of index 2 in vector 300C is set to 0, as it corresponds to a non-zero value in vector 300B that is not the first non-zero value in the next column.
[0061] As another example, the bit at index 3 in vector 300C is set to 1 because the corresponding nonzero value (value 1 at index 3 in vector 300B) is the first nonzero value in the next column (specifically, column 2) of matrix 300A. The sequence in vector 300C follows this described pattern until all nonzero values in all columns have been represented.
[0062] Vector 305C corresponds to control values for configuring the input crossbar for the sparse shard. For each non-zero value in shard input matrix 300A, vector 305C specifies the "y" coordinate of the non-zero value in sub-matrix 300A. In this example, the "y" dimension runs vertically up and down sparse matrix 300A, but in other examples the "y" dimension may be defined differently, e.g., horizontally. For example, the value "3" of element zero in vector 305C corresponds to the "y" coordinate of the non-zero value "1" in the first column of matrix 300A. As another example, the value "4" of element six in vector 305C corresponds to the value "1" at the bottom of the fourth column of matrix 300A.
[0063] Vector 310C configures how the sums produced by the adder circuits of the sparse shards are arranged by the output crossbar to produce the shard output vector. The control value corresponds to the
[0064] By the mathematical definition of matrix multiplication, each value at location (x,y) of the sparse matrix is multiplied by the value of the input vector at location y. The result of this multiplication is added to the output at location x. The input crossbar uses vector 305C to order nonzero values in the same column to adjacent multiplier circuits in the sparse shard. The output crossbar uses vector 310C to order the calculated sums in the shard output vector, which represents the product of the multiplication of the shard input matrix and the shard input vector by the sparse shard.
[0065] Returning to FIG. 2, the crossbar 215 may be configured to receive the control values 230 and arrange each value for the shard input vector 205 to be received as an input to one or more multiplication circuits that match a corresponding column of the shard input matrix when the shard input vector is multiplied by the shard input matrix.
[0066] Crossbar 235 may be configured to receive one or more sums from the summation circuits and rearrange the received sums to obtain the correct output shard vectors that correspond to the multiplication of the input shard matrices and the input shard vectors. As with crossbar 215, any of a variety of different techniques for implementing crossbar 235 as one or more circuits may be applied.
[0067] As described herein, the segment marker is configured to gate inputs between adjacent summing circuits depending on the gate input value of the segment marker. For example, if the segment marker receives a control value of one, the segment marker may prevent an output from a first summing circuit adjacent to the segment marker from being passed as an input to a second summing circuit adjacent to the segment marker. If the segment marker receives a control value of zero, the segment marker passes the output from the first summing circuit to the second summing circuit (or vice versa, depending on the implementation). This configuration of the segment marker corresponds to adding only sums corresponding to non-zero values in the same column apart from sums of non-zero values in different columns. As described herein with reference to FIG. 2, the one or more sums may be passed to the crossbar 235 and rearranged to generate the shard output vector 240.
[0068] The crossbar 235 can be configured to determine whether to discard or accept an input and rearrange the input to match its correct position in the shard output vector 240. As shown in FIG. 2, each summing circuit can pass an output to the crossbar 235 (indicated by an arrow to the crossbar 235). If the next segment marker after a summing circuit is not gated, the running sum between adjacent summing circuits has not finished and the crossbar 235 can discard the output to that summing circuit. If the next segment marker is gated (or there is no segment marker in the case of the last summing circuit 225N), the running sum for that column is finished and the crossbar 235 receives the running sum as an input to become part of the output shard vector 240. By knowing which sums to ignore and which sums to include as part of the shard output vector 240, the crossbar 235 can accurately track the column sums to generate output vectors of various lengths, up to a maximum dimension C.
[0069] FIG. 4 illustrates an example of a computation of a sparse shard 400 that receives a shard input matrix 410, a shard input vector 405, and a control value 430 for the shard input matrix 410.
[0070] Consider an example of values for shard input vector 405 and shard input matrix 410: .
[0071]
[0132] (Vector 405) [0 3 0 2 0 4 1 0 0] (matrix 410) The corresponding control vectors 430-432 for matrix 410 in this example are: [1 0 1 1] (control vector 430) [1 2 0 1] (control vector 431) [0 0 1 2] (control vector 432) It is.
[0072] For ease of explanation, the multiplication circuits 40A-D are abbreviated as multipliers A-D, the segment markers 43A-C are abbreviated as segment markers A-C, and the addition circuits 42A-D are abbreviated as adders A-D.
[0073] Based on the non-zero values of matrix 410, multiplier A is loaded with a value of 2, multiplier B is loaded with a value of 1, multiplier C is loaded with a value of 3, and multiplier D is loaded with a value of 4. Note that in this example, the sparse shard only includes four multipliers A-D and four adders A-D.
[0074] Crossbar 415 receives vector 405 having vector values 1, 3, and 2. Dashed lines 45A, solid lines 45B, and dotted lines 45C are shown to indicate the path of data from shard input vector 405 to shard output vector 440. Crossbar 415 receives control vector 431, each value of which corresponds to the "y" coordinate of a respective non-zero value in shard input matrix 410. Values of control vector 431 include 1, 2, 0, and 1. Note that sparse input matrix 410 has dimensions 3×3, so the values of the "y" coordinate range from 0 to 2. Control vector 431 specifies which of the values of shard input vector 405 should be sent to which multiplier for each of multipliers A-D.
[0075] For example, the first value of control vector 431 is 1, corresponding to the “y” coordinate of the first non-zero value in shard input matrix 410. Because “y” coordinate 1 is the second coordinate (after zero), crossbar 415 routes the second value of shard input vector 405 to the first multiplier (here, multiplier A). The second value of control vector 431 is 2, corresponding to the “y” coordinate of the second non-zero value in shard input matrix 410. Crossbar 405 may then be configured to route the third value of shard input vector 405 to multiplier B. As another example, the third value of control vector 431 is 0, corresponding to the next non-zero value having a “y” coordinate of 0. Crossbar 415 routes the first value of shard input vector 405 to multiplier C.
[0076] For the first column of matrix 410, crossbar 415 directs the value 3 to multiplier A and the value 2 to multiplier B. For the second column of matrix 410, crossbar 415 directs the value 1 to multiplier C. For the third and final column of matrix 410, crossbar 415 directs the value 3 to multiplier D. The products of multipliers A to D are 6 (3×2) for multiplier A, 2 (2×1) for multiplier B, 3 (1×3) for multiplier C, and 12 (3×4) for multiplier D.
[0077] Next, adders A-D receive the products calculated by multipliers A-D. Adder A receives the product from multiplier A, which is 6. The first control value (1) is discarded. Since there is no adder before adder A, adder A passes the sum to segment marker A. The second value of control value 430 is zero, so segment marker A is not gated. Adder B receives the current sum (6) from adder A and adds it to the product of multiplier B (2). Since segment marker B is gated, the output of adder B (8) is passed to crossbar 420 (discarded). 420 (shown as dashed line 45C). Finally, multiplier D receives the product of multiplier D (12) and, because it is the last adder in the sparse shard, automatically passes its output (12) to crossbar 420 (shown as solid line 45B). Segment marker B gates the output from adder B, so adder C adds a zero to the product of multiplier C (3). Because segment marker C is gated, the output of adder C (3) is passed to crossbar 420 (shown as dashed line 45C). Finally, multiplier D receives the product of multiplier D (12) and, because it is the last adder in the sparse shard, automatically passes its output (12) to crossbar 420 (shown as solid line 45B).
[0078] Crossbar 420 rearranges the received sums 8, 3, and 12 according to the correct order for outputting output shard vector 440. Crossbar 420 receives control vector 432 with values 0, 0, 2, and 1. As described herein with reference to FIGS. 3A-3C, the value of the control vector for output crossbar 420 corresponds to a non-zero valued "x" coordinate position. As with the "y" coordinate, the value ranges from 0 to 2 in this example. The first two values of control vector 432 are 0. Thus, crossbar 420 routes the first received sum to the first element of output shard vector 440. The next value after 0 for crossbar 432 is 2. Crossbar 432 is configured to route the second sum received from adders A-D to the second element of output vector 440 (indicated by line 45C) and route the third sum to the element (indicated by line 45B). In some examples, crossbar 420 is configured to skip consecutive duplicate control values in vector 432, such as shown in vector 432 where the first two are zero. In some examples, rather than skipping consecutive duplicate control values, crossbar 420 is configured to perform an inclusive OR operation on the received input sums and output the results of this inclusive OR operation in the positions in output vector 440 that correspond to the consecutive duplicate control values.
[0079] For example, line 46 indicates a potential input source of adder A to output crossbar 420. Because segment marker 43A has a value of zero, the output of adder A to crossbar 420 is suppressed, e.g., masked or set to zero. Instead, the output of adder A is passed through segment marker 43A to adder B. In some examples, when output crossbar 420 receives control value 432, output crossbar 420 performs an inclusive OR operation on the first sum (having a value of zero) received (through line 46) and the second sum (having a value of eight from adder B) received (through line 45A). Crossbar 420 may be configured to perform the inclusive OR operation to output a non-zero operand. Crossbar 420 performs an inclusive OR operation on the received sums (e.g., zero). OR 8) and then outputs 8 and places the result in the first position in output vector 440. In some examples, output crossbar 420 can receive individual outputs from at least some of adders A-D and sum them.
[0080] The summation circuits may be implemented according to any of a variety of different circuit configurations for adding consecutive numerical ranges that define segments, with each segment corresponding to the values in a respective column of the shard input matrix processed by the corresponding sparse shard. For example, the summation circuits of sparse shards 200 or 400 may be implemented as one or more sequential segmented summation circuits (e.g., as illustrated by sparse shards 200 or 400) for performing sequential segmented summation of products by multiplying the non-zero values of each column by the respective values of the shard input vector. In some implementations, the summation circuits may be configured to perform the summation tree as parallel segmented summation circuits. Individual summation circuits may be configured to sum corresponding inputs in parallel and pass the sum to an output crossbar and / or adjacent summation circuit according to the gate values of any intervening segment markers, as described herein.
[0081] A parallel segmented sum circuit can reduce the latency of the circuit compared to a sequential segmented sum circuit, especially when the number of terms being summed is large. The control values for gating allow for parallel segmented summation because at least the range of values to be summed can be tracked according to the gate values of the segment markers that gate inputs between summation circuits corresponding to values in different segments and allow inputs between summation circuits corresponding to added values in the same segment.
[0082] In some implementations, output crossbar 420 is configured to receive the individual sums calculated by each adder, add the sums, and use control vector 432 to direct the sums to a corresponding element of output shard vector 440. If two or more sums are directed to the same element, for example if control vector 432 contains duplicates of the same value, such as the first zero in vector 432, then output crossbar 420 is configured to add each received sum so that they are directed to the same element in output shard vector 440, rather than receiving a single sum as shown by line 45A in FIG.
[0083] Example of how to FIG. 5 is a flow diagram of an example process 500 for multiplying a submatrix of a sparse matrix on a sparse shard by a system input vector according to aspects of the disclosure. For ease of explanation, the submatrix of the sparse matrix is referred to herein as a shard input matrix. FIG. 9 illustrates an example process for partitioning an input sparse matrix into multiple submatrices. For example, a sparse shard such as sparse shard 200 of FIG. 2 performs process 500.
[0084] A sparse shard receives a shard input matrix, according to block 510. The shard input matrix is within a predetermined dimensionality threshold and has a non-zero value count less than or equal to a predetermined maximum non-zero threshold.
[0085] In accordance with block 520, the sparse shard receives a shard input vector that includes multiple vector values. The shard input vector is a sub-vector of a system input vector, which may be generated by a pre-processing engine as part of a pre-processing input to the system, as described herein with reference to Figures 1 and 9.
[0086] In accordance with block 530, the sparse shard generates one or more products of a respective vector value multiplied by a respective non-zero value. As described herein with reference to Figures 2 and 4, the sparse shard may be configured with a control value corresponding to the location of the non-zero value in the shard input matrix. Based on the control value, the sparse shard may be configured to direct the received vector value of the partial vector to a corresponding adjacent multiplier circuit in which the non-zero values along the same column of the input shard matrix are stored. Figure 8A described herein illustrates one example of a process for configuring a sparse shard with a control value for a shard input matrix.
[0087] In accordance with block 540, the sparse shard generates one or more sums of the one or more products. As described herein with reference to FIG. 2, the sparse shard includes a number of summing circuits configured to receive inputs from corresponding multiplier circuits. The summing circuits are further configured to sum these inputs along adjacent summing circuits until a segment marker is reached that is configured to gate adjacent circuits to the segment marker. The sum of the summing circuit inputs up to the segment marker may be passed to a crossbar, such as crossbar 235 of FIG. 2.
[0088] From this one or more sums, the sparse shard generates a shard output vector that is the product of the shard input vector multiplied by the shard input matrix. The sparse shard is sensitive to how the operands of the product generated by the multiplication circuit are ordered by the first crossbar. Depending on which totals have been received, the received totals may be rearranged, for example through a second crossbar.
[0089] 6 is a flow diagram of an example of a process 600 for generating a system output vector that represents a product of system input vectors and a system input matrix from multiple sparse shards. A system of sparse shards, such as the sparse matrix multiplication system 100 of FIG. 1, can perform the process 600.
[0090] In accordance with block 610, for each group of sparse shards along the column dimension of the array of sparse shards, the shard output vectors of each sparse shard in the group are summed to generate a column output vector for the group. FIG. 7C, described below, illustrates an example of grouping sparse shards according to their respective shard input matrices. The dimension along which the groups of sparse shards are formed may vary depending on the implementation. For example, depending on the direction in which the system input matrix and system input vectors are provided to the array of sparse shards, the groups may be along the rows of the array as opposed to the columns.
[0091] In accordance with block 620, the system concatenates each column output vector to generate a system output vector. The system output vector is the product of the system input matrix multiplied by the system input vector. In some implementations, as described in more detail herein with reference to FIG. 9, the system may receive a system input matrix in which the columns of the matrix are permuted, e.g., to more evenly distribute the occurrence of non-zero values among the sub-matrices assigned to the sparse shards. In those implementations, the system may be configured to rearrange the elements of the concatenated system output vector to reverse the original permutation. The system may receive data defining the permutation as part of receiving the system input matrix, the control values, and the data defining the partitions of the system input matrix.
[0092] 7A-7C and corresponding description illustrate an example multiplication between an example sparse matrix 700 and a vector 750. For purposes of illustration, the multiplication is described as being performed for a system having 16 sparse shards of a 4×4 array.
[0093] 7A illustrates an example system input matrix 700 and a system input vector 750. In this figure, system input matrix 700 is shown with integer values, although it is understood that the elements may be other values, such as, for example, floating point values. Also, in Figures 7A-7C, non-zero valued elements of the various matrices that comprise system input matrix 700 are shown as shaded cells.
[0094] 7B is a diagram illustrating the partitioning of a system input matrix 700 and a system input vector 750. In this example, the system input matrix 700 is partitioned into 16 sub-matrices 700A-P, one sub-matrix for each sparse shard of the 4×4 array. The system input vector 750 is partitioned into four sub-vectors 750A-D, one sub-vector for each column of the sparse shard of the 4×4 array.
[0095] An example of mapping the partial vectors and partial matrices to 16 sparse shards (referred to as sparse shards A to P) is shown in Table 1 below.
[0096] [Table 1]
[0097] FIG. 7C illustrates a partitioned system input matrix 700 and a system output vector 770 that represents the product of the system input matrix multiplied by a system input vector 750. FIG. 7C shows the partitioned matrix grouped along columns 705A-D. As described herein with reference to FIG. 6, the system may add the shard output vectors for each column of an array of sparse shards to generate a column output vector. In FIG. 7C, column output vectors 710A-D correspond to columns 705A-D. The system may concatenate the column output vectors 710A-D to generate the system output vector as the product of the system input matrix multiplied by the system input vector.
[0098] 8A-8B are flow diagrams of example processes 600A-B for configuring sparse shards and performing matrix multiplication using one or more control values of input matrices according to aspects of the disclosure.
[0099] FIG. 8A illustrates a block diagram of a shard input matrix for sharding using one or more control values according to an aspect of the disclosure. 8 is a flow diagram of an example of a process 800A for configuring a sparse shard. A sparse shard, such as sparse shard 200 of FIG. 2, can perform process 800A.
[0100] According to block 810, the sparse shard receives one or more control values that specify the location of non-zero values along each column of the shard input matrix.
[0101] In accordance with block 820, the sparse shard loads the non-zero values of the shard input matrix into registers of the multiplier circuits. As described herein with reference to FIG. 2, the sparse shard may implement a number of multiplier circuits equal to a predetermined maximum non-zero threshold for the system. The sparse shard may load the non-zero values in the order in which they appear when reading the shard input matrix along a predetermined read direction, such as from left to right.
[0102] In accordance with block 830, the sparse shard loads the one or more control values into the sparse shard's crossbar. A first crossbar, such as crossbar 815 of FIG. 2, may be configured to receive the control values and arrange each value of the shard input vector to be received as an input to one or more multiplier circuits that match the columns of the corresponding shard input matrix by which the vector values are multiplied as part of the matrix-vector multiplication. A second crossbar, such as crossbar 235 of FIG. 2, may be configured to receive one or more sums from the summation circuits and rearrange the received sums to obtain a correct output shard vector corresponding to the multiplication of the input shard matrix and the input shard vector.
[0103] In accordance with block 840, the sparse shard loads the one or more control values into one or more segment markers configured to gate an input to a summation circuit based on the value of the control value.
[0104] FIG. 8B is a flow diagram of an example of a process for performing matrix-vector multiplication using sparse shards configured according to process 800A of FIG. 8A.
[0105] According to block 850, the sparse shard receives and loads the non-zero values of the input shard matrix.
[0106] In accordance with block 860, the sparse shard receives the vector values of the shard input vector and sends these vector values to a multiplication circuit along with the non-zero values along the same column of the shard input matrix.
[0107] In accordance with block 870, the sparse shard generates one or more segmented sums from adjacent summation circuits that are not gated by a segment marker. As described herein with reference to Figures 2 and 8A, the sparse shard can aggregate sums between summation circuits that are not gated by a segment marker and configure the segment marker with a control value to gate summation circuits that represent calculations from different columns of the shard input matrix. When the sparse shard reaches a segment marker with an active gating bit, the sparse shard passes the segmented sums to a second crossbar configured to rearrange the segmented sums to generate the shard output vector.
[0108] The sparse shard generates a shard output vector from the one or more segmented sums in accordance with block 880. As described herein with reference to Figures 2 and 8A, the crossbar may be configured with a control value to receive one or more segmented sums from the summation circuitry of the sparse shard. The crossbar may further configure the shard output vector to receive the one or more segmented sums from the summation circuitry of the sparse shard. to generate the correct shard output vector representing the product of the shard input matrix multiplied by the shard input vector.
[0109] 9 is a flow diagram of an example of a process 900 for generating a submatrix from a system input matrix according to an aspect of the disclosure. One or more processors in one or more locations may perform the process 900. For example, a pre-processing engine such as pre-processing engine 150 of FIG. 1 may perform the process 900.
[0110] According to block 910, the pre-processing engine receives a system input matrix. The system input matrix may be, for example, a matrix such as that shown in system input matrix 700 of FIG. 7A.
[0111] In accordance with block 920, the pre-processing engine partitions the system input matrix into a number of candidate sub-matrices. As part of the partitioning, the pre-processing engine may receive parameters specifying a predetermined dimensionality threshold and one or more parameter values indicating the number of sparse shards implemented by the system receiving the sub-matrix. For example, the pre-processing engine may be configured to generate 16 candidate sub-matrices (for a 4×4 array of sparse shards) within a dimensionality threshold of 8 rows by 8 columns. In some examples, the pre-processing engine may receive updated parameter values, e.g., for pre-processing inputs across different systems having different dimensionality thresholds and / or configurations of sparse shards.
[0112] In some implementations, before the pre-processing engine partitions the system input matrix, the pre-processing engine permutes columns of the system input matrix to distribute non-zero values evenly among the candidate sub-matrices. For example, if non-zero values occur more frequently on one side of the input matrix than on the other side by more than a predetermined tolerance, the pre-processing engine may be configured to reorder the columns of the input matrix such that the occurrence of non-zero values is more spread out, resulting in a more even distribution among the sparse shards after partitioning.
[0113] If the pre-processing engine performs this ordering, it passes data representing the ordering as an additional input to the system, e.g., as part of the partition data sent to a system having an array of sparse shards. The system may be configured to reorder the elements of the system output vector according to the ordering to make the output vector match the output of the system input vector multiplied with the system input matrix before the columns of the system input matrix were permuted.
[0114] Permuting columns of the system input matrix can improve the overall speed at which the system input matrix is processed by a system of sparse shards. For example, permuting columns can allow each sparse shard to be used more efficiently, especially in some instances where some sparse shards may receive shard input matrices having only zero values while other shards may receive shard input matrices having only non-zero values or a number of non-zero values up to a non-zero threshold.
[0115] The pre-processing engine determines whether there is a candidate sub-matrix with a non-zero value count greater than a predetermined non-zero threshold, per block 930. If the pre-processing engine determines there is a candidate sub-matrix with a non-zero value count greater than a predetermined non-zero threshold, then, per block 940, the pre-processing engine repartitions the sub-matrix along the same row or column as the candidate sub-matrix.
[0116] As shown in FIG. 7B, the sub-matrices may be organized along columns and rows based on the location of their values in the input matrix. For example, if sub-matrix 700J has values higher than a non-zero threshold, If the pre-processing engine determines that submatrix 700J contains a nonzero count, the pre-processing engine may repartition the submatrix along the rows of submatrix 700J (including submatrix 700I, 700K, and 700L) and / or along the columns of submatrix 700J (including submatrix 700B, 700F, and 700N). In performing the repartitioning, the pre-processing engine uses a predetermined dimensionality threshold and performs the repartitioning such that the number of candidate submatrices remains unchanged. For example, the pre-processing may split the candidate submatrices and redistribute the rows / columns of the submatrix along the rows / columns of the determined candidate submatrices.
[0117] After repartitioning the submatrices according to blocks 930 and 940, the pre-processing engine again determines whether there are any candidate submatrices with a nonzero value count greater than the nonzero threshold. The pre-processing engine may repeat the determination and repartitioning according to blocks 930 and 940 until it determines that there are no candidate submatrices with a nonzero value count that exceeds the maximum nonzero threshold, and proceed to block 950.
[0118] As shown in block 950, the pre-processing engine partitions the system input vector. Each sub-vector of the system input vector is input to each of the rows or columns of the sparse shards, depending on how the buses feeding the array of sparse shards are arranged. The pre-processing engine partitions the vector such that the vector dimensions can be multiplied by the matrices of the received sparse shards, e.g., by valid dimensions for mathematically valid matrix multiplication.
[0119] For example, as shown in Figures 7B-7C and Table 1, vector 750 is partitioned into sub-vectors 750A-D, each of which is passed as an input to one or more sparse shards. Also in Figure 7B, each dimension of each sub-vector 750A-D has the correct dimension for multiplication with a corresponding sub-matrix 700A-P. For example, sub-vector 750B is 1x2 (row x column), and each of sub-matrices 700E-H has two rows, allowing valid matrix multiplication between sub-vector 750B and sub-matrices 700E-H.
[0120] As indicated in block 960, the pre-processing engine generates control values for each candidate sub-matrix. As described with reference to Figures 3A-3C, the pre-processing engine may generate a vector of control values indicating a starting non-zero value for each column of the matrix. The pre-processing engine repeats this generation for each candidate sub-matrix to generate corresponding control values for each sub-matrix.
[0121] The pre-processing engine outputs the control values and the candidate sub-matrices, according to block 970. The pre-processing engine may output the control values and data specifying the partition of the system input matrix, for example, to a system 100 as shown in FIG.
[0122] Example Computing Environment FIG. 10 is a block diagram of an example computing environment for implementing a sparse matrix multiplication system 100 and a pre-processing engine 150 according to aspects of the disclosure. The pre-processing engine 150 can be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 1015. The user computing device 1012 and the server computing device 1015 can be communicatively coupled to one or more storage devices 1030 via a network 1060. The storage device(s) 1030 can be a combination of volatile and non-volatile memory and can be in the same or different physical location as the computing devices 1012, 1015. For example, the storage device(s) 1030 can be a hard drive, solid state drive, tape drive, optical storage, memory cards, ROM, RAM, DVD, CD-ROM, writable memory, and read-only memory. The present invention may include any type of non-transitory computer-readable medium capable of storing information, such as read-only memory.
[0123] The server computing device 1015 may include one or more processors 1013 and memory 1014. The memory 1014 may store information accessible to the processor(s) 1013, including instructions 1021 that may be executed by the processor(s) 1013. The memory 1014 may also include data 1023 that the processor(s) 1013 may retrieve, manipulate, or store. The memory 1014 may be a type of non-transitory computer-readable medium that may store information accessible to the processor(s) 1013, such as volatile and non-volatile memory. The processor(s) 1013 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FGPAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).
[0124] The server computing device 1015 may implement the sparse matrix multiplication system 100 in hardware, for example as a system on a chip. The system 100 may be implemented as part of a physical chip slotted into or installed in the server computing device 1015. The system 100 is configured to communicate with other components of the server computing device 1015.
[0125] The instructions 1021 may include one or more instructions that, when executed by the processor(s) 1013, cause the one or more processors to perform an operation defined by the instructions. The instructions 1021 may be stored in object code format for direct processing by the processor(s) 1013, or in other formats including interpretable scripts or collections of independent source code modules that are interpreted on-demand or pre-compiled. The instructions 1021 may include instructions for implementing a sparse shard 400 consistent with aspects of the disclosure. The pre-processor engine 105 may be executed with the processor(s) 1013 and / or with other processors located remotely from the server computing device 1015.
[0126] The data 1023 may be retrieved, stored, or modified by the processor(s) 1013 according to the instructions 1021. The data 1023 may be stored in computer registers, in a relational or non-relational database as a table with multiple different fields and records, or as a JSON, YAML, proto, or XML document. The data 1023 may also be formatted into a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. Additionally, the data 1023 may include information sufficient to identify relevant information or information used by a function to compute the relevant data, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memory, including other network locations, etc.
[0127] The user computing device 1012 may be configured similarly to the server computing device 1015, having one or more processors 1016, memory 1017, instructions 1018, and data 1019. The user computing device 1012 may also include a user output 1026 and a user input 1024. The user input 1024 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touch screen, a microphone, and a sensor.
[0128] The server computing device 1015 may be configured to transmit data to the user computing device 1012, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 1026. The user output 1026 may also be used to display an interface between the user computing device 1012 and the server computing device 1015. The user output 1026 may alternatively or additionally include one or more speakers, transducers or other audio output, a haptic interface or other tactile feedback that provides non-visual and non-audible information to a platform user of the user computing device 1012.
[0129] Although FIG. 10 illustrates the processors 1013, 1016 and memories 1014, 1017 as being internal to the computing devices 1015, 1012, the components described herein, including the processors 1013, 1016 and memories 1014, 1017, may include multiple processors and memories that may operate in different physical locations and are not internal to the same computing device. For example, some of the instructions 1021, 1018 and data 1023, 1019 may be stored on a removable SD card and other parts may be stored in a read-only computer chip. Some or all of the instructions and data may be stored in a location that is physically separate from the processors 1013, 1016 but accessible to the processors 1013, 1016. Similarly, the processors 1013, 1016 may include a collection of processors that may perform simultaneous and / or sequential operations. Each of the computing devices 1015, 1012 may include one or more internal clocks that provide timing information that may be used to time operations and programs executed by the computing devices 1015, 1012.
[0130] The server computing device 1015 may be configured to receive requests to process data from the user computing devices 1012. For example, the environment 1000 may be part of a computing platform configured to provide various services to users via various user interfaces and / or APIs that expose platform services. As part of executing the services, the server computing device 1015 may process the received data using the system 100. For example, if the service is training a machine learning model, the server computing device 1015 may be configured to use the system 100 to perform a multiplication operation as part of training the machine learning model.
[0131] The devices 1012, 1015 may be capable of direct and indirect communication over the network 1060. The devices 1015, 1012 may set up listening sockets that may accept initiating connections to send and receive information. The network 1060 itself may include a variety of configurations and protocols, such as the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks that use communication protocols proprietary to one or more companies. The network 1060 may support a variety of short-range and long-range connections. The short-range and long-range connections may be made over different bandwidths, such as 2.402 GHz to 2.480 GHz (commonly associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (commonly associated with the Wi-Fi® communication protocol), or with various communication standards, such as the LTE® standard for wireless broadband communication. The network 1060 may also or alternatively support wired connections between the devices 1012, 1015, such as via various types of Ethernet® connections.
[0132] 10 shows one server computing device 1015 and one user computing device 1012, it is understood that aspects of the disclosure can be implemented according to a variety of different configurations and quantities of computing devices, such as in a paradigm for sequential or parallel processing, or via a distributed network of multiple devices. In some implementations, aspects of the disclosure can be performed on one device, and any combination thereof. Furthermore, while the pre-processing engine and the sparse matrix multiplication system 100 are shown as being implemented on the same server computing device 1015, in some implementations, the pre-processing engine 150 is implemented on one or more server computing devices and / or user computing devices 1012 that are separate from the server computing device 1015.
[0133] Aspects of the disclosure can be implemented as digital circuitry, as one or more computer programs in a computer-readable storage medium, or as a combination of one or more of the above. The computer-readable storage medium may be non-transitory, for example, as one or more instructions executable by a computing device and stored on a tangible storage device.
[0134] In this specification, the phrase "configured to" is used in different contexts relating to a computer system, hardware, or part of a computer program, engine, or module. When a system is said to be configured to perform one or more operations, this means that the system has installed thereon appropriate software, firmware, and / or hardware that, when operated, causes the system to perform the one or more operations. When a piece of hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, when operated, receive an input and generate an output corresponding to the one or more operations according to the input. When a computer program, engine, or module is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform the one or more operations.
[0135] Although the operations illustrated in the figures and recited in the claims are shown in a particular order, it is understood that the operations may be performed in an order different from that shown, and that some operations may be omitted, performed multiple times, and / or performed in parallel and / or concurrently with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring separation of the components. The described components, modules, programs, and engines may be integrated as one system or may be part of multiple systems.
[0136] Unless otherwise stated, the above alternatives are not mutually exclusive and may be implemented in various combinations to achieve specific advantages. These and other variations and combinations of the above features may be utilized without departing from the subject matter defined by the claims, and therefore the above description of the embodiments should be interpreted as illustrating, but not limiting, the subject matter defined by the claims. In addition, the provision of the embodiments described herein, as well as clauses such as "such as" and "including," should not be interpreted as limiting the subject matter of the claims to a particular embodiment, but rather the embodiment is intended to show only one of many possible implementations. Furthermore, the same reference numbers in various figures may identify the same or similar elements.
Claims
1. A sparse shard including a plurality of multiplication circuits, the sparse shard comprising: a shard input matrix including a number of non-zero values equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to a number of the plurality of multiplier circuits, the multiplier circuits receiving each non-zero value of the shard input matrix, the sparse shard further comprising: receiving a shard input vector containing multiple vector values; generating products of vector values multiplied by the respective non-zero values of the shard input matrix by the plurality of multiplication circuits; the system is configured to use one or more of the products to generate a shard output vector that is the product of applying the shard input vector with the shard input matrix.
2. The system of claim 1, wherein the shard input matrix has a dimension less than or equal to a predetermined dimensionality threshold corresponding to a maximum matrix input size.
3. The sparse shard is one of a plurality of sparse shards, the plurality of sparse shards comprising: receiving a number of shard input matrices that are submatrices of the system input matrix; receiving a plurality of shard input vectors that are subvectors of a system input vector; 3. A system according to claim 1 or 2, configured to generate a system output vector representing the product of said system input vector applied to said system input matrix.
4. The system of claim 3, wherein when generating the system output vector, the multiple sparse shards are further configured to concatenate their respective shard output vectors to generate the system output vector.
5. A system as described in any one of claims 1 to 4, wherein the multiplication circuit is coupled to a register containing the respective non-zero value.
6. A system described in any one of claims 1 to 5, wherein the predetermined non-zero threshold is a maximum non-zero threshold.
7. The sparse shard further includes a crossbar circuit, the crossbar circuit comprising: receiving the plurality of vector values of the shard input vector; 7. A system according to claim 1, configured to provide as input to each of the plurality of multiplication circuits one vector value of the plurality of vector values in accordance with one or more control values.
8. The system of any one of claims 1 to 7, wherein the sparse shard is further configured to load non-zero values in a same column of the shard input matrix into registers of adjacent multiplier circuits of the plurality of multiplier circuits.
9. 10. The system of claim 8, wherein the sparse shard is further configured to receive one or more control values that specify positions of non-zero values along columns of the shard input matrix.
10. The system of any one of claims 1 to 9, wherein the sparse shard further comprises a plurality of summation circuits configured to generate one or more sums of one or more of the products of vector values.
11. The system of claim 10 , wherein the plurality of summing circuits form a parallel segmented summing circuit.
12. The sparse shard further comprises a crossbar circuit, the crossbar circuit comprising: receiving said one or more sums, 12. The system of claim 10 or 11, configured to arrange the one or more sums according to one or more control values to generate the sharded output vector.
13. The system of claim 12 , wherein the crossbar circuitry forms a Benesh network.
14. A method comprising: The method further comprises: a sparse shard including a plurality of multiplier circuits receiving a shard input matrix including a number of non-zero values less than or equal to a predetermined non-zero threshold, the non-zero threshold corresponding to a number of the plurality of multiplier circuits, the multiplier circuits receiving each non-zero value of the shard input matrix; the non-zero threshold corresponding to a number of the plurality of multiplier circuits; receiving, by the sparse shard, a shard input vector including a plurality of vector values; the plurality of multiplication circuits generating products of vector values multiplied by the respective non-zero values of the sharded input matrix; the sparse shard using one or more of the products to generate a shard output vector that is a product of applying the shard input vector with the shard input matrix.
15. The method of claim 14, wherein the shard input matrix has a dimension less than or equal to a predetermined dimensionality threshold corresponding to a maximum matrix input size.
16. The method of claim 15, wherein a plurality of sparse shards, of which the sparse shard is one, receive a plurality of shard input matrices that are submatrices of a system input matrix; the plurality of sparse shards receiving a plurality of shard input vectors that are sub-vectors of a system input vector; The method of claim 14 or 15, further comprising: the plurality of sparse shards generating a system output vector that represents a product of applying the system input vector to the system input matrix.
17. The method of claim 16, wherein generating the system output vector further comprises the multiple sparse shards concatenating their respective shard output vectors to generate the system output vector.
18. A method according to any one of claims 14 to 17, wherein the multiplication circuit is coupled to a register containing the respective non-zero value.
19. A method according to any one of claims 14 to 18, wherein the predetermined non-zero threshold is a maximum non-zero threshold.
20. One or more computer programs that, when executed by a system having a sparse shard including a plurality of multiplication circuits, cause the system to perform a method according to any one of claims 14 to 19.
