Sparse matrix multiplication in hardware

Abstract

The present disclosure relates to sparse matrix multiplication in hardware. Methods, systems, and apparatuses, including computer-readable storage media, are provided for sparse matrix multiplication. A system for matrix multiplication includes an array of sparse tiles. Each sparse tile can be configured to receive an input submatrix and an input subvector, where the input submatrix has a number of non-zero values that is equal to or less than a predetermined maximum non-zero threshold. The sparse tile can compute, through a plurality of multiplier circuits, one or more products of a vector value multiplied by respective non-zero values of the input submatrix. The sparse tile can generate a tile output vector that is an output of the sparse tile and uses the one or more products, the tile output vector being a product of applying the tile input vector to the tile input matrix.

Description

[0001] Case Analysis

[0002] This application is a divisional application of patent application No. 202111665133.7 filed on December 30, 2021, entitled "Sparse Matrix Multiplication in Hardware". Technical Field

[0003] This disclosure relates to sparse matrix multiplication in hardware. Background Technology

[0004] A sparse matrix is ​​a matrix in which the ratio of zero to non-zero values ​​is relatively high. Different sparse matrices can have different degrees of sparsity based on the ratio of zero to non-zero values. A matrix with a higher ratio of zero to non-zero values ​​is said to have higher sparsity compared to a matrix with a lower ratio.

[0005] A neural network is a machine learning model that includes one or more non-linear operation layers to predict outputs in response to received inputs. In addition to input and output layers, some neural networks also include one or more hidden layers. The output of each hidden layer can be fed into another hidden layer or the output layer of the neural network. Each layer of a neural network can generate a corresponding output from the received input based on the values ​​of one or more model parameters used for that layer. Model parameters can be weights or biases determined by a training algorithm to enable the neural network to generate accurate outputs. The model parameter values ​​used for layers in a neural network can be represented as elements of a matrix or tensor. Summary of the Invention

[0006] This disclosure relates to sparse matrix-dense vector multiplication in hardware.

[0007] One aspect of this disclosure provides a system comprising: a sparse slice including a plurality of multiplier circuits, wherein the sparse slice is configured to: receive a slice input matrix including a plurality of nonzero values ​​equal to or less than a predetermined maximum nonzero threshold; receive a slice input vector including a plurality of vector values; for each of the multiplier circuits, receive a corresponding nonzero value of the slice input matrix; generate one or more products by the plurality of multiplier circuits multiplying the vector values ​​by the corresponding nonzero values ​​of the slice input matrix; and generate a slice output vector as the output of the sparse slice and using the one or more products, the slice output vector being a product of applying the slice input vector to the slice input matrix.

[0008] Another aspect of this disclosure provides one or more non-transitory computer-readable storage media storing instructions that, when executed by a system comprising a plurality of sparse slices, cause the system to perform operations including: receiving a sliced ​​input matrix and a sliced ​​input vector by a sparse slice comprising a plurality of multiplier circuits, wherein the sliced ​​input matrix comprises a plurality of non-zero values ​​equal to or less than a predetermined maximum non-zero threshold, and the sliced ​​input vector comprises a plurality of vector values; receiving a corresponding non-zero value of the sliced ​​input matrix for each of the multiplier circuits; generating one or more products by the plurality of multiplier circuits of the sparse slice, wherein the corresponding vector value is multiplied by the corresponding non-zero value; and generating a sliced ​​output vector as an output to the sparse slice and using the one or more products, the sliced ​​output vector being a product of applying the sliced ​​input vector to the sliced ​​input matrix.

[0009] Another aspect of this disclosure provides a method comprising: receiving a piecewise input matrix and a piecewise input vector by a sparse piecewise segment comprising a plurality of multiplier circuits, wherein the piecewise input matrix comprises a plurality of nonzero values ​​equal to or less than a predetermined maximum nonzero threshold, and the piecewise input vector comprises a plurality of vector values; receiving a corresponding nonzero value of the piecewise input matrix for each of the multiplier circuits; generating one or more products by the plurality of multiplier circuits of the sparse piecewise segment multiplying the corresponding vector value by the corresponding nonzero value of the piecewise input matrix; and generating a piecewise output vector as the output of the sparse piecewise segment and using the one or more products, the piecewise output vector being a product of applying the piecewise input vector to the piecewise input matrix.

[0010] The foregoing and other aspects may optionally include one or more of the following features, individually or in combination. One implementation may include a combination of all of the following features.

[0011] The length of the segmented output vector is greater than one.

[0012] The sparse slice is one of a plurality of sparse slices, which are configured to: receive a plurality of sliced ​​input matrices as submatrices of the system input matrix; receive a plurality of sliced ​​input vectors as subvectors of the system input vector; and generate a system output vector from the plurality of sparse slices, the system output vector representing the product of applying the system input vector to the system input matrix.

[0013] The plurality of sparse fragments are arranged as a systolic array, the systolic array comprising one or more groups of sparse fragments along the column dimension of the systolic array; and wherein, in order to generate the system output vector, the one or more processors are further configured to: for each group along the column dimension of the systolic array, add each sparse fragment in the group together with the corresponding fragment output vector to generate the corresponding column output vector; and concatenate the corresponding column output vectors of each group to generate the system output vector.

[0014] Each multiplier circuit is coupled to a corresponding register, which includes the corresponding non-zero value from the corresponding slice input matrix of the sparse slice.

[0015] The number of multiplier circuits in the plurality of multiplier circuits is equal to the predetermined maximum non-zero threshold.

[0016] The sparse sharding further includes a cross-switching circuit, and wherein the sparse sharding is further configured to: receive the plurality of vector values ​​of the sharding input vector by the cross-switching circuit; and send a vector value of the plurality of vector values ​​as input to each of the plurality of multiplier circuits and by the cross-switching circuit.

[0017] The sparse slicing is further configured to load non-zero values ​​of the same column in the slicing input matrix into the registers of adjacent multiplier circuits in the plurality of multiplier circuits.

[0018] The sparse sharding is further configured to receive one or more control values, the one or more control values ​​specifying at least the position of a non-zero value along each column of the sharding input matrix; and wherein the cross-switching circuitry for the sparse sharding is further configured to: receive the one or more control values; and, based on the one or more control values, send a vector value to an adjacent multiplier circuitry to be multiplied by a non-zero value along the same column of the sharding input matrix.

[0019] The sparse slicing further includes a plurality of adder circuits, wherein the sparse slicing further includes one or more segment markers, wherein each segment marker is configured to select the input of a corresponding adder circuit among the plurality of adder circuits based on the value of a corresponding control value loaded in the segment marker; and wherein the sparse slicing is further configured to: load at least a portion of the one or more control values ​​in the one or more segment markers, wherein adder circuits for non-zero values ​​in a first column of the slicing input matrix are selected by receiving inputs from adjacent adder circuits, the inputs including non-zero values ​​in a second column of the slicing input matrix that are different from the first column; and generate one or more sums of the one or more products by the plurality of adder circuits, wherein each of the one or more sums is a corresponding piecewise sum of multiplying one or more non-zero values ​​of a column of the slicing input matrix with one or more corresponding values ​​of the slicing input vector.

[0020] The plurality of adder circuits form a parallel segmented summation circuit, wherein each of the one or more segmented sums is a sum to the output of an adjacent adder circuit that is not selected by a segment marker.

[0021] The cross-switching circuit is a first cross-switching circuit; and wherein the sparse sharding further includes a second cross-switching circuit, the second cross-switching circuit being configured to: receive the one or more segments and, according to the one or more control values, arrange the one or more segments to generate a corresponding sharding output vector for the sparse sharding.

[0022] The second cross-sectional switching circuit can form a Beneath (Beneath) The network, wherein the piecewise input matrix is ​​a square matrix.

[0023] Another aspect of this disclosure provides a system comprising one or more processors; and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving an input matrix comprising zero and non-zero values; and partitioning the input matrix into a plurality of submatrices, wherein the number of non-zero values ​​in each submatrix is ​​equal to or less than a predetermined maximum non-zero threshold, and wherein the dimension of each submatrix is ​​equal to or less than a predetermined dimension threshold.

[0024] The operation may further include generating one or more corresponding control values ​​for each submatrix, the one or more corresponding control values ​​specifying the position of non-zero values ​​along each column of the submatrix.

[0025] The operation may further include: sending each submatrix and one or more corresponding control values ​​for each submatrix to a plurality of sparse slices, the plurality of sparse slices being configured to process each submatrix and the corresponding control values ​​for the submatrix. Each of the plurality of sparse slices is configured to: receive a submatrix, the one or more corresponding control values ​​for the submatrix, and at least a portion of an input vector, and generate a corresponding output slice vector, the corresponding output slice vector representing the product of a portion of the submatrix and the input vector. According to the system of claim 21, partitioning the input matrix includes partitioning the matrix into a plurality of submatrices equal to the number of sparse slices in the plurality of sparse slices.

[0026] Other aspects of this disclosure include corresponding systems, apparatuses, and computer programs stored on one or more non-transitory computer-readable storage media. Attached Figure Description

[0027] Figure 1 This is a block diagram of an example system including a sparsely segmented array according to aspects of this disclosure.

[0028] Figure 2 This is a block diagram illustrating an example of sparse fragmentation based on aspects of this disclosure.

[0029] Figure 3A This is a diagram of an example piecewise input matrix used for sparse piecewise partitioning.

[0030] Figure 3B This is a diagram illustrating the vector of non-zero values ​​of the example sliced ​​input matrix.

[0031] Figure 3C This is a diagram illustrating the vector of control values ​​used to illustrate the sliced ​​input matrix.

[0032] Figure 4 The illustration shows an example matrix multiplication that receives a sparsely sliced ​​input matrix, a sliced ​​input vector, and control values ​​for the sliced ​​input matrix.

[0033] Figure 5 This is a flowchart illustrating an example process for multiplying a submatrix of a sparse matrix on a sparse piece with a system input vector, according to aspects of this disclosure.

[0034] Figure 6 This is a flowchart illustrating an example process of generating a system output vector representing the product of the system input vector and the system input matrix from multiple sparse slices.

[0035] Figure 7A This is a diagram illustrating the example system input matrix and system input vector.

[0036] Figure 7BThis is a diagram illustrating the partitioning of the system input matrix and system input vector.

[0037] Figure 7C This is a diagram of the system input matrix and system output vector of the partition, where the system output vector represents the product of the system input matrix and the system input vector.

[0038] Figure 8A This is a flowchart of an example process for configuring sparse sharding using one or more control values ​​of a sharded input matrix, according to aspects of this disclosure.

[0039] Figure 8B It is used according to Figure 8A The flowchart shows an example process for performing matrix-vector multiplication using sparse partitioning configured in the process configuration.

[0040] Figure 9 This is a flowchart illustrating an example process for generating a submatrix from a system input matrix, according to aspects of this disclosure.

[0041] Figure 10 This is a block diagram of an example computational environment for implementing a sparse matrix multiplication system according to aspects of this disclosure. Detailed Implementation

[0042] Overview

[0043] Various aspects of this disclosure relate to a system comprising one or more integrated circuits configured for sparse matrix-dense vector multiplication. The system, comprising multiple sparse slices, can receive submatrices and subvectors of an input system matrix and vector for each sparse slice. Each sparse slice may be at least part of an integrated circuit and implement several arithmetic units, such as multiplier circuitry and adder circuitry. Each sparse slice is configured to receive a submatrix having several non-zero values ​​equal to or less than a maximum non-zero threshold, which may be predetermined when the system is implemented, for example, as a chip including sparse slices and optionally one or more other components as described herein.

[0044] Each sparse slice can additionally receive metadata, such as the control values ​​described herein, which the sparse slice can use to direct different inputs to each multiplier or adder circuit and to gating when the output from one unit is passed to another. By configuring the sparse slices according to the received metadata, these sparse slices can efficiently process input submatrices of arbitrary size up to a predetermined dimension threshold and output the product as a vector of length corresponding to the input.

[0045] As part of the receiving submatrix, each sparse slice can receive one or more control values, which represent the position of a non-zero value at each column of the submatrix. Using these control values, the sparse slice can be configured to adjust how the individual values ​​of the submatrix and subvectors are multiplied, added, and arranged so that the sparse slice generates a sliced ​​output vector representing the product of the submatrix and subvectors. The system can be further configured to generate from the sliced ​​output vector of each slice a system output vector representing the product of applying the system input vector to the system input matrix (e.g., by multiplying the system input vector by the system input matrix).

[0046] This disclosure also relates to systems for preprocessing sparse matrices for processing by an array of sparse slices. A system with one or more processors can be configured to partition an input matrix into multiple submatrices, wherein the dimension of each submatrix is ​​equal to or less than a predetermined dimension threshold of the maximum matrix input size for a specified sparse slice. As part of the partitioning, the system can identify whether any submatrix includes a number of nonzero values ​​greater than a predetermined maximum nonzero threshold, and in response, repartition the submatrix along the same rows or columns as the identified submatrix. The system can repeat this process until submatrices are generated for each sparse slice such that the dimension of the submatrix is ​​within the dimension threshold and the number of nonzero values ​​in the submatrix is ​​less than or equal to the predetermined nonzero threshold.

[0047] Systems implemented according to aspects of this disclosure can perform workloads involving repetitive multiplications of sparse matrices and vectors more efficiently. For example, a system-on-chip (SoC) device implementing aspects of this disclosure can generate products of sparse matrix-vector multiplications in fewer processing cycles than conventional methods, at least because the device omits redundant "multiply-by-zero" computations of zero elements in the matrix, which do not contribute to the final product. While other methods may experience increased performance loss for larger matrices with high sparsity ratios, systems implemented as described herein can use these matrices to compute products even more efficiently, at least because a higher proportion of redundant computations are omitted as the sparsity ratio of the input matrix increases.

[0048] Some workloads, such as executing or training neural networks, heavily rely on performing matrix multiplication. Hardware accelerators or other devices can perform certain operations, such as matrix multiplication, efficiently, but often have limitations on processing power. For example, a device might require a predefined sparsity ratio between the zero and non-zero values ​​of the matrix, or have strict limitations on the size of the input to be processed. While matrix multiplication is a common type of computation for many neural network workloads, accelerators with processing limitations are limited in the number of types of workloads (such as workloads involving matrix multiplication of matrices with different sparsity ratios) that the accelerator can support.

[0049] This disclosure provides a system for sparse matrix multiplication that can flexibly handle matrices of varying sizes and sparsity ratios. The sparse slices configured for sparse matrix multiplication can discard zero values ​​of the input matrix while still tracking the positions of non-zero values ​​across multiple columns in a given input submatrix. The sparse slices can be configured with only as many registers and multiplier circuitry as a predetermined maximum non-zero threshold, and only store the non-zero values ​​of the input submatrix in memory. As a result, sparse slices can store and process input data more efficiently and with fewer resources compared to storing the full-size matrix.

[0050] Furthermore, sparse piecewise operations can be configured to arrange non-zero values ​​and their products when multiplied by the values ​​of the input vectors along adjacent multiplier and adder circuits, respectively. Sparse piecewise operations can be configured to efficiently generate piecewise sums for each column of the input matrix by combining the outputs of these adjacent circuits, and rearrange the piecewise sums to generate a piecewise output vector. Sparse piecewise operations can preserve the order of non-zero values ​​to accurately add and multiply these values ​​with corresponding values ​​of the input vectors, and accurately generate a piecewise output vector as the product of the input submatrix and the input vector. Sparse piecewise operations can perform multiplication on the input matrix without requiring preprocessing to change the shape of the input matrix.

[0051] The system may also include one or more processors configured to preprocess the sparse matrix for multiplication by multiple sparse slices with the input vector. The system may support different configurations of sparse slices, including generating different numbers of submatrices depending on the number of available sparse slices, and generating submatrices for different maximum non-zero thresholds.

[0052] The aspects of this disclosure as described herein can be implemented by a system comprising one or more processors and multiple sparse slices. This system can be implemented, for example, as a chip on a server computing device installed in a data center, such as a computing platform.

[0053] Example system

[0054] Figure 1 This is a block diagram of an example system 100 including an array 101 of sparse slices 101A-101P. The array 101 may be at least a part of a sparse matrix multiplication system (such as sparse matrix multiplication system 100) configured to perform matrix multiplication on matrix inputs.

[0055] As referenced in this article Figure 2In more detail, a sparse slice is a collection of circuits configured to perform arithmetic operations, arrange inputs and outputs between circuits performing arithmetic operations, and / or gate inputs and outputs between other circuits. For example, a sparse slice can be configured to perform matrix multiplication between rectangular slices or portions of a sparse matrix and linear slices or portions of a vector. As described in more detail herein, a sparse slice can receive any of a variety of different slices or portions of a sparse matrix, wherein any number of non-zero values ​​are present in said slice or portion up to a predetermined maximum threshold.

[0056] The set of circuits can be configured to control, as described herein, the state of each gate (i.e., open or closed); the operands to be computed; and / or the arrangement of the inputs to each circuit and the outputs from each circuit. The configuration may depend at least in part on the dimension of the piecewise input matrix in which the sparse pieces are configured to receive as inputs, and / or on the arrangement and number of non-zero values ​​in the piecewise input matrix. Each sparse piece may be a separate circuit component of a system configured to multiply sparse matrices with vectors.

[0057] Each sparse slice 101A-101P is configured to communicate with two or more other sparse slices in array 101. A sparse slice can communicate with its immediate neighbors, for example, neighbors immediately before or after the sparse slice along a dimension defined by the rectangular arrangement of the sparse slices 101A-101P. Connections between sparse slices can be implemented, for example, via a bus or one or more circuit interconnects that physically connect the sparse slices to their neighbors. Each sparse slice 101A-101P can be implemented as one or more circuits configured to receive at least a portion of a matrix and at least a portion of a vector and generate an output vector representing the product of the input matrix and vector. In some examples, the sparse slices 101A-101P are organized as a pulsating array, but in various implementations, array 101 is typically configured or arranged according to a rectangular arrangement of the sparse slices 101A-101P.

[0058] Array 101 may be at least part of a system-on-a-chip that implements several components and integrated circuits on a circuit board or other material. Array 101 may be mounted as part of a computing device and configured to interact with other components of the device, including memory, processor, network components, and / or peripheral devices. For example, array 101 may receive a system input vector 105 and a system input matrix 110 from one or more memory devices implemented as part of system 100. System 100 may generate a system output vector 115 as output, representing the product of the system input vector 105 and the system input matrix 110.

[0059] In some examples, the system output vector 115 can be fed as input to other devices or components implementing system 100. For example, if the system output vector 115 is a product of model parameter values ​​multiplied by a vector input of the neural network, the system output vector 115 can be fed as input to one or more processors configured to compute activation functions for the output vector 115.

[0060] System 100 may receive system input vector 105 and system input matrix 110 from preprocessing engine 150. Preprocessing engine 150 may be implemented on one or more computing devices, which may or may not be the devices implementing system 100. As an example, system input matrix 110 may include values ​​representing at least a portion of the model parameter values ​​used for the neural network. System input matrix 110 may be part of a more complex data structure, such as a multidimensional array or tensor. System 100 may be configured to receive each matrix, such as a three-dimensional tensor or matrix, corresponding to at least a portion of a larger data structure, and generate a corresponding output when multiplied by system input vector 105. System input vector 105 may be, for example, the input to a trained neural network whose model parameter values ​​are at least partially represented by the values ​​of system input matrix 110.

[0061] The preprocessing engine 150 can be configured to process the system input matrix 110 to generate one or more control values ​​111 and matrix partition data 112. The preprocessing engine 150 can receive the system input matrix 110 in various formats used for storing sparse matrices, such as its full row and column form (with all non-zero and zero values). As other examples, the preprocessing engine 150 can receive the system input matrix 110 according to a compressed sparse column format, a coordinate list format, or any of various other formats used for storing sparse matrices.

[0062] In some examples where the preprocessing engine 150 receives the system input matrix 110 in its complete form, the preprocessing engine 150 is configured to convert the system input matrix 110 into a format predetermined to be suitable for storage in memory. For example, before conversion to the predetermined format, the preprocessing engine 150 may remove zero values ​​from the matrix and generate control values ​​that are used to track the position of non-zero values ​​relative to their original positions in the matrix.

[0063] As referenced in this article Figure 2The control values, described in more detail, are used to configure each sparse slice to receive a subvector of the system input vector 105 and multiply it by a submatrix of the system input matrix 110. Partition data 112 specifies how the system input matrix 110 should be partitioned into submatrices. Each submatrix is ​​received as input at the corresponding sparse slice, and partition data 112 specifies as many partitions of submatrices as there are sparse slices in array 101.

[0064] The preprocessing engine 150 can be configured to generate vector partitioned data 106. (See reference...) Figures 7A to 7C In more detail, sparse slices along the same rows or columns can receive the same subvectors as the input for multiplication with the corresponding submatrix.

[0065] Although system input matrix 110 and system input vector 105 are shown as being fed along the left and right sides of system 100, the exact location and orientation of the buses that feed input data to system 100 can vary depending on the implementation. For example, based on the location of other components on the same chip as system 100, the bus or circuit interconnects used to feed input to system 100 and receive output from system 100 can be differently oriented or positioned to illustrate the location of those other components.

[0066] Figure 2 This is a block diagram of an example sparse fragment 200 based on various aspects of this disclosure. For example, sparse fragments 101A-101P of system 100 can each be implemented as described herein with reference to sparse fragment 200.

[0067] Sparse slice 200 is configured to receive slice input vector 205 and slice input matrix 210. Slice input vector 205 includes one or more vector values ​​and may have a maximum dimension of 1×R. Slice input matrix includes one or more zero values ​​and one or more non-zero values ​​and has a maximum dimension of R×C. R (rows) and C (columns) are predetermined dimension thresholds corresponding to the maximum input size of the vector / matrix that the sparse slice can receive as input. In different implementations, sparse slices can be configured for different dimensions R and C. R and C may be equal to or different from each other, and different sparse slices can be implemented for different dimensions, for example, in response to the nature of the data of different workloads to be processed by the sparse slice. The sparse slices of the array of sparse slices can be configured to receive input within the same maximum dimension threshold.

[0068] Sparse slice 200 can be configured to receive slice input vector 205 and slice input matrix 210 in a predetermined address range in memory associated with sparse slice 200. For example, sparse slice 200 is configured to automatically retrieve slice input vector 205 from a first address range in coupled memory and slice input matrix 210 from the same or different address ranges in memory. A device or component implementing system 100 having sparse slice 200 can be configured to send slice input matrix and slice input vector to a location in memory corresponding to each of the one or more sparse slices implemented by the system. For example, after generating the processing system input matrix and / or system input vector, preprocessing engine 150 can be configured to store individual slice input matrices and slice input vectors at an address range corresponding to each sparse slice (including sparse slice 200).

[0069] In addition to the maximum dimension threshold, sparse slice 200 is configured to also receive sliced ​​input matrices having a non-zero count equal to or less than a predetermined maximum non-zero threshold. Similar to the dimension threshold, a maximum non-zero threshold, which varies depending on the implementation, can be set for sparse slice 200 and its corresponding array. For example, if the system is configured to process workloads whose data typically includes processing matrices with a high sparsity ratio, the sparse slice system can be configured with a relatively high maximum non-zero threshold. As described herein, the number of multiplier and adder circuits in a sparse slice corresponds to its maximum non-zero threshold; therefore, sparse slices with relatively low maximum non-zero thresholds can be constructed with fewer circuits compared to sparse slices with higher maximum non-zero thresholds.

[0070] Sparse slice 200 may include cross switches 215 and multiplier circuits 220. Cross switches 215 are configured to receive a maximum of R vector values ​​and distribute these values ​​across N multiplier circuits 220, where N equals the maximum non-zero threshold of the sparse slice 200. Multiplier circuits 220A-220C, 220N are shown; however, it should be understood that in different implementations, sparse slice 200 may include more or fewer multiplier circuits.

[0071] The cross switch 215 can be implemented as any one or more circuits configured to receive input and pass it to one or more destinations, which themselves can be other circuits such as multiplier circuit 220. Multiplier circuit 220 can be implemented according to any of a variety of different techniques for performing hardware multiplication between two operands. The first operand for the multiplier circuit can be a vector value received by the cross switch 215. The second operand for the multiplier circuit can be a non-zero value from the sliced ​​input matrix 210. Each non-zero value is loaded into the corresponding registers 221A-221C, 221N of the corresponding multiplier circuits 220A-220C, 220N. As many multiplier circuits as there is a maximum non-zero threshold, providing available multiplier circuitry for each sliced ​​input matrix within the non-zero threshold. Each multiplier circuit multiplies the non-zero value stored in its corresponding register with the vector value received by the cross switch 215.

[0072] The sparse segmentation 200 may also include adder circuits 225. Each adder circuit 225A-225C, 225N is configured to receive input from a corresponding multiplier circuit. A segment marker is located in the middle of each adder circuit. Adder circuits 225A-225C, 225N and segment markers 226A-226C, 226N-1 are shown in... Figure 2 However, as with multiplier circuit 220, it should be understood that the number of adder circuits and segment markers can vary depending on the implementation.

[0073] Adder circuits can be implemented using any technique for hardware addition of two operands. The first operand from the adder circuit can be a product received from the multiplier circuit. For example, multiplier circuit 220A passes a product of vector values ​​and non-zero values ​​to adder circuit 225A. Segment markers are circuitry or other hardware components configured to gate inputs between adjacent adder circuits based on the gate input value.

[0074] Control value 230 may be received as input along with slice input matrix 210 and slice input vector 205, and used to configure one or more of cross switches 215, segment markers 226, and / or cross switches 235. Control value 230 may be a sequence of values, each corresponding to a corresponding non-zero value in slice input matrix 210. Control value 230 may include values ​​of a first type, such as 1, which may correspond to a non-zero value as the first non-zero value in the corresponding column of the slice input matrix. Control value 230 may include values ​​of a second type, such as 0, which may correspond to a zero value preceding one or more other non-zero values ​​in the same column of the slice input matrix. (See references herein.) Figures 3A to 3CAs described herein, control value 230 may also include one or more vectors, as described herein, that can be used to configure sparse sharding for processing the values ​​of sharded input matrix 210 and sharded input vector 205. Figure 4 The corresponding description in this paper clarifies example matrix multiplication using sparse partitioning.

[0075] In some implementations, the cross switches 215 and 235 can be implemented using different maximum dimension thresholds of the sparse partitions 200. For example, when dimensions R and C are equal or approximately equal, then the cross switch 235 can be implemented using any technique for rearranging the cross of a square or approximately square input, such as a Beneathian network. (network) was implemented.

[0076] Figures 3A to 3C The illustration shows a piecewise input matrix 300A, a non-zero value vector 300B in the piecewise input matrix, and a control value vector 300C corresponding to the piecewise input matrix 700A. Figure 3C Additional control vectors 305C and 310C are also shown.

[0077] Figure 3A This is a diagram of an example pieced input matrix 300A used for sparse pieced distribution. For clarity, non-zero values ​​are shown as shaded cells. For clarity, indices are provided along the columns and rows of pieced input matrix 300A and along vectors 300B and 300C. For example, in matrix 300A, the value at row 2, column 4 (2,4) is 1.

[0078] Figure 3B This is a diagram of a vector 300B representing the non-zero values ​​of the example sliced ​​input matrix 300A. The non-zero values ​​in vector 300B correspond to the order in which the non-zero values ​​in sliced ​​input matrix 300A appear when read from left to right, but the exact reading order may vary depending on the implementation, such as from right to left.

[0079] Figure 3C This is an illustration of vectors 300C, 305C, and 310C used for the control values ​​of the example sliced ​​input matrix 300A. In some examples, vectors 300C, 305C, and 310C may be part of the same vector in a predetermined order, and the sparse slices may be configured as described herein to use the predetermined order when receiving vectors of these control values ​​and configuring different components of the sparse slices according to vectors 300C, 305C, and 310C.

[0080] Vector 300C corresponds to the control value for the segment marker used to configure sparse fragmentation. The control value (a bit in this example) of 1 in vector 300C corresponds to the start of a new column in matrix 300A. The value at index 0 can be automatically set to 1 as the start of the control value in vector 300C. In some implementations, the initial control value can be omitted by the sparse fragmentation process and assumed to be constant. Hardware implementations can simplify their circuitry by utilizing the fact that this value is known to be 1. The value at index 1 in vector 300C is also set to 1 to correspond to the value at index 1 in vector 300B, which is the first non-zero value in the next column of matrix 300A. The value at index 2 in vector 300C is set to 0 because it corresponds to a non-zero value in vector 300B that is not the first non-zero value in the next column.

[0081] As another example, the bit at index 3 in vector 300C is set to 1 because the corresponding non-zero value (value 1 at index 3 in vector 300B) is the first non-zero value in the next column (specifically the second column) of matrix 300A. The sequence in vector 300C follows this described pattern until all non-zero values ​​in all columns have been represented.

[0082] Vector 305C corresponds to the control value used to configure the input crossbar switch for sparse slicing. Vector 305C specifies the "y" coordinate of the non-zero value within submatrix 300A for each non-zero value in the slicing input matrix 300A. In this example, the "y" dimension moves vertically up and down along the sparse matrix 300A, but in other examples, the "y" dimension can be defined differently, for example, horizontally. For example, the value "3" at element zero in vector 305C corresponds to the "y" coordinate of the non-zero value "1" in the first column of matrix 300A. As another example, the value "4" at element six in vector 305C corresponds to the value "1" at the bottom of the fourth column of matrix 300A.

[0083] Vector 310C corresponds to the control value used to configure how the sparsely sliced ​​adder circuitry generates and how the output cross switches are arranged to generate the sliced ​​output vector.

[0084] According to the mathematical definition of matrix multiplication, each value at position (x, y) in the sparse matrix is ​​multiplied by the value at position y in the input vector. The result of the multiplication is added to the output at position x. The input crossbar switch uses vector 305C to arrange the non-zero values ​​of the same column into adjacent multiplier circuits in the sparse slice. The output crossbar switch uses vector 310C to arrange the calculated sums in the correct order in the sliced ​​output vector, thus representing the product of the sliced ​​input matrix and sliced ​​input vector multiplied by the sparse slice.

[0085] return Figure 2The cross switch 215 can be configured to receive control value 230 and arrange each value of the sliced ​​input vector 205 to be received as input to one or more multiplier circuits matching the corresponding sliced ​​input matrix column when the sliced ​​input vector is multiplied by the sliced ​​input matrix.

[0086] The cross switch 235 can be configured to receive one or more sums from an adder circuit and rearrange the received sums to obtain the correct output slice vector corresponding to the multiplication of the input slice matrix with the input slice vector. Like the cross switch 215, any of a variety of different techniques can be applied to implement the cross switch 235 as one or more circuits.

[0087] As described herein, segment markers are configured to gate inputs between adjacent adder circuits depending on the gate input value of the segment marker. For example, when a segment marker receives a control value of 1, it prevents the output from the first adder circuit adjacent to that segment marker from being passed as input to the second adder circuit adjacent to that segment marker. When a segment marker receives a control value of 0, it passes the output from the first adder circuit to the second adder circuit (or, in some implementations, the reverse). This configuration of segment markers corresponds to adding only the sums corresponding to the non-zero values ​​in the same column, separated from the sums of the non-zero values ​​in different columns. (See references herein.) Figure 2 As described, the one or more of them can be passed to the cross switch 235 and rearranged to generate the sliced ​​output vector 240.

[0088] The cross switch 235 can be configured to determine whether to discard or receive an input and rearrange that input to match its correct position in the sliced ​​output vector 240. For example... Figure 2 As shown, each adder circuit can pass its output to the cross switch 235 (indicated by the arrows pointing to the cross switch 235). If the next segment marker after the adder circuit is not selected, the cross switch 235 can discard the output to that adder circuit because the running sum between adjacent adder circuits has not yet finished. If the next segment marker is selected (or if there is no segment marker, in the case of the last adder circuit 225N), the column running sum has been completed, and the cross switch 235 receives the running sum as input to be part of the output slice vector 240. By tracking which sums to ignore and which sums to include as part of the sliced ​​output vector 240, the cross switch 235 can precisely track column sums to generate output vectors of varying lengths up to the maximum dimension C.

[0089] Figure 4The illustration shows an example computation of receiving a sliced ​​input matrix 410, a sliced ​​input vector 405, and sparse slices 400 for control values ​​430 of the sliced ​​input matrix 410.

[0090] Consider example values ​​for piecewise input vector 405 and piecewise input matrix 410: (Vector 405) (Matrix 410).

[0091] In this example, the corresponding control vectors 430-432 for matrix 410 are: (Control vector 430) (Control vector 431) (Control vector 432) For ease of description, multiplier circuits 40A-40D are abbreviated as multiplier AD, segment markers 43A-43C are abbreviated as segment marker AC, and adder circuits 42A-42D are abbreviated as adder AD.

[0092] Based on the non-zero values ​​of matrix 410, multiplier A is loaded with the value 2, multiplier B with the value 1, multiplier C with the value 3, and multiplier D with the value 4. Note that for this example, the sparse partitioning consists of only four multipliers AD and four adders AD.

[0093] The cross switch 415 receives vector 405 with vector values ​​1, 3, and 2. Dashed lines 45A, solid lines 45B, and dashed lines 45C are shown to illustrate the path of data from the fragmented input vector 405 to the fragmented output vector 440. The cross switch 415 receives control vector 431, where each value corresponds to the "y" coordinate of the corresponding non-zero value in the fragmented input matrix 410. Values ​​in control vector 431 include 1, 2, 0, and 1. Note that the "y" coordinate values ​​range from 0 to 2 because the sparse input matrix 410 has a dimension of 3×3. Control vector 431 specifies to each multiplier in multiplier AD which value from the fragmented input vector 405 should be sent to which multiplier.

[0094] For example, the first value in control vector 431 is 1, corresponding to the "y" coordinate of the first non-zero value in piecewise input matrix 410. Because the "y" coordinate 1 is the second coordinate (after zero), the cross switch 415 routes the second value of piecewise input vector 405 to the first multiplier, here multiplier A. The second value of control vector 431 is 2, corresponding to the "y" coordinate of the second non-zero value in piecewise input matrix 410. The cross switch 405 can be configured to then route the third value of piecewise input vector 405 to multiplier B. As another example, the third value in control vector 431 is 0, corresponding to the next non-zero value after the "y" coordinate is 0. The cross switch 415 routes the first value of piecewise input vector 405 to multiplier C.

[0095] For the first column of matrix 410, the crossbar switch 415 routes the value 3 to multiplier A and the value 2 to multiplier B. For the second column of matrix 410, the crossbar switch 415 routes the value 1 to multiplier C. For the third and last columns of matrix 410, the crossbar switch 415 routes the value 3 to multiplier D. The product of multipliers A and D is as follows: 6 (3×2) for multiplier A, 2 (2×1) for multiplier B, 3 (1×3) for multiplier C, and 12 (3×4) for multiplier D.

[0096] Next, adder AD receives the product calculated by multiplier AD. Adder A receives the product from multiplier A, which is 6. The first control value (1) is discarded. Adder A has no previous adders, so it passes the sum to segment marker A. Segment marker A is not selected because the second value of control value 430 is zero. Adder B receives the current sum (6) from adder A and adds it to the product (2) of multiplier B. Segment marker B is selected, so the output (8) of adder B is passed to cross switch 420 (indicated by dashed line 45A). Adder C does not add the product (3) of multiplier C to anything because segment marker B is selected from the output of adder B. Segment marker C is selected, so the output (3) of adder C is passed to cross switch 420 (indicated by dashed line 45C). Finally, multiplier D receives the product (12) of multiplier D, and because it is the last adder in the sparse slice, it automatically passes its output (12) to the cross switch 420 (indicated by solid line 45B).

[0097] The cross switch 420 rearranges the received bits 8, 3, and 12 according to the correct order used for outputting the output slice vector 440. The cross switch 420 receives control vector 432 with values ​​0, 0, 2, and 1. (See references herein.) Figures 3A to 3CThe values ​​of the control vector used for the output cross switch 420 correspond to the "x" coordinate positions of non-zero values. Like the "y" coordinates, in this example, the values ​​range from 0 to 2. The first two values ​​of the control vector 432 are 0. The cross switch 420 therefore routes the received first sum to the first element of the output slice vector 440. The next value following the zero in the cross switch 432 is 2. The cross switch 432 is configured to route the second sum received from the adder AD to the second element of the output vector 440 (as shown by line 45C) and the third sum to the third element (as shown by line 45B). In some examples, the cross switch 420 is configured to skip consecutive repetitive control values ​​in the vector 432, such as the vector 432 with the first two zeros. In some examples, instead of skipping the continuous repetitive control value, the cross switch 420 is configured to perform an inclusive OR operation on the received input and output the result of the inclusive OR operation to the position of the output vector 440 corresponding to the continuous repetitive control value.

[0098] For example, line 46 shows a potential input source from adder A to output cross switch 420. Because segment marker 43A has a zero value, the output from adder A to cross switch 420 is suppressed, for example, masked or set to zero. The output of adder A is instead passed to adder B via segment marker 43A. In some examples, when output cross switch 420 receives control value 432, output cross switch 420 performs an implication OR operation on the received first sum (via line 46, with a value of 0) and the received second sum (via line 45A, with a value of 8 from adder B). Cross switch 420 can be configured to perform an implication OR operation to output a non-zero operand. Cross switch 420 will output 8 after performing an implication OR operation (e.g., 0 or 8) on the received sums and pass the result to a first position in output vector 440. In some examples, output cross switch 420 can receive individual outputs from at least some adders AD and compute the sum of these individual outputs.

[0099] The adder circuit can be implemented using any of a variety of different circuit configurations for adding consecutive ranges of numbers in a defined segment, each segment corresponding to a value in a corresponding column of the sliced ​​input matrix processed by the corresponding sparse slice. For example, the adder circuit of sparse slice 200 or 400 can be implemented as one or more sequential slice summation circuits (e.g., as shown by sparse slice 200 or 400) for performing sequential slice summation of the product by multiplying the non-zero values ​​in each column with the corresponding values ​​in the sliced ​​input vector. In some implementations, the adder circuit can be configured to perform the summation tree as a parallel slice summation circuit. Individual adder circuits can be configured to add corresponding inputs in parallel and, as described herein, pass their sum to the output crossbar switch and / or adjacent adder circuits based on a gate value marked by any intermediate segment.

[0100] Parallel segmented summation circuits can reduce the time delay of circuits compared to sequential segmented summation circuits, especially when the number of terms to be summed is large. Control values ​​used to gating segment markers allow for parallel segmented summation, at least because the range of values ​​to be summed together can be tracked based on the gate value of the segment markers, which gating inputs between adder circuits corresponding to values ​​in different segments and allowing inputs between adder circuits corresponding to values ​​to be added in the same segment.

[0101] In some implementations, the output crossover switch 420 is configured to receive the individual sums calculated by each adder and use control vector 432 to sum these sums and route them to the corresponding elements of the output slice vector 440. If two or more sums are routed to the same element, for example, if control vector 432 includes a copy of the same value as the first zero in vector 432, then the output crossover switch is configured to sum each received sum to be routed to the same element in the output slice vector 440, instead of as... Figure 4 As shown in line 45A, it receives a single and.

[0102] Example Method

[0103] Figure 5 This is a flowchart of an example process 500 for multiplying a submatrix of a sparse matrix on a sparse piece with a system input vector, according to various aspects of this disclosure. For ease of description, the submatrix of the sparse matrix is ​​referred to as the piecewise input matrix. Reference Figure 9 This paper describes an example process for partitioning an input sparse matrix into multiple submatrices. For example, sparse partitioning (such as...) Figure 2 The sparse fragmentation (200) execution process (500).

[0104] According to box 510, sparse fragmentation receives fragmented input matrices. The fragmented input matrix is ​​within a predetermined dimension threshold and has a count of non-zero values ​​equal to or less than a predetermined maximum non-zero threshold.

[0105] According to box 520, sparse fragmentation receives fragmented input vectors comprising multiple vector values. The fragmented input vector is a subvector of the system input vector, which can be generated as part of the input to the preprocessing system by the preprocessing engine, as referenced herein. Figure 1 and Figure 9 As described.

[0106] According to box 530, sparse partitioning generates one or more products of the corresponding vector value and the corresponding non-zero value. (See references in this document.) Figure 2 and Figure 4 As described, sparse slicing can be configured with control values ​​corresponding to the positions of non-zero values ​​in the slicing input matrix. Based on these control values, sparse slicing can be configured to route the input vector values ​​of subvectors to the corresponding adjacent multiplier circuits, storing them along with non-zero values ​​along the same columns of the input slicing matrix. Figure 8A An example procedure is shown for configuring sparse sharding using control values ​​of the sharded input matrix.

[0107] According to box 540, sparse partitioning generates one or more sums of the one or more products. (See references herein.) Figure 2 The described sparse slicing includes multiple adder circuits configured to receive inputs from corresponding multiplier circuits. The adder circuits are further configured to add these inputs along adjacent adder circuits until a segment marker is reached, which is set to a gate circuit adjacent to the segment marker. The sum of the adder circuit inputs up to the segment marker can be passed to a crossbar switch, for example... Figure 2 The 235 horizontal and vertical switch.

[0108] Sparse slicing generates a sliced ​​output vector from the sums of one or more sums as a product of multiplying the sliced ​​input vector by the sliced ​​input matrix. For example, by means of a second cross switch, sparse slicing can rearrange the received sums depending on how the operands of the product generated by the multiplier circuit are sorted by the first cross switch.

[0109] Figure 6 This is a flowchart of an example process 600 for generating a system output vector representing the product of the system input vector and the system input matrix from multiple sparse partitions. Sparse partitioned systems (such as...) Figure 1 The sparse matrix multiplication system 100 can execute process 600.

[0110] According to box 610 and for each group of sparse fragments along the column dimension of the array of sparse fragments, the fragment output vectors of each sparse fragment in the group are summed to generate the column output vector of that group. The following... Figure 7C This illustrates example grouping of sparse pieces based on the corresponding piecewise input matrix. The dimensions along which the groups form the sparse pieces can vary depending on the implementation. For example, depending on the direction in which the system input matrix and system input vector are fed into the array of sparse pieces, the groups can be along rows of the array rather than columns.

[0111] According to box 620, the system concatenates the column output vectors to generate the system output vector. The system output vector is the product of the system input matrix and the system input vector. In some implementations and as referenced herein... Figure 9 In more detail, the system can receive a system input matrix in which the columns of the matrix are permuted, for example, to distribute the occurrence of non-zero values ​​more evenly across submatrices assigned to sparse partitions. In those implementations, the system can be configured to rearrange the elements of the concatenated system output vector to reverse the original permutations. The system can receive reordered data as part of receiving the system input matrix, control values, and data reordered for partitions of the system input matrix.

[0112] Figures 7A to 7C The corresponding description clarifies an example multiplication between an example sparse matrix 700 and a vector 750. For illustrative purposes, the multiplication is described as being performed on a system with a 4×4 array having 16 sparse partitions.

[0113] Figure 7A This is an illustration of example system input matrix 700 and system input vector 750. In this illustration, system input matrix 700 is shown with integer values; however, it should be understood that the elements may be other values, such as floating-point values. Additionally, in... Figures 7A to 7C In the diagram, non-zero value elements of various matrices, including the system input matrix 700, are shown as shaded cells.

[0114] Figure 7B This is a diagram illustrating the partitioning of the system input matrix 700 and the system input vector 750. In this example, the system input matrix 700 is partitioned into sixteen submatrices 700A-700P, one submatrix for each sparse slice of the 4×4 array. The system input vector 750 is partitioned into four subvectors 750A-750D, one subvector for each column of the 4×4 array of the sparse slices.

[0115] An example mapping of subvectors and submatrices to sixteen sparse partitions (referred to as sparse partitioning AP) is shown in Table 1 below:

[0116] Table 1

[0117] Figure 7C This is a diagram of the partitioned system input matrix 700 and the system output vector 770, which represents the product of the system input matrix and the system input vector 750. Figure 7C This shows a partitioned matrix grouped along columns 705A-705D. (See references herein.) Figure 6 As described, the system can sum the sliced ​​output vectors of each column of a sparsely sliced ​​array and generate a column output vector. Figure 7C In the diagram, column output vectors 710A-710D correspond to columns 705A-705D. The system can concatenate column output vectors 710A-710D to generate a system output vector, which is the product of the system input matrix and the system input vector.

[0118] Figures 8A to 8B These are flowcharts of example processes 600A-600B for configuring sparse partitioning and performing matrix multiplication using one or more control values ​​of an input matrix, according to various aspects of this disclosure.

[0119] Figure 8A This is a flowchart of an example process 800A for configuring sparse partitioning using one or more control values ​​of a partitioned input matrix, according to various aspects of this disclosure. Sparse partitioning (such as...) Figure 2 The sparse partitioning (200) can execute the 800A process.

[0120] According to box 810, sparse slicing receives one or more control values ​​that specify the location of non-zero values ​​along each column of the slicing input matrix.

[0121] According to box 820, sparse piecewise operations load the non-zero values ​​of the piecewise input matrix into the registers of the multiplier circuit. (See references in this document.) Figure 2 As described, sparse slicing can implement a number of multiplier circuits equal to a predetermined maximum non-zero threshold of the system. Sparse slicing can load non-zero values ​​in the order they appear when the sliced ​​input matrix is ​​read along a predetermined read direction (e.g., from left to right).

[0122] According to box 830, the sparse slice loads one or more control values ​​into the crossbar switch of the sparse slice. The first crossbar switch (such as...) Figure 2 The cross switch 815 can be configured to receive control values ​​and arrange each value of the sliced ​​input vector as an input to one or more multiplier circuits, which are matched with a corresponding sliced ​​input matrix column, wherein the corresponding sliced ​​input matrix column is multiplied by the vector values ​​as part of a matrix-vector multiplication. A second cross switch (such as...) Figure 2The cross switch 235 can be configured to receive one or more sums from the adder circuit and rearrange the received sums to obtain a correct output slice vector, which corresponds to multiplying the input slice matrix by the input slice vector.

[0123] According to block 840, sparse slicing loads one or more control values ​​into one or more segment tags, which are configured to gate the input of the adder circuit based on the values ​​of these control values.

[0124] Figure 8B It is used for use according to Figure 8A The flowchart shows an example process of performing matrix-vector multiplication using sparse partitioning configured in the 800A.

[0125] According to box 850, sparse fragmentation receives and loads the non-zero values ​​of the input fragmentation matrix.

[0126] According to box 860, the sparse slice receives the vector values ​​of the slice input vector along with the non-zero values ​​along the same column of the slice input matrix and sends them to the multiplier circuit.

[0127] According to box 870, sparse fragmentation generates one or more segmented sums from adjacent adder circuits that have never been gated by segment tags. (See references in this document.) Figure 2 and Figure 8A As described, sparse slicing can aggregate sums between adder circuits not selected by segment markers, and uses control values ​​to configure segment markers to select adder circuits representing computations from different columns of the slicing input matrix. When a sparse slice reaches a segment marker with active gate bits, the sparse slice passes the segmented sums to a second crossbar switch, which is configured to rearrange the segmented sums and generate a slicing output vector.

[0128] According to box 880, sparse partitioning generates a partitioned output vector from the one or more segments. (See references herein.) Figure 2 and Figure 8A As described, control values ​​can be used to configure the crossover switch to receive one or more segmented sums from a sparsely segmented adder circuit. The crossover switch can be further configured to rearrange the segmented sums to generate a correctly segmented output vector that represents the product of multiplying the segmented input matrix by the segmented input vector.

[0129] Figure 9 This is a flowchart of an example process 900 for generating a submatrix from a system input matrix, according to various aspects of this disclosure. One or more processors in one or more locations may execute process 900. For example, a preprocessing engine (such as...) Figure 1 The preprocessing engine (150) can perform 900 processes.

[0130] According to box 910, the preprocessing engine receives the system input matrix. The system input matrix can be, for example, as shown in box 910. Figure 7A The system input matrix 700 is shown as the matrix.

[0131] According to box 920, the preprocessing engine partitions the system input matrix into multiple candidate submatrices. As part of the partitioning, the preprocessing engine may receive parameters specifying a predetermined dimension threshold, and one or more parameter values ​​indicating the number of sparse partitions implemented by the system receiving the submatrices. For example, the preprocessing engine may be configured to generate 16 candidate submatrices within an 8-row by 8-column dimension threshold (to obtain a 4×4 array of sparse partitions). In some examples, the preprocessing engine may receive updated parameter values, for example, to preprocess the input across different system preprocessing thresholds and / or configurations with different sparse partitions.

[0132] In some implementations, before the preprocessing engine partitions the system input matrix, it permutes the columns of the system input matrix to distribute non-zero values ​​evenly across the candidate submatrices. For example, if non-zero values ​​occur more frequently on one side of the input matrix than on the other side, the preprocessing engine can be configured to change the order of the columns of the input matrix, making the occurrence of non-zero values ​​more dispersed and thus more evenly distributed across the sparse partitions after partitioning.

[0133] If the preprocessing engine performs this sorting, it passes the sorted data as additional input to the system, for example, as part of partitioned data sent to a system with a sparsely sharded array. The system can be configured to reorder the elements of the system output vector according to this sorting so that the output vector matches the output of multiplying the system input matrix and the system input vector before the columns of the system input matrix are permuted.

[0134] Permuting the columns of the system input matrix can improve the overall rate of system processing of the input matrix by sparsely partitioned systems. For example, by permuting the columns, each sparse partition can be used more efficiently, especially when, in some examples, some sparse partitions can receive partitioned input matrices with only zero values, while other partitions can receive partitioned input matrix values ​​with only non-zero values, or can receive non-zero values ​​up to a non-zero threshold.

[0135] According to box 930, the preprocessing engine determines whether there exists a candidate submatrix with a non-zero value count greater than a predetermined non-zero threshold. According to box 940, if the preprocessing engine determines that there exists a candidate submatrix with a non-zero value count greater than the predetermined non-zero threshold, the preprocessing engine repartitions the submatrix along the same rows or columns as the candidate submatrix.

[0136] like Figure 7BAs shown, submatrices can be organized along columns and rows based on the position of their values ​​within the input matrix. For example, if the preprocessing engine determines that submatrix 700J contains a count of non-zero values ​​above a non-zero threshold, the preprocessing engine can repartition the submatrix along its rows (including submatrices 700I, 700K, and 700L) and / or along its columns (including submatrices 700B, 700F, and 700N). When performing repartitioning, the preprocessing engine uses a predetermined dimension threshold and performs the repartitioning such that the number of candidate submatrices remains unchanged. For example, the preprocessing engine can split the candidate submatrices and redistribute the rows / columns of the submatrices along the determined rows / columns of the candidate submatrices.

[0137] According to boxes 930 and 940, after repartitioning the submatrix, the preprocessing engine again determines whether there are candidate submatrixes with a non-zero count greater than the non-zero threshold. The preprocessing engine can repeat the determination and repartitioning according to boxes 930 and 940 until it is determined that there are no candidate submatrixes with a non-zero count exceeding the maximum non-zero threshold, and proceeds to box 950.

[0138] As shown in box 950, the preprocessing engine partitions the system input vector. Depending on the bus arrangement of the array feeding the sparse slices, each subvector of the system input vector is input to each row or column of the sparse slice. The preprocessing engine partitions the vector so that the vector dimension can be multiplied by a matrix in the receiving sparse slice, for example, a matrix with a dimension that is mathematically valid for matrix multiplication.

[0139] For example, Figures 7B to 7C As shown in Table 1, vector 750 is partitioned into subvectors 750A-750D, and each of these subvectors is passed as input to one or more sparse partitions. Similarly, in... Figure 7B In this matrix, each dimension of each subvector 750A-750D has the correct dimension for multiplication with the corresponding submatrix 700A-700P. For example, subvector 750B is 1×2 (rows × columns), and each of the submatrixes 700E-700H has 2 rows, thus allowing efficient matrix multiplication between subvector 750B and submatrixes 700E-700H.

[0140] As shown in box 960, the preprocessing engine generates control values ​​for each candidate submatrix. (See reference...) Figures 3A to 3C As described, the preprocessing engine can generate a vector of control values ​​indicating the initial non-zero values ​​of each column of a matrix. The preprocessing engine repeats this generation process for each candidate submatrix, thereby generating corresponding control values ​​for each submatrix.

[0141] According to box 970, the preprocessing engine outputs control values ​​and candidate submatrices. The preprocessing engine can, for example,... Figure 1 The system 100 shown outputs control values ​​and data for the partitions of the specified system input matrix.

[0142] Example computing environment

[0143] Figure 10 This is a block diagram of an example computing environment implementing a sparse matrix multiplication system 100 and a preprocessing engine 150 according to aspects of this disclosure. The preprocessing engine 150 may be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 1015. User computing device 1012 and server computing device 1015 may be communicatively coupled to one or more storage devices 1030 via a network 1060. Storage device 1030 may be a combination of volatile and non-volatile memory and may be located in the same or different physical locations as computing devices 1012 and 1015. For example, storage device 1030 may include any type of non-transitory computer-readable medium capable of storing information, such as hard disk drives, solid-state drives, magnetic tape drives, optical storage devices, memory cards, ROM, RAM, DVDs, CD-ROMs, writable and read-only memories.

[0144] Server computing device 1015 may include one or more processors 1013 and memory 1014. Memory 1014 may store information accessible by processor 1013, including instructions 1021 executable by processor 1013. Memory 1014 may also include data 1023 retrievable, manipulated, or stored by processor 1013. Memory 1014 may be a non-transitory computer-readable medium capable of storing information accessible by processor 1013, such as volatile and non-volatile memory. Processor 1013 may include one or more central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FGPAs), and / or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0145] The server computing device 1015 can implement the sparse matrix multiplication system 100 in hardware, for example, as a system-on-a-chip. The system 100 can be implemented as part of a physical chip inserted into or installed in the server computing device 1015. The system 100 is configured to communicate with other components of the server computing device 1015.

[0146] Instructions 1021 may include one or more instructions that, when executed by processor 1013, cause the processor to perform actions defined by those instructions. Instructions 1021 may be stored in object code format for direct processing by processor 1013, or in other formats, including a collection of interpretable scripts or standalone source code modules that are interpreted on demand or pre-compiled. Instructions 1021 may include instructions for implementing sparse fragmentation 400 consistent with aspects of this disclosure. The preprocessor engine 105 may be executed using processor 1013 and / or other processors located remotely from server computing device 1015.

[0147] Processor 1013 can retrieve, store, or modify data 1023 according to instruction 1021. Data 1023 can be stored in computer registers, as a table with multiple distinct fields and records in a relational or non-relational database, or as a JSON, YAML, proto, or XML document. Data 1023 can also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, data 1023 may include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary code, pointers, references to data stored in other memory, including other network locations, or information used by functions to calculate relevant data.

[0148] User computing device 1012 may also be configured similarly to server computing device 1015, and may include one or more processors 1016, memory 1017, instructions 1018, and data 1019. User computing device 1012 may also include user output 1026 and user input 1024. User input 1024 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, mouse, mechanical actuator, soft actuator, touchscreen, microphone, and sensors.

[0149] Server computing device 1015 can be configured to transmit data to user computing device 1012, and user computing device 1012 can be configured to display at least a portion of the received data on a display implemented as part of user output 1026. User output 1026 can also be used to display the interface between user computing device 1012 and server computing device 1015. User output 1026 may alternatively or additionally include one or more speakers, transducers or other audio outputs, haptic interfaces, or other haptic feedback that provides non-visual and non-auditory information to the platform user of user computing device 1012.

[0150] Although Figure 10Processors 1013, 1016 and memories 1014, 1017 are illustrated as being within computing devices 1015, 1012. However, the components including processors 1013, 1016 and memories 1014, 1017 described herein may include multiple processors and memories that may be in different physical locations and operate outside the same computing device. For example, some of instructions 1021, 1018 and data 1023, 1019 may be stored on a removable SD card, while others may be stored within a read-only computer chip. Some or all of the instructions and data may be stored physically away from processors 1013, 1016 but still accessible by them. Similarly, processors 1013, 1016 may include a collection of processors capable of performing concurrent and / or sequential operations. Computing devices 1015, 1012 may each include one or more internal clocks that provide timing information for time measurement of operations and programs run by computing devices 1015, 1012.

[0151] Server computing device 1015 can be configured to receive requests for processing data from user computing device 1012. For example, environment 1000 can be part of a computing platform configured to provide various services to users through various user interfaces and / or APIs that expose platform services. As part of executing services, server computing device 1015 can use system 100 to process incoming data. For example, if the service is training a machine learning model, server computing device 1015 can be configured to use system 100 to perform multiplication operations as part of training the machine learning model.

[0152] Devices 1012 and 1015 can communicate directly and indirectly via network 1060. Devices 1015 and 1012 can be configured to accept listening sockets for initiating connections to send and receive information. Network 1060 itself can include various configurations and protocols, including the Internet, World Wide Web, Intranet, Virtual Private Network, Wide Area Network, Local Area Network, and private networks using one or more proprietary communication protocols. Network 1060 can support a variety of short-range and long-range connections. Short-range and long-range connections can occur at different bandwidths, such as 2.402 GHz to 2.480 GHz (typically associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (typically associated with the Wi-Fi® communication protocol); or with various communication standards, such as the LTE® standard for wireless broadband communication. Additionally or alternatively, network 1060 can also support wired connections between devices 1012 and 1015, including via various types of Ethernet connections.

[0153] Although Figure 10The illustration shows a single server computing device 1015 and a user computing device 1012. It should be understood that aspects of this disclosure can be implemented according to various configurations and numbers of computing devices, including in examples for sequential or parallel processing, or through a distributed network of multiple devices. In some implementations, aspects of this disclosure can be executed on a single device or any combination thereof. Furthermore, although the preprocessing engine and sparse matrix multiplication system 100 shown are implemented on the same server computing device 1015, in some implementations, the preprocessing engine 150 is implemented on one or more server computing devices and / or user computing devices 1012, separate from the server computing device 1015.

[0154] The aspects of this disclosure may be implemented in digital circuits, computer-readable storage media, as one or more computer programs, or a combination of the foregoing. The computer-readable storage medium may be non-transitory, for example, as one or more instructions executable by a computing device and stored on a tangible storage device.

[0155] In this specification, the phrase "configured to" is used in various contexts relating to a computer system, hardware, or computer program, engine, or module. When a system is referred to as being configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed on the system that, when running, causes the system to perform said one or more operations. When some hardware is referred to as being configured to perform one or more operations, this means that the hardware includes one or more circuits that, during operation, receive input and, based on said input, produce output corresponding to said one or more operations. When a computer program, engine, or module is referred to as being configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause said one or more computers to perform said one or more operations.

[0156] Although the operations shown in the accompanying drawings and recounted in the claims are illustrated in a specific order, it should be understood that these operations may be performed in a different order than shown, and some operations may be omitted, performed more than once, and / or performed in parallel and / or simultaneously with other operations. Furthermore, the separation of different system components configured to perform different operations should not be construed as requiring the separation of components. The described components, modules, programs, and engines may be integrated as a single system or as part of multiple systems.

[0157] Unless otherwise stated, the above alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the examples should be understood illustratively rather than as limiting the subject matter defined by the claims. Furthermore, the provision of the examples described herein and the use of terms such as "such as," "comprising," etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, these examples are intended to illustrate only one of many possible implementations. Additionally, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A system comprising: Sparse partitioning, wherein the sparse partitioning includes multiple multiplier circuits, wherein the sparse partitioning is configured as follows: A sliced ​​input matrix is ​​received, the sliced ​​input matrix including a number of non-zero values ​​equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits in the plurality of multiplier circuits, wherein the multiplier circuits receive the corresponding non-zero values ​​of the sliced ​​input matrix. Receive a fragmented input vector, wherein the fragmented input vector includes multiple vector values; One or more products are generated by multiplying vector values ​​generated by the plurality of multiplier circuits with the corresponding non-zero values ​​of the piecewise input matrix; and Using the one or more products, a piecewise output vector is generated, which is the product of applying the piecewise input vector to the piecewise input matrix.

2. The system of claim 1, wherein the piecewise input matrix has a dimension equal to or less than a predetermined dimension threshold corresponding to the maximum matrix input size.

3. The system according to claim 1, wherein, The sparse partition is one of the plurality of sparse partitions, which are configured as follows: Receive multiple sliced ​​input matrices that serve as submatrices of the system input matrix; Receives multiple sliced ​​input vectors as subvectors of the system input vector; and Generate a system output vector, which represents the product of applying the system input vector to the system input matrix.

4. The system according to claim 3, wherein, In generating the system output vector, the plurality of sparse slices are further configured to concatenate the corresponding slice output vectors to generate the system output vector.

5. The system according to claim 1, wherein, The multiplier circuit is coupled to a register, which includes the corresponding non-zero value.

6. The system according to claim 1, wherein, The predetermined non-zero threshold is the maximum non-zero threshold.

7. The system according to claim 1, wherein, The sparse sharding further includes a cross-sectional switching circuit, which is configured to: Receive the plurality of vector values ​​of the sliced ​​input vector; and Send a vector value from the plurality of vector values ​​according to one or more control values.

8. The system according to claim 1, wherein, The sparse slicing is further configured to load non-zero values ​​of the same column in the slicing input matrix into the registers of adjacent multiplier circuits in the plurality of multiplier circuits.

9. The system according to claim 8, wherein, The sparse sharding is further configured to receive one or more control values ​​that specify the positions of non-zero values ​​along the columns of the sharding input matrix.

10. The system according to claim 1, wherein, The sparse partitioning further includes a plurality of adder circuits configured to generate one or more sums of the one or more products of the vector values.

11. The system according to claim 10, wherein, The multiple adder circuits form a parallel segmented summation circuit.

12. The system according to claim 10, wherein, The sparse sharding further includes a cross-sectional switching circuit, which is configured to: Receive one or more of the above and, The one or more sums are arranged according to one or more control values ​​to generate the fragmented output vector.

13. The system according to claim 12, wherein, The cross-sectional switching circuit forms a Bene )network.

14. A method comprising: A sparsely distributed input matrix is ​​received by a plurality of multiplier circuits, the sparsely distributed input matrix including a plurality of non-zero values ​​equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits in the plurality of multiplier circuits, wherein the multiplier circuits receive the corresponding non-zero values ​​of the sparsely distributed input matrix. The sparse partition receives a partitioned input vector, which includes multiple vector values. One or more products are generated by multiplying the vector values ​​generated by the plurality of multiplier circuits with the corresponding non-zero values ​​of the piecewise input matrix; as well as The sparse partitions are used with one or more products to generate a partitioned output vector, which is the product of applying the partitioned input vector to the partitioned input matrix.

15. The method according to claim 14, wherein, The piecewise input matrix has a dimension equal to or less than a predetermined dimension threshold corresponding to the maximum matrix input size.

16. The method of claim 14, further comprising: The plurality of sparse partitions, one of which is the sparse partition, receive a plurality of partitioned input matrices as submatrices of the system input matrix; The multiple sparse slices receive multiple slice input vectors as subvectors of the system input vector; as well as The system output vector is generated from the plurality of sparse slices, and the system output vector represents the product of applying the system input vector to the system input vector.

17. The method according to claim 16, wherein, Generating the system output vector further includes concatenating the corresponding sparse output vectors of the plurality of sparse fragments to generate the system output vector.

18. The method according to claim 14, wherein, The multiplier circuit is coupled to a register, which includes the corresponding non-zero value.

19. The method of claim 14, wherein, The predetermined non-zero threshold is the maximum non-zero threshold.

20. One or more non-transitory computer-readable storage media storing instructions that, when executed by a system comprising sparse fragments including a plurality of multiplier circuits, cause the system to perform operations including: A sliced ​​input matrix is ​​received, the sliced ​​input matrix including a number of non-zero values ​​equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits in the plurality of multiplier circuits, wherein the multiplier circuits receive the corresponding non-zero values ​​of the sliced ​​input matrix. Receive a fragmented input vector, wherein the fragmented input vector includes multiple vector values; One or more products are generated by multiplying the vector values ​​generated by the plurality of multiplier circuits with the corresponding non-zero values ​​of the piecewise input matrix; as well as Using the one or more products, a piecewise output vector is generated, which is the product of applying the piecewise input vector to the piecewise input matrix.

Images