Sparse matrix multiplication in hardware

Abstract

The invention relates to sparse matrix multiplication in hardware. Methods, systems, and apparatus, including computer readable storage media, are provided for sparse matrix multiplication. A system for matrix multiplication includes an array of sparse slices. Each sparse slice may be configured to receive an input sub-matrix and an input sub-vector, wherein the input sub-matrix has a number of non-zero values equal to or less than a predetermined maximum non-zero threshold. The sparse fragmentation may compute one or more products of vector values multiplied by respective non-zero values of the input sub-matrix by a plurality of multiplier circuits. The sparse sharding may generate a sharding output vector that is an output to the sparse sharding and uses the one or more products, the sharding output vector being a product that applies the sharding input vector to the sharding input matrix.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202610149851.5 (22) Application Date 2021.12.30 (30) Priority Data 17 / 329,259 2021.05.25 US (62) Divisional Application Data 202111665133.7 2021.12.30 (71) Applicant Google LLC Address USA (72) Inventor Rainer Alvin Pope (74) Patent Agency Zhongyuan Xinda Intellectual Property Agency Co., Ltd. 11219 Patent Attorney Park Jindan Zhou Yarong (51) Int.Cl. G06F 17 / 16 (2006.01) G06N 3 / 0464 (2023.01) G06N 3 / 08 (2023.01) (54) Title of Invention: Sparse Matrix Multiplication in Hardware (57) Abstract This disclosure relates to sparse matrix multiplication in hardware. Methods, systems, and apparatuses for sparse matrix multiplication are provided, including a computer-readable storage medium. The system for matrix multiplication includes an array of sparse slices. Each sparse slice can be configured to receive an input submatrix and an input subvector, wherein the input submatrix has a plurality of nonzero values ​​equal to or less than a predetermined maximum nonzero threshold. The sparse slice can compute one or more products by multiplying the vector values ​​with the corresponding nonzero values ​​of the input submatrix using a plurality of multiplier circuits. The sparse slice can generate a sliced ​​output vector as the output of the sparse slice and using the one or more products, the sliced ​​output vector being a product of applying the sliced ​​input vector to the sliced ​​input matrix. Claims 2 pages, Description 19 pages, Drawings 14 pages, CN 122153225 A 2026.06.05 CN 1 22 15 32 25 A 1. A system comprising: a sparse partition, the sparse partition including a plurality of multiplier circuits, wherein the sparse partition is configured to: receive a partitioned input matrix, the partitioned input matrix including a plurality of non-zero values ​​equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits in the plurality of multiplier circuits, wherein the multiplier circuits receive corresponding non-zero values ​​of the partitioned input matrix; receive a partitioned input vector, the partitioned input vector including a plurality of vector values; generate one or more products by the plurality of multiplier circuits multiplying the vector values ​​by the corresponding non-zero values ​​of the partitioned input matrix; and generate a partitioned output vector using the one or more products, the partitioned output vector being a product of applying the partitioned input vector to the partitioned input matrix. 2. The system of claim 1, wherein the piecewise input matrix has a dimension equal to or less than a predetermined dimension threshold corresponding to the maximum matrix input size.3. The system of claim 1, wherein the sparse slice is one of the plurality of sparse slices, the plurality of sparse slices being configured to: receive a plurality of sliced ​​input matrices as submatrices of a system input matrix; receive a plurality of sliced ​​input vectors as subvectors of a system input vector; and generate a system output vector, the system output vector representing a product of applying the system input vectors to the system input matrices. 4. The system of claim 3, wherein in generating the system output vector, the plurality of sparse slices are further configured to concatenate corresponding sliced ​​output vectors to generate the system output vector. 5. The system of claim 1, wherein the multiplier circuit is coupled to a register, the register including the corresponding non-zero value. 6. The system of claim 1, wherein the predetermined non-zero threshold is a maximum non-zero threshold. 7. The system of claim 1, wherein the sparse slice further includes a cross-switch circuit, the cross-switch circuit being configured to: receive the plurality of vector values ​​of the sliced ​​input vectors; and send a vector value from the plurality of vector values ​​according to one or more control values. 8. The system of claim 1, wherein the sparse slice is further configured to load non-zero values ​​of the same column in the sliced ​​input matrix into registers of adjacent multiplier circuits in the plurality of multiplier circuits. 9. The system of claim 8, wherein the sparse slice is further configured to receive one or more control values ​​specifying the position of a non-zero value along a column of the sliced ​​input matrix. 10. The system of claim 1, wherein the sparse slice further includes a plurality of adder circuits configured to generate one or more sums of the one or more products of vector values. 11. The system of claim 10, wherein the plurality of adder circuits form a parallel piecewise summation circuit. 12. The system of claim 10, wherein the sparse slice further includes a cross-switching circuit configured to: receive the one or more sums and, according to one or more control values, arrange the one or more sums to generate the sliced ​​output vector. 13. The system of claim 12, wherein the cross-switching circuit forms a Bene network. Claims 1 / 2 Page 2 CN 122153225 A 14. A method comprising: receiving a sparse input matrix by a sparse shard comprising a plurality of multiplier circuits, the sharded input matrix comprising a plurality of nonzero values ​​equal to or less than a predetermined nonzero threshold, the nonzero threshold corresponding to the number of multiplier circuits in the plurality of multiplier circuits, wherein the multiplier circuits receive the corresponding nonzero values ​​of the sharded input matrix;The method comprises: receiving a piecewise input vector from the sparse piecewise segment, the piecewise input vector comprising a plurality of vector values; generating one or more products by the plurality of multiplier circuits of the vector values ​​multiplied by corresponding non-zero values ​​of the piecewise input matrix; and generating a piecewise output vector from the sparse piecewise segment using the one or more products, the piecewise output vector being a product of applying the piecewise input vector to the piecewise input matrix. 15. The method of claim 14, wherein the piecewise input matrix has a dimension equal to or less than a predetermined dimension threshold corresponding to a maximum matrix input size. 16. The method of claim 14, further comprising: receiving a plurality of piecewise input matrices as submatrices of a system input matrix from the plurality of sparse pieces, the plurality of sparse pieces receiving a plurality of piecewise input vectors as subvectors of a system input vector; and generating a system output vector from the plurality of sparse pieces, the system output vector representing a product of applying the system input vector to the system input vector. 17. The method of claim 16, wherein generating the system output vector further comprises concatenating the corresponding slice output vectors by the plurality of sparse slices to generate the system output vector. 18. The method of claim 14, wherein the multiplier circuitry is coupled to a register, the register including the corresponding non-zero value. 19. The method of claim 14, wherein the predetermined non-zero threshold is a maximum non-zero threshold. 20. One or more non-transitory computer-readable storage media storing instructions that, when executed by a system comprising sparse fragments including a plurality of multiplier circuits, cause the system to perform operations comprising: receiving a fragmented input matrix comprising a plurality of non-zero values ​​equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits in the plurality of multiplier circuits, wherein the multiplier circuits receive corresponding non-zero values ​​of the fragmented input matrix; receiving a fragmented input vector comprising a plurality of vector values; generating one or more products by the plurality of multiplier circuits multiplying the vector values ​​by corresponding non-zero values ​​of the fragmented input matrix; and using the one or more products to generate a fragmented output vector, the fragmented output vector being a product of applying the fragmented input vector to the fragmented input matrix. Claims 2 / 2 Page 3 CN 122153225 A Sparse Matrix Multiplication in Hardware

[0001] Divisional Application Description

[0002] This application is a divisional application of patent application No. 202111665133.7, filed on December 30, 2021, entitled "Sparse Matrix Multiplication in Hardware". Technical Field

[0003] This disclosure relates to sparse matrix multiplication in hardware.Background Art

[0004] A sparse matrix is ​​a matrix in which the ratio of zero values ​​to non-zero values ​​is high. Different sparse matrices can have different degrees of sparsity based on the ratio of zero values ​​to non-zero values. A matrix with a high ratio of zero values ​​to non-zero values ​​is said to have higher sparsity compared to a matrix with a lower ratio.

[0005] A neural network is a machine learning model that includes one or more layers of nonlinear operations to predict outputs in response to received inputs. In addition to input and output layers, some neural networks also include one or more hidden layers. The output of each hidden layer can be input to another hidden layer or the output layer of the neural network. Each layer of a neural network can generate a corresponding output from the received input based on the values ​​of one or more model parameters used for that layer. Model parameters can be weights or biases determined by a training algorithm to enable the neural network to generate accurate outputs. The model parameter values ​​used for layers of a neural network can be represented as elements of a matrix or tensor. Summary of the Invention

[0006] Aspects of this disclosure relate to sparse matrix-dense vector multiplication in hardware.

[0007] One aspect of this disclosure provides a system comprising: a sparse partition, the sparse partition including a plurality of multiplier circuits, wherein the sparse partition is configured to: receive a partitioned input matrix, the partitioned input matrix including a plurality of non-zero values ​​equal to or less than a predetermined maximum non-zero threshold; receive a partitioned input vector, the partitioned input vector including a plurality of vector values; for each of the multiplier circuits, receive a corresponding non-zero value of the partitioned input matrix; generate one or more products by the plurality of multiplier circuits multiplying the vector values ​​by the corresponding non-zero values ​​of the partitioned input matrix; and generate a partitioned output vector as the output of the sparse partition and using the one or more products, the partitioned output vector being a product of applying the partitioned input vector to the partitioned input matrix.

[0008] Another aspect of this disclosure provides one or more non-transitory computer-readable storage media storing instructions that, when executed by a system comprising a plurality of sparse slices, cause the system to perform operations including: receiving a sliced ​​input matrix and a sliced ​​input vector by a sparse slice comprising a plurality of multiplier circuits, wherein the sliced ​​input matrix comprises a plurality of non-zero values ​​equal to or less than a predetermined maximum non-zero threshold, and the sliced ​​input vector comprises a plurality of vector values; receiving a corresponding non-zero value of the sliced ​​input matrix for each of the multiplier circuits; generating one or more products by the plurality of multiplier circuits of the sparse slice, wherein the corresponding vector value is multiplied by the corresponding non-zero value; and generating a sliced ​​output vector as an output to the sparse slice and using the one or more products, the sliced ​​output vector being a product of applying the sliced ​​input vector to the sliced ​​input matrix. (Specification 1 / 19 pages 4 CN)122153225 A

[0009] Another aspect of this disclosure provides a method comprising: receiving a piecewise input matrix and a piecewise input vector by a sparse piecewise segment comprising a plurality of multiplier circuits, wherein the piecewise input matrix comprises a plurality of nonzero values ​​equal to or less than a predetermined maximum nonzero threshold, and the piecewise input vector comprises a plurality of vector values; receiving a corresponding nonzero value of the piecewise input matrix for each of the multiplier circuits in the multiplier circuits; generating one or more products by multiplying the corresponding vector value with the corresponding nonzero value of the piecewise input matrix by the plurality of multiplier circuits of the sparse piecewise segment; and generating a piecewise output vector as the output of the sparse piecewise segment and using the one or more products, the piecewise output vector being a product of applying the piecewise input vector to the piecewise input matrix.

[0010] The foregoing and other aspects may optionally include one or more of the following features individually or in combination. One implementation may include a combination of all of the following features.

[0011] The length of the piecewise output vector is greater than one.

[0012] The sparse slice is one of a plurality of sparse slices, the sparse slice being configured to: receive a plurality of sliced ​​input matrices as submatrices of the system input matrix; receive a plurality of sliced ​​input vectors as subvectors of the system input vector; and generate a system output vector from the plurality of sparse slices, the system output vector representing a product of applying the system input vector to the system input matrix.

[0013] The plurality of sparse slices are arranged as a systolic array, the systolic array comprising one or more groups of sparse slices along the column dimension of the systolic array; and wherein, in order to generate the system output vector, the one or more processors are further configured to: for each group along the column dimension of the systolic array, add each sparse slice in the group together with a corresponding sliced ​​output vector to generate a corresponding column output vector; and concatenate the corresponding column output vectors of each group to generate the system output vector.

[0014] Each multiplier circuit is coupled to a corresponding register, the corresponding register including the corresponding nonzero value from the corresponding sliced ​​input matrix of the sparse slice.

[0015] The number of multiplier circuits in the plurality of multiplier circuits is equal to the predetermined maximum non-zero threshold.

[0016] The sparse slicing further includes a cross-switching circuit, and wherein the sparse slicing is further configured to: receive the plurality of vector values ​​of the slicing input vector by the cross-switching circuit; and send a vector value from the plurality of vector values ​​as input to each of the plurality of multiplier circuits and by the cross-switching circuit.

[0017] The sparse slicing is further configured to: load non-zero values ​​of the same column in the slicing input matrix into the registers of adjacent multiplier circuits in the plurality of multiplier circuits.

[0018] The sparse sharding is further configured to receive one or more control values, the one or more control values ​​specifying at least the position of a non-zero value along each column of the sharding input matrix; and wherein the cross-switching circuitry for the sparse sharding is further configured to: receive the one or more control values; and, based on the one or more control values, send a vector value to an adjacent multiplier circuitry to be multiplied by a non-zero value along the same column of the sharding input matrix.

[0019] The sparse slicing further includes a plurality of adder circuits, wherein the sparse slicing further includes one or more segment markers, wherein each segment marker is configured to select the input of a corresponding adder circuit among the plurality of adder circuits based on the value of a corresponding control value loaded in the segment marker; and wherein the sparse slicing is further configured to: load at least a portion of the one or more control values ​​in the one or more segment markers, wherein the adder circuit for non-zero values ​​in a first column of the slicing input matrix is ​​selected by receiving an input from an adjacent adder circuit, the input including non-zero values ​​in a second column of the slicing input matrix that is different from the first column; and generate one or more sums of the one or more products by the plurality of adder circuits, wherein each of the one or more sums is a corresponding piecewise sum of multiplying one or more non-zero values ​​of a column of the slicing input matrix with one or more corresponding values ​​of the slicing input vector.

[0020] The plurality of adder circuits form a parallel piecewise summation circuit, and wherein each of the one or more piecewise sums is a sum to the output of an adjacent adder circuit not selected by a segment marker.

[0021] The cross-switching circuit is a first cross-switching circuit; and wherein the sparse sharding further includes a second cross-switching circuit configured to: receive the one or more piecewise sums, and arrange the one or more piecewise sums according to the one or more control values ​​to generate a corresponding sharded output vector of the sparse shard.

[0022] The second cross-switching circuit may form a Bene network, and wherein the sharded input matrix is ​​a square matrix.

[0023] Another aspect of this disclosure provides a system comprising one or more processors; and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including: receiving an input matrix comprising zero and non-zero values; and partitioning the input matrix into a plurality of submatrices, wherein the number of non-zero values ​​in each submatrix is ​​equal to or less than a predetermined maximum non-zero threshold, and wherein the dimension of each submatrix is ​​equal to or less than a predetermined dimension threshold.

[0024] The operation may further include, for each submatrix, generating one or more corresponding control values, the one or more corresponding control values ​​specifying the positions of non-zero values ​​along each column of the submatrix.

[0025] The operation may further include: sending each submatrix and the one or more corresponding control values ​​for each submatrix to a plurality of sparse slices, the plurality of sparse slices being configured to process each submatrix and the corresponding control values ​​for the submatrix. Each of the plurality of sparse slices is configured to: receive a submatrix, the one or more corresponding control values ​​for the submatrix, and at least a portion of an input vector, and generate a corresponding output slice vector, the corresponding output slice vector representing the product of a portion of the submatrix and the input vector. The system of claim 21, wherein partitioning the input matrix includes partitioning the matrix into a plurality of submatrixes equal to the number of sparse slices in the plurality of sparse slices.

[0026] Other aspects of this disclosure include corresponding systems, apparatuses, and computer programs stored on one or more non-transitory computer-readable storage media.

[0027] FIG1 is a block diagram of an example system including an array of sparse slices according to aspects of the present disclosure.

[0028] FIG2 is a block diagram of an example sparse slice according to aspects of the present disclosure.

[0029] FIG3A is a diagram of an example slice input matrix for sparse slices.

[0030] FIG3B is a diagram of a vector of non-zero values ​​of the example slice input matrix.

[0031] FIG3C is a diagram of a vector of control values ​​for the example slice input matrix.

[0032] FIG4 illustrates an example matrix multiplication by receiving a slice input matrix, a slice input vector, and control values ​​for the slice input matrix.

[0033] FIG5 is a flowchart of an example process for multiplying a submatrix of a sparse matrix on a sparse slice with a system input vector according to aspects of the present disclosure.

[0034] FIG6 is a flowchart of an example process for generating a system output vector representing the product of a system input vector and a system input matrix from multiple sparse slices.

[0035] FIG. 7A is a diagram of an example system input matrix and system input vector.

[0036] FIG. 7B is a diagram of partitions of the system input matrix and system input vector.

[0037] FIG. 7C is a diagram of partitions of the system input matrix and system output vector, wherein the system output vector represents the product of multiplying the system input matrix and system input vector.

[0038] FIG. 8A is a flowchart of an example process for configuring sparse partitioning using one or more control values ​​of the partitioned input matrix according to aspects of the present disclosure.

[0039] FIG. 8B is a flowchart of an example process for performing matrix-vector multiplication using sparse partitioning configured according to the process of FIG. 8A.

[0040] Figure 9 is a flowchart of an example process for generating a submatrix from a system input matrix according to aspects of the present disclosure.

[0041] Figure 10 is a block diagram of an example computational environment for implementing a sparse matrix multiplication system according to aspects of the present disclosure. Detailed Description

[0042] Overview

[0043] Aspects of the present disclosure relate to a system comprising one or more integrated circuits configured for sparse matrix-dense vector multiplication. A system with multiple sparse slices can receive a submatrix and a subvector of an input system matrix and vector for each sparse slice. Each sparse slice may be at least part of an integrated circuit and implement several arithmetic units, such as multiplier circuitry and adder circuitry. Each sparse slice is configured to receive a submatrix having several non-zero values ​​equal to or less than a maximum non-zero threshold, which may be predetermined when the system is implemented, for example, as a chip including sparse slices and optionally one or more other components as described herein.

[0044] Each sparse slice may additionally receive metadata, such as control values ​​described herein, which the sparse slice can use to direct different inputs to each multiplier circuit or adder circuit and to gating when the output from one unit is passed to another unit. By configuring the sparse slices according to the received metadata, these sparse slices can efficiently process input submatrices of arbitrary size up to a predetermined dimension threshold and output the product as a vector of length corresponding to the input.

[0045] As part of receiving the submatrices, each sparse slice may receive one or more control values ​​that indicate the position of a non-zero value at each column in the submatrices. Using these one or more control values, the sparse slices can be configured to adjust how the individual values ​​of the submatrices and subvectors are multiplied, added, and arranged so that the sparse slices generate sliced ​​output vectors representing the products of the submatrices and subvectors. The system can be further configured to generate from the sliced ​​output vectors of each slice a system output vector representing the product of applying the system input vector to the system input matrix (e.g., by multiplying the system input vector by the system input matrix).

[0046] Aspects of this disclosure also relate to a system for preprocessing sparse matrices for processing by an array of sparse slices. A system with one or more processors can be configured to partition an input matrix into multiple submatrices, wherein the dimension of each submatrix is ​​equal to or less than a predetermined dimension threshold of the maximum matrix input size for a specified sparse slice. As part of the partitioning, the system can identify whether any submatrix includes a number of non-zero values ​​greater than a predetermined maximum non-zero threshold, and in response, repartition the submatrix along the same rows or columns as the identified submatrix. The system can repeat this process until submatrices are generated for each sparse slice such that the dimension of the submatrix is ​​within the dimension threshold, and the number of non-zero values ​​in the submatrix is ​​less than or equal to the predetermined non-zero threshold. (Specification 4 / 19 pages 7 CN)122153225 A

[0047] Systems implemented according to aspects of this disclosure can perform workloads involving repetitive multiplications of sparse matrices and vectors more efficiently. For example, a device implementing a system-on-chip (SoC) according to aspects of this disclosure can generate a product of sparse matrices and vectors in fewer processing cycles than conventional methods, at least because the device omits redundant "multiply-by-zero" computations of the zero elements in the matrix, which do not contribute to the final product. While other methods may experience increased performance loss for larger matrices with a high sparsity ratio, systems implemented as described herein can use these matrices to compute products even more efficiently, at least because a higher proportion of redundant computations are omitted as the sparsity ratio of the input matrix increases.

[0048] Some workloads, such as executing or training neural networks, heavily rely on performing matrix multiplications. Hardware accelerators or other devices can perform certain operations, such as matrix multiplications, efficiently, but often with limitations on processing power. For example, a device may require a predefined sparsity ratio between the zero and non-zero values ​​of a matrix, or have strict limitations on the size of the input to be processed. While matrix multiplication is a common type of computation for many neural network workloads, accelerators with constraints are limited in the number of types of workloads they can support, such as workloads involving matrix multiplication of matrices with different sparsity ratios.

[0049] Aspects of this disclosure provide a system for sparse matrix multiplication that can flexibly handle matrices of different sizes and sparsity ratios. The sparse slices of a system configured for sparse matrix multiplication can discard the zero values ​​of the input matrix while still tracking the positions of non-zero values ​​in a given input submatrix across multiple columns. The sparse slices can be configured with only as many registers and multiplier circuitry as a predetermined maximum non-zero threshold and store only the non-zero values ​​of the input submatrix in memory. As a result, sparse slices can store and process input data more efficiently and with fewer resources compared to storing a full-size matrix.

[0050] Furthermore, sparse slicing can be configured to arrange non-zero values ​​and their products when multiplied with the values ​​of the input vectors along adjacent multiplier and adder circuits, respectively. Sparse slicing can be configured to efficiently generate piecewise sums for each column of the input matrix by combining the outputs of these adjacent circuits, and rearrange the piecewise sums to generate a sliced ​​output vector. Sparse slicing can preserve the order of non-zero values ​​to accurately add and multiply these values ​​with corresponding values ​​of the input vectors, and accurately generate a sliced ​​output vector as the product of the input submatrix and the input vector. Sparse slicing can perform multiplication on the input matrix without requiring preprocessing operations to change the shape of the input matrix.

[0051] The system may also include one or more processors configured to preprocess sparse values.The matrix is ​​to be multiplied by the input vector by multiple sparse slices. The system can support different configurations of sparse slices, including generating different numbers of submatrices depending on the number of available sparse slices, and generating submatrices for different maximum non-zero thresholds.

[0052] The aspects of this disclosure as described herein can be implemented by a system including one or more processors and multiple sparse slices. The system can be implemented, for example, as a chip on a server computing device installed in a data center of a computing device such as a computing platform.

[0053] Example System

[0054] FIG1 is a block diagram of an example system 100 including an array 101 of sparse slices 101A-101P. Array 101 can be at least a portion of a sparse matrix multiplication system (such as sparse matrix multiplication system 100) configured to perform matrix multiplication on matrix inputs.

[0055] As described in more detail herein with reference to FIG2, a sparse slice is a collection of circuits configured to perform arithmetic operations, arrange inputs and outputs between circuits performing arithmetic operations, and / or gate inputs and outputs between other circuits. For example, sparse slices can be configured to perform matrix multiplication between rectangular slices or portions of a sparse matrix and linear slices or portions of a vector. As described in more detail herein, sparse slices can receive any of a variety of different slices or portions of a sparse matrix, wherein any number of non-zero values ​​are present in said slice or portion up to a predetermined maximum threshold.

[0056] The set of circuits can be configured to control: the state of each gate (i.e., open or closed); the operands to be computed; and / or the arrangement of the inputs to each circuit and the outputs from each circuit. The configuration may depend at least in part on the dimension of the sliced ​​input matrix in which the sparse slices are configured to be received as inputs, and / or on the arrangement and number of non-zero values ​​in the sliced ​​input matrix. Each sparse slice can be a separate circuit component of a system configured to multiply a sparse matrix with a vector.

[0057] Each sparse slice 101A-101P is configured to communicate with two or more other sparse slices in array 101. A sparse slice can communicate with its immediate neighbors, for example, neighbors immediately before or after the sparse slice along a dimension defined by the rectangular arrangement of the sparse slices 101A-101P. Connections between sparse slices can be implemented, for example, via a bus or one or more circuit interconnects that physically connect the sparse slices to their neighbors. Each sparse slice 101A-101P can be implemented as one or more circuits configured to receive at least a portion of a matrix and at least a portion of a vector and generate an output vector representing the product of the input matrix and vector. In some...In the example, sparse slices 101A-101P are organized as a systolic array, but in various implementations, array 101 is typically configured or arranged according to a rectangular arrangement of sparse slices 101A-101P.

[0058] Array 101 may be at least part of a system-on-a-chip that implements several components and integrated circuits on a circuit board or other material. Array 101 may be mounted as part of a computing device and configured to interact with other components of the device, including memory, processor, network components, and / or peripheral devices. For example, array 101 may receive system input vector 105 and system input matrix 110 from one or more memory devices implemented as part of system 100. System 100 may generate a system output vector 115 as an output representing the product of multiplying system input vector 105 and system input matrix 110.

[0059] In some examples, system output vector 115 may be fed as input to other devices or components of the device implementing system 100. For example, if the system output vector 115 is a product of model parameter values ​​multiplied by a vector input of the neural network, the system output vector 115 can be fed as input to one or more processors configured to compute activation functions for the output vector 115.

[0060] System 100 may receive system input vector 105 and system input matrix 110 from preprocessing engine 150. Preprocessing engine 150 may be implemented on one or more computing devices, which may or may not be the devices implementing system 100. As an example, system input matrix 110 may include values ​​representing at least a portion of the model parameter values ​​used for the neural network. System input matrix 110 may be part of a more complex data structure, such as a multidimensional array or tensor. System 100 may be configured to receive each matrix, such as a three-dimensional tensor or matrix, corresponding to at least a portion of a larger data structure, and generate a corresponding output when multiplied with system input vector 105. System input vector 105 may be, for example, the input of a trained neural network whose model parameter values ​​are at least partially represented by the values ​​of system input matrix 110.

[0061] The preprocessing engine 150 can be configured to process the system input matrix 110 to generate one or more control values ​​111 and matrix partition data 112. The preprocessing engine 150 can receive the system input matrix 110 in various formats for storing sparse matrices, such as its complete row and column form (with all non-zero and zero values). As other examples, the preprocessing engine 150 can receive the system input matrix 110 according to a compressed sparse column format, a coordinate list format, or any of various other formats for storing sparse matrices.

[0062] In some examples where the preprocessing engine 150 receives the system input matrix 110 in its complete form, see page 6 / 19 of the preprocessing engine specification, 9 CN.122153225 A 150 is configured to convert the system input matrix 110 into a format pre-determined to be suitable for storage in memory. For example, before conversion to the predetermined format, the preprocessing engine 150 may remove zero values ​​from the matrix and generate control values ​​that track the position of non-zero values ​​relative to their original positions in the matrix.

[0063] The control values, as described in more detail herein with reference to FIG2, are used to configure each sparse slice to receive a subvector of the system input vector 105 and multiply it by a submatrix of the system input matrix 110. Partition data 112 is data specifying how the system input matrix 110 should be partitioned into submatrices. Each submatrix is ​​received as input at the corresponding sparse slice, and partition data 112 specifies partitions of the submatrices as many as the sparse slices in array 101.

[0064] The preprocessing engine 150 may be configured to generate vector partition data 106. As described in more detail with reference to Figures 7A through 7C, sparse slices along the same rows or columns can receive the same subvectors as the input for multiplication with the corresponding submatrix.

[0065] Although the system input matrix 110 and system input vector 105 are shown as being fed along the left and right sides of system 100, the exact location and orientation of the bus that feeds the input data to system 100 can vary depending on the implementation. For example, based on the location of other components on the same chip as system 100, the bus or circuit interconnects used to feed input to system 100 and receive output from system 100 can be oriented or positioned differently to illustrate the location of those other components.

[0066] Figure 2 is a block diagram of an example sparse slice 200 according to various aspects of the present disclosure. For example, sparse slices 101A-101P of system 100 can each be implemented as described herein with reference to sparse slice 200.

[0067] Sparse slice 200 is configured to receive slice input vector 205 and slice input matrix 210. The sliced ​​input vector 205 includes one or more vector values ​​and may have a maximum dimension of 1×R. The sliced ​​input matrix includes one or more zero values ​​and one or more non-zero values ​​and has a maximum dimension of R×C. R (rows) and C (columns) are predetermined dimension thresholds corresponding to the maximum input size of the vector / matrix that the sparse slice can receive as input. In different implementations, sparse slices may be configured for different dimensions R and C. R and C may be equal to or different from each other, and different sparse slices may be implemented for different dimensions, for example, in response to the nature of the data of different workloads that the sparse slices are configured to process. The sparse slices of the array of sparse slices may be configured to receive input within the same maximum dimension threshold.

[0068] The sparse slice 200 may be configured to be within a predetermined address range in memory associated with the sparse slice 200.The fragment input vector 205 and fragment input matrix 210 are received. For example, sparse fragment 200 is configured to automatically retrieve fragment input vector 205 from a first address range in coupled memory and fragment input matrix 210 from the same or different address range in memory. A device or component of an apparatus implementing system 100 having sparse fragment 200 may be configured to send fragment input matrix and fragment input vector to a location in memory corresponding to each of one or more sparse fragments implemented by the system. For example, after generating the processing system input matrix and / or system input vector, preprocessing engine 150 may be configured to store individual fragment input matrices and fragment input vectors at an address range corresponding to each sparse fragment (including sparse fragment 200).

[0069] In addition to a maximum dimension threshold, sparse fragment 200 is configured to also receive a fragment input matrix having a non-zero count equal to or less than a predetermined maximum non-zero threshold. Similar to the dimension threshold, a maximum non-zero threshold can be set for the sparse shard 200 and its corresponding array, depending on the implementation. For example, if the system is configured to process workloads whose data typically includes processing matrices with a high sparsity ratio, the sparse sharding system can be configured with a relatively high maximum non-zero threshold. As described herein, the number of multiplier and adder circuits in a sparse shard corresponds to its maximum non-zero threshold, so a sparse shard with a relatively low maximum non-zero threshold can be constructed with fewer circuits compared to a sparse shard with a higher maximum non-zero threshold.

[0070] The sparse shard 200 may include a cross switch 215 and a multiplier circuit 220. The cross switch 215 is configured to receive a maximum of R vector values ​​and distribute these values ​​across N multiplier circuits 220, where N is equal to the maximum non-zero threshold of the sparse shard 200. Multiplier circuits 220A-220C, 220N are shown, but it should be understood that in different implementations, sparse fragmentation 200 may include more or fewer multiplier circuits.

[0071] Cross-switch 215 can be implemented as any one or more circuits configured to receive input and pass the input to one or more destinations, which themselves may be other circuits such as multiplier circuit 220. Multiplier circuit 220 can be implemented according to any of a variety of different techniques for performing hardware multiplication between two operands. The first operand for the multiplier circuit can be a vector value received by cross-switch 215. The second operand for the multiplier circuit can be a non-zero value from fragmented input matrix 210. Each non-zero value is loaded into the corresponding registers 221A-221C, 221N of the corresponding multiplier circuits 220A-220C, 220N. The number of multipliers is as many as the maximum non-zero threshold.The circuit provides a usable multiplier circuit for each sliced ​​input matrix within a non-zero threshold. Each multiplier circuit multiplies a non-zero value stored in its corresponding register with a vector value received by the cross switch 215.

[0072] The sparse slice 200 may also include adder circuits 225. Each adder circuit 225A-225C, 225N is configured to receive input from the corresponding multiplier circuit. In the middle of each adder circuit is a segment marker. Adder circuits 225A-225C, 225N and segment markers 226A-226C, 226N-1 are shown in FIG2, but as with multiplier circuit 220, it should be understood that the number of adder circuits and segment markers may vary depending on the implementation.

[0073] The adder circuits may be implemented using any technique for hardware addition of two operands. The first operand from the adder circuit may be the product received from the multiplier circuit. For example, multiplier circuit 220A passes a product of vector values ​​and non-zero values ​​to adder circuit 225A. Segment markers are circuitry or other hardware components configured to gate inputs between adjacent adder circuits based on the gate input value.

[0074] Control value 230 may be received as input along with slice input matrix 210 and slice input vector 205, and is used to configure one or more of cross switches 215, segment markers 226, and / or cross switches 235. Control value 230 may be a sequence of values, each corresponding to a corresponding non-zero value in slice input matrix 210. Control value 230 may include values ​​of a first type, such as 1, which may correspond to a non-zero value as the first non-zero value of its corresponding column in the slice input matrix. Control value 230 may include values ​​of a second type, such as 0, which may correspond to a zero value preceding one or more other non-zero values ​​in the same column of the slice input matrix. As described herein with reference to Figures 3A through 3C, control value 230 may also include one or more vectors, as described herein, that can be used to configure sparse partitioning to process the values ​​of partitioned input matrix 210 and partitioned input vector 205. Figure 4 and its corresponding description herein illustrate an example matrix multiplication using sparse partitioning.

[0075] In some implementations, cross switches 215, 235 may be implemented using different maximum dimension thresholds of sparse partitioning 200. For example, when dimensions R and C are equal or approximately equal, then cross switch 235 may be implemented according to any technique used for cross rearranging square or approximately square inputs, such as a Bene network.

[0076] Figures 3A through 3C illustrate an example partitioned input matrix 300A, a non-zero value vector 300B in the partitioned input matrix, and a cross switch 205.The control value vector 300C corresponding to the sliced ​​input matrix 700A. Figure 3C also shows additional control vectors 305C and 310C.

[0077] Figure 3A is a diagram of an example sliced ​​input matrix 300A for sparse slices. For clarity, non-zero values ​​are shown as shaded cells. For clarity, indices are provided along the columns and rows of the sliced ​​input matrix 300A and along vectors 300B and 300C. For example, in matrix 300A, the value at row 2, column 4 (2,4) is 1.

[0078] Figure 3B is a diagram of the vector 300B of non-zero values ​​of an example sliced ​​input matrix 300A. The non-zero values ​​in vector 300B correspond to the order in which non-zero values ​​appear in the sliced ​​input matrix 300A when read from left to right, but the exact reading order may vary depending on the implementation, for example, from right to left.

[0079] Figure 3C is a diagram of vectors 300C, 305C, and 310C for control values ​​of an example sliced ​​input matrix 300A. In some examples, vectors 300C, 305C, and 310C may be part of the same vector in a predetermined order, and the sparse slices may be configured to use the predetermined order when receiving vectors of these control values ​​and configuring different components of the sparse slices according to vectors 300C, 305C, and 310C, as described herein.

[0080] Vector 300C corresponds to the control value for segment markers used to configure the sparse slices. A control value (a bit in this example) of 1 in vector 300C corresponds to the beginning of a new column in matrix 300A. The value at index 0 may be automatically set to 1 as the beginning of vector 300C of control values. In some implementations, the initial control value may be omitted by the processing of the sparse slices and assumed to be a constant. Hardware implementations can simplify their circuitry by utilizing the fact that this value is known to be 1. The value at index 1 in vector 300C is also set to 1 to correspond to the value at index 1 in vector 300B, which is the first non-zero value in the next column of matrix 300A. The value at index 2 in vector 300C is set to 0 because it corresponds to a non-zero value in vector 300B that is not the first non-zero value in the next column.

[0081] As another example, the bit at index 3 in vector 300C is set to 1 because the corresponding non-zero value (value 1 in index 3 of vector 300B) is the first non-zero value in the next column (specifically the second column) of matrix 300A. The sequence in vector 300C follows this described pattern until all non-zero values ​​in all columns are represented.

[0082] Vector 305C corresponds to the control value used to configure the input crossbar switch for sparse slicing. Vector 305C is for slicing inputEach non-zero value in matrix 300A specifies the "y" coordinate of the non-zero value within submatrix 300A. In this example, the "y" dimension moves vertically up and down along the sparse matrix 300A, but in other examples, the "y" dimension can be defined differently, for example, horizontally. For example, the value "3" at element zero in vector 305C corresponds to the "y" coordinate of the non-zero value "1" in the first column of matrix 300A. As another example, the value "4" at element six in vector 305C corresponds to the value "1" at the bottom of the fourth column of matrix 300A.

[0083] Vector 310C corresponds to the control values ​​used to configure how the adder circuit generated by the sparse slices is arranged by the output crossbar switches to generate sliced ​​output vectors.

[0084] According to the mathematical definition of matrix multiplication, each value at position (x, y) in the sparse matrix is ​​multiplied by the value at position y in the input vector. The result of the multiplication is added to the output at position x. The input crossover switch uses vector 305C to arrange non-zero values ​​of the same column into adjacent multiplier circuits in the sparse slice. The output crossover switch uses vector 310C to arrange the calculated sums in the correct order in the sliced ​​output vector, thereby representing the product of multiplying the sliced ​​input matrix and the sliced ​​input vector by the sparse slice.

[0085] Referring back to FIG2, the crossover switch 215 can be configured to receive control value 230 and arrange each value of the sliced ​​input vector 205 as the input to one or more multiplier circuits matching the corresponding sliced ​​input matrix column when the sliced ​​input vector is multiplied by the sliced ​​input matrix.

[0086] The crossover switch 235 can be configured to receive one or more sums from the adder circuit and rearrange the received sums to obtain the correct output slice vector corresponding to multiplying the input slice matrix by the input slice vector. As with the crossover switch 215, any of a variety of different techniques can be applied to implement the crossover switch 235 as one or more circuits.

[0087] As described herein, segment markers are configured to gate inputs between adjacent adder circuits depending on the gate input value of the segment marker. For example, when a segment marker receives a control value of 1, it prevents the output from the first adder circuit adjacent to the segment marker from being passed as input to the second adder circuit adjacent to the segment marker. When a segment marker receives a control value of 0, it passes the output from the first adder circuit to the second adder circuit (or, in some implementations, the reverse). This configuration of the segment markers corresponds to adding only the sums corresponding to the non-zero values ​​in the same column, separated from the sums of the non-zero values ​​in different columns. As described herein with reference to FIG. 2, one or more sums can be passed to the cross switch 235 and rearranged to generate the sliced ​​output vector 240.

[0088] Cross switch 235 can be configured to determine whether to discard an input or accept an input and rearrange it to match its correct position in the sliced ​​output vector 240. As shown in Figure 2, each adder circuit can pass its output to cross switch 235 (indicated by the arrows pointing to cross switch 235). If the next segment marker after an adder circuit is not selected, cross switch 235 can discard the output to that adder circuit because the running sum between adjacent adder circuits has not yet finished. If the next segment marker is selected (or if there is no segment marker, in the case of the last adder circuit 225N), the running sum of the column has been completed, and cross switch 235 receives the running sum as input to be part of the output sliced ​​vector 240. By tracking which sums to ignore and which sums to include as part of the sliced ​​output vector 240, cross switch 235 can accurately track column sums to generate output vectors of varying lengths up to the maximum dimension C.

[0089] Figure 4 illustrates an example computation of a sparse slice 400 receiving a sliced ​​input matrix 410, a sliced ​​input vector 405, and control values ​​430 for the sliced ​​input matrix 410.

[0090] Consider example values ​​for the sliced ​​input vector 405 and the sliced ​​input matrix 410: (vector 405) (matrix 410).

[0091] The corresponding control vectors 430-432 for matrix 410 in this example are: (control vector 430) (control vector 431) (control vector 432) For ease of description, multiplier circuits 40A-40D are abbreviated as multiplier A-D, segment labels 43A-43C are abbreviated as segment labels A-C, and adder circuits 42A-42D are abbreviated as adder A-D.

[0092] Based on the non-zero values ​​of matrix 410, multiplier A is loaded with the value 2, multiplier B is loaded with the value 1, multiplier C is loaded with the value 3, and multiplier D is loaded with the value 4. Note that for this example, the sparse sharding only includes four multipliers A-D and four adders A-D.

[0093] Cross switch 415 receives vector 405 with vector values ​​1, 3, and 2. Dashed lines 45A, solid lines 45B, and dashed lines 45C are shown to illustrate the path of data from sharded input vector 405 to sharded output vector 440. Cross switch 415 receives control vector 431, each value in control vector 431 corresponding to the “y” coordinate of the corresponding non-zero value in sharded input matrix 410. Values ​​in control vector 431 include 1, 2, 0, and 1. Note that the value of the “y” coordinate ranges from 0 to 2 because sparse input matrix 410 has a dimension of 3×3. Control vector 431 specifies which value from slice input vector 405 should be sent to each of the multipliers in multipliers A-D.

[0094] For example, the first value in control vector 431 is 1, corresponding to the "y" coordinate of the first non-zero value in piecewise input matrix 410. Because the "y" coordinate 1 is the second coordinate (after zero), the cross switch 415 routes the second value of piecewise input vector 405 to the first multiplier, here multiplier A. The second value of control vector 431 is 2, corresponding to the "y" coordinate of the second non-zero value in piecewise input matrix 410. The cross switch 405 can be configured to then route the third value of piecewise input vector 405 to multiplier B. As another example, the third value in control vector 431 is 0, corresponding to the next non-zero value after the "y" coordinate is 0. The cross switch 415 routes the first value of piecewise input vector 405 to multiplier C.

[0095] For the first column of matrix 410, the cross switch 415 routes the value 3 to multiplier A and the value 2 to multiplier B. For the second column of matrix 410, the cross switch 415 routes the value 1 to multiplier C. For the third and last columns of matrix 410, the cross switch 415 routes the value 3 to multiplier D. The product of multipliers A-D is as follows: 6 (3×2) for multiplier A, 2 (2×1) for multiplier B, 3 (1×3) for multiplier C, and 12 (3×4) for multiplier D.

[0096] Next, adders A-D receive the product calculated by multipliers A-D. Adder A receives the product from multiplier A, which is 6. The first control value (1) is discarded. Adder A has no previous adders, so it will pass the sum to segment marker A. Segment marker A is not selected because the second value of control value 430 is zero. Adder B receives the current sum (6) from adder A and adds it to the product (2) of multiplier B. Segment marker B is selected, so the output (8) of adder B is passed to cross switch 420 (indicated by dashed line 45A). Adder C will not add the product (3) of multiplier C to anything because segment marker B is selected from the output of adder B. Segment marker C is selected, so the output (3) of adder C is passed to cross switch 420 (indicated by dashed line 45C). Finally, multiplier D receives the product (12) of multiplier D, and because it is the last adder in the sparse slice, it automatically passes its output (12) to cross switch 420 (indicated by solid line 45B).

[0097] The cross switch 420 rearranges the received bits 8, 3, and 12 according to the correct order for outputting the output slice vector 440. The cross switch 420 receives a control vector 432 with values ​​0, 0, 2, and 1. As described herein with reference to Figures 3A through 3C, the values ​​of the control vectors used to output the cross switch 420 correspond to non-zero "x" coordinate positions. Like the "y" coordinates,In this example, the value ranges from 0 to 2. The first two values ​​of control vector 432 are 0. Cross switch 420 therefore routes the received first sum to the first element of output slice vector 440. The next value following the zero in cross switch 432 is 2. Cross switch 432 is configured to route the second sum received from adder AD to the second element of output vector 440 (as shown by line 45C) and the third sum to the third element (as shown by line 45B). In some examples, cross switch 420 is configured to skip consecutive repetitive control values ​​in vector 432, for example, as shown in vector 432 with the first two zeros. In some examples, instead of skipping consecutive repetitive control values, cross switch 420 is configured to perform an inclusive OR operation on the received input sum and output the result of the inclusive OR operation to the position of output vector 440 corresponding to the consecutive repetitive control value.

[0098] For example, line 46 shows the potential input source from adder A to output cross switch 420. Because segment marker 43A has a zero value, the output of adder A to cross switch 420 is suppressed, for example, masked or set to zero. The output of adder A is instead passed to adder B via segment marker 43A. In some examples, when output cross switch 420 receives control value 432, output cross switch 420 performs an implication OR operation on the received first sum (via line 46, with a value of 0) and the received second sum (via line 45A, with a value of 8 from adder B). Cross switch 420 can be configured to perform an implication OR operation to output a non-zero operand. Cross switch 420 will output 8 after performing an implication OR operation (e.g., 0 or 8) on the received sums and will pass the result to a first position in output vector 440. In some examples, output cross switch 420 can receive individual outputs from at least some adders A-D and compute the sum of these individual outputs.

[0099] The adder circuit can be implemented according to any of a variety of different circuit configurations for adding consecutive ranges of numbers in a defined segment, each segment corresponding to a value in a corresponding column of the sliced ​​input matrix processed by the corresponding sparse slice. For example, the adder circuit of sparse slice 200 or 400 can be implemented as one or more sequential segmented summation circuits (e.g., as shown by sparse slice 200 or 400) for performing sequential segmented summation of the product by multiplying the non-zero values ​​in each column with the corresponding values ​​in the sliced ​​input vector. In some implementations, the adder circuit can be configured to perform the summation tree as a parallel segmented summation circuit. Individual adder circuits can be configured to add corresponding inputs in parallel and, as described herein, pass their sum to the output crossbar switch and / or adjacent adder circuits according to a gate value marked by any intermediate segment.

[0100] Parallel segmented summation circuits can reduce the time delay of the circuit relative to sequential segmented summation circuits, especially when the number of terms to be summed is large. Control values ​​for gating segment markers allow for parallel segmented summation, at least because the range of values ​​to be summed together can be tracked based on the threshold value of the segment markers, which gating inputs between adder circuits corresponding to values ​​in different segments and allowing inputs between adder circuits corresponding to values ​​to be added in the same segment.

[0101] In some implementations, the output crossover switch 420 is configured to receive the individual sums calculated by each adder, and a control vector 432 is used to add these sums and route them to the corresponding elements of the output slice vector 440. If two or more sums are routed to the same element, for example, control vector 432 includes a copy of the same value as the first zero in vector 432, the output crossover switch is configured to add each received sum that is to be routed to the same element in output slice vector 440, instead of receiving a single sum as shown by line 45A in FIG4.

[0102] Example Method

[0103] FIG5 is a flowchart of an example process 500 for multiplying a submatrix of a sparse matrix on a sparse slice with a system input vector according to aspects of the present disclosure. For ease of description, the submatrix of the sparse matrix is ​​referred to as the slice input matrix. An example process for partitioning an input sparse matrix into multiple submatrices is described herein with reference to FIG9. For example, sparse slice (such as sparse slice 200 of FIG2) performs process 500.

[0104] According to block 510, the sparse slice receives a slice input matrix. The slice input matrix is ​​within a predetermined dimension threshold and has a count of nonzero values ​​equal to or less than a predetermined maximum nonzero threshold.

[0105] According to block 520, the sparse slice receives a sliced ​​input vector comprising multiple vector values. The sliced ​​input vector is a subvector of the system input vector, which may be generated as part of the input of the preprocessing engine preprocessing system, as described herein with reference to Figures 1 and 9.

[0106] According to block 530, the sparse slice generates one or more products of corresponding vector values ​​multiplied by corresponding non-zero values. As described herein with reference to Figures 2 and 4, the sparse slice may be configured with control values ​​corresponding to the positions of non-zero values ​​in the sliced ​​input matrix. Based on this control value, the sparse slice may be configured to route the incoming vector values ​​of the subvectors to the corresponding adjacent multiplier circuits, stored together with non-zero values ​​along the same column of the input slice matrix. Figure 8A described herein illustrates an example process for configuring the sparse slice using the control value of the sliced ​​input matrix.

[0107] According to block 540, the sparse slice generates one or more sums of the one or more products. As described herein with reference to Figure 2, sparse slicing comprises multiple adder circuits configured to receive inputs from corresponding multiplier circuits. AdditionThe adder circuits are further configured to add these inputs along adjacent adder circuits until a segment marker is reached, which is set to the gate circuit adjacent to the marker. The sum of the adder circuit inputs up to the segment marker can be passed to a cross switch, such as cross switch 235 of FIG2.

[0108] Sparse partitioning generates a partitioned output vector from the one or more sums as a product of multiplying the partitioned input vector with the partitioned input matrix. For example, by a second cross switch, the sparse partitioning can rearrange the received sums depending on how the operands of the product generated by the multiplier circuits are ordered by the first cross switch.

[0109] FIG6 is a flowchart of an example process 600 for generating a system output vector representing the product of the system input vector and the system input matrix from multiple sparse partitions. A sparse partitioning system (such as the sparse matrix multiplication system 100 of FIG1) can perform process 600.

[0110] According to block 610 and for each group of sparse pieces along the column dimension of the array of sparse pieces, the piecewise output vectors of each sparse piece in the group are summed to generate the columnwise output vector of the group. Figure 7C below shows an example grouping of sparse pieces according to the corresponding piecewise input matrix. The dimension along which the group of sparse pieces forms can vary depending on the implementation. For example, depending on the direction in which the system input matrix and the system input vector are fed into the array of sparse pieces, the group can be along the rows of the array instead of the columns. Specification 12 / 19 pages 15 CN 122153225 A

[0111] According to block 620, the system concatenates each columnwise output vector to generate a system output vector. The system output vector is the product of the system input matrix and the system input vector. In some implementations and as described in more detail herein with reference to Figure 9, the system can receive a system input matrix in which the columns of the matrix are permuted, for example, to make the occurrence of nonzero values ​​more evenly distributed across the submatrices assigned to the sparse pieces. In those implementations, the system can be configured to rearrange the elements of the cascaded system output vector to reverse the original permutation. The system can receive data that is restricted to a reordered form as part of receiving the system input matrix, control values, and data that is restricted to partitions of the system input matrix.

[0112] Figures 7A through 7C and the corresponding descriptions illustrate an example multiplication between an example sparse matrix 700 and a vector 750. For illustrative purposes, the multiplication is described as being performed on a system with a 4×4 array of 16 sparse partitions.

[0113] Figure 7A is an illustration of an example system input matrix 700 and a system input vector 750. In this illustration, the system input matrix 700 is shown with integer values, but it should be understood that the elements may be other values, such as floating-point values. Additionally, in Figures 7A through 7C, the non-zero value elements of the various matrices that include the system input matrix 700 are shown as shaded cells.

[0114] Figure 7B is a diagram of the partitioning of the system input matrix 700 and the system input vector 750. In this example, the system input matrix 700 is partitioned into sixteen submatrices 700A-700P, one submatrix for each sparse slice of the 4×4 array. The system input vector 750 is partitioned into four subvectors 750A-750D, one subvector for each column of the 4×4 array of the sparse slices.

[0115] An example mapping of the subvectors and submatrices to the sixteen sparse slices (referred to as sparse slices A-P) is shown in Table 1 below: Specification 13 / 19 Page 16 CN 122153225 A

[0116] Table 1

[0117] Figure 7C is a diagram of the partitioned system input matrix 700 and the system output vector 770 representing the product of the system input matrix and the system input vector 750. Figure 7C shows the partitioned matrix grouped along columns 705A-705D. As described herein with reference to Figure 6, the system can sum the slice output vectors of each column of the sparse slice array and generate a column output vector. In Figure 7C, column output vectors 710A-710D correspond to columns 705A-705D. The system can concatenate column output vectors 710A-710D to generate a system output vector as a product of multiplying the system input matrix and the system input vector.

[0118] Figures 8A and 8B are flowcharts of example processes 600A-600B for configuring sparse slices and performing matrix multiplication using one or more control values ​​of the input matrix according to various aspects of the present disclosure.

[0119] Figure 8A is a flowchart of example process 800A for configuring sparse slices using one or more control values ​​of the slice input matrix according to various aspects of the present disclosure. Sparse slices (such as sparse slice 200 of Figure 2) can perform the process of 800A.

[0120] According to block 810, the sparse slice receives one or more control values ​​specifying the positions of non-zero values ​​along each column of the slice input matrix.

[0121] According to block 820, the sparse slice loads the non-zero values ​​of the slice input matrix into registers of multiplier circuits. As described herein with reference to FIG2, the sparse slice can implement a plurality of multiplier circuits equal to a predetermined maximum non-zero threshold of the system. The sparse slice can load the non-zero values ​​in the order they appear when the slice input matrix is ​​read along a predetermined read direction (e.g., from left to right).

[0122] According to block 830, the sparse slice loads the one or more control values ​​into a cross switch of the sparse slice. A first cross switch (such as cross switch 815 of FIG2) can be configured to receive control values ​​and arrange each value of the slice input vector as being received as input to one or more multiplier circuits, which are associated with a corresponding cross switch.The corresponding sliced ​​input matrix column is matched, wherein the corresponding sliced ​​input matrix column is multiplied by the vector value as part of the matrix-vector multiplication. A second cross switch (such as cross switch 235 of FIG2) can be configured to receive one or more sums from the adder circuit and rearrange the received sums to obtain the correct output slice vector, which corresponds to multiplying the input slice matrix by the input slice vector.

[0123] According to block 840, the sparse slice loads the one or more control values ​​into one or more segment tags, which are configured to gate the input of the adder circuit based on the values ​​of these control values.

[0124] FIG8B is a flowchart of an example process for performing matrix-vector multiplication using sparse slices configured according to process 800A of FIG8A.

[0125] According to block 850, the sparse slice receives and loads non-zero values ​​of the input slice matrix.

[0126] According to block 860, the sparse slice receives the vector values ​​of the sliced ​​input vector along with non-zero values ​​along the same columns of the sliced ​​input matrix and sends them to the multiplier circuit.

[0127] According to block 870, the sparse slice generates one or more segmented sums from adjacent adder circuits that are not selected by segment markers. As described herein with reference to Figures 2 and 8A, the sparse slice can aggregate sums between adder circuits that are not selected by segment markers and uses control values ​​to configure segment markers to select adder circuits representing computations from different columns of the sliced ​​input matrix. When the sparse slice reaches a segment marker with active gate bits, the sparse slice passes the segmented sums to a second crossover switch, which is configured to rearrange the segmented sums and generate a sliced ​​output vector.

[0128] According to block 880, the sparse slice generates a sliced ​​output vector from the one or more segmented sums. As described herein with reference to Figures 2 and 8A, control values ​​can be used to configure the crossover switch to receive one or more segmented sums from a sparsely segmented adder circuit. The crossover switch can be further configured to rearrange the segmented sums to generate a correctly segmented output vector, which represents the product of multiplying the segmented input matrix by the segmented input vector.

[0129] Figure 9 is a flowchart of an example process 900 for generating a submatrix from a system input matrix according to various aspects of the present disclosure. One or more processors at one or more locations can perform process 900. For example, a preprocessing engine (such as preprocessing engine 150 of Figure 1) can perform process 900.

[0130] According to block 910, the preprocessing engine receives a system input matrix. The system input matrix can be, for example, a matrix as shown with system input matrix 700 of Figure 7A.

[0131] According to block 920, the preprocessing engine partitions the system input matrix into a plurality of candidate submatrices. As part of the partitioning, the preprocessing engine can receive parameters specifying a predetermined dimension threshold, as well as instructions implemented by the system receiving the submatrices.One or more parameter values ​​for the number of sparse partitions. For example, the preprocessing engine can be configured to generate 16 candidate submatrices within an 8-row by 8-column dimensional threshold (to obtain a 4×4 array of sparse partitions). In some examples, the preprocessing engine can receive updated parameter values, for example, for different system preprocessing inputs with different dimensional thresholds and / or configurations across sparse partitions.

[0132] In some implementations, before the preprocessing engine partitions the system input matrix, the preprocessing engine permutates columns in the system input matrix to make non-zero values ​​evenly distributed in the candidate submatrices. For example, if non-zero values ​​occur more frequently on one side of the input matrix than on the other side than a predetermined tolerance, the preprocessing engine can be configured to change the order of columns in the input matrix so that the occurrence of non-zero values ​​is more spread out and thus more evenly distributed in the sparse partitions after partitioning.

[0133] If the preprocessing engine performs this sorting, the preprocessing engine passes the sorted data as additional input to the system, for example, as part of the partitioned data sent to a system with an array of sparse slices. The system can be configured to reorder the elements of the system output vector according to this sorting so that the output vector matches the output of multiplying the system input matrix and the system input vector before the columns of the system input matrix are permuted.

[0134] Permuting the columns of the system input matrix can improve the overall rate at which the system input matrix is ​​processed by a sparsely sliced ​​system. For example, by permuting the columns, each sparse slice can be used more efficiently, especially when, in some examples, some sparse slices may receive sliced ​​input matrices with only zero values, while other slices may receive sliced ​​input matrix values ​​with only non-zero values, or may receive up to a non-zero threshold of non-zero values.

[0135] According to block 930, the preprocessing engine determines whether there exists a candidate submatrix with a count of non-zero values ​​greater than a predetermined non-zero threshold. According to box 940, if the preprocessing engine determines that there is a candidate submatrix with a count of non-zero values ​​greater than a predetermined non-zero threshold, the preprocessing engine repartitions the submatrix along the same rows or columns as the candidate submatrix.

[0136] As shown in FIG7B, the submatrix can be organized along columns and rows based on the position of its values ​​in the input matrix. For example, if the preprocessing engine determines that submatrix 700J includes a count of non-zero values ​​greater than the non-zero threshold, the preprocessing engine can repartition the submatrix along the rows of submatrix 700J (including submatrixes 700I, 700K, and 700L) and / or along the columns of submatrix 700J (including submatrixes 700B, 700F, and 700N). When performing repartitioning, the preprocessing engine uses a predetermined dimension threshold and performs repartitioning such that the number of candidate submatrixes remains unchanged. For example, the preprocessing can split the candidate submatrix.The preprocessing engine redistributes the rows / columns of the submatrix along the rows / columns of the determined candidate submatrix, according to blocks 930 and 940.

[0137] After repartitioning the submatrix, the preprocessing engine again determines whether there is a candidate submatrix with a non-zero value count greater than a non-zero threshold. The preprocessing engine can repeat the determination and repartitioning according to blocks 930 and 940 until it is determined that there is no candidate submatrix with a non-zero value count exceeding the maximum non-zero threshold, and proceeds to block 950.

[0138] As shown in block 950, the preprocessing engine partitions the system input vector. Depending on the arrangement of the bus of the array feeding the sparse slices, each subvector of the system input vector is input to each row or column of the sparse slice. The preprocessing engine partitions the vectors such that the vector dimension can be multiplied by the matrix in the receiving sparse slice, for example, the matrix has a valid dimension for mathematically valid matrix multiplication.

[0139] For example, as shown in Figures 7B to 7C and Table 1, vector 750 is partitioned into subvectors 750A-750D, each of which is passed as input to one or more sparse slices. Also in Figure 7B, each dimension of each subvector 750A-750D has the correct dimension for multiplication with the corresponding submatrix 700A-700P. For example, subvector 750B is 1×2 (rows × columns), and each of the submatrixes 700E-700H has 2 rows, thus allowing efficient matrix multiplication between subvector 750B and submatrixes 700E-700H.

[0140] As shown in box 960, the preprocessing engine generates control values ​​for each candidate submatrix. As described with reference to Figures 3A to 3C, page 16 / 19 of the specification, CN 122153225 A, the preprocessing engine can generate a vector of control values ​​indicating the initial non-zero values ​​of each column of the matrix. The preprocessing engine repeats this generation for each candidate submatrix, thereby generating a corresponding control value for each submatrix.

[0141] According to block 970, the preprocessing engine outputs the control value and the candidate submatrix. The preprocessing engine may, for example, output the control value and data specifying the partitions of the system input matrix to the system 100 shown in FIG1. ​​

[0142] Example Computing Environment

[0143] FIG10 is a block diagram of an example computing environment for implementing a sparse matrix multiplication system 100 and a preprocessing engine 150 according to aspects of the present disclosure. The preprocessing engine 150 may be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 1015. User computing device 1012 and server computing device 1015 may be communicatively coupled to one or more storage devices 1030 via network 1060. Storage device 1030 may be a combination of volatile and non-volatile memory and may be located in the same or different physical locations as computing devices 1012, 1015.Storage device 1030 may include any type of non-transitory computer-readable medium capable of storing information, such as hard disk drives, solid-state drives, tape drives, optical storage devices, memory cards, ROM, RAM, DVDs, CD-ROMs, writable and read-only memories.

[0144] Server computing device 1015 may include one or more processors 1013 and memory 1014. Memory 1014 may store information accessible by processor 1013, including instructions 1021 executable by processor 1013. Memory 1014 may also include data 1023 retrievable, manipulated, or stored by processor 1013. Memory 1014 may be a non-transitory computer-readable medium capable of storing information accessible by processor 1013, such as volatile and non-volatile memory. Processor 1013 may include one or more central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FGPAs), and / or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0145] The server computing device 1015 may implement the sparse matrix multiplication system 100 in hardware, for example as a system-on-a-chip. The system 100 may be implemented as part of a physical chip inserted into or mounted in the server computing device 1015. The system 100 is configured to communicate with other components of the server computing device 1015.

[0146] Instructions 1021 may include one or more instructions that, when executed by the processor 1013, cause the processor to perform actions defined by those instructions. Instructions 1021 may be stored in an object code format for direct processing by the processor 1013, or stored in other formats, including a collection of interpretable scripts or standalone source code modules that are interpreted on demand or pre-compiled. Instructions 1021 may include instructions for implementing sparse fragmentation 400 consistent with aspects of this disclosure. The preprocessor engine 105 may be executed using the processor 1013 and / or other processors located remotely from the server computing device 1015.

[0147] Processor 1013 may retrieve, store, or modify data 1023 according to instruction 1021. Data 1023 may be stored in computer registers, as a table with multiple different fields and records in a relational or non-relational database, or as a JSON, YAML, proto, or XML document. Data 1023 may also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, data 1023 may include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary code, pointers, references to data stored in other memory, including other network locations, or information used by functions to calculate relevant data.

[0148] User computing device 1012 may also be configured similarly to server computing device 1015, having one or more processors 1016, memory 1017, instructions 1018, and data 1019. User computing device 1012 may also include user output 1026 and user input 1024. User input 1024 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, mouse, mechanical actuator, soft actuator, touchscreen, microphone, and sensor.

[0149] Server computing device 1015 may be configured to transmit data to user computing device 1012, and user computing device 1012 may be configured to display at least a portion of the received data on a display implemented as part of user output 1026. User output 1026 may also be used to display an interface between user computing device 1012 and server computing device 1015. User output 1026 may alternatively or additionally include one or more speakers, transducers or other audio outputs, haptic interfaces, or other haptic feedback that provides non-visual and non-auditory information to the platform user of user computing device 1012.

[0150] Although FIG10 illustrates processors 1013, 1016 and memories 1014, 1017 as being within computing devices 1015, 1012, the components including processors 1013, 1016 and memories 1014, 1017 described herein may include multiple processors and memories that may be in different physical locations and not operate within the same computing device. For example, some of instructions 1021, 1018 and data 1023, 1019 may be stored on a removable SD card, while others may be stored within a read-only computer chip. Some or all of the instructions and data may be stored in a location physically remote from processors 1013, 1016 but still accessible by processors 1013, 1016. Similarly, processors 1013, 1016 may include a collection of processors capable of performing concurrent and / or sequential operations. Computing devices 1015, 1012 may each include one or more internal clocks that provide timing information, which can be used for time measurement of operations and programs run by computing devices 1015, 1012.

[0151] Server computing device 1015 may be configured to receive requests to process data from user computing device 1012. For example, environment 1000 may be part of a computing platform configured to provide various services to users through various user interfaces and / or APIs that expose platform services. As part of performing services, server computing device 1015 may use system 100 to process incoming data. For example, if the service is training a machine learning model, the server computing device...1015 can be configured to perform multiplication operations using system 100 as part of training a machine learning model.

[0152] Devices 1012 and 1015 can communicate directly and indirectly via network 1060. Devices 1015 and 1012 can be configured to accept listening sockets for initiating connections to send and receive information. Network 1060 itself can include various configurations and protocols, including the Internet, World Wide Web, Intranet, Virtual Private Network, Wide Area Network, Local Area Network, and private networks using one or more proprietary communication protocols. Network 1060 can support a variety of short-range and long-range connections. Short-range and long-range connections can be made on different bandwidths, such as 2.402 GHz to 2.480 GHz (typically associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (typically associated with the Wi-Fi® communication protocol); or with various communication standards, such as the LTE® standard for wireless broadband communication. Alternatively, network 1060 may also support wired connections between devices 1012 and 1015, including via various types of Ethernet connections.

[0153] Although a single server computing device 1015 and a user computing device 1012 are shown in FIG. 10, it should be understood that aspects of this disclosure can be implemented according to various configurations and numbers of computing devices, including in examples for sequential or parallel processing, or through a distributed network of multiple devices. In some implementations, aspects of this disclosure can be executed on a single device or any combination thereof. Furthermore, although the preprocessing engine and sparse matrix multiplication system 100 shown are implemented as on the same server computing device 1015, in some implementations, the preprocessing engine 150 is implemented on one or more server computing devices and / or user computing devices 1012 separate from the server computing device 1015.

[0154] Aspects of this disclosure can be implemented in digital circuitry, in a computer-readable storage medium, as one or more computer programs, or in a combination of one or more of the foregoing. Computer-readable storage media can be non-transitory, for example, as one or more instructions executable by a computing device and stored on a tangible storage device.

[0155] In this specification, the phrase "configured to" is used in various contexts relating to a computer system, hardware, or computer program, engine, or module. When a system is referred to as being configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed on the system that, when running, enables the system to perform the one or more operations. When some hardware is referred to as being configured to perform one or more operations, this means that the hardware includes one or more circuits that, during operation, receive input and generate corresponding instructions based on the input.The output of one or more operations. When a computer program, engine, or module is referred to as being configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform the one or more operations.

[0156] Although the operations shown in the figures and recited in the claims are shown in a particular order, it should be understood that these operations may be performed in a different order than shown, and some operations may be omitted, performed more than once, and / or performed in parallel and / or simultaneously with other operations. Furthermore, the separation of different system components configured to perform different operations should not be construed as requiring the separation of components. The described components, modules, programs, and engines may be integrated together as a single system or as part of multiple systems.

[0157] Unless otherwise stated, the above alternative examples are not mutually exclusive but may be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the examples should be understood by way of illustration rather than by way of limitation of the subject matter defined by the claims. Furthermore, the examples described herein and the terms expressed as "such as," "including," etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, these examples are intended to illustrate only one of many possible implementations. Additionally, the same reference numerals in different figures may identify the same or similar elements. Instruction manual page 19 / 19, 22 CN 122153225 A, Figure 1; Instruction manual figure 1 / 14 page, 23 CN 122153225 A, Figure 2; Instruction manual figure 2 / 14 page, 24 CN 122153225 A, Figure 3A, Figure 3B; Instruction manual figure 3 / 14 page, 25 CN 122153225 A, Figure 3C; Instruction manual figure 4 / 14 page, 26 CN 122153225 A, Figure 4; Instruction manual figure 5 / 14 page, 27 CN 122153225 A, Figure 5; Instruction manual figure 6 / 14 page, 28 CN 122153225 A, Figure 6; Instruction manual figure 7 / 14 page, 29 CN 122153225 A, Figure 7A; Instruction manual figure 8 / 14 page, 30 CN 122153225 A, Figure 7B; Instruction manual figure 9 / 14 page, 31 CN 122153225 A Figure 7C, Appendix to the Instruction Manual, Page 10 / 14, 32 CN 122153225 A Figure 8A, Appendix to the Instruction Manual, Page 11 / 14, 33 CN122153225 A Figure 8B Description Drawings Page 12 / 14 34 CN 122153225 A Figure 9 Description Drawings Page 13 / 14 35 CN 122153225 A Figure 10 Description Drawings Page 14 / 14 36 CN 122153225 A ABSTRACT The present disclosure is directed to sparse matrix multiplication in hardware. Methods, systems, and apparatuses, including computer-readable storage media, for sparse matrix multiplication are provided. A system for matrix multiplication includes an array of sparse shards. Each sparse shard can be configured to receive an input sub-matrix and an input sub-vector, where the input sub-matrix has a number of non-zero values equal to or less than a predetermined maximum non-zero threshold. The sparse shard can, by a plurality of multiplier circuits, compute one or more products of vector values multiplied with respective non-zero values of the input sub-matrix. The sparse shard can generate, as output from the sparse shard and using the one or more products, a shard output vector that is the product of applying the shard input vector to the shardinput matrix.

Claims

1. A system comprising: Sparse partitioning, wherein the sparse partitioning includes multiple multiplier circuits, wherein the sparse partitioning is configured as follows: A sliced ​​input matrix is ​​received, the sliced ​​input matrix including a number of non-zero values ​​equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits in the plurality of multiplier circuits, wherein the multiplier circuits receive the corresponding non-zero values ​​of the sliced ​​input matrix. Receive a fragmented input vector, wherein the fragmented input vector includes multiple vector values; One or more products are generated by multiplying vector values ​​generated by the plurality of multiplier circuits with the corresponding non-zero values ​​of the piecewise input matrix; and Using the one or more products, a piecewise output vector is generated, which is the product of applying the piecewise input vector to the piecewise input matrix.

2. The system of claim 1, wherein the piecewise input matrix has a dimension equal to or less than a predetermined dimension threshold corresponding to the maximum matrix input size.

3. The system according to claim 1, wherein, The sparse partition is one of the plurality of sparse partitions, which are configured as follows: Receive multiple sliced ​​input matrices that serve as submatrices of the system input matrix; Receives multiple sliced ​​input vectors as subvectors of the system input vector; and Generate a system output vector, which represents the product of applying the system input vector to the system input matrix.

4. The system according to claim 3, wherein, In generating the system output vector, the plurality of sparse slices are further configured to concatenate the corresponding slice output vectors to generate the system output vector.

5. The system according to claim 1, wherein, The multiplier circuit is coupled to a register, which includes the corresponding non-zero value.

6. The system according to claim 1, wherein, The predetermined non-zero threshold is the maximum non-zero threshold.

7. The system according to claim 1, wherein, The sparse sharding further includes a cross-sectional switching circuit, which is configured to: Receive the plurality of vector values ​​of the sliced ​​input vector; and Send a vector value from the plurality of vector values ​​according to one or more control values.

8. The system according to claim 1, wherein, The sparse slicing is further configured to load non-zero values ​​of the same column in the slicing input matrix into the registers of adjacent multiplier circuits in the plurality of multiplier circuits.

9. The system according to claim 8, wherein, The sparse sharding is further configured to receive one or more control values ​​that specify the positions of non-zero values ​​along the columns of the sharding input matrix.

10. The system according to claim 1, wherein, The sparse partitioning further includes a plurality of adder circuits configured to generate one or more sums of the one or more products of the vector values.

11. The system according to claim 10, wherein, The multiple adder circuits form a parallel segmented summation circuit.

12. The system according to claim 10, wherein, The sparse sharding further includes a cross-sectional switching circuit, which is configured to: Receive one or more of the above and, The one or more sums are arranged according to one or more control values ​​to generate the fragmented output vector.

13. The system according to claim 12, wherein, The cross-sectional switching circuit forms a Bene )network.

14. A method comprising: A sparsely distributed input matrix is ​​received by a plurality of multiplier circuits, the sparsely distributed input matrix including a plurality of non-zero values ​​equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits in the plurality of multiplier circuits, wherein the multiplier circuits receive the corresponding non-zero values ​​of the sparsely distributed input matrix. The sparse partition receives a partitioned input vector, which includes multiple vector values. One or more products are generated by multiplying the vector values ​​generated by the plurality of multiplier circuits with the corresponding non-zero values ​​of the piecewise input matrix; as well as The sparse partitions are used with one or more products to generate a partitioned output vector, which is the product of applying the partitioned input vector to the partitioned input matrix.

15. The method according to claim 14, wherein, The piecewise input matrix has a dimension equal to or less than a predetermined dimension threshold corresponding to the maximum matrix input size.

16. The method of claim 14, further comprising: The plurality of sparse partitions, one of which is the sparse partition, receive a plurality of partitioned input matrices as submatrices of the system input matrix; The multiple sparse slices receive multiple slice input vectors as subvectors of the system input vector; as well as The system output vector is generated from the plurality of sparse slices, and the system output vector represents the product of applying the system input vector to the system input vector.

17. The method according to claim 16, wherein, Generating the system output vector further includes concatenating the corresponding sparse output vectors of the plurality of sparse fragments to generate the system output vector.

18. The method according to claim 14, wherein, The multiplier circuit is coupled to a register, which includes the corresponding non-zero value.

19. The method of claim 14, wherein, The predetermined non-zero threshold is the maximum non-zero threshold.

20. One or more non-transitory computer-readable storage media storing instructions that, when executed by a system comprising sparse fragments including a plurality of multiplier circuits, cause the system to perform operations including: A sliced ​​input matrix is ​​received, the sliced ​​input matrix including a number of non-zero values ​​equal to or less than a predetermined non-zero threshold, the non-zero threshold corresponding to the number of multiplier circuits in the plurality of multiplier circuits, wherein the multiplier circuits receive the corresponding non-zero values ​​of the sliced ​​input matrix. Receive a fragmented input vector, wherein the fragmented input vector includes multiple vector values; One or more products are generated by multiplying the vector values ​​generated by the plurality of multiplier circuits with the corresponding non-zero values ​​of the piecewise input matrix; as well as Using the one or more products, a piecewise output vector is generated, which is the product of applying the piecewise input vector to the piecewise input matrix.

Images