Accelerator for sparse matrix multiplication in neural networks

Abstract

The present application describes an accelerator, a computer system and a method for tensor product calculation for facilitating circuit design. The method can include: dividing a weight tensor into a block matrix; rearranging the block matrix in the weight tensor to obtain a rearranged weight tensor; calculating a bit mask, the bit mask including a bit matrix, bits in the bit matrix corresponding to elements in the rearranged weight tensor; removing zero elements in the rearranged weight tensor and packing non-zero elements in the rearranged weight tensor; generating a compact activation tensor based on the bit mask and an activation tensor; and performing tensor multiplication based on the compact weight tensor and the compact activation tensor. Wherein, the rearrangement step effectively reduces the fan-out between the activation and the corresponding weight, and reducing the fan-out can reduce the wiring length, thereby reducing the energy consumption of signal transmission.

Description

Technical Field

[0001] This invention generally relates to improving the storage and computation efficiency of neural network applications involving sparse matrix multiplication. Background Technology

[0002] Neural networks (NNs) form the foundation of many modern artificial intelligence (AI) applications, such as image and video recognition, recommender systems, image classification, medical image analysis, and natural language processing. NNs can also be applied to various scenarios, including autonomous driving, cancer detection, and running complex games. A typical neural network may consist of a series of convolutional layers performing convolution operations, which are computationally intensive and energy-intensive. A typical convolutional layer may involve matrix computations on one or more activation (or input) tensors and one or more weight tensors.

[0003] In recent years, various methods have been developed to improve the computational efficiency of neural networks. These methods achieve computational efficiency improvements by introducing sparsity into the convolution process of neural networks. In some cases, pruning the weight tensor reduces the number of non-zero weights, thereby reducing the total computational cost with negligible precision loss, and thus improving the computational efficiency of the neural network. Based on the distribution pattern of zero elements in the pruned tensor, various existing neural network accelerators can efficiently store and access pruned tensors. However, to ensure the design efficiency of neural network accelerator circuits, current technologies only support a very limited number of zero element distribution patterns. To address this issue, this invention discloses a novel, easily hardware-implementable sparse accelerator that supports a wider variety of sparse patterns in sparse matrix computation. Summary of the Invention

[0004] Various embodiments of this specification may include hardware circuits, systems, and methods for performing efficient sparse matrix multiplication using designs that are easy to implement in hardware.

[0005] In some aspects, the techniques described herein relate to a computer-implemented method comprising: receiving a weight tensor and an activation tensor at a layer of a neural network; partitioning the weight tensor into a block matrix; rearranging the block matrices in the weight tensor to obtain a rearranged weight tensor; computing a bitmask comprising a bit matrix in which bits correspond to elements in the rearranged weight tensor, wherein each bit includes a bit value indicating whether the corresponding element in the rearranged weight tensor is a zero element or a non-zero element; removing zero elements from the rearranged weight tensor and packing the non-zero elements in the rearranged weight tensor to obtain a compact weight tensor; generating a compact activation tensor based on the bitmask and the activation tensor; and performing tensor multiplication based on the compact weight tensor and the compact activation tensor to generate the output tensor of that layer in the neural network.

[0006] In some respects, the techniques described herein relate to a computer-implemented method that also includes pruning the weight tensor such that the number of non-zero elements in every j consecutive elements is in the range a to b, where j, a, and b are integers, and a <= b <= j.

[0007] In some respects, the techniques described in this paper relate to a computer-implemented method where j equals the number of rows within a block.

[0008] In some respects, the technique described herein relates to a computer-implemented method in which computing a bitmask comprising a bit matrix corresponding to elements in a weight tensor includes: computing a bitmask based on a pruned weight tensor, wherein each element in the pruned weight tensor corresponds to a bit in the bitmask.

[0009] In some respects, the techniques described herein relate to a computer-implemented method in which a compact version of the stored bitmask and weight tensor occupies less memory space compared to storing the weight tensor.

[0010] In some respects, the techniques described herein relate to a computer-implemented method in which each block in a block matrix has the same rectangular or square shape.

[0011] In some respects, the techniques described herein relate to a computer-implemented method in which rearranging a block matrix includes: keeping the first column of the block matrix unchanged; rotating the second column of the block matrix along the column direction by the size of a block row; and rotating the third column of the block matrix along the column direction by twice the size of a block row.

[0012] In some respects, the techniques described herein relate to a computer-implemented method in which a bitmask comprises a bit block matrix, wherein the bit blocks in the bit block matrix correspond to blocks in the block matrix of the weight tensor.

[0013] In some respects, the techniques described herein relate to a computer-implemented method in which generating a compact activation tensor based on a bitmask and an activation tensor includes: dividing a first row of activations in the activation tensor into multiple segments of activations, wherein the multiple segments of activations correspond to bit blocks in the first row of bit blocks in a bit block matrix; and copying one or more activations from the multiple segments of activations based on the non-zero bits in the first row of bit blocks to generate a first set of compact one or more rows of activations.

[0014] In some respects, the technique described herein relates to a computer-implemented method in which copying one or more activations of a multi-segment activation based on non-zero bits in a first row bit block includes: for one activation of the multi-segment activation, identifying the position of a non-zero bit in each column of the corresponding bit block in the first row bit block; copying the activation corresponding to the non-zero bit in each column of the corresponding bit block; and compressing the copied activation into a first set of compact one-line or multi-line activations.

[0015] In some respects, the techniques described herein relate to a computer-implemented method that further includes: rotating a segment of activations in a first row of activations by a segment size; copying the rotated segment of activations based on non-zero bits in a second row of bit blocks to generate a second set of compact one- or multiple-row activations; and compressing the first set of compact one- or multiple-row activations and the second set of compact one- or multiple-row activations into an activation cache for multiplication with a compact weight tensor.

[0016] In some respects, the technique described herein relates to a computer-implemented method in which a weight tensor comprises j columns, and a partitioning from the weight tensor to a block matrix divides the j columns into k segments, where k and j are integers, j is divisible by k, and 1 < k < j.

[0017] In some respects, the technique described herein relates to a computer-implemented method in which rearranging the block matrix in the weight tensor reduces the fan-out of each activation by a factor of j / k, thereby reducing power consumption and signal loss in the circuitry used to multiply the weight tensor by the activation tensor.

[0018] In some aspects, the techniques described herein relate to a hardware accelerator for improving the computational efficiency of multiplying weight tensors and activation tensors in a layer of a neural network. The hardware accelerator includes: a weight tensor compression circuit configured to partition a weight tensor into a block matrix, rearrange the block matrices in the weight tensor to obtain a rearranged weight tensor, remove zero-value elements from the rearranged weight tensor, and pack the non-zero elements in the rearranged weight tensor to obtain a compact weight tensor; a bitmask generation circuit configured to compute a bitmask including a bit matrix, the bits of which correspond to elements in the rearranged weight tensor, wherein each bit includes a bit value indicating whether the corresponding element in the rearranged weight tensor is a zero element or a non-zero element; an activation tensor compression circuit configured to generate a compact activation tensor based on the bitmask and the activation tensor; and a computation circuit configured to perform tensor multiplication based on the compact weight tensor and the compact activation tensor to generate the output tensor of that layer in the neural network.

[0019] In some respects, the technique described herein relates to a hardware accelerator in which weight tensors are pruned such that the number of non-zero elements in every j consecutive elements is in the range a to b, where j, a, and b are integers, a <= b <= j, and the j consecutive elements come from the same column of a block.

[0020] In some respects, the techniques described herein relate to a hardware accelerator in which, in order to rearrange the block matrix in the weight tensor, the weight tensor compression circuit is also configured to: keep the first column of the block matrix unchanged; rotate the second column of the block matrix along the column direction by the size of one row of the block; and rotate the third column of the block matrix along the column direction by twice the size of one row of the block.

[0021] In some aspects, the techniques described herein relate to a hardware accelerator in which a bitmask comprises a bit block matrix, the bit blocks in which each bit block corresponds to a block in a block matrix of a weight tensor. To generate a compact activation tensor based on the bitmask and the activation tensor, the activation tensor compression circuit is further configured to: divide the first row of activations in the activation tensor into multiple segments, each segment corresponding to a bit block in the first row of the bit block matrix; and copy the multiple segments of activations based on the non-zero bits in the first row of the bit blocks to generate a first set of compact one or more rows of activations. The copying includes: for the i-th activation in one segment of the multiple segments of activation, copying the activation based on the non-zero bits in the i-th row of the bit block corresponding to that segment of activation.

[0022] In some respects, the techniques described herein relate to a hardware accelerator in which, in order to generate a compact activation tensor based on a bitmask and an activation tensor, an activation tensor compression circuit is further configured to: rotate multiple activation segments in a first row of activations by a segment size; and copy the rotated multiple activation segments based on non-zero bits in a second row of bit blocks to generate a second set of compact one or more rows of activations.

[0023] In some respects, the techniques described herein relate to a hardware accelerator in which, in order to generate a compact activation tensor based on a bitmask and an activation tensor, an activation tensor compression circuit is further configured to compress a first set of compact one-line or more-line activations and a second set of compact one-line or more-line activations into an activation buffer for multiplication with a compact weight tensor.

[0024] These and other features of the systems, methods, and hardware devices disclosed herein, the methods of operation and functions of the associated structural elements, and the economic efficiency of the combination and manufacture of components will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings. The drawings form part of this specification, wherein the same reference numerals denote corresponding components in the drawings. However, it should be understood that the drawings are for illustration and description only and are not intended to limit the invention. Attached Figure Description

[0025] Figure 1 An exemplary schematic diagram of a hardware environment according to some embodiments is shown, which is used to implement tensor products that are easy to implement in hardware.

[0026] Figure 2A An exemplary tensor product calculation in a neural network according to some embodiments is shown.

[0027] Figure 2B A tensor product pipeline that is easy to implement in hardware is shown according to some embodiments.

[0028] Figure 2C A comparison between wiring with a larger fan-out and wiring with a smaller fan-out is shown according to some embodiments.

[0029] Figure 3 An exemplary workflow for preprocessing sparse input tensors is shown according to some embodiments, wherein a tensor product pipeline that is easy to implement in hardware is employed.

[0030] Figure 4 An exemplary tensor product pipeline that is easy to implement in hardware is shown according to some embodiments.

[0031] Figure 5 An exemplary system diagram is shown according to some embodiments for implementing a tensor product pipeline that is easy to implement in hardware.

[0032] Figure 6 An exemplary method for tensor product that is easy to implement in hardware is shown according to some embodiments. Detailed Implementation

[0033] This specification is provided to enable any person skilled in the art to make and use the illustrated embodiments, and is given in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the illustrated embodiments, but is accorded the widest scope consistent with the principles and features disclosed herein.

[0034] In neural network (NN) and deep learning (DL) applications, tensor product computation is one of the most fundamental operations, leveraging the properties of tensors to model related concepts. A typical tensor product involves multiplying the activation tensor by the weight tensor to extract features from the activation tensor. To improve performance, it is common practice to prune tensors and inject sparsity, thereby accelerating tensor product computation without sacrificing significant accuracy.

[0035] Ideally, pruning methods should focus on the semantics and values ​​of elements in a tensor, rather than the distribution of zero elements introduced by pruning. This type of pruning can yield high precision and high sparsity while resulting in a fully irregular sparsity tensor. Unfortunately, in practice, the distribution of zero elements introduced by pruning can directly impact the design of the underlying hardware, such as tensor product accelerators or engines. For example, existing circuit designs are inefficient at storing and accessing non-zero elements in tensors with a fully irregular sparsity distribution (e.g., requiring complex logic to address completely randomly distributed non-zero elements, causing computational elements (PEs) to remain idle during computation and reducing utilization), thus offering little to no acceleration of the computation process. Therefore, existing pruning methods often restrict tensor pruning patterns to block sparsity or N:M sparsity patterns.

[0036] In block sparsity mode, tensors can be divided into blocks / regions, where some blocks / regions are completely pruned to zero elements, but others are not pruned. This coarse-grained pruning method can lead to low sparsity and low computational precision (e.g., if 80% of the elements are pruned, the remaining 20% ​​will be restricted to certain specific blocks). In N:M sparsity mode, every M elements within N elements (where N and M are integers, and N>=M) are retained, and all other elements (NM elements) are pruned to zero. However, N:M sparsity mode algorithms lack flexibility. For example, for feature extraction, some regions within the tensor may not be semantically important, but the algorithm will still retain M non-zero elements in these regions, which can lead to low sparsity and low precision.

[0037] Figure 1 An exemplary schematic diagram of a hardware environment according to some embodiments is shown, which is used to implement tensor products that are easy to implement in hardware.

[0038] like Figure 1 As shown, the hardware environment includes a memory pool 210, processing circuitry 220, and tensor product acceleration circuitry 230. It should be understood that... Figure 1 The component layout in the hardware environment is shown for illustrative purposes, but this layout can be implemented differently depending on the actual hardware configuration. In some embodiments, the tensor product acceleration circuit 230 can be implemented as a separate hardware accelerator, independent of the processing circuitry 220 (e.g., one or more CPUs or GPUs). In some embodiments, the tensor product acceleration circuit 230 can be implemented as part of the processing circuitry 220 (e.g., part of one or more CPUs or GPUs) to improve memory management efficiency. Memory pool 210 can refer to external storage devices, system RAM, other types of memory resources, or any combination of the foregoing.

[0039] In some embodiments, the processing circuitry 220 may include one or more processors 222 and a cache 221 shared by the one or more processors 222. Each processor 222 may include an instruction fetch unit (IFU) 223, an instruction decode unit (IDU) 224, an instruction transfer unit (ITU) 225, and an instruction execution unit (IEU) 226.

[0040] In some embodiments, IFU 223 can fetch instructions or data to be executed from memory pool 210 to register set 229. In some embodiments, the instructions or data to be executed can first be fetched into cache 221 and then sent to IFU 223 via microcontroller unit (MCU) 227. After obtaining the instructions or data, processing circuit 220 enters the instruction decoding stage. IDU 224 decodes the obtained instructions according to a predetermined instruction format to determine operand fetch information, wherein the operands are used to execute the obtained instructions. In some embodiments, operand fetch information may include real-time data, registers, or pointers or addresses of other software / hardware that provide the operands.

[0041] In some embodiments, ITU 225 can be configured to receive decoded instructions from IDU 224 and perform instruction scheduling and management. ITU 225 can efficiently distribute instructions to different IDUs 226 for parallel processing. In some embodiments, after ITU 225 distributes an instruction to an IDU 226, that IDU 226 can execute the instruction.

[0042] In some embodiments, the tensor product acceleration circuit 230 may receive instructions from the processing circuit 220, access data from the memory pool 210, and perform tensor product calculations. The tensor product acceleration circuit 230 may send the tensor product result (e.g., the output tensor) back to the processing circuit 220 to continue the remaining computation. For example, the tensor product result corresponding to one layer of a neural network (NN) may be used as input for the next tensor product calculation of the next layer of the neural network. The tensor product acceleration circuit 230 may be implemented as a hardware accelerator or engine to improve the computational efficiency of tensor products. In this context, the improved efficiency is manifested not only in faster computation speeds but also in smaller memory footprints and a more streamlined hardware implementation for circuit design.

[0043] In some embodiments, the tensor product acceleration circuit 230 may include a weight tensor processing module 232, a bitmask generation module 233, an activation tensor processing module 234, and a calculation module 235. The following description is based on the assumption that tensor product calculation involves weight tensors and activation tensors, wherein the weight tensors are being or have already been sparsely pruned, and the activation tensors may be sparse or non-sparse. This assumption may be adjusted to encompass other cases where the activation tensors are being or have already been sparsely pruned, and the weight tensors may be sparse or non-sparse. In some embodiments, Figure 1 Some of the modules listed can be implemented outside of the tensor product acceleration circuit 230. For example, the weight tensor processing module 232 can be implemented in the processing circuit 220, which means that the weight tensors are rearranged and compressed in the processing circuit 220 before being sent to the tensor product acceleration circuit 230 for tensor product calculation.

[0044] In some embodiments, the weight tensor processing module 232 can be configured to receive weight tensors and activation tensors for tensor product calculation. The weight tensors and activation tensors may originate from a layer of a neural network, and the tensor product calculation may be part of a convolution process. In some cases, the weight tensors and / or activation tensors may be pruned, and the weight tensors and / or activation tensors may include zero-value elements and non-zero-value elements. The weight tensor processing module 232 can also be configured to divide the weight tensor into a block matrix (e.g., blocks having the same rectangular or square shape) and rearrange the block matrix in the weight tensor to obtain a rearranged weight tensor. Rearranging the block matrix may include: keeping the first column of the block matrix unchanged; rotating the second column of the block matrix along the column direction by one row size of the block; and rotating the third column of the block matrix along the column direction by twice the row size of the block. The purpose of the rearrangement includes facilitating circuit design by reducing the fan-out of each node (e.g., a register file storing activations) and reducing the total wiring distance. Figure 2B An exemplary rearrangement process is further illustrated.

[0045] In some embodiments, the bitmask generation module 223 can be configured to compute a bitmask using a bit matrix, wherein the bits in the bit matrix correspond to elements in the rearranged weight tensor. Each bit in the bitmask includes a bit value that indicates whether the corresponding element in the rearranged weight tensor is a zero element or a non-zero element.

[0046] In some embodiments, after calculating the bitmask based on the rearranged weight tensor, the weight tensor processing module 232 may also be configured to remove zero elements from the rearranged weight tensor and pack the non-zero elements in the rearranged weight tensor to obtain a compact weight tensor. "Packaging" as used herein may refer to removing any space occupied by zero-value weights. In some embodiments, removing space occupied by zero-value weights may include shifting the remaining weights in the same column up by one space. Figure 3 The schematic process of packing weight tensors is illustrated.

[0047] Bitmasks provide information about the position of non-zero elements within the rearranged weight tensor, while compact weight tensors provide the actual non-zero values, eliminating the need to store any zero-value elements or their corresponding indices. Therefore, bitmasks and compact weight tensors can be used together to reconstruct all the information of the original weight tensor with a smaller memory footprint. In many cases, storing bitmasks and compact versions of the weight tensor requires less memory than storing the weight tensor itself. For example, each element (zero or non-zero) in the original weight tensor might occupy 32 bits (e.g., 4 bytes for integers), while each element in the bitmask might occupy 1 bit. Assuming the original weight tensor contains X elements with x non-zero elements, storing the original weight tensor would require 32X bits, while storing the bitmask and compact weight tensor would only require X+32x bits. This means that storing bitmasks and compact weight tensors can save memory space as long as the sparsity of the pruned weight tensor is greater than 5%.

[0048] In some embodiments, the activation tensor processing module 234 can be configured to generate a compact activation tensor based on a bitmask and the original activation tensor. It is important to note that generating a compact activation tensor differs from generating a compact weight tensor. A compact weight tensor is generated by removing zero-value elements and packing (e.g., compressing) non-zero-value elements into a compact form; however, a compact activation tensor is generated based on non-zero bits in the bitmask. For example, if an activation corresponds to a zero-value bit in the bitmask, that activation is removed; if an activation corresponds to a non-zero bit in the bitmask, that activation is retained. Using this approach, activations removed from the activation tensor may include non-zero-value activations, while activations retained in the compact activation tensor may also include zero-value activations. In other words, the removal process here only retains those activations that need to be multiplied by non-zero weights in the weight tensor, while discarding those activations corresponding to zero-value weights in the weight tensor. After the removal process, the activations can be compressed or packed into a compact activation tensor. In this article, "packaging" can refer to shifting the remaining active elements in the same column up by the size of one element after removing the active element. Figure 4 The schematic process of packing activation tensors is shown.

[0049] In some embodiments, computation module 235 can be configured to perform tensor multiplication and addition (MAC operations) based on compact weight tensors and compact activation tensors to generate the output tensor of that layer in the neural network. Since both operators (compact weight tensors and compact activation tensors) are in compact format, the output tensor can also be in compact format. In some embodiments, if it is necessary to decompress the compact output tensor, a bitmask can be used to insert zero-value elements to reconstruct the uncompressed output tensor. In some embodiments, the compact output tensor, along with the bitmask, can be input into the next layer of the neural network for the next round of computation.

[0050] Figure 2A An exemplary tensor product computation in a neural network according to some embodiments is illustrated. For ease of description, Figure 2A The diagram in the image decomposes matrix-matrix multiplication between two matrices into simple vector-matrix multiplication. For example... Figure 2A As shown, vector (1*i) represents activation segment 200, which can represent a row of activations or a portion of a row of activations in the activation tensor. Matrix (i*j) represents weight tensor 210. Known matrix multiplication involves multiplying activation segment 200 by each column of weights in weight tensor 210. Specifically, this multiplication involves multiplying each activation in activation segment 200 by the corresponding weight in weight tensor 210 and summing all products to generate an element of the output vector. This means that each activation in activation segment 200 will be multiplied by j weights in the corresponding row of weight tensor 210. For example, the first activation vector [0] will be multiplied by all weights in the first row of weight tensor 210. Thus, the fan-out of the first activation (and any other activations) in the activation segment is j. In circuit design, to minimize total wiring distance and power consumption (e.g., power consumption due to wiring distance), the register file storing the first activation can be arranged in the middle of multiple register files storing the weights of the first row. However, in this design, the distance between the first activation and the first weight (or the last weight) in the first row is much longer than the distance between the first activation and the weights in the middle of the row. Transmitting signals over a longer distance consumes more power and may cause various timing problems (e.g., requiring more complex clock synchronization techniques). Therefore, Figure 2B An improved method is shown to reduce fan-out, thereby reducing power consumption and the likelihood of various timing problems or even signal loss.

[0051] Figure 2B A tensor product pipeline that is easy to implement in hardware is illustrated according to some embodiments. It should be noted that... Figure 2B The design shown includes tiling the tensor into multiple blocks and rearranging these blocks to facilitate hardware design. Figure 2BThe pipeline shown can be applied to tensors with or without sparsity. For simplicity, [the following is an example of a pipeline]. Figure 2B The description ignores sparsity within the tensor, focusing instead on tiling and rearrangement processes. For example... Figure 3 and 4 As shown, this design can be applied to use cases involving at least sparse weight tensors.

[0052] In some embodiments, one of the two tensors involved in the tensor product calculation can be divided into blocks. For example... Figure 2B As shown in the example above, the weight tensor 230 can be divided into multiple blocks 252. Here, the weight tensor 230 is a 12x12 matrix, and each block 252 is a 4x4 block; therefore, the weight tensor 230 is divided into a matrix with 3x3 blocks 252. Next, the matrix of blocks 252 can be rearranged, thereby reducing the fan-out of each activation in the activation segment 220 during tensor product computation. From the above... Figure 2A As described above, without rearrangement, each activation in activation segment 220 can be copied to all weights in the corresponding row (e.g., using an amplifier for copying) for multiplication and addition, in which case the fanout of each activation is equal to the row size of weight tensor 230 (12 in this example).

[0053] In some embodiments, rearranging block 252 may include: keeping the first column of the block matrix unchanged; rotating the second column of the block matrix along the column direction by the size of one row of the block; and rotating the third column of the block matrix along the column direction by twice the size of one row of the block, and so on. Figure 2B The diagram illustrates the rearrangement, where weight tensor 230 is transformed into the rearranged weight tensor 250. After the rearrangement, the first activation (A0, where "A" represents activation) in activation segment 240 may only need to be copied to the four weights in the first row of the first block (w[0,0], w[0,1], w[0,2], and w[0,3], where "w" represents weight, the first index is the row index, and the second index is the column index); similarly, activation A1 needs to be copied to weights w[1,0]-w[1,3], activation A2 needs to be copied to weights w[2,0]-w[2,3], and activation A3 needs to be copied to weights w[3,0]-w[3,3]; and the fifth activation in activation segment 240 may only need to be copied to the four weights in the first row of the second block. Therefore, the fanout of each activation in activation segment 240 is reduced to the size of one row of the block (4 in this example), rather than the size of one row of weight tensor 250.

[0054] Because of the small fan-out, the register file storing the active A0 can be placed in the middle of multiple register files storing w[0,0]-w[0,3], the register file storing the active A4 can be placed in the middle of multiple register files storing w[0,4]-w[0,7], and the register file storing the active A8 can be placed in the middle of multiple register files storing w[0,8]-w[0,11]. See [link to relevant documentation]. Figure 2C The example shown is wiring 273 with a smaller fanout. In contrast, for a larger fanout (i.e., no rearrangement), the register file storing A0 needs to be placed in the middle of multiple register files storing w[0,0]-w[0,11], where the distance between A0 and the weights at both ends is much greater than the distance between A0 and the middle weights. See [reference needed]. Figure 2C The example shown is a wiring 272 with a large fanout. For example... Figure 2C As shown, in routing 273 with a smaller fanout, the average distance between the activations and weights is smaller compared to routing 272 with a larger fanout. Therefore, routing with a smaller fanout effectively reduces the total routing distance between the activations and their corresponding weights. Since transmitting signals over longer wires consumes more power and results in higher delays, routing with a smaller fanout saves overall power consumption and reduces the likelihood of various timing problems or even signal loss.

[0055] In some embodiments, after the activation segment 240 is multiplied by the first row block in the rearranged weight tensor 250, the activation segment 240 can be rearranged before being multiplied by the next row block. For example... Figure 2B As shown, the activation segment 240 can be divided into multiple segments based on the row size of the blocks in the rearranged weight tensor 250, and these segments can be rotated by the size of one segment in one direction to obtain the rotated activation segment 260. For the purpose of tensor computation, the segments in the rotated activation segment 260 correspond to the blocks in the next row.

[0056] Figure 3 An exemplary workflow for preprocessing sparse input tensors is illustrated according to some embodiments, wherein preprocessing is performed using a tensor product pipeline that is easy to implement in hardware. In some embodiments, "preprocessing" means that it can be implemented outside of a tensor product computation accelerator. Figure 3 The workflow in this context refers to the process implemented before the actual tensor computations between tensors are performed. In other embodiments, the "preprocessing" can be implemented as part of a tensor product computation accelerator.

[0057] like Figure 3As shown, after receiving a sparse tensor (or matrix), a tiling operation can be performed to divide the sparse tensor into a block matrix, where the blocks can have a rectangular or square shape. As an example, the sparse tensor is a weight tensor from a neural network layer. The weight tensor can have j columns, and dividing the weight tensor into a block matrix involves dividing that j columns into k segments (each segment corresponding to a block), where k and j are integers, j is divisible by k, and 1 < k < j.

[0058] In some embodiments, the block matrix in the weight tensor can be rearranged to reduce the fan-out of each activation in the activation tensor by a factor of j / k, thereby reducing power consumption and signal loss in the circuitry used to multiply the weight tensor and activation tensor. Figure 2B The exemplary rearrangement process is explained in detail.

[0059] After rearrangement, the weight tensors can be compressed / packed. Compression consists of two stages: (1) compression within each row block, and (2) compression across different row blocks. In the compression stage within each row block, each row block is locally compressed. For example, in the first row block, all zero-value weights are removed, and all remaining non-zero weights are compressed in the column direction so that there is no space between any two non-zero weights in the same column of this row block. In the compression stage across different row blocks, the locally compressed blocks are further compressed in the column direction so that there is no space between any two non-zero weights in the same column. The result of compression is a densely packed weight tensor.

[0060] Densely packed weight tensors contain all non-zero weights, and the location information of these non-zero weights is necessary for correct computation. Therefore, a bitmask can be constructed based on the rearranged weight tensor. The bitmask can be the same size as the rearranged weight tensor and includes multiple bits, each corresponding to one of the weights (zero or non-zero) in the rearranged weight tensor. Each bit can use a binary value to represent whether the corresponding weight is zero or non-zero. Since each element in the bitmask occupies only one bit of memory, the bitmask is lightweight.

[0061] In some embodiments, such as in Figure 3 At the beginning of the workflow shown, the received tensors can be pruned to zero out unnecessary features (e.g., weights less than a threshold can be pruned). The pruning method can be designed to facilitate workflow execution; for example, the received tensors can be pruned so that the resulting densely packed matrix has a rectangular shape. For instance, pruning can include pruning the weight tensors so that, after pruning, the number of non-zero elements in every j consecutive elements in the same column is in the range a to b, where j, a, and b are integers, a <= b <= j, and j equals the number of rows in a block. Figure 3 As shown, each block is a 4x4 square (4 rows), and pruning of the input tensor results in columns of four elements having 1 to 2 non-zero elements. In this case, j=4, a=1, and b=2.

[0062] Figure 4 A tensor product pipeline that is easy to implement in hardware is shown according to some embodiments. Figure 4 The process shown is based on the assumption that the weight tensor has been divided into blocks, rearranged based on the blocks, and compressed into a packaged weight tensor 460, and that a bitmask 410 has been calculated (based on the uncompressed and rearranged weight tensors).

[0063] As described above, bitmask 410 includes a bit matrix, where the bits in the bit matrix correspond to elements in the partitioned and rearranged weight tensor, and bitmask 410 can be used to generate a compact activation tensor based on the original activation tensor. In some embodiments, compact multi-row activations can be constructed based on bitmask 410 using each row of activations in the original activation tensor (stored in activation register 430). The bitmask can be viewed as a bit block matrix, where each bit block corresponds to a block matrix of the (partitioned and rearranged) weight tensor, and each row of bit blocks can be used as a wizard to select activations from a row of activations. The selected activations can form a compressed activation cluster 440. For example, a compact activation tensor can be constructed by: partitioning the first row of activations in the activation tensor into multiple segments, where each segment corresponds to a bit block in the first row of bit blocks in the bit block matrix; and copying the multiple segments of activations based on the non-zero bits in the first row of bit blocks to generate a first set of compact one-row or multi-row activations. The copying process may include: for one of the multiple activations, identifying the position of the non-zero bit in each column of the corresponding bit block; copying the activation based on the non-zero bit in each column of the corresponding bit block; and compressing the copied activation into a first set of compact one or more rows of activation.

[0064] by Figure 4For example, a row of bit blocks from bitmask 410 consists of eight 4*4 bit blocks, and a row of activations is divided into multiple activation segments, each segment containing four activations. The first segment 422 corresponds to the first bit block 412. For the first segment 422, the first column of the first bit block 412 is checked, where the first and third bits are non-zero. Based on the non-zero bits in the first column of the first bit block 412, the first and third activations in the first segment 422 are copied and compressed into the first column 442 of the compressed activation cluster 440. Next, the second column of the first bit block 412 is checked, where the second and fourth bits are non-zero. Subsequently, the corresponding second and fourth activations from the first segment are copied and compressed into the second column of the compressed activation cluster 440. This process continues until all columns of the first bit block 412 have been checked. Afterward, the process proceeds to the next segment of activation in the activation register 430.

[0065] In some embodiments, after processing the first row of activations stored in the activation register 430 (for constructing the first few rows of the compressed activation cluster 440 based on the bitmask 410), the activation rearranger 420 can rearrange the first row of activations to match the next row of bit blocks and generate the next few rows of the compressed activation cluster 440.

[0066] Subsequently, the compressed activation clusters 440 generated through multiple iterations can be compressed into a packed activation tensor 450. This packed activation tensor 450 includes all activations corresponding to the non-zero bits in the bitmask 410, where activations can include both zero and non-zero activations. The packed activation tensor 450 and the packed weight tensor 460 can then be sent to a multiplication-accumulation (MAC) gate for computation. The MAC gate can generate an output tensor, which can be stored in an output accumulation buffer 470 for subsequent computation.

[0067] Figure 5 An exemplary system diagram 500 is shown according to some embodiments for implementing a tensor product pipeline that is easy to implement in hardware. The components in system diagram 500 are for illustrative purposes only. Depending on the implementation, system diagram 500 may include more, fewer, or alternative components. In some embodiments, system diagram 500 may refer to a pipeline for performing tensor product calculations between weight tensors and activation tensors (e.g., in a layer of a neural network). This pipeline transforms the weight and activation tensors to reduce fan-out in circuit design. In the field of integrated circuit (IC) design, "fan-out" can refer to the maximum number of digital inputs that the output of a single logic gate can feed / drive. Reducing fan-out helps reduce the overall power consumption of the circuit, reduce signal propagation delay, and reduce the probability of signal timing problems or even signal loss problems.

[0068] In some embodiments, the system diagram 500 may include a weight input module 510, which is configured to receive a weight tensor. In some embodiments, the weight tensor may be a sparse tensor that has undergone pruning. In some embodiments, pruning may be implemented by the weight input module 510. The pruning process may include: dividing the weight tensor into block matrices, and pruning the weight tensor such that the number of non-zero elements in every j consecutive elements is within the range of a to b, where j, a, and b are integers, a < b <= j, and j is equal to the number of rows within a block. Compared with the pruning method in the existing N:M sparsity pattern, this pruning method provides greater flexibility. In particular, the pruning method in the existing N:M sparsity pattern requires retaining M non-zero elements within every N elements, regardless of whether these N elements are in a necessary region or a non-necessary region within the weight tensor. Herein, "necessary" and "non-necessary" refer to whether the corresponding feature is important (e.g., for object detection, edge / corner features are important / necessary). In contrast, the pruning method provided by the present invention allows for pruning blocks in the non-necessary region with a larger margin, i.e., retaining a smaller number of non-zero elements (e.g., using the lower limit a of the above range), and pruning blocks in the necessary region within the weight tensor with a smaller margin, i.e., retaining a larger number of non-zero elements (e.g., using the upper limit b of the above range). In some embodiments, the weight tensor, as well as the relevant information of the necessary or non-necessary region corresponding to the weight tensor, may be received by the weight input module 510 or other modules in the system diagram 500.

[0069] In some embodiments, the pruned weight tensor may be rearranged in the weight rearrangement module 512. The rearrangement may include: rearranging blocks in different columns within the weight tensor at different distances. For example, after dividing the weight tensor into a block matrix of weights, the rearrangement may include: keeping the first column of the block matrix unchanged; rotating the second column of the block matrix by the size of one row of the block in the column direction; and rotating the third column of the block matrix by twice the size of one row of the block in the column direction; continuing this process until all columns (except the first column) of the block matrix have been rearranged. The rearrangement helps reduce the fan-out between each activation and the corresponding weight. For more details, reference can be made to Figure 2B .

[0070] After pruning and rearranging the weight tensor, a bitmask may be generated by the weight bitmask module 520. The bitmask may include a bit matrix, where the bits in the bit matrix correspond to the elements in the rearranged weight tensor, and each bit includes a bit value indicating whether the corresponding element in the rearranged weight tensor is zero or a non-zero element. In some embodiments, the bits in the bitmask may be further divided into bit blocks, which respectively correspond to the blocks within the rearranged weight tensor. Thereafter, the bit blocks may be used to construct a packed activation tensor.

[0071] The weight removal module 514 can remove zero-value elements from the rearranged weight tensor. Removal can be performed iteratively across all rows of the weight tensor. After each zero-value element is removed, the remaining elements in the same column of the weight tensor (below the removed element) are shifted up to occupy the space previously occupied by the removed element.

[0072] After removing all zero-value elements from the rearranged weight tensor, the weight packing module 516 obtains a packed weight tensor (i.e., a compressed version of the rearranged weight tensor), which retains the non-zero elements from the rearranged weight tensor. If the rearranged weight tensor is sparse, storing the bitmask and the packed weight tensor will occupy less memory space compared to storing the rearranged weight tensor.

[0073] While processing the weight tensor via modules 510, 512, 514, 516, and 520, the activation tensor can also be processed in parallel. For example, activation input module 530 can obtain the activation tensor (the method / system described herein does not require the activation tensor to be sparse). At activation selection module 532, the activation tensor can be compressed based on a bitmask, where the bitmask stores the position information of non-zero weights in the rearranged weight tensor, and the weight tensor is rearranged based on weight blocks. At activation selection module 532, compression of the activation tensor may include: dividing the first row of activations in the activation tensor into multiple activation segments, where each segment corresponds to a bit block in the first row of the bit block matrix; and copying the multiple activation segments based on the non-zero bits in the first row of the bit blocks to generate a first set of compact one or more rows of activations. In some embodiments, copying multi-segment activations based on non-zero bits in the first row bit block includes: for one segment of activations, identifying the position of a non-zero bit in each column of the corresponding bit block in the first row bit block; copying the activation based on the non-zero bits in each column of the corresponding bit block; and compressing the copied activations into a first set of compact one-line or multi-line activations.

[0074] In some embodiments, before generating more compact multi-row activations using the second row bit block and the first row activations, the activation rearrangement module 540 can rotate / rearrange a segment of the first row activations. After rotation, the rotated multi-segment activations can be copied based on the non-zero bits in the second row bit block, thereby generating more compact multi-row activations. Subsequently, the first set of compact one-row or multi-row activations and the second set of compact one-row or multi-row activations can be compressed into the activation cache 534 as a packed activation tensor.

[0075] After the packed activation tensor and the packed weight tensor are ready, they can be fed into the multiply-accumulate module 536 to perform calculations (e.g., multiplication and addition) to generate an output tensor, which will be provided to the output module 542. In some embodiments, the output tensor can be used as the input to the next layer of the neural network.

[0076] Figure 6 An exemplary method 600 for tensor product that is easy to implement in hardware according to some embodiments is shown. This method 600 can be implemented Figure 1 in the environment shown and can be performed by Figure 1-5 a device, apparatus or system shown (such as Figure 1 the tensor product acceleration circuit 230 in). Depending on the implementation, method 600 may include more, fewer or alternative steps performed in various orders or in parallel.

[0077] Step 610 in method 600 includes receiving a weight tensor and an activation tensor in a layer of a neural network.

[0078] Step 620 in method 600 includes partitioning the weight tensor into block matrices.

[0079] Step 630 in method 600 includes rearranging the block matrices in the weight tensor to obtain a rearranged weight tensor. In some embodiments, each block in the block matrix has the same rectangular or square shape. In some embodiments, rearranging the block matrix includes: keeping the first column of the block matrix unchanged; rotating the second column of the block matrix by the size of one row of the block in the column direction; and rotating the third column of the block matrix by twice the size of one row of the block in the column direction. In some embodiments, the weight tensor includes j columns, and partitioning the weight tensor into block matrices includes partitioning the j columns into k segments, where k and j are integers, j is divisible by k, and 1 < k < j. In some embodiments, rearranging the block matrices in the weight tensor reduces the fan-out of each activation by a factor of j / k, thereby reducing the power consumption and signal loss of the circuit for multiplying the weight tensor by the activation tensor.

[0080] Step 640 in method 600 includes calculating a bit mask, which includes a bit matrix, where the bits in the bit matrix correspond to the elements in the rearranged weight tensor, and each bit includes a bit value indicating whether the corresponding element in the rearranged weight tensor is a zero or a non-zero element. In some embodiments, the bit mask includes a bit block matrix, where the bit blocks in the bit block matrix respectively correspond to the blocks in the block matrix of the weight tensor.

[0081] Step 650 in method 600 includes removing zero elements from the rearranged weight tensor and packing the non-zero elements in the rearranged weight tensor to obtain a compact weight tensor.

[0082] Step 660 in method 600 includes generating a compact activation tensor based on a bitmask and an activation tensor. In some embodiments, generating a compact activation tensor based on a bitmask and an activation tensor includes: dividing a first row of activations in the activation tensor into multiple segments, wherein each segment corresponds to a bit block in a first row of bit blocks of a bit block matrix; and copying one or more activations from the multiple segments based on non-zero bits in the first row of bit blocks to generate a first set of compact one or more rows of activations. In some embodiments, copying one or more activations from the multiple segments based on non-zero bits in the first row of bit blocks includes: for one segment of activation, identifying a non-zero bit position in each column of the corresponding bit block in the first row of bit blocks; copying the activation corresponding to the non-zero bit in each column of the corresponding bit block; and compressing the copied activations into a first set of compact one or more rows of activations. In some embodiments, method 600 may further include: rotating the multi-segment activations in the first row of activations by a segment size; copying the rotated multi-segment activations based on the non-zero bits in the second row of bit blocks to generate a second set of compact one-line or multi-line activations; and compressing the first set of compact one-line or multi-line activations and the second set of compact one-line or multi-line activations into an activation cache to multiply with a compact weight tensor.

[0083] Step 670 of method 600 includes performing tensor multiplication based on compact weight tensors and compact activation tensors to generate the output tensor of that layer in the neural network.

[0084] In some embodiments, method 600 may further include: pruning the weight tensor such that the number of non-zero elements in every j consecutive elements is in the range a to b, where j, a, and b are integers, a <= b <= j, and j equals the number of rows in a block. In some embodiments, a bitmask is computed based on the pruned weight tensor, wherein each element in the pruned weight tensor corresponds to a bit in the bitmask.

[0085] In some embodiments, a compact version of storing bitmasks and weight tensors will occupy less memory space compared to storing weight tensors.

[0086] Each process, method, and algorithm described in the foregoing sections may be embodied in a code module executed by one or more computer systems or a computer processor including computer hardware, and may be fully or partially automated by the code module. The processes and algorithms may be implemented, partially or entirely, in dedicated circuitry.

[0087] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (in whole or in part) disclosed herein, or aspects that contribute to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium that may include multiple instructions to cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0088] Specific embodiments disclosed herein also provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to the steps in any of the method embodiments disclosed above. Specific embodiments disclosed herein also provide a non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause one or more processors to perform operations corresponding to the steps in any of the method embodiments disclosed above.

[0089] The embodiments disclosed herein can be implemented through a cloud platform, server, or group of servers (collectively referred to as the "service system") that interacts with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that may have the platform application installed.

[0090] The various features and processes described above can be used independently of each other or in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this invention. Some implementations may omit certain method or process blocks. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a different order than specifically disclosed above, or multiple blocks or states may be combined in a single block or state. Additionally, exemplary blocks or states may be executed serially, in parallel, or otherwise, and blocks or states may be added to or removed from the exemplary embodiments disclosed above. The exemplary systems and components described herein may be configured differently from those described; for example, elements may be added, removed, or rearranged compared to the exemplary embodiments disclosed above.

[0091] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. This algorithm may include program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium), and may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly program the computer to perform the function, but may learn from training data to build a predictive model for performing that function.

[0092] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation that performs one or more of the operations or functions described herein.

[0093] Similarly, the methods described herein can be implemented at least in part by a processor, employing one or more specific processors. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by the processor. Furthermore, one or more processors can also support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some operations can be performed by a group of computers (e.g., machines including processors) that can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0094] The execution of certain operations can be distributed across processors, rather than residing within a single machine, but deployed across multiple machines. In some exemplary embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other exemplary embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0095] In this specification, multiple instances can implement the components, operations, or structures described as single instances. Although a single operation of one or more methods is shown and described as a separate operation, one or more of these single operations can be performed simultaneously and not necessarily in the order shown. Structures and functions presented as separate components in the exemplary configuration can also be implemented as composite structures or components. Similarly, structures and functions presented as single components can also be implemented as multiple separate components. It should be understood that all variations, modifications, additions, and improvements mentioned, as well as those not mentioned, fall within the scope of this document.

[0096] Although the subject matter of the invention has been described with reference to specific exemplary embodiments, various modifications and changes can be made to these embodiments without departing from the broad scope of the embodiments of the invention. Embodiments of the invention may be referred to herein individually or collectively by the term "invention," which is merely for convenience and is not intended to voluntarily limit the scope of this application to any single disclosure or concept, should more than one be disclosed in practice.

[0097] The embodiments shown herein have been described in sufficient detail above to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, the detailed description should not be construed as limiting, and the scope of the various embodiments is defined only by the full scope of the appended claims together with the equivalents given by the claims.

[0098] Any process description, element, or block depicted in the flowcharts described herein and / or in the accompanying drawings should be understood as potentially representing a module, segment, or section of code comprising one or more executable instructions for implementing specific logical functions or steps within the process. Those skilled in the art will understand that alternative implementations are included within the scope of the embodiments described herein, wherein, depending on the functionality involved, elements or functions may be omitted, performed out of the order shown or discussed, including elements or functions performed substantially simultaneously or in reverse order.

[0099] As used herein, unless otherwise expressly indicated or the context otherwise indicates, "or" is inclusive rather than exclusive. Therefore, in this document, unless otherwise expressly indicated or the context otherwise indicates, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C." Furthermore, unless otherwise expressly indicated or the context otherwise indicates, "and" is both common and individual. Therefore, in this document, unless otherwise expressly indicated or the context otherwise indicates, "A and B" means "common or separate A and B." Additionally, multiple instances may be provided for a resource, operation, or structure described herein as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are shown in the context of a particular illustrative configuration. The allocation of other functionalities is predictable and may fall within the scope of various embodiments of the invention. Generally, structures and functionalities presented as separate resources in exemplary configurations can be implemented as combined structures or resources. Similarly, structures and functionalities presented as single resources can be implemented as multiple separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the invention as represented by the appended claims. Therefore, the description and drawings are considered illustrative rather than restrictive.

[0100] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, such as “may,” “will,” “may,” or “can,” unless specifically stated otherwise or understood in the context in which they are used, is generally intended to convey that some embodiments include certain features, elements, and / or steps, while other embodiments do not include these features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, elements, and / or steps are included or will be performed in any particular embodiment, with or without user input or prompting.

Claims

1. A computer-implemented method applied to a tensor product acceleration circuit, comprising: Each layer of the neural network receives weight tensors and activation tensors; The weight tensor is divided into block matrices; The block matrix is ​​rearranged to obtain a rearranged weight tensor. The rearranged weight tensor is used to reduce the physical fan-out of the register storing the activation tensor. The register of the activation tensor can be arranged at the position of the corresponding block matrix. Calculate a bitmask, the bitmask comprising a bit matrix, wherein the bits in the bit matrix correspond to the elements in the rearranged weight tensor, wherein the elements in the rearranged weight tensor include one or more zero elements and one or more non-zero elements, and each bit in the bit matrix indicates whether the corresponding element in the rearranged weight tensor is a zero element or a non-zero element. Remove zero elements from the rearranged weight tensor and pack the non-zero elements in the rearranged weight tensor to obtain a compact weight tensor. A compact activation tensor is generated based on the bitmask and the activation tensor, wherein the compact activation tensor is generated by copying and compressing multiple activation segments stored in the register of the activation tensor. The bitmask includes a bit block matrix, and the bit blocks in the bit block matrix correspond to the blocks in the block matrix of the weight tensor. Generating a compact activation tensor based on the bitmask and the activation tensor includes dividing the first row of activations in the activation tensor into multiple activation segments, each of which corresponds to a bit block in the first row of bit blocks in the bit block matrix, and copying one or more activations from the multiple activation segments based on the non-zero bits in the first row of bit blocks to generate a first set of compact one or more rows of activations. as well as Tensor multiplication is performed based on the compact weight tensor and the compact activation tensor to generate the output tensor of the layer of the neural network.

2. The computer-implemented method according to claim 1 further includes: The weight tensor is pruned such that the number of non-zero elements in every j consecutive elements is in the range a to b, where j, a, and b are integers, and a <= b <= j.

3. The computer-implemented method according to claim 2, wherein, j equals the number of rows in a block.

4. The computer-implemented method according to claim 2, wherein, Calculating the bitmask includes: The bitmask is calculated based on the pruned weight tensor, wherein each element in the pruned weight tensor corresponds to a bit in the bitmask.

5. The computer-implemented method according to claim 1, wherein, The compact version of storing the bitmask and weight tensor occupies less memory space compared to storing the weight tensor.

6. The computer-implemented method according to claim 1, wherein, Each block in the block matrix has the same rectangular or square shape.

7. The computer-implemented method according to claim 1, wherein, Rearranging the block matrix includes: Keep the first column of the block matrix unchanged; Rotate the second column of the block matrix by the size of one row of the block along the column direction; and Rotate the third column of the block matrix along the column direction by twice the size of a block row.

8. The computer-implemented method according to claim 1, wherein, Based on the non-zero bits in the first row block, one or more of the activations in the multi-segment activations include: For one of the multiple activations, identify the position of the non-zero bit in each column of the corresponding bit block in the first row bit block; Copy the activation corresponding to the non-zero bit in each column of the corresponding bit block; and Compress the copied activations into a first set of compact one-line or multi-line activations.

9. The computer-implemented method according to claim 8, further comprising: Rotate the multiple active segments in the first row by the size of one segment; Multi-segment activations are generated based on non-zero bit copy rotations in the second row bit block to produce a second set of compact one-line or multi-line activations. as well as The first set of compact one-line or multiple-line activations and the second set of compact one-line or multiple-line activations are compressed into a hardware activation cache for multiplication with the compact weight tensor.

10. The computer-implemented method according to claim 1, wherein, The weight tensor comprises j columns, and dividing the weight tensor into a block matrix involves dividing the j columns into k segments, where k and j are integers, j is divisible by k, and 1 <k<j。 11. The computer-implemented method according to claim 1, wherein, Rearranging the block matrix in the weight tensor reduces the fan-out of each activation, thereby reducing the total wiring distance and power consumption of the circuitry used to multiply the weight tensor by the activation tensor.

12. A hardware accelerator applied to a tensor product acceleration circuit to improve the computational efficiency of multiplying weight tensors and activation tensors in a layer of a neural network, comprising: The weighted tensor compression circuit is configured as follows: The weight tensor is divided into block matrices; The block matrix in the weight tensor is rearranged to obtain the rearranged weight tensor. The rearranged weight tensor is used to reduce the physical fan-out of the register storing the activation tensor. The register of the activation tensor can be arranged at the position of the corresponding block matrix. as well as Remove zero-value elements from the rearranged weight tensor and pack the non-zero elements in the rearranged weight tensor to obtain a compact weight tensor. The bitmask generation circuit is configured as follows: Calculate a bitmask, the bitmask comprising a bit matrix, wherein the bits in the bit matrix correspond to the elements in the rearranged weight tensor, wherein the elements in the rearranged weight tensor include one or more zero elements and one or more non-zero elements, and each bit in the bit matrix indicates whether the corresponding element in the rearranged weight tensor is a zero element or a non-zero element. The tensor compression circuit is activated and configured as follows: A compact activation tensor is generated based on the bitmask and the activation tensor, wherein the compact activation tensor is generated by copying and compressing multiple activation segments stored in the register of the activation tensor. The bitmask includes a bit block matrix, and the bit blocks in the bit block matrix correspond to the blocks in the block matrix of the weight tensor. Generating a compact activation tensor based on the bitmask and the activation tensor includes dividing the first row of activations in the activation tensor into multiple activation segments, each of which corresponds to a bit block in the first row of bit blocks in the bit block matrix, and copying one or more activations from the multiple activation segments based on the non-zero bits in the first row of bit blocks to generate a first set of compact one or more rows of activations. as well as The computing circuit is configured as follows: Tensor multiplication is performed based on the compact weight tensor and the compact activation tensor to generate the output tensor of the layer of the neural network.

13. The hardware accelerator according to claim 12, wherein, The weight tensor is pruned as follows: the number of non-zero elements in every j consecutive elements is in the range a to b, where j, a, and b are integers, a <= b <= j, and j equals the number of rows in a block.

14. The hardware accelerator according to claim 12, wherein, To rearrange the block matrix in the weight tensor, the weight tensor compression circuit is further configured to: Keep the first column of the block matrix unchanged; Rotate the second column of the block matrix by the size of one row of the block along the column direction; and Rotate the third column of the block matrix along the column direction by twice the size of a block row.

15. The hardware accelerator of claim 12, wherein one or more activations of the multiple segment activations are copied based on non-zero bits in the first row-bit block to generate a first set of compact row or multiple row activations, wherein the copying comprises: For one of the multiple activations, identify the position of the non-zero bit in each column of the corresponding bit block in the first row bit block; Copy the activation corresponding to the non-zero bit in each column of the corresponding bit block; and Compress the copied activations into a first set of compact one-line or multi-line activations.

16. The hardware accelerator of claim 15, wherein, In order to generate the compact activation tensor based on the bitmask and the activation tensor, the activation tensor compression circuit is further configured to: Rotate the multiple active segments in the first row by the size of one segment; as well as Multi-segment activations are generated based on the non-zero bit copy rotation in the second row bit block to produce a second set of compact one-line or multi-line activations.

17. The hardware accelerator of claim 16, wherein, In order to generate the compact activation tensor based on the bitmask and the activation tensor, the activation tensor compression circuit is further configured to compress the first set of compact one-line or more-line activations and the second set of compact one-line or more-line activations into an activation buffer for multiplication with the compact weight tensor.

18. A non-transitory computer-readable storage medium for use in a tensor product acceleration circuit, the storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform an operation, the operation comprising: Each layer of the neural network receives weight tensors and activation tensors; The weight tensor is divided into block matrices; The block matrix is ​​rearranged to obtain a rearranged weight tensor. The rearranged weight tensor is used to reduce the physical fan-out of the register storing the activation tensor. The register of the activation tensor can be arranged at the position of the corresponding block matrix. Calculate a bitmask, the bitmask comprising a bit matrix, wherein the bits in the bit matrix correspond to the elements in the rearranged weight tensor, wherein the elements in the rearranged weight tensor include one or more zero elements and one or more non-zero elements, and each bit in the bit matrix indicates whether the corresponding element in the rearranged weight tensor is a zero element or a non-zero element. Remove zero elements from the rearranged weight tensor and pack the non-zero elements in the rearranged weight tensor to obtain a compact weight tensor. A compact activation tensor is generated based on the bitmask and the activation tensor, wherein the compact activation tensor is generated by copying and compressing multiple activation segments stored in the register of the activation tensor. The bitmask includes a bit block matrix, and the bit blocks in the bit block matrix correspond to the blocks in the block matrix of the weight tensor. Generating a compact activation tensor based on the bitmask and the activation tensor includes dividing the first row of activations in the activation tensor into multiple activation segments, each of which corresponds to a bit block in the first row of bit blocks in the bit block matrix, and copying one or more activations from the multiple activation segments based on the non-zero bits in the first row of bit blocks to generate a first set of compact one or more rows of activations. as well as Tensor multiplication is performed based on the compact weight tensor and the compact activation tensor to generate the output tensor of the layer of the neural network.

Images