LSTM hardware accelerator based on column-by-column maximum pruning

Abstract

The present invention discloses an LSTM hardware accelerator based on column-by-column maximization pruning, comprising a vector processing unit, a matrix-vector computing core, a parallel-input and serial-output shift register, a weight storage module, and an activation function module. The vector processing unit encodes the input vector and hidden-layer output vector into sparse vectors, and uses AIDX to represent the positional encoding of non-zero elements therein. The weight storage module transmits matching non-zero weight parameters from an on-chip memory in parallel according to AIDX. The matrix-vector computing core performs matrix calculations using the non-zero elements of the sparse vectors and the non-zero weight parameters indexed by the weight storage module. The shift register caches the calculation results of the matrix-vector computing core and transmits them serially to the activation function module. The activation function module is responsible for calculating sigmoid and tanh functions and element-by-element operations. The present invention ensures that the regular distribution of the sparse matrix structure after pruning matches the parallel computing characteristics of the hardware, thereby improving resource utilization.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer hardware acceleration, and in particular relates to an LSTM hardware accelerator based on column-by-column maximization pruning. Background Art

[0002] Long Short-Term Memory (LSTM) is a variant of Recurrent Neural Network (RNN) widely used in sequence task processing, especially in tasks such as speech recognition and sentiment analysis. Compared with traditional RNN, LSTM introduces three gates (input gate, forget gate, output gate) and a cell state. The corresponding formulas are shown in Equations (1)-(6), and the schematic diagram is shown in Figure 1 As shown in Figure 2, these mechanisms enable LSTM to better handle long-term dependencies in sequences.

[0003] ;

[0004] Where W is the weight matrix, b is the bias matrix, x is the data input at the current moment, y is the current output, c is the cell state, h is the hidden state, σ is the sigmoid function and tanh tangent function, and ⊙ is the matrix dot product.

[0005] However, the gating units introduced in LSTM significantly increase computational overhead and storage requirements, a problem that is particularly prominent in resource-constrained edge computing platforms and high-real-time scenarios. To address this challenge, researchers have proposed a variety of model compression techniques, such as pruning, to reduce memory access and computational complexity.

[0006] Pruning techniques reduce memory requirements by removing unimportant connections between neurons in the model and attempting to delete or prune redundant and non-critical weights. This results in a sparse weight matrix that occupies less memory than the original dense matrix, thus achieving model compression. In recent years, to address workload imbalance caused by irregular weight sparsity patterns after model pruning, structured pruning has been applied to RNN accelerators. For example, column-balanced target dropout (CBTD) pruning ensures that the number of non-zero weights in each column of a sparse matrix is ​​equal, resolving the load imbalance issue and achieving a throughput of up to 9.4 Top / s. However, this method does not account for the error sensitivity of gate units, leaving room for further compression. The block-balanced sparsity (BBS) method divides the rows of the weight matrix into blocks of equal length and then applies fine-grained pruning to each block to ensure that the number of non-zero elements in each group is equal. The matrix-vector multiplication (MxV) operation accounts for the majority of LSTM computation time. While this method achieves sparse matrices, the vectors remain dense, suggesting further room for acceleration.

[0007] In order to improve the accelerator's inference speed, increasing vector sparsity is also an effective way to reduce the LSTM accelerator's memory access requirements. Inspired by incremental networks, sparse vectors are generated by storing the calculation results of the model's previous time step, further reducing the amount of matrix-vector multiplication calculations. The generation of sparse vectors is based on the difference between the input vector and the vector of the previous time step. Elements with a difference less than a set threshold are set to zero. As shown in the following formula, given an input vector x t To output y t :

[0008] ;

[0009] in is the increment of the input vector between two time steps. W represents the weight matrix. When the difference between the inputs of adjacent time steps is too small, it is set to 0, achieving vector sparsification. Therefore, formulas (1)-(6) can be rewritten as the following formula:

[0010] ;

[0011] in:

[0012] .

[0013] LSTMs based on the DN algorithm can achieve high vector sparsity. However, when combined with pruned sparse weights, workload balance becomes unbalanced. This is due to the uneven distribution of non-zero values ​​across matrix columns, and the dynamic and random distribution of non-zero elements in the vector at each time step. Consequently, existing LSTM accelerators often suffer from low hardware utilization when deploying LSTM models with sparse weights and vectors, making them difficult to use in real-world applications.

[0014] At the hardware accelerator architecture level, G. Nan et al. proposed a hybrid engine LSTM accelerator that supports both structured and sparse matrix MxV operations. However, this results in more complex hardware architecture and difficulty in timing matching. Z. Que et al. proposed a column-by-column LSTM hardware accelerator architecture that effectively alleviates the dependencies between input data in RNN networks and enables a fully pipelined structure. However, this results in significant latency when the hidden layer dimensions are small.

[0015] Existing LSTM model compression technology and FPGA-based LSTM acceleration technology have the following main problems:

[0016] 1. Existing structured pruning technologies make it difficult to adapt sparse matrices to incremental sparse vectors. This is because the non-zero values ​​of the weight matrix are unevenly distributed across columns. Matrix operations combined with sparse vectors can lead to unbalanced workloads on the hardware platform.

[0017] 2. Existing structured pruning algorithms for LSTM do not explore the impact of different weight matrices on model accuracy, and there is a potential structured pruning space with higher sparsity.

[0018] 3. Existing incremental network technologies convert dense vectors into sparse vectors, which reduces storage and computational overhead during inference. However, they lack an efficient dedicated hardware module to encode dense vectors and then generate sparse vectors and their address indices in a pipelined manner.

[0019] 4. The existing LSTM hardware accelerator architecture suffers from data dependency issues due to the cyclic nature of the model. This means that the calculation of the current time step requires the output of past time steps. This introduces unnecessary delays in the inference process, resulting in low hardware utilization and limiting the actual throughput of the accelerator. Summary of the Invention

[0020] The present invention provides an LSTM hardware accelerator based on column-by-column maximization pruning, which further compresses the LSTM model while ensuring its accuracy.

[0021] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0022] An LSTM hardware accelerator based on column-by-column maximization pruning, comprising: a vector processing unit, a matrix-vector calculation core, a parallel-input and serial-output shift register, a weight storage module, and an activation function module;

[0023] The vector processing unit is used to: and the hidden layer output vector Encoded as a sparse vector and , and use AIDX to represent the position encoding of non-zero elements of the sparse vector;

[0024] The weight storage module is used to: send matching non-zero weight parameters from the on-chip memory in parallel according to the position code AIDX;

[0025] The matrix vector calculation core is used to: perform matrix calculation using the non-zero elements of the sparse vector transmitted by the vector processing unit and the non-zero weight parameters indexed by the weight storage module;

[0026] The shift register is used to: cache the calculation results of the matrix vector calculation core and transmit them to the activation function module in a serial manner;

[0027] The activation function module is responsible for calculating sigmoid, tanh functions and element-by-element operations.

[0028] Furthermore, a LSTM acceleration method based on column-by-column maximization pruning is implemented, wherein the LSTM acceleration method includes:

[0029] Step 1: Classify each weight matrix included in LSTM into sensitive and non-sensitive types according to its sensitivity to LSTM accuracy;

[0030] Step 2: Set different sparsity combination values ​​for sensitive and non-sensitive weight matrices, perform column-by-column maximization pruning on each LSTM weight matrix according to each sparsity combination value, and then determine the optimal sparsity combination value based on the corresponding LSTM accuracy after pruning;

[0031] Step 3: Maximize the pruning of each LSTM weight matrix column by column according to the optimal sparsity combination value to achieve LSTM acceleration.

[0032] Furthermore, the weight matrices of LSTM, including the weight matrices of the input gate, forget gate, cell state, and output gate, are classified as sensitive, and the weight matrices of the cell state are classified as non-sensitive.

[0033] Furthermore, the specific process of step 2 includes:

[0034] Step 2.1: The sparsity of sensitive and non-sensitive weight matrices 、 Set to a low preset level, the current maximum overall sparsity The default value is 0; Step 2.2, with the current sparsity For each non-sensitive weight matrix of LSTM Perform column-by-column pruning based on the current sparsity For each sensitive weight matrix of LSTM Perform column-by-column pruning;

[0035] Step 2.3: Train the pruned LSTM and calculate the model accuracy of LSTM and overall sparsity ;

[0036] Step 2.4, determine whether it is satisfied and ,in is the accuracy of the pre-trained model, is the acceptable threshold of model accuracy;

[0037] If satisfied, combine the current sparsity As the current optimal sparsity combination, and the current Assign to ;

[0038] Step 2.5, set the current sparsity Increase the interval α. If it is still less than 100% after the increase, return to step 2.2; otherwise, go to step 2.6.

[0039] Step 2.6, the current sparsity Increase the interval α. If it is still less than 100% after the increase, the sparsity If it is set back to the preset low level, return to step 2.2, otherwise go to step 2.7;

[0040] Step 2.7, output the current optimal sparsity combination.

[0041] Furthermore, the vector processing unit includes: an input cache, a vector sparse conversion module, an incremental scheduler, and an output cache;

[0042] The input buffer, for the input vector and the hidden layer output vector Vectors constructed by concatenation and transmitted in data stream mode Cache;

[0043] The vector sparse conversion module processes sequentially in each clock cycle: the subtractor calculates the vectors of two adjacent time steps 、 The difference between ; Then the difference It is definitely worth it and through the comparator Compare with the pre-stored threshold θ; then output according to the comparison result through the selector: , then the difference Output as the incremental value and its address index, otherwise the incremental value is set to 0 for output;

[0044] The incremental scheduler extracts the non-zero incremental value and the address index thereof output by the vector sparse conversion module, and stores them in two independent output caches respectively.

[0045] Furthermore, the vector of the previous time step , stored in the random access storage based on the lookup table, and the comparison result output by the comparator is used to control the write use signal of the random access storage, and the comparator is in When it is greater than the threshold value θ, the output is high level.

[0046] Furthermore, the matrix vector calculation core adopts a block parallel computing structure, including several processing units PE, several addition trees and an output cache; each PE includes a multiplier and a random access memory based on a lookup table;

[0047] First, all PEs parallelize the incoming increment vector Perform multiplication and addition operations with weights;

[0048] Then, each PE obtains the past accumulation result of the corresponding address from its own random access storage according to the address index of the vector processing unit output cache, and accumulates it with the current multiplication and addition result. The accumulation result is stored in the random access storage;

[0049] The calculation results of all PEs are summed through the addition tree and the summation result is sent to the output buffer.

[0050] Furthermore, the weight storage module allocates storage blocks by column for the weight merging matrix after the sparse operation, and each column corresponds to one storage block for dedicated storage; and when the output cache of the vector processing unit is not empty, the weight storage module reads out the weights in parallel, and the number of weight parameters read out in parallel for each column is VP, and the EP column is read out at the same time; wherein, the weight merging matrix after the sparse operation is obtained by stacking the gated weight matrices of the LSTM from top to bottom and then performing the sparse operation.

[0051] Furthermore, the activation function module adopts a six-stage pipeline structure, the input is the matrix operation result sent serially by the shift register, and the output is the activation value of the hidden layer. Specifically:

[0052] S1: Perform sigmoid function operation on the matrix operation result of the input gate to obtain the output ;

[0053] S2: Perform tanh function operation on the matrix operation results of candidate cells to obtain output ;

[0054] S3: Perform sigmoid function operation on the matrix operation result of the forget gate to obtain the forget gate output ; At the same time and Multiply to get the intermediate result R1;

[0055] S4: Perform sigmoid function operation on the matrix operation result of the output gate to obtain the output ; At the same time, the forget gate output Compared with the cell state at the previous time step t-1 After multiplication, add it to R1 to get the cell state at the current time step t ;

[0056] S5: Cell status Perform tanh function operation to obtain the intermediate result R2;

[0057] S6: Multiply by R2 to get the hidden layer activation value .

[0058] Furthermore, the activation function module adopts a six-stage pipeline structure, the input is the matrix operation result sent serially by the shift register, and the output is the activation value of the hidden layer. Specifically:

[0059] S1: Perform tanh function operation on the matrix operation results of candidate cells to obtain output ;

[0060] S2: Perform sigmoid function operation on the matrix operation result of the input gate to obtain the output ;

[0061] S3: Perform sigmoid function operation on the matrix operation result of the forget gate to obtain the forget gate output ; At the same time and Multiply to get the intermediate result R1;

[0062] S4: Perform sigmoid function operation on the matrix operation result of the output gate to obtain the output ; At the same time, the forget gate output Compared with the cell state at the previous time step t-1 After multiplication, add it to R1 to get the cell state at the current time step t ;

[0063] S5: Cell status Perform tanh function operation to obtain the intermediate result R2;

[0064] S6: Multiply by R2 to get the hidden layer activation value .

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] 1. To address the problem that existing sparse vector processing technologies cannot adapt to the dynamic distribution of non-zero elements, the LSTM hardware accelerator of the present invention has a vector encoding unit that generates sparse vectors and their address indexes in a pipelined manner, supports the dynamic distribution of non-zero elements in sparse vectors, and effectively improves computing efficiency.

[0067] 2. In response to the data dependency problem caused by the cyclic characteristics of the model itself in the existing LSTM hardware accelerator architecture, the activation function module of the LSTM hardware accelerator of the present invention alleviates the data dependency between LSTM time steps, realizes a full pipeline structure, and improves the effective throughput by skipping redundant calculations.

[0068] 3. The present invention is based on the LSTM method of column-by-column maximization pruning, which prunes the weight matrix column by column to ensure that it has column-balanced properties, effectively supports sparse vectors to realize matrix parallel operations, and effectively improves computing efficiency.

[0069] 4. The present invention is based on the LSTM method of column-by-column maximization pruning. By exploring the sensitivity of the sparsity of each LSTM weight matrix to the model accuracy, different degrees of target sparsity are applied to each weight matrix for pruning, thereby ensuring the model accuracy while further improving the overall sparsity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a schematic diagram of an LSTM unit.

[0071] Figure 2 It is the pseudo code of the adaptive pruning rate adjustment (CMP) algorithm of the present invention.

[0072] Figure 3 It is the pseudo code of the column-wise pruning (CP) algorithm of the present invention.

[0073] Figure 4 This is the overall hardware architecture of the LSTM hardware accelerator of the present invention.

[0074] Figure 5 It is the VPU module of the LSTM hardware accelerator of the present invention.

[0075] Figure 6 This is the MVM kernels structure of the LSTM hardware accelerator of the present invention.

[0076] Figure 7 This is the WMEM structure of the LSTM hardware accelerator of the present invention.

[0077] Figure 8 This is the PISO structure of the LSTM hardware accelerator of the present invention.

[0078] Figure 9 This is the activation pipeline structure of the LSTM hardware accelerator of the present invention. DETAILED DESCRIPTION

[0079] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, provides a detailed implementation method and a specific operation process, and further explains the technical solution of the present invention.

[0080] Example 1

[0081] This embodiment provides an LSTM acceleration method based on column-by-column maximum pruning, including:

[0082] Step 1: Classify each weight matrix included in LSTM into sensitive type and non-sensitive type according to its sensitivity to LSTM accuracy.

[0083] LSTM contains four weight matrices: (Input Gate), (Forgotten Gate), (cell status), (Output gate) The embodiment of the present invention adopts column-by-column maximization pruning technology, which is divided into two categories according to the sensitivity of each weight matrix to the accuracy of the LSTM model: one is error-sensitive matrix (Its sparsification will significantly affect the accuracy of the model, and the pruning rate needs to be limited); the other type is the error-insensitive matrix (Its sparsification has less impact on accuracy and has a higher potential pruning rate.) Among them, the weight matrix of the cell state is Directly affects cell status The update of , its error will propagate through the time step, resulting in cumulative error, so it is classified as an error-sensitive matrix, while the other three weight matrices mainly control information screening and are less sensitive to error propagation, so they are classified as non-sensitive matrices.

[0084] Step 2: Set different sparsity combination values ​​for sensitive and non-sensitive weight matrices, perform column-by-column maximization pruning on each LSTM weight matrix according to each sparsity combination value, and then determine the optimal sparsity combination value based on the corresponding LSTM accuracy after pruning.

[0085] refer to Figure 2 The pseudo code of the adaptive pruning rate adjustment (CMP) algorithm shown in Figure 2 specifically includes:

[0086] Step 2.1: The sparsity of sensitive and non-sensitive weight matrices 、 Set to a low preset level, the current maximum overall sparsity The default value is 0;

[0087] Step 2.2, with the current sparsity For each non-sensitive weight matrix of LSTM Perform column-by-column pruning based on the current sparsity For each sensitive weight matrix of LSTM Column-by-column pruning is performed to ensure that the number of non-zero elements in each column is equal, thereby reducing computational complexity.

[0088] refer to Figure 3 The pseudo code of the column-by-column pruning (CP) algorithm shown includes:

[0089] (1) Process the weight matrix column by column, and arrange all elements in each column in descending order of the absolute value of the weight;

[0090] (2) Prune the weight element with the smallest absolute value of γ (target sparsity) and retain the weight elements of the first (1-γ), that is, set the γ weight elements in each column to zero, in order to ensure that the number of non-zero elements in each column is consistent, where 0 < γ < 100%;

[0091] (3) Output a sparse weight matrix W with column-balanced properties p .

[0092] Step 2.3: Train the pruned LSTM and calculate the model accuracy of LSTM and overall sparsity .

[0093] Step 2.4, determine whether it is satisfied and ,in is the accuracy of the pre-trained model, is the acceptable threshold of model accuracy;

[0094] If satisfied, combine the current sparsity As the current optimal sparsity combination, and the current Assign to ;Right now , ; is the current optimal sensitive sparsity and insensitive sparsity;

[0095] Step 2.5, set the current sparsity Increase the interval α. If it is still less than 100% after the increase, return to step 2.2; otherwise, go to step 2.6.

[0096] Step 2.6, the current sparsity Increase the interval α. If it is still less than 100% after the increase, return to step 2.2 and set the sparsity Set back to the preset low level. Otherwise, proceed to step 2.7;

[0097] Step 2.7, output the current optimal sparsity combination.

[0098] Step 3: Maximize the pruning of each LSTM weight matrix column by column according to the optimal sparsity combination value to achieve LSTM acceleration.

[0099] The present invention aims to adapt to the dynamic characteristics of non-zero elements in sparse vectors and improve the efficiency of matrix operations when deploying LSTM on hardware platforms. LSTM is optimized by:

[0100] 1. Dynamic pruning: Adaptively adjust the pruning rate based on the sensitivity of each weight matrix to the model accuracy (error sensitivity).

[0101] 2. Hardware compatibility: Ensure that the pruned sparse matrix structure is regularly distributed, match the hardware parallel computing characteristics, and improve resource utilization.

[0102] 3. Maximum sparsity: Achieve maximum overall sparsity of the weight matrix while maintaining model accuracy.

[0103] Example 2

[0104] This embodiment provides an LSTM hardware accelerator that implements the LSTM acceleration method described in Example 1. Figure 4 As shown, it includes: vector processing unit (VPU), matrix vector calculation kernel (MVM kernels), parallel input serial output shift register (PISO), weight storage module (WMEM) and activation function module (AFB).

[0105] 1. Vector Processing Unit (VPU).

[0106] The vector processing unit (VPU) is used to convert the input vector and the hidden layer output vector Encoded as a sparse vector (Right now and ), and AIDX is used to represent the position encoding of non-zero elements of the sparse vector.

[0107] The VPU module is as follows Figure 5 The VPU module includes an input FIFO, a vector sparse conversion module DSM, an incremental scheduler, and an output FIFO. The input of the VPU module is the sequence input vector of the LSTM. and the hidden layer output vector , It is fed by the PS side of FPGA via AXI interface. The output of the previous time step is directly connected to the input FIFO, and the output is a sparse vector and its position index.

[0108] S1: vector and Concatenate to form the input vector and transmitted to the VPU module in the form of data stream until the First-In-First-Out (FIFO) storage buffer is filled.

[0109] S2: The FIFO module is subsequently connected to the DSM module. The DSM will process the data in sequence in each clock cycle. The M elements in .

[0110] S3: In DSM, the difference between the two time step vector elements is calculated by a subtractor, and then the absolute value of the difference is taken. The difference is compared with the pre-stored threshold θ through a comparator. If it is greater than the threshold θ, the difference (i.e., the increment) and its address index will be retained; otherwise, the output increment value will be set to 0.

[0111] S4: The output of the comparator controls the write enable signal of the random access memory (LUTRAM). The comparator is in |∆S t When the value is greater than the threshold value θ, a high level state is generated.

[0112] S5: The DSM module is then connected to the incremental scheduler. The non-zero elements and address indices in the FIFO are stored in two separate FIFO buffers: FIFO_V is used to store the increment value, and FIFO_I is used to store the position index.

[0113] 2. Matrix-vector kernels (MVM kernels).

[0114] The matrix vector computation kernel (MVM kernels) is used to perform matrix calculations using the non-zero elements of the sparse vectors transmitted by the vector processing unit and the non-zero weight parameters indexed by the weight storage module. This module is a block parallel computing structure composed of EP*VP processing units PE. PE is a MAC unit synthesized from a digital signal processing (DSP) module that can perform up to 16-bit multiplication by 16-bit and 48-bit accumulation. In addition, each PE has a dedicated LUTRAM to cache its corresponding partial accumulation for operations between non-zero elements and weights, such as Figure 6 As shown, the module input is the output of FIFO_V and the output of WMEM, and the module output is the result of the matrix operation involved in the gating unit.

[0115] S1: When the FIFO of the VPU module is not empty, the increment vector Together with the weights stored in WMEM, they are passed into the PE unit in parallel and multiplication and addition operations are performed using its multiplier-accumulator.

[0116] S2: Each PE obtains the past accumulation result of the corresponding address from the LUTRAM of the PE according to the address index of FIFO_I, and accumulates it with the current multiplication and addition result of its own multiplier and adder. The accumulation result is stored in the LUTRAM of the PE.

[0117] S3: The calculation results of all PEs are summed through the addition tree and the summation result is sent to the output buffer.

[0118] S4: The output buffer of the adder is fed into the subsequent activation module to generate the activation value.

[0119] In matrix multiplication, a matrix is ​​a×b and a vector is b×1. Due to hardware resource limitations, parallel computing units (such as DSPs) are insufficient to process a×b calculations simultaneously. Therefore, block operations are required. The horizontal length of a parallel computing block is EP, and the column length is VP, meaning the block size is EP×VP. Sparsity is related to the matrix size a×b. EP×VP should be less than the total number of DSPs in the hardware device, and a / VP and b / EP should be integers to improve hardware computing efficiency. Furthermore, to achieve full pipeline operation and minimize latency, EP should be minimized. For example, in the embodiment of the present invention, the block size of the computing unit is 2×512.

[0120] 3. Weight Storage Module (WMEM).

[0121] The weight storage module (WMEM) is used to send the matching non-zero weight parameters from the on-chip memory in parallel according to the position code AIDX. In the weight storage module of this embodiment, the weight matrix is ​​divided into blocks in a column manner, and each column of non-zero weights is used as a weight storage block. The input of this module is the output of FIFO_I of VPU, and the output is the non-zero weight, such as Figure 7 shown.

[0122] S1: The weight matrices of the four gates in LSTM are stacked from top to bottom into a matrix Ws, which is then allocated and encoded column by column after the sparse operation.

[0123] S2: Each nonzero element in each column of the sparse matrix is ​​treated as a submatrix. This means the sparse matrix is ​​split into N submatrices, where N = the length of the input vector x + the length of the hidden layer vector h. Each submatrix is ​​stored in a dedicated WMEM block in its corresponding matrix operation array to avoid conflicts between block storage. The increment vector is split in the same manner as the stacked weight matrix.

[0124] S3: When the FIFO_I of the VPU is not empty, WMEM reads out the weights in parallel. The number of weights read out in parallel for each column is VP, and the EP column is read out at the same time.

[0125] 4. Shift register (PISO).

[0126] The shift register (PISO) is used to cache the calculation results of the matrix vector calculation core and transmit them to the activation function module in a serial manner. The PISO of this embodiment is composed of Pa serial 16-bit shift registers, such as Figure 8As shown in the figure, the input is the matrix operation result of the MVM kernels and the control enable signal Enable, and the output is used as the input of the AFB module. It should be noted that HID is the hidden layer unit length of the LSTM network and Pa is the activation parallelism.

[0127] S1: After the matrix vector calculation core completes the matrix operation, the results are cached in the shift register in parallel.

[0128] S2: The matrix operation results to be activated are output in sequence through the enable signal control, and Pa matrix operation results will be sent in parallel in each clock cycle.

[0129] S3: When the data in all shift registers are empty, the next batch of matrix operation results are stored again, and then the steps of S2 are repeated.

[0130] 5. Activation function module (AFB).

[0131] The activation function module (AFB) is responsible for calculating the sigmoid, tanh functions, and element-by-element operations. Since a series of element-by-element multiplication operations are required in the activation function calculation in LSTM, in order to complete this part of the operation and maximize the use of hardware resources, this embodiment designs the activation function operation and element-by-element operation into a six-stage pipeline structure (S1-S6), as shown in the following example. Figure 9 The input of this pipeline is the matrix operation result sent serially by PISO, and the output is the activation value of the hidden layer.

[0132] S1: Perform sigmoid function operation on the matrix operation result of the input gate to obtain the output .

[0133] S2: Perform tanh function operation on the matrix operation results of candidate cells to obtain output ; This step S2 can be exchanged with S1 in order.

[0134] S3: Perform sigmoid function operation on the matrix operation result of the forget gate to obtain the forget gate output ; At the same time and Multiply to get the intermediate result R1.

[0135] S4: Perform sigmoid function operation on the matrix operation result of the output gate to obtain the output ; At the same time, the forget gate output Compared with the cell state at the previous time step t-1 After multiplication, add it to R1 to get the cell state at the current time step t .

[0136] S5: Cell status Perform tanh function operation to obtain the intermediate result R2.

[0137] S6: Multiply by R2 to get the hidden layer activation value .

[0138] This pipeline design prevents activation function operations from being occupied at the same time. In addition, because the activation function module is designed using the CODIC algorithm, it can complete different activation function calculations by configuring the selection signal. This ensures that all activation function calculation requirements of the LSTM can be met with fewer activation modules. For example, by arranging sigmoid and tanh calculations in different pipeline stages, the activation function module can perform calculations when sigmoid is needed, and perform tanh calculations when tanh is needed by changing the selection signal. This reduces implementation complexity and enables the reuse of activation function modules, thereby saving hardware resources.

[0139] The above embodiments are preferred embodiments of the present invention. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present invention, these changes or improvements should fall within the scope of protection required by the present invention.

Claims

1. An LSTM hardware accelerator based on column-by-column maximization pruning, characterized in that: include: Vector processing unit, matrix-vector calculation core, parallel input and serial output shift register, weight storage module and activation function module; The vector processing unit is used to: and the hidden layer output vector Encoded as a sparse vector and , and use AIDX to represent the position encoding of non-zero elements of the sparse vector; The weight storage module is used to: send matching non-zero weight parameters from the on-chip memory in parallel according to the position code AIDX; The matrix vector calculation core is used to: perform matrix calculation using the non-zero elements of the sparse vector transmitted by the vector processing unit and the non-zero weight parameters indexed by the weight storage module; The shift register is used to: cache the calculation results of the matrix vector calculation core and transmit them to the activation function module in a serial manner; The activation function module is responsible for calculating sigmoid, tanh functions and element-by-element operations; The LSTM hardware accelerator is used to implement an LSTM acceleration method based on column-by-column maximization pruning, and the LSTM acceleration method includes: Step 1: Classify each weight matrix included in LSTM into sensitive and non-sensitive types according to its sensitivity to LSTM accuracy; Step 2: Set different sparsity combination values ​​for sensitive and non-sensitive weight matrices, perform column-by-column maximization pruning on each LSTM weight matrix according to each sparsity combination value, and then determine the optimal sparsity combination value based on the corresponding LSTM accuracy after pruning; Step 2.1: The sparsity of sensitive and non-sensitive weight matrices 、 Set to a low preset level, the current maximum overall sparsity The default value is 0; Step 2.2, with the current sparsity For each non-sensitive weight matrix of LSTM Perform column-by-column pruning based on the current sparsity For each sensitive weight matrix of LSTM Perform column-by-column pruning; Step 2.3: Train the pruned LSTM and calculate the model accuracy of LSTM and overall sparsity ; Step 2.4, determine whether it is satisfied and ,in is the accuracy of the pre-trained model, is the acceptable threshold of model accuracy; If satisfied, combine the current sparsity As the current optimal sparsity combination, and the current Assign to ; Step 2.5, set the current sparsity Increase the interval α. If it is still less than 100% after the increase, return to step 2.2; otherwise, go to step 2.

6. Step 2.6, the current sparsity Increase the interval α. If it is still less than 100% after the increase, the sparsity If it is set back to the preset low level, return to step 2.2, otherwise go to step 2.7; Step 2.7, output the current optimal sparsity combination; Step 3: Maximize the pruning of each LSTM weight matrix column by column according to the optimal sparsity combination value to achieve LSTM acceleration.

2. The LSTM hardware accelerator according to claim 1, characterized in that The weight matrices of LSTM include the weight matrices of the input gate, forget gate, cell state, and output gate. The weight matrix of the cell state is classified as sensitive, and the remaining weight matrices are classified as non-sensitive.

3. The LSTM hardware accelerator according to claim 1, characterized in that The vector processing unit includes: an input cache, a vector sparse conversion module, an incremental scheduler, and an output cache; The input buffer, for the input vector and the hidden layer output vector Vectors constructed by concatenation and transmitted in data stream mode Cache; The vector sparse conversion module processes sequentially in each clock cycle: the subtractor calculates the vectors of two adjacent time steps 、 The difference between ; Then the difference It is definitely worth it and through the comparator Compare with the pre-stored threshold θ; then output according to the comparison result through the selector: , then the difference Output as the incremental value and its address index, otherwise the incremental value is set to 0 for output; The incremental scheduler extracts the non-zero incremental value and the address index thereof output by the vector sparse conversion module, and stores them in two independent output caches respectively.

4. The LSTM hardware accelerator according to claim 3, characterized in that The vector of the previous time step , stored in the random access storage based on the lookup table, and the comparison result output by the comparator is used to control the write use signal of the random access storage, and the comparator is in When it is greater than the threshold value θ, the output is high level.

5. The LSTM hardware accelerator according to claim 1, characterized in that The matrix vector calculation core adopts a block parallel computing structure, including several processing units (PEs), several addition trees and an output cache; each PE includes a multiplier-adder and a random access memory implemented based on a lookup table; First, all PEs parallelize the incoming increment vector Perform multiplication and addition operations with weights; Then, each PE obtains the past accumulation result of the corresponding address from its own random access storage according to the address index of the vector processing unit output cache, and accumulates it with the current multiplication and addition result. The accumulation result is stored in the random access storage; The calculation results of all PEs are summed through the addition tree and the summation result is sent to the output buffer.

6. The LSTM hardware accelerator according to claim 1, characterized in that The weight storage module allocates storage blocks by column for the weight merging matrix after the sparse operation, and each column corresponds to one storage block for dedicated storage; and when the output cache of the vector processing unit is not empty, the weight storage module reads out the weights in parallel, and the number of weight parameters read out in parallel for each column is VP, and the EP column is read out at the same time; wherein, the weight merging matrix after the sparse operation is obtained by stacking the gated weight matrices of the LSTM from top to bottom and then performing the sparse operation.

7. The LSTM hardware accelerator according to claim 1, characterized in that The activation function module adopts a six-stage pipeline structure. The input is the matrix operation result sent serially by the shift register, and the output is the activation value of the hidden layer. Specifically: S1: Perform sigmoid function operation on the matrix operation result of the input gate to obtain the output ; S2: Perform tanh function operation on the matrix operation results of candidate cells to obtain output ; S3: Perform sigmoid function operation on the matrix operation result of the forget gate to obtain the forget gate output ; At the same time and Multiply to get the intermediate result R1; S4: Perform sigmoid function operation on the matrix operation result of the output gate to obtain the output ; At the same time, the forget gate output Compared with the cell state at the previous time step t-1 After multiplication, add it to R1 to get the cell state at the current time step t ; S5: Cell status Perform tanh function operation to obtain the intermediate result R2; S6: Multiply by R2 to get the hidden layer activation value .

8. The LSTM hardware accelerator according to claim 1, characterized in that The activation function module adopts a six-stage pipeline structure. The input is the matrix operation result sent serially by the shift register, and the output is the activation value of the hidden layer. Specifically: S1: Perform tanh function operation on the matrix operation results of candidate cells to obtain output ; S2: Perform sigmoid function operation on the matrix operation result of the input gate to obtain the output ; S3: Perform sigmoid function operation on the matrix operation result of the forget gate to obtain the forget gate output ; At the same time and Multiply to get the intermediate result R1; S4: Perform sigmoid function operation on the matrix operation result of the output gate to obtain the output ; At the same time, the forget gate output Compared with the cell state at the previous time step t-1 After multiplication, add it to R1 to get the cell state at the current time step t ; S5: Cell status Perform tanh function operation to obtain the intermediate result R2; S6: Multiply by R2 to get the hidden layer activation value .

Images