A LSTM accelerator and acceleration method based on systolic array

By adopting a pulsating array-based accelerator in the LSTM network, using sparse weight matrix and input matrix to flow between processing units, combined with block cyclic matrix compression and block balance pruning algorithm, the inefficiency of the LSTM network on resource-limited devices is solved. The problem of large storage needs has achieved high parallelism and high throughput computing performance improvement.

CN117610624BActive Publication Date: 2024-11-12NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311535097.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-16
Publication Date
2024-11-12
Estimated Expiration
2043-11-16

AI Technical Summary

Technical Problem

When deploying existing LSTM networks on embedded devices with limited resources, there are problems such as inefficiency and huge storage requirements. Traditional accelerators cannot effectively adapt to the compressed LSTM network model, resulting in idle computing units and unreasonable data flow arrangements, resulting in The overall efficiency is not high.

Method used

Using an LSTM accelerator based on a pulsating array, the multiplication operation of the weight matrix and the input vector is converted into matrix multiplication through a vector-matrix conversion module, and processed in the pulsating array. The sparse weight matrix and the input matrix flow between the processing unit PE. Combining block cyclic matrix compression and block balance pruning algorithm, data structure and parallelism are optimized to achieve high parallelism and high throughput operations.

Benefits of technology

Effectively improve the performance of LSTM accelerator, improve data utilization efficiency and accelerator's computing performance. Through high parallelism and high throughput operating state, the problems of inefficiency and large storage requirements of traditional LSTM networks on resource-limited devices are solved. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117610624B_ABST
    Figure CN117610624B_ABST
Patent Text Reader

Abstract

The present invention discloses an LSTM accelerator based on a systolic array, the accelerator includes a vector-matrix conversion module, a systolic array, an Element-wise module, a control module and a storage module; the vector-matrix conversion module is responsible for converting an input vector into a matrix form and mapping it to an input data stream of the systolic array; the systolic array is responsible for the multiplication operation of the sparse weight matrix of the LSTM gating unit and the input matrix, and is composed of a plurality of PE modules, and the input data stream and the weight data stream flow between the PE modules, highly multiplexing data, and effectively avoiding frequent data reading. The present invention also discloses an acceleration method for an LSTM accelerator based on a systolic array, the present invention converts the original matrix-vector multiplication operation into a matrix-matrix multiplication operation by optimizing the data structure, and processes the multiplication operation of the sparse weight matrix and the input matrix through the systolic array, realizes the high parallelism and high throughput of the PE module, thereby improving the computing performance of the accelerator.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence recurrent neural network, and in particular to an LSTM accelerator based on a systolic array and an acceleration method. Background Art

[0002] Long Short-Term Memory (LSTM) is a classic variation of Recurrent Neural Network (RNN) and is often used to process sequence data, such as text analysis, speech recognition, and language translation. Compared with traditional RNN networks, LSTM networks have more powerful memory functions and can better handle long sequences and long-term dependencies. The key feature of LSTM is that it introduces gating units and cell states to control the flow of information. The introduction of gating units enables LSTM to capture sequence information with longer time intervals and maintain long-term dependencies, avoiding the problem of gradient disappearance or explosion. At the same time, the introduction of gating units has led to an increasing amount of computation and storage complexity in LSTM networks, resulting in low efficiency in traditional general-purpose processors when executing LSTM models. The huge storage requirements also restrict the deployment of LSTM models in resource-limited embedded devices.

[0003] Field Programmable Gate Array (FPGA) has the characteristics of programmability, high parallelism, and low power consumption. Compared with traditional CPUs and GPUs, FPGA has higher flexibility and shorter development cycles. It can be designed for models with different network structures, and can effectively improve the computing efficiency of LSTM through reasonable parallel computing, algorithm mapping, and memory optimization.

[0004] Due to the huge memory requirements and low computational efficiency of LSTM, various compression methods for LSTM networks can effectively reduce the number of LSTM parameters and computational complexity, but at the same time may also cause problems such as decreased accuracy and unbalanced load. In addition, LSTM accelerators generally only use simple parallel and pipeline designs, which cannot be applied to compressed LSTM network models. There are cases where the computing units are idle or the data flow is arranged unreasonably, resulting in low overall model efficiency. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an LSTM accelerator and acceleration method based on a systolic array. By reasonably optimizing the data structure, the key multiplication operation of the weight matrix and the input vector in the LSTM is converted into a matrix-matrix multiplication operation, and a systolic array is used for processing. The weight data and input data in the systolic array flow between the basic processing units PE, thereby achieving high parallelism and high throughput, and effectively improving the performance of the accelerator.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] According to the present invention, a LSTM accelerator based on a systolic array includes a weight cache module, a vector-matrix conversion module, K systolic arrays and an Element-wise module, wherein the systolic array includes M×N processing unit PE modules, M and N are both integers greater than 1, and the number of systolic arrays is the same as the number of LSTM gating units in the LSTM model; wherein,

[0008] A vector-matrix conversion module, used to convert the input vector into a matrix form, obtain the input matrix and output it to the systolic array;

[0009] The systolic array is used to receive the weight matrix from the weight cache module and the input matrix from the vector-matrix conversion module. The weight matrix and the input matrix flow between the PE modules. The weight matrix is ​​pruned to obtain a sparse weight matrix. The multiplication operation of the sparse weight matrix of the LSTM gating unit and the input matrix is ​​completed in the systolic array, and the output vector is sent to the Element-wise module.

[0010] Element-wise module, which receives the vector from the systolic array output and calculates the cell state value and hidden value of the current time step.

[0011] As a further optimization solution of the LSTM accelerator based on the systolic array described in the present invention, the PE module includes J sparse weight multiplication and accumulation SMAC modules, and the SMAC module is used to process the multiplication and accumulation operation of a single hidden node of the LSTM unit.

[0012] As a further optimization solution of the LSTM accelerator based on the systolic array described in the present invention, the SMAC module includes a selector, a shifter and an adder, and the SMAC module is used to complete decoding, shifting and adding operations.

[0013] As a further optimization scheme of the LSTM accelerator based on the systolic array described in the present invention, the parallelism J of the SMAC module depends on the block size and sparsity of the block balanced sparse pruning.

[0014] Based on the above-mentioned acceleration method of the LSTM accelerator based on the systolic array, including the calculation method of the systolic array, the calculation method of the systolic array is as follows:

[0015] Step 1, K systolic arrays correspond to K LSTM gating units, and the systolic array is used to perform corresponding multiplication operations on the sparse weight matrix and the input matrix output by the vector-matrix conversion module;

[0016] Step 2: The PE module receives the parameters in the sparse weight matrix of the weight cache module and the parameters in the input matrix, or the parameters in the sparse weight matrix of the adjacent PE module and the parameters in the input matrix, and the SMAC module inside the PE module performs decoding and multiplication and accumulation operations;

[0017] Step 3: After the PE module completes the operation of the current cycle, it retains the part and result of the current operation, and passes the parameters of the current sparse weight matrix and the parameters in the input matrix to the adjacent PE module for the adjacent PE module to perform decoding and multiplication and accumulation operations;

[0018] Step 4: After receiving the parameters of the sparse weight matrix and the parameters in the input matrix, the adjacent PE module repeats the above steps 2 and 3 until the systolic array completes the multiplication of the sparse weight matrix of the LSTM gating unit and the input matrix; and passes the partial sum result in the PE module to the Element-wise module to calculate the current cell state value and hidden value. The partial sum result in the PE module refers to the output vector of the systolic array.

[0019] As a further optimization scheme of the acceleration method of the LSTM accelerator based on the systolic array described in the present invention, firstly, a block circulant matrix compression algorithm is used to compress the original weight matrix in the LSTM network, the weight matrix is ​​divided into blocks, and a circulant matrix is ​​used to replace the original weight matrix, and the precision loss is compensated by retraining, the circulant matrix is ​​represented by the first row vector, and the first row vector of each block circulant matrix is ​​rearranged and stored;

[0020] Secondly, a block balanced pruning algorithm is used to prune the first row of stored vectors. Each row vector retains the same sparsity, and the non-zero weights in the sparse weight matrix after compression and pruning are encoded. The index value and parameter value are combined and stored. The index value is stored in the high bit and the parameter value is stored in the low bit. The high-order index value is used to decode the corresponding input data value in the input vector.

[0021] Finally, exponential quantization is used to quantize the non-zero weights in the compressed and pruned sparse weight matrix into powers of 2. In the subsequent hardware acceleration process, shift operations can be used instead of multiplication operations.

[0022] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0023] The present invention mainly performs hardware acceleration on the forward reasoning process of the sparse LSTM neural network after compression and pruning. In order to improve the data utilization efficiency and accelerator performance, the original matrix-vector multiplication operation in the LSTM is converted into a matrix-matrix multiplication operation through a vector-matrix conversion module, and the structure and parallelism of the basic processing unit PE in the systolic array are reasonably arranged according to the block size of the circulant matrix and the sparsity of the block balance pruning. The pipeline design is reasonably used, and the weight data and input data flow between the PEs of the systolic array, the weights and input data are efficiently reused, and different PEs run simultaneously, thereby achieving a high parallelism and high throughput operation state, and effectively improving the computing performance of the accelerator. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is the architecture diagram of the LSTM accelerator based on the systolic array;

[0025] Figure 2 It is the LSTM network compression flow chart;

[0026] Figure 3a is a diagram of the systolic array structure. Figure 3b It is a schematic diagram of the PE structure of the basic component unit of the systolic array;

[0027] Figure 4 This is a schematic diagram of the SMAC structure. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] like Figure 1 , a LSTM accelerator based on a systolic array, the LSTM accelerator includes a vector-matrix conversion module, a systolic array, an Element-wise module, a storage module and a control module;

[0030] The LSTM network is compressed by a compression algorithm, which effectively reduces the storage requirements of the LSTM network model deployed on embedded devices and accelerates the calculation process. The vector-matrix conversion module is responsible for converting the input vector into a matrix form and mapping it to the input data stream of the systolic array. The systolic array consists of M×N PE modules, which receive weights and input data from the weight cache module and the vector-matrix conversion module. The data flows between PEs and performs operations with high parallelism to complete the multiplication of the sparse weight matrix of the LSTM gating unit and the input matrix. The PE module consists of J SMAC modules, which are responsible for the multiplication and accumulation operations of the LSTM hidden nodes. The parallelism of the SMAC module depends on the block size and sparsity of the block balance pruning.

[0031] The element-wise module consists of tanh and sigmoid modules, which are responsible for operations such as the activation function of the LSTM gate output vector.

[0032] The storage module includes off-chip DRAM, weight cache module, input cache module, and output cache module. The off-chip DRAM stores off-chip data. The weight cache module stores the weight parameters required for LSTM model calculation, including weight values ​​and weight index values. The input cache module is used to cache input data read from the off-chip DRAM. The output cache module includes c t Cache and h t Cache, used to store the LSTM cell state value output c at each time step t and hidden value output h t .

[0033] The control module outputs an enable signal, an accelerator system state conversion signal, and a read / write control signal of the storage unit to control the operation of the entire accelerator.

[0034] like Figure 2 , the multi-hybrid compression algorithm includes the following processes:

[0035] Step 1: First, build the original LSTM network model and obtain the weight matrix of the original LSTM network model by training the corresponding data set.

[0036] Step 2: Use the block circulant matrix to replace the original LSTM model weight matrix, retrain the replaced LSTM network, update the weight parameters, and restore the accuracy loss.

[0037] Step 3: Perform block-balanced pruning on the retrained weight matrix. The size of the pruned block is consistent with the size of the first row vector of the circulant matrix. Each weight block retains the same sparsity and is retrained until the accuracy loss is restored.

[0038] Step 4: quantize the non-zero weight parameters after pruning, and quantize the parameters to the power of 2. During this process, retrain continuously to compensate for the loss of accuracy.

[0039] Step 5: Encode the non-zero weights and store the index values ​​and weight values ​​at the same time.

[0040] like Figure 3a, which is a schematic diagram of the structure of the systolic array. The systolic array consists of M×N PE units. Adjacent PE units are separated only by registers, and the communication distance is short. The systolic array receives data from the weight cache module and the vector-matrix conversion module. The input data and weight parameters flow directionally between adjacent PE units in the systolic array. The input data is reused by the PE units in each row, and the weight parameters are reused by the PE units in each column. Different PE units operate simultaneously, with a high degree of parallelism, which accelerates the calculation process and improves the overall throughput of the system.

[0041] like Figure 3b , is a schematic diagram of the structure of the basic unit PE of the systolic array. PE mainly consists of SMAC module, adder and register. PE receives data from cache or adjacent PE for operation. In this example, the 7-bit weight parameter contains 4-bit weight value and 3-bit index value, and the input vector contains 8 input data, such as Figure 4 As shown in the figure, the data is decoded in the SMAC module, and the corresponding input data is decoded by the index value of the non-zero weight, and the multiplication and accumulation operation is performed with the weight value. The multiplication operation is realized by shifting, and the result of the multiplication and accumulation is saved in the partial sum register. After completing the multiplication and accumulation operation of all weights and inputs, the PE unit adds the partial sum results in all SMAC units to obtain the output value of the current node, and passes it to the Element-wise module for operation, outputting the cell state value and hidden value of the current time step.

[0042] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. An acceleration method for an LSTM accelerator based on a systolic array, characterized in that: The LSTM accelerator includes a weight cache module, a vector-matrix conversion module, K systolic arrays and an Element-wise module, wherein the systolic array includes M×N processing unit PE modules, M and N are both integers greater than 1, and the number of systolic arrays is the same as the number of LSTM gating units in the LSTM model; wherein, A vector-matrix conversion module, used to convert the input vector into a matrix form, obtain the input matrix and output it to the systolic array; The systolic array is used to receive the weight matrix from the weight cache module and the input matrix from the vector-matrix conversion module. The weight matrix and the input matrix flow between the PE modules. The weight matrix is ​​pruned to obtain a sparse weight matrix. The multiplication operation of the sparse weight matrix of the LSTM gating unit and the input matrix is ​​completed in the systolic array, and the output vector is sent to the Element-wise module. Element-wise module, which receives the vector from the systolic array output and calculates the cell state value and hidden value of the current time step; The systolic array is calculated as follows: Step 1, K systolic arrays correspond to K LSTM gating units, and the systolic array is used to perform corresponding multiplication operations on the sparse weight matrix and the input matrix output by the vector-matrix conversion module; Step 2: The PE module receives the parameters in the sparse weight matrix of the weight cache module and the parameters in the input matrix, or the parameters in the sparse weight matrix of the adjacent PE module and the parameters in the input matrix, and the SMAC module inside the PE module performs decoding and multiplication and accumulation operations; Step 3: After the PE module completes the operation of the current cycle, it retains the part and result of the current operation, and passes the parameters of the current sparse weight matrix and the parameters in the input matrix to the adjacent PE module for the adjacent PE module to perform decoding and multiplication and accumulation operations; Step 4: After receiving the parameters of the sparse weight matrix and the parameters in the input matrix, the adjacent PE module repeats the above steps 2 and 3 until the systolic array completes the multiplication operation of the sparse weight matrix of the LSTM gating unit and the input matrix; and passes the partial sum result in the PE module to the Element-wise module to calculate the current cell state value and hidden value. The partial sum result in the PE module refers to the output vector of the systolic array. Firstly, the block circulant matrix compression algorithm is used to compress the original weight matrix in the LSTM network. The weight matrix is ​​divided into blocks and replaced with a circulant matrix. The precision loss is compensated by retraining. The circulant matrix is ​​represented by the first row vector. The first row vector of each block circulant matrix is ​​rearranged and stored. Secondly, the block balanced pruning algorithm is used to prune the first row vector stored. Each row vector retains the same sparsity, and the non-zero weights in the sparse weight matrix after compression and pruning are encoded. The index value and parameter value are combined and stored. The high bit stores the index value and the low bit stores the parameter value. The high bit index value is used to decode the corresponding input data value in the input vector. Finally, exponential quantization is used to quantize the non-zero weights in the compressed and pruned sparse weight matrix into powers of 2, and shift operations are used instead of multiplication operations in the subsequent hardware acceleration process.

2. The acceleration method of the LSTM accelerator based on the systolic array according to claim 1, characterized in that: The PE module includes J sparse weight multiplication and accumulation SMAC modules, which are used to process the multiplication and accumulation operations of a single hidden node of the LSTM unit.

3. The acceleration method of the LSTM accelerator based on the systolic array according to claim 1, characterized in that: The SMAC module includes a selector, a shifter and an adder. The SMAC module is used to complete decoding, shifting and addition operations.

4. The acceleration method of the LSTM accelerator based on a systolic array according to claim 1, characterized in that: The parallelism J of the SMAC module depends on the block size and sparsity of the block-balanced sparse pruning.