Efficient Vision-LSTM (Long Short Term Memory) hardware accelerator based on FPGA (Field Programmable Gate Array)

Abstract

The invention discloses an efficient Vision-LSTM (Long Short Term Memory) hardware accelerator based on an FPGA (Field Programmable Gate Array), which comprises a multi-head mLSTM cell calculation unit, the multi-head mLSTM cell calculation unit comprises a plurality of mLSTM cell calculation units which are connected in parallel, and each mLSTM cell calculation unit comprises a QKMatCalc calculation subunit, a CVMatCalc calculation subunit, a DMatCalc calculation subunit, a CMatCalc calculation subunit, a first Norm calculation subunit and a V pipeline buffer unit, according to the method, efficient reasoning calculation of the ViL model can be achieved, compared with a GPU standard, the performance expression of 0.90-1.19 x is achieved, and the energy efficiency is improved by 26.80 x-29.57 x.

Description

Technical Field

[0001] The present invention belongs to the field of FPGA data processing technology, and more specifically, relates to an efficient Vision-LSTM hardware accelerator based on FPGA. Background Art

[0002] Currently, if Figure 2 , Figure 3 As shown in Figure 1, Vision-LSTM (ViL), based on the xLSTM building block, has attracted much attention due to its linear computation and storage complexity, as well as its excellent performance in computer vision fields such as medical imaging. The performance of Vision-LSTM (ViL) in the imagenet-1k recognition task even exceeds the SSM-based model and the ViT model after extensive optimization. Therefore, the Vil model is considered to be a promising visual task model.

[0003] However, since there are a large number of computing units of different types in the ViL model, and the computing modes of each computing unit are very different, its computing flow is quite complicated. In addition, there are a considerable number of large-span residual connection mechanisms in the ViL model, which makes its calculation process have a lot of data dependencies, further increasing the computational complexity. Therefore, general computing units such as CPUs and GPUs cannot perform ViL reasoning calculations efficiently. In the actual running test process, this problem is directly manifested in the excessive processing time consumed in the calculation part with low computational volume. Directly deploying the ViL model on the GPU will result in high deployment costs.

[0004] In addition, during the inference calculation process of the ViL model, the internal mLSTM Cell part of the calculation involves a large number of lower triangular matrix calculations; this means that nearly half of the calculations in the mLSTM Cell can be saved, and these redundant calculations account for about 20% of the entire ViL model calculations. However, general computing units such as CPUs and GPUs cannot make good use of this computing feature to optimize ViL inference calculations, resulting in a large number of redundant calculations in ViL inference calculations on these platforms, which in turn leads to low efficiency of ViL inference calculations. Summary of the invention

[0005] In order to overcome the problems of low computing resource utilization and high deployment cost caused by directly deploying the ViL model on a GPU in the prior art, the present invention provides an efficient Vision-LSTM hardware accelerator based on FPGA.

[0006] The primary purpose of the present invention is to solve the above technical problems. The technical solution of the present invention is as follows:

[0007] The present invention provides an efficient Vision-LSTM hardware accelerator based on FPGA, the accelerator comprising:

[0008] A multi-head mLSTM cell computing unit, wherein the multi-head mLSTM cell computing unit includes a plurality of mLSTM cell computing units connected in parallel;

[0009] The mLSTM cell calculation unit includes a QKMatCalc calculation subunit, a CVMatCalc calculation subunit, a DMatCalc calculation subunit, a CMatCalc calculation subunit, a first Norm calculation subunit, and a V pipeline buffer unit;

[0010] Among them, the Gate signal is input into the DMatCalc calculation subunit, and the DMatCalc calculation result is output;

[0011] Input the matrix K and matrix Q into the QKMatCalc calculation subunit, and output the QKMatCalc calculation result;

[0012] Input the matrix V into the V pipeline buffer unit and output the V pipeline buffer result;

[0013] Input the calculation results of DMatCalc and QKMatCalc into the CMatCalc calculation subunit, and output the calculation results of CMatCalc;

[0014] Input the CMatCalc calculation result and the V pipeline buffer result into the CVMatCalc calculation subunit, and output the CVMatCalc calculation result;

[0015] The CVMatCalc calculation result is input into the first Norm calculation subunit, and the first Norm calculation result is output.

[0016] Furthermore, the QKMatCalc calculation subunit includes a K buffer, a Q buffer, several P-VPU modules, and an output buffer; the calculation flow adopts a K-stationary method, stores the value of the matrix K in each P-VPU calculation structure through the K buffer, flows the value of the matrix Q between each P-VPU calculation structure through the Q buffer, and connects the matrix Q with the matrix K T The result is output through the output buffer.

[0017] Furthermore, the CVMatCalc calculation subunit includes a C buffer, a V buffer, several packed row accumulation units, and an output buffer; using a C-stationary method, the value of the matrix C is stored in each packed row accumulation unit through the C buffer, and the value of the matrix V flows between each packed row accumulation unit through the V buffer, thereby performing matrix multiplication on the matrix C and the matrix V, and the result is output through the output buffer.

[0018] Further, the hardware accelerator also includes an external storage access interface, a second Norm calculation unit, an UpProj linear calculation unit, a residual connection writing unit, a ConvACT sequence two-dimensional convolution unit, a QKVProj linear calculation unit, an IFGate linear calculation unit, and an X pipeline buffer unit;

[0019] Wherein, the storage read data stream of the external storage access interface is input into the second Norm calculation unit through the input read port, and the second Norm calculation result is output;

[0020] The weight stream of the external storage access interface is input into the residual connection write unit, and the processed external weight stream is output. The processed external weight stream and the second Norm calculation result are input into the UpProj linear calculation unit, and the UpProj linear calculation result and the storage write data stream are output. The storage write data stream is input into the external storage access interface through the residual connection write unit;

[0021] Input the processed external weight stream and UpProj linear calculation results into the ConvACT sequence two-dimensional convolution unit to obtain the ConvACT sequence two-dimensional convolution result, and input the ConvACT sequence two-dimensional convolution result into the external storage access interface through the residual connection write unit as storage write data;

[0022] Input the UpProj linear calculation result into the X pipeline buffer unit to obtain the X pipeline buffer result;

[0023] Input the X pipeline buffer result and the two-dimensional convolution result of the ConvACT sequence into the QKVProj linear calculation unit, output the QKVProj linear calculation result, input the matrix K and matrix Q in the QKVProj linear calculation result into the QKMatCalc calculation subunit input of the multi-head mLSTM cell calculation unit, and input the matrix V in the QKVProj linear calculation result into the V pipeline buffer input of the multi-head mLSTM cell calculation unit;

[0024] The weight flow of the external storage access interface and the linear calculation result of QKVProj are input into the IFGate linear calculation unit, the IFGate linear calculation result is output, and the IFGate linear calculation result is input as the Gate signal into the DMatCalc calculation sub-unit input end of the multi-head mLSTM cell calculation unit.

[0025] Furthermore, the processing of the input data by the first Norm calculation subunit and the second Norm calculation unit includes: calculating the mean, calculating the standard deviation, and linear output.

[0026] Furthermore, the processing of input data by the ConvACT sequence two-dimensional convolution unit includes row buffering, convolution kernel, and SiLU calculation.

[0027] Furthermore, the hardware accelerator further includes a residual connection reading unit, a DownProj linear calculation unit, a first residual connection calculation unit, and a second residual connection calculation unit;

[0028] Input the weight stream and storage read data stream of the external storage access interface into the residual connection read unit, output the processed weight stream and the processed storage read data stream, input the processed weight stream into the weight buffer of the first residual connection calculation unit, perform element-by-element multiplication operation with the processed storage read data stream, and obtain the element-by-element multiplication operation result;

[0029] Input the first Norm calculation result into the first residual connection calculation unit, perform element-by-element addition operation on the element-by-element multiplication operation result, and obtain the element-by-element addition operation result;

[0030] Perform SiLU calculation on the processed weight stream, and perform element-by-element addition operation on the result obtained and the result of element-by-element addition operation to obtain the first residual connection calculation result;

[0031] Input the first residual connection calculation result and the processed weight stream into the DownProj linear calculation unit, and output the DownProj linear calculation result;

[0032] The DownProj linear calculation result and the processed storage read data stream are input into the second residual connection calculation unit, and the output is input into the external storage access interface through the output write port as the storage write data stream.

[0033] Furthermore, the UpProj linear calculation unit, the QKVProj linear calculation unit, the IFGate linear calculation unit, and the DownProj linear calculation unit each include one or more P-MPU structures.

[0034] Further, the P-MPU structure includes several parallel connected P-VPU modules, weight buffers, activation value buffers, accumulation tree modules, and output buffers, and the P-VPU module is a cascade calculation submodule with a level of N based on a packed DSP MAC calculation method;

[0035] The activation value is input into the activation value buffer for processing to obtain activation value buffer data, the activation value buffer data is input into several parallel connected P-VPU modules, the weight value in the weight buffer is combined for calculation, the calculation result is input into the accumulation tree module for accumulation operation, and output through the output buffer.

[0036] Furthermore, a lookup table method is used to process nonlinear function calculations in the calculation process.

[0037] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0038] The inventors proposed an efficient Vision-LSTM hardware accelerator based on FPGA, which can realize efficient reasoning calculation of ViL model. Compared with GPU benchmark, it achieves 0.90~1.19x performance and 26.80x~29.57x improvement in energy efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to make the purpose and technical solution of the present invention clearer, the present invention provides the following drawings and descriptions:

[0040] Figure 1 A schematic diagram of a multi-head mLSTM cell computing unit provided in an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of the Vision-LSTM model;

[0042] Figure 3 This is a schematic diagram of the mLSTM layer structure in the Vision-LSTM model;

[0043] Figure 4 A schematic diagram of analyzing the amount of computation that can be saved in the ViL layer provided in an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of the calculation load distribution of the QKMatCalc calculation subunit provided in an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of the structure of a QKMatCalc calculation subunit provided in an embodiment of the present invention;

[0046] Figure 7A schematic diagram of the structure of the CVMatCalc calculation subunit provided in an embodiment of the present invention;

[0047] Figure 8 A schematic diagram of a packed row accumulation unit in a CVMatCalc calculation subunit provided in an embodiment of the present invention;

[0048] Fig. 9 A schematic diagram of computing load distribution of the CVMatCalc computing subunit provided in an embodiment of the present invention;

[0049] Fig.10 A schematic diagram of the structure of a hardware accelerator provided by an embodiment of the present invention;

[0050] Fig.11 A schematic diagram of a P-VPU computing structure provided in an embodiment of the present invention;

[0051] Fig.12 A schematic diagram of data input of a P-VPU computing structure provided in an embodiment of the present invention;

[0052] Fig.13 A schematic diagram of a P-MPU structure provided in an embodiment of the present invention;

[0053] Fig.14 A schematic diagram of data flow of a hardware accelerator provided by an embodiment of the present invention;

[0054] Fig.15 A schematic diagram of a partial on-chip residual buffer flow provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0057] The present invention provides an efficient Vision-LSTM hardware accelerator based on FPGA, such as Figure 1 The figure shows a structure diagram of an efficient Vision-LSTM hardware accelerator system based on FPGA, wherein the accelerator includes:

[0058] A multi-head mLSTM cell computing unit, wherein the multi-head mLSTM cell computing unit includes a plurality of mLSTM cell computing units connected in parallel;

[0059] The mLSTM cell calculation unit includes a QKMatCalc calculation subunit, a CVMatCalc calculation subunit, a DMatCalc calculation subunit, a CMatCalc calculation subunit, a first Norm calculation subunit, and a V pipeline buffer unit.

[0060] Among them, the Gate signal is input into the DMatCalc calculation subunit, and the DMatCalc calculation result is output;

[0061] Input the matrix K and matrix Q into the QKMatCalc calculation subunit, and output the QKMatCalc calculation result;

[0062] Input the matrix V into the V pipeline buffer unit and output the V pipeline buffer result;

[0063] Input the calculation results of DMatCalc and QKMatCalc into the CMatCalc calculation subunit, and output the calculation results of CMatCalc;

[0064] Input the CMatCalc calculation result and the V pipeline buffer result into the CVMatCalc calculation subunit, and output the CVMatCalc calculation result;

[0065] The CVMatCalc calculation result is input into the first Norm calculation subunit, and the first Norm calculation result is output.

[0066] The DMatCalc is an abbreviation of D-Matrix-Calculation, the CMatCalc is an abbreviation of CM-Matrix-Calculation, the QKMatCalc is an abbreviation of QK-Matrix-Calculation, and the CVMatCalc is an abbreviation of CV-Matrix-Calculation.

[0067] In the calculation of mLSTM Cell, assume that the sequence dimension inside each head is s, f and i are the forget gate input and input gate input of the complete sequence, Q, K, V are s×d-dimensional inputs, and the calculation process is roughly as shown in the following formula, defined:

[0068]

[0069] Among them, F ij is a matrix, i and j are the indices of the matrix, σ(·) is the sigmoid function, fk is the gating function.

[0070]

[0071] Among them, I ij is a matrix, i and j are the indices of the matrix, and i_j represents the control parameters related to the input / hidden state of the jth step.

[0072] Taking the logarithm of the above matrix F and adding it to the matrix I, we get:

[0073] D=logF+I

[0074] Perform numerical stabilization on D in softmax and get:

[0075] D′=exp(D-max(D))

[0076] Among them, D′ is a non-negative matrix after masking and maintains the lower triangular structure. Combining the similarity score of QKT with the lower triangular structure of D′, we get:

[0077]

[0078] Among them, ⊙ represents element-by-element multiplication. After normalization, the expression is as follows:

[0079]

[0080] By multiplying the normalized attention weights C with the value matrix V, we get the output:

[0081] H=CV

[0082] From the above formula, we can see that the matrix D is actually a lower triangular matrix, which leads to the fact that in the calculation of the C′ matrix, only the calculation of the lower triangular part is valid, while the calculation of the upper triangular part is an unnecessary calculation multiplied by 0, which means that QK T The calculation of only needs to calculate the lower triangular part, which directly saves about half of the calculation amount for this step. The lower triangular calculation exists in the form of linear calculation (which can be disassembled into vector inner product). In addition, since the C matrix is ​​also a lower triangular matrix, this feature can also be used in the calculation of H = CV to save about half of the calculation amount.

[0083] like Figure 4 As shown, the calculations of UpProj, DownProj, QKMatCalc, CVMatCalc, etc. in the figure can save about 20% of the calculation amount for the entire ViLlayer by utilizing the characteristics of the above lower triangular calculations.

[0084] More specifically, the QKMatCalc calculation subunit includes a K buffer, a Q buffer, several P-VPU modules, and an output buffer; the lower triangular matrix calculation design of the QKMatCalc calculation subunit: in the four heads of the mLSTM cell, the matrix Q of size s×d and the matrix k of size d×s T Multiply and perform the operation MatQK=QK T , the Q matrix is ​​expressed as Q = (q 0 ,q 1 ,…,q s-1 ) T , K matrix is ​​expressed as K = (k 0 ,k 1 ,…,k s-1 ) T , then the calculation can be expressed as This means that several P-VPUs can be used to process the calculation in parallel, and only elements i≤j need to be calculated. Therefore, the calculation flow can be designed to achieve the calculation with only half of the computing resources while the effective computing throughput remains unchanged. In order to ensure high data reuse rate during calculation, the K-stationary calculation flow design is adopted. During the calculation process, the values ​​of the K matrix are stored in each computing unit, and only the values ​​of the Q matrix flow between the computing units, so as to reduce the data movement between the computing units, thereby reducing the power consumption of the overall hardware computing unit and improving the data reuse rate during calculation.

[0085] by Figure 5 The computing load is distributed to each P-VPU in a certain way to ensure the high utilization rate of each P-VPU. Figure 6 The QKMatCalc calculation subunit is designed in the manner shown to achieve efficient calculation of this part.

[0086] like Figure 1 As shown, the DMatCalc calculation subunit includes a Gate value accumulation module, a row maximum value module, a normalization module, an exp calculation module, and an output buffer module, wherein input data passes through the Gate value accumulation module, and the output accumulation result data is respectively input into the row maximum value module and the normalization module; the output of the row maximum value module and the accumulation result data pass through the normalization module and the exp calculation module in sequence, and are output together with the output of the row maximum value module through the output buffer module.

[0087] The CMatCalc calculation subunit includes an element-by-element multiplication module, a row maximum value module, a normalization module, and an output buffer module; the element-by-element multiplication module performs element-by-element multiplication calculation processing on the outputs of the DMatCalc calculation subunit and the QKMatCalc calculation subunit respectively, and inputs the obtained results into the row maximum value module for row maximum value calculation processing, and inputs the output of the row maximum value module and the output of the element-by-element multiplication module into the normalization module for normalization processing, and the normalization processing result is output through the output buffer module.

[0088] like Figure 7 As shown, the CVMatCalc calculation subunit includes a C buffer, a V buffer, several packed row accumulation units, and an output buffer; using the C-stationary method, the values ​​of the matrix C with a size of s×s are stored in each packed row accumulation unit through the C buffer, and the values ​​of the matrix V with a size of s×d flow between each packed row accumulation unit through the V buffer, so that the matrix C is multiplied by the matrix V to obtain an output matrix H with a size of s×d and output it through the output buffer.

[0089] The H matrix is ​​represented as H = (h 0 ,h 1 ,h 2 ,…,h s-1 ) T , V matrix is ​​expressed as V = (v 0 ,v 1 ,v 2 ,…,v s-1 ) T , c ij is represented as the jth element in the i-th row of the c matrix, then the calculation can be expressed as That is, each row of the V matrix is ​​divided into ij To improve the calculation efficiency, this embodiment can make c ij v j and c kl v j With (c ij ×2 a +c kl )v j , and use the packed row accumulation unit for calculation, such as Figure 8 Similarly, in order to ensure high data reuse rate during calculation, the C-stationary calculation flow design is adopted. During the calculation process, the values ​​of the C matrix are stored in each packed row accumulation unit, and only the values ​​of the V matrix flow between each packed row accumulation unit. Fig. 9 The computing load is distributed to each package row accumulation unit in a manner to ensure high utilization of each package row accumulation unit.

[0090] More specifically, if Fig.10 As shown, the hardware accelerator also includes an external storage access interface, a second Norm calculation unit, an UpProj linear calculation unit, a residual connection write unit, a ConvACT sequence two-dimensional convolution unit, a QKVProj linear calculation unit, an IFGate linear calculation unit, and an X pipeline buffer unit;

[0091] Wherein, the storage read data stream of the external storage access interface is input into the second Norm calculation unit through the input read port, and the second Norm calculation result is output;

[0092] The weight stream of the external storage access interface is input into the residual connection write unit, and the processed external weight stream is output. The processed external weight stream and the second Norm calculation result are input into the UpProj linear calculation unit, and the UpProj linear calculation result and the storage write data stream are output. The storage write data stream is input into the external storage access interface through the residual connection write unit;

[0093] Input the processed external weight stream and UpProj linear calculation results into the ConvACT sequence two-dimensional convolution unit to obtain the ConvACT sequence two-dimensional convolution result, and input the ConvACT sequence two-dimensional convolution result into the external storage access interface through the residual connection write unit as storage write data;

[0094] More specifically, the processing performed by the ConvACT sequence two-dimensional convolution unit on the input data includes line buffering, convolution kernel, and SiLU calculation;

[0095] Input the UpProj linear calculation result into the X pipeline buffer unit to obtain the X pipeline buffer result;

[0096] Input the X pipeline buffer result and the two-dimensional convolution result of the ConvACT sequence into the QKVProj linear calculation unit, output the QKVProj linear calculation result, input the matrix K and matrix Q in the QKVProj linear calculation result into the QKMatCalc calculation subunit input of the multi-head mLSTM cell calculation unit, and input the matrix V in the QKVProj linear calculation result into the V pipeline buffer input of the multi-head mLSTM cell calculation unit;

[0097] The weight flow of the external storage access interface and the linear calculation result of QKVProj are input into the IFGate linear calculation unit, the IFGate linear calculation result is output, and the IFGate linear calculation result is input as the Gate signal into the DMatCalc calculation sub-unit input end of the multi-head mLSTM cell calculation unit.

[0098] More specifically, the IFGate linear calculation unit includes a P-MPUIGate module, a P-MPUFGate module, and an LS calculation module. The output of the QKVProj linear calculation unit is respectively input into the P-MPUIGate module and the P-MPUFGate module of the IFGate linear calculation unit, and the P-MPUIGate calculation result and the P-MPUFGate calculation result are output respectively. The P-MPUFGate calculation result is input into the LS calculation module, and the LS calculation result is output. The LS calculation result and the P-MPUIGate calculation result are output through the data path.

[0099] More specifically, the processing of the input data by the first Norm calculation subunit and the second Norm calculation unit includes: calculating the mean, calculating the standard deviation, and linear output.

[0100] More specifically, the hardware accelerator further includes a residual connection reading unit, a DownProj linear calculation unit, a first residual connection calculation unit, and a second residual connection calculation unit;

[0101] Input the weight stream and storage read data stream of the external storage access interface into the residual connection read unit, output the processed weight stream and the processed storage read data stream, input the processed weight stream into the weight buffer of the first residual connection calculation unit, perform element-by-element multiplication operation with the processed storage read data stream, and obtain the element-by-element multiplication operation result;

[0102] Input the first Norm calculation result into the first residual connection calculation unit, perform element-by-element addition operation on the element-by-element multiplication operation result, and obtain the element-by-element addition operation result;

[0103] Perform SiLU calculation on the processed weight stream, and perform element-by-element addition operation on the result obtained and the result of element-by-element addition operation to obtain the first residual connection calculation result;

[0104] Input the first residual connection calculation result and the processed weight stream into the DownProj linear calculation unit, and output the DownProj linear calculation result;

[0105] The DownProj linear calculation result and the processed storage read data stream are input into the second residual connection calculation unit, and the output is input into the external storage access interface through the output write port as the storage write data stream.

[0106] More specifically, the UpProj linear calculation unit, the QKVProj linear calculation unit, the IFGate linear calculation unit, and the DownProj linear calculation unit each include one or more P-MPU structures.

[0107] More specifically, the P-MPU structure includes several parallel connected P-VPU modules, weight buffers, activation value buffers, accumulation tree modules, and output buffers, and the P-VPU module is a cascade calculation submodule with a level of N based on a packed DSP MAC calculation method;

[0108] The activation value is input into the activation value buffer for processing to obtain activation value buffer data, the activation value buffer data is input into several parallel connected P-VPU modules, the weight value in the weight buffer is combined for calculation, the calculation result is input into the accumulation tree module for accumulation operation, and output through the output buffer.

[0109] For the linear calculation part, it can basically be decomposed into several vector-matrix multiplications, and the vector-matrix multiplication can be further decomposed into several pairs of vector inner product operations. Consider the multiplication of the N-dimensional activation value vector a and the N×P-dimensional weight matrix W, where W = (w 1 ,w 2 ,w 3 ,…,w P ), then the calculation can be decomposed into P N-dimensional vector inner products o i =aw i At this point, a cascade calculation submodule with N levels based on the packed DSP MAC calculation method can be created, forming the following Fig.11 A P-VPU (packed vector process unit) is shown, and Fig.12 The input is sent into the computing structure in a manner to perform packed vector inner product calculations, achieving a computing throughput of completing two vector inner product operations in a single cycle, which can greatly improve the utilization of hardware resources and the computing throughput of each computing unit.

[0110] On the basis of P-VPU, when a large-dimensional packed vector inner product calculation that cannot be handled by a single P-VPU occurs, the calculation submodules can be expanded in parallel to form the following Fig.13 A larger P-MPU unit is used to handle this situation, so as to handle the inner product calculation of packed vectors of arbitrary dimension N.

[0111] More specifically, a lookup table method is used to process nonlinear function calculations in the calculation process, and the nonlinear function calculations include SiLU calculations, normalization calculations, and exp calculations.

[0112] Since there are a large number of nonlinear functions, such as SiLU function, logsigmoid function and exp function, directly performing these nonlinear functions will make the hardware complexity of these parts higher. Under the low-precision neural network calculation load, the input bit width and output bit width of these nonlinear function calculations can be determined. On this basis, for the nonlinear function calculations with not too large bit widths, this embodiment can be implemented in the form of a lookup table (LUT) to achieve nonlinear function calculations with low hardware complexity and low resource consumption.

[0113] In the calculation of the ViL Layer, there are several residual connections with large spans. In the hardware accelerator design, due to the data dependency caused by the residual connection mechanism, the current calculation data must be retained until the calculation of the residual connection is completed. In the conventional streaming accelerator design, this means that a large amount of hardware storage resources are consumed to achieve pipeline buffering of these data. When the span of the residual connection that needs to be buffered is large to a certain extent, the amount of hardware storage resources required for this design is unacceptable. Therefore, this embodiment proposes a half-layer mapping method, which can use external storage as a buffering medium to avoid excessive hardware storage resource requirements caused by residual connections. Figure 7 As shown, for each large-span residual connection that needs to be processed, we can only do partial data buffering on the chip, and temporarily store the data in external storage during the rest of the time, so as to process the large-span residual connection in the ViL layer calculation with acceptable hardware storage resource consumption.

[0114] The present invention designs an accelerator pipeline data flow and builds a complete ViL hardware accelerator, such as Fig.10 , and completed design verification and performance comparison on the Xilinx Alveo U250 platform.

[0115] In this embodiment, if Fig.10 As shown, the accelerator architecture of the present invention achieves efficient processing of each part of the computing task by customizing specialized hardware processing units for different computing tasks. The present invention adopts a full pipeline design to build the entire accelerator to ensure that the system can maintain a high throughput. The working method of the accelerator is as follows.

[0116] When the computation starts, the input data storage and reading unit will send the data into the accelerator pipeline in the form of a stream, such as Fig.14 As shown, the input tensor X of size S×D will be split into several row vector units of size 1×u as a calculation fragment; and this will be used as the unit for calculating input and output, such as Fig.15As shown, D must be divisible by u and is fed into the accelerator pipeline sequentially in each clock cycle. Therefore, the throughput of the system can be expressed as follows:

[0117]

[0118] In order to ensure that there is no data blocking in the hardware accelerator pipeline, the data and computing throughput of all hardware modules in the accelerator pipeline are designed according to this standard.

[0119] During the pipeline calculation process, the data related to some large-span residual links will be sent to the external storage for temporary storage through the residual link storage write unit. In the subsequent stages of the pipeline calculation, these data will be read back through the residual link storage read unit and the calculation related to the residual connection will be completed. After all calculations are completed, the data stream will be sent to the output storage write unit to be written back to the external storage.

[0120] This embodiment uses Verilog HDL to deploy QVil, and uses Vivado 2021.1 to deploy it to the XilinxFPGA Alveo U250 platform, and implements it at a target frequency of 240MHz. The corresponding resource utilization and power consumption are shown in Table 1. LUT (Look-Up Table), FF (Flip-Flop), DSP (Digital Signal Processor), BRAM (Block RAM) are resource utilization, reflecting the design's use of FPGA hardware resources, Freq. (Frequency), Power (power consumption), and Throughput (throughput) are performance indicators, indicating the accelerator's operating performance and power consumption characteristics. The hardware accelerator uses a computational fragment of size U=4 for pipeline design. In order to reduce the impact of the delay in accessing external memory on the hardware pipeline, all residual connection external storage read / write modules use pingpong buffer when accessing external storage.

[0121] Table 1

[0122]

[0123] Since Vision-LSTM currently does not work with other FPGA or ASIC hardware accelerators, CPU and GPU will be used as benchmarks and performance will be compared with them. This example uses the pre-trained model vil2_tiny16_e400_in1k in the pytorch2.3.1, CUDA12.1.0 environment to perform CPU and GPU reasoning of the ViL model. Among them, the CPU benchmark is AMD Ryzen ThreadRipper 3960X, and the GPU baseline is NVIDIA RTX 3090. The experimental results are shown in Table 2, where Batchsize is the amount of data processed by each computing task, Throughput is the throughput, ApproximatePower is the power consumption during operation, and Performance / Power is the energy efficiency ratio.

[0124] The GPU power consumption is measured using nvidia-smi, and the CPU power consumption is measured using the ryzen-power tool.

[0125] Table 2

[0126]

[0127] From the comparison results, it can be concluded that the hardware accelerator designed by the present invention has a great improvement over the CPU benchmark, and compared with the GPU benchmark, it has achieved a performance of 0.90X to 1.19X, and has an improvement of 26.80X to 29.57X in energy efficiency.

[0128] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An efficient Vision-LSTM hardware accelerator based on FPGA, characterized in that: The accelerator comprises: A multi-head mLSTM cell computing unit, wherein the multi-head mLSTM cell computing unit includes a plurality of mLSTM cell computing units connected in parallel; The mLSTM cell calculation unit includes a QKMatCalc calculation subunit, a CVMatCalc calculation subunit, a DMatCalc calculation subunit, a CMatCalc calculation subunit, a first Norm calculation subunit, and a V pipeline buffer unit; Among them, the Gate signal is input into the DMatCalc calculation subunit, and the DMatCalc calculation result is output; Input the matrix K and matrix Q into the QKMatCalc calculation subunit, and output the QKMatCalc calculation result; Input the matrix V into the V pipeline buffer unit and output the V pipeline buffer result; Input the calculation results of DMatCalc and QKMatCalc into the CMatCalc calculation subunit, and output the calculation results of CMatCalc; Input the CMatCalc calculation result and the V pipeline buffer result into the CVMatCalc calculation subunit, and output the CVMatCalc calculation result; The CVMatCalc calculation result is input into the first Norm calculation subunit, and the first Norm calculation result is output.

2. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 1, characterized in that: The QKMatCalc calculation subunit includes a K buffer, a Q buffer, several P-VPU modules, and an output buffer; the calculation flow adopts a K-stationary method, stores the value of the matrix K in each P-VPU calculation structure through the K buffer, flows the value of the matrix Q between each P-VPU calculation structure through the Q buffer, and connects the matrix Q with the matrix K T The result is output through the output buffer.

3. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 1, characterized in that: The CVMatCalc calculation subunit includes a C buffer, a V buffer, several packed row accumulation units, and an output buffer; using the C-stationary method, the value of the matrix C is stored in each packed row accumulation unit through the C buffer, and the value of the matrix V flows between each packed row accumulation unit through the V buffer, so that the matrix C is multiplied by the matrix V, and the result is output through the output buffer.

4. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 1, characterized in that: The hardware accelerator also includes an external storage access interface, a second Norm calculation unit, an UpProj linear calculation unit, a residual connection writing unit, a ConvACT sequence two-dimensional convolution unit, a QKVProj linear calculation unit, an IFGate linear calculation unit, and an X pipeline buffer unit; Wherein, the storage read data stream of the external storage access interface is input into the second Norm calculation unit through the input read port, and the second Norm calculation result is output; The weight stream of the external storage access interface is input into the residual connection write unit, and the processed external weight stream is output. The processed external weight stream and the second Norm calculation result are input into the UpProj linear calculation unit, and the UpProj linear calculation result and the storage write data stream are output. The storage write data stream is input into the external storage access interface through the residual connection write unit; Input the processed external weight stream and UpProj linear calculation results into the ConvACT sequence two-dimensional convolution unit to obtain the ConvACT sequence two-dimensional convolution result, and input the ConvACT sequence two-dimensional convolution result into the external storage access interface through the residual connection write unit as storage write data; Input the UpProj linear calculation result into the X pipeline buffer unit to obtain the X pipeline buffer result; Input the X pipeline buffer result and the two-dimensional convolution result of the ConvACT sequence into the QKVProj linear calculation unit, output the QKVProj linear calculation result, input the matrix K and matrix Q in the QKVProj linear calculation result into the QKMatCalc calculation subunit input of the multi-head mLSTM cell calculation unit, and input the matrix V in the QKVProj linear calculation result into the V pipeline buffer input of the multi-head mLSTM cell calculation unit; The weight flow of the external storage access interface and the linear calculation result of QKVProj are input into the IFGate linear calculation unit, the IFGate linear calculation result is output, and the IFGate linear calculation result is input as the Gate signal into the DMatCalc calculation sub-unit input end of the multi-head mLSTM cell calculation unit.

5. An efficient Vision-LSTM hardware accelerator based on FPGA according to claim 1 or 4, characterized in that: The processing of the input data by the first Norm calculation subunit and the second Norm calculation unit includes: calculating the mean, calculating the standard deviation, and linear output.

6. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 4, characterized in that: The processing of input data by the ConvACT sequence two-dimensional convolution unit includes row buffering, convolution kernel, and SiLU calculation.

7. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 4, characterized in that: The hardware accelerator further includes a residual connection reading unit, a DownProj linear calculation unit, a first residual connection calculation unit, and a second residual connection calculation unit; Input the weight stream and storage read data stream of the external storage access interface into the residual connection read unit, output the processed weight stream and the processed storage read data stream, input the processed weight stream into the weight buffer of the first residual connection calculation unit, perform element-by-element multiplication operation with the processed storage read data stream, and obtain the element-by-element multiplication operation result; Input the first Norm calculation result into the first residual connection calculation unit, perform element-by-element addition operation on the element-by-element multiplication operation result, and obtain the element-by-element addition operation result; Perform SiLU calculation on the processed weight stream, and perform element-by-element addition operation on the result obtained and the result of element-by-element addition operation to obtain the first residual connection calculation result; Input the first residual connection calculation result and the processed weight stream into the DownProj linear calculation unit, and output the DownProj linear calculation result; The DownProj linear calculation result and the processed storage read data stream are input into the second residual connection calculation unit, and the output is input into the external storage access interface through the output write port as the storage write data stream.

8. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 7, characterized in that: The UpProj linear calculation unit, the QKVProj linear calculation unit, the IFGate linear calculation unit, and the DownProj linear calculation unit all include one or more P-MPU structures.

9. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 8, characterized in that: The P-MPU structure includes several parallel connected P-VPU modules, weight buffers, activation value buffers, accumulation tree modules, and output buffers. The P-VPU module is a cascade calculation submodule with a level of N based on a packed DSP MAC calculation method. The activation value is input into the activation value buffer for processing to obtain activation value buffer data, the activation value buffer data is input into several parallel connected P-VPU modules, the weight value in the weight buffer is combined for calculation, the calculation result is input into the accumulation tree module for accumulation operation, and output through the output buffer.

10. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 1, characterized in that: The nonlinear function calculation in the calculation process is processed by using the lookup table method.

Images