An efficient FPGA-based vision-lstm hardware accelerator

Abstract

The application discloses a kind of high-efficiency Vision-LSTM hardware accelerator based on FPGA, comprising: multi-head mLSTM cell computing unit, the multi-head mLSTM cell computing unit includes several mLSTM cell computing unit parallel connection, the mLSTM cell computing unit includes QKMatCalc computing subunit, CVMatCalc computing subunit, DMatCalc computing subunit, CMatCalc computing subunit, first Norm computing subunit, V pipeline buffer unit, can realize the efficient inference calculation of ViL model, compared with GPU benchmark, it reaches 0.90~1.19x performance, and there is 26.80x~29.57x improvement in energy efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of FPGA data processing technology, and more specifically, relates to an efficient Vision-LSTM hardware accelerator based on FPGA. Background Art

[0002] Currently, if Figure 2 、 Figure 3 As shown in Figure 2, the Vision-LSTM (ViL) model, based on the xLSTM building block, has attracted considerable attention due to its linear computational and storage complexity, as well as its outstanding performance in computer vision fields such as medical imaging. In the ImageNet-1k recognition task, the Vision-LSTM (ViL) model even outperformed models based on the SSM and the heavily optimized ViT model. Therefore, the ViL model is considered a promising model for vision tasks.

[0003] However, due to the large number of different types of computing units in the ViL model, and the significant differences in the computing modes of each computing unit, its computational flow is quite complex. In addition, there are a considerable number of large-span residual connection mechanisms in the ViL model, which results in a large amount of data dependency in its computational process, further increasing the computational complexity. Therefore, general-purpose computing units such as CPUs and GPUs cannot perform ViL inference calculations efficiently. In the actual operation test process, this problem is directly manifested in the excessive processing time consumed in the computational part with lower computational complexity. Directly deploying the ViL model on the GPU will result in high deployment costs.

[0004] Furthermore, during the ViL model's inference process, the mLSTM cells within it perform a significant amount of lower triangular matrix calculations. This means that nearly half of the computational effort in the mLSTM cells could be saved, representing approximately 20% of the total ViL model computation. However, general-purpose computing units such as CPUs and GPUs cannot effectively utilize this computational feature to optimize ViL inference. This results in a significant amount of redundant computation on these platforms, leading to low ViL inference efficiency. Summary of the Invention

[0005] In order to overcome the problems of low computing resource utilization and high deployment cost caused by directly deploying the ViL model on a GPU in the prior art, the present invention provides an efficient Vision-LSTM hardware accelerator based on FPGA.

[0006] The primary purpose of the present invention is to solve the above technical problems, and the technical solutions of the present invention are as follows:

[0007] The present invention provides an efficient Vision-LSTM hardware accelerator based on FPGA, the accelerator comprising:

[0008] A multi-head mLSTM cell computing unit, wherein the multi-head mLSTM cell computing unit includes several mLSTM cell computing units connected in parallel;

[0009] The mLSTM cell calculation unit includes a QKMatCalc calculation subunit, a CVMatCalc calculation subunit, a DMatCalc calculation subunit, a CMatCalc calculation subunit, a first Norm calculation subunit, and a V pipeline buffer unit;

[0010] Among them, the Gate signal is input into the DMatCalc calculation subunit, and the DMatCalc calculation result is output;

[0011] Input matrix K and matrix Q into the QKMatCalc calculation subunit and output the QKMatCalc calculation result;

[0012] Input the matrix V into the V pipeline buffer unit and output the V pipeline buffer result;

[0013] Input the calculation results of DMatCalc and QKMatCalc into the CMatCalc calculation subunit, and output the calculation results of CMatCalc;

[0014] Input the CMatCalc calculation result and the V pipeline buffer result into the CVMatCalc calculation subunit, and output the CVMatCalc calculation result;

[0015] The CVMatCalc calculation result is input into the first Norm calculation subunit, and the first Norm calculation result is output.

[0016] Furthermore, the QKMatCalc calculation subunit includes a K buffer, a Q buffer, several P-VPU modules, and an output buffer; the calculation flow adopts a K-stationary method, storing the value of the matrix K in each P-VPU calculation structure through the K buffer, and flowing the value of the matrix Q between each P-VPU calculation structure through the Q buffer, and combining the matrix Q with the matrix K. T The result is output through the output buffer.

[0017] Furthermore, the CVMatCalc calculation subunit includes a C buffer, a V buffer, several packed row accumulation units, and an output buffer; using the C-stationary method, the value of the matrix C is stored in each packed row accumulation unit through the C buffer, and the value of the matrix V flows between each packed row accumulation unit through the V buffer, thereby performing matrix multiplication on the matrix C and the matrix V, and the result is output through the output buffer.

[0018] Furthermore, the hardware accelerator also includes an external storage access interface, a second Norm calculation unit, an UpProj linear calculation unit, a residual connection write unit, a ConvACT sequence two-dimensional convolution unit, a QKVProj linear calculation unit, an IFGate linear calculation unit, and an X pipeline buffer unit;

[0019] The storage read data stream of the external storage access interface is input into the second Norm calculation unit through the input read port, and the second Norm calculation result is output;

[0020] The weight stream of the external storage access interface is input into the residual connection write unit, which outputs the processed external weight stream. The processed external weight stream and the second Norm calculation result are input into the UpProj linear calculation unit, which outputs the UpProj linear calculation result and the storage write data stream. The storage write data stream is input into the external storage access interface through the residual connection write unit;

[0021] Input the processed external weight stream and UpProj linear calculation results into the ConvACT sequence two-dimensional convolution unit to obtain the ConvACT sequence two-dimensional convolution result, and input the ConvACT sequence two-dimensional convolution result into the external storage access interface through the residual connection write unit as storage write data;

[0022] Input the UpProj linear calculation result into the X pipeline buffer unit to obtain the X pipeline buffer result;

[0023] Input the X pipeline buffer result and the two-dimensional convolution result of the ConvACT sequence into the QKVProj linear calculation unit, output the QKVProj linear calculation result, input the matrix K and matrix Q in the QKVProj linear calculation result into the QKMatCalc calculation sub-unit input of the multi-head mLSTM cell calculation unit, and input the matrix V in the QKVProj linear calculation result into the V pipeline buffer input of the multi-head mLSTM cell calculation unit;

[0024] The weight flow of the external storage access interface and the QKVProj linear calculation result are input into the IFGate linear calculation unit, the IFGate linear calculation result is output, and the IFGate linear calculation result is input as the Gate signal into the DMatCalc calculation sub-unit input end of the multi-head mLSTM cell calculation unit.

[0025] Furthermore, the processing of the input data by the first Norm calculation subunit and the second Norm calculation unit includes: calculating the mean, calculating the standard deviation, and linear output.

[0026] Furthermore, the processing of input data by the ConvACT sequence two-dimensional convolution unit includes row buffering, convolution kernel, and SiLU calculation.

[0027] Furthermore, the hardware accelerator further includes a residual connection reading unit, a DownProj linear calculation unit, a first residual connection calculation unit, and a second residual connection calculation unit;

[0028] Input the weight stream and storage read data stream of the external storage access interface into the residual connection read unit, output the processed weight stream and the processed storage read data stream, input the processed weight stream into the weight buffer of the first residual connection calculation unit, perform element-by-element multiplication operation with the processed storage read data stream, and obtain the element-by-element multiplication operation result;

[0029] Input the first Norm calculation result into the first residual connection calculation unit, perform element-by-element addition operation on the element-by-element multiplication result, and obtain the element-by-element addition operation result;

[0030] Perform SiLU calculation on the processed weight stream, and perform element-by-element addition operation on the result obtained to obtain the first residual connection calculation result;

[0031] Input the first residual connection calculation result and the processed weight stream into the DownProj linear calculation unit, and output the DownProj linear calculation result;

[0032] The DownProj linear calculation result and the processed storage read data stream are input into the second residual connection calculation unit, and the output is input into the external storage access interface through the output write port as the storage write data stream.

[0033] Furthermore, the UpProj linear calculation unit, the QKVProj linear calculation unit, the IFGate linear calculation unit, and the DownProj linear calculation unit each include one or more P-MPU structures.

[0034] Furthermore, the P-MPU structure includes several parallel-connected P-VPU modules, a weight buffer, an activation value buffer, an accumulation tree module, and an output buffer. The P-VPU module is a cascade calculation submodule with N levels based on a packed DSP MAC calculation method.

[0035] The activation value is input into the activation value buffer for processing to obtain activation value buffer data, the activation value buffer data is input into several parallel connected P-VPU modules, and the weight value in the weight buffer is combined for calculation. The calculation result is input into the accumulation tree module for accumulation operation and output through the output buffer.

[0036] Furthermore, a lookup table method is used to process nonlinear function calculations in the calculation process.

[0037] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0038] The inventors proposed an efficient FPGA-based Vision-LSTM hardware accelerator, which can realize efficient inference calculation of the ViL model. Compared with the GPU benchmark, it achieves a performance improvement of 0.90-1.19x and an improvement of 26.80-29.57x in energy efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to make the purpose and technical solution of the present invention clearer, the present invention provides the following drawings and descriptions:

[0040] Figure 1 Schematic diagram of a multi-head mLSTM cell computing unit provided in an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of the Vision-LSTM model;

[0042] Figure 3 This is a schematic diagram of the mLSTM layer structure in the Vision-LSTM model;

[0043] Figure 4 A schematic diagram illustrating analysis of the amount of computation that can be saved in the ViL layer provided in an embodiment of the present invention;

[0044] Figure 5 A schematic diagram illustrating the calculation load distribution of the QKMatCalc calculation subunit provided in an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of the structure of the QKMatCalc calculation subunit provided in an embodiment of the present invention;

[0046] Figure 7A schematic diagram of the structure of the CVMatCalc calculation subunit provided in an embodiment of the present invention;

[0047] Figure 8 A schematic diagram of a packed row accumulation unit in a CVMatCalc calculation subunit provided in an embodiment of the present invention;

[0048] Figure 9 A schematic diagram of the computational load distribution of the CVMatCalc computation subunit provided in an embodiment of the present invention;

[0049] Figure 10 A schematic diagram of the structure of a hardware accelerator provided in an embodiment of the present invention;

[0050] Figure 11 A schematic diagram of the P-VPU computing structure provided in an embodiment of the present invention;

[0051] Figure 12 A schematic diagram of data input for a P-VPU computing structure provided in an embodiment of the present invention;

[0052] Figure 13 A schematic diagram of a P-MPU structure provided in an embodiment of the present invention;

[0053] Figure 14 A schematic diagram of data flow of a hardware accelerator provided by an embodiment of the present invention;

[0054] Figure 15 A schematic diagram of a portion of an on-chip residual buffer flow provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0057] The present invention provides an efficient Vision-LSTM hardware accelerator based on FPGA, such as Figure 1 Figure 2 shows the structure of an efficient FPGA-based Vision-LSTM hardware accelerator system. The accelerator includes:

[0058] A multi-head mLSTM cell computing unit, wherein the multi-head mLSTM cell computing unit includes several mLSTM cell computing units connected in parallel;

[0059] The mLSTM cell calculation unit includes a QKMatCalc calculation subunit, a CVMatCalc calculation subunit, a DMatCalc calculation subunit, a CMatCalc calculation subunit, a first Norm calculation subunit, and a V pipeline buffer unit.

[0060] Among them, the Gate signal is input into the DMatCalc calculation subunit, and the DMatCalc calculation result is output;

[0061] Input matrix K and matrix Q into the QKMatCalc calculation subunit and output the QKMatCalc calculation result;

[0062] Input the matrix V into the V pipeline buffer unit and output the V pipeline buffer result;

[0063] Input the calculation results of DMatCalc and QKMatCalc into the CMatCalc calculation subunit, and output the calculation results of CMatCalc;

[0064] Input the CMatCalc calculation result and the V pipeline buffer result into the CVMatCalc calculation subunit, and output the CVMatCalc calculation result;

[0065] The CVMatCalc calculation result is input into the first Norm calculation subunit, and the first Norm calculation result is output.

[0066] The DMatCalc is an abbreviation of D-Matrix-Calculation, the CMatCalc is an abbreviation of CM-Matrix-Calculation, the QKMatCalc is an abbreviation of QK-Matrix-Calculation, and the CVMatCalc is an abbreviation of CV-Matrix-Calculation.

[0067] In the calculation of the mLSTM Cell computing unit, assume that the sequence dimension within each head is s, f and i are the forget gate input and input gate input of the complete sequence, Q, K, and V are s×d-dimensional inputs. The calculation process is roughly as shown in the following formula and definition:

[0068]

[0069] Among them, F ij is a matrix, i and j are the indices of the matrix, σ(·) is the sigmoid function, fk is the gating function.

[0070]

[0071] Among them, I ij is a matrix, i and j are the indices of the matrix, and i_j represents the control parameters related to the input / hidden state of the jth step.

[0072] Taking the logarithm of the above matrix F and adding it to the matrix I, we get:

[0073] D=logF+I

[0074] Perform numerical stabilization on D in softmax and get:

[0075] D′=exp(D-max(D))

[0076] Where D′ is a non-negative matrix after masking and maintains the lower triangular structure. Combining the similarity score of QKT with the lower triangular structure of D′, we get:

[0077]

[0078] Where ⊙ represents element-by-element multiplication. After normalization, the expression is as follows:

[0079]

[0080] By multiplying the normalized attention weights C with the value matrix V, we get the output:

[0081] H=CV

[0082] From the above formula, we can see that the matrix D is actually a lower triangular matrix, which leads to the fact that in the calculation of the C′ matrix, only the calculation of the lower triangular part is valid, while the calculation of the upper triangular part is an unnecessary calculation multiplied by 0, which means that QK T The calculation only requires the lower triangular results, saving about half the computational effort for this step. Lower triangular calculations are linear (and can be broken down into vector inner products). Furthermore, since the C matrix is ​​also a lower triangular matrix, this property can be exploited in the H = CV step to save about half the computational effort.

[0083] like Figure 4 As shown in the figure, the calculations of UpProj, DownProj, QKMatCalc, CVMatCalc, etc. for the entire ViLlayer can save about 20% of the calculation amount by utilizing the characteristics of the above lower triangular calculations.

[0084] More specifically, the QKMatCalc calculation subunit includes a K buffer, a Q buffer, several P-VPU modules, and an output buffer; the lower triangular matrix calculation design of the QKMatCalc calculation subunit is: in the four heads of the mLSTM cell, the matrix Q of size s×d and the matrix k of size d×s T Multiply, do the operation MatQK=QK T , the Q matrix is ​​expressed as Q = (q0,q1,…,q s-1 ) T , K matrix is ​​represented as K=(k0,k1,…,k s-1 ) T , then the calculation can be expressed as This means that several P-VPUs can be used to process the calculation in parallel, and only elements i≤j need to be calculated. Therefore, the calculation flow can be designed to achieve the calculation with only half of the computing resources while keeping the effective computing throughput unchanged. In order to ensure high data reuse rate during calculation, a K-stationary calculation flow design is adopted. During the calculation process, the value of the K matrix is ​​stored in each computing unit, and only the value of the Q matrix is ​​allowed to flow between computing units, so as to reduce the data movement between computing units, thereby reducing the power consumption of the overall hardware computing unit and improving the data reuse rate during calculation.

[0085] by Figure 5 The computing load is distributed to each P-VPU in a way to ensure high utilization of each P-VPU. Figure 6 The QKMatCalc calculation subunit is designed in the manner shown to achieve efficient calculation of this part.

[0086] like Figure 1 As shown, the DMatCalc calculation subunit includes a Gate value accumulation module, a row maximum module, a normalization module, an exp calculation module, and an output buffer module, wherein the input data passes through the Gate value accumulation module, and the output accumulation result data is respectively input into the row maximum module and the normalization module; the output of the row maximum module and the accumulation result data pass through the normalization module and the exp calculation module in sequence, and are output together with the output of the row maximum module through the output buffer module.

[0087] The CMatCalc calculation subunit includes an element-by-element multiplication module, a row maximum module, a normalization module, and an output buffer module; the element-by-element multiplication module performs element-by-element multiplication calculation processing on the outputs of the DMatCalc calculation subunit and the QKMatCalc calculation subunit respectively, and inputs the obtained results into the row maximum module for row maximum calculation processing, and inputs the output of the row maximum module and the output of the element-by-element multiplication module into the normalization module for normalization processing, and the normalization processing result is output through the output buffer module.

[0088] like Figure 7 As shown, the CVMatCalc calculation subunit includes a C buffer, a V buffer, several packed row accumulation units, and an output buffer; using the C-stationary method, the values ​​of the matrix C with a size of s×s are stored in each packed row accumulation unit through the C buffer, and the values ​​of the matrix V with a size of s×d flow between each packed row accumulation unit through the V buffer, so that the matrix C is multiplied by the matrix V to obtain an output matrix H with a size of s×d and output it through the output buffer.

[0089] Denote the H matrix as H = (h0,h1,h2,…,h s-1 ) T , V matrix is ​​expressed as V=(v0,v1,v2,…,v s-1 ) T , c ij Represented as the jth element in the i-th row of the c matrix, the calculation can be expressed as That is, each row of the V matrix is ​​divided into two groups according to c ij To improve the calculation efficiency, this embodiment can use c ij v j and c kl v j (c ij ×2 a +c kl )v j , and use the packed row accumulation unit for calculation, such as Figure 8 Similarly, in order to ensure high data reuse rate during calculation, the C-stationary calculation flow design is adopted. During the calculation process, the values ​​of the C matrix are stored in each packed row accumulation unit, and only the values ​​of the V matrix flow between each packed row accumulation unit, and the C matrix is ​​used as the calculation flow. Figure 9 The computing load is distributed to each package row accumulation unit in a manner to ensure high utilization of each package row accumulation unit.

[0090] More specifically, if Figure 10As shown, the hardware accelerator also includes an external storage access interface, a second Norm calculation unit, an UpProj linear calculation unit, a residual connection write unit, a ConvACT sequence two-dimensional convolution unit, a QKVProj linear calculation unit, an IFGate linear calculation unit, and an X pipeline buffer unit;

[0091] The storage read data stream of the external storage access interface is input into the second Norm calculation unit through the input read port, and the second Norm calculation result is output;

[0092] The weight stream of the external storage access interface is input into the residual connection write unit, which outputs the processed external weight stream. The processed external weight stream and the second Norm calculation result are input into the UpProj linear calculation unit, which outputs the UpProj linear calculation result and the storage write data stream. The storage write data stream is input into the external storage access interface through the residual connection write unit;

[0093] Input the processed external weight stream and UpProj linear calculation results into the ConvACT sequence two-dimensional convolution unit to obtain the ConvACT sequence two-dimensional convolution result, and input the ConvACT sequence two-dimensional convolution result into the external storage access interface through the residual connection write unit as storage write data;

[0094] More specifically, the processing performed by the ConvACT sequence two-dimensional convolution unit on the input data includes line buffering, convolution kernel, and SiLU calculation;

[0095] Input the UpProj linear calculation result into the X pipeline buffer unit to obtain the X pipeline buffer result;

[0096] Input the X pipeline buffer result and the two-dimensional convolution result of the ConvACT sequence into the QKVProj linear calculation unit, output the QKVProj linear calculation result, input the matrix K and matrix Q in the QKVProj linear calculation result into the QKMatCalc calculation sub-unit input of the multi-head mLSTM cell calculation unit, and input the matrix V in the QKVProj linear calculation result into the V pipeline buffer input of the multi-head mLSTM cell calculation unit;

[0097] The weight flow of the external storage access interface and the QKVProj linear calculation result are input into the IFGate linear calculation unit, the IFGate linear calculation result is output, and the IFGate linear calculation result is input as the Gate signal into the DMatCalc calculation sub-unit input end of the multi-head mLSTM cell calculation unit.

[0098] More specifically, the IFGate linear calculation unit includes a P-MPUIGate module, a P-MPUFGate module, and an LS calculation module. The output of the QKVProj linear calculation unit is respectively input into the P-MPUIGate module and the P-MPUFGate module of the IFGate linear calculation unit, and the P-MPUIGate calculation results and the P-MPUFGate calculation results are respectively output. The P-MPUFGate calculation results are input into the LS calculation module, and the LS calculation results are output. The LS calculation results and the P-MPUIGate calculation results are output through the data path.

[0099] More specifically, the processing of the input data by the first Norm calculation subunit and the second Norm calculation unit includes: calculating the mean, calculating the standard deviation, and linear output.

[0100] More specifically, the hardware accelerator further includes a residual connection reading unit, a DownProj linear calculation unit, a first residual connection calculation unit, and a second residual connection calculation unit;

[0101] Input the weight stream and storage read data stream of the external storage access interface into the residual connection read unit, output the processed weight stream and the processed storage read data stream, input the processed weight stream into the weight buffer of the first residual connection calculation unit, perform element-by-element multiplication operation with the processed storage read data stream, and obtain the element-by-element multiplication operation result;

[0102] Input the first Norm calculation result into the first residual connection calculation unit, perform element-by-element addition operation on the element-by-element multiplication result, and obtain the element-by-element addition operation result;

[0103] Perform SiLU calculation on the processed weight stream, and perform element-by-element addition operation on the result obtained to obtain the first residual connection calculation result;

[0104] Input the first residual connection calculation result and the processed weight stream into the DownProj linear calculation unit, and output the DownProj linear calculation result;

[0105] The DownProj linear calculation result and the processed storage read data stream are input into the second residual connection calculation unit, and the output is input into the external storage access interface through the output write port as the storage write data stream.

[0106] More specifically, the UpProj linear calculation unit, the QKVProj linear calculation unit, the IFGate linear calculation unit, and the DownProj linear calculation unit each include one or more P-MPU structures.

[0107] More specifically, the P-MPU structure includes several parallel-connected P-VPU modules, a weight buffer, an activation value buffer, an accumulation tree module, and an output buffer. The P-VPU module is a cascade calculation submodule with N levels based on a packed DSP MAC calculation method.

[0108] The activation value is input into the activation value buffer for processing to obtain activation value buffer data, the activation value buffer data is input into several parallel connected P-VPU modules, and the weight value in the weight buffer is combined for calculation. The calculation result is input into the accumulation tree module for accumulation operation and output through the output buffer.

[0109] For the linear calculation part, it can basically be decomposed into several vector-matrix multiplications, and the vector-matrix multiplication can be further decomposed into several pairs of vector inner product operations. Consider the multiplication of the N-dimensional activation value vector a and the N×P-dimensional weight matrix W, where W=(w1,w2,w3,…,w P ), then the calculation can be decomposed into P N-dimensional vector inner products o i =aw i At this point, a cascade calculation submodule with N levels based on the packed DSP MAC calculation method can be created, forming the following Figure 11 A P-VPU (packed vector process unit) is shown, and Figure 12 The input is sent into the computing structure in a way to perform packed vector inner product calculation, achieving the computational throughput of completing two vector inner product operations in a single cycle, which can greatly improve the utilization of hardware resources and the computational throughput of each computing unit.

[0110] On the basis of P-VPU, when a large-dimensional packed vector inner product calculation that cannot be handled by a single P-VPU occurs, the calculation submodules can be expanded in parallel to form the following Figure 13 A larger P-MPU unit is used to handle this situation, so as to handle the inner product calculation of packed vectors of arbitrary dimension N.

[0111] More specifically, a lookup table method is used to process nonlinear function calculations in the calculation process, where the nonlinear function calculations include SiLU calculations, normalization calculations, and exp calculations.

[0112] Due to the presence of a large number of nonlinear functions, such as SiLU function, logsigmoid function and exp function, directly performing these nonlinear functions will make the hardware complexity of these parts higher. However, under the low-precision neural network calculation load, the input bit width and output bit width of these nonlinear function calculations can be determined. On this basis, for the nonlinear function calculations with these bit widths not too large, the present embodiment can adopt the lookup table (LUT) method to implement them, so as to achieve nonlinear function calculations with low hardware complexity and low resource consumption.

[0113] In the calculation of ViL Layer, there are several residual connections with large spans. In the hardware accelerator design, due to the data dependency caused by the residual connection mechanism, the current calculation data must be retained until the calculation of the residual connection is completed. In the conventional streaming accelerator design, this means that a large amount of hardware storage resources are consumed to achieve pipeline buffering of these data. When the span of the residual connection that needs to be buffered is large to a certain extent, the amount of hardware storage resources required for this design is unacceptable. Therefore, this embodiment proposes a half-layer mapping method, which can use external storage as a buffering intermediary to avoid excessive hardware storage resource requirements caused by residual connections. Figure 7 As shown in Figure 1, for each large-span residual connection that needs to be processed, we can only partially buffer the data on-chip, and temporarily store the data in external storage during the rest of the time, so as to process the large-span residual connection in the ViL layer calculation with acceptable hardware storage resource consumption.

[0114] The present invention will design the accelerator pipeline data flow and build a complete ViL hardware accelerator, such as Figure 10 , and completed design verification and performance comparison on the Xilinx Alveo U250 platform.

[0115] In this embodiment, if Figure 10 The accelerator architecture of the present invention is shown in Figure 2. By customizing specialized hardware processing units for different computing tasks, efficient processing of each component is achieved. The present invention utilizes a fully pipelined design to build the entire accelerator, ensuring the system maintains high throughput. The accelerator operates as follows.

[0116] When the computation starts, the input data storage and reading unit will send the data into the accelerator pipeline in the form of a stream, such as Figure 14 As shown, the input tensor X of size S×D will be split into several row vector units of size 1×u as a calculation fragment; and used as the unit for calculation input and output, as shown in Figure 15As shown, where D must be divisible by u and is fed into the accelerator pipeline sequentially in each clock cycle. Therefore, the throughput of the system can be expressed as follows:

[0117]

[0118] In order to ensure that there is no data congestion in the hardware accelerator pipeline, the data and computing throughput of all hardware modules in the accelerator pipeline are designed according to this standard.

[0119] During the pipeline computation, data related to some large-span residual links is temporarily stored in external storage via the residual link storage write unit. Later in the pipeline computation, this data is read back via the residual link storage read unit to complete the residual connection computation. After all computations are complete, the data stream is sent to the output storage write unit for writing back to external storage.

[0120] This embodiment uses Verilog HDL to deploy QVil, and uses Vivado 2021.1 to deploy it to the XilinxFPGA Alveo U250 platform, and implements it at a target frequency of 240MHz. The corresponding resource utilization and power consumption are shown in Table 1. LUT (Look-Up Table), FF (Flip-Flop), DSP (Digital Signal Processor), and BRAM (Block RAM) are resource utilization, reflecting the design's use of FPGA hardware resources. Freq. (Frequency), Power (power consumption), and Throughput (throughput) are performance indicators, indicating the accelerator's operating performance and power consumption characteristics. The hardware accelerator uses a computational fragment of size U=4 for pipeline design. In order to reduce the impact of the delay in accessing external memory on the hardware pipeline, all residual connection external storage read / write modules use pingpong buffer when accessing external storage.

[0121] Table 1

[0122]

[0123] Since Vision-LSTM currently lacks other FPGA or ASIC hardware accelerators, CPUs and GPUs are used as benchmarks for performance comparison. This example uses the pre-trained model vil2_tiny16_e400_in1k in the PyTorch 2.3.1, CUDA 12.1.0 environment to perform CPU and GPU inference of the ViL model. The CPU benchmark is the AMD Ryzen Threadripper 3960X, and the GPU baseline is the NVIDIA RTX 3090. The experimental results are shown in Table 2, where BatchSize is the amount of data processed per computational task, Throughput is the throughput, ApproximatePower is the runtime power consumption, and Performance / Power is the energy efficiency ratio.

[0124] The GPU power consumption was measured using nvidia-smi, and the CPU power consumption was measured using the ryzen-power tool.

[0125] Table 2

[0126]

[0127] From the comparison results, it can be concluded that the hardware accelerator designed by the present invention has a great improvement compared with the CPU benchmark, and compared with the GPU benchmark, it has achieved a performance of 0.90X to 1.19X, and has an improvement of 26.80X to 29.57X in energy efficiency.

[0128] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. An efficient Vision-LSTM hardware accelerator based on FPGA, characterized by: The accelerator comprises: A multi-head mLSTM cell computing unit, wherein the multi-head mLSTM cell computing unit includes several mLSTM cell computing units connected in parallel; The mLSTM cell calculation unit includes a QKMatCalc calculation subunit, a CVMatCalc calculation subunit, a DMatCalc calculation subunit, a CMatCalc calculation subunit, a first Norm calculation subunit, and a V pipeline buffer unit; Among them, the Gate signal is input into the DMatCalc calculation subunit, and the DMatCalc calculation result is output; Input matrix K and matrix Q into the QKMatCalc calculation subunit and output the QKMatCalc calculation result; Input the matrix V into the V pipeline buffer unit and output the V pipeline buffer result; Input the calculation results of DMatCalc and QKMatCalc into the CMatCalc calculation subunit, and output the calculation results of CMatCalc; Input the CMatCalc calculation result and the V pipeline buffer result into the CVMatCalc calculation subunit, and output the CVMatCalc calculation result; Input the CVMatCalc calculation result into the first Norm calculation subunit and output the first Norm calculation result; The hardware accelerator also includes an external storage access interface, a second Norm calculation unit, an UpProj linear calculation unit, a residual connection write unit, a ConvACT sequence two-dimensional convolution unit, a QKVProj linear calculation unit, an IFGate linear calculation unit, and an X pipeline buffer unit; The storage read data stream of the external storage access interface is input into the second Norm calculation unit through the input read port, and the second Norm calculation result is output; The weight stream of the external storage access interface is input into the residual connection write unit, which outputs the processed external weight stream. The processed external weight stream and the second Norm calculation result are input into the UpProj linear calculation unit, which outputs the UpProj linear calculation result and the storage write data stream. The storage write data stream is input into the external storage access interface through the residual connection write unit; Input the processed external weight stream and UpProj linear calculation results into the ConvACT sequence two-dimensional convolution unit to obtain the ConvACT sequence two-dimensional convolution result, and input the ConvACT sequence two-dimensional convolution result into the external storage access interface through the residual connection write unit as storage write data; Input the UpProj linear calculation result into the X pipeline buffer unit to obtain the X pipeline buffer result; Input the X pipeline buffer result and the two-dimensional convolution result of the ConvACT sequence into the QKVProj linear calculation unit, output the QKVProj linear calculation result, input the matrix K and matrix Q in the QKVProj linear calculation result into the QKMatCalc calculation sub-unit input of the multi-head mLSTM cell calculation unit, and input the matrix V in the QKVProj linear calculation result into the V pipeline buffer input of the multi-head mLSTM cell calculation unit; The weight flow of the external storage access interface and the QKVProj linear calculation result are input into the IFGate linear calculation unit, the IFGate linear calculation result is output, and the IFGate linear calculation result is input as the Gate signal into the DMatCalc calculation sub-unit input end of the multi-head mLSTM cell calculation unit.

2. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 1, characterized in that: The QKMatCalc calculation subunit includes a K buffer, a Q buffer, several P-VPU modules, and an output buffer; the calculation flow adopts a K-stationary method, storing the value of the matrix K in each P-VPU calculation structure through the K buffer, and flowing the value of the matrix Q between each P-VPU calculation structure through the Q buffer. With the matrix The result is output through the output buffer.

3. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 1, characterized in that: The CVMatCalc calculation subunit includes a C buffer, a V buffer, several packed row accumulation units, and an output buffer; using the C-stationary method, the values ​​of the matrix C are stored in each packed row accumulation unit through the C buffer, and the values ​​of the matrix V flow between each packed row accumulation unit through the V buffer, thereby With the matrix Perform matrix multiplication and output the result through the output buffer.

4. The efficient FPGA-based Vision-LSTM hardware accelerator according to claim 1, characterized in that The processing of the input data by the first Norm calculation subunit and the second Norm calculation unit includes: calculating the mean, calculating the standard deviation, and linear output.

5. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 1, characterized in that The processing of input data by the ConvACT sequence two-dimensional convolution unit includes row buffering, convolution kernel, and SiLU calculation.

6. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 1, characterized in that The hardware accelerator further includes a residual connection reading unit, a DownProj linear calculation unit, a first residual connection calculation unit, and a second residual connection calculation unit; Input the weight stream and storage read data stream of the external storage access interface into the residual connection read unit, output the processed weight stream and the processed storage read data stream, input the processed weight stream into the weight buffer of the first residual connection calculation unit, perform element-by-element multiplication operation with the processed storage read data stream, and obtain the element-by-element multiplication operation result; Input the first Norm calculation result into the first residual connection calculation unit, perform element-by-element addition operation on the element-by-element multiplication result, and obtain the element-by-element addition operation result; Perform SiLU calculation on the processed weight stream, and perform element-by-element addition operation on the result obtained to obtain the first residual connection calculation result; Input the first residual connection calculation result and the processed weight stream into the DownProj linear calculation unit, and output the DownProj linear calculation result; The DownProj linear calculation result and the processed storage read data stream are input into the second residual connection calculation unit, and the output is input into the external storage access interface through the output write port as the storage write data stream.

7. The efficient FPGA-based Vision-LSTM hardware accelerator according to claim 6, characterized in that The UpProj linear calculation unit, the QKVProj linear calculation unit, the IFGate linear calculation unit, and the DownProj linear calculation unit each include one or more P-MPU structures.

8. The efficient Vision-LSTM hardware accelerator based on FPGA according to claim 7, characterized in that: The P-MPU structure includes several parallel connected P-VPU modules, weight buffers, activation value buffers, accumulation tree modules, and output buffers. The P-VPU module is a cascade calculation submodule with N levels based on the packed DSP MAC calculation method. The activation value is input into the activation value buffer for processing to obtain activation value buffer data, the activation value buffer data is input into several parallel connected P-VPU modules, and the weight value in the weight buffer is combined for calculation. The calculation result is input into the accumulation tree module for accumulation operation and output through the output buffer.

9. The efficient FPGA-based Vision-LSTM hardware accelerator according to claim 1, characterized in that The nonlinear function calculation in the calculation process is processed using the lookup table method.

Images