High-flow-degree MambaV2 hardware acceleration system based on calculation sequence optimization of FPGA (Field Programmable Gate Array)

Abstract

The invention, which relates to the field of FPGA and machine learning, discloses a high-flow-degree MambaV2 hardware acceleration system with an optimized calculation sequence based on FPGA, comprising a preceding stage module, a tensor expansion module, a hidden layer state calculation module, an output state calculation module, a residual multiplication normalization module and an output linear layer module. The calculation process of the MambaV2 algorithm is divided into a plurality of pipeline levels, and a data stream is output in a pipeline form. According to the method, an original algorithm structure is not changed, the operation sequence of a hidden layer state equation in an original algorithm is changed, an algorithm for optimizing the calculation sequence is provided, the number of times of operation is reduced, cache consumption is reduced, the MambaV2 model is deployed and reasoned on an FPGA, the edge scene deployment requirement is met, customized acceleration of the FPGA is utilized, and data streams are adjusted, so that the edge scene deployment efficiency is improved. The pipeline design of the whole algorithm is realized, the utilization rate of computing resources is improved, the problem of data blocking is solved, and computing delay and cache consumption are reduced.

Description

Technical Field

[0001] The present invention relates to the field of FPGA and machine learning technology, and more specifically, to a high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization. Background Art

[0002] The Mamba model is an important architecture that has emerged in the field of machine learning in recent years, especially when dealing with large-scale data and complex computing tasks. With the rapid development of deep learning, traditional model architectures and optimization methods face problems such as high consumption of computing resources and long training time. In order to meet these challenges, the Mamba model came into being, aiming to provide more efficient computing performance and flexible architecture design, and later evolved into the MambaV2 model, which has been widely used in natural language processing (NLP), computer vision (CV) and other fields.

[0003] As the complexity and scale of the MambaV2 model increase, the number of parameters increases dramatically, and the demand for computing resources also continues to rise. How to efficiently deploy the MambaV2 model on hardware has become a difficult problem that needs to be solved. Although GPU is a traditional deep learning accelerator, in some scenarios, FPGA is considered to be a better hardware choice, especially in terms of performance, energy efficiency and flexibility. Deploying the MambaV2 model on FPGA has its specific technical advantages and application requirements. FPGA has the advantage of energy efficiency and can realize customized hardware acceleration, which is in great demand in edge computing application scenarios.

[0004] However, there are few mainstream designs that deploy the MambaV2 model on FPGA. Most of the existing technical implementations focus on GPUs for training and reasoning. The main reason is that GPUs have mature parallel computing architectures and extensive development support. However, GPUs have obvious limitations when dealing with some specific scenarios, and FPGA platforms provide new possibilities. Therefore, how to deploy the MambaV2 model on FPGA to meet the deployment requirements of edge scenarios while optimizing data flow, reducing computing latency, improving computing resource utilization, and reducing storage resource consumption has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the problem in the prior art that how to deploy the MambaV2 model on FPGA to meet the deployment requirements of edge scenarios while optimizing data flow, reducing computing delay, improving computing resource utilization and reducing storage resource consumption has become an urgent problem to be solved, and to provide a high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] A high-pipeline MambaV2 hardware acceleration system based on FPGA calculation sequence optimization includes: a front-end module, a tensor expansion module, a hidden layer state calculation module, an output state calculation module, a residual multiplication normalization module and an output linear layer module. The hardware acceleration system divides the MambaV2 model algorithm calculation process into several pipeline stages and outputs data streams in a pipeline form;

[0008] The front-stage module operates on the input data stream and outputs the tensor data stream required for the subsequent pipeline stage to operate;

[0009] The tensor expansion module is the first pipeline stage, which is used to perform the dBx tensor expansion operation process after optimizing the calculation order in the MambaV2 model algorithm to reduce the number of calculations and reduce cache consumption;

[0010] The hidden layer state calculation module is a second pipeline stage, which is used to implement the multiplication and addition operation when calculating the hidden layer state to obtain the hidden layer state;

[0011] The output state calculation module is a third pipeline stage, used to calculate the output state according to the hidden layer state;

[0012] The residual product normalization module is a fourth pipeline stage and a fifth pipeline stage, and performs a residual product operation and a normalization operation on the output state through two stages of pipelines to obtain an output vector;

[0013] The output linear layer module is the sixth pipeline stage, which performs dimension transformation on the output vector to obtain a result vector.

[0014] As a preferred solution, the tensor expansion module is used to complete the dBx tensor expansion operation process, that is, to calculate the second item dBx(t) of the addition operation in the hidden layer state. The expression for calculating the hidden layer state h is:

[0015] h(t)=dA*h(t-1)+dBx(t)

[0016] Wherein, h(t) is the hidden layer state at time t, dBx(t) is the influence of x on the hidden layer state, dA is the data in the tensor data stream, obtained by discretizing the input matrix A in the input data stream, and the discretization expression is as follows:

[0017] dA=e A*dt

[0018] Wherein, dt is the data in the tensor data stream;

[0019] The dBx tensor expansion operation process after optimizing the calculation order in the MambaV2 model algorithm includes firstly performing tensor expansion on the data dt and x to obtain an intermediate result tensor, and then performing tensor expansion on the intermediate result tensor and B to obtain dBx, where dt, x and B are all data in the tensor data stream. The specific calculation steps are: firstly performing tensor expansion on dt and x, and performing dot multiplication operation on the elements of the vector dt and the elements of the vector with dimension (1, p) in the tensor x in the h dimension to obtain an intermediate result tensor with dimension (h, p). The calculation process is expressed as:

[0020] einsum(h,hp→hp,dt,x)

[0021] Among them, h represents the number of heads in the model, and p represents the dimension of the head;

[0022] Then, the intermediate result tensor of dimension (h, p) is used to perform tensor expansion with B, and each element in the tensor dx of dimension (h, p) is multiplied with the vector B of dimension (1, n) to obtain the tensor dBx. The calculation process is expressed as:

[0023] einsum(hp,n→hpn,dx,B)

[0024] Here, n represents the hidden layer dimension.

[0025] As a preferred solution, the tensor expansion module divides the data of dimension (h, p, n) into h*p period outputs, each period outputs data of dimension (1, n), and subsequent modules perform pipeline operations based on this data stream of dimension (1, n).

[0026] As a preferred solution, the reduction in the number of calculations and cache consumption is specifically as follows:

[0027] The total number of multiplications required for dBx tensor expansion in the original MambaV2 model algorithm is h*n+h*n*p. Assuming that each element in the tensor consumes 1 storage unit, the total cache consumption is h+n+h*p+h*n storage units.

[0028] After optimizing the calculation order, the total number of multiplications required for dBx tensor expansion is h*p+h*n*p, and the total cache consumption is h+n+2*h*p storage units. In the MambaV2 model algorithm, n represents the hidden layer dimension, which is generally 128, and p represents the head dimension, which is generally 64. By optimizing the calculation order, the number of multiplication calculations is greatly reduced and the consumption of storage resources is reduced.

[0029] As a preferred solution, the hidden layer state calculation module includes an on-chip hidden layer state cache, each on-chip cache has a depth of h*p and vector expanded data of a width of (1, n) for storing the previous hidden layer state.

[0030] As a preferred solution, the hidden layer state calculation module completes the multiplication and addition operation in the expression for calculating the hidden layer state h;

[0031] Among them, the calculation process of the term dA*h(t-1) in the expression for calculating the hidden layer state h is as follows: the input of the hidden layer state calculation module comes from the data stream with a dimension of (1, n) output by the tensor expansion module and the data dA in the tensor data stream calculated by the previous module, the dimension of the hidden layer state h is (h, p, n), the dimension of dA is (1, h), and the dimension of dBx is (h, p, n). During the calculation process, dA*h(t-1) adopts a broadcast mechanism to transform the dimension of dA into (h, 1, 1) and then perform a dot multiplication operation with h(t-1).

[0032] As a preferred solution, the hidden layer state calculation module uses n multipliers to perform parallel calculations, performs multiplication and addition operations in a single cycle, and outputs a data stream with a dimension of (1, n). Subsequent modules perform pipeline operations based on this data stream with a dimension of (1, n).

[0033] As a preferred solution, the output state calculation module is used to complete the calculation of the output state y, and the calculation process is expressed as:

[0034] y(t)=Ch(t)+Dx(t)

[0035] Wherein, y(t) is the output state at time t, C is the tensor data in the tensor data stream, and D is the matrix stored in the on-chip cache of the FPGA;

[0036] During the calculation process, the tensor broadcast mechanism is used to perform inner product operations. The products of the elements between the vectors are calculated in parallel by n multipliers, and then the inner product is calculated through the addition tree. The output state calculation module, i.e., the third-level pipeline, folds the addition operation to realize internal pipeline. The output state y tensor (h, p) is equivalent to (1, ed), ed=h*p. The output state y is divided into h*p cycles, and each cycle outputs data with a dimension of (1, 1). The subsequent modules perform pipeline operations according to this data stream with a dimension of (1, 1).

[0037] As a preferred solution, the residual multiplication normalization module completes the calculation of the following formula:

[0038] Y(t)=RMSnorm(y(t)*z)

[0039] Wherein, Y(t) represents the output vector, and the residual z is the data in the tensor data stream;

[0040] In the residual multiplication and normalization module, the residual product operation part is the fourth pipeline stage, and the normalization operation part is the fifth pipeline stage, wherein the operation of the nonlinear function in the normalization operation part is implemented by a table lookup method, and the residual multiplication and normalization module outputs data with a dimension of (1, 1) in each cycle, and the subsequent modules will perform pipeline operations based on this data stream with a dimension of (1, 1).

[0041] As a preferred solution, the output linear layer module performs a dimension transformation on the vector, that is, a vector with a dimension of (1, ed) is multiplied by a matrix with a dimension of (ed, d), where ed = h*p, each element in the vector is multiplied and accumulated one by one with each column element of the matrix, and d multipliers are used to perform the accumulation operation to finally obtain a result vector. In a single cycle, only one element is input into the input vector to match the data flow inputted by the previous pipeline stage, thereby realizing pipeline design and avoiding data flow blockage.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. Without changing the original algorithm, the present invention changes the operation order of the hidden layer state equation in the MambaV2 model algorithm, and proposes an algorithm for optimizing the calculation order, which can reduce the number of operations and reduce the consumption of storage resources.

[0044] 2. The high-streamline MambaV2 hardware accelerator system of the present invention realizes the deployment of the inference MambaV2 model on the FPGA, which can meet the deployment requirements of edge scenarios.

[0045] 3. The present invention utilizes customized acceleration of FPGA and realizes pipeline design of the overall algorithm by adjusting the data flow, thereby improving the utilization rate of computing resources, solving the problem of data blocking, reducing computing delay, and reducing the consumption of on-chip cache. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The overall framework diagram of a high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to an embodiment of the present application;

[0047] Figure 2 A schematic diagram of the structure of the dBx tensor expansion module of an embodiment of the present application

[0048] Figure 3 A schematic diagram of the structure of a hidden layer state calculation module according to an embodiment of the present application;

[0049] Figure 4A schematic diagram of the structure of an output state calculation module of an embodiment of the present application;

[0050] Figure 5 This is a schematic diagram of the structure of a residual multiplication normalization module according to an embodiment of the present application;

[0051] Figure 6 Schematic diagram of the structure of the output linear layer module of an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0053] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0054] Example 1

[0055] See also Figure 1 Embodiment 1 of the present application provides a high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization, including:

[0056] The hardware acceleration system divides the MambaV2 model algorithm calculation process into several pipeline stages and outputs the data stream in the form of pipeline, including the pre-stage module, dBx tensor expansion module, hidden layer state calculation module, output state y calculation module, residual multiplication normalization module and output linear layer module;

[0057] The front-end module operates on the input data stream and outputs the tensor data stream required for the subsequent pipeline stages to operate;

[0058] The dBx tensor expansion module is the first pipeline stage, which is used to perform the dBx tensor expansion operation process after optimizing the calculation order in the MambaV2 model algorithm to reduce the number of calculations and reduce cache consumption;

[0059] The hidden layer state calculation module is the second pipeline stage, which is used to implement the multiplication and addition operation when calculating the hidden layer state to obtain the hidden layer state;

[0060] The output state y calculation module is the third pipeline stage, which is used to calculate the output state according to the hidden layer state;

[0061] The residual product normalization module is the fourth pipeline stage and the fifth pipeline stage. Through the two-stage pipeline, the residual product operation and the normalization operation are performed on the output state to obtain the output vector;

[0062] The output linear layer module is the sixth pipeline stage, which transforms the dimension of the output vector to obtain the result vector.

[0063] In the original MambaV2 model algorithm, the input vector u is first mapped to the input linear layer, and then causal convolution and activation function operations are performed to obtain x, B, C, dt and z, and then the input matrix A is discretized:

[0064] dA=e A*dt (1)

[0065] Then calculate the hidden layer state:

[0066] h(t)=dA*h(t-1)+dBx(t) (2)

[0067] Then calculate the output state:

[0068] y(t)=Ch(t)+Dx(t) (3)

[0069] Perform residual multiplication and normalization operations:

[0070] Y(t)=RMSnorm(y(t)*z) (4)

[0071] Finally, the output linear layer mapping is performed.

[0072] The present invention does not discuss the calculation order of the previous module, and assumes that the data flow calculation of the previous module is completed. It only discusses the calculation data flow after formula (2) and the hardware implementation of the hardware accelerator system. The specific calculation process of the influence of x on the hidden layer state dBx(t) in formula (2) in the original MambaV2 model algorithm is as follows:

[0073] Use tensors dt and B to perform tensor expansion to discretize dB, and then perform tensor expansion with tensor x to obtain dBx. The specific process is as follows:

[0074] einsum(h,n,hp→hpn",dt,B,x) (5)

[0075] Among them, n represents the hidden layer dimension, h represents the number of heads in the model, and p represents the dimension of the head. Specifically, the vector with dimension (1, h) in dt and the vector with dimension (1, n) in B are outer-producted to obtain a matrix with dimension (h, n). Then, in each h dimension, the vector with dimension (1, n) and the vector with dimension (1, p) in x are outer-producted to finally obtain a tensor with dimension (h, p, n). This is the complete tensor expansion process of the original MambaV2 model algorithm. At this point, it can be known that: first, the tensor expansion of vector dt and vector B requires h*n multiplications; then, the tensor expansion with vector x requires h*n*p multiplications. Therefore, the number of multiplications required for the complete tensor expansion of dBx in the original MambaV2 model algorithm is h*n+h*n*p.

[0076] Assuming that each element in the tensor consumes 1 storage unit, the cache consumption of the initial vector is h+n+h*p storage units, and the cache of the intermediate result needs to consume h*n storage units. Therefore, the total cache consumption is h+n+h*p+h*n storage units.

[0077] The calculation order of dBx tensor expansion operation process is optimized and modified. This part of the operation is implemented in the tensor expansion module. The specific calculation process after optimizing the calculation order is as follows:

[0078] First, let dt and x be expanded as tensors.

[0079] einsum(h,hp→hp,dt,x) (6)

[0080] In the h dimension, the elements of the vector dt are multiplied with the vector elements of the tensor x with the dimension (1, p) to obtain the intermediate result tensor dx with the dimension (h, p). The number of multiplications is h*p, and the cache of the intermediate result requires h*p storage units. Then, this intermediate result tensor dx with the dimension (h, p) is used to expand the tensor with B.

[0081] einsum(hp,n→hpn,dx,B) (7)

[0082] To multiply each element of the tensor dx of dimension (h, p) with the vector B of dimension (1, n), h*p*n multiplications need to be performed. Therefore, the number of multiplications required for the complete process of dBx tensor expansion after optimizing and modifying the calculation order is h*p+h*n*p, and the total cache consumption is h+n+2*h*p storage units. In the MambaV2 model algorithm, n represents the hidden layer dimension, which is generally 128, and p represents the dimension of the head in the model, which is generally 64. Therefore, by modifying the calculation order, the number of multiplications can be greatly reduced, and the consumption of storage resources can also be reduced.

[0083] See also Figure 1 , which is an overall architecture block diagram of a high-pipeline MambaV2 hardware acceleration system based on FPGA calculation sequence optimization in this embodiment 1. In order to improve acceleration performance, reduce calculation delay, and improve the utilization rate of computing units, this application proposes a pipeline design scheme to implement the MambaV2 hardware acceleration system architecture, and divides the calculation part after the front-stage module into six pipeline stages. The first pipeline stage is the dBx tensor expansion module, the second pipeline stage is the hidden layer state calculation module, the third pipeline stage is the output state y calculation module, the fourth pipeline stage and the fifth pipeline stage are the residual multiplication normalization modules, and the last pipeline stage is the output linear layer module, which realizes the overall pipeline design, optimizes the data flow, reduces the calculation delay, improves the utilization rate of the computing unit, and reduces the consumption of storage resources.

[0084] The front-end module includes: the input vector u and the weight matrix cache are first mapped to the input linear layer, and then the causal convolution and activation function operations are performed respectively to obtain the tensor data streams x, B, C, dt and z, and then the input matrix A is discretized to obtain dA, where the input matrix A is a randomly generated matrix obtained through training and stored in the on-chip cache of the FPGA.

[0085] The dBx tensor expansion module executes the dBx tensor expansion process after optimizing and modifying the calculation order of formula (2). In the process of executing the expansion of the intermediate tensors dx and B, the calculation is completed in the form of pipeline output data with h*p cycles of pipeline output dimension (1, n). The subsequent modules are pipelined designed according to this data flow.

[0086] The hidden layer state calculation module implements the multiplication and addition operation of formula (2). The calculation module contains an on-chip hidden layer state cache. Each on-chip cache has a depth of h*p and a width of (1, n) vector expanded data, which is used to store the previous hidden layer state. The input is the data stream from the dBx tensor expansion module (1, n) and dA calculated by the previous module. The dimension of the hidden layer state is (h, p, n), the dimension of dA is (1, h), and the dimension of dBx is (h, p, n). In formula (2), dA*h(t-1) uses a broadcast mechanism to transform the dimension of dA to (h, 1, 1) and then perform a dot multiplication operation with h(t-1). During the calculation process, n multipliers and adders are used for parallel calculation, and the multiplication and addition operation is performed in a single cycle. The output is a data stream with a dimension of (1, n), which realizes a pipeline design and reduces latency.

[0087] The output state y calculation module implements the operation of formula (3), which involves using the tensor broadcast mechanism to perform inner product operations. The products of the elements between vectors are calculated in parallel by n multipliers, and then the inner product is calculated through the addition tree. This pipeline stage will fold the addition operation to achieve internal pipeline, and the pipeline output dimension is (1, 1) elements.

[0088] The residual product normalization module implements the operation of formula (4), performs residual product operation through the fourth pipeline stage, and performs normalization operation through the fifth pipeline stage, wherein the normalization of the nonlinear function in the normalization operation is implemented by a table lookup method.

[0089] Finally, there is the output linear layer module, which transforms the vector dimension. Its operation can be regarded as the multiplication of a vector and a matrix. Specifically, a vector of dimension (1, ed) is multiplied by a matrix of dimension (ed, d), where ed = h*p, that is, each element in the vector is multiplied and accumulated one by one with each column element of the matrix, and finally the result vector is obtained. A pipeline design is adopted, and d multipliers are used for accumulation operations. In a single cycle, only one element needs to be input into the input vector, which just matches the data flow of the previous input, thus realizing the pipeline design and solving the problem of data flow congestion.

[0090] In this embodiment 1, the process of applying the hardware acceleration system to perform an inference acceleration task is as follows:

[0091] The weight matrix and input vector are loaded from the external memory into the on-chip storage of the FPGA. The front-stage module calculates the tensors B, x, dt, dA and C required by the subsequent pipeline stages through operations. Then the dBx tensor expansion module and the hidden layer state calculation module complete the operation of formula (2). The output state y calculation module completes the operation of formula (3). The residual multiplication normalization module completes the operation of formula (4). The output linear layer module transforms the dimension of the output vector to obtain the result vector. Then the result vector is used as the input vector again to continue iteration.

[0092] Example 2

[0093] This embodiment 2 is based on embodiment 1 and further describes the dBx tensor expansion module therein. Figure 2 The figure below is a schematic diagram of the structure of the dBx tensor expansion module. Combined with the algorithm process after the calculation order is optimized, the workflow of the dBx tensor expansion module is explained in detail with a calculation process of a pipeline output vector. The details are as follows:

[0094] The previous module inputs the calculated tensors dt, B, and x into the on-chip cache of the FPGA, and then executes the algorithm after optimizing and modifying the calculation order. A row of data with a dimension of (1, h) output from the dt cache and a matrix data with a dimension of (h, p) output from the x cache are input into the multiplier array together, and the output data with a dimension of (h, p) is stored in the dx cache. After that, in each single cycle, the dx cache outputs one element, and the B cache outputs a column of data with a dimension of (1, n), which is input into the multiplier array, and then outputs data with a dimension of (1, n). In essence, the output data with a dimension of (h, p, n) is divided into h*p cycles, and each cycle outputs data with a dimension of (1, n). The subsequent modules perform pipeline operations based on this (1, n) data stream.

[0095] The other technical details and implementation steps of this embodiment 2 are the same as those of embodiment 1 and will not be repeated here.

[0096] Example 3

[0097] Based on Example 1, this Example 3 further describes the hidden layer state calculation module, as follows:

[0098] like Figure 3As shown, it is a schematic block diagram of the structure of the hidden layer state calculation module. This module completes the calculation of formula (2) in Example 1. The second term of the addition in formula (2) has been calculated by the dBx tensor expansion module. The input of the hidden layer state calculation module is a data stream with a dimension of (1, n) from the dBx tensor expansion module, and the previous module has calculated the dA tensor and stored it in the dA cache. The hidden layer state calculation module contains a hidden layer state cache, which stores the previous hidden layer state. In order to adapt to the data stream with a dimension of (1, n), the depth of each on-chip hidden layer state cache is h*p, and the data width is the vector expansion data of (1, n). There are a total of layerCount hidden layer caches, depending on the number of layers of the deployment algorithm MambaV2, and the corresponding cache will be accessed according to the current number of layers. When the dBx tensor expansion module inputs valid vector data with a dimension of (1, n), the dBx tensor expansion module will output an enable signal one cycle in advance. This enable signal will be input into the dA cache and the hidden layer state cache. In the next cycle, the dBx tensor expansion module outputs a valid vector data with a dimension of (1, n), and the dA cache and the hidden layer state cache will also output corresponding data. These data are input into the multiplier array together to obtain an output valid vector with a dimension of (1, n), realizing the pipeline processing of data. At this time, the hidden layer state, i.e., the tensor with a dimension of (h, p, n), is split into h*p cycles, and each cycle outputs a data with a dimension of (1, n). The subsequent modules continue to perform pipeline operations based on this data stream with a dimension of (1, n), thereby realizing the pipeline of the entire data stream.

[0099] The other technical details and implementation steps of this embodiment 3 are the same as those of embodiment 1 and will not be repeated here.

[0100] Example 4

[0101] Based on Example 1, this Example 4 further describes the output state y calculation module, as follows:

[0102] like Figure 4 As shown, it is a schematic block diagram of the structure of the output state y calculation module. This module completes the operation of formula (3) in Example 1. The previous module calculates x to give data and valid signals, which are input into the output state y calculation module, and multiplied with the matrix D (obtained and stored in the same way as the input matrix A) stored in the cache on the FPGA chip to calculate the Dx tensor stored in the cache. The vector data with a dimension of (1, n) output by the hidden layer h calculation module and the vector data with a dimension of (1, n) input by the tensor C of the previous module are input into the multiplier array, and the element product between the vectors is calculated in parallel, and then the inner product of the two vectors is calculated through a balanced addition tree:

[0103] (1, n)@(n, 1)=(1, 1)

[0104] Then add it to the Dx tensor element in the cache to calculate the result. In essence, the output state y, which is a tensor of dimension (h, p) (equivalent to (1, ed), ed = h*p), is split into h*p cycles, and each cycle outputs a data of dimension (1, 1). The subsequent modules continue to perform pipeline operations based on this data stream of dimension (1, 1), thereby realizing the pipeline of the entire data stream.

[0105] The other technical details and implementation steps of this embodiment 4 are the same as those of embodiment 1 and will not be repeated here.

[0106] Example 5

[0107] Based on Example 1, this Example 5 further describes the residual multiplication normalization module, which is as follows:

[0108] like Figure 5 As shown, it is a schematic block diagram of the structure of the residual multiplication normalization module. This module is responsible for completing the operation of formula (4) in Example 1. In the initial stage, the normalized weight data in the external storage will be input into the cache. Afterwards, the previous module calculates the residual z and provides data and valid signals, which are input into the residual z cache. When the output state y calculation module outputs valid data with a dimension of (1, 1), an enable signal is also given to the residual z cache one cycle in advance. In the next cycle, the valid data and the residual data read out of the cache will perform residual multiplication operations. Then, the result is input into the lookup table to complete the operation of the nonlinear function, and then multiplied by the normalized weight read from the cache, and finally a data with a dimension of (1, 1) is obtained. Subsequent modules will continue to perform pipeline operations based on this (1, 1) data stream.

[0109] The other technical details and implementation steps of this embodiment 5 are the same as those of embodiment 1 and will not be repeated here.

[0110] Example 6

[0111] Based on Example 1, this Example 6 further describes the output linear layer module, as follows:

[0112] like Figure 6As shown in the figure, it is a schematic block diagram of the structure of the output linear layer module. This module is responsible for transforming the dimension of the output vector, and its operation can be regarded as the multiplication of a vector and a matrix. Specifically, a vector with a dimension of (1, ed) is multiplied by a matrix with a dimension of (ed, d), that is, each element in the vector is multiplied and accumulated one by one with each column element of the matrix, and finally a result vector is obtained. During the calculation process, a pipeline design is adopted, and d multipliers are used for accumulation operations. In a single cycle, the residual multiplication normalization module outputs an element with a dimension of (1, 1) to the multiplier-adder array, and the weight matrix cache outputs a row of data to the multiplier-adder array. After ed (h*p) cycles, the operation is completed and the result vector with a dimension of (1, d) is output, thereby realizing the pipeline design. At this point, the data flow completes the pipeline design and solves the problem of data flow congestion.

[0113] The other technical details and implementation steps of this embodiment 6 are the same as those of embodiment 1 and will not be repeated here.

[0114] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization, characterized in that: The hardware acceleration system includes: a front-end module, a tensor expansion module, a hidden layer state calculation module, an output state calculation module, a residual multiplication normalization module and an output linear layer module. The hardware acceleration system divides the MambaV2 model algorithm calculation process into several pipeline stages and outputs data streams in a pipeline form; The front-stage module operates on the input data stream and outputs the tensor data stream required for the subsequent pipeline stage to operate; The tensor expansion module is the first pipeline stage, which is used to execute the dbx tensor expansion operation process after optimizing the calculation order in the MambaV2 model algorithm to reduce the number of calculations and reduce cache consumption; The hidden layer state calculation module is a second pipeline stage, which is used to implement the multiplication and addition operation when calculating the hidden layer state to obtain the hidden layer state; The output state calculation module is a third pipeline stage, used to calculate the output state according to the hidden layer state; The residual product normalization module is a fourth pipeline stage and a fifth pipeline stage, and performs a residual product operation and a normalization operation on the output state through two stages of pipelines to obtain an output vector; The output linear layer module is the sixth pipeline stage, which performs dimension transformation on the output vector to obtain a result vector.

2. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 1, characterized in that: The tensor expansion module is used to complete the dbx tensor expansion operation process, that is, to calculate the second term dbx(t) of the addition operation in the hidden layer state h. The expression for calculating the hidden layer state h is: h(t)=dA*h(t-1)+dbx(t) Wherein, h(t) is the hidden layer state at time t, dbx(t) is the influence of x on the hidden layer state, dA is the data in the tensor data stream, obtained by discretizing the input matrix A in the input data stream, and the discretization expression is as follows: dA=e A*dt Wherein, dt is the data in the tensor data stream; The dbx tensor expansion operation process after optimizing the calculation order in the MambaV2 model algorithm includes firstly performing tensor expansion on data dt and x to obtain an intermediate result tensor, and then performing tensor expansion on the intermediate result tensor and B to obtain dbx, where dt, x and B are all data in the tensor data stream. The specific calculation steps are: firstly performing tensor expansion on dt and x, and performing dot multiplication operation on the elements of vector dt and the elements of the vector with dimension (1, p) in tensor x in dimension h to obtain an intermediate result tensor with dimension (h, p). The calculation process is expressed as: einsum(h,hp→hp,dt,x) Among them, h represents the number of heads in the model, and p represents the dimension of the head; Then, the intermediate result tensor of dimension (h, p) is used to perform tensor expansion with B, and each element in the tensor dx of dimension (h, p) is multiplied with the vector B of dimension (1, n) to obtain the tensor dbx. The calculation process is expressed as: einsum(hp,n→hpn,dx,B) Here, n represents the hidden layer dimension.

3. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The tensor expansion module divides the data of dimension (h, p, n) into h*p period outputs, and each period outputs data of dimension (1, n). The subsequent modules perform pipeline operations based on this data stream of dimension (1, n).

4. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The reduction of the number of calculations and cache consumption is specifically as follows: The total number of multiplications required for dbx tensor expansion in the original MambaV2 model algorithm is h*n+h*n*p. Assuming that each element in the tensor consumes 1 storage unit, the total cache consumption is h+n+h*p+h*n storage units. After optimizing the calculation order, the total number of multiplications required for dbx tensor expansion is h*p+h*n*p, and the total cache consumption is h+n+2*h*p storage units. In the MambaV2 model algorithm, n represents the hidden layer dimension, which is generally 128, and p represents the head dimension, which is generally 64. By optimizing the calculation order, the number of multiplication calculations is greatly reduced and the consumption of storage resources is reduced.

5. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The hidden layer state calculation module includes an on-chip hidden layer state cache, each of which has a depth of h*p and a vector-expanded data of width (1, n) for storing the previous hidden layer state.

6. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The hidden layer state calculation module completes the multiplication and addition operation in the expression for calculating the hidden layer state; Among them, the calculation process of the term dA*h(t-1) in the expression for calculating the hidden layer state h is as follows: the input of the hidden layer state calculation module comes from the data stream with a dimension of (1, n) output by the tensor expansion module and the data dA in the tensor data stream calculated by the previous module, the dimension of the hidden layer state h is (h, p, n), the dimension of dA is (1, h), and the dimension of dbx is (h, p, n). During the calculation process, dA*h(t-1) adopts a broadcast mechanism to transform the dimension of dA into (h, 1, 1) and then perform a dot multiplication operation with h(t-1).

7. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 6, characterized in that: The hidden layer state calculation module uses n multipliers to perform parallel calculations, performs multiplication and addition operations in a single cycle, and outputs a data stream with a dimension of (1, n). Subsequent modules perform pipeline operations based on the data stream with a dimension of (1, n).

8. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The output state calculation module is used to complete the calculation of the output state y, and the calculation process is expressed as: y(t)=Ch(t)+Dx(t) Wherein, y(t) is the output state at time t, C is the tensor data in the tensor data stream, and D is the matrix in the input data stream; During the calculation process, the tensor broadcast mechanism is used to perform inner product operations. The products of the elements between the vectors are calculated in parallel by n multipliers, and then the inner product is calculated through the addition tree. The output state calculation module, i.e., the third-level pipeline, folds the addition operation to realize internal pipeline. The output state y tensor (h, p) is equivalent to (1, ed), ed=h*p. The output state y is divided into h*p cycles, and each cycle outputs data with a dimension of (1, 1). The subsequent modules perform pipeline operations according to this data stream with a dimension of (1, 1).

9. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 6, characterized in that: The residual multiplication normalization module completes the calculation of the following formula: Y(t)=RMSnorm(y(t)*z) Wherein, Y(t) represents the output vector, and the residual z is the data in the tensor data stream; In the residual multiplication and normalization module, the residual product operation part is the fourth pipeline stage, and the normalization operation part is the fifth pipeline stage, wherein the operation of the nonlinear function in the normalization operation part is implemented by a table lookup method, and the residual multiplication and normalization module outputs data with a dimension of (1, 1) in each cycle, and the subsequent modules will perform pipeline operations based on this data stream with a dimension of (1, 1).

10. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The output linear layer module transforms the dimension of the vector, that is, a vector with a dimension of (1, ed) is multiplied by a matrix with a dimension of (ed, d), each element in the vector is multiplied and accumulated with each column element of the matrix one by one, and d multipliers are used to perform accumulation operations to finally obtain a result vector. In a single cycle, only one element is input into the input vector, which matches the data flow inputted by the previous pipeline stage, thereby realizing pipeline design and avoiding data flow blockage.

Images