A high-parallelism MambaV2 hardware acceleration system based on FPGA for computing order optimization

Abstract

The application discloses a high-parallelism MambaV2 hardware acceleration system based on FPGA and optimized in calculation sequence, relates to the fields of FPGA and machine learning, and comprises a front-stage module, a tensor expansion module, a hidden layer state calculation module, an output state calculation module, a residual multiplication and normalization module and an output linear layer module, wherein the MambaV2 algorithm calculation process is divided into a plurality of pipeline stages and data streams are output in a pipeline form. The application does not change the original algorithm structure, but changes the operation sequence of the hidden layer state equation in the original algorithm, proposes an algorithm optimized in calculation sequence, reduces the number of operations, reduces the cache consumption, realizes the deployment of a reasoning MambaV2 model on an FPGA, meets the deployment requirements of edge scenes, utilizes the customized acceleration of the FPGA, realizes the pipeline design of the overall algorithm through the adjustment of data streams, improves the utilization rate of calculation resources, solves the data blocking problem, and reduces the calculation delay and the cache consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of FPGA and machine learning, more particularly, to a high pipelined MambaV2 hardware acceleration system based on FPGA computing sequence optimization. BACKGROUND

[0002] Mamba model is an important architecture in the field of machine learning in recent years, especially in handling large-scale data and complex computing tasks. With the rapid development of deep learning, traditional model architecture and optimization method faces the problems of large consumption of computing resources, long training time, etc. In order to solve these challenges, Mamba model emerges as the times require, aiming to provide more efficient computing performance and flexible architecture design, and then evolves into MambaV2 model, which has been widely used in natural language processing (NLP), computer vision (CV) and other fields.

[0003] With the increase of model complexity and size, the parameter quantity of MambaV2 model increases sharply, and the demand for computing resources also rises. How to efficiently deploy MambaV2 model on hardware has become a difficult problem to be solved. Although GPU is the traditional deep learning accelerator, in some scenarios, FPGA is considered as a better hardware choice, especially in some performance, energy efficiency and flexibility. Deploying MambaV2 model on FPGA has its specific technical advantages and application needs. FPGA has the advantage of energy efficiency ratio, and can realize customized hardware acceleration, which has a large demand in edge computing application scenarios.

[0004] However, current mainstream design has few MambaV2 models deployed on FPGA, and existing technical implementation mostly focuses on training and inference on GPU. The main reason is that GPU has mature parallel computing architecture and extensive development support. However, GPU has obvious limitations in dealing with some specific scenarios, while FPGA platform provides new possibilities. Therefore, how to deploy MambaV2 model on FPGA to meet the deployment needs of edge scenarios while optimizing data flow, reducing computing delay, improving computing resource utilization and reducing storage resource consumption has become a problem to be solved. SUMMARY

[0005] The purpose of the present application is to overcome the problem of how to deploy MambaV2 model on FPGA to meet the deployment needs of edge scenarios while optimizing data flow, reducing computing delay, improving computing resource utilization and reducing storage resource consumption in the prior art, and to provide a high pipelined MambaV2 hardware acceleration system based on FPGA computing sequence optimization.

[0006] To achieve the above object, the technical scheme of the present application is as follows:

[0007] A high-parallelism MambaV2 hardware acceleration system based on FPGA and optimized in calculation sequence, comprising a front-stage module, a tensor expansion module, a hidden layer state calculation module, an output state calculation module, a residual product normalization module and an output linear layer module, wherein the hardware acceleration system divides the calculation process of the MambaV2 model algorithm into several pipeline stages and outputs data streams in a pipeline form;

[0008] The front-stage module operates on input data streams and outputs tensor data streams required for the operation of subsequent pipeline stages;

[0009] The tensor expansion module is the first pipeline stage and is used to perform the dBx tensor expansion operation process in the MambaV2 model algorithm after the calculation sequence is optimized to reduce the number of calculations and the cache consumption;

[0010] The hidden layer state calculation module is the second pipeline stage and is used to implement the multiplication and addition operation when calculating the hidden layer state to obtain the hidden layer state;

[0011] The output state calculation module is the third pipeline stage and is used to calculate the output state according to the hidden layer state;

[0012] The residual product normalization module is the fourth and fifth pipeline stages, which performs residual product operation and normalization operation on the output state through two-stage pipeline to obtain an output vector;

[0013] The output linear layer module is the sixth pipeline stage, which performs dimension transformation on the output vector to obtain a result vector.

[0014] As a preferred scheme, the tensor expansion module is used to complete the dBx tensor expansion operation process, i.e., to calculate the second term dBx(t) of the addition operation in the hidden layer state, and the expression for calculating the hidden layer state h is as follows:

[0015] h(t)=dA*h(t-1)+dBx(t)

[0016] wherein h(t) is the hidden layer state at time t, dBx(t) is the influence of x on the hidden layer state, dA is data in the tensor data stream, which is obtained by discretizing the input matrix A in the input data stream, and the discretization expression is as follows:

[0017] dA=e A*dt

[0018] wherein dt is the data in the tensor data stream;

[0019] The dBx tensor expansion operation process after optimizing the calculation order in the MambaV2 model algorithm includes first performing tensor expansion on the data dt and x to obtain an intermediate result tensor, and then performing tensor expansion on the intermediate result tensor and B to obtain dBx. dt, x, and B are all data in the tensor data stream. The specific calculation steps are: first performing tensor expansion on dt and x, and performing a dot product operation on the elements of the vector dt and the elements of the vector with dimension (1, p) in the tensor x in the h dimension to obtain an intermediate result tensor with dimension (h, p). The calculation process is expressed as:

[0020] einsum(h,hp→hp,dt,x)

[0021] Where h represents the number of heads in the model and p represents the dimension of the head;

[0022] Then, the intermediate result tensor of dimension (h, p) is used to perform tensor expansion with B, and each element in the tensor dx of dimension (h, p) is multiplied by the vector B of dimension (1, n) to obtain the tensor dBx. The calculation process is expressed as:

[0023] einsum(hp,n→hpn,dx,B)

[0024] Here, n represents the hidden layer dimension.

[0025] As a preferred solution, the tensor expansion module divides the data of dimension (h, p, n) into h*p cycles for output, and each cycle outputs data of dimension (1, n). The subsequent modules perform pipeline operations based on this data stream of dimension (1, n).

[0026] As a preferred solution, the reduction in the number of calculations and cache consumption is specifically as follows:

[0027] The total number of multiplications required for dBx tensor expansion in the original MambaV2 model algorithm is h*n+h*n*p. Assuming that each element in the tensor consumes 1 storage unit, the total cache consumption is h+n+h*p+h*n storage units.

[0028] After optimizing the calculation order, the total number of multiplications required for dBx tensor expansion is h*p+h*n*p, and the total cache consumption is h+n+2*h*p storage units. In the MambaV2 model algorithm, n represents the hidden layer dimension, which is generally 128, and p represents the head dimension, which is generally 64. By optimizing the calculation order, the number of multiplication calculations is greatly reduced and the consumption of storage resources is reduced.

[0029] As a preferred solution, the hidden layer state calculation module comprises on-chip hidden layer state caches, each of which has a depth of h*p and a width of a vector-expanded data of (1, n) for storing previous hidden layer states.

[0030] As a preferred solution, the hidden layer state calculation module completes the multiply-add operation in the expression of the calculated hidden layer state h.

[0031] The calculation of the dA*h(t-1) term in the expression of the calculated hidden layer state h is as follows: the input of the hidden layer state calculation module comes from the data stream of dimension (1, n) output by the tensor expansion module and the data dA in the tensor data stream calculated by the previous module, the dimension of the hidden layer state h is (h, p, n), the dimension of dA is (1, h), and the dimension of dBx is (h, p, n); in the calculation process, dA*h(t-1) adopts a broadcast mechanism to transform the dimension of dA into (h, 1, 1) and then performs a dot product operation with h(t-1).

[0032] As a preferred solution, the hidden layer state calculation module adopts n multiply-adders for parallel calculation, performs a multiply-add operation in a single cycle, and outputs a data stream of dimension (1, n), and the subsequent module performs pipeline operation according to the data stream of dimension (1, n).

[0033] As a preferred solution, the output state calculation module is configured to complete the calculation of the output state y, and the calculation process is represented as:

[0034] y(t) = Ch(t) + Dx(t)

[0035] wherein y(t) is the output state at time t, C is a tensor data in the tensor data stream, and D is a matrix stored in an on-chip cache of an FPGA.

[0036] In the calculation process, the inner product operation is performed using the broadcast mechanism of the tensor, the product of elements between vectors is calculated through n multipliers in parallel, and then the inner product is calculated through an addition tree; the output state calculation module, i.e., the third stage of the pipeline, folds the addition operation to realize internal pipelining, and the output state y tensor has a dimension of (h, p) and is equivalent to (1, ed), ed = h*p; the output state y is divided into h*p periods, and a data of dimension (1, 1) is output in each period; and the subsequent module performs pipeline operation according to the data stream of dimension (1, 1).

[0037] As a preferred solution, the residual multiplication and normalization module completes the calculation of the following formula:

[0038] Y(t) = RMSnorm(y(t)*z)

[0039] Wherein, Y(t) represents an output vector, and the residual z is data in the tensor data stream;

[0040] In the residual multiplication and normalization module, the residual multiplication operation part is the fourth pipeline stage, and the normalization operation part is the fifth pipeline stage, wherein the operation of the nonlinear function in the normalization operation part is realized by a lookup table method, and the residual multiplication and normalization module outputs one dimension (1, 1) data per cycle, and the subsequent module will perform pipeline operation according to the dimension (1, 1) data stream.

[0041] As a preferred scheme, the output linear layer module performs dimension transformation on the vector, that is, a vector with a dimension of (1, ed) is multiplied by a matrix with a dimension of (ed, d), wherein ed=h*p, each element in the vector is multiplied with each column element of the matrix one by one and accumulated, d adders are used for accumulation operation, and finally a result vector is obtained, in a single cycle, only one element of the input vector is input, which matches the data stream input by the previous pipeline stage, so that the pipeline design is realized, and data flow blockage is avoided.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] 1. The present application changes the operation sequence of the hidden layer state equation in the MambaV2 model algorithm without changing the original algorithm, and proposes an algorithm for optimizing the calculation sequence, which can reduce the number of operations and reduce the consumption of storage resources.

[0044] 2. The high-flow MambaV2 hardware accelerator system of the present application realizes deployment of the inference MambaV2 model on the FPGA, and can meet the deployment requirements of the edge scene.

[0045] 3. The present application utilizes the customized acceleration of FPGA, realizes the pipeline design of the overall algorithm by adjusting the data stream, improves the utilization rate of computing resources, solves the problem of data blocking, reduces the calculation delay, and also reduces the consumption of on-chip cache. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The present application is a high-flow MambaV2 hardware acceleration system for optimizing the calculation sequence of FPGA-based computing, and the overall framework diagram thereof is shown in the figure;

[0047] Figure 2 The structure diagram of the dBx tensor expansion module of the embodiment of the present application is shown in the figure

[0048] Figure 3 The structure diagram of the hidden layer state calculation module of the embodiment of the present application is shown in the figure

[0049] Figure 4A structural diagram of an output state calculation module of an embodiment of the present application;

[0050] Figure 5 A structural diagram of a residual multiplication and normalization module of an embodiment of the present application;

[0051] Figure 6 A structural diagram of an output linear layer module of an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0053] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, and therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0054] Embodiment 1

[0055] Please refer to Figure 1 The embodiment 1 of the present application provides a high pipelining MambaV2 hardware acceleration system based on FPGA for calculation sequence optimization, comprising:

[0056] The front stage module, the dBx tensor expansion module, the hidden layer state calculation module, the output state y calculation module, the residual multiplication and normalization module and the output linear layer module, the hardware acceleration system divides the MambaV2 model algorithm calculation process into several pipelining stages and adopts pipelining form to output data stream;

[0057] The front stage module operates on the input data stream and outputs the tensor data stream required for the operation of the subsequent pipelining stage;

[0058] The dBx tensor expansion module is the first pipelining stage, which is used to execute the dBx tensor expansion operation process after the calculation sequence is optimized in the MambaV2 model algorithm to reduce the calculation times and reduce the cache consumption;

[0059] The hidden layer state calculation module is the second pipelining stage, which is used to realize the multiplication and addition operation when calculating the hidden layer state to obtain the hidden layer state;

[0060] The output state y calculation module is the third pipelining stage, which is used to calculate the output state according to the hidden layer state;

[0061] The residual multiplication and normalization module is the fourth pipelining stage and the fifth pipelining stage, which performs residual multiplication operation and normalization operation on the output state through two-stage pipelining to obtain the output vector;

[0062] The output linear layer module is the sixth flow level, which performs dimension transformation on the output vector to obtain a result vector.

[0063] In the original MambaV2 model algorithm, the input vector u is first mapped by the input linear layer, then the causal convolution and activation function operation are performed respectively to obtain x, B, C, dt and z, and then the input matrix A is discretized:

[0064] dA = e A*dt (1)

[0065] Then, the hidden layer state is calculated:

[0066] h(t) = dA * h(t-1) + dBx(t) (2)

[0067] Then, the output state is calculated:

[0068] y(t) = Ch(t) + Dx(t) (3)

[0069] Residual multiplication and normalization operations are performed:

[0070] Y(t) = RMSnorm(y(t) * z) (4)

[0071] Finally, the output linear layer mapping is performed.

[0072] The present application does not discuss the calculation order of the previous module, and defaults that the data stream operation of the previous module is completed, only the operation data stream of formula (2) and thereafter, and the hardware implementation of the hardware accelerator system are discussed. In the original MambaV2 model algorithm, the specific calculation process of the influence dBx(t) of x on the hidden layer state in formula (2) is as follows:

[0073] Tensor expansion is performed using tensors dt and B to obtain dB, and then tensor expansion is performed with tensor x to obtain dBx. The specific process is represented as follows:

[0074] einsum(h, n, hp→hpn", dt, B, x) (5)

[0075] wherein n represents the hidden layer dimension, h represents the number of heads in the model, and p represents the dimension of the head, specifically, the outer product of the (1, h) dimension vector in dt and the (1, n) dimension vector in B is performed to obtain a (h, n) dimension matrix, and then the outer product of the (1, n) dimension vector and the (1, p) dimension vector in x is performed under each h dimension to finally obtain a (h, p, n) dimension tensor, which is the complete tensor expansion process of the original MambaV2 model algorithm. At this time, it can be known that: first, the tensor expansion of the vector dt and the vector B needs to perform h*n multiplications; and then the tensor expansion with the vector x needs to perform h*n*p multiplications. Therefore, the number of multiplications required for the complete tensor expansion of dBx in the original MambaV2 model algorithm is h*n+h*n*p.

[0076] Assuming that each element in the tensor consumes 1 storage unit, the cache consumption of the initial vector is h+n+h*p storage units, and the cache consumption of the intermediate result is h*n storage units. Therefore, the total cache consumption is h+n+h*p+h*n storage units.

[0077] The calculation order of the dBx tensor expansion operation process is optimized and modified, and this part of operation is implemented in the tensor expansion module. The specific calculation process after optimizing the calculation order is as follows:

[0078] First, let dt and x perform tensor expansion,

[0079] einsum(h,hp→hp,dt,x) (6)

[0080] The elements of the vector dt and the vector elements of dimension (1, p) in the tensor x are multiplied under the h dimension to obtain an intermediate result tensor dx of dimension (h, p), at this time the number of multiplications is h*p, and the cache consumption of the intermediate result is h*p storage units. Then the intermediate result tensor dx of dimension (h, p) is used for tensor expansion with B,

[0081] einsum(hp,n→hpn,dx,B) (7)

[0082] The product of each element in the tensor dx of dimension (h, p) and the vector B of dimension (1, n) is performed, which needs to perform h*p*n multiplications. Therefore, the complete process of the dBx tensor expansion after optimizing and modifying the calculation order needs h*p+h*n*p multiplications, and the total cache consumption is h+n+2*h*p storage units. In the MambaV2 model algorithm, n represents the hidden layer dimension, which is generally 128, and p represents the dimension of the head in the model, which is generally 64. Therefore, by modifying the calculation order, the number of multiplications can be greatly reduced, and the consumption of storage resources can also be reduced.

[0083] Referring to Figure 1 , for the overall architecture block diagram of an FPGA-based computing sequence optimization of a high-parallelism MambaV2 hardware acceleration system in Embodiment 1, in order to improve acceleration performance, reduce computing delay, and improve computing unit utilization, the application proposes a pipeline design scheme to implement the MambaV2 hardware acceleration system architecture, which divides the computing part after the previous module into six pipeline stages, the first pipeline stage is the dBx tensor expansion module, the second pipeline stage is the hidden layer state computing module, the third pipeline stage is the output state y computing module, the fourth and fifth pipeline stages are the residual multiplication and normalization module, and the last pipeline stage is the output linear layer module, which realizes the overall pipeline design, optimizes the data flow, reduces the computing delay, improves the utilization of the computing unit, and reduces the consumption of storage resources.

[0084] The previous module includes input vectors u and weight matrix cache, which are first mapped to the input linear layer, then subjected to causal convolution and activation function operation to obtain tensor data streams x, B, C, dt and z, and then the input matrix A is discretized to obtain dA, wherein the input matrix A is a randomly generated matrix obtained through training and stored in the on-chip cache of the FPGA.

[0085] The dBx tensor expansion module performs the dBx tensor expansion process of the optimized and modified computing sequence of formula (2). In the process of expanding the intermediate tensor dx and B, the data with a dimension of (1, n) is output in a pipeline manner for h*p cycles to complete the calculation, and the subsequent modules are designed in a pipeline manner according to this data stream.

[0086] The hidden layer state computing module implements the multiplication and addition operation of formula (2). The computing module includes on-chip hidden layer state cache, each on-chip cache has a depth of h*p and a width of (1, n) vector expanded data for storing the previous hidden layer state. The input is the data stream from the dBx tensor expansion module (1, n) and the dA calculated by the previous module. The dimension of the hidden layer state is (h, p, n), the dimension of dA is (1, h), and the dimension of dBx is (h, p, n). In formula (2), dA*h(t-1) adopts a broadcast mechanism to transform the dimension of dA to (h, 1, 1) and then perform dot multiplication with h(t-1). During the execution of the computing process, n multipliers are used for parallel computation, and multiplication and addition operations are performed in a single cycle to output a data stream with a dimension of (1, n), realizing pipeline design and reducing delay.

[0087] The output state y calculation module implements the operation of formula (3), which involves using the tensor broadcast mechanism to perform inner product operations. The products of the elements between vectors are calculated in parallel by n multipliers, and then the inner product is calculated through the addition tree. This pipeline stage will fold the addition operation to achieve internal pipeline, and the pipeline output dimension is (1, 1).

[0088] The residual product normalization module implements the operation of formula (4), performs the residual product operation through the fourth pipeline stage, and performs the normalization operation through the fifth pipeline stage. Among them, the normalization of the nonlinear function in the normalization operation is realized by the table lookup method.

[0089] Finally, the output linear layer module transforms the vector dimensions. This operation can be considered a vector-matrix multiplication. Specifically, a vector of dimension (1, ed) is multiplied by a matrix of dimension (ed, d), where ed = h*p. This means that each element in the vector is multiplied and added to each column of the matrix, resulting in a final result vector. A pipelined design employs d multipliers and accumulators for accumulation. In a single cycle, only one element of the input vector needs to be input, which perfectly matches the data flow of the previous input, thus enabling a pipelined design and resolving data flow congestion issues.

[0090] In this embodiment 1, the process of applying the hardware acceleration system to perform an inference acceleration task is as follows:

[0091] The weight matrix and input vector are loaded from the external memory into the on-chip memory of the FPGA. The front-stage module calculates the tensors B, x, dt, dA and C required by the subsequent pipeline stages through operations. Then, the dBx tensor expansion module and the hidden layer state calculation module complete the operation of formula (2), the output state y calculation module completes the operation of formula (3), and the residual multiplication normalization module completes the operation of formula (4). The output linear layer module transforms the output vector to obtain the result vector, and then the result vector is used as the input vector again to continue iteration.

[0092] Example 2

[0093] This embodiment 2 is based on embodiment 1 and further explains the dBx tensor expansion module. Figure 2 The figure below is a schematic diagram of the structure of the dBx tensor expansion module. Combined with the algorithm process after the calculation order is optimized, the workflow of the dBx tensor expansion module is explained in detail through the calculation process of a pipeline output vector. The details are as follows:

[0094] The front stage module inputs the calculated tensor dt, B and x into the on-chip cache of FPGA, and then executes the algorithm after optimizing and modifying the calculation sequence. The data with the dimension of (1, h) output from the dt cache and the matrix data with the dimension of (h, p) output from the x cache are input into the multiplier array together, the data with the dimension of (h, p) output are stored into the dx cache, and then every single period, one element output from the dx cache and one column data with the dimension of (1, n) output from the B cache are input into the multiplier array, and then the data with the dimension of (1, n) are output. In essence, the data with the dimension of (h, p, n) output are divided into h*p periods of output, and the data with the dimension of (1, n) are output every period, and the subsequent module performs pipelining operation according to the (1, n) data stream.

[0095] The other technical details and implementation steps of this embodiment 2 are the same as those of embodiment 1, which will not be repeated here.

[0096] Embodiment 3

[0097] This embodiment 3 is based on embodiment 1, and further describes the hidden layer state calculation module, as follows:

[0098] As Figure 3As shown, it is a schematic block diagram of the structure of the hidden layer state calculation module. This module completes the calculation of formula (2) in Example 1. The second term of the addition in formula (2) has been calculated by the dBx tensor expansion module. The input of the hidden layer state calculation module is a data stream with a dimension of (1, n) from the dBx tensor expansion module, and the previous module has calculated the dA tensor and stored it in the dA cache. The hidden layer state calculation module contains a hidden layer state cache, which stores the previous hidden layer state. In order to adapt to the data stream with a dimension of (1, n), the depth of each on-chip hidden layer state cache is h*p, and the data width is the vector expanded data of (1, n). There are a total of layerCount hidden layer caches, which depend on the number of layers of the deployment algorithm MambaV2. The corresponding cache will be accessed according to the current number of layers. When the dBx tensor expansion module inputs valid vector data with a dimension of (1, n), the dBx tensor expansion module will output an enable signal one cycle in advance. This enable signal will be input into the dA cache and the hidden layer state cache. In the next cycle, the dBx tensor expansion module outputs a valid vector of dimension (1, n). Simultaneously, the dA buffer and the hidden layer state buffer also output corresponding data. This data is fed into the multiplier-accumulator array, resulting in a valid output vector of dimension (1, n), enabling data pipeline processing. At this point, the hidden layer state (i.e., the tensor of dimension (h, p, n)) is split into h*p cycles, each of which outputs a data of dimension (1, n). Subsequent modules continue to perform pipeline operations based on this (1, n) data stream, thus streamlining the entire data flow.

[0099] Other technical details and implementation steps of this embodiment 3 are the same as those of embodiment 1 and will not be repeated here.

[0100] Example 4

[0101] This embodiment 4 is based on embodiment 1 and further describes the output state y calculation module, as follows:

[0102] like Figure 4 As shown, it is a schematic block diagram of the structure of the output state y calculation module. This module completes the operation of formula (3) in Example 1. The previous module calculates x to give data and valid signals, inputs them into the output state y calculation module, and multiplies them with the matrix D stored in the cache on the FPGA chip (the same way as the input matrix A is obtained and stored) to calculate the Dx tensor stored in the cache. The vector data with a dimension of (1, n) output by the hidden layer h calculation module and the vector data with a dimension of (1, n) input by the tensor C of the previous module are input into the multiplier array, and the element products between the vectors are calculated in parallel, and then the inner product of the two vectors is calculated through a balanced addition tree:

[0103] (1, n)@(n, 1)=(1, 1)

[0104] The result is then added to the Dx tensor elements in the cache to calculate the result. Essentially, the output state y, a tensor of dimension (h, p) (equivalently (1, ed), where ed = h*p), is split into h*p cycles, with each cycle outputting a data stream of dimension (1, 1). Subsequent modules continue to perform pipeline operations based on this data stream of dimension (1, 1), thus streamlining the entire data flow.

[0105] Other technical details and implementation steps of this embodiment 4 are the same as those of embodiment 1 and will not be repeated here.

[0106] Example 5

[0107] This embodiment 5 is based on embodiment 1 and further describes the residual multiplication and normalization module, as follows:

[0108] like Figure 5 As shown, it is a structural schematic block diagram of the residual multiplication normalization module. This module is responsible for completing the operation of formula (4) in Example 1. In the initial stage, the normalized weight data in the external storage will be input into the cache. Afterwards, the previous module calculates the residual z and provides data and valid signals, which are input into the residual z cache. When the output state y calculation module outputs valid data with a dimension of (1, 1), an enable signal is also given to the residual z cache one cycle in advance. In the next cycle, the valid data and the residual data read out of the cache will perform the residual multiplication operation. Then, the result is input into the lookup table to complete the operation of the nonlinear function, and then multiplied by the normalized weight read from the cache, and finally a data with a dimension of (1, 1) is obtained. The subsequent modules will continue to perform pipeline operations based on this (1, 1) data stream.

[0109] Other technical details and implementation steps of this embodiment 5 are the same as those of embodiment 1 and will not be repeated here.

[0110] Example 6

[0111] Based on Example 1, this Example 6 further describes the output linear layer module, as follows:

[0112] like Figure 6As shown, it is a structural schematic block diagram of the output linear layer module. This module is responsible for dimension transformation of the output vector, and its operation can be regarded as the multiplication of vector and matrix. Specifically, a vector with a dimension of (1, ed) is multiplied by a matrix with a dimension of (ed, d), that is, each element in the vector is multiplied by each column element of the matrix one by one and accumulated, and finally the result vector is obtained. In the calculation process, a pipeline design is adopted, and d multipliers are used for accumulation operation. In a single cycle, the residual multiplication normalization module outputs an element with a dimension of (1, 1) to the multiplier array, the weight matrix cache outputs a row of data to the multiplier array, and the operation is completed after ed(h*p) cycles, and a result vector with a dimension of (1, d) is output, thereby realizing the pipeline design, and thus the data flow completes the pipeline design, solving the problem of data flow blockage.

[0113] The other technical details and implementation steps of this embodiment 6 are the same as those of embodiment 1, and will not be repeated here.

[0114] Obviously, the above embodiments of the application are only examples for clearly illustrating the application, and are not intended to limit the implementation manner of the application. Based on the above description, those skilled in the art can make other different forms of changes or modifications. Here, all the implementation manners are not required or can not be exhausted. Any modification, equivalent replacement and improvement made within the spirit and principle of the application should be included in the protection scope of the claims of the application.

Claims

1. A high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization, characterized in that: The hardware acceleration system includes: a front-end module, a tensor expansion module, a hidden layer state calculation module, an output state calculation module, a residual multiplication normalization module, and an output linear layer module. The hardware acceleration system divides the MambaV2 model algorithm calculation process into several pipeline stages and outputs data streams in a pipeline form; The front-stage module operates on the input data stream and outputs the tensor data stream required for the subsequent pipeline stage to perform operations; The tensor expansion module is the first pipeline stage, which is used to perform the dBx tensor expansion operation process after optimizing the calculation order in the MambaV2 model algorithm to reduce the number of calculations and reduce cache consumption; The hidden layer state calculation module is the second pipeline stage, which is used to implement the multiplication and addition operation when calculating the hidden layer state to obtain the hidden layer state; The output state calculation module is a third pipeline stage, used to calculate the output state according to the hidden layer state; The residual multiplication and normalization modules are the fourth and fifth pipeline stages, and perform residual product operations and normalization operations on the output state through two stages of pipelines to obtain an output vector; The output linear layer module is the sixth pipeline stage, which performs dimension transformation on the output vector to obtain a result vector; The dBx tensor expansion operation process after optimizing the calculation order in the MambaV2 model algorithm includes first performing tensor expansion on the data dt and x to obtain an intermediate result tensor, and then performing tensor expansion on the intermediate result tensor and B to obtain dBx. dt, x, and B are all data in the tensor data stream. The specific calculation steps are: first performing tensor expansion on dt and x, and then performing a dot product operation on the elements of the vector dt and the elements of the vector with dimension (1, p) in the tensor x in the h dimension to obtain an intermediate result tensor with dimension (h, p). The calculation process is expressed as: Where h represents the number of heads in the model and p represents the dimension of the head; Then, the intermediate result tensor of dimension (h, p) is used to perform tensor expansion with B, and each element in the tensor dx of dimension (h, p) is multiplied by the vector B of dimension (1, n) to obtain the tensor dBx. The calculation process is expressed as: Here, n represents the hidden layer dimension.

2. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 1, characterized in that: The tensor expansion module is used to complete the dBx tensor expansion operation process, that is, to calculate the second term of the addition operation in the hidden layer state h , the expression for calculating the hidden layer state h is: in, is the hidden layer state at time t, is the influence of x on the hidden layer state, dA is the data in the tensor data stream, which is obtained by discretizing the input matrix A in the input data stream. The discretization expression is as follows: Wherein, dt is the data in the tensor data stream.

3. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The tensor expansion module divides the data of dimension (h, p, n) into h The output is p cycles, and each cycle outputs data of dimension (1, n). Subsequent modules perform pipeline operations based on this data stream of dimension (1, n).

4. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: Reduce the number of calculations and cache consumption, specifically: The total number of multiplications required for the dBx tensor expansion in the original MambaV2 model algorithm is h n + h n p, assuming that each element in the tensor consumes 1 storage unit, the total cache consumption is h + n + h p + h n storage units; The total number of multiplications required for the dBx tensor expansion after optimizing the calculation order is h p + h n p, the total cache consumption is h + n + 2 h There are p storage units. In the MambaV2 model algorithm, n represents the hidden layer dimension, which is generally 128, and p represents the head dimension, which is generally 64. By optimizing the calculation order, the number of multiplication calculations is greatly reduced and the consumption of storage resources is reduced.

5. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The hidden layer state calculation module includes an on-chip hidden layer state cache, each of which has a cache depth of h p, the expanded data of a vector with a width of (1, n), is used to store the previous hidden layer state.

6. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The hidden layer state calculation module completes the multiplication and addition operation in the expression for calculating the hidden layer state; Among them, dA in the expression for calculating the hidden layer state h The calculation process of h(t-1) is as follows: the input of the hidden layer state calculation module comes from the data stream with the dimension of (1, n) output by the tensor expansion module and the data dA in the tensor data stream calculated by the previous module. The dimension of the hidden layer state h is (h, p, n), the dimension of dA is (1, h), and the dimension of dBx is (h, p, n). During the calculation process, dA h(t-1) uses a broadcast mechanism to transform the dimension of dA to (h, 1, 1) and then perform a dot product operation with h(t-1).

7. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 6, characterized in that: The hidden layer state calculation module uses n multipliers to perform parallel calculations, performs multiplication and addition operations in a single cycle, and outputs a data stream with a dimension of (1, n). Subsequent modules perform pipeline operations based on this data stream with a dimension of (1, n).

8. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 2, characterized in that: The output state calculation module is used to complete the calculation of the output state y, and the calculation process is expressed as: in, is the output state at time t, C is the tensor data in the tensor data stream, and D is the matrix in the input data stream; During the calculation process, the tensor broadcast mechanism is used to perform the inner product operation. The product of the elements between the vectors is calculated in parallel by n multipliers, and then the inner product is calculated through the addition tree. The output state calculation module, that is, the third-level pipeline, folds the addition operation to realize the internal pipeline and outputs the state y tensor (h, p), which is equivalent to (1, ed), ed = h p, split the output state y into h There are p cycles, and each cycle outputs data with a dimension of (1, 1). Subsequent modules perform pipeline operations based on this data stream with a dimension of (1, 1).

9. The high-flow MambaV2 hardware acceleration system based on FPGA calculation sequence optimization according to claim 6, characterized in that: The residual multiplication and normalization module completes the calculation of the following formula: in, represents the output vector, and the residual z is the data in the tensor data stream; In the residual multiplication and normalization module, the residual product operation part is the fourth pipeline stage, and the normalization operation part is the fifth pipeline stage. The operation of the nonlinear function in the normalization operation part is implemented by a table lookup method. The residual multiplication and normalization module outputs a data with a dimension of (1, 1) in each cycle, and the subsequent modules will perform pipeline operations based on this data stream with a dimension of (1, 1).

10. The high-flow MambaV2 hardware acceleration system based on FPGA and optimized calculation sequence according to claim 2, characterized in that: The output linear layer module transforms the vector dimension. That is, a vector of dimension (1, ed) is multiplied by a matrix of dimension (ed, d). Each element in the vector is multiplied and accumulated with each column element of the matrix one by one. D multipliers are used for accumulation operation to finally obtain the result vector. In a single cycle, only one element is input into the input vector, matching the data flow of the previous pipeline stage input, thereby realizing pipeline design and avoiding data flow congestion.

Images