An FPGA-based Transformer hardware accelerator

Abstract

The application discloses a kind of based on FPGA's Transform hardware accelerator;It includes data preprocessing module, computing unit and controller;Computing unit includes matrix processing unit and nonlinear processing unit;Nonlinear processing unit includes activation function unit, layer standardization module and Softmax module;Data preprocessing module is deployed on CPU, controller and computing unit are deployed on FPGA, data preprocessing module is preprocessed to the text needing processing, is converted into integer sequence and is input into FPGA, data is scheduled in computing unit by controller and completes computing process, finally obtains output result.The application only needs to use less logic resource when calculating, can obtain result in shorter time, while not affecting the final calculation accuracy.

Description

Technical Field

[0001] This invention belongs to the field of hardware accelerator technology, specifically, it relates to an FPGA-based Transformer hardware accelerator. Background Technology

[0002] Transformer is a deep learning algorithm that has emerged in recent years. Initially used for machine translation, Transformer and its derivatives have been applied to many other fields, such as BERT for natural language understanding, GPT for language generation models, and ViT for image processing. Unlike traditional text processing algorithms, Transformer introduces an attention mechanism, which allows it to capture textual dependencies with large word spacing, thus improving accuracy. Furthermore, the attention mechanism supports parallel computation, further enhancing its practicality.

[0003] Traditionally, deep learning algorithms are deployed using CPUs or GPUs. However, as algorithms become increasingly large-scale, CPUs with lower parallelism cannot efficiently complete the inference process. Meanwhile, the development of edge computing has made power efficiency increasingly important, and the high power consumption of GPUs limits their application at the edge. FPGAs, as programmable devices, can simultaneously meet the requirements of parallelism and low power consumption, making them an ideal edge computing device.

[0004] Sehoon Kim et al. used quantization to compress the Transformer-derived model BERT in their paper, *Kim S, Gholami A, Yao Z, et al. I-bert: Integer-only bert quantization[C] / / International conference on machine learning. PMLR,2021: 5506-5518*. The authors employed a full quantization scheme. Specifically, for linear layers, weights and activations were compressed to 8 bits. For nonlinear layers, approximation functions were used to replace the nonlinear functions in the model, resulting in performance improvements. However, this work still requires computation of the complete model and does not remove redundant parts, thus leaving room for improvement in computational efficiency and hardware consumption.

[0005] Zejian Liu et al., in their paper "Hardware acceleration of fully quantized BERT for efficient natural language processing [C] / / 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2021: 513-516," used quantization to compress the model and designed a multiplication unit to calculate the high and low bits separately, reducing the hardware consumption of the model. However, this design still only uses quantization for computation and cannot fully optimize model computation and hardware. Summary of the Invention

[0006] To address the issues of high computational latency and excessive storage and computational resource consumption in existing Transformer algorithms, this invention employs model compression techniques such as quantization and pruning to design an FPGA hardware architecture for implementing the Transformer algorithm. This invention uses a polynomial function that includes only multiply-accumulate operations to approximate the nonlinear functions in the model, thereby improving the model's speed. Furthermore, this invention requires fewer logic resources during computation, enabling results to be obtained in a shorter time without compromising final computational accuracy. The specific technical solution of this invention is as follows.

[0007] An FPGA-based Transformer hardware accelerator includes a data preprocessing module, a computing unit, and a controller. The data preprocessing module is deployed on a CPU, while the controller and computing unit are deployed on the FPGA. The data preprocessing module preprocesses the text to be processed, converting it into an integer sequence that is input into the FPGA. The data in the computing unit undergoes computation under the scheduling of the controller to obtain the final output result.

[0008] The computational unit includes a matrix processing unit and a nonlinear processing unit;

[0009] The matrix processing unit is used to compute matrix multiplication in multi-head attention mechanisms and feedforward neural networks, as well as residual connections before normalization in the execution layer. The matrix processing unit includes a matrix multiplication unit and a matrix addition unit. The matrix multiplication unit adopts a systolic array structure. In the matrix multiplication unit, the input buffer is connected to the systolic array, intermediate buffer, and accumulator in sequence. The input buffer of the matrix multiplication unit divides the input matrix into a combination of multiple small matrices according to the size of the systolic array unit. The small matrices are processed by the systolic array to obtain partial results of matrix multiplication. The partial results are stored in the intermediate buffer, which is connected to the accumulator. Whenever the systolic array obtains a new partial result, the accumulator adds the new result to the result currently stored in the intermediate buffer and stores it again in the intermediate buffer. After the matrix multiplication unit completes the multiplication of each row of small matrices, the intermediate buffer passes the result to the data buffer. After all rows of small matrices have been computed, the data buffer combines the results of each row to obtain the final matrix multiplication result.

[0010] The nonlinear processing unit includes an activation function unit, a layer normalization module, and a softmax module. The activation function unit is calculated using a polynomial approximation method. The layer normalization module is used to perform layer normalization calculations on the results of the multi-head attention mechanism. It first calculates the mean, variance, and the square root of the variance to obtain the standard deviation. Then, it normalizes the input matrix based on the mean and standard deviation, multiplies it by the weights of the layer normalization module, and adds the bias of the layer normalization module. The softmax module is calculated using a polynomial fitting method.

[0011] The controller is used to adjust the cycle time of the intermediate buffer and accumulator in the matrix multiplication unit according to the size of the input matrix.

[0012] In this invention, a weight cache and a bias cache are deployed on the FPGA. The weight cache and the bias cache are used to store the weights and biases of the model, respectively. The weight cache and the bias cache are connected to the matrix processing unit.

[0013] In this invention, a data cache is deployed on the FPGA, and the input and output data of the computing unit are stored in the data cache.

[0014] In this invention, an LN cache is deployed on the FPGA for the weights and biases of the storage layer normalization module.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0016] This invention, based on quantization, pruning, and hardware optimization, designs an FPGA-based Transformer hardware accelerator. This accelerator reduces the storage space required for a single parameter from 32 bits to 8 bits, lowering memory consumption. Simultaneously, it removes structures from the model that have little impact on accuracy, improving computational efficiency and reducing hardware consumption. Furthermore, it employs parallel computing strategies such as unrolling and pipelines to fully utilize FPGA hardware resources, reducing hardware idle time and maximizing computational efficiency and throughput.

[0017] This invention can be used in the field of language processing at the edge, such as mobile devices and edge devices for acquiring and processing text. Attached Figure Description

[0018] Figure 1 This describes the calculation process of the Transformer.

[0019] Figure 2 It is a comparison between the true value and the fitted value of the exponential function.

[0020] Figure 3 It is the system architecture of the Transformer accelerator. Detailed Implementation

[0021] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] This invention uses model compression techniques such as quantization and pruning to obtain an optimized Transformer model, then writes design code, and uses the HLS design strategy: pipeline, unroll to optimize the hardware architecture, finally obtaining a computing hardware architecture for implementing the Transformer model.

[0023] The specific calculation process of Transformer is as follows: Figure 1 As shown. The main computational functions required by the Transformer are the multi-head attention mechanism and the feedforward neural network, thus requiring the design of sub-modules including matrix multiplication, addition, softmax, and layer normalization. Due to the large size of the Transformer model, model compression methods are needed to reduce computational overhead. This invention mainly uses two model compression methods: quantization and pruning. The quantization process employs a linear symmetric quantization scheme. The weight parameters are converted from 32-bit floating-point numbers to 8-bit integers through a linear transformation, and the bias parameters are converted from 32-bit floating-point numbers to 16-bit integers. This linear transformation only includes scaling. In the specific calculation process, the 8-bit weights are multiplied by the 8-bit input to obtain a 16-bit activation value. The activation value is added to the 16-bit bias, and the result is multiplied by the total quantization coefficient S.

[0024]

[0025] Where S w S x S a These represent the quantization coefficients of the weights, inputs, and activation values, respectively. The pruning process employs an interval pruning strategy, removing one layer at a time to achieve a pruning rate of 50%. Before pruning, the model is trained through adaptive pruning to improve its robustness to pruning.

[0026] The final designed hardware architecture is as follows Figure 3 As shown, the hardware includes a data preprocessing module, a computing unit, a controller, a weight cache, a bias cache, a data cache, and an LN cache. The data preprocessing module is deployed on the CPU, while the computing unit, controller, weight cache, bias cache, data cache, and LN cache are deployed on the FPGA. The data preprocessing module preprocesses the text to be processed, converting it into an integer sequence that is input into the FPGA. The data in the computing unit completes the calculation process through the scheduling of the controller, and finally obtains the output result. The computing unit includes a matrix processing unit and a nonlinear processing unit.

[0027] In handling nonlinear functions, this invention employs a polynomial approximation method. This approach allows for the simultaneous introduction of quantization operations, further optimizing the nonlinear calculation process. Specifically, the nonlinear operations in the model mainly include exponential and arithmetic square root operations. In the exponential operation, this invention uses a quadratic function to fit a portion of the exponential function, and then obtains the final calculation result through methods such as shifting. Figure 2 Comparing the actual and fitted values ​​of the exponent, we can see that the difference is very small. The square root calculation uses an iterative algorithm, which can efficiently calculate the square root of integers. Sufficient data experiments have shown that calculating the square root of an integer up to 16 digits requires at most five iterations. By performing a shift operation before calculating the square root, a highly accurate result can be obtained.

[0028] The following explains how to use this invention. Figure 3 The architecture shown is implemented Figure 1 The calculation process is shown. The controller controls... Figure 3 The hardware in, according to Figure 1 The computation process is as follows: First, the text data is input into the CPU. After passing through the data preprocessing module, the text is decomposed into combinations of words in the vocabulary. Each word in the vocabulary corresponds to an integer, so the text will be expressed as a sequence of integers, which is input into the FPGA and stored in the data buffer.

[0029] Figure 1The left side is called the encoder. The data is first input into the encoder and processed by word embedding. The weights related to word embedding are stored in the weight cache. The data cache accesses the weight cache to obtain the embedded representation of the text. The weights related to position encoding are also stored in the weight cache. The embedded representation and position encoding are input into the matrix addition unit, the two are added together and the result is stored in the data cache.

[0030] The above results are then used to calculate the multi-head attention mechanism. This part mainly includes matrix multiplication, matrix addition, and softmax operations combined in a certain order. The relevant weight data is stored in a weight cache, and the bias data is stored in a bias cache. The order of these operations is scheduled by the controller. In the matrix multiplication stage, the input cache of the matrix multiplication unit divides the input matrix into a combination of multiple smaller matrices according to the size of the systolic array unit. These smaller matrices pass through the systolic array to obtain partial results of the matrix multiplication. These partial results are stored in an intermediate cache connected to an accumulator. Whenever the systolic array obtains a new partial result, the accumulator adds the new result to the result currently stored in the intermediate cache and stores it back in the intermediate cache. After the matrix multiplication unit completes the multiplication of each row of smaller matrices, the intermediate cache passes the result to the data cache. After all rows of smaller matrices have been calculated, the data cache combines the results of each row to obtain the final matrix operation result. Matrix addition mainly refers to the addition of the multiplication result to the bias. The matrix addition unit accesses the data in the bias cache and the data cache, adds them together, and stores the result back in the data cache. When softmax is executed, the data cache passes the data to the softmax module, which calculates the result using a polynomial fitting method and stores the result in the data cache.

[0031] The results of the multi-head attention mechanism are then processed through layer normalization, which takes place within the layer normalization module. This module first calculates the mean, variance, and the square root of the variance to obtain the standard deviation. Then, it normalizes the input matrix based on the mean and standard deviation, multiplies it by the LN weights, adds the LN bias, and returns the result to the data cache. Both the LN weights and LN bias are stored in the LN cache.

[0032] Next, the feedforward neural network performs computations. In the feedforward neural network, matrix multiplication and addition are performed first, similar to the matrix multiplication and addition operations in the multi-head attention mechanism. Then, the activation function is calculated in the activation function unit, which also uses a polynomial approximation method. The result is returned to the data buffer. Following the feedforward neural network is layer normalization, which is calculated similarly to the layer normalization after the multi-head attention mechanism. The computational results of these four layers—multi-head attention mechanism, layer normalization, feedforward neural network, and layer normalization—are called the encoder output and are stored in the data buffer.

[0033] Figure 1 The right side of the encoder is called the decoder. The encoder's output is not directly used as the decoder's input, but rather fed into the decoder's second-layer multi-head attention mechanism. The decoder's output is a probability distribution representing the probability of each word in the vocabulary. The word with the highest probability is used as the output of the current loop. The outputs of all previous loops are concatenated to form the decoder's input. The decoder loops in this manner until an end-of-loop marker is displayed. The masked multi-head attention mechanism in the decoder adds a masking matrix, which is automatically generated in the data buffer based on the loop cycle. Otherwise, the computational flow of the decoder's multi-head attention mechanism is exactly the same as that in the encoder. After the masked multi-head attention mechanism, layer normalization, multi-head attention mechanism, layer normalization, feedforward neural network, and layer normalization, the result is input into the linear layer. The linear layer performs matrix multiplication and matrix addition calculations, and the result is input into the softmax unit. This step is performed in the softmax unit, and the result is stored in the data buffer.

Claims

1. An FPGA-based Transform hardware accelerator, characterized in that, It comprises a data preprocessing module, a calculation unit and a controller; the data preprocessing module is deployed on a CPU, the controller and the calculation unit are deployed on an FPGA, the data preprocessing module pre-processes the text to be processed, converts it into an integer sequence and inputs it into the FPGA, the data is scheduled by the controller in the calculation unit to complete the calculation process, and finally the output result is obtained; wherein: The calculation unit comprises a matrix processing unit and a nonlinear processing unit. The matrix processing unit comprises a matrix multiplication unit and a matrix addition unit, the matrix multiplication unit is used to calculate the matrix multiplication in the multi-head attention mechanism and the feedforward neural network, and the matrix addition unit performs the residual connection before the layer normalization; the matrix multiplication unit adopts a systolic array structure, in the matrix multiplication unit, the input buffer is connected with the systolic array, the intermediate buffer and the accumulator in sequence; the input buffer of the matrix multiplication unit divides the input matrix into a combination of multiple small matrices according to the size of the systolic array, the small matrices obtain partial results of the matrix multiplication through the systolic array, the partial results are stored in the intermediate buffer, the intermediate buffer is connected with the accumulator, whenever the systolic array obtains new partial results, the accumulator adds the new partial results to the results currently stored in the intermediate buffer and re-stores them in the intermediate buffer, when the matrix multiplication unit completes the multiplication of each row of small matrices, the intermediate buffer transmits the results to the data buffer, and after the calculation of all rows of small matrices is completed, the data buffer combines the results of each row to obtain the final matrix multiplication result; The nonlinear processing unit comprises an activation function unit, a layer normalization module and a Softmax module; the activation function unit uses a polynomial approximation method to calculate; the layer normalization module is used to calculate the layer normalization of the result of the multi-head attention mechanism, which firstly calculates the mean, the variance and the square root of the variance to obtain the standard deviation, then normalizes the input matrix according to the mean and the standard deviation, multiplies the input matrix by the weight of the layer normalization module and adds the bias of the layer normalization module; the Softmax module uses a polynomial approximation method to calculate; The controller is used to adjust the cycle period of the intermediate buffer and the accumulator in the matrix multiplication unit according to the size of the input matrix.

2. The Transformer hardware accelerator of claim 1, wherein, The FPGA is deployed with a weight buffer and a bias buffer, the weight buffer and the bias buffer are used to store the weights and biases of the model respectively, and the weight buffer and the bias buffer are connected with the matrix processing unit respectively.

3. The Transformer hardware accelerator of claim 1, wherein, The FPGA is deployed with a data buffer, and the input and output data of the calculation unit are stored in the data buffer.

4. The Transformer hardware accelerator of claim 1, wherein, The FPGA is deployed with an LN buffer, which is used to store the weights and biases of the layer normalization module.

Images