A BERT network accelerator based on FPGA bit serial systolic array

Abstract

This invention discloses a BERT network accelerator based on an FPGA-based bit-serial systolic array, comprising: an input data controller, a systolic array operation block, an intermediate data controller, an intermediate data buffer, an output encoder, and an output controller. This accelerator uses weighted data that has undergone bit-consistent quantization and compression for computation. The systolic array operation block employs bit-serial PE units, flexibly adapting to mixed-precision quantization and matrix multiplication calculations with low effective bit counts. The intermediate data controller is compatible with computations with and without BIAS and drives the intermediate data buffer. The output encoder performs bit-consistent quantization and compression on-chip on the data from the intermediate data buffer before sending it to the output controller. This accelerator fully utilizes the bit sparsity in the weighted data for bit-serial multiplication calculations and on-chip data compression, reducing logic and storage resource consumption; its compatibility with multiple computation modes improves the versatility of the BERT network FPGA accelerator and reduces computational costs.

Description

Technical Field

[0001] This invention relates to the fields of digital integrated circuits, electronic information and artificial intelligence, and in particular to a BERT network accelerator based on an FPGA bit serial systolic array. Background Technology

[0002] In recent years, with the development of Natural Language Processing (NLP), Transformer architectures based on pre-trained models, such as BERT (Bidirectional Encoder Representations from Transformers), have become the benchmark for various language understanding tasks. While these models excel in accuracy, their high computational complexity and large number of parameters make efficient inference on resource-constrained edge computing devices difficult, significantly limiting accelerator performance improvements. Because BERT network parameters contain a large number of invalid zero bits, traditional accelerator architectures often ignore the bit sparsity in the data, leading to storage bottlenecks and computational inefficiencies. Therefore, considering bit-consistent quantization and compression of parameters to remove invalid zero bits can reduce parameter storage pressure and computational costs while ensuring operational correctness. Summary of the Invention

[0003] The purpose of this invention is to address the problem of insufficient utilization of invalid data bits in existing neural networks, which leads to parameter storage bottlenecks and excessively high multiplication computation costs. This invention proposes a bit-serial systolic array BERT network accelerator based on FPGA. This accelerator uses data that has undergone bit consistency quantization and compression to complete bit-serial computation and on-chip encoding of parameters, enabling the accelerator to make full use of the bit sparsity of the data.

[0004] To achieve the above-mentioned technical objectives, the technical solution implemented by this invention is as follows:

[0005] A BERT network accelerator based on an FPGA bit serial systolic array is characterized by comprising: an input data controller, a systolic array arithmetic block, an intermediate data controller, an intermediate data buffer, an output encoder, and an output controller.

[0006] The input data controller is used to transfer data from the off-chip memory to the systolic array arithmetic block. It consists of a BRAM and a control section. The BRAM includes an input data cache, a weight data cache, and a BIAS cache.

[0007] The systolic array operation block includes m×n bit serial PE units, m×n accumulators and m multiplexers (MUX), where m is the number of rows and n is the number of columns. Each bit serial PE unit and accumulator is combined into a tile. Each bit serial PE unit consists of a shift register and an adder. Each accumulator consists of an adder and a bit truncation unit.

[0008] The intermediate data controller, consisting of a Moore state machine and an adder, is used to control the reading and writing of intermediate data cache data and the addition of BIAS data with the calculation result.

[0009] The intermediate data cache is a BRAM, which is used to store BIAS data and the data after adding the calculation result to BIAS. The number of BRAMs is the same as the number of rows of the systolic array operation block.

[0010] The output encoder is used to perform bit consistency quantization on the calculated data and compress and encode it into a data format that can be processed by the pulsating array operation block.

[0011] The output controller is used to write the result data to off-chip memory.

[0012] Optionally, the input controller can index the data storage units according to the matrix block information.

[0013] Optionally, in the systolic array operation block, the accumulator can be activated after the data computation is completed and truncated, or it can be directly output. When calculating the multi-head self-attention in the BERT network, the accumulator directly outputs the result to the multiplexer; when calculating the feedforward layer in the BERT network, the accumulator outputs the activated result to the multiplexer.

[0014] Optionally, the intermediate data controller can be configured to directly write the calculated data to the intermediate data cache, or to add the calculated data to the BIAS and then write it to the intermediate data cache. This is used when calculating Q×K. T and Matrix P×W G During the calculation of the systolic array, the intermediate data controller does not write to the BIAS, but directly writes the calculation result of the systolic array to the intermediate data cache. When performing other calculation tasks, the intermediate data controller writes the BIAS to the intermediate data cache during the calculation of the systolic array operation block. After the calculation of the systolic array operation block is completed, the BIAS is read out, added to the calculation result, and then written back to the intermediate data cache in sequence.

[0015] Optionally, the output encoder has on-chip compression and pass-through characteristics. In calculating X×W in multi-head self-attention... K and X×W VWhen the output encoder performs bit-consistent quantization and compression on the received data, it sends the data to the output controller. When performing other computational tasks, the output encoder does not work and directly sends the results to the output controller.

[0016] A bit-consistent quantization and compression technique for FPGA hardware is characterized by non-uniform quantization of BERT network parameters, ensuring that each parameter has the same number of valid bits, particularly when each parameter has only one valid bit. Weight information is then reconstructed based on the position (offset) of the valid bits. The compressed parameters do not require decompression and can be directly computed in the bit-serialized systolic array operation block, reducing resource overhead.

[0017] The advantages and beneficial effects of the technical solution adopted in this invention are as follows:

[0018] This system extracts the position information of valid bits from the network weight parameters, compresses the data to reduce storage capacity, and allows the compressed data to be directly computed in the systolic array operation block, improving computational efficiency. The intermediate data controller can simultaneously control the operation of the systolic array operation block, the reading and writing of BIAS data, and the addition operation, thereby improving resource utilization efficiency and reducing output latency. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the present invention;

[0020] Figure 2 This is a schematic diagram of the structure of the pulsating array arithmetic block;

[0021] Figure 3 This is a schematic diagram of the structure of a bit serial PE unit; Detailed Implementation

[0022] like Figure 1 The structure shown in this embodiment relates to a BERT network accelerator based on an FPGA bit serial systolic array, including: an input data controller, a systolic array arithmetic block, an intermediate data controller, an intermediate data buffer, an output encoder, and an output controller.

[0023] The input data controller is used to transfer data from off-chip memory to the systolic array computation block and cache input, weight, and BIAS data within the controller. The input data controller consists of a control section and BRAM storage resources. The reading order is as follows: input feature map from width to height and then to the number of heads for multi-head self-attention; weights from width to height and then to the number of heads for multi-head self-attention; BIAS from height to the number of heads for multi-head self-attention.

[0024] The structure of the pulse array operation block is as follows: Figure 2As shown, it includes m×n bit serial PE units, m×n accumulators, and m multiplexers, where m is the number of rows and n is the number of columns. Each bit serial PE unit and accumulator is combined into a tile. Each bit serial PE unit consists of a shift register and an adder, and each accumulator consists of an adder and a bit truncation unit. Each multiplexer receives the output of one row of accumulators. Its internal counter increments with the clock and is compared with the length of the external input sequence. The counter data is input to the MUX to sequentially select a specific accumulator in the corresponding row and output the calculation result.

[0025] The structure of the bit-serial PE unit is as follows: Figure 3 As shown, the number of valid bits can be configured to be 2 or 1. Regardless of the configuration, the shifter will always have 2 bits. When the number of valid bits is configured to be 2, as shown... Figure 3 (a) A0 is 16-bit input data, and W0 is 9-bit weight data, where the highest bit is the sign bit and the rest are the position information of the valid bits. The position information of the two valid bits is fed into two shifters for calculation; when the number of valid bits is configured to be 1, as shown in the figure... Figure 3 (b) A0' is 16-bit input data, and W0' is 5-bit weight data, where the highest bit is the sign bit, and the rest are the position information of the valid bits. Since the value is too small when only one valid bit is retained in the weight parameter, the data needs to be multiplied by 1.5, that is, the position information of the only valid bit needs to be input into shifter 1 for calculation. At the same time, the position information is subtracted by 1 and then input into shifter 2 for calculation. By configuring the number of valid bits, the bit serial PE unit can support mixed precision quantization parameters.

[0026] The intermediate data controller is composed of a Moore state mechanism and can be configured to directly write calculated data into the intermediate data cache, or add calculated data to BIAS and then write it into the intermediate data cache, or read data from the intermediate data cache by an external signal. In calculating Q×K... T and Matrix P×W G During the initial computation, the intermediate data controller does not write to the BIAS but directly writes the result to the intermediate data cache. When performing other computational tasks, the intermediate data controller writes the BIAS to the intermediate data cache during the computation of the systolic array operation block. After the systolic array operation block finishes computation, the BIAS is read out, added to the computation result, and then written back to the intermediate data cache in sequence. When writing the computation result to the intermediate data cache, the intermediate data controller calculates the BRAM address information based on the matrix block location information.

[0027] The intermediate data cache is composed of BRAMs, the number of BRAMs corresponds to the number of rows of the systolic array operation block, each row of the systolic array operation block corresponds to one BRAM, the bit width of the BRAM is the same as the bit width of the input data, and the depth is the length of the result matrix.

[0028] The output encoder can be configured to reconstruct data with 2 or 1 valid bits, extract the position information of the valid bits, and combine them with the sign bit to form compressed data. The compressed data possesses the same characteristics as the weights and can be re-entered into the pulsation array operation block for matrix multiplication with the input data. Simultaneously, the output encoder has a pass-through characteristic, allowing data to be transmitted only when calculating the 0×W values ​​in the multi-head self-attention. K and X×W V When the data is received, the output encoder performs bit-consistent quantization and compression, and sends it to the output controller. When calculating other matrix multiplications, the output encoder does not work and directly sends the result to the output controller.

[0029] The output controller sends the data received from the output encoder to the off-chip memory unit via an external bus, thus no longer occupying valuable on-chip storage resources.

Claims

1. A BERT network accelerator based on an FPGA bit-serial systolic array, characterized in that, include: Input data controller, pulse array arithmetic block, intermediate data controller, intermediate data buffer, output encoder and output controller; The input data controller is used to buffer the data stream from the external memory and transfer it to the systolic array operation block. It consists of the BRAM of the input data controller and a control part. The BRAM of the input data controller includes an input data cache, a weight data cache and a BIAS cache. The systolic array arithmetic block includes m×n bit serial PE units, m×n accumulators, and m multiplexers, where m is the number of rows and n is the number of columns. Each bit serial PE unit and accumulator is combined into a tile. Each bit serial PE unit consists of a shift register and an adder. Each accumulator consists of an adder and a bit truncation unit. Each multiplexer receives the output of one row of accumulators. An internal counter increments with the clock and is compared with the length of the external input sequence. The counter data is input into the multiplexer to select the specific accumulator in the corresponding row and output the calculation result. The intermediate data controller consists of a Moore state machine and an adder, and is used to control the reading and writing of intermediate data cache data and the addition of BIAS data and calculation results. The intermediate data cache is used to store the calculation results of the systolic array operation block and the calculation result after adding BIAS. The number of BRAMs in the intermediate data cache is the same as the number of rows of the systolic array operation block. The output encoder is used to perform bit consistency quantization on the calculated data and compress and encode it into a data format that can be processed by the pulsating array operation block. The output controller is used to write the result data to the off-chip memory; The intermediate data controller is configured to either write the computed data directly to the intermediate data cache, or add the computed data to the BIAS and then write it to the intermediate data cache. In calculating Q×K T And Matrix P×W G At this time, the intermediate data controller will not write to BIAS, but will directly write the calculation results of the pulsating array to the intermediate data cache. While performing other computing tasks, the intermediate data controller writes BIAS into the intermediate data cache during the computation of the pulsating array arithmetic block. After the systolic array operation block finishes its calculation, the BIAS is read out, added to the calculation result, and then written back to the intermediate data cache in sequence. The structure of the bit-serial PE unit can be configured with 2 or 1 valid bits. Regardless of the configuration, there are always 2 shifters. When the number of valid bits is configured to be 2, A0 is 16-bit input data and W0 is 9-bit weight data, where the highest bit is the sign bit and the rest are the position information of the valid bits. The position information of the two valid bits is fed into the two shifters for calculation. When the number of valid bits is configured to be 1, A0' is 16-bit input data and W0' is 5-bit weight data, where the highest bit is the sign bit and the rest are the position information of the valid bits. Since the value is too small when only one valid bit is retained in the weight parameter, the data needs to be multiplied by 1.5, that is, the unique position information of the valid bit is input to shifter 1 for calculation. At the same time, the position information is subtracted by 1 and then input to shifter 2 for calculation.

2. The BERT network accelerator based on FPGA bit-serial systolic array according to claim 1, characterized in that, The input data controller can index the data storage units based on the block information of the input / weight matrix.

3. The BERT network accelerator based on FPGA bit-serial systolic array according to claim 1, characterized in that, In the pulsating array operation block, the accumulator is activated or directly output after the data calculation is completed and truncated; when calculating the multi-head self-attention in the BERT network, the accumulator directly outputs the result to the multiplexer; when calculating the feedforward layer in the BERT network, the accumulator outputs the activated result to the multiplexer.

4. The BERT network accelerator based on FPGA bit-serial systolic array according to claim 1, characterized in that, The output encoder features on-chip compression and pass-through transmission; in calculating X×W in multi-head self-attention... K and X×W V When the output encoder performs bit-consistent quantization and compression on the received data, it sends the data to the output controller. When performing other computational tasks, the output encoder does not work and directly sends the results to the output controller.

5. A hardware-friendly bit-consistency quantization and compression method, the method being based on the BERT network accelerator based on an FPGA bit-serial systolic array as described in any one of claims 1-4, characterized in that, The BERT network parameters are non-uniformly quantized to ensure that each parameter has the same number of valid bits. When each parameter has only one valid bit, the weight information is reconstructed based on the position of the valid bit. The compressed parameters do not need to be decompressed and can be directly entered into the bit serial systolic array operation block for calculation, reducing resource consumption.

Images