FPGA-based reconfigurable systolic array tensor IP core

Abstract

The invention discloses a reconfigurable systolic array tensor IP core based on an FPGA. The architecture comprises a core controller, a shift distributor, a data processing array and a four-stage data cache. The tensor core can adapt to general tensor multiplication and addition operations in Prefill and Decode stages of the generative neural network at the same time through reconfigurable design. The core controller receives the tensor multiplication instruction and the configuration information and decouples the tensor multiplication instruction and the configuration information according to a data stream to generate a plurality of block tensor multiplication instructions. The shift distributor decodes a block tensor multiplication instruction, decoded mode information controls data communication between a first-level cache and a second-level cache (Distrided RAM and Bram), a control signal of a data processing array is generated, and the control signal enters the array together with data to flow. And the data processing array is connected with different secondary BRAM caches according to the received configuration information and is used for processing data in a bubble-free pulsating flow manner. The four-stage data cache realizes the multi-stage pinpong operation of the loading of the operation data of the tensor operation core, and realizes the multi-stage pipeline design of the data from the off-chip HBM to the on-chip URAM, from the URAM to the BRAM and from the BRAM to the operation unit. Aiming at a single-user edge end generation type network reasoning task, a core controller generates a data processing unit mode configuration signal in a fine-grained (tensor element level) manner, and the data processing unit mode configuration signal is transmitted by a shift distributor and flows in parallel with operation data, so that the mode configuration efficiency and the hardware resource utilization rate are greatly improved, the problem of high fan-out caused by broadcast control is reduced, and the user experience is improved. A control flow is designed by means of the structural characteristics of the systolic array, and a complex control algorithm is realized with low resource occupation.

Description

Technical Field

[0001] The invention relates to the technical fields of digital integrated circuits, electronic information and artificial intelligence, and in particular to a reconfigurable systolic array tensor IP core based on FPGA. Background Art

[0002] Since the advent of Chat-gpt, generative large language models have entered the public eye. The demand for them in life, work, scientific research, and military fields has increased explosively. As the R&D investment in large language models gradually increases and the model training hardware is upgraded, more excellent basic models have gradually emerged in academia and industry, and they are becoming more and more perfect in applications such as machine translation, text summarization, dialogue systems, and question-answering systems. In the field of generative large language models, the basic model is generally composed of a stack of Transformer Decoder structures or their deformations. The acceleration effect ultimately achieved by deploying them on a single-user device at the edge is limited by the large difference in activation size in different stages of the Decoder structure. The activation operations of the two main stages in the generative language model reasoning process: Prefill and Decoding are two-dimensional tensor multiplication and addition and one-dimensional tensor multiplication and addition respectively. It is difficult to use the unified tensor operation array of a single-user device at the edge to be compatible with the efficient deployment of the two tensor operations, which poses a great challenge to the reasoning process of the model. Therefore, we consider reconfiguring the pulsating tensor operation array with high data reuse, while supporting tensor operations of different scales and dimensions in the two stages of the reasoning process, achieving higher hardware utilization at the same array scale, and thus improving the reasoning speed of the unified generative model. Summary of the invention

[0003] The purpose of the present invention is to address the problem that in the existing neural network hardware acceleration technology, the use of a unified tensor operation array causes the generative neural network to be incompatible in the two stages of Prefill and Decoding, resulting in poor algorithm deployment parallelism and low hardware utilization. A reconfigurable systolic array tensor IP core based on FPGA is proposed. This IP introduces array splitting and reconstruction, mode configuration and activation broadcast mechanism on the basis of the original two-dimensional systolic array, so that the two stages in the generative model reasoning process in the edge single-user scenario are compatible, thereby achieving higher hardware utilization.

[0004] In order to achieve the above technical objectives, the technical solution implemented by the present invention is as follows:

[0005] A reconfigurable systolic array tensor IP core based on FPGA, characterized by comprising: a core controller, a shift distributor, a data processing array and a four-level data cache.

[0006] The core controller is used to receive tensor multiplication and addition instructions and decouple them according to data flow to generate multiple block tensor multiplication and addition instructions.

[0007] The shift distributor is used to decode the block tensor multiplication instruction. The decoded mode information controls the data communication between the first and second caches (Distributed RAM and Bram), generates the control signal of the data processing array, and enters the array flow together with the data.

[0008] The data processing array is used to connect different secondary BRAM caches according to configuration information, process data in a bubble-free pulsating stream, and return output results to the secondary cache in different formats.

[0009] The four-level data cache implements multi-level pingpong operations for loading data of the tensor operation core, and realizes a multi-level pipeline design from off-chip HBM to on-chip URAM, URAM to BRAM, and BRAM to the operation unit.

[0010] Optionally, the core controller includes a two-level instruction stack, the first-level cache IP peripheral tensor operation instructions, and the second-level cache tensor decoupling each operation instruction, the decoupled instructions include the second-level instruction tensor operation dimension information, the input and output addresses of the second-level cache BRAM, the array call effective dimension, the output transposition enable, the P / D mode configuration, and the start and end flag information of the first and second-level instructions.

[0011] Optionally, the shift distributor performs periodic shifting of control signals and start and end flags, and the shifting mode resonates in parallel with the pulsation array input mode, reducing the complexity of the generation of control signals of the core controller. The shift distributor performs different connection configurations of the secondary cache BRAM and the primary cache Distributed RAM of the computing unit in different modes (Prefill or Decoding), controls the interaction between the left column and the top row of the array and the secondary cache belonging to the row and column in the Prefill stage; controls the broadcast interaction between the reconfigurable primary cache of the computing unit of the array and its corresponding secondary cache in the Decoding stage. Synchronize control signals and data to a common streaming pulsation mode and transmit them in parallel to the computing array.

[0012] Optionally, the data processing array performs tensor multiplication, bias accumulation, and residual accumulation in a bidirectional pulsating manner, and is equipped with a post-multiplication-addition quantization module to quantize intermediate results. It supports horizontal and vertical flow of quantized accumulation results, and matrix transposition is performed without cache reshape. The output direction of the multiplication-accumulation result is set to the left and top to reduce the output startup delay of the operation result.

[0013] Optionally, the four-level data cache caches four dimensions of data from bottom to top respectively, the fourth-level HBM caches the entire model parameters, the third-level URAM caches single-layer model parameters and vocabulary, logit weights, the second-level BRAM caches operator weights and fusion operator weights, and the first-level Distributed RAM temporarily caches a set of activations, weights and intermediate results.

[0014] A bubble-free tensor operation data flow based on FPGA is characterized in that the complete tensor operation instruction is decoupled through the block splitting result of the complete tensor operation, and the block tensor operation instruction is generated. The corresponding control signal generated by the split instruction synchronizes the data pulsation form, is transmitted to the reconfigurable systolic array in a streaming pulsation manner, and flows in each operation unit. The control signal adopts a two-level flow mode, the first level flows from the core controller to the shift distributor, and the second level flows from the shift distributor to the operation unit of the corresponding row or column. The control signal in the shift distributor controls the secondary cache corresponding to the array row or column, and activates and weights the unicast and broadcast combination input respectively at different stages of model reasoning. The control signal along with the data flow determines the calculation behavior of the operation unit, the flow path and time of the input and output data.

[0015] The technical solution adopted by the present invention has the following advantages and beneficial effects:

[0016] Through the parallel flow of systolic array reconstruction and control flow and data flow, the unified and efficient deployment of the two-stage (Prefill, Decoding) tensor operation of Transformer Decoder on the edge single-user device is realized, which improves the hardware utilization and reduces the inference delay of the generative model. Through the two-level control signal bidirectional flow, the logic complexity of the core controller control signal generation is reduced, the fan-out scale of the controller control signal is reduced, and the data source and operation behavior of each operation unit in different modes are accurately controlled. By activating broadcast in the Decoding stage, the number of data loading and memory access in the secondary cache is reduced. Through the tensor instruction decoupling planning, each group of decoupled block tensor instructions gives priority to obtaining the complete row of the output tensor, which is conducive to the fusion and pipeline execution of the tensor core and other row vector operations in the generative language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0018] Figure 2 This is a schematic diagram of the core controller structure;

[0019] Figure 3 It is a schematic diagram of the shift distributor structure;

[0020] Figure 4It is a schematic diagram of the structure of the computing unit; DETAILED DESCRIPTION

[0021] like Figure 1 The overall structure shown is schematically shown. This example involves a reconfigurable systolic array tensor IP core based on FPGA, and its architecture includes: a core controller, a shift distributor, a data processing array and a four-level data cache.

[0022] The core controller is as follows Figure 2 , used to receive tensor multiplication and addition instructions and decouple them according to the data stream, generate multiple block tensor multiplication and addition instructions, and generate the state control signal of the corresponding instruction at the same time, and transmit it to the horizontal and vertical shift distributors in parallel with the instruction. Specifically, the core controller architecture is divided into an instruction stream and a control stream. The instruction stream decouples the IP call instructions cached in the first-level instruction cache stack, and splits the large-scale matrix multiplication and addition into multiple block matrix multiplication and addition according to the data stream form required by the output matrix. The second-level instruction information includes the second-level instruction tensor operation dimension information, the input and output addresses of the second-level cache BRAM, the array call effective dimension, the output transposition enable, and the P / D mode configuration. The decoupled second-level instructions are cached in the second-level instruction stack, and the decoupled multiple instructions are decoded and transmitted to the corresponding horizontal and vertical shift distributors. While the instruction stream is executed, the control stream generates the instruction status flags corresponding to the first-level instructions and the decoupled second-level instructions in the instruction state controller according to the instruction valid flag bit, and controls the emission and submission of the two-level instructions.

[0023] The shift allocator is as follows Figure 3 , use the decoded block tensor multiplication instruction to generate a tensor operation control signal. The various mode information in the control signal controls the data communication between the first and second level caches (Distributed RAM and BRAM), generates a control signal for the data processing array, and enters the array pulsating flow together with the data. Specifically, the decoded control signal is sent from the cache connected to the upper-level core controller to the control signal generation module, which generates: tensor end bit flag, operation flag, transposition flag, inference stage flag, array start scale control signal. The decoded control signal is also transmitted to the second level cache by the second level cache interface, passing the input data address and data length information of the decoupled instruction. The output data of the second level cache and the generated control signal are packaged and fused in the data control synchronizer, and sent to the data processing array for pulsating flow.

[0024] Each computing unit in the data processing array is as follows: Figure 4, the data processing array connects different secondary BRAM caches according to the configuration information, and processes data in a bubble-free pulsating streaming manner. Specifically, the data processing array includes a plurality of two-dimensional operation units, and the number of the operation units is configurable. Each operation unit includes two bidirectional buses: a horizontal inflow and outflow bus and a vertical inflow and outflow bus, which support data pulsation between different units in the array. The unit includes four horizontal and vertical caches in the upper, lower, left and right directions as the intermediate cache for bus transmission. Each operation unit receives the flag signal assigned by its shift distributor: stage flag, bias / residual flag, start flag, last bit flag, transposition flag, thereby determining the corresponding operation form and the corresponding input and output flow direction. Each unit performs tensor multiplication, bias accumulation, and residual accumulation in a bidirectional pulsating manner. It is equipped with a multiplication and addition post-quantization module, supports output horizontal and vertical flow directions, and performs matrix transposition without cache reshape. The output direction of the multiplication and accumulation result is set to the left and top, which reduces the output startup delay of the operation result.

[0025] The four-level data cache implements multi-level pingpong operations for loading data of tensor core operations, and the hardware supports multi-level pipeline design of data from off-chip HBM to on-chip URAM, URAM to BRAM, and BRAM to operation unit. Specifically, the four-level data cache caches data of different dimensions respectively, the fourth-level HBM caches the entire model parameters, the third-level URAM caches single-layer model parameters and vocabulary, logit weights, the second-level BRAM caches operator weights and fusion operator weights, and the first-level Distributed RAM temporarily caches a set of activations, weights and intermediate results. Through the four-level data cache, a pipeline design of data loading, storage and data operation of different granularities is performed to realize a unified parameter cache transmission mechanism.

Claims

1. A reconfigurable systolic array tensor IP core based on FPGA, characterized in that: include: Core controller, shift allocator, data processing array and four-level data cache. The core controller is used to receive tensor multiplication and addition instructions and decouple them according to data flow to generate multiple block tensor multiplication and addition instructions. The shift distributor is used to decode the block tensor multiplication instruction. The decoded mode information controls the data communication between the first and second caches (Distributed RAM and Bram), generates the control signal of the data processing array, and enters the array flow together with the data. The data processing array is used to connect different secondary BRAM caches according to configuration information, process data in a bubble-free pulsating stream, and return output results to the secondary cache in different formats. The four-level data cache implements multi-level pingpong operations for loading data of the tensor operation core, and realizes a multi-level pipeline design from off-chip HBM to on-chip URAM, URAM to BRAM, and BRAM to the operation unit.

2. The FPGA-based reconfigurable systolic array tensor IP core according to claim 1, characterized in that: The core controller includes a two-level instruction stack, a first-level cache IP peripheral tensor operation instruction, and a second-level cache tensor decoupling each operation instruction, the decoupled instruction includes the second-level instruction tensor operation dimension information, the input and output address of the second-level cache BRAM, the array call effective dimension, the output transposition enable, the P / D mode configuration, and the start and end flag information of the first and second-level instructions.

3. The FPGA-based reconfigurable systolic array tensor IP core according to claim 1, characterized in that: The shift distributor performs periodic shifting of control signals and start and end flags, and the shifting mode resonates in parallel with the pulsation array input mode, reducing the complexity of the generation of control signals of the core controller. The shift distributor performs different connection configurations of the secondary cache BRAM and the primary cache Distributed RAM of the computing unit in different modes (Prefill or Decoding), controls the interaction between the left column and the top row of the array and the secondary cache belonging to the row and column in the Prefill stage; controls the broadcast interaction between the reconfigurable primary cache of the computing unit and its corresponding secondary cache in the Decoding stage. Synchronize control signals and data to a common streaming pulsation mode and transmit them to the computing array in parallel.

4. The FPGA-based reconfigurable systolic array tensor IP core according to claim 1, characterized in that: The data processing array performs tensor multiplication, bias accumulation, and residual accumulation in a bidirectional pulsating manner. It is equipped with a post-multiplication and addition quantization module to quantize intermediate results. It supports horizontal and vertical flow of quantized accumulation results, and performs matrix transposition without cache reshape. The output direction of the multiplication and accumulation results is set to the left and top, reducing the output startup delay of the calculation results.

5. The FPGA-based reconfigurable systolic array tensor IP core according to claim 1, characterized in that: The four-level data cache caches four dimensions of data from bottom to top. The fourth-level HBM caches the entire model parameters, the third-level URAM caches single-layer model parameters and vocabulary, logit weights, the second-level BRAM caches operator weights and fusion operator weights, and the first-level Distributed RAM temporarily caches a set of activations, weights and intermediate results.

6. A bubble-free tensor computing data flow based on FPGA, characterized in that: Through the block splitting results of the complete tensor operation, the complete tensor operation instructions are guided to decouple and generate block tensor operation instructions. The corresponding control signals generated by the split instructions synchronize the data pulsation form, are streamed and pulsated to the reconfigurable systolic array, and flow in each operation unit. The control signal adopts a two-level flow method, the first level flows from the core controller to the shift distributor, and the second level flows from the shift distributor to the operation unit of the corresponding row or column. The control signal in the shift distributor controls the secondary cache corresponding to the array row or column, and activates and weights the unicast and broadcast combination input at different stages of model reasoning. The control signal with the data flow determines the calculation behavior of the operation unit, the flow path and time of the input and output data.

Images