Eso-LMs hardware acceleration system for dynamic sequence length block attention calculation based on FPGA

Abstract

The invention discloses an auto-regression and mask diffusion double-normal-form fusion language model hardware acceleration system based on an FPGA (Field Programmable Gate Array), and relates to the field of FPGA and machine learning. The invention provides a model core calculation process-oriented hardware acceleration architecture by aiming at a language model fusing double normal forms of an autoregression model and a mask diffusion model and adopting the characteristics of revising an attention mechanism and parallel generation of a KV cache. The method disclosed by the invention is implemented by taking an Eso-LMs (Esolic Language Models) model as an example. The system comprises a layer normalization and adaptive layer normalization modulation module, a QKV projection module, a rotation position coding application module, a KV cache management module, a multi-head attention calculation module, an output projection and residual connection module and a multi-layer perceptron module. By optimizing the calculation sequence and the data flow of each module in the model calculation process and adopting the pipeline parallel and resource reuse technology, the efficient reasoning acceleration of the language model fusing the autoregression and mask diffusion double normal forms on the FPGA platform is realized, and the model reasoning speed is obviously improved. According to the system, the advantage of FPGA customizable hardware acceleration is fully exerted, the utilization efficiency of hardware resources is improved, data pipeline blockage is eliminated, calculation delay and storage overhead are reduced by reconstructing the data flow direction and constructing a whole-process pipeline processing architecture, and the system is suitable for the deployment requirement of an edge calculation scene.

Description

Technical Field

[0001] This invention relates to the field of FPGA (Field Programmable Gate Array) hardware acceleration technology, specifically to a hardware acceleration system for Eso-LMs (Esoteric Language Models) based on dynamic sequence length block attention computation using FPGA, which is used to efficiently accelerate the inference of Eso-LMs models that integrate autoregressive and mask diffusion dual paradigms on the FPGA platform. Background Technology

[0002] With the rapid development of deep learning technology, large language models (LMs) have achieved remarkable results in tasks such as natural language processing and text generation. However, traditional large language models suffer from high computational complexity, large memory consumption, and long inference latency during the inference process, especially in edge computing scenarios where these limitations are even more pronounced.

[0003] Eso-LMs (Esoteric Language Models) is a novel language model that integrates autoregressive (AR) and masked diffusion (MDM) paradigms, with its core structure based on the DiffusionTransformer (DiT) architecture. Unlike traditional autoregressive or diffusion models, Eso-LMs features dynamic sequence length, variable window size, partial masking, hybrid generation modes, and dynamically growing key-value caches during inference. Specifically, the sequence length gradually increases from the initial window to the complete sequence, rather than being fixed; the window size dynamically adjusts according to the generation stage, supporting a hybrid generation mode of diffusion and autoregressive stages; in attention computation, historical tokens are not masked, and only tokens in the current window are subject to causal masking, unlike traditional attention mechanisms that use full-sequence masking or no masking at all; the model alternates between diffusion and autoregressive stages during inference, dynamically switching between the two modes within the same loop; each layer requires maintaining an independent key-value cache, with the cache sequence length dynamically increasing, necessitating an efficient cache management mechanism.

[0004] These characteristics make it difficult to implement Eso-LMs models efficiently on general-purpose GPUs or CPUs. Traditional neural network acceleration solutions are typically designed for fixed sequence lengths with static hardware resource allocation, which cannot adapt to the dynamic sequence length characteristics of Eso-LMs. When the sequence length grows dynamically, cache allocation and block partitioning strategies need to be dynamically adjusted, which existing solutions struggle to support. Eso-LMs models require multi-layered maintenance of independent key-value caches, with the cache sequence length growing dynamically. Traditional solutions typically use fixed window sizes or static allocation strategies, which cannot fully utilize the tiered storage resources of FPGAs (URAM / BRAM and DDR), resulting in low cache access efficiency. In terms of attention computation, the query vector sequence length of Eso-LMs models is relatively fixed (current window), while the value vector sequence length grows dynamically. Traditional solutions such as FlashAttention are mainly designed for GPU parallel processing of multiple tokens, using multiple streaming multiprocessors for parallel computation, with activation functions locally reset within blocks, which is unsuitable for the resource-constrained environment of FPGA single-token processing units. The Eso-LMs model applies causal masks only to the current window, and historical tokens are not restricted by masks. Traditional masking hardware typically implements full-sequence masks or no masks at all, which cannot support the selective application of partial masks, leading to wasted computational resources. During inference, the Eso-LMs model needs to dynamically switch between diffusion and autoregressive modes, requiring dynamic adjustments to window size and processing strategies. Traditional solutions typically use fixed generation modes and cannot support dynamic switching between mixed modes. Furthermore, the Eso-LMs model uses gated residual connections, requiring the storage of two skip connection values ​​(before MSA and before MLP). Traditional solutions typically implement simple additive residual connections, which cannot efficiently achieve hardware fusion of gated residual connections.

[0005] Therefore, there is an urgent need for a dedicated FPGA hardware acceleration system for the characteristics of Eso-LMs models, which can achieve efficient inference acceleration of Eso-LMs models on FPGA platforms through dynamic hardware architecture adjustment, hierarchical storage optimization, block computation optimization, multi-level pipeline depth optimization and efficient resource reuse. Summary of the Invention

[0006] The technical problem to be solved by this invention is: how to design an FPGA hardware acceleration system for dynamic sequence length optimization, taking into account the characteristics of Eso-LMs models that integrate autoregressive and masked diffusion dual paradigms during inference, such as dynamic growth of sequence length, variable window size, and mixed generation modes. Through dynamic hardware architecture adjustment, hierarchical storage optimization, block computation optimization, multi-level pipeline depth optimization, and efficient resource reuse, the system can achieve efficient inference acceleration of Eso-LMs models on the FPGA platform.

[0007] To address the aforementioned technical problems, this invention provides a hardware acceleration system for Eso-LMs based on FPGA with dynamic sequence length block attention computation. Taking advantage of the Diffusion Transformer architecture of the Eso-LMs model, the system divides the computation process of each Diffusion Transformer Block into several pipeline stages and outputs the data stream in a pipelined manner.

[0008] The hardware acceleration system includes the cascaded execution of multiple Diffusion Transformer Blocks. The calculation process of each Block includes: QKV projection module, rotation position encoding application module, KV cache management module, block attention calculation module, output projection and gated residual connection module, and multilayer perceptron module. The system adopts a single token processing unit design to process multiple tokens in the window sequentially, and supports dynamic window size and hybrid generation mode.

[0009] The QKV projection module is a first pipeline stage and a second pipeline stage, including a first-layer normalization and adaptive layer normalization modulation module and a QKV projection module. It is used to perform layer normalization processing and adaptive layer normalization modulation on the input feature vector, and then project the modulated feature vector into a query vector Q, a key vector K and a value vector V through a linear transformation.

[0010] The rotation position encoding application module is the third pipeline stage, used to apply rotation position encoding to the first half of the dimensions of the query vector Q and the key vector K, while the value vector V is not subject to rotation position encoding.

[0011] The KV cache management module is the fourth pipeline stage, used to merge the current key vector K and value vector V with the key vector and value vector of the historical cache, so as to realize the dynamic update of the KV cache.

[0012] The block attention calculation module consists of a fifth, a sixth, and a seventh pipeline. Through these three pipelines, the block attention calculation performs transpose multiplication of the query vector and the key vector, activation function normalization, and weighted vector aggregation operations, respectively.

[0013] The output projection and gated residual connection module is the eighth pipeline stage, which is used to transform the attention output through linear projection and perform residual connection with the input feature vector through a gating mechanism.

[0014] The multilayer perceptron module consists of a ninth, tenth, eleventh, and twelfth pipeline stage, including a second-layer normalization and adaptive layer normalization modulation module, a multilayer perceptron module, and a second-gated residual connection module. These modules are used to perform layer normalization processing and adaptive layer normalization modulation on the input feature vector. Through two pipeline stages, a first-layer linear transformation, GELU activation, and a second-layer linear transformation are performed. Finally, a gated mechanism is used to perform a residual connection with the input feature vector to obtain the output feature vector.

[0015] As a preferred embodiment, the KV cache management module adopts a hierarchical cache architecture of L1 (URAM / BRAM, low-latency access) and L2 (DDR, high-capacity storage). It dynamically adjusts the cache allocation strategy based on the current sequence length and window position, implementing predictive prefetching and incremental update mechanisms. The dynamically sequence-length-driven cache allocation control unit dynamically calculates the L1 cache capacity (storing the K / V of the most recent N tokens) and the L2 cache capacity (storing the K / V of the complete sequence) based on the current sequence length and window position (last_k_start, curr_k_end), and dynamically adjusts the allocation strategy according to the access pattern, rather than using a fixed cache size or static allocation. The incremental update calculation unit only calculates and stores the K / V of newly generated tokens at the hardware level, reusing historical caches to reduce computation and storage overhead. The predictive prefetch control unit predicts the next possible K / V position based on last_k_start, curr_k_end, and num_clean_and_mask, preloading it at the hardware level to hide DDR access latency.

[0016] As a preferred embodiment, the block-based attention calculation module uses a block-based computing hardware unit to divide the dynamically growing K sequence into fixed-size blocks. The GEMM unit calculates the transpose multiplication of the query vector and the key vector block by block. The calculation result for each block is stored in on-chip BRAM / URAM, avoiding the need to calculate the complete transpose multiplication matrix of the query vector and the key vector at once, thus reducing on-chip cache capacity requirements. The prefix cumulative activation function register uses dedicated registers to maintain cross-block activation function statistics (global maximum register max_global_reg and normalized denominator accumulation register sum_exp_reg). These registers are updated after processing each block, supporting cross-block activation function normalization and avoiding the need to store the complete attention score matrix in DDR. The historical block and current window block separation processing control unit includes a historical block processing path and a current window block processing path. When the hardware control logic detects a historical block (num_gen_before_last part), it skips the mask application unit and directly performs the transpose multiplication of the query vector and the key vector. The result can be cached in the on-chip BRAM. When the hardware control logic detects the current window block (num_clean_and_mask part), it enables the mask application unit, fully calculates and applies the causal mask, and avoids pipeline stalls caused by conditional branches through mask pre-computation and fast application units. The on-chip fusion computing pipeline implements a three-stage pipeline in the on-chip cache (BRAM / URAM), from the transpose multiplication of the query vector and the key vector to the activation function unit to the weighted vector aggregation unit. Data flows on-chip, reducing DDR access. After each block is processed, the intermediate result cache is released immediately, reusing BRAM resources.

[0017] As a preferred embodiment, the output projection and gated residual connection module integrates gated multiplication and residual addition into a single hardware unit to achieve the fused calculation of gate * new_value + skip_value. Data flows on-chip (gate value, new_value, and skip_value are read from on-chip BRAM / URAM, and the calculation result is directly written back), avoiding the intermediate result being written back to DDR, thus reducing memory access latency and bandwidth consumption.

[0018] As a preferred embodiment, the hardware acceleration system supports cascaded execution of multiple Diffusion Transformer Blocks. It achieves efficient resource reuse and pipelined parallelism through time-reused computation units, pipelined parameter loading, and inter-layer data flow optimization. Multiple blocks share the same set of computation units (GEMM unit, block attention computation unit, prefix cumulative activation function register, mask application unit, etc.), reducing hardware resource consumption through time reuse, and allowing multiple blocks to sequentially use the same set of hardware resources. When computing the current block, the hardware control unit preloads the parameters (weight matrix, adaptive layer normalization parameters, etc.) of the next block from DDR to the on-chip BRAM / URAM, hiding parameter loading latency and achieving pipelined parallelism between computation and parameter loading. Data transfer between blocks is achieved through a FIFO buffer, supporting pipelined execution. The output data of the current block is passed to the next block via FIFO, realizing pipelined cascading of multiple blocks.

[0019] As a preferred embodiment, the hardware acceleration system implements a multi-level pipeline architecture design, with sub-pipelines within each level, realizing a complete computational pipeline from feature vector input to feature output. The module-level pipeline fusion integrates multiple consecutive operations into the same pipeline stage, reducing the number of pipeline stages and intermediate data caching requirements. For example, a three-level fusion pipeline can be used for the transpose multiplication unit of the query vector and key vector, the activation function unit, and the weighted vector aggregation unit. Gated residual calculation, layer normalization, and activation functions can be integrated into the same pipeline. The pipeline depth balancing optimization allows the hardware control unit to dynamically adjust the pipeline depth based on the computational latency of each module, balancing the processing time of each pipeline stage, avoiding pipeline bottlenecks, and maximizing throughput. Data dependency processing and pipeline stall avoidance involve implementing a hardware data dependency detection unit. Data forwarding and register renaming techniques are used to avoid pipeline stalls caused by data dependencies. For example, pre-caching and forwarding of skip join values ​​avoids waiting for residual join calculations to complete. The dynamic pipeline configuration enables the hardware control unit to dynamically adjust the pipeline configuration based on the current sequence length, window size, and generation mode. For example, it can reduce the number of blocks for short sequences and increase the number of blocks for long sequences to optimize pipeline throughput.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. Optimization of dynamic sequence length block attention computation: Through hardware implementation of prefix cumulative activation function, separation of historical blocks and current window blocks, and on-chip fusion computation pipeline, efficient dynamic sequence length block attention computation is achieved, which significantly reduces on-chip cache capacity requirements and computation latency.

[0022] 2. Dynamic Sequence Length Optimization and Tiered Storage Management: Through tiered KV cache allocation and adaptive block control logic driven by dynamic sequence length, the cache allocation strategy and block strategy are dynamically adjusted according to the sequence length, which significantly reduces cache access latency and bandwidth consumption, and improves hardware resource utilization and inference efficiency.

[0023] 3. Multi-level pipeline architecture and efficient resource reuse: Through multi-level pipeline architecture design, module-level pipeline fusion and resource reuse technology, a complete computation pipeline from feature vector input to output is realized, which significantly improves system throughput and resource utilization and reduces hardware resource consumption. Attached Figure Description

[0024] Figure 1 This is an overall architecture diagram of an Eso-LMs hardware acceleration system based on FPGA for dynamic sequence length block attention computation, as an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of the structure of the QKV projection module according to an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the structure of the rotation position encoding application module in an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the structure of the KV cache management module in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the block attention calculation module according to an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the output projection and gated residual connection module according to an embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the structure of a multilayer perceptron module according to an embodiment of this application. Detailed Implementation

[0031] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0032] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0033] Example 1

[0034] Please see Figure 1 Embodiment 1 of this application provides an Eso-LMs hardware acceleration system for dynamic sequence length block attention computation based on FPGA, including: a QKV projection module, a rotation position encoding application module, a KV buffer management module, a block attention computation module, an output projection and gated residual connection module, and a multilayer perceptron module. The hardware acceleration system divides the Eso-LMs model algorithm computation process into several pipeline stages and outputs the data stream in a pipelined manner.

[0035] The QKV projection module is the first and second pipeline stages. It is used to perform layer normalization and adaptive layer normalization modulation on the input feature vector, and then project the modulated feature vector into a query vector Q, a key vector K, and a value vector V through a linear transformation.

[0036] The rotation position encoding application module is the third pipeline level. It is used to apply rotation position encoding to the first half of the dimensions of the query vector Q and the key vector K, while the value vector V is not subject to rotation position encoding.

[0037] The KV cache management module is the fourth pipeline stage, used to merge the current key vector K and value vector V with the key vector and value vector of the historical cache, so as to realize the dynamic update of the KV cache.

[0038] The block attention calculation module consists of a fifth, sixth, and seventh pipeline stage. Through these three pipeline stages, the block attention calculation performs transpose multiplication of the query vector and key vector, activation function normalization, and weighted V aggregation operations, respectively.

[0039] The output projection and gated residual connection module is the eighth pipeline stage. It is used to transform the attention output through linear projection and then perform residual connection with the input feature vector through a gating mechanism.

[0040] The multilayer perceptron module consists of nine, ten, eleven, and twelfth pipeline stages, used to perform layer normalization and adaptive layer normalization modulation on the input feature vector. It performs a first-layer linear transformation, GELU activation, and a second-layer linear transformation through two pipeline stages. Finally, it performs a residual connection with the input feature vector through a gating mechanism to obtain the output feature vector.

[0041] Please see Figure 1This is a block diagram of the overall architecture of an Eso-LMs hardware acceleration system based on FPGA for dynamic sequence length block attention computation in Embodiment 1. To improve acceleration performance, reduce computation latency, and increase the utilization of computing units, this application proposes a pipelined design to implement the Eso-LMs hardware acceleration system architecture. The computation process of each Diffusion Transformer Block is divided into twelve pipeline stages. The first and second pipeline stages are QKV projection modules, the third pipeline stage is a rotation position encoding application module, the fourth pipeline stage is a KV cache management module, the fifth, sixth, and seventh pipeline stages are block attention computation modules, the eighth pipeline stage is an output projection and gated residual connection module, and the ninth, tenth, eleventh, and twelfth pipeline stages are multilayer perceptron modules. This realizes the overall pipelined design, optimizes the data flow, reduces computation latency, improves the utilization of computing units, and reduces the consumption of storage resources.

[0042] The system employs a single-token processing unit design, sequentially processing multiple tokens within a window, and supports dynamic window size and hybrid generation modes. Multiple blocks share the same set of computing units, reducing hardware resource consumption through time multiplexing. While computing the current block, parameters for the next block are pre-loaded from DDR, achieving pipelined parallelism between computation and parameter loading. Data transfer between blocks is achieved through FIFO buffering, supporting pipelined cascading of multiple block layers.

[0043] In this embodiment 1, the process of applying this hardware acceleration system to accelerate inference tasks is as follows: The weight matrix and input feature vector are loaded from external memory into the on-chip memory of the FPGA. The QKV projection module performs layer normalization and adaptive layer normalization modulation on the input feature vector. Then, the modulated feature vector is projected into a query vector Q, a key vector K, and a value vector V through a linear transformation. The rotation position encoding application module applies rotation position encoding to the first half of the dimensions of the query vector Q and the key vector K. The KV cache management module merges the current key vector K and value vector V with the historical cached key vector and value vector. The block attention calculation module performs block attention calculation through a three-stage pipeline. The block attention computation performs transpose multiplication of the query vector and key vector, activation function normalization, and weighted V aggregation operations. The output projection and gated residual connection module transforms the attention output through linear projection and performs residual connection with the input feature vector through a gating mechanism. The multilayer perceptron module performs layer normalization and adaptive layer normalization modulation on the input feature vector. It performs the first layer linear transformation, GELU activation, and the second layer linear transformation through a two-stage pipeline. Finally, it performs residual connection with the input feature vector through a gating mechanism to obtain the output feature vector. The output feature vector is then used as the input of the next block and the iteration continues until the computation of all layers is completed.

[0044] Example 2

[0045] This embodiment 2 is based on embodiment 1, and further describes the QKV projection module therein. For example... Figure 2 The diagram shown is a structural schematic of the QKV projection module.

[0046] The first-layer normalization and adaptive layer normalization modulation module is the first pipeline stage, used to perform layer normalization processing on the input feature vector, and obtain the adaptive layer normalization modulation parameters by looking up a table according to the time conditions, and modulate the normalized feature vector. The module includes a layer normalization unit, an adaptive layer normalization modulation parameter lookup table storage unit, and a lookup table and multiply-accumulate fusion unit.

[0047] The key calculation formula for layer normalization is as follows:

[0048] in:

[0049] x is the input feature vector

[0050] mean is the average value of the input feature vector along the last dimension.

[0051] var is the variance of the input feature vector in the last dimension.

[0052] eps is a small constant used for numerical stability (usually taken as 1e-5).

[0053] x_norm is the normalized feature vector.

[0054] The adaptive layer normalized modulation parameter lookup table storage unit pre-calculates the modulation parameters (shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp) for all diffusion steps and stores them in BRAM / URAM. Since the time condition is fixed during inference (sigma=0), the modulation parameters can be pre-calculated, avoiding real-time computation overhead.

[0055] The table lookup and multiply-accumulate fusion unit integrates table lookup operations with modulation calculations within a single pipeline stage, reducing intermediate data access. The modulation calculation formula is as follows:

[0056] in:

[0057] x_norm is the normalized feature vector.

[0058] `scale` represents the adaptive layer normalization modulation parameters (scaling parameters).

[0059] `shift` represents the adaptive layer normalization modulation parameter (offset parameter).

[0060] x_modulated is the modulated feature vector.

[0061] The QKV projection module is the second pipeline stage, used to project the modulated feature vector into a query vector Q, a key vector K, and a value vector V through a linear transformation.

[0062] The formula for calculating QKV projection is as follows:

[0063] in:

[0064] x is the input feature vector (the modulated feature vector).

[0065] W_qkv is the QKV projection weight matrix.

[0066] W_qkv.T or W_qkv^{\top} is the transpose of the weight matrix.

[0067] `qkv_raw` is the original tensor after QKV projection, containing the concatenated query vector Q, key vector K, and value vector V. Then, a rearrangement operation separates the QKV tensor into the query vector Q, key vector K, and value vector V.

[0068] To optimize the input of the first-layer block, the module detects whether all positions of the input are the same (e.g., all positions in the early diffusion stage are MASK token embeddings). If they are the same, the QKV projection is calculated only once and then copied to all positions, saving computation.

[0069] Other technical details and implementation steps of this embodiment 2 are the same as those of embodiment 1, and will not be repeated here.

[0070] Example 3

[0071] This embodiment 3, based on embodiment 1, further explains the rotation position encoding application module. For example... Figure 3 The diagram shown is a structural schematic of the rotation position encoding application module.

[0072] The module includes a partial-dimensional rotation position encoding unit, a dedicated rotate_half unit, a cos / sin lookup table optimization unit, and a Q / K parallel processing unit. The partial-dimensional rotation position encoding unit only processes rotations in the first head_dim / 2 dimensions; the second head_dim / 2 dimensions are passed directly, reducing computation by approximately half. For each position pos and each attention head, the rotation calculation is as follows: a rotation matrix is ​​applied to the first head_dim / 2 dimensions, using cosine and sinine values; the second head_dim / 2 dimensions are passed directly without rotation.

[0073] The output is a rotated encoded Q / K vector with dimensions [batch_size, seq_len, n_heads, head_dim]. Each cycle outputs a data set with dimensions [1, n_heads, head_dim], and subsequent modules perform pipelined operations based on these dimensions.

[0074] The dedicated rotate_half unit hardware implements the data rearrangement logic from [x0,x1,x2,x3...] to [-x1,x0,-x3,x2...]. It uses simple data rearrangement logic instead of multiplication and executes it continuously in the pipeline, reducing pipeline pauses.

[0075] The sine and cosine value lookup table optimization unit performs table lookups based on position indices, supporting dynamic sequence lengths. Sine and cosine values ​​are stored in BRAM / URAM, and the lookup operation is integrated with rotation calculations in the pipeline, avoiding redundant calculations.

[0076] The Q / K parallel processing unit enables Q and K to apply rotational position encoding in parallel (because they use the same sine and cosine values), improving processing efficiency and supporting pipelined parallel execution.

[0077] Other technical details and implementation steps of this embodiment 3 are the same as those of embodiment 1, and will not be repeated here.

[0078] Example 4

[0079] This embodiment 4, based on embodiment 1, further explains the KV cache management module. For example... Figure 4 The diagram shown is a structural schematic of the KV cache management module.

[0080] The KV cache management module includes a dynamically sequence length-driven cache allocation control unit, an L1 / L2 hierarchical cache storage unit, a double-buffered KV cache unit, an incremental update calculation unit, a predictive prefetch control unit, and a cache line aligned access optimization unit.

[0081] The dynamic sequence length-driven cache allocation control unit dynamically adjusts the L1 / L2 cache allocation strategy based on the current sequence length and window position (last_k_start, curr_k_end), rather than using a fixed cache size or static allocation.

[0082] Cache capacity calculation: L1 cache capacity: L1_capacity = min(seq_len, L1_max_size) (where L1_max_size is the maximum capacity of the L1 cache), L2 cache capacity: L2_capacity = seq_len - L1_capacity. For the current window position (last_k_start, curr_k_end), the hardware control unit determines: if curr_k_end <= L1_capacity, then the key-value pairs of the current window are stored in the L1 cache; if curr_k_end > L1_capacity, then part of the key-value pairs of the current window are stored in the L1 cache and part are stored in the L2 cache.

[0083] The L1 / L2 hierarchical cache storage unit includes an L1 cache (URAM / BRAM, storing the key-value pairs of the most recent N tokens, low-latency access) and an L2 cache (DDR, storing the complete sequence of key-value pairs, high-capacity storage), optimized for FPGA resource characteristics. The L1 cache is used to store the most recently accessed key-value pairs, supporting low-latency access; the L2 cache is used to store the complete sequence of key-value pairs, supporting high-capacity storage.

[0084] The dual-buffered KV cache unit implements read-write separation through dual buffering, avoiding access conflicts and supporting pipelined parallelism. Specifically, it uses two independent cache buffers: one for reading historical K / V values ​​and the other for writing new K / V values, with read-write separation achieved through hardware control logic.

[0085] The incremental update calculation unit only calculates and stores the key-value pairs (K / V) of newly generated tokens at the hardware level, reusing historical caches to reduce computational and storage overhead. Specifically, for each newly generated token, the hardware control unit calculates the K / V value of the new token (using QKV projection and rotational position encoding), writes the K / V value to the KV cache (selecting L1 or L2 cache based on sequence length), and updates the cache index and sequence length counter. K / V values ​​of historical tokens are read directly from the cache without needing to be recalculated.

[0086] The predictive prefetch control unit uses last_k_start, curr_k_end, and num_clean_and_mask to predict the next K / V location that may be accessed, and loads it in advance at the hardware level, hiding DDR access latency.

[0087] The next step is to calculate the window position: next_last_k_start = curr_k_end, next_curr_k_end = curr_k_end + num_clean_and_mask. Then, the key-value pairs in the range [next_last_k_start, next_curr_k_end] are loaded from DDR into the L1 cache in advance.

[0088] The cache line alignment access optimization unit optimizes the DDR access mode by batch merging access requests from adjacent locations, thereby improving bandwidth utilization. Specifically, the hardware control unit merges multiple access requests from adjacent locations into a batch access, utilizing the cache line alignment feature of DDR to improve access efficiency.

[0089] The other technical details and implementation steps of this embodiment 4 are the same as those of embodiment 1, and will not be repeated here.

[0090] Example 5

[0091] This embodiment 5, based on embodiment 1, further explains the block attention calculation module. For example... Figure 5 The diagram shown is a schematic of the block attention calculation module.

[0092] The block attention calculation module includes a block calculation hardware unit, a prefix cumulative activation function register, a historical block and current window block separation processing control unit, an on-chip fusion calculation pipeline, adaptive block control logic, and a block-level KV cache access optimization unit.

[0093] The block-based computation hardware unit is the fifth pipeline stage. It divides the dynamically growing K sequence into fixed-size blocks and uses the GEMM unit to compute Q @ K_block^T block by block. The computation result of each block is stored in on-chip BRAM / URAM, avoiding the need to compute the entire Q @ K^T matrix at once and reducing on-chip cache capacity requirements. For the Q sequence length being num_clean_and_mask (current window, relatively fixed) and the K sequence length being num_gen_before_last + num_clean_and_mask (dynamically growing), the hardware control unit divides the K sequence into multiple fixed-size blocks and computes the attention score block by block.

[0094] The prefix cumulative activation function register is the sixth pipeline stage. It uses dedicated registers to maintain cross-block activation function statistics (global maximum register max_global_reg and normalized denominator accumulation register sum_exp_reg). These registers are updated after each block is processed, which supports cross-block activation function normalization and avoids the need to store the complete attention score matrix in DDR.

[0095] For each block_i (i = 0, 1, ..., num_blocks-1), the calculation process and formula are as follows:

[0096] Attention Score:

[0097] Update global maximum value:

[0098] Calculate the normalized denominator:

[0099] Accumulated normalized denominator:

[0100] Calculate the normalized attention weights:

[0101] in:

[0102] Q is the query vector

[0103] K_block_i is the key vector of the i-th block.

[0104] scores_block_i is the attention score matrix for the i-th block.

[0105] max_global_reg is the global maximum value register.

[0106] sum_exp_reg is the normalized denominator accumulation register.

[0107] exp_scores_block_i is the normalized denominator for the i-th block.

[0108] `attn_weights_block_i` represents the normalized attention weights for the i-th block.

[0109] block_i is the i-th block

[0110] The historical block and current window block separation processing control unit includes a historical block processing path and a current window block processing path. When the hardware control logic detects a historical block (num_gen_before_last part), it skips the mask application unit and directly performs Q@K^T calculation, with the result cached in on-chip BRAM. When the hardware control logic detects the current window block (num_clean_and_mask part), it enables the mask application unit to fully calculate and apply the causal mask. The mask pre-computation and fast application unit avoids pipeline stalls caused by conditional branches. Specifically, the mask pre-computation and fast application unit pre-generates an upper triangular mask matrix (dynamically generated based on num_clean_and_mask) in hardware, stores it in on-chip BRAM, and quickly applies the mask using a dedicated mask adder (adding the mask value to the attention score instead of skipping the calculation), thus avoiding pipeline stalls caused by conditional branches.

[0111] The on-chip fusion computing pipeline implements a three-stage pipeline from the Q@K^T computation unit to the activation function unit to the weighted V aggregation unit in the on-chip cache (BRAM / URAM). Data flows on-chip, reducing DDR access. After each block is processed, the intermediate result cache is released immediately, reusing BRAM resources.

[0112] The weighted V aggregation unit is the seventh pipeline stage, which performs matrix multiplication on the attention weights and the value vector V to obtain the attention output.

[0113] For each block_i, the calculation formula is as follows:

[0114] Then sum up the output of all blocks:

[0115] in:

[0116] `attn_weights_block_i` represents the normalized attention weights for the i-th block.

[0117] V_block_i is the value vector of the i-th block.

[0118] `attn_output_block_i` represents the attention output of the i-th block.

[0119] attn_output is the final attention output after accumulating all blocks.

[0120] The adaptive block control logic dynamically selects the block strategy based on the sequence length of K and the on-chip cache capacity (available BRAM / URAM resources). In short sequence mode, a one-time computation mode is selected, where all K data is loaded into the on-chip cache and processed in one go. In long sequence mode, a block mode is selected, with the block size dynamically calculated based on the on-chip cache capacity and sequence length, dynamically adjusting the pipeline configuration to optimize throughput.

[0121] The block-level KV cache access optimization unit reads historical block K / V data from the KV cache (L1 / L2 hierarchical cache) block by block, using DMA or a dedicated cache controller for batch reading, reducing random access overhead and supporting pipelined data flow.

[0122] Other technical details and implementation steps of this embodiment 5 are the same as those of embodiment 1, and will not be repeated here.

[0123] Example 6

[0124] This embodiment 6, based on embodiment 1, further explains the connection module between the output projection and the gated residual. For example... Figure 6 The diagram shown is a structural schematic of the module connecting the output projection and the gated residual.

[0125] The module includes an output projection unit and a gated residual connection unit. The output projection unit is used to perform matrix multiplication on the attention output feature vector and the weight matrix W_out.

[0126] The formula for calculating the output projection is as follows:

[0127] in:

[0128] attn_output is the attention output feature vector.

[0129] W_out is the output projection weight matrix.

[0130] output is the projected feature vector.

[0131] The gated residual connection unit integrates gated multiplication and residual addition into a single hardware unit, enabling the fusion calculation of gate * new_value + skip_value. Data flows on-chip (gate value, new_value, and skip_value are read from on-chip BRAM / URAM, and the calculation result is written back directly), avoiding the intermediate result being written back to DDR, thus reducing memory access latency and bandwidth consumption.

[0132] The formula for calculating gated residual connections is as follows:

[0133] in:

[0134] gate is the gate value

[0135] new_value is the output projected feature vector.

[0136] skip_value is the input feature vector (the feature vector before multi-head self-attention).

[0137] output is the output feature vector after the gated residual connection.

[0138] Other technical details and implementation steps of this embodiment 6 are the same as those of embodiment 1, and will not be repeated here.

[0139] Example 7

[0140] This embodiment 7, based on embodiment 1, further explains the multilayer perceptron module. For example... Figure 7 The diagram shown is a structural schematic of a multilayer perceptron module.

[0141] The second-layer normalization and adaptive layer normalization modulation module is the ninth pipeline stage, used to perform layer normalization on the input feature vector and apply adaptive layer normalization modulation parameters for modulation. This module includes a layer normalization unit, an adaptive layer normalization modulation parameter lookup table storage unit, and a lookup and multiply-accumulate fusion unit. The layer normalization unit calculates the mean and variance of the input feature vector in the last dimension, normalizes the feature vector, and applies learnable weight parameters for scaling. The specific implementation is the same as the first-layer normalization unit. The adaptive layer normalization modulation parameter lookup table storage unit pre-calculates the modulation parameters (shift_mlp, scale_mlp, gate_mlp) for all diffusion steps and stores them in BRAM / URAM. If all blocks share the adaptive layer normalization parameters, only one lookup table is needed, reducing storage resource consumption. The lookup and multiply-accumulate fusion unit integrates the lookup operation with the modulation calculation (x * (1+scale) + shift) in a single pipeline stage, reducing intermediate data access.

[0142] The multilayer perceptron module has a tenth and an eleventh pipeline stage. Through these two pipeline stages, the modulated feature vector undergoes a first-layer linear transformation, GELU activation, and a second-layer linear transformation, respectively. The module includes a first-layer linear transformation unit, a GELU activation function unit, and a second-layer linear transformation unit.

[0143] The formula for calculating the first-level linear transformation is as follows:

[0144] in:

[0145] x is the input feature vector (the modulated feature vector).

[0146] W_mlp1 is the weight matrix of the first-level linear transformation.

[0147] mlp_hidden is the hidden layer feature vector after the first linear transformation.

[0148] The GELU activation function unit and the second-level linear transformation unit constitute the eleventh pipeline stage. These two pipeline stages perform GELU activation and the second-level linear transformation on the output of the first-level linear transformation, respectively. The GELU activation function is implemented using a lookup table, avoiding the use of floating-point units and reducing hardware resource consumption.

[0149] The second-level linear transformation is calculated as follows:

[0150] in:

[0151] mlp_hidden is the hidden layer feature vector after the first linear transformation.

[0152] GELU is the Gaussian Error Linear Unit activation function, implemented using a lookup table.

[0153] W_mlp2 is the weight matrix of the second-level linear transformation.

[0154] mlp_output is the output feature vector after the second linear transformation.

[0155] The second gated residual connection module is the twelfth pipeline stage, used to perform residual connection between the multilayer perceptron output and the input feature vector through a gating mechanism to obtain the output feature vector.

[0156] The module includes a gated residual fusion hardware unit, a gated value hardware calculation unit, a double skip value cache management hardware, a conditional skip control logic, a gated residual pipeline fusion unit, and a gated multiplier resource reuse unit.

[0157] The gated residual fusion hardware unit integrates gated multiplication and residual addition into a single hardware unit, enabling the fusion calculation of gate * new_value + skip_value. Data flows on-chip (gate value, new_value, and skip_value are read from on-chip BRAM / URAM, and the calculation result is written back directly), avoiding the need to write intermediate results back to DDR, thus reducing memory access latency and bandwidth consumption.

[0158] The gate value hardware calculation unit enables the calculation of the gate value (gate) to use a hardware lookup table or piecewise linear approximation to implement nonlinear functions such as sigmoid, avoiding the use of floating-point arithmetic units, reducing hardware resource consumption. The gate value can be quantized into INT8 or fixed-point format for storage, and the lookup operation and subsequent calculations are integrated into the pipeline.

[0159] The dual-skip value cache management hardware uses on-chip BRAM / URAM to cache two skip values ​​(x_skip and x_skip2), adopts a dual-port BRAM or time-division multiplexing strategy, supports pipelined execution of two gated residual connections of multi-head self-attention and multilayer perceptron, optimizes cache access mode, reduces cache conflicts, and pre-caches and forwards skip values ​​to avoid pipeline pauses caused by waiting for residual connection calculation to complete.

[0160] The conditional skip control logic hardware detection unit detects whether the gate value is less than a preset threshold. If the gate value is very small, the hardware control logic directly outputs skip_value through the MUX selector, skipping the multiplication operation of gate * new_value, saving multiplier resources and power consumption, reducing computational latency, and avoiding pipeline stalls caused by conditional branches.

[0161] The gated residual pipeline fusion unit integrates gated residual calculation with upstream and downstream modules (such as layer normalization and activation functions) in the same hardware pipeline. Gate calculation, multiplication, addition, and subsequent processing are executed continuously in the pipeline, reducing pipeline pauses and intermediate data caching requirements.

[0162] The gated multiplier resource reuse unit enables two gated residual connections after multi-head self-attention and after multilayer perceptron to share the same set of gated multiplier hardware resources. By using time multiplexing, hardware resource consumption is reduced. The hardware control unit schedules the two gated residual connections to use the multiplier sequentially. In a single cycle, the input vector inputs only one element, matching the data flow input from the previous pipeline stage, thereby realizing pipeline design and avoiding data flow congestion.

[0163] The other technical details and implementation steps of this embodiment 7 are the same as those of embodiment 1, and will not be repeated here.

[0164] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A hardware acceleration system for Eso-LMs based on FPGA for dynamic sequence length block attention computation, characterized in that, include: The hardware acceleration system comprises a first-layer normalization and adaptive layer normalization modulation module, a QKV projection module, a rotation position encoding application module, a KV buffer management module, a block attention calculation module, an output projection and gated residual connection module, a second-layer normalization and adaptive layer normalization modulation module, an MLP module, and a second gated residual connection module. The system divides the computation process of each Transformer Block in the Eso-LMs model into 12 pipeline stages and outputs the data stream in a pipelined manner. The first layer normalization and adaptive layer normalization modulation module is the first pipeline stage, which is used to perform layer normalization processing on the input feature vector, and obtain the adaptive layer normalization modulation parameters by looking up a table according to the time conditions, and modulate the normalized feature vector. The QKV projection module is the second pipeline stage, used to project the modulated feature vector into a query vector Q, a key vector K, and a value vector V through a linear transformation. The rotation position encoding application module is the third pipeline stage, used to apply rotation position encoding to the first head_dim / 2 dimensions of the query vector Q and the key vector K, while the value vector V is not subject to rotation position encoding. The KV cache management module is the fourth pipeline stage, used to merge the current key vector K and value vector V with the key vector and value vector of the historical cache, so as to realize the dynamic update of the KV cache. The block attention calculation module consists of a fifth, sixth, and seventh pipeline stage. The block attention calculation is performed through three pipeline stages, which respectively perform Q@K^T calculation, activation function normalization, and weighted V aggregation operations. The output projection and gated residual connection module is the eighth pipeline stage, which is used to transform the attention output through linear projection and perform residual connection with the input feature vector through a gate mechanism. The second layer normalization and adaptive layer normalization modulation module is the ninth pipeline stage, which is used to perform layer normalization processing on the input feature vector and apply adaptive layer normalization modulation parameters for modulation. The MLP module consists of a tenth and an eleventh pipeline stage, which performs a first-level linear transformation, GELU activation, and a second-level linear transformation on the modulated feature vector through the two pipeline stages. The second gated residual connection module is the twelfth pipeline stage, used to perform residual connection between the MLP output and the input feature vector through a gating mechanism to obtain the output feature vector.

2. The Eso-LMs hardware acceleration system for dynamic sequence length block attention computation based on FPGA according to claim 1, characterized in that, The first-layer normalization and adaptive layer normalization modulation module includes a layer normalization unit, an adaptive layer normalization modulation parameter lookup table storage unit, and a lookup table and multiplication-addition fusion unit. The layer normalization unit is used to calculate the mean and variance of the input feature vector and normalize the feature vector. The adaptive layer normalization modulation parameter lookup table storage unit pre-calculates the modulation parameters for all diffusion steps and stores them in BRAM / URAM. The modulation calculation formula is as follows: in: x is the normalized eigenvector. `scale` represents the adaptive layer normalization modulation parameters (scaling parameters). `shift` represents the adaptive layer normalization modulation parameter (offset parameter). The table lookup and multiply-accumulate fusion unit integrates table lookup operations with modulation calculations into a single pipeline.

3. The Eso-LMs hardware acceleration system for dynamic sequence length block attention computation based on FPGA according to claim 1, characterized in that, The QKV projection module performs matrix multiplication on the input feature vector and the weight matrix W_qkv to obtain a QKV tensor. The QKV tensor is then separated into a query vector Q, a key vector K, and a value vector V through a rearrangement operation.

4. The Eso-LMs hardware acceleration system based on FPGA for dynamic sequence length block attention computation according to claim 1, characterized in that, The rotation position encoding application module includes a partial-dimensional rotation position encoding unit, a dedicated rotate_half unit, a cos / sin lookup table optimization unit, and a Q / K parallel processing unit. The partial-dimensional rotation position encoding unit only processes the rotation in the first head_dim / 2 dimensions, while the second head_dim / 2 dimensions are directly processed, reducing computation by 50%. The dedicated rotate_half unit implements the data rearrangement logic [x0,x1,x2,x3...] to [-x1,x0,-x3,x2...] in hardware, using simple data rearrangement logic instead of multiplication operations to reduce hardware resource consumption. The sine / cosine lookup table optimization unit looks up the table based on the position index, with sine and cosine values ​​pre-stored in BRAM / URAM, supporting dynamic sequence length and avoiding redundant calculations. The Q / K parallel processing unit enables parallel application of rotation position encoding in Q and K, improving processing efficiency and supporting pipelined parallel execution.

5. The Eso-LMs hardware acceleration system for dynamic sequence length block attention computation based on FPGA according to claim 1, characterized in that, The KV cache management module includes a dynamically sequence-length driven cache allocation control unit, an L1 / L2 hierarchical cache storage unit, a double-buffered KV cache unit, an incremental update calculation unit, a predictive prefetch control unit, and a cache line aligned access optimization unit. The dynamic sequence length-driven cache allocation control unit dynamically adjusts the L1 / L2 cache allocation strategy based on the current sequence length and window position, rather than using a fixed cache size or static allocation, thus making full use of the FPGA's hierarchical storage resources. The L1 / L2 hierarchical cache storage unit includes L1 cache (URAM / BRAM, low-latency access) and L2 cache (DDR, high-capacity storage), optimized for the characteristics of FPGA resources. The dual-buffered KV cache unit implements read-write separation with dual buffering to avoid access conflicts and support pipeline parallelism; the incremental update calculation unit only calculates and stores the K / V of newly generated tokens at the hardware level, reuses historical cache, and reduces calculation and storage overhead. The predictive prefetch control unit predicts the next K / V position that may be accessed based on the window position and loads it in advance at the hardware level, hiding DDR access latency; the cache line alignment access optimization unit optimizes the DDR access mode, batches and merges access requests from adjacent positions, and improves bandwidth utilization.

6. The Eso-LMs hardware acceleration system for dynamic sequence length block attention computation based on FPGA according to claim 1, characterized in that, The block attention calculation module includes a block calculation hardware unit, a prefix cumulative activation function register, a historical block and current window block separation processing control unit, an on-chip fusion calculation pipeline, adaptive block control logic, and a block-level KV cache access optimization unit. The block-based computation hardware unit is the fifth pipeline stage. It divides the dynamically growing K sequence into fixed-size blocks and uses GEMM units to compute Q @ K_block^T block by block. The computation result of each block is stored in on-chip BRAM / URAM, avoiding the need to compute the entire Q @ K^T matrix at once and reducing on-chip cache capacity requirements. The block attention score calculation formula is as follows: in: Q is the query vector K_block is the key vector of the block. K_block^T is the transpose of the block key vector. The prefix cumulative activation function register is the sixth pipeline stage. It uses dedicated registers to maintain cross-block activation function statistics (max_global_reg and sum_exp_reg). These registers are updated after each block is processed, which supports cross-block activation function normalization and avoids the need to store the complete attention score matrix in DDR. The historical block and current window block separation processing control unit includes a historical block processing path and a current window block processing path. When the hardware control logic detects a historical block, it skips the mask application unit and directly performs Q@K^T calculation. The result can be cached in the on-chip BRAM. When the hardware control logic detects the current window block, it enables the mask application unit to fully calculate and apply the causal mask. The pipeline stalls caused by conditional branches are avoided through mask pre-computation and fast application units. After each block is processed, the intermediate result cache is released immediately and the BRAM resources are reused. The weighted V aggregation unit is the seventh pipeline stage, which performs matrix multiplication operations on the attention weights and the value vector V. The adaptive block control logic dynamically selects the block strategy based on the sequence length of K and the on-chip cache capacity. The short sequence mode selects the one-time calculation mode, and the long sequence mode selects the block mode. The block size is dynamically calculated based on the on-chip cache capacity and sequence length, and the pipeline configuration is dynamically adjusted to optimize throughput. The block-level KV cache access optimization unit reads historical block K / V data from the KV cache block by block, and uses DMA or a dedicated cache controller for batch reading to reduce random access overhead and support pipelined data flow.

7. The Eso-LMs hardware acceleration system for dynamic sequence length block attention computation based on FPGA according to claim 1, characterized in that, The output projection and gated residual connection module includes an output projection unit and a gated residual connection unit; the calculation formula for the gated residual connection is as follows: in: gate is the gate value new_value is the output projected feature vector. skip_value is the input feature vector (the feature vector before multi-head self-attention). The output projection unit is used to perform matrix multiplication operations on the attention output feature vector and the weight matrix W_out; the gated residual connection unit integrates gated multiplication and residual addition into a hardware unit to realize the fusion calculation of gate * new_value + skip_value. The data flows on-chip (gate value, new_value, and skip_value are read from on-chip BRAM / URAM, and the calculation result is directly written back), avoiding the intermediate result being written back to DDR, reducing memory access latency and bandwidth consumption.

8. The Eso-LMs hardware acceleration system for dynamic sequence length block attention computation based on FPGA according to claim 1, characterized in that, The second-layer normalization and adaptive layer normalization modulation module includes a layer normalization unit, an adaptive layer normalization modulation parameter lookup table storage unit, and a lookup and multiply-accumulate fusion unit. The adaptive layer normalization modulation parameter lookup table storage unit pre-calculates the modulation parameters for all diffusion steps and stores them in BRAM / URAM. If all blocks share AdaLN parameters, only one lookup table is needed, reducing storage resource consumption. The lookup and multiply-accumulate fusion unit integrates the lookup operation with the modulation calculation (x * (1+scale) + shift) in a single pipeline stage, reducing intermediate data access and pipeline pauses.

9. The Eso-LMs hardware acceleration system for dynamic sequence length block attention computation based on FPGA according to claim 1, characterized in that, The MLP module includes a first-layer linear transformation unit, a GELU activation function unit, and a second-layer linear transformation unit. The first-layer linear transformation unit is the tenth pipeline stage, used to perform matrix multiplication operations between the modulated feature vector and the weight matrix W_mlp1. The GELU activation function unit and the second-layer linear transformation unit are the eleventh pipeline stages. Through two pipeline stages, the output of the first-layer linear transformation is subjected to GELU activation and the second-layer linear transformation, respectively. The GELU activation function is implemented using a lookup table method, avoiding the use of floating-point arithmetic units and reducing hardware resource consumption.

10. The Eso-LMs hardware acceleration system based on FPGA for dynamic sequence length block attention computation according to claim 1, characterized in that, The second gated residual connection module includes a gated residual fusion hardware unit, a gated value hardware calculation unit, a double skip value cache management hardware, conditional skip control logic, a gated residual pipeline fusion unit, and a gated multiplier resource reuse unit. The gated residual fusion hardware unit merges gated multiplication and residual addition into one hardware unit to achieve the fusion calculation of gate * new_value + skip_value. Data flows on-chip, avoiding writing intermediate results back to DDR, reducing memory access latency and bandwidth consumption. The gate value hardware calculation unit calculates the gate value using hardware lookup table or piecewise linear approximation to implement nonlinear functions such as sigmoid, avoiding the use of floating-point arithmetic units and reducing hardware resource consumption (DSP, LUT). The gate value can be quantized into INT8 or fixed-point format for storage, and the lookup table operation is integrated with subsequent calculations in the pipeline. The dual-skip value cache management hardware uses on-chip BRAM / URAM to cache two skip values ​​(x_skip and x_skip2), adopts a dual-port BRAM or time-sharing multiplexing strategy, supports pipelined execution of two gated residual connections, MSA and MLP, optimizes cache access mode, reduces cache conflicts, and caches and forwards skip values ​​in advance to avoid pipeline pauses caused by waiting for residual connection calculation to complete. The conditional skip control logic hardware detection unit detects whether the gate value is less than a preset threshold. If the gate value is very small, the hardware control logic directly outputs skip_value through the MUX selector, skipping the multiplication operation of gate * new_value, saving multiplier resources and power consumption, reducing computational latency, and avoiding pipeline stalls caused by conditional branches. The gated residual pipeline fusion unit integrates gated residual calculation with upstream and downstream modules (such as normalization modules and activation functions) into the same hardware pipeline. Gate calculation, multiplication, addition, and subsequent processing are executed continuously in the pipeline, reducing pipeline pauses and intermediate data caching requirements. The two gated residual connections after MSA and MLP share the same set of gated multiplier hardware resources. Time reuse reduces hardware resource consumption, and the hardware control unit schedules the two gated residual connections to use the multiplier sequentially.

Images