FPGA-based MTLA-Transformer hardware accelerator
By designing an FPGA-based MTLA-Transformer hardware accelerator, the problems of computational resource scheduling and memory access bandwidth of the MTLA-Transformer model on the FPGA platform are solved, realizing efficient incremental inference and resource optimization, which is suitable for edge computing scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUN YAT SEN UNIV
- Filing Date
- 2026-04-24
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies struggle to efficiently deploy the MTLA-Transformer model on FPGA platforms, exhibiting limitations in computational resource scheduling, high memory access bandwidth, and low hardware throughput, thus restricting its application in edge computing and resource-constrained scenarios.
Design an FPGA-based MTLA-Transformer hardware accelerator, including a controller module, an attention computation module, and a feedforward network module. By optimizing data flow, improving computational resource utilization, and reducing storage resource consumption, and by employing parallel computing and pre-computation techniques, achieve efficient incremental inference of the MTLA-Transformer model.
It significantly improves the inference efficiency and hardware resource utilization of the MTLA-Transformer model on FPGA, reduces computational latency and storage overhead, and meets the needs of edge deployment.
Smart Images
Figure CN122154768A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of FPGA and artificial intelligence inference acceleration technology, and more specifically, to an FPGA-based MTLA-Transformer hardware accelerator for implementing incremental inference computation of MTLA-Transformer. It improves inference throughput and reduces model computation and off-chip storage access overhead through mechanisms such as on-chip cache control, KV cache management, data scheduling, offline computation, and operator fusion. Background Technology
[0002] In recent years, the Transformer architecture has made significant progress in natural language processing, computer vision, speech recognition and synthesis, and has been widely adopted in applications such as machine translation, dialogue generation, image understanding, video analysis, and multimodal content generation. The MTLA-Transformer model introduces a multi-head temporal latent attention mechanism into the traditional Transformer framework, reducing KV cache storage and memory access overhead and improving model inference performance in long sequence incremental inference scenarios.
[0003] The MTLA-Transformer primarily involves two computational steps during incremental inference: multi-head temporal latent attention computation and a feedforward network. Multi-head Temporal Latent Attention (MTLA) introduces the concept of temporal weights, weighting and compressing key-value vectors across multiple time steps according to these weights, thereby compressing the multi-time-step key-value representation into a single-time-step latent representation. The feedforward network (FFN) performs linear transformations, activations, and further linear transformations on the output of the attention sublayer to generate the final output of that layer. Overall, this model achieves computational efficiency optimization and information representation integration for long-sequence incremental inference through attention compression and feedforward network processing.
[0004] With the rapid growth in the scale and computational demands of deep learning models, traditional general-purpose processors face challenges such as high power consumption, limited throughput, and heavy storage pressure in high-performance inference tasks. FPGAs, as a programmable hardware acceleration platform, offer highly customizable parallel computing capabilities, low power consumption, and flexible on-chip memory management. They can be customized to design data paths and computing units based on the characteristics of model computation, thereby optimizing resource utilization while maintaining performance. Therefore, FPGAs are gaining increasing attention in deep learning model acceleration, edge inference, and low-latency applications, and offer new possibilities for efficiently implementing temporal attention models like the MTLA-Transformer.
[0005] In incremental inference, strict forward dependencies exist between time steps, making parallelization impossible at the time step level and pipelined operations across time steps. This further restricts the scheduling of on-chip cache and computing resources, placing higher demands on off-chip memory access bandwidth and hardware throughput. MTLA-Transformer introduces temporal weights through a multi-head temporal latent attention mechanism, weighting and compressing the key-value vectors across multiple time steps. This reduces KV cache usage and redundant computation overhead, enabling efficient incremental inference execution even with limited on-chip cache and computing resources, significantly improving the model's hardware feasibility.
[0006] Currently, implementations of MTLA-Transformer models on FPGA platforms are still relatively lacking. Existing technologies mostly rely on GPUs for model training and inference. Although GPUs possess mature parallel computing capabilities and a well-developed software ecosystem, their high power consumption and fixed architecture limit the flexibility of practical applications in edge computing or resource-constrained scenarios. Therefore, there is an urgent need for a method to efficiently deploy MTLA-Transformer on FPGAs, fully utilizing on-chip computing and caching resources, optimizing data flow paths, reducing computational latency, and minimizing storage overhead while ensuring performance. Summary of the Invention
[0007] This invention provides an FPGA-based MTLA-Transformer hardware accelerator designed to meet the deployment needs of edge computing scenarios. It optimizes data flow, improves computing resource utilization, reduces computational latency, and minimizes storage resource consumption through targeted hardware design. A dedicated FPGA hardware accelerator system is implemented by leveraging the characteristics of the MTLA-Transformer.
[0008] To achieve the above objectives, the technical solution of the present invention is as follows: The FPGA-based MTLA-Transformer model hardware accelerator comprises a controller module, an attention computation module, and a feedforward network module. Each module contains smaller modules responsible for different functions. Through the unified scheduling of on-chip and off-chip storage by the controller module, and combined with the collaborative processing of MTLA attention computation and the feedforward network module, this invention can achieve efficient incremental inference of the MTLA-Transformer model with limited on-chip resources, thereby improving inference efficiency and hardware resource utilization and meeting edge deployment requirements.
[0009] As a preferred embodiment, the main functions of the controller module include: unified scheduling and management of data transmission, real-time control of data flow and computation process, and completion of data reading and writing between on-chip cache and off-chip memory. Among these, data reading and writing between on-chip cache and off-chip memory is the most important function handled by the controller. The on-chip cache is a high-speed, small-capacity storage resource within the FPGA, used to temporarily store intermediate data and reduce memory access latency; the off-chip memory is a large-capacity external storage resource used to store model parameters and KV cache data, and exchanges data with the FPGA via a bus.
[0010] As a preferred embodiment, the attention calculation module is used to perform MTLA calculations, and its calculation formula during incremental inference is as follows: in, This indicates that a token has been entered. This indicates that the compression has undergone temporal and low-rank decomposition. Key-value vector, Represents the position after encoding Query vector, This indicates that the data has undergone temporal and low-rank decomposition and compression, and positional encoding has been applied. Key vector, This represents the weight matrix obtained from MTLA training. This represents the dimension of each attention head, used for softmax scaling. This represents the final calculated output of the MTLA attention calculation.
[0011] As a preferred embodiment, the attention calculation module includes a reusable matrix multiplication and addition calculation array, a position encoding submodule, a HyperNet calculation submodule, a KV cache management submodule, a Softmax submodule, and a residual normalization submodule, for implementing hardware acceleration on the FPGA.
[0012] As a preferred embodiment, the reusable matrix multiply-addressable array performs matrix multiply-address operations using a systolic array. Its input supports various data formats, including optional weight matrices, data matrices, and data vectors. By configuring parameters such as the array's operation mode, input data source, weight parameters, and output write-back address, the array can be flexibly reused at different inference stages to perform matrix multiply-address operations in processes such as QK projection, Q-mapping calculation, attention score calculation, V-projection, and output projection.
[0013] As a preferred embodiment, the matrix multiplication and addition array employs parallel computation for QK projection operations to improve the throughput of the hardware accelerator. The specific QK projection operation process is as follows: in For query vector, For key vectors, Let Q be the projection weight matrix. Let K be the projected weight matrix. Since the two projections do not affect each other, the parallel computation acceleration scheme can provide low latency and fully utilize the two computational blocks of the matrix multiplication and addition array. Meanwhile, After separating the non-encoded portion, Linergy is used for normalization. The normalization formula is as follows: in, As input features, For channel-related scaling parameters, For channel-related translation parameters, This is for output features. Using operator fusion is mathematically equivalent and avoids the multiplication of the input feature vector with a scalar.
[0014] As a preferred embodiment, the matrix multiply-add array employs parallel computation to improve hardware acceleration throughput when calculating attention scores. The specific attention calculation is divided into two parts: non-positionally encoded score calculation and positionally encoded score calculation, with the following formulas: in, This represents the query vector without rotational position encoding. This represents the key vector without applied rotational position encoding. This represents the query vector to which rotational position encoding has been applied. This represents the key vector to which rotational position encoding is applied. For non-positionally encoded score results, The positional coding score is the result of the two scores, which are then added together to obtain the final MTLA score.
[0015] As a preferred embodiment, the matrix multiplication and addition array requires operator fusion when performing output projection calculations. By using the reciprocal of the fusion scaling factor to optimize the Softmax operation, this part of the calculation fusion is completed offline, thus not consuming on-chip hardware resources of the FPGA and not affecting the overall computing throughput.
[0016] As a preferred embodiment, the location encoding submodule implements... Vector sum Vector position encoding. In GPU inference, position encoding involves two steps: rotation factor calculation and rotation transformation. The position encoding submodule avoids tedious rotation factor calculations on the FPGA through pre-computation, only requiring the reading of pre-computation results for rotation transformation encoding. It also employs a parallel architecture to reduce computational latency and improve inference efficiency.
[0017] As a preferred embodiment, the HyperNet computation submodule implements the HyperNetwork computation in MTLA-Transformer. HyperNetwork is a neural network in MTLA used to compute KV cache compression weights, consisting of two fully connected layers and a sigmoid activation function. The HyperNet computation submodule incorporates a pre-computation cache for the positional encoding vectors, pre-completing one fully connected layer in the original inference computation and using the saved result to complete the overall network output.
[0018] As a preferred embodiment, the KV cache management submodule calculates the KV vector at the current time step and updates it in the cache for subsequent inference. It is worth noting that, in order to reduce the Softmax scaling factor... In FPGA accelerators, the KV vector update strategy differs from that in GPU computing. The scaling factor is pre-multiplied into the KV vector at each step to avoid multiplying the large matrix by the scaling factor after the fraction calculation, which would introduce a lot of computation and cause hardware architecture blockage.
[0019] As a preferred embodiment, the attention calculation module further includes a Softmax submodule and a residual normalization submodule. The Softmax submodule is used to normalize the attention score matrix to obtain attention weights, and the residual normalization submodule is used to perform residual connection on the attention output and complete the normalization process, thereby improving the stability and convergence performance of the MTLA-Transformer model inference.
[0020] As a preferred embodiment, the feedforward network module includes a first linear transformation submodule, an activation submodule, a second linear transformation submodule, and a residual connection normalization submodule. The first linear transformation submodule is used to perform dimensionality-up mapping on the input features, the activation submodule is used to introduce nonlinear expressive power, the second linear transformation submodule is used to map the features back to the original dimension, and the residual connection normalization submodule is used to perform residual connection and normalization processing on the output to improve the network's expressive power and computational stability.
[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0022] 1. This invention enables flexible scheduling and data reuse management of on-chip cache and off-chip storage through a controller, effectively reducing memory access overhead and improving overall computing throughput.
[0023] 2. This invention uses a pulsating array to implement parallel matrix multiplication and addition operations, and performs fusion processing on the projection operator, thereby reducing intermediate data movement and reducing hardware resource pressure.
[0024] 3. The present invention pre-calculates and stores the position encoding results, avoiding repeated calculations during the inference stage, thereby reducing computational complexity and improving execution efficiency.
[0025] 4. This invention performs offline pre-computation and storage of HyperNet output, reducing the amount of data reading and computation during the online inference stage, thereby improving system real-time performance and resource utilization.
[0026] 5. This invention combines storage scheduling optimization, parallel computing architecture, and pre-computation of key operators to significantly improve accelerator throughput performance and reduce hardware resource overhead while ensuring computational accuracy. Attached Figure Description
[0027] Figure 1 This is a schematic diagram of the overall architecture of an FPGA-based MTLA-Transformer hardware accelerator according to this application;
[0028] Figure 2 This is a schematic diagram of the controller module structure of this application;
[0029] Figure 3 This is a schematic diagram illustrating the scheduling of the controller module of this application during accelerator operation;
[0030] Figure 4 This is a schematic diagram of the reusable matrix multiply-accumulate array structure in the attention calculation module of this application;
[0031] Figure 5 This is a schematic diagram of the position encoding submodule in the attention calculation module of this application;
[0032] Figure 6 This is a schematic diagram of the HyperNet submodule structure in the attention calculation module of this application;
[0033] Figure 7 This is a schematic diagram of the KV cache management submodule in the attention calculation module of this application. Detailed Implementation
[0034] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0035] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0036] Example 1
[0037] The overall hardware design of the FPGA-based MTLA-Transformer hardware accelerator is as follows: Figure 1 As shown, the three main modules include a controller module, an attention computation module, and a feedforward network module. The controller module is responsible for reading the input embedding vector and weight parameters from off-chip memory into the on-chip cache and scheduling the execution of each computation module according to the time step order. The attention computation module employs a multi-head temporal latent attention mechanism, using temporal weights to weight and compress the key-value vectors of multiple time steps, thereby improving inference efficiency while ensuring computational correctness. The feedforward network module performs linear transformations, activations, and linear transformations on the output of the attention sub-layer to generate the final output of that layer. Each module consists of many sub-modules. The attention module includes a reusable matrix multiplication and addition computation array, a position encoding sub-module, a HyperNet computation sub-module, a KVcache management sub-module, a Softmax sub-module, and a residual normalization sub-module. The feedforward network module includes a first linear transformation sub-module, an activation sub-module, a second linear transformation sub-module, and a residual connection normalization sub-module.
[0038] Through the collaborative work of the modules, this invention can efficiently execute incremental inference of the MTLA-Transformer model with limited on-chip resources, significantly improving the performance and resource utilization of the hardware implementation, and providing a feasible hardware acceleration solution for edge deployment.
[0039] Example 2
[0040] This embodiment 2 is based on embodiment 1, and further explains the controller module therein. During the incremental inference process of the MTLA-Transformer model, a lot of data needs to be pre-loaded into the FPGA's off-chip memory. This data includes weight matrices, pre-calculated results, and KV cache, etc. The controller module is responsible for reading the data from the off-chip memory to the on-chip cache in a timely manner during the accelerator inference operation, so that the necessary input can be received more quickly during computation. Its working principle and structure are as follows: Figure 2As shown. Meanwhile, the KV cache is updated as the time step increases. The updated data is temporarily stored in the on-chip cache, but due to the limited storage resources of the on-chip cache, the controller module will promptly write it to the off-chip memory for updating.
[0041] The controller module works in close coordination with each computing module, and the specific operational logic that changes with inference time is as follows: Figure 3 As shown, the controller module overlaps the data read and write operations of the on-chip cache and off-chip memory with the computation processes of each computing module, thereby achieving parallel execution of data transmission and computation, reducing or avoiding additional latency to the critical path of inference, and completing the read and write updates of cached data without blocking the computation process of the computing modules, thus improving inference throughput.
[0042] Example 3
[0043] This embodiment 3, based on embodiment 1, further explains the attention calculation module. The attention calculation module breaks down the MTLA operation, and its calculation formula is as follows: By decomposing the computation steps, we can divide them into: QK projection, position encoding, HyperNet computation, KV cache update, Q-map computation, attention score computation, Softmax computation, V projection, output projection, and residual normalization. Therefore, the attention computation module can be decomposed into a reusable matrix multiplication and addition computation array, a position encoding submodule, a HyperNet computation submodule, a KV cache management submodule, a Softmax submodule, and a residual normalization submodule.
[0044] Reusable matrix multiplication and addition array hardware structure such as Figure 4 As shown. The matrix multiplication-addition computation array consists of a first computation block and a second computation block. Under the unified scheduling and control of the controller module, the two computation blocks can be configured to work in parallel mode or time-division multiplexing mode. The main computational objectives of the matrix multiplication-addition computation array are QK projection, Q-mapping calculation, attention score calculation, V projection, and output projection.
[0045] Optimizations in the QK projection include parallel computation and operator fusion. Parallel computation assigns the Q-projection and K-projection to two separate computation blocks, increasing accelerator throughput. Operator fusion occurs in the normalized structure after the K-projection; the Lineargy normalization operation is as follows: The specific integration method is as follows: The weight matrix obtained after fusion Weight matrix before fusion The data volume is equal, so it will not introduce additional storage pressure.
[0046] In the process of attention score calculation, this invention adopts a parallel computing optimization strategy, which unfolds and calculates the non-positional encoding score calculation path and the positional encoding score calculation path in parallel and synchronously, thereby reducing serial waiting time and improving the throughput of attention score calculation.
[0047] The optimization in the output projection is operator fusion. The specific formula for calculating the output projection is: in, The output features of the attention module are represented. This represents the output projection weight matrix. This represents the final MTLA attention result. To complement the optimizations of the Softmax operation by the Softmax submodule and KV cache management submodule in the MTLA-Transformer hardware accelerator, The reciprocal of the scaling factor needs to be merged. The method is as follows: Specifically, The updated weight matrix is a fixed constant related to the attention head dimension. and With the same number of parameters, there is no increase in FPGA storage.
[0048] The submodule structure of the position encoding submodule is as follows: Figure 5 As shown, the rotation factor calculation required for position encoding is fused through pre-computation, and parallel processing is used to improve operating efficiency.
[0049] The formula for calculating the twitch factor is as follows: in, Indicates a position constant. For the vector dimension that needs to be encoded at position, For dimensional indexing, For frequency, The phase angle, The rotation factor is a parameter that depends only on the position and vector dimension, and is independent of the specific value of the vector encoding. Since the vector has a fixed degree during incremental inference, the rotation factor at different positions can be calculated in advance, and the on-chip computation can be reduced by using the FPGA lookup table.
[0050] The formula for calculating rotation transformation is as follows: in, The vectors that need to be position-encoded are grouped pairwise along adjacent dimensions. The positional encoding result is formed by splitting and merging all pairwise grouped results to form a positional encoding vector. According to the above formula, the positional encoding submodule contains multiple small-dimensional matrix multipliers and adders for parallel processing of operations between 2*2 matrices and vectors. Each encoding operation retrieves a pre-calculated 2*2 rotation matrix from a lookup table.
[0051] The HyperNet compute submodule structure is as follows: Figure 6 As shown, by utilizing pre-computation results to reduce hardware computation, the hardware accelerator can complete incremental inference tasks while avoiding repeated computations of the fully connected layer.
[0052] Specifically, the calculation formula for HyperNetwork during the incremental inference phase is as follows: in, This represents the position-encoded input content vector. This represents the positional encoding vector for temporal compression. This describes the weights calculated and output by HyperNetwork. Because... It is only related to position and is independent of the input vector. The weights obtained from computational training are fixed, therefore the weights for storing all locations can be pre-computed. As a result, further operator fusion was performed to... The weights and biases of the fully connected layer are fused with the pre-computation results and directly read and called in the HyperNet computation submodule, avoiding multiple high-dimensional fully connected layer calculations on the FPGA.
[0053] The structure of the KV cache management submodule is as follows: Figure 7 As shown, this is used for updating and managing access to the key-value cache during incremental inference. This requires multiplying the updated KV values by a scaling factor. This is in conjunction with the overall optimization of the Softmax operation. In MTLA, the Softmax operation involves calculating the scores of both the non-positionally encoded and positionally encoded portions, as follows: After calculating the fraction, the steps of Softmax are as follows: The scaling factor should be calculated in advance during the two-step fraction calculation process. Multiplication does not affect the result of the Softmax operation, therefore it can be performed separately. and The scaling factor is applied to the two parts that are read and updated via the KV cache. The fusion process involves updating the KV vector to be stored in the KV cache at each time step. This update method eliminates the need for Softmax scaling on every element of the attention score result matrix; instead, scaling is performed on the vector in each update step. This significantly reduces computational cost and eliminates the need for repeated scaling calculations, achieving hardware acceleration.
[0054] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. An FPGA-based MTLA-Transformer hardware accelerator, characterized in that, The hardware accelerator module includes a controller module, an attention computation module, and a feedforward network module. The accelerator design is achieved through hardware-friendly methods such as module reuse, operator fusion, parallel processing, and pre-computation lookup, and incremental inference computation is optimized on FPGA. The controller module is used to uniformly schedule the on-chip cache and off-chip memory during incremental inference, and coordinate data loading, computation execution and key-value cache updates according to the time step sequence. The attention calculation module is used to perform multi-head temporal latent attention operations on the input embedding vector. The attention calculation module includes a reusable matrix multiplication and addition calculation array. The matrix multiplication and addition calculation array is configured to perform projection calculation, attention score calculation and output projection calculation at different inference stages. Normalization scaling, position encoding related scaling or Softmax scaling coefficients are fused into the matrix multiplication and addition operation through operator fusion to reduce the number of independent operators and intermediate data access. The feedforward network module is used to perform linear transformation, nonlinear activation, and linear transformation on the output of the attention sublayer, and form a residual connection with the input. Each module is implemented on an FPGA and connected via a configurable data path. It is uniformly scheduled and executed by the controller module, thereby completing efficient incremental inference computation of the MTLA-Transformer model under limited on-chip resources.
2. The FPGA-based MTLA-Transformer model hardware accelerator according to claim 1, characterized in that, The controller module is used to read the input embedding vector and related weight parameters from the off-chip memory to the on-chip cache, and coordinate the calculation order and data supply of the attention calculation module and the feedforward network module to ensure that the data is supplied to each calculation module in the order of the time step sequence. At the same time, it manages the KV cache update and access timing during incremental inference, supports the continuous processing of multi-step sequences, and schedules the execution of each sub-module according to hardware resources and task requirements to optimize the computing throughput and data utilization.
3. The FPGA-based MTLA-Transformer model hardware accelerator according to claim 1, characterized in that, The attention calculation module includes a reusable matrix multiply-accumulate array, a position encoding submodule, a HyperNet calculation submodule, a KV cache management submodule, a Softmax submodule, and a residual normalization submodule. The reusable matrix multiply-accumulate array comprises a first calculation block and a second calculation block. These two blocks operate in parallel computing or time-division multiplexing under the scheduling of the controller module. By configuring the operation mode, input data source, weight parameters, and output write-back address of the matrix multiply-accumulate array, it can be used to perform matrix multiply-accumulate operations in QK projection, Q-mapping calculation, attention score calculation, V projection, and output projection at different inference stages. The position encoding submodule applies a Rotation Position Encoding (RoPE) transformation to the position encoding portions of the query vector and key vector. The HyperNet calculation submodule calculates temporal compression weights based on the current position and input content. The KV cache management submodule updates or calculates the compressed KV cache during incremental inference. The Softmax submodule normalizes the attention score to obtain the attention weights. The residual normalization submodule adds the projected multi-head Value representation to the input embedding vector, performs residual addition, and normalizes the result to generate the attention sublayer output.
4. The FPGA-based MTLA-Transformer model hardware accelerator according to claim 1, characterized in that, The feedforward network module includes a first linear transformation submodule, an activation submodule, a second linear transformation submodule, and a residual connection normalization submodule. The first linear transformation submodule maps the output of the attention sublayer to a high-dimensional representation. The activation submodule applies a nonlinear activation function to the high-dimensional representation. The second linear transformation submodule maps the activated representation back to the embedding dimension. The residual connection submodule adds the mapped output to the input attention sublayer output and performs normalization processing to generate the final output of the layer.
5. The MTLA-Transformer model attention calculation module according to claim 3, characterized in that, The reusable matrix multiplication and addition array can perform parallel operations on Q-projection and KV-projection when performing QKV projection, and can use operator fusion to incorporate the normalized scalar parameters that need to be processed later into the weight matrix of the projection in advance. The QK projection submodule applies the formula to the input embedding vector: in For query vector, The input vector is the key vector, and both operations are linear matrix multiplication and addition operations with only weights. The QK projection submodule further adopts a block processing strategy, dividing the input vector into several sub-blocks, and implementing matrix multiplication operations on the FPGA through a systolic array, thereby improving the parallel computing efficiency of multi-head projection and reducing on-chip memory access pressure. Since the two operations are not directly related, they can be allocated to the first and second computing blocks respectively for parallel operation in the matrix multiplication and addition array. The calculation incorporates the scalar multiplication operations from the subsequent normalized Lineargy using a fusion operator. In the matrix, to reduce computational load, Lineargy is operated on in MTLA as follows: in, As input features, For channel-related scaling parameters, For channel-related translation parameters, This is for output features. Using operator fusion to incorporate channel-related scaling parameters into the original weight matrix does not increase hardware parameter storage but effectively reduces computational load.
6. The MTLA-Transformer model attention calculation module according to claim 3, characterized in that, The reusable matrix multiply-add computation array performs attention score calculations using parallel data paths for both the score calculation path without rotation position encoding and the score calculation path with rotation position encoding. The results of the two calculations are then merged to generate the attention score, thereby reducing serial waiting time and improving the throughput of attention score calculations. The formula for calculating the score without rotational position coding is as follows: in, This represents the query vector without rotational position encoding. This represents a key vector without rotational position encoding. The formula for calculating the score with applied rotational position coding is as follows: in, This represents the query vector to which rotational position encoding has been applied. This represents the key vector to which rotational position encoding is applied. The operations are not directly related. In the matrix multiplication and addition calculation array, they can be allocated to the first calculation block and the second calculation block for parallel operation. The results of the two calculations are merged to generate an attention score, thereby reducing the serial waiting time and improving the throughput of attention score calculation.
7. The MTLA-Transformer model attention calculation module according to claim 3, characterized in that, When performing output projection, the reusable matrix multiply-accumulate array pre-fuses the output projection matrix with the reciprocal of the scaling factor required for the attention score Softmax. This reduces the computational load caused by Softmax scaling while ensuring the accuracy of the calculation results and keeping the weight size stored on the FPGA unchanged.
8. The MTLA-Transformer model attention calculation module according to claim 3, characterized in that, The position encoding submodule applies a rotation position encoding (RoPE) transformation to the portion of the query vector and key value vector used for position encoding, and stores the transformation result in advance through a lookup table. Combined with the advance scheduling of the controller module, the lookup table value is directly read in the hardware to avoid complex trigonometric function calculations, thereby achieving hardware acceleration in position encoding compared to GPU incremental inference.
9. The MTLA-Transformer model attention calculation module according to claim 3, characterized in that, The HyperNet computation submodule is used to generate temporal compression weights, and its computation process includes: processing the position encoding vector. With content vector Perform fully connected mappings respectively to obtain , ,pass and The correlation between the two is calculated by the inner product and then the weights are obtained by the Sigmoid function. The specific calculation formula for the position encoding vector C is as follows: Among them, the position encoding vector Relying solely on temporal location and independent of input content, location-dependent intermediate results are pre-calculated and stored as a lookup table offline. During incremental inference, these results are retrieved from the table at the current location, only affecting the content vector. After performing a vector dot product and bias addition, the weights are obtained via Sigmoid, thereby reducing the number of operations in the fully connected layer during the inference phase and the amount of data access during weight reuse, thus achieving hardware acceleration of the HyperNet computation submodule and optimization of storage access during inference.
10. The MTLA-Transformer model attention calculation module according to claim 3, characterized in that, The KV cache management submodule reads the pre-loaded compressed key-value cache on-chip, which includes both position-encoded and non-position-encoded KV data. During incremental inference, it updates or appends new compressed KV data. Before updating, it pre-absorbs the scaling factor required for the attention score Softmax, eliminating the need for repeated scaling calculations at each time step during incremental inference, thereby reducing computational load and increasing attention score computation throughput. The controller module schedules KV cache access according to the inference stage, allowing attention calculations at each time step to directly access all required KV data, thus reducing off-chip access latency and improving overall inference throughput.