Accelerated calculation control method and system for executing Transform model by on-chip storage unit
By employing a Tile local residency mechanism and sparse scheduling with dynamic quantization on the FPGA, the problem of separating storage and computation in the Transformer model is solved, achieving efficient data residency and computation, reducing DDR access frequency and power consumption, and maintaining model accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TAIYUAN UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2026-01-21
- Publication Date
- 2026-05-15
AI Technical Summary
In existing technologies, Transformer models on GPU/CPU platforms suffer from the problem of separation of in-memory and computation, resulting in high data migration overhead and high power consumption. FPGA accelerators cannot effectively utilize sparsity and quantization optimization, and dedicated IMC devices are costly and difficult to integrate, making it difficult to achieve a near-in-memory computing effect.
The Tile local residency mechanism is adopted to divide the input tensor into tiles and reside them in the on-chip storage unit. Combined with sparse mask scheduling and dynamic quantization, matrix operations are completed through a close-range computing array. Intermediate results are stored on-chip until the computation is completed, avoiding DDR access.
Significantly reduces DDR access frequency, lowers inference latency and power consumption, improves system response speed and energy efficiency, maintains model accuracy, and is suitable for commercial FPGA implementation.
Smart Images
Figure CN122045127A_ABST
Abstract
Description
Technical Field
[0001] This invention provides an accelerated computing control method and system for executing Transformer models in on-chip memory cells, belonging to the field of data processing technology for executing Transformer models in memory cells. Background Technology
[0002] The Transformer model is a deep learning model that uses multi-head attention as its core computational unit. It can replace recurrent neural networks and convolutional neural networks, and consists of an encoder and a decoder. It has been widely used in industrial applications of machine learning and computer vision. The data processing flow based on the Transformer model mainly includes query-key dot product calculation, softmax normalization, value weighted summation, and generating the final output. This calculation process involves a large number of matrix operations, with a computational complexity of O(n²), where n is the sequence length. However, this data processing process has the following technical problems:
[0003] I. On traditional GPU / CPU platforms, the Transformer inference process suffers from a severe problem of separation of storage and computation:
[0004] Model parameters (weight matrix) and activation values need to be frequently transferred between external DDR memory and computing units, resulting in data round-trip overhead. Especially when dealing with large sequence lengths, DDR access bandwidth becomes a performance bottleneck, and frequent data movement leads to a significant increase in power consumption, which is not conducive to edge deployment.
[0005] Second, the attention behavior is unbalanced; the attention mechanism of the Transformer has inherent sparsity, which is reflected in:
[0006] The importance of different tokens or different attention heads varies significantly.
[0007] A large number of low-contribution attention scores are calculated, but they have little actual impact on the final result;
[0008] Existing accelerators typically employ intensive computing patterns, which cannot effectively utilize this sparsity, resulting in wasted computation.
[0009] III. Existing FPGA Transformer accelerators have the following problems, including:
[0010] When using a systolic array or DPU (deep learning processing unit) architecture, the data flow is a "fetch data → compute → store back → fetch again" pattern. After each operator is executed, the intermediate results are immediately written back to DDR. The data lifecycle is limited to the single operator and cannot achieve cross-operator data residency, resulting in repeated access to external storage.
[0011] Sparsity and quantization are usually treated as independent optimization techniques, decoupled from the hardware architecture. Existing accelerators cannot uniformly schedule sparse computation and quantization operations at the hardware level, resulting in limited optimization effects.
[0012] Softmax operations typically require global normalization, which leads to the separation of computation and data access. This necessitates additional DDR round trips to complete the normalization, disrupting pipeline coherence.
[0013] Fourth, in-memory computing (IMC) chips (such as devices based on RRAM, PCM, and crossbar arrays) have proposed a direction to reduce data migration overhead. However, the manufacturing cost of dedicated IMC devices is much higher than that of commercial FPGAs. Currently, it is difficult to obtain mature IMC chip products on the market, and they are difficult to integrate with the existing FPGA ecosystem, making it difficult to achieve commercial deployment.
[0014] Therefore, how to achieve near-in-memory computing on commercial FPGAs, enabling data residency and computation through architectural innovation without the need for dedicated IMC devices, reducing DDR round trips, improving inference throughput and energy efficiency, while maintaining model accuracy, has become a pressing technical problem for the industry. Summary of the Invention
[0015] To address the technical problems existing in the prior art, the present invention provides an accelerated computation control method for executing Transformer models in on-chip memory cells, comprising the following computation control steps:
[0016] Step 1: Divide the tensors of the storage cells on the input chip into tiles according to a preset dimension;
[0017] Step 2: Load each tile into the on-chip storage unit and reside there;
[0018] Step 3: In a computing array that is closely coupled to the on-chip storage cells, perform matrix multiplication and accumulation processing on the tile;
[0019] Step 4: Skip the calculation of the corresponding tile according to the row or column sparse mask. The sparse mask scheduling module performs head reorganization and load balancing scheduling. The sparse mask scheduling module generates sparse masks through Top-K selection or fixed thresholds, and skips the entire tile for low contribution rows or columns without generating empty calculation cycles.
[0020] Step 5: Perform dynamic quantization and dequantization within the tile. During dynamic quantization, each tile independently calculates the quantization parameters scale and fractional bits. The quantization and dequantization operations are completed within the local BRAM.
[0021] Step 6: Store the intermediate results of the tile in the on-chip storage unit until the calculation is complete.
[0022] In step one, the memory cell on the chip adopts a double buffering mechanism, including a first buffer block and a second buffer block. The first buffer block is used for the calculation of the current tile, and the second buffer block is used for the preloading of the next tile.
[0023] In step three, the computing array includes a DSP / LUT array coupled to BRAM or URAM, which does not access external storage during the lifetime of a single tile.
[0024] An accelerated computing control system for executing Transformer models in on-chip memory cells includes:
[0025] The on-chip memory cells are used to store tile data;
[0026] A computational array, closely coupled to the storage unit, is used to perform matrix multiplication and accumulation of tiles;
[0027] The sparse scheduling module is used to skip the calculation of the corresponding tile based on the sparse mask;
[0028] The quantization module is used to perform dynamic quantization and dequantization within the tile;
[0029] The control module is used to retain the intermediate results of the tile in on-chip storage until the calculation is completed;
[0030] The Softmax module performs exponent calculation and normalization through lookup tables and accumulators, and the intermediate accumulated values are stored in the on-chip storage area corresponding to the tile.
[0031] The present invention has the following advantages over the prior art:
[0032] I. The Transformer model-accelerated computation control method used in this invention completes the entire computation process in on-chip storage through a tile local resident mechanism, which can significantly reduce the number of DDR accesses and reduce the frequency of external storage access.
[0033] Second, by reducing data round-trip overhead, end-to-end inference latency can be effectively reduced, and system response speed can be improved.
[0034] Third, reducing data transfer power consumption can effectively reduce overall power consumption and improve the system's energy efficiency ratio;
[0035] Fourth, by using tile-level dynamic quantization, we can maintain model accuracy while ensuring computational efficiency, thus achieving a balance between accuracy and efficiency.
[0036] Fifth, this control scheme is implemented entirely based on commercial FPGA, requiring no special components, and has good manufacturability and commercialization capabilities. Attached Figure Description
[0037] The present invention will be further described below with reference to the accompanying drawings:
[0038] Figure 1 This is a schematic diagram of the topology of the FPGA system of the present invention;
[0039] Figure 2 This is a schematic diagram of the Tile lifecycle state transition of the present invention;
[0040] Figure 3 This is a flowchart illustrating the steps of sparse jump scheduling in this invention;
[0041] Figure 4 This is a schematic diagram of the near-field computing unit (PE) of the present invention;
[0042] Figure 5 This is a schematic diagram of the structure of Softmax according to the present invention. Detailed Implementation
[0043] like Figures 1 to 5 As shown, this invention provides a near-memory computation acceleration method and system for implementing Transformer network inference on an FPGA. The technical problem to be solved is: how to implement data resident computation in Transformer attention inference on an FPGA without dedicated IMC devices, reduce DDR round trip access, improve inference throughput and energy efficiency, and maintain model accuracy. This invention is used in deep learning applications such as object detection, image recognition, and visual inference in the fields of artificial intelligence hardware acceleration and embedded computing.
[0044] This invention provides a Transformer inference acceleration method and device based on on-chip storage and near-field execution. It employs sparse scheduling, tile local caching, near-field PE computation, and tile dynamic quantization to construct a near-in-memory computation-integrated Attention acceleration architecture. The core architectural features include:
[0045] The Tile local residency mechanism is adopted, including:
[0046] The Q, K, V matrix of the Transformer is divided into tiles of fixed size (e.g., 32×64, 64×64).
[0047] Each tile resides in the FPGA's BRAM (Block RAM) or URAM (Ultra RAM) using a double-buffered mechanism;
[0048] The calculation is only completed near BRAM / URAM, and the calculation results are not returned to DDR until the entire tile calculation process is completed.
[0049] The data lifecycle spans the entire tile computation process, rather than a single operator.
[0050] The algorithm employs a line-level skipping mechanism, including:
[0051] Leveraging the natural sparsity of the attention matrix, perform whole-tile skipping on low-contribution rows / columns;
[0052] Tile that needs to be skipped is identified by using a sparse mask, thus avoiding empty computation cycles;
[0053] It enables dynamic allocation of computing resources to avoid ineffective computation.
[0054] The Tile-level Dynamic Fixed-Point Quantization (DFQ) mechanism is employed, including:
[0055] Each tile independently calculates quantization parameters (scale and fractional bits);
[0056] Quantization and dequantization operations are performed within the local BRAM, avoiding global re-quantization.
[0057] Different tiles can be quantized with different precision to achieve fine-grained precision control.
[0058] Employing Near-memory Processing Elements (PEs), including:
[0059] DSP (Digital Signal Processor) or LUT (Look-Up Table) resources are directly connected to BRAM / URAM;
[0060] A computational system is formed that incorporates weighted residency and activation flow: tile → local PE → tile output → downstream;
[0061] During the lifecycle of a single tile, the computing unit does not access external storage (DDR).
[0062] Implemented using Softmax local accumulation and LUT, including:
[0063] Softmax's exponentiation (exp) and normalization operations are both performed within a tile;
[0064] It is implemented using lookup tables (LUTs) and accumulators to avoid DDR round-trip;
[0065] Intermediate accumulated values are stored in the on-chip storage area corresponding to the tile.
[0066] In summary, the Transformer model accelerated computation control method provided by this invention is based on the data lifecycle running through a tile, rather than operator-based round-trip DDR. It adopts sparse scheduling and architecture binding, rather than independent optimization. Quantization and computation are completed locally to avoid global weighting. Softmax is executed locally and there is no DDR round-trip.
[0067] like Figure 1 As shown, in an embodiment of the present invention, the hardware platform configuration used by the FPGA system is as follows:
[0068] The FPGA board uses the Xilinx ZCU104 development board, with the FPGA model being Xilinx Zynq UltraScale+XCZU7EV-2FFVC1156. Key resources include:
[0069] DSPSlices: Approximately 1500-1600;
[0070] BRAM: Approximately 500-600 (each 36Kb);
[0071] URAM: Approximately 60-80 (each 288Kb; ZCU104 only has 86 URAMs, so BRAMs need to be used in combination).
[0072] LUTs: Approximately 200K-300K;
[0073] FF: Approximately 400K-600K units;
[0074] The installed external storage is:
[0075] DDR4 SDRAM: 4GB, used to store model weights and input data;
[0076] AXIDMA: Used for high-speed data transfer between DDR and FPGA.
[0077] Figure 1 This demonstrates the data flow path and the connection relationships of the main functional modules in the FPGA system, and has the following characteristics:
[0078] The computing area is close to the storage area: the close-range computing unit (3) is directly connected to the Tile buffer module (2) to realize close-range computing and reduce data migration delay;
[0079] Data flows in one direction: data flows from external storage (1) → Tile buffer (2) → nearby computing unit (3) → Softmax module (4) → output buffer (5) → downstream processing (6), avoiding round trips to external storage;
[0080] High-speed on-chip interconnect: Each module is connected through a high-speed interconnect within the FPGA, resulting in low latency and high bandwidth;
[0081] Double buffering mechanism: The Tile buffer module adopts a double buffering design to achieve pipelined parallelism of computation and data loading;
[0082] The functions of each control module in the diagram are as follows:
[0083] External storage module (DDR) is used to store model weights and input data;
[0084] The Tile buffer module includes double buffer 1 and double buffer 2, which are implemented using BRAM / URAM.
[0085] The Near-memory PE consists of multiple parallel processing units (PE1, PE2, PE3, PE4), and each PE contains DSP resources;
[0086] The Softmax module includes an ExplUT (lookup table), an accumulator tree, and a divider;
[0087] The output tile buffer is implemented using BRAM / URAM;
[0088] Downstream processing modules are used for processing in the next attention layer or FFN (feedforward neural network);
[0089] The meanings of each data flow path are as follows:
[0090] Path A: External storage → Tile buffer module (via AXIDMA interface);
[0091] Path B: Tile buffer module → Near-field computing unit (via on-chip high-speed interconnect);
[0092] Path C: Near-field computation unit → Softmax module;
[0093] Path D: Softmax module → Output Tile buffer;
[0094] Path E: Output Tile buffer → Downstream processing module.
[0095] Figure 4 This is a detailed structural diagram of the near-field computing unit (PE) used in this invention. The diagram shows the internal structure of the DSP array, BRAM connections, and pipeline design, wherein:
[0096] Module 1 (BRAM / URAM interface): Directly connects to on-chip storage (BRAM or URAM) to store Q, K, and V tile data, including TileQBuffer, TileKBuffer, and TileVBuffer;
[0097] Module 2 (Quantization Module, TileDFQ): Performs dynamic fixed-point quantization on each tile, independently calculating quantization parameters (scale and fractional bits), including two parallel quantization sub-modules;
[0098] Module 3 (DSP array): Performs matrix multiplication and accumulation operations, including multiple parallel-working DSPs (DSP1, DSP2, DSP3, DSP4) to achieve high-parallelism computing;
[0099] Module 4 (Accumulator Tree): Accumulates the intermediate results of the DSP array, using a tree structure to achieve efficient accumulation;
[0100] Module 5 (Output Buffer): Stores the calculation results, implemented using BRAM / URAM, and prepares them for transmission to the Softmax module;
[0101] Module 6 (Control Logic): Schedules the computation process, controls the data flow and computation timing, and controls the operation of each module through control signals (dashed lines).
[0102] The meanings of each data flow path are as follows:
[0103] Data path 1: BRAM / URAM interface → quantization module (TileQ, K, V data streams);
[0104] Data path 2: Quantization module → DSP array (quantized data stream);
[0105] Data path 3: DSP array → accumulator tree (matrix multiplication and accumulation result stream);
[0106] Data path 4: Accumulator tree → Output buffer (final calculation result stream);
[0107] Control path: Control logic → Functional modules (control signals, represented by dashed lines);
[0108] Near-range calculations have the following characteristics:
[0109] Proximity connection (marked between modules 1 and 3): The DSP array is directly connected to the BRAM / URAM, which has low latency and high bandwidth, enabling proximity computing;
[0110] Pipeline design (marked on the data path): Quantization, computation, and accumulation pipeline execution improve system throughput;
[0111] Local computation (labeled on the overall structure): All computations are performed on-chip within the lifecycle of a single tile, without accessing external storage (DDR).
[0112] Figure 5 This is a schematic diagram of the hardware implementation structure of the Softmax module used in this invention. The diagram shows the complete calculation process, including the LUT lookup table and the accumulator, wherein:
[0113] Module 1 (Input Tile Buffer): Stores the attention score matrix from the Proximal Computation Unit (PE), including multiple score values such as Score1, Score2 to ScoreN;
[0114] Module 2 (ExpLUT, lookup table): Implements the exponentiation operation exp(x), avoiding complex floating-point operations through a lookup table, including multiple ExpLUTs (ExpLUT1, ExpLUT2) that work in parallel.
[0115] Module 3 (Accumulator Tree): Accumulates all exp values and calculates the global sum sum(exp(x_j)). It uses a tree structure to achieve efficient accumulation and includes multiple accumulators (Acc1, Acc2, Acc3, Acc4).
[0116] Module 4 (Global Accumulator): Accumulates the results of all branches to obtain the final global sum, which is used for normalization calculation;
[0117] Module 5 (Divider): Performs normalized division, calculates exp(x_i) / sum(exp(x_j)), and implements Softmax normalization;
[0118] Module 6 (Output Buffer): Stores the normalized attention weights, implemented using BRAM / URAM, ready to be passed to the Value weighting module;
[0119] Module 7 (Control Logic): Schedules the Softmax calculation process, controls the data flow and calculation timing, and controls the operation of each module through control signals (dashed lines).
[0120] The meanings of each data flow path are as follows:
[0121] Data path 1: Input Tile buffer → ExpLUT (Attention Score Data Stream);
[0122] Data path 2: ExpLUT → Accumulator tree (exponent calculation result stream);
[0123] Data path 3: Accumulator tree → Global accumulator (partial accumulation result stream);
[0124] Data path 4: Global accumulator → Divider (global and result stream);
[0125] Data path 5: ExpLUT → Divider (exponential value stream, used for normalization);
[0126] Data path 6: Divider → Output buffer (normalized attention weight flow);
[0127] Control path: Control logic → Functional modules (control signals, represented by dashed lines).
[0128] The Softmax module has the following characteristics when processing data:
[0129] Local execution (marked on the overall structure): All calculations are completed within the tile, and intermediate results are stored in on-chip storage (BRAM / URAM), without accessing external storage (DDR).
[0130] LUT implementation (labeled on module 2): Use lookup tables to implement exponentiation, avoid complex floating-point operations, improve computational efficiency and hardware implementation feasibility;
[0131] Pipeline design (marked on the data path): exp calculation, accumulation, and normalization pipeline execution improve system throughput.
[0132] like Figure 2 As shown, the Tile design parameters used in this invention are as follows:
[0133] Tile size: The Q / K / V block size specification is selected as 64×64. The 64×64 tile can make full use of the storage capacity of BRAM / URAM, match the computing power of the DSP array, and balance the computing parallelism and storage overhead.
[0134] A double buffering mechanism is adopted: each tile uses two BRAM / URAM blocks to implement double buffering. The working mode is: one buffer block is used for the calculation of the current tile, and the other buffer block is used for the preloading of the next tile, realizing the pipeline parallelism of calculation and data loading.
[0135] Figure 2 This demonstrates the transition relationships and conditions between the various states of a single tile throughout its complete lifecycle from loading to output, including:
[0136] State 1 (IDLE): The system is in an idle state, waiting for requests to be processed;
[0137] State 2 (LOAD): Load tile data from external memory (DDR) to on-chip memory (BRAM / URAM);
[0138] State 3 (QUANTIZE): Perform dynamic quantization on the tile, and calculate the quantization parameters (scale and fractional bits).
[0139] State 4 (COMPUTE): Perform matrix multiplication and accumulation (Q·K^T) to generate the attention score matrix;
[0140] State 5 (SKIP): When the sparse mask is 0, the calculation of this tile is skipped, and no calculation cycle is generated;
[0141] State 6 (SOFTMAXACCUMULATE): Performs Softmax normalization, including exponentiation (exp) and normalization;
[0142] Status 7 (WRITENEXT): Write the calculation results to the downstream processing module.
[0143] The conditions for each state transition are:
[0144] Transition condition 1: State 1 → State 2, triggered by receiving a processing request;
[0145] Transition condition 2: State 2 → State 3, triggered by data loading completion;
[0146] Transition condition 3: State 3 → State 4, triggered by the completion of quantization parameter calculation;
[0147] Transition condition 4a: State 4 → State 5, triggered by a sparse mask of 0 (invalid tile);
[0148] Transition condition 4b: State 4 → State 6, triggered by a sparse mask of 1 (valid tile).
[0149] Transition condition 5: State 5 → State 7, triggered by the completion of the skip operation;
[0150] Transition condition 6: State 6 → State 7, triggered by the completion of Softmax normalization;
[0151] Transition condition 7: State 7 → State 1, triggered by the completion of downstream processing and preparation for the next tile.
[0152] like Figure 3 As shown, the sparse scheduling parameters of this invention meet the following requirements:
[0153] Row sparsity threshold satisfies: Top-K selection or fixed threshold, where:
[0154] Top-K: Selects the K largest values in each row (e.g., K=32);
[0155] Fixed threshold: Rows / columns below the threshold are skipped;
[0156] Depending on the model characteristics, the sparsity is usually between 30% and 70%.
[0157] Load balancing satisfies:
[0158] Number of PEs: Based on FPGA resources, configure 4 parallel PEs (standard configuration for codecs).
[0159] Scheduling strategy: Dynamically allocate tiles to each PE based on sparse mask to ensure load balancing.
[0160] The quantization parameter accuracy meets the following requirements:
[0161] Activation value: INT8 (8-bit integer);
[0162] Weight: INT6 (6-bit integer);
[0163] Selection criteria:
[0164] The INT8 activation value strikes a good balance between accuracy and efficiency.
[0165] INT6 weights can further compress the model size while maintaining an acceptable loss of accuracy.
[0166] Tile-level dynamic quantization:
[0167] Quantization parameter calculation: The scale and zero-point are calculated independently for each tile;
[0168] The quantization formula is: quantized_value = round(float_value / scale) + zero_point;
[0169] The inverse quantization formula is: float_value = (quantized_value - zero_point) * scale.
[0170] Figure 3 The tile jump scheduling process from sparse mask generation to result update is demonstrated, including:
[0171] Step 1 (Mask Generation): Based on the attention score matrix, generate a sparse mask using Top-K selection or a fixed threshold to identify valid and invalid tiles;
[0172] Step 2 (TileIndex Identification): Identify the tile indices that need to be skipped based on the sparse mask and build a list of valid tiles;
[0173] Step 3 (PE scheduling): Dynamically allocate valid tiles to available processing units (PEs) to achieve load balancing scheduling;
[0174] Judgment steps: Determine the validity of the tile, and select different processing paths based on the judgment result;
[0175] Step 4a (Skip Calculation): For invalid tiles, perform a skip calculation operation to skip the calculation of that tile and not generate a calculation cycle;
[0176] Step 4b (Normal Calculation): For valid tiles, perform normal matrix multiplication and accumulation calculation;
[0177] Step 5 (Result Update): Update the calculation results and scheduling status to prepare for the next scheduling cycle;
[0178] Loop check: Determine if there are still tiles to be processed. If so, return to step 1 to continue scheduling.
[0179] The tile jump scheduling process has the following characteristics:
[0180] Dynamic load balancing (marked in step 3): Dynamically allocate computing resources to each PE based on sparsity to ensure that the processing load of each PE is balanced.
[0181] Zero-overhead skip calculation (marked in step 4a): Skipping the operation does not generate a calculation cycle and directly enters the next valid tile, improving calculation efficiency;
[0182] Multi-PE parallel processing (marked in step 4b): Supports multiple PEs to process different tiles in parallel, improving system throughput.
[0183] The data stream implementation in this embodiment of the invention includes the following steps:
[0184] Step 1: Load tile in DDR:
[0185] Load Q, K, and V tile data from DDR via AXIDMA;
[0186] Data is written to the first buffer block of the BRAM / URAM double buffer;
[0187] Step 2: Calculate Q·K locally T :
[0188] Perform matrix multiplication of Q and K in near-range PE;
[0189] The results are stored in BRAM / URAM and not written back to DDR;
[0190] Step 3: LUTSoftmax:
[0191] Use a lookup table to perform exponentiation;
[0192] Normalization is performed within the tile, and intermediate results are stored on-chip.
[0193] Step 4: Weighted output of Value:
[0194] The V matrix is weighted and summed using the Softmax result;
[0195] The results are stored in the output tile buffer;
[0196] Step 5: Write the Tile output downstream of the pipeline:
[0197] Write the output tile to the downstream processing module (such as the next attention layer or FFN).
[0198] At the same time, the next tile is preloaded into the second buffer block of the double buffer.
[0199] The assembly line is designed as follows:
[0200] Production line stages: 5-7;
[0201] Pipeline parallelism: Multiple tiles are processed simultaneously at different stages;
[0202] Throughput: The computation of multiple tiles can be processed per clock cycle.
[0203] Performance benefits include: Through the tile local resident mechanism, the entire calculation process is completed in on-chip storage, which can significantly reduce the number of DDR accesses, reduce the frequency of external storage access and bandwidth utilization;
[0204] Reducing data round-trip overhead can effectively reduce end-to-end inference latency, shorten single-tile processing time, and improve system response speed;
[0205] Reducing data migration power consumption can effectively lower overall power consumption and DDR access power consumption, thereby improving the system's energy efficiency ratio.
[0206] By using tile-level dynamic quantization, the loss of model accuracy can be controlled within an acceptable range. Quantization error can be effectively controlled through tile-level dynamic quantization, achieving a balance between accuracy and efficiency.
[0207] The near-memory computing acceleration method provided by this invention can be applied in the following scenarios:
[0208] (1) Target detection:
[0209] Applied to the RT-DETR-R18 model, the model structure adopts a ResNet-18 backbone network and a Transformer encoder-decoder, with an input size of 640×640 pixels. The sequence length is determined according to the input image size, and the inference speed can meet the requirements of real-time processing. The mAP (mean accuracy) is comparable to that of the floating-point model.
[0210] (2) Image recognition:
[0211] When applied to models such as Vision Transformer (ViT) and DeiT, it can significantly improve inference speed when the input is 224×224 or 384×384 pixels, while keeping the accuracy loss within an acceptable range.
[0212] (3) Visual reasoning:
[0213] Based on the multimodal Transformer model, end-to-end latency can be effectively reduced when inputting images and text sequences.
[0214] This invention employs a combination of tile-based resident near-computation structure, sparse scheduling, and local Softmax to achieve data lifecycle throughout the entire tile, rather than operator-based round-trip DDR. Sparse scheduling is bound to the architecture for hardware-level optimization. Quantization and computation are performed locally to avoid global weighting. Softmax is used for local execution, eliminating DDR round-trip. This invention is applicable to Transformer models with multi-head attention mechanisms, covering various application scenarios such as object detection, image recognition, and visual reasoning. It adopts a data flow path of DDR→Tile buffer→near-distance PE→Softmax→output, and the technical solution is universal and not limited to specific models.
[0215] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for accelerating computational control of on-chip memory cells executing Transformer models, characterized in that: The calculation and control steps include the following: Step 1: Divide the tensors of the storage cells on the input chip into tiles according to a preset dimension; Step 2: Load each tile into the on-chip storage unit and reside there; Step 3: In a computing array that is closely coupled to the on-chip storage cells, perform matrix multiplication and accumulation processing on the tile; Step 4: Skip the calculation of the corresponding tile according to the row or column sparse mask. The sparse mask scheduling module performs head reorganization and load balancing scheduling. The sparse mask scheduling module generates sparse masks through Top-K selection or fixed thresholds, and skips the entire tile for low contribution rows or columns without generating empty calculation cycles. Step 5: Perform dynamic quantization and dequantization within the tile. During dynamic quantization, each tile independently calculates the quantization parameters scale and fractional bits. The quantization and dequantization operations are completed within the local BRAM. Step 6: Store the intermediate results of the tile in the on-chip storage unit until the calculation is complete.
2. The accelerated computation control method for executing Transformer models in on-chip memory units according to claim 1, characterized in that: In step one, the memory cell on the chip adopts a double buffering mechanism, including a first buffer block and a second buffer block. The first buffer block is used for the calculation of the current tile, and the second buffer block is used for the preloading of the next tile.
3. The accelerated computation control method for executing Transformer models in on-chip memory units according to claim 1, characterized in that: In step three, the computing array includes a DSP / LUT array coupled to BRAM or URAM, which does not access external storage during the lifetime of a single tile.
4. An accelerated computing control system for executing Transformer models in on-chip memory units, characterized in that: include: The on-chip memory cells are used to store tile data; A computational array, closely coupled to the storage unit, is used to perform matrix multiplication and accumulation of tiles; The sparse scheduling module is used to skip the calculation of the corresponding tile based on the sparse mask; The quantization module is used to perform dynamic quantization and dequantization within the tile; The control module is used to retain the intermediate results of the tile in on-chip storage until the calculation is completed; The Softmax module performs exponent calculation and normalization through lookup tables and accumulators, with intermediate accumulated values stored in the on-chip storage area corresponding to the tile.