Hardware-friendly Transform column balance pruning model compression and efficient deployment method
By employing a hardware-software co-optimization approach and utilizing technologies such as column balancing pruning algorithms and lightweight storage formats, the deployment challenges of the Transformer model on edge devices were solved, achieving efficient deployment and improved computational efficiency.
Patent Information
- Application Number
- CN202511691136.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-18
- Publication Date
- 2026-02-24
AI Technical Summary
The large number of parameters in the Transformer model makes it difficult to deploy on resource-constrained edge devices. Existing pruning strategies are incompatible with hardware architecture, resulting in unbalanced computational load and low operating efficiency.
A hardware-software co-optimization approach is adopted, which reduces the number of parameters and computational complexity through a column-balanced structured pruning algorithm. Combined with a lightweight parameter storage format, a computational data buffer, a systolic array computation block, a vector computation unit, a nonlinear operator unit, and a data flow controller, a hardware-friendly Transformer model deployment is achieved.
It enables efficient deployment of the Transformer model on resource-constrained devices, reduces parameter storage pressure, ensures load balancing, and improves hardware computing efficiency.
Smart Images
Figure CN121562702A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of digital integrated circuits, electronic information and deep learning technologies, and in particular to a Transformer software-hardware co-optimization method based on structured pruning. Background Technology
[0002] In recent years, with the development of Natural Language Processing (NLP), Transformer architectures based on pre-trained models, such as BERT (Bidirectional Encoder Representations from Transformers), have become the benchmark for various language understanding tasks. While these models excel in accuracy, their high performance is accompanied by an exponential increase in the number of model parameters and a doubling of computational complexity, making them difficult to deploy on resource-constrained edge devices. To address these issues, a common approach is to compress the model using pruning. However, traditional pruning algorithms lack suitable hardware architectures, leading to unbalanced workloads on processing units and reduced efficiency. Therefore, we propose a hardware-software co-optimization method to fully leverage the reduction in parameter size and computational complexity after pruning, thereby achieving efficient deployment of Transformer models. Summary of the Invention
[0003] The purpose of this invention is to address the problems of Transformer models having a large number of parameters, making them difficult to deploy on resource-constrained edge devices, and the incompatibility between existing pruning strategies and hardware architectures, which lead to load imbalance and low operating efficiency due to the failure to fully utilize the sparsity after pruning during the computation process. This invention proposes a software and hardware co-optimization method to fully utilize the sparsity after pruning, thereby achieving efficient deployment of Transformer models.
[0004] To achieve the above-mentioned technical objectives, the technical solution implemented by this invention is as follows:
[0005] A hardware-friendly Transformer column-balanced pruning model compression and efficient deployment method is characterized by comprising: a model compression algorithm, a lightweight parameter storage format, a computational data buffer, a systolic array computation block, a vector computation unit, a nonlinear operator unit, a data flow controller, and a DMA unit.
[0006] The model compression algorithm reduces the number of model parameters and computational complexity by using a matrix-block-based column-balanced structured pruning algorithm in a loop to generate regular sparsity.
[0007] The lightweight parameter storage format includes single instruction multiple data stream storage and mask-encoded storage. Single instruction multiple data stream storage is used for activation parameters and sparse parameters, while mask-encoded storage is used for sparse parameter indexes to optimize the parameter storage format and further reduce the storage pressure of parameter indexes.
[0008] The computational data buffer consists of RAM and includes an input data buffer, a BIAS buffer, an intermediate data buffer, and a result data buffer.
[0009] The pulsating array operation block includes There are PE units, where m is the number of rows and n is the number of columns. Each PE unit consists of a compression parameter decoder, a MAC, and registers. The decoder consists of two decoding modules used in sparse matrix multiplication mode. The MAC consists of a multiplier and an accumulator.
[0010] The vector operation unit includes a Bias addition calculation module, a maximum value calculation module, an average value calculation module, and a data shaper.
[0011] The nonlinear operator unit includes a Softmax calculation unit and a LayerNorm calculation unit, which are used for nonlinear calculations during the calculation process.
[0012] The dataflow controller includes a pulsating array input controller, a pulsating array output controller, and a DMA controller. It is used to flexibly schedule the input of computational data into the pulsating array computation blocks, vector computation units, or nonlinear operator units in the data buffer, and to ensure that control data is correctly stored in the RAM of the data buffer.
[0013] The DMA unit, designed based on the AXI_FULL interface protocol, is used for data transfer between off-chip DDR and on-chip cache.
[0014] Optionally, the vector operation unit can be configured according to the calculation process to directly output the calculated data, add the calculated data to Bias and then output it, or calculate the maximum value of each row of calculated data or calculate the average value of each row of calculated data.
[0015] Optionally, the nonlinear operator unit can be configured to perform Softmax or LayerNorm operations.
[0016] The advantages and beneficial effects of the technical solution adopted in this invention are as follows:
[0017] This technology achieves optimized Transformer model deployment through a hardware-software co-design, reducing parameter storage pressure, ensuring balanced workload, and improving hardware computing efficiency. Its core principle is "algorithm-optimized hardware, hardware-adapted algorithm." It uses a lightweight Transformer model algorithm to guide the design of dedicated hardware operators, and the hardware operators are adapted to the lightweight algorithm's computation through a hardware-software co-optimization architecture, thereby enabling efficient Transformer model deployment. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the structure of the present invention;
[0019] Figure 2 A schematic diagram of the balanced pruning process;
[0020] Figure 3 A schematic diagram illustrating single instruction multiple data stream storage, mask encoding storage, and decoding rules;
[0021] Figure 4 This is a schematic diagram of the structure of the pulsating array operation block and PE; Detailed Implementation
[0022] like Figure 1 The structure shown in this embodiment relates to a hardware-friendly Transformer column-balanced pruning model compression and efficient deployment method, including: a model compression algorithm, a lightweight parameter storage format, a computational data buffer, a systolic array computation block, a vector computation unit, a nonlinear operator unit, a data flow controller, and a DMA unit.
[0023] The model compression algorithm process is as follows: Figure 2 As shown, during the pruning process, the element with the smallest absolute value in each column of the block is used as the pruning target. A training-pruning-retraining loop is used to gradually increase the sparsity of the model, avoiding the significant performance drop caused by pruning a large number of weights at once. Furthermore, during iterative pruning when the pruning rate is less than 50%, only two parameters out of four adjacent parameters are removed to ensure structured sparse parameters after pruning. The specific steps are as follows:
[0024] Step 1: Determine the initial matrix block size. (where m is a multiple of 4, initially set to 4), using the densely pre-trained fine-tuned model as the initial optimal model for pruning and generating a model parameter mask bitmap (freezing the pruned parameters during training).
[0025] Step 2: Set a model accuracy threshold and train the model. Keep all models that exceed the threshold and select the model with the highest accuracy as the optimal model. Repeat steps 1 and 2.
[0026] Step 3: If no model has an accuracy exceeding the threshold after training, increase the parameter block size and repeat steps 1 and 2. Terminate pruning-training when the matrix block size is 32×n.
[0027] The lightweight parameter storage format is as follows: Figure 3 As shown. This method is used in a lightweight model with a pruning rate of 50%. Figure 3 (a) is a single-instruction multiple-data stream storage method, which concatenates the non-zero bits of four adjacent column vector elements into a single element for storage, achieving a compact transformation from vector storage to scalar storage. Similarly, for dense input parameters, the bits of four adjacent row vector elements are concatenated into a single element for storage. Figure 3 (b) For mask encoding storage, there are only 6 forms of adjacent column vector mask arrangements of length 4 in each block. Each form is encoded into 1-6 according to its characteristics. The encoding of the 6 forms can be completed using 3-bit index, and decoding can be completed with simple shifting and addition. Figure 3 (c) represents the decoding rules for each encoding.
[0028] The computational data buffer is responsible for storing Input, Weight, and Index data in the Input data buffer, storing Bias data in the Bias buffer, storing the computation results of the vector operation unit and the nonlinear operator unit in the intermediate data buffer, and storing the final computation result in the result data buffer.
[0029] The structure of the pulse array operation block is as follows: Figure 4 As shown, including There are PE units. Here, m is the number of rows and n is the number of columns. Each PE unit supports two modes: sparse matrix multiplication and dense matrix multiplication. It controls whether the PE performs sparse matrix multiplication (cmd=0) or dense matrix multiplication (cmd=1) via the cmd signal. The Decoder is responsible for decoding the index and compressed weights into a mask and sparse weights, using the index and compressed weights as input. The index is decoded into a mask through shifting or shifting and adding, while the weights are decoded into sparse weights through bit concatenation based on the current index. The MAC module is responsible for performing multiplication and accumulation operations. Each MAC module consists of four multipliers and seven adders. Each multiplier is responsible for inputting the multiplication of the corresponding bits of the weights. Its enable signal is controlled by the mask generated by the Decoder and the cmd signal. The register module is responsible for storing the input data and passing it to the next PE.
[0030] The vector operation unit performs bias addition, maximum value calculation, and average value calculation on the output data of the systolic array according to the calculation process, and finally outputs the data after passing through a data shaper. The data shaper is responsible for shaping the completed data into the same format as the Single Instruction Multiple Data Stream (SID) storage for subsequent use in the systolic array operation block. The calculation functions executed in different processes are as follows: The systolic array operation module executes... After dense matrix multiplication, the computational data from the vector operation unit is directly output to the buffer; after the systolic array operation module performs sparse matrix multiplication, such as... The computational data from the vector operation unit will be accumulated with Bias and written to the cache; the pulsating array operation module executes... After dense matrix multiplication, the vector operation unit calculates the maximum value of each row of calculated data and writes it to the cache; the systolic array operation module then executes...
[0031] After sparse matrix multiplication, the computational data of the vector operation unit is accumulated twice by Bias and the average value of each row is calculated before being written to the cache.
[0032] The nonlinear operator unit consists of the Softmax calculation unit and the LayerNorm calculation unit, which perform different functional calculations on the input data according to the calculation process. The Softmax calculation unit is composed of an adder and a shifter. The serial input data is converted into a base-2 exponential operation by the arithmetic block, and finally outputs a normalized probability value. The LayerNorm calculation unit consists of two sequential calculation modules. The first module is responsible for calculation... and The first module is responsible for calculating the square root and reciprocal operations in the denominator; the second module is responsible for calculating the final result. .
[0033] The dataflow controller comprises a systolic array input controller, a systolic array output controller, and a DMA controller, all composed of a Moore's state machine. The systolic array input controller is responsible for controlling the reading of computational data from the input data buffer and inputting it into the systolic array computation block. The systolic array output controller is responsible for configuring the vector computation unit functions according to the computation flow and controlling its computation results to be stored in the intermediate data buffer. The DMA controller is responsible for controlling the DMA unit to move external DDR data and store it in the input data buffer to update the input data buffer data for the next computation, as well as moving the final computation result to external DDR for storage.
[0034] The DMA unit is responsible for moving the Input, Weight, Index, and Bias required for the operation from the off-chip DDR to the input data buffer and the Bias buffer, and for moving the calculation results from the result buffer to the off-chip DDR.
Claims
1. A hardware-friendly method for compressing and efficiently deploying a Transformer column-balanced pruning model, characterized in that, include: Model compression algorithm, lightweight parameter storage format, computational data buffer, systolic array computation block, vector operation unit, nonlinear operator unit, data flow controller, DMA unit; The model compression algorithm is used to reduce the number of model parameters and computational complexity, and to generate regular sparsity of weight parameters. The lightweight parameter storage format includes a single instruction multiple data stream storage format and a mask-encoded storage format, which are used to optimize the model parameter storage format and reduce the storage pressure of sparse parameter indexes. The computational data buffer is composed of RAM and includes an input data buffer, a bias buffer, an intermediate data buffer, and an output data buffer, used to store Input, Weight, Index, Bias, intermediate calculation results, and final calculation results. The pulsating array operation block includes There are PE units, where m is the number of rows and n is the number of columns. Each PE unit consists of a compression parameter decoder, a MAC, and a register. The decoder consists of two decoding modules and is used in sparse matrix multiplication mode. The MAC consists of a multiplier and an accumulator. The vector operation unit includes a Bias addition calculation module, a maximum value calculation module, an average value calculation module, and a data shaper, which are used to perform various operations on the output results of the pulsating array; The nonlinear operator unit includes a Softmax calculation module and a LayerNorm calculation module, which are used to perform nonlinear operations on the input data; The data flow controller includes a systolic array input controller, a systolic array output controller, and a DMA controller, which are used to flexibly schedule the computational data in the data buffer to input it to the systolic array operation block, vector operation unit, or nonlinear operator unit, and control the data to be correctly stored in the RAM of the data buffer. The DMA unit, designed based on the AXI_FULL interface protocol, is used for data transfer between off-chip DDR and on-chip cache.
2. The hardware-friendly Transformer column-balanced pruning model compression and efficient deployment method according to claim 1, characterized in that, The model compression algorithm uses a matrix-blocked column-balanced structured pruning strategy to perform iterative pruning operations, reducing the number of model parameters and computational complexity, and making the weight parameters regular and sparse.
3. The hardware-friendly Transformer column-balanced pruning model compression and efficient deployment method according to claim 1, characterized in that, The lightweight parameter storage format achieves compact compression of activation parameters and sparse parameters based on the single instruction multiple data stream storage format, converts vector storage to scalar storage, performs sparse parameter index compression based on mask encoding storage format to reduce its storage pressure, and performs decoding by shifting or shifting and adding one according to different encoding cases.
4. The hardware-friendly Transformer column-balanced pruning model compression and efficient deployment method according to claim 1, characterized in that, The computational data buffer stores different data input into it into corresponding internal buffers. Specifically, Input, Weight, and Index are stored in the input data buffer, Bias is stored in the Bias buffer, intermediate calculation results are stored in the intermediate data buffer, and the final calculation result is stored in the output data buffer.
5. The hardware-friendly Transformer column-balanced pruning model compression and efficient deployment method according to claim 1, characterized in that, The pulsating array arithmetic block performs matrix multiplication on the input data. During dense matrix multiplication, the input data is the data in the intermediate result buffer, and the PE unit is in dense matrix multiplication mode, with the decoder acting as a register. During sparse matrix multiplication, the input data is the data in the input data buffer, and the PE unit is in sparse matrix multiplication mode, with the decoder acting as a sparse parameter decoder. During the operation, the PE unit controls the bubble-free switching between dense and sparse matrix multiplication modes via the cmd signal.
6. The hardware-friendly Transformer column-balanced pruning model compression and efficient deployment method according to claim 1, characterized in that, The vector operation unit supports addition calculation with or without bias, maximum value calculation, average value calculation, and data shaping functions. The data shaper shapes the calculation results into a single instruction multiple data stream storage format for output.
7. The hardware-friendly Transformer column-balanced pruning model compression and efficient deployment method according to claim 1, characterized in that, The nonlinear operator unit performs Softmax or LayerNorm operations on the input data.
8. The hardware-friendly Transformer column-balanced pruning model compression and efficient deployment method according to claim 1, characterized in that, The data flow controller is controlled by a pulse array input controller to control the data reading process in the input data buffer and input it to the pulse array arithmetic block for calculation; the pulse array output controller is used to configure the functional mode of the vector calculation unit and control its output results to be stored in the intermediate data buffer. The DMA controller controls the DMA unit to move data from the off-chip DDR to the input data buffer and the bias buffer, and moves the data from the result buffer to the off-chip DDR for storage.