Transform hardware acceleration method and accelerator based on hybrid precision quantization and huffman coding
Patent Information
- Application Number
- CN202610414223.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-31
- Publication Date
- 2026-06-26
AI Technical Summary
When the Swin Transformer model is deployed on a resource-constrained hardware platform, the computational burden and storage bandwidth requirements are high. Existing mixed-precision quantization methods have large computational overhead and do not fully incorporate structured or entropy coding, resulting in limited compression ratios.
A genetic algorithm is used to obtain a mixed-precision quantization configuration scheme, and Huffman coding is combined to perform lossless variable-length encoding on the quantized model. Model compression is achieved through a quantization configuration prediction network and a Huffman encoder.
The search speed of the quantization configuration scheme was improved, the computational overhead was reduced, and the model was further compressed while maintaining the accuracy, thus achieving a balance between compression efficiency and inference speed.
Smart Images
Figure CN122287750A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and integrated circuits, and specifically relates to a Transformer hardware acceleration method and accelerator based on mixed precision quantization and Huffman coding. Background Technology
[0002] The Swin Transformer model incurs a significant computational burden due to its numerous matrix multiplication and addition operations, attention weight calculations, and feedforward networks (FFNs). Furthermore, its multi-layered stacked Transformer modules and high-dimensional feature representations place high demands on storage bandwidth and energy consumption. These factors make the deployment of Swin Transformer on resource-constrained hardware platforms (such as FPGAs, ASICs, and edge devices) challenging.
[0003] In the Swing Transformer, components such as in-window and cross-window attention mechanisms, multilayer perceptrons, layer normalization, and cross-stage feature fusion each have their own characteristics in terms of computation mode and numerical dynamic range.
[0004] While mixed-precision quantization (MMQ) can allocate appropriate bit widths to different network layers in the Swing Transformer, finely adapting to the varying tolerances of quantization errors across layers and concentrating limited bit resources on precision-sensitive modules to achieve a better precision-efficiency tradeoff, existing MMQ methods largely rely on search strategies for bit width configuration. This typically requires evaluating a large number of candidate configurations, resulting in high computational overhead and low search efficiency, making it difficult to apply efficiently to large-scale models. Furthermore, most existing MMQ schemes only use quantization as a model compression method, without fully integrating further compression techniques such as structured encoding or entropy coding, thus limiting the overall model compression rate. Summary of the Invention
[0005] This invention provides a Transformer hardware acceleration method and accelerator based on mixed-precision quantization and Huffman coding. It uses neural networks to intelligently obtain a mixed-precision quantization configuration scheme according to the actual target, thereby improving the search speed of the quantization configuration scheme and reducing the computational overhead. Moreover, it uses Huffman coding to perform lossless variable-length encoding on the quantized model, further compressing the model while maintaining the same precision.
[0006] To achieve the above technical objectives, the present invention adopts the following technical solution:
[0007] A hardware acceleration method for Transformer based on mixed-precision quantization and Huffman coding includes:
[0008] S1, for the Transformer network layer, uses a genetic algorithm to obtain several configuration schemes for mixed-precision quantization;
[0009] S2, perform mixed precision quantization on each network layer of Transformer based on each quantization configuration scheme, and obtain a KL divergence for each quantization configuration scheme;
[0010] S3, using the quantized configuration scheme and the corresponding KL divergence as the output label and input features respectively, train a multilayer perceptron to obtain the quantized configuration prediction network;
[0011] S4: Receive the target KL divergence value set by the user, and use the quantization configuration prediction network to obtain the corresponding quantization configuration scheme;
[0012] S5, based on the quantization configuration scheme obtained in S4, performs mixed-precision quantization on each network layer of the Transformer;
[0013] S6 uses Huffman coding to encode and compress all quantized weights, and then stores the compressed weights off-chip.
[0014] Furthermore, a genetic algorithm is used to obtain several configuration schemes for mixed-precision quantization, specifically including:
[0015] S1.1: Randomly generate N candidate quantization configuration schemes, each quantization configuration scheme defining the quantization bit width of each layer of the Transformer;
[0016] S1.2: For each current quantization configuration scheme, perform mixed precision quantization at each layer of the Transformer, and calculate the KL divergence between the quantized model and the original model on the softmax output of the validation data, which is used as the fitness of the quantization configuration scheme.
[0017] S1.3: Using a tournament selection mechanism, two groups are randomly selected from the current population, each with x members. The quantization scheme with the lowest KL divergence value in each of the two groups is taken as the parent generation. A new generation of quantization schemes is generated through crossover and mutation.
[0018] Specifically, the crossover operation involves swapping the bit widths of different quantization configuration schemes, while the mutation operation involves randomly perturbing the bit widths of any network layer in the quantization configuration scheme.
[0019] S1.4: Repeat step S1.3 for a total of m rounds to obtain m quantized configuration schemes that have undergone multiple generations of evolution;
[0020] S1.5: Repeat steps S1.2-S1.4 for a total of n rounds to finally obtain m×n quantization configuration schemes.
[0021] Furthermore, the Transformer hardware acceleration method also includes step S7: using a two-level lookup table to decode the codewords after weight encoding and compression.
[0022] Furthermore, an improved two-level lookup table is used to decode the weighted, compressed codewords, specifically including:
[0023] Step S7.1: Based on the length distribution of all codes in the Huffman coding table and the characteristics of the common high-order prefix, determine the depth of the first-level lookup table L1, the depth of the second-level lookup table L2, and the length of the window to be decoded.
[0024] Step S7.2: Obtain the Huffman codeword from the Huffman encoded stream according to the length of the window to be decoded, and use it as the current Huffman codeword to be decoded;
[0025] Step S7.3: Decode the current Huffman codeword using a first-level lookup table to obtain the decoded weights, code length, and valid bits. If decoding the current Huffman codeword using the first-level lookup table is invalid, shift the Huffman encoded stream according to the depth of the first-level lookup table to obtain a Huffman codeword as the current Huffman codeword. Then, decode the current Huffman codeword using a second-level lookup table to obtain the decoded weights, code length, and valid bits.
[0026] Step S7.4: Shift the Huffman code stream according to the code length of the decoded codeword to obtain the next Huffman codeword to be decoded, and return to step S7.3 to decode the next Huffman codeword to be decoded.
[0027] Furthermore, the specific method for step S7.1 is as follows:
[0028] First, determine the maximum length of all codes. ,according to Determine the segmentation location: If If it is an integer, then... Determine the dividing position; otherwise, and As candidate segmentation locations;
[0029] Then, if If the value is an integer, then the code length in the Huffman coding table is greater than or equal to... Count the number of bits whose common high bit is 1 for all codewords. and will Determine the depth of the first-level lookup table, and Determine the depth of the second-level lookup table;
[0030] if For non-integer values, among two candidate split positions, the one that results in a smaller total depth of the two-level lookup table is selected as the final split position. The total depth of the two-level lookup table is calculated as follows: for code lengths greater than or equal to... Count the number of bits whose common high bit is 1 for all codewords. The depths of the corresponding first-level and second-level lookup tables are respectively The total depth of the two-level lookup table is ; and for code lengths greater than or equal to in the Huffman coding table Count the number of bits whose common high bit is 1 for all codewords. The depths of the corresponding first-level and second-level lookup tables are respectively The total depth of the two-level lookup table is ;
[0031] Finally, let the depth of the first-level lookup table be... The depth of the second-level lookup table is The length of the window to be decoded is .
[0032] Furthermore, the Transformer specifically refers to the Swing Transformer.
[0033] A Transformer hardware accelerator, deployed on an FPGA, is used to implement a Transformer model, wherein the Transformer model is obtained using the Transformer hardware acceleration method based on mixed-precision quantization and Huffman coding as described above.
[0034] Furthermore, the Transformer accelerator includes: a top-level control module, a PE array, a nonlinear unit, an input buffer, a weight buffer, a KV buffer, and a decoder;
[0035] The top-level control module is used to: control the input and output data of the accelerator and the data buffer;
[0036] The PE array is used to implement matrix operation of the Transformer model;
[0037] The nonlinear unit is used to perform layernorm, softmax, and GELU calculations in the Transformer model;
[0038] The input buffer is used to store the input data of the accelerator, the Q matrix calculated by the PE array, and part of the output of the nonlinear unit.
[0039] The weight buffer is used to retrieve weight data from on-chip memory in parallel and then send it to the decoder.
[0040] The KV buffer is used to cache the K and V matrices generated by the PE array;
[0041] The Huffman decoder is used to perform Huffman decoding on the weight data in the weight buffer.
[0042] Furthermore, the PE array uses a DSP and LUT to jointly complete multiple multiplication operations. Specifically, all MAC units are grouped into pairs, and the two MAC units in each group have a common multiplier, resulting in several dual-MAC calculation groups composed of dual-MAC units and unpaired single-MAC units. Each dual-MAC calculation group is implemented in parallel using a single DSP, while the single-MAC unit is implemented using lookup table logic.
[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0044] (1) The genetic algorithm sampling used in the mixed precision quantization of this invention effectively improves the quality of samples, increases search efficiency, and reduces computational overhead.
[0045] (2) The present invention uses genetic algorithm sampling to improve the mixed precision quantization, which can automatically explore the quantization sensitivity of each layer of SwinTransformer, and perform quantization compression on the model while ensuring accuracy.
[0046] (3) The present invention performs lossless variable-length encoding on the quantized model through Huffman coding, and further compresses the model while maintaining the same accuracy.
[0047] (4) This invention designs a low-overhead, high-throughput Huffman decoder, thereby achieving a true balance between compression efficiency and inference speed while ensuring the model accuracy is intact.
[0048] (5) The Swin Transformer hardware device based on DSP packaging technology implemented in this invention significantly improves DSP utilization and achieves a significant increase in throughput while maintaining the integrity and accuracy of model functions. Attached Figure Description
[0049] Figure 1 This is a flowchart of the Transformer hardware acceleration method described in the embodiments of this application;
[0050] Figure 2 This is a flowchart illustrating how a genetic algorithm is used to obtain a quantitative configuration scheme in an embodiment of this application.
[0051] Figure 3 This is a schematic diagram of Huffman decoding as described in the embodiments of this application;
[0052] Figure 4 This is the overall hardware architecture of the Transformer accelerator described in the embodiments of this application;
[0053] Figure 5 This is a network structure diagram of the Swin Transformer module in the Swin Transformer model;
[0054] Figure 6 This is a schematic diagram of DSP packaging as described in the embodiments of this application. Detailed Implementation
[0055] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.
[0056] Example 1
[0057] This embodiment provides a hardware acceleration method for Transformers based on mixed-precision quantization and Huffman coding, applied to the Swing Transformer model. The aim is to improve model compression and hardware computation efficiency. The framework process is as follows: Figure 1 As shown. Specifically, the model is systematically optimized through the following two key stages: (1) Automated quantization strategy search: Using the Kullback-Leibler (KL) divergence between the quantized model and the original model on the softmax output as the evaluation index, the optimal quantization bit width configuration of each layer of the Swin Transformer model is automatically searched, significantly reducing storage occupation while maintaining model accuracy as much as possible. (2) Further compression using entropy coding: Based on the distribution characteristics of the weights of the quantized network layers, Huffman coding is used to perform lossless compression, further improving the compression ratio of the Swin Transformer model and reducing storage occupation.
[0058] Specifically, the Transformer hardware acceleration method based on mixed-precision quantization and Huffman coding includes:
[0059] S1, for the Transformer network layer, uses a genetic algorithm to obtain several configuration schemes for mixed-precision quantization.
[0060] Among them, several configuration schemes for mixed-precision quantization are obtained using genetic algorithms, as referenced. Figure 2 As shown, it specifically includes:
[0061] S1.1: Randomly generate N candidate quantization configuration schemes, each quantization configuration scheme defining the quantization bit width of each layer of the Transformer;
[0062] S1.2: For each current quantization configuration scheme, perform mixed precision quantization at each layer of the Transformer, and calculate the KL divergence between the quantized model and the original model on the softmax output of the validation data, which is used as the fitness of the quantization configuration scheme.
[0063] S1.3: Using a tournament selection mechanism, two groups are randomly selected from the current population, each with x members. The quantized configuration scheme with the best fitness (lowest KL divergence) in each of these two groups is selected as the parent generation. A new generation of quantized configuration schemes is generated through crossover and mutation.
[0064] Specifically, the crossover operation involves swapping the bit widths of different quantization configuration schemes, while the mutation operation involves randomly perturbing the bit widths of any network layer in the quantization configuration scheme.
[0065] S1.4: Repeat step S1.3 for a total of m rounds to obtain m quantized configuration schemes that have undergone multiple generations of evolution;
[0066] S1.5: Repeat steps S1.2-S1.4 for a total of n rounds to finally obtain m×n diverse and high-quality quantitative configuration schemes, forming a configuration pool.
[0067] S2, perform mixed-precision quantization on each network layer of Transformer based on each quantization configuration scheme, and obtain a KL divergence for each quantization configuration scheme.
[0068] All the quantization configuration schemes obtained in the above process and their corresponding KL divergences are combined to form a supervised learning dataset, where the KL divergence is the input feature and the quantization configuration scheme is the output label.
[0069] S3 uses the quantized configuration scheme and the corresponding KL divergence as the output label and input features, respectively, to train a multilayer perceptron (MLP) to obtain the quantized configuration prediction network. This MLP learns the mapping relationship from the target KL divergence to the specific quantized configuration scheme, thereby replacing the time-consuming evolutionary search and achieving rapid prediction of suitable quantized configuration schemes.
[0070] S4 receives the target KL divergence value set by the user and uses the quantization configuration prediction network to obtain the corresponding quantization configuration scheme.
[0071] S5, based on the quantization configuration scheme obtained in S4, performs layer-by-layer mixed precision quantization on each network layer of the Transformer.
[0072] S6 uses Huffman coding to encode and compress all quantized weights.
[0073] After mixed-precision quantization, the model weights of each Transformer network layer typically exhibit a non-uniform discrete distribution. Utilizing this characteristic, this invention statistically analyzes the frequencies of all quantized weights and constructs a Huffman coding table. The weight values are then replaced with corresponding variable-length Huffman codes. By assigning short codes to high-frequency values and long codes to low-frequency values, further lossless compression of the Transformer model file is achieved, thereby alleviating the pressure on the accelerator's on-chip storage (SRAM) and reducing the amount of data transferred between the accelerator and external memory (such as DRAM).
[0074] S7 uses a two-level lookup table to decode the codewords after the weights have been encoded and compressed.
[0075] In this embodiment, Huffman coding is used to encode the quantization weights, and a corresponding Huffman decoder is used to complete the decoding operation. Traditional methods use a single-level lookup table to perform the decoding operation, which stores all prefixes of the maximum code length L. However, this results in a lookup table with a very large size. When the Huffman coding table is large, it becomes unacceptable for resource-constrained FPGAs. Considering that longer Huffman codes have more leading 1s, this embodiment employs an improved two-level lookup table to decode the weighted, compressed codewords. Referring to Table 1, the specific steps include:
[0076] Step S7.1: Based on the length distribution of all codes in the Huffman coding table and the characteristics of the common high-order prefix, determine the depth of the first-level lookup table L1, the depth of the second-level lookup table L2, and the length of the lookup table address.
[0077] First, determine the maximum length of all codes. ,according to Determine the segmentation location: If If it is an integer, then... Determine the dividing position; otherwise, and As candidate segmentation locations.
[0078] Then, if If the value is an integer, then the code length in the Huffman coding table is greater than or equal to... Count the number of bits whose common high bit is 1 for all codewords. and will Determine the depth of the first-level lookup table, and Determine the depth of the second-level lookup table;
[0079] if For non-integer values, among two candidate split positions, the one that results in a smaller total depth of the two-level lookup table is selected as the final split position. The total depth of the two-level lookup table is calculated as follows: for code lengths greater than or equal to... Count the number of bits whose common high bit is 1 for all codewords. The depths of the corresponding first-level and second-level lookup tables are respectively The total depth of the two-level lookup table is ; and for code lengths greater than or equal to in the Huffman coding table Count the number of bits whose common high bit is 1 for all codewords. The depths of the corresponding first-level and second-level lookup tables are respectively The total depth of the two-level lookup table is .
[0080] Finally, let the depth of the first-level lookup table be... The depth of the second-level lookup table is Then the length of the lookup table address is .
[0081] For example Figure 3 The Huffman coding table has a maximum code length of 10. Starting from code length 6, the first 5 codewords of all subsequent code lengths are all 1s, thus determining the L1 depth to be... L2 is .
[0082] For example, Figure 3 The maximum code length is 11 (corresponding to "0" and "2"). Finding the dividing positions near code length 11 / 2, such as code lengths 5 (corresponding to "12" and "13") and 6 (corresponding to "5"), we can observe that the first three bits of all subsequent code lengths starting from code length 5 are 1. Therefore, we can determine the depth of the first-level lookup table to be... The second-level lookup table is Where 5 is the dividing position, and 8 equals the maximum code length of 11 minus the number of leading 1s of 3. This two-level lookup table is still very large, so the initial verification code length is 6, starting from... Figure 3 We can observe that starting from code length 6, the first 5 bits of all subsequent code lengths are 1. Therefore, we can similarly determine the depth of the first-level lookup table as follows: The second-level lookup table is Compared to the case without pruning, this two-level lookup table... The value has become much smaller. Therefore, the final division position is determined to be 6.
[0083] Step S7.2: Obtain the Huffman codeword from the Huffman encoded stream according to the length of the lookup table address, and use it as the current Huffman codeword to be decoded.
[0084] Step S7.3: Decode the current Huffman codeword using a first-level lookup table to obtain the decoded codeword, its code length, and the valid decoded bits. If decoding the current Huffman codeword using the first-level lookup table is invalid, shift the Huffman encoded stream according to the depth of the first-level lookup table to obtain a Huffman codeword as the current Huffman codeword, and then decode the current Huffman codeword using a second-level lookup table to obtain the decoded codeword, its code length, and the valid decoded bits.
[0085] Step S7.4: Shift the Huffman code stream according to the code length of the decoded codeword to obtain the next Huffman codeword to be decoded, and return to step S7.3 to decode the next Huffman codeword to be decoded.
[0086] For example, according to the Huffman coding table in Table 1, addresses 00000 to 00111 in the first-level lookup table L1 are decoded to obtain a0101, where 'a' is the codeword of the quantization weight, 010 is the code length of the quantization weight, and '1' is the valid decoded bit. As another example, addresses 01000 to 01111 in the first-level lookup table L1 are decoded to obtain b0101. Furthermore, addresses 11000 to 11111 in the second-level lookup table L2 are decoded to obtain f0101, and address 00010 is decoded to obtain g1011.
[0087] Example 2
[0088] This embodiment provides a Transformer hardware accelerator deployed on an FPGA to implement a Transformer model. The Transformer model is obtained using the Transformer hardware acceleration method based on mixed-precision quantization and Huffman coding described in Embodiment 1.
[0089] Specifically, such as Figure 4 As shown, the Transformer hardware accelerator in this embodiment adds a Huffman decoder to the existing Transformer hardware accelerator, which includes: a top-level control module, a PE array, a nonlinear unit, an input buffer, a weight buffer, a KV buffer, a quantization module, a Huffman decoder, and a comparator.
[0090] The top-level control module is used to control the input and output data of the accelerator and the data buffer (including weight buffer, KV buffer and input buffer).
[0091] The PE array is used to implement matrix operation of the Transformer model;
[0092] The nonlinear unit is used to perform layernorm, softmax, and GELU calculations in the Transformer model;
[0093] The input buffer is used to store the input data of the accelerator, the Q matrix calculated by the PE array, and part of the output of the nonlinear unit.
[0094] The weight buffer is used to retrieve weight data from on-chip memory in parallel and then send it to the decoder.
[0095] The KV buffer is used to cache the K and V matrices generated by the PE array;
[0096] The quantization module is used to quantize the calculation results of the PE array into 8 bits to complete the calculations in the subsequent steps.
[0097] The Huffman decoder is used to perform Huffman decoding on the weight data in the weight buffer.
[0098] The comparator is used to select the data required by the next module (input buffer or PE array) from its two respective input ports.
[0099] The aforementioned top-level control module, PE array, nonlinear unit, input buffer, weight buffer, KV buffer, quantization module, and comparator work together to realize the working principle of the Transformer model, which are all existing technologies. This embodiment innovatively adds a Huffman decoder after the weight buffer to decode the weight data in the weight buffer (compressed and stored off-chip / on-chip in the Transformer hardware acceleration method based on mixed precision quantization and Huffman coding described in Embodiment 1, and obtained from on-chip by the weight buffer) to realize the Transformer model function.
[0100] This embodiment deploys a hardware accelerator on an FPGA to implement the compressed SwingTransformer model described in Embodiment 1. For example... Figure 4 , Figure 5As shown, in each MSA (Multi-Head Self-Attention) module, the inputs at both ends of the PE array come from the quantized weights decoded by the Huffman decoder and the input features processed by LayerNorm, respectively. Based on these inputs, the PE array generates three matrices: Q (query), K (key), and V (value). The Q matrix is stored in the input buffer, while the K and V matrices are stored in the KV buffer. Then, the Q and K matrices are input into the PE array for computation to obtain the attention score matrix A; this matrix is normalized by Softmax and temporarily stored in the input buffer. Next, the normalized attention score matrix S is multiplied and added to the V matrix to obtain the output matrix O, which is then stored in the input buffer again. After the MSA computation is completed, the result remains in the input buffer. Subsequently, the MSA output is normalized by the LayerNorm layer and then fed into the MLP (Multilayer Perceptron) for feedforward computation, ultimately generating the output of the core swin transformer module in the Swin Transformer model. This output is finally returned to the input buffer, thus completing all computational steps.
[0101] In a better embodiment of the Transformer hardware accelerator, the PE array uses a DSP and a LUT to jointly complete multiple multiplication operations. Specifically, all MAC units are grouped into pairs to obtain several dual-MAC calculation groups consisting of dual-MAC units and a small number of unpaired single-MAC units. Each dual-MAC calculation group is implemented in parallel using a single DSP, while the single-MAC unit is implemented using LUT logic.
[0102] The PE array is a block parallel computing architecture capable of performing multiple multiplication operations (MACs). To improve hardware parallelism, this design uses a DSP and a lookup table (LUT) to jointly complete a group of MAC units. Specifically, the MAC units are paired up to form several computation groups consisting of dual MACs, and a small number of unpaired single MAC units. The dual MAC computation groups are implemented in parallel using DSP resources to fully leverage the high throughput and energy efficiency of the DSP; the remaining single MAC operations are implemented using LUT logic, achieving a better balance between DSP utilization and overall resource overhead. The core role of the DSP is to execute multiplication-addition operations at the hardware level in the most efficient and fastest way. The LUT is essentially a small memory that stores the pre-calculated results of multiplication operations, using the multiplier as a lookup condition to find the product result in the pre-stored results. A LUT with M inputs can pre-store... The system finds and outputs the corresponding result based on the input M-bit signal as the address.
[0103] S1: The inputs to the PE array come from the weight buffer, KV buffer, input buffer, and nonlinear unit, respectively. When the top-level control module sends a signal, the corresponding module passes the input to the PE array for multiplication and addition operations.
[0104] S2: Current Swin Transformer accelerators typically execute one MAC on each DSP. Since the DSP is the main performance bottleneck of the hardware accelerator, improving its utilization efficiency is crucial. Therefore, this embodiment employs DSP packing technology, enabling a single DSP to complete two MACs simultaneously, thereby significantly improving computational efficiency. Figure 6 As shown, taking an 8-bit input as an example, this scheme maps multipliers A and B to bits 0–7 and 18–25 of the DSP's large input port, respectively, while mapping the multiplier C to bits 0–7 of the small input port. Through parallel multiplication operations within the DSP, two product results, AC and BC, can be obtained simultaneously at the output port, where AC is located at bits 0–15 and BC is located at bits 18–33.
[0105] S3: All PE calculation results are accumulated by an external adder tree, quantized to 8 bits, and finally sent to the corresponding module.
[0106] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.
Claims
1. A hardware acceleration method for Transformer based on mixed-precision quantization and Huffman coding, characterized in that, include: S1, for the Transformer network layer, uses a genetic algorithm to obtain several configuration schemes for mixed-precision quantization; S2, perform mixed precision quantization on each network layer of Transformer based on each quantization configuration scheme, and obtain a KL divergence for each quantization configuration scheme; S3, using the quantized configuration scheme and the corresponding KL divergence as the output label and input features respectively, train a multilayer perceptron to obtain the quantized configuration prediction network; S4: Receive the target KL divergence value set by the user, and use the quantization configuration prediction network to obtain the corresponding quantization configuration scheme; S5, based on the quantization configuration scheme obtained in S4, performs mixed-precision quantization on each network layer of the Transformer; S6 uses Huffman coding to encode and compress all quantized weights, and then stores the compressed weights off-chip.
2. The Transformer hardware acceleration method based on mixed-precision quantization and Huffman coding according to claim 1, characterized in that, Several configuration schemes for mixed-precision quantization are obtained using a genetic algorithm, specifically including: S1.1: Randomly generate N candidate quantization configuration schemes, each quantization configuration scheme defining the quantization bit width of each layer of the Transformer; S1.2: For each current quantization configuration scheme, perform mixed precision quantization at each layer of the Transformer, and calculate the KL divergence between the quantized model and the original model on the softmax output of the validation data, which is used as the fitness of the quantization configuration scheme. S1.3: Using a tournament selection mechanism, two groups are randomly selected from the current population, each with x members. The quantization scheme with the lowest KL divergence value in each of the two groups is taken as the parent generation. A new generation of quantization schemes is generated through crossover and mutation. Specifically, the crossover operation involves swapping the bit widths of different quantization configuration schemes, while the mutation operation involves randomly perturbing the bit widths of any network layer in the quantization configuration scheme. S1.4: Repeat step S1.3 for a total of m rounds to obtain m quantized configuration schemes that have undergone multiple generations of evolution; S1.5: Repeat steps S1.2-S1.4 for a total of n rounds to finally obtain m×n quantization configuration schemes.
3. The method of claim 1, wherein, It also includes step S7: using a two-level lookup table to decode the codewords after the weights have been encoded and compressed.
4. The method of claim 3, wherein the method is characterized by, An improved two-level lookup table is used to decode the weighted, compressed codewords, specifically including: Step S7.1: Based on the length distribution of all codes in the Huffman coding table and the characteristics of the common high-order prefix, determine the depth of the first-level lookup table L1, the depth of the second-level lookup table L2, and the length of the window to be decoded. Step S7.2: Obtain the Huffman codeword from the Huffman encoded stream according to the length of the window to be decoded, and use it as the current Huffman codeword to be decoded; Step S7.3: Decode the current Huffman codeword using a first-level lookup table to obtain the decoded weights, code length, and valid bits. If decoding the current Huffman codeword using the first-level lookup table is invalid, shift the Huffman encoded stream according to the depth of the first-level lookup table to obtain a Huffman codeword as the current Huffman codeword. Then, decode the current Huffman codeword using a second-level lookup table to obtain the decoded weights, code length, and valid bits. Step S7.4: Shift the Huffman code stream according to the code length of the decoded codeword to obtain the next Huffman codeword to be decoded, and return to step S7.3 to decode the next Huffman codeword to be decoded.
5. The Transformer hardware acceleration method based on mixed-precision quantization and Huffman coding according to claim 4, characterized in that, The specific method for step S7.1 is as follows: First, determine the maximum length of all codes. ,according to Determine the segmentation location: If If it is an integer, then... Determine the dividing position; otherwise, and As candidate segmentation locations; Then, if If the value is an integer, then the code length in the Huffman coding table is greater than or equal to... Count the number of bits whose common high bit is 1 for all codewords. and will Determine the depth of the first-level lookup table, and Determine the depth of the second-level lookup table; if For non-integer values, among two candidate split positions, the one that results in a smaller total depth of the two-level lookup table is selected as the final split position. The total depth of the two-level lookup table is calculated as follows: for code lengths greater than or equal to... Count the number of bits whose common high bit is 1 for all codewords. The depths of the corresponding first-level and second-level lookup tables are respectively The total depth of the two-level lookup table is ; and for code lengths greater than or equal to in the Huffman coding table Count the number of bits whose common high bit is 1 for all codewords. The depths of the corresponding first-level and second-level lookup tables are respectively The total depth of the two-level lookup table is ; Finally, let the depth of the first-level lookup table be... The depth of the second-level lookup table is The length of the window to be decoded is .
6. The method of claim 1, wherein, The Transformer specifically refers to the Swing Transformer.
7. A Transformer hardware accelerator deployed on an FPGA, characterized in that, This is used to implement the Transformer model, which is obtained using the Transformer hardware acceleration method based on mixed precision quantization and Huffman coding as described in any one of claims 1-6.
8. The Transform accelerator of claim 7, wherein, include: Top-level control module, PE array, nonlinear unit, input buffer, weight buffer, KV buffer and decoder; The top-level control module is used to: control the input and output data of the accelerator and the data buffer; The PE array is used to implement matrix operation of the Transformer model; The nonlinear unit is used to perform layernorm, softmax, and GELU calculations in the Transformer model; The input buffer is used to store the input data of the accelerator, the Q matrix calculated by the PE array, and part of the output of the nonlinear unit. The weight buffer is used to retrieve weight data from on-chip memory in parallel and then send it to the decoder. The KV buffer is used to cache the K and V matrices generated by the PE array; The Huffman decoder is used to perform Huffman decoding on the weight data in the weight buffer.
9. The Transform accelerator of claim 8, wherein, The PE array uses a DSP and LUT to perform multiple multiplication operations. Specifically, all MAC units are paired up and grouped into groups. The two MAC units in each group have a common multiplier, resulting in several dual-MAC calculation groups composed of dual MAC units and unpaired single MAC units. Each dual-MAC calculation group is implemented in parallel using a single DSP, while the single MAC unit is implemented using lookup table logic.