FPGA model reasoning hardware adaptation method based on Qwen-2 low-bit quantization technology

Abstract

The embodiment of the invention discloses an FPGA (Field Programmable Gate Array) model reasoning hardware adaptation method based on a Qwen-2 low-bit quantization technology. According to the embodiment of the invention, quantitative perception training is carried out on a Transform layer of a Qwen-2 model so as to keep reasoning precision; converting the quantized weight and activation value into a fixed-point format supported by the FPGA and generating a quantization parameter table; according to DSP and BRAM resource distribution of the FPGA, mapping low-bit matrix operation to a DSP unit, storing quantization parameters in the BRAM, and planning a task scheduling sequence; a parallel pipeline is designed for an attention layer and a feed-forward layer, low-bit storage optimization is carried out on key value cache, and dynamic quantization adjustment is carried out on an activation value; and finally, deploying to an FPGA for testing, and performing closed-loop optimization on a hardware mapping strategy or quantization granularity based on delay and resource data. According to the method, the model precision is effectively maintained while the storage and calculation overhead is remarkably reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model inference technology, specifically to an FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, large-scale pre-trained models have demonstrated outstanding performance in fields such as natural language processing and computer vision. Qwen-2, as an advanced large-scale language model, possesses strong semantic understanding and generation capabilities and is widely used in dialogue systems, content generation, and intelligent question answering. However, such models typically have a large number of parameters and computational complexity, facing significant latency and resource pressure in real-time inference scenarios, making them difficult to deploy directly on resource-constrained edge devices or embedded systems.

[0003] Field-Programmable Gate Arrays (FPGAs), as a reconfigurable hardware platform, possess high parallel computing capabilities, low power consumption, and flexible configuration, making them suitable for accelerating neural network inference tasks. However, deploying large models that primarily rely on floating-point operations directly to FPGAs faces challenges such as limited hardware resources, limited storage bandwidth, and low computational efficiency. Traditional floating-point mapping methods not only consume significant amounts of digital signal processor (DSP) and block RAM resources but also fail to fully leverage the parallel computing advantages of FPGAs.

[0004] To reduce the resource requirements of model deployment, model quantization techniques are widely used. By reducing the numerical precision of weights and activation values ​​(e.g., quantizing from 32-bit floating-point to 8-bit or 4-bit integers), model size and computational overhead are significantly reduced at the cost of minimal precision. However, existing quantization methods often have the following limitations:

[0005] Simple quantization leads to a loss of accuracy: Some methods only perform post-training quantization on the model without considering quantization error during the training phase, which can easily cause a significant decrease in model accuracy.

[0006] Static hardware resource allocation: When deploying on an FPGA, a fixed resource allocation strategy is often used, which cannot dynamically adjust computing units and storage resources according to the characteristics of each layer of the model, resulting in low resource utilization.

[0007] Lack of pipeline optimization for large model structures: For the self-attention mechanism and feedforward network layer in Transformer-type models such as Qwen-2, existing solutions often adopt a sequential execution method, which does not make full use of the pipeline and parallel computing capabilities of FPGA, thus limiting inference throughput;

[0008] Separation of quantization parameters from hardware mapping: Most methods treat quantization and hardware deployment as independent stages, failing to achieve end-to-end collaborative optimization from quantization training to hardware resource scheduling, making it difficult to achieve the best balance between accuracy and efficiency. Summary of the Invention

[0009] To address these issues, this invention provides an FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology, which solves the technical problems of large accuracy loss, rigid hardware resource utilization, insufficient parallel computing, and overall performance bottlenecks caused by the fragmented deployment and optimization process of traditional quantization methods.

[0010] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0011] According to a first aspect of the present invention, an FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology is provided, the method comprising:

[0012] We performed low-bit quantization-aware training on the Transformer layer of the Qwen-2 model to simulate the impact of low-bit numerical representation on weights and activation values ​​during training, in order to preserve the model's inference accuracy.

[0013] The quantized Qwen-2 model weights and activation parameters are converted into a fixed-point format supported by FPGA, and a quantization parameter table containing scaling factors and zero-point information is generated.

[0014] Based on the DSP and BRAM resource distribution of the FPGA, the low-bit matrix multiplication operation of the Qwen-2 model is mapped to the DSP unit, the quantization parameter table is stored in the BRAM, and the scheduling order of the computing tasks at each layer is planned.

[0015] We designed a parallel pipeline for the attention layer and feed-forward layer of the Qwen-2 model, optimized the key-value pair cache of the attention layer with low-bit storage, and dynamically quantized and adjusted the activation values ​​of the feed-forward layer.

[0016] The adapted model is deployed to the FPGA for inference testing. Data on computational latency and resource utilization for each layer is collected. If the latency of a certain layer exceeds a preset threshold, the hardware mapping strategy or quantization granularity of that layer is adjusted. The optimization is repeated until the performance requirements are met.

[0017] Furthermore, the low-bit quantization-aware training includes:

[0018] Define the quantization function for weights and activations, adopt a symmetric or asymmetric quantization scheme, and set the quantization bit width;

[0019] The Straight-Through Estimator is used to handle gradient backpropagation during training;

[0020] A quantization error regularization term is added to the loss function to compensate for the accuracy loss introduced by quantization.

[0021] Furthermore, the generation of the quantization parameter table includes:

[0022] For weight tensors, directly calculate their minimum and maximum values;

[0023] For the activation value tensor, perform forward inference using a representative calibration dataset to count its minimum and maximum values;

[0024] Calculate the scaling factor and zero point based on the minimum and maximum values, and store them as a structured parameter table.

[0025] Furthermore, the mapping method for the low-bit matrix multiplication operation is as follows:

[0026] Convert the 4-bit quantized weights and inputs into integer format;

[0027] The product sum operation is performed using the built-in multiplier of the DSP unit and configured in pipelined mode to improve throughput.

[0028] Furthermore, the scheduling order is planned using a greedy algorithm based on the critical path to minimize the overall inference latency under DSP resource constraints.

[0029] Furthermore, the low-bit storage optimization of the attention layer key-value pair cache includes:

[0030] A 4-bit unified quantization scheme is adopted, and the quantization parameters are statically determined based on the global range of the cache tensor;

[0031] Store the cache as an integer index and use it for real-time dequantization during inference.

[0032] Furthermore, the dynamic quantization adjustment of the activation value of the Feed-Forward layer includes:

[0033] In each forward propagation batch, the mean and standard deviation of the activation values ​​are calculated in real time;

[0034] The scaling factor is dynamically calculated based on the standard deviation, and the activation value is quantized.

[0035] Furthermore, the hardware mapping adjustment strategy includes increasing the parallelism factor of the computation at this layer.

[0036] Furthermore, adjusting the quantization granularity includes reducing the quantization bit width of the layer's weights or activation values.

[0037] Furthermore, the quantization parameter table is a structured file, in which each record includes a tensor identifier, a scaling factor, and zero-point information.

[0038] The embodiments of the present invention have the following advantages:

[0039] This invention employs quantization-aware training on the Transformer layer of the Qwen-2 model to maintain inference accuracy. The quantized weights and activation values ​​are converted to a fixed-point format supported by the FPGA, and a quantization parameter table is generated. Based on the FPGA's DSP and BRAM resource distribution, low-bit matrix operations are mapped to DSP units, quantization parameters are stored in BRAM, and a task scheduling order is planned. Parallel pipelines are designed for the attention layer and feedforward layer, low-bit storage optimization is performed on the key-value cache, and dynamic quantization adjustment is implemented for the activation values. Finally, the model is deployed to the FPGA for testing, and closed-loop optimization of the hardware mapping strategy or quantization granularity is performed based on latency and resource data until performance requirements are met. This invention significantly reduces storage and computational overhead while effectively maintaining model accuracy. It is suitable for edge inference scenarios with stringent real-time and energy efficiency requirements, such as intelligent security and autonomous driving, and can solve the problems of high resource consumption, high inference latency, and significant loss of quantization accuracy when deploying large language models on FPGAs. Attached Figure Description

[0040] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0041] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0042] Figure 1 A flowchart illustrating an FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology, provided for an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the application process of an FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology, provided in an embodiment of the present invention. Detailed Implementation

[0044] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Traditional floating-point arithmetic methods perform model inference directly on the FPGA. However, this approach suffers from high resource consumption and slow processing speed because FPGAs are inefficient at handling floating-point operations. Another method, simple quantization, merely reduces the number of bits in the model without considering the impact of quantization on model accuracy, leading to a significant drop in inference precision. Furthermore, static resource allocation methods allocate FPGA resources according to a fixed pattern, failing to adjust flexibly based on model characteristics and actual conditions, resulting in low resource utilization. Finally, methods that execute computational tasks sequentially lack parallel processing mechanisms, leading to long inference times.

[0046] To address the technical challenges of traditional quantization methods, such as significant accuracy loss, rigid hardware resource utilization, insufficient parallel computing, and fragmented deployment and optimization processes, which lead to overall performance bottlenecks.

[0047] refer to Figure 1 and Figure 2 This invention discloses an FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology. The core idea is to achieve model and FPGA hardware adaptation by using low-bit quantization technology and reasonable allocation of hardware resources around the Qwen-2 model.

[0048] In practice, the first step is to perform low-bit quantization-aware training on the Transformer layer of the Qwen-2 model to simulate the impact of low-bit numerical representation on weights and activation values, ensuring inference accuracy. Next, the quantized parameters are converted to a fixed-point format supported by the FPGA, generating a quantization parameter table. Then, based on the FPGA's resource distribution, low-bit matrix multiplication operations are mapped to DSP units, the parameter table is stored in BRAM, and the computation task scheduling order is planned. Parallel pipelines are designed for the attention layer and feed-forward layer, and the relevant data storage and activation values ​​are optimized. Finally, the model is deployed to the FPGA for testing, and performance is continuously optimized based on performance data.

[0049] We performed low-bit quantization-aware training on the Transformer layer of the Qwen-2 model to simulate the impact of low-bit numerical representation on weights and activation values ​​during training, in order to preserve the model's inference accuracy.

[0050] The quantized Qwen-2 model weights and activation parameters are converted into a fixed-point format supported by FPGA, and a quantization parameter table containing scaling factors and zero-point information is generated.

[0051] Based on the DSP and BRAM resource distribution of the FPGA, the low-bit matrix multiplication operation of the Qwen-2 model is mapped to the DSP unit, the quantization parameter table is stored in the BRAM, and the scheduling order of the computing tasks at each layer is planned.

[0052] We designed a parallel pipeline for the attention layer and feed-forward layer of the Qwen-2 model, optimized the key-value pair cache of the attention layer with low-bit storage, and dynamically quantized and adjusted the activation values ​​of the feed-forward layer.

[0053] The adapted model is deployed to the FPGA for inference testing. Data on computational latency and resource utilization for each layer is collected. If the latency of a certain layer exceeds a preset threshold, the hardware mapping strategy or quantization granularity of that layer is adjusted. The optimization is repeated until the performance requirements are met.

[0054] 1) Perform low-bit quantization-aware training on the Transformer layer of the Qwen-2 model. During training, simulate the impact of low-bit numerical representation on weights and activation values ​​to preserve the model's inference accuracy. The specific implementation method for this step is as follows:

[0055] Implementing low-bit quantization-aware training in the Transformer layer of the Qwen-2 model requires modifying the training process to simulate quantization effects. First, define the quantization functions for the weights and activations, choosing a bit width (e.g., 4-bit or 8-bit) using a symmetric or asymmetric quantization scheme, and setting the quantization step size parameter. The quantization operation simulates the low-bit representation through approximation functions, such as the weights. Quantization output:

[0056]

[0057] in It is a dynamically adjusted step size, calculated as follows: , The value range is ensured by representing the number of bits, simulating the range of integer values. The step size is initialized based on the statistical distribution of the weights and updated during training to adapt to model changes.

[0058] Integrating a Straight-Through Estimator in the training loop handles gradient backpropagation, avoiding the quantization non-differentiability problem. The gradient of the quantized layer is approximated as:

[0059]

[0060] By directly propagating the upstream gradient without modifying the quantization operation, efficient backpropagation is maintained. Similarly, activation values... Quantification:

[0061]

[0062] STE is also used, among which Adaptive computation based on activation range. During training, quantization modules are inserted into the self-attention layer and feedforward layer of the Transformer, and computation is performed after each forward propagation simulation of quantization.

[0063] The fine-tuning process uses the original loss function, such as cross-entropy. Combined with the quantization error regularization term:

[0064]

[0065] Weight Control the regularization strength. Total loss:

[0066]

[0067] Optimize model parameters to compensate for the accuracy loss introduced by quantization. After training, evaluate inference accuracy to align with the floating-point baseline, and monitor the average accuracy difference using the validation set to ensure it remains within 1%.

[0068] 2) Convert the quantized Qwen-2 model weights and activation parameters to a fixed-point format supported by the FPGA, generating a quantization parameter table containing scaling factors and zero-point information. The specific implementation method for this step is as follows:

[0069] For the weight tensor of the Qwen-2 model, the weight parameters remain static after quantization. The minimum and maximum values ​​of each tensor element are directly calculated without input data dependency. The tensor is traversed using NumPy or PyTorch libraries, and the `min()` and `max()` functions are called to obtain these values. For the activation value tensor, its values ​​change dynamically and need to be calculated during the quantization calibration phase. A representative calibration dataset (such as a subset of CIFAR-10 or task-specific samples) is prepared, and batch forward inference is run using PyTorch's quantization API. On multiple input batches, the `MinMaxObserver` module is used to record the element-level minimum and maximum values ​​of each activation tensor and aggregate them into global statistics. Ensure the dataset covers typical inference scenarios to avoid overfitting or distribution shift; a dataset size of at least 1000 samples is recommended. The entire statistical process is automated using a Python script, outputting the `min_val` and `max_val` values ​​of each tensor to an intermediate file.

[0070] Calculate the scaling factor S and zero point Z based on the values ​​of min_val and max_val. Define the target range of fixed-point numbers for the FPGA, such as the range of 8-bit unsigned integers. and The scaling factor S is expressed by the formula:

[0071]

[0072] The calculation uses the denominator to represent the quantized dynamic range. The formula for calculating the zero point Z is:

[0073]

[0074] The `round` function ensures that Z is an integer, achieving a precise mapping from floating-point to integer. For symmetric tensors (such as those with zero mean weights), a simplified approach is used: calculating the absolute maximum value. ,Then and set All calculations are performed using floating-point operations, and S > 0 is verified to avoid division by zero errors.

[0075] Generate a quantization parameter table containing metadata for all tensors in the model. Each entry defines a tensor identifier (e.g., "block1.linear_weight" or "layer3.relu_activation"), a scaling factor S, and a zero point Z. The parameter table structure is in dictionary or list form and is exported to a file using Python's csv or json module. For example, a CSV file might have each line formatted as tensor_name, scale, zero_point. During deployment, this file is integrated into the FPGA development environment (e.g., a Vivado project) and loaded via a hardware description language parser. Ensure numerical precision is aligned with FPGA resources, for example, using float32 to store S and int8 to store Z, and verify data integrity using checksums.

[0076] 3) Based on the DSP and BRAM resource distribution of the FPGA, the low-bit matrix multiplication operations of the Qwen-2 model are mapped to DSP units, the quantization parameter table is stored in the BRAM, and the scheduling order of each layer of computation tasks is planned. The specific implementation method for this step is as follows:

[0077] First, the DSP and BRAM resource distribution of the FPGA is analyzed. Based on the specifications of the target device, such as the Xilinx Zynq Ultrascale+, available DSP slices (e.g., each DSP supports 18x25-bit multiplication operations) and BRAM blocks (36Kb capacity per block) are identified. A resource utilization report is generated using a resource assessment tool to identify the DSP cell distribution density (e.g., 4 DSPs per logic block) and the BRAM address space layout, ensuring that quantization parameter storage does not conflict before model deployment. The quantization parameter table includes scaling factors. and zero point Parameters are stored in BRAM for low-latency access, each occupying an 8-bit fixed-point format, and optimized for contiguous storage to reduce addressing overhead. Matrix multiplication operations use 4-bit low-bit quantization, and weights... and input Through the formula:

[0078]

[0079]

[0080] When processed as an integer type, the low-bit product sum operation is implemented using the built-in multiplier of the DSP unit, as shown in the formula:

[0081]

[0082] The computation output is calculated, with the DSP configured in pipelined mode to improve throughput. The scheduling order of computational tasks at each layer is planned using a greedy algorithm based on the critical path, with resource constraints defined. ,in It is the first Layer multiplication operands This is the requirement for each DSP operation. Ensure scheduling does not exceed limits and introduce inter-layer delay:

[0083]

[0084] Optimize the execution sequence to minimize overall inference latency. During implementation, a hardware description language needs to be written in the Vivado HLS tool to verify that resource utilization meets the requirements of each layer of the model and avoids resource contention. The entire process is automated through script deployment to ensure improved real-time performance.

[0085] 4) Design parallel pipelines for the attention layer and feed-forward layer of the Qwen-2 model, optimize the key-value pair cache of the attention layer with low-bit storage, and dynamically quantize and adjust the activation values ​​of the feed-forward layer. The specific implementation method for this step is as follows:

[0086] In the inference process of the Qwen-2 model, a parallel pipeline mechanism is designed to overlap the computation of the attention layer and the feed-forward layer. Utilizing pipeline parallelism, the model is divided into multiple stages: when processing the current input token, the attention layer performs self-attention computation and updates the key-value cache, while the feed-forward layer processes the output of the previous token in parallel. This is achieved through an asynchronous execution queue, where the computation graph is split into independent tasks and allocated to GPU cores by the scheduler. Specifically, the output of the attention layer serves as the input to the feed-forward layer, but the pipeline allows the feed-forward layer to process the (t-1)th token simultaneously while the attention layer is computing the t-th token, thus reducing overall latency. The performance model uses Amdahl's Law to evaluate parallel efficiency: , in It is the serial portion of time. This is the parallel portion of the time. This design requires hardware support for multi-core synchronization, and after optimization, the throughput is improved by approximately 30%.

[0087] For the key-value pair cache of the attention layer, low-bit storage optimization is implemented to compress memory footprint. The key (K) and value (V) caches employ a unified quantization scheme with a fixed bit width of 4 bits (b=4). Quantization parameters are statically determined based on the global range of the cache tensor: during model initialization, the maximum and minimum values ​​of K and V are collected, and the step size is calculated. ,in represent or The quantized cache is stored as an integer index:

[0088]

[0089] And in real time dequantization during inference This saves 75% of storage space, and the error is controlled within 1% through offline calibration.

[0090] For the activation values ​​of the Feed-Forward layer, dynamic quantization is applied to reduce computational precision. This layer activation occurs after the first linear transformation (e.g., the output of the GELU function), generating an intermediate tensor. Dynamic adaptation of quantization parameters: Statistical values ​​are calculated in real time for each forward propagation batch. and:

[0091]

[0092] in Set the batch size. Set the scaling factor. , This is a hyperparameter (default value 2). Then it is quantized as follows:

[0093]

[0094] And input the second-level linear transform. This adaptive method adjusts the precision according to the input distribution, and the mean square error of the quantization noise is:

[0095]

[0096] in It uses a dynamic step size. Experiments show that activation memory is reduced by 50% with an accuracy loss of less than 0.5%. All implementations are integrated into the model inference framework, supporting mixed precision operation of FP16 and INT8.

[0097] 5) Deploy the adapted model to the FPGA for inference testing, collect computational latency and resource utilization data for each layer. If the latency of a certain layer exceeds the threshold, adjust the hardware mapping strategy or quantization granularity for that layer, and iterate until the performance requirements are met. The specific implementation method of this step is as follows:

[0098] The adapted model is deployed to the target FPGA platform and implemented using hardware acceleration frameworks such as Xilinx Vitis or Intel OpenVINO. The model is converted to register-transfer-level (RTL) code using High-Level Synthesis (HLS) tools, specifying clock frequency constraints and resource allocation parameters. During the synthesis and implementation phases, bitstream files are generated and loaded onto the FPGA development board, ensuring that input / output interfaces are correctly configured to handle the inference data stream. Embedded monitoring logic, including on-chip counters and timestamp units, is configured to capture timing information in real time. When running inference tests, a standardized test dataset is input, and multiple iterations are performed to eliminate noise, reading the raw data via JTAG or PCIe interfaces.

[0099] Perform tests and collect performance metrics for each layer: Calculate latency by measuring the difference between the start and end timestamps of each layer's operation, defined as... ,in and Representing the first Layer startup and completion times. Resource utilization data is extracted from the comprehensive report, including the utilization rates of lookup tables (LUTs), block RAM (BRAM), and digital signal processors (DSPs), calculated using the following formula:

[0100]

[0101] Indicates the first The amount of resources occupied by the layer. This represents the total resource capacity of the FPGA. All data is stored in a log file and automatically parsed using Python or TCL scripts.

[0102] Compare the delay of each layer With preset threshold Threshold Based on the application scenario, such as real-time inference requirements ms. If detected This triggers an optimization loop. Optimization focuses on the problem layer, adjusting hardware mapping strategies such as increasing the parallelism factor. The new delay is calculated as follows:

[0103]

[0104] It is an operand. For clock frequency. Or modify the quantization granularity, for example, by changing the weight bit width from... Position dropped to The quantization error model is as follows: , For the layer correlation coefficient, it is necessary to verify that the accuracy loss does not exceed the allowable range. .

[0105] After application optimization, the model is redeployed to the FPGA, and the testing and comparison process is repeated. This iterative process continues until all layers meet the requirements. The system monitors resource utilization to ensure it does not exceed the FPGA's limit. Performance metrics are recorded for each iteration, and convergence conditions such as the maximum number of iterations or the minimum improvement threshold are evaluated. The final output is an optimization report, including the final delay distribution and a resource usage heatmap.

[0106] This invention is applied to scenarios requiring model inference on FPGAs, enabling efficient inference with low-bit quantization when using Qwen-2 models. Utilizing an FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology, it is applicable to real-time scenarios with high demands for computing resources and inference speed, such as real-time target identification and tracking in video footage in intelligent security monitoring systems, and rapid judgment of road conditions and obstacles in autonomous driving scenarios.

[0107] Compared to traditional floating-point arithmetic methods, the use of low-bit quantization technology significantly reduces resource consumption and improves computation speed. Compared to simple quantization methods, quantization-aware training simulates the effects of low bits, preserving model inference accuracy. Compared to static resource allocation methods, this patent dynamically allocates resources based on FPGA resource distribution and model characteristics, improving resource utilization. Compared to sequential execution methods, a parallel pipeline design shortens inference time.

[0108] This invention has wide applications in fields such as artificial intelligence, intelligent monitoring, and autonomous driving. By using an FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology, model inference can be completed in a short time, improving customer service efficiency.

[0109] In smart home scenarios, devices need to recognize and process users' voice commands in real time. This adaptation method enables devices to respond more quickly and accurately, improving the user experience.

[0110] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A hardware adaptation method for FPGA model inference based on Qwen-2 low-bit quantization technology, characterized in that, The method includes: We performed low-bit quantization-aware training on the Transformer layer of the Qwen-2 model to simulate the impact of low-bit numerical representation on weights and activation values ​​during training, in order to preserve the model's inference accuracy. The quantized Qwen-2 model weights and activation parameters are converted into a fixed-point format supported by FPGA, and a quantization parameter table containing scaling factors and zero-point information is generated. Based on the DSP and BRAM resource distribution of the FPGA, the low-bit matrix multiplication operation of the Qwen-2 model is mapped to the DSP unit, the quantization parameter table is stored in the BRAM, and the scheduling order of the computing tasks at each layer is planned. We designed a parallel pipeline for the attention layer and feed-forward layer of the Qwen-2 model, optimized the key-value pair cache of the attention layer with low-bit storage, and dynamically quantized and adjusted the activation values ​​of the feed-forward layer. The adapted model is deployed to the FPGA for inference testing. Data on computational latency and resource utilization for each layer is collected. If the latency of a certain layer exceeds a preset threshold, the hardware mapping strategy or quantization granularity of that layer is adjusted. The optimization is repeated until the performance requirements are met.

2. The FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology as described in claim 1, characterized in that, The low-bit quantization-aware training includes: Define the quantization function for weights and activations, adopt a symmetric or asymmetric quantization scheme, and set the quantization bit width; The Straight-Through Estimator is used to handle gradient backpropagation during training; A quantization error regularization term is added to the loss function to compensate for the accuracy loss introduced by quantization.

3. The FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology as described in claim 1, characterized in that, The generated quantization parameter table includes: For weight tensors, directly calculate their minimum and maximum values; For the activation value tensor, perform forward inference using a representative calibration dataset to count its minimum and maximum values; Calculate the scaling factor and zero point based on the minimum and maximum values, and store them as a structured parameter table.

4. The FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology as described in claim 1, characterized in that, The mapping method for the low-bit matrix multiplication operation is as follows: Convert the 4-bit quantized weights and inputs into integer format; The product sum operation is performed using the built-in multiplier of the DSP unit and configured in pipelined mode to improve throughput.

5. The FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology as described in claim 1, characterized in that, The scheduling order is planned using a greedy algorithm based on the critical path to minimize the overall inference latency under DSP resource constraints.

6. The FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology as described in claim 1, characterized in that, The attention layer key-value pair cache low-bit storage optimization includes: A 4-bit unified quantization scheme is adopted, and the quantization parameters are statically determined based on the global range of the cache tensor; Store the cache as an integer index and use it for real-time dequantization during inference.

7. The FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology as described in claim 1, characterized in that, The dynamic quantization adjustment of the Feed-Forward layer activation value includes: In each forward propagation batch, the mean and standard deviation of the activation values ​​are calculated in real time; The scaling factor is dynamically calculated based on the standard deviation, and the activation value is quantized.

8. The FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology as described in claim 1, characterized in that, The hardware mapping adjustment strategy includes increasing the parallelism factor of the computation at this layer.

9. The FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology as described in claim 1, characterized in that, The adjustment of quantization granularity includes reducing the quantization bit width of the layer's weights or activation values.

10. The FPGA model inference hardware adaptation method based on Qwen-2 low-bit quantization technology as described in claim 1, characterized in that, The quantization parameter table is a structured file, in which each record includes a tensor identifier, scaling factor, and zero-point information.

Images