A resnet network quantization and simulation reasoning method for FPGA deployment

Abstract

The application discloses a ResNet network quantization and simulation reasoning method for FPGA deployment, which effectively alleviates the numerical overflow and precision reduction problems caused by direct rounding in the traditional quantization method. The application performs quantization conversion on the weights and activation values of the ResNet model to adapt to the FPGA, so that the model maintains high precision under the condition of limited FPGA resources. In addition, the application designs a simulation reasoning method without hardware verification, which can test the performance and precision of the quantization model in the FPGA simulation environment, reduces the complex steps of hardware debugging and verification, and greatly improves the development efficiency and reduces the resource consumption.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence hardware acceleration technology, and specifically to a ResNet network quantization and simulation inference method for FPGA deployment. Background Art

[0002] With the rapid development of deep learning and neural network technologies, the residual network architecture (ResNet) has significantly improved the performance of deep neural networks in the field of deep learning. The introduction of the residual module effectively alleviates the problems of vanishing and exploding gradients in deep networks, thereby enhancing the stability of the training process. Furthermore, residual network architectures like ResNet18, with their small number of parameters and excellent classification performance, perform well on resource-rich hardware such as GPUs and have been widely used in various computing tasks.

[0003] However, due to the limitations of FPGAs in terms of computing and storage resources, deploying ResNet directly on FPGAs for efficient inference faces challenges. Traditional neural network quantization techniques reduce computing and storage resource consumption by converting the model's weights and biases from floating-point numbers to low-precision integers. Neural network quantization is mainly divided into quantization-aware training (QAT) and post-training quantization (PTQ). In post-training quantization methods, the quantization range is usually determined by least squares or histograms to minimize the loss of accuracy during the quantization process. Although post-training quantization technology can effectively reduce the computing and storage requirements of neural networks, the direct rounding of floating-point numbers may cause overflows in model weights. When deployed on resource-constrained hardware, this overflow can reduce computing speed and energy efficiency, thereby affecting the device's real-time processing capabilities.

[0004] Numerous studies have explored and optimized post-training quantization methods for ResNet models. For example, publication number CN113657267A discloses a semi-supervised person re-identification model, method, and apparatus. This proposal introduces an average teacher model during the ResNet model pre-training phase for mutual learning and supervision, improving model accuracy and achieving better performance before loading onto an FPGA. While this approach alleviates the overfitting problem of the ResNet model to some extent, its optimization focuses on distillation of the full-precision model and pseudo-label generation, failing to fully consider FPGA resource constraints. Publication number CN1162302A discloses an FPGA-based intelligent capture system and method. This system trains a ResNet18 model and uses TensorRT for quantization and compilation, generating a quantized model suitable for deployment on FPGA edge devices. TensorRT quantization calibrates the activation range of the data statistical model and quantizes weights and activations to INT8 format, significantly improving inference speed and reducing memory requirements. However, if the calibration dataset is insufficient or unrepresentative, TensorRT quantization can result in significant accuracy loss.

[0005] Furthermore, current model deployment processes for FPGAs typically directly load quantized models onto hardware for verification to evaluate their performance in real-world applications. This approach is not only time-consuming, but also lacks a comprehensive assessment of the model's runtime environment, which can lead to significant variations in the inference efficiency and energy efficiency of quantized models on different hardware platforms. Summary of the Invention

[0006] The purpose of this invention is to provide a ResNet network quantization and simulation inference method for FPGA deployment, aiming to optimize model performance while fully considering the resource limitations of FPGA hardware and reducing dependence on calibration datasets, thereby achieving more efficient and stable model deployment effects.

[0007] In order to achieve the above-mentioned objectives, the present invention provides a ResNet network quantization and simulation inference method for FPGA deployment, comprising the following steps:

[0008] S1. Load the ResNet model and adjust the model structure, including the operation of integrating the BN layer, to adapt to FPGA deployment requirements;

[0009] S2. Read the full-precision image and perform preprocessing. Use the adjusted model to perform forward inference on the preprocessed image, and use hook functions to capture the input and output data of each layer.

[0010] S3. Based on the captured data, the quantization factor calculation is completed layer by layer as the initial value of the model quantization factor;

[0011] S4. Perform reverse retrieval and adjust the quantization of convolution operations and activation functions to optimize the selection of quantization factors for each layer;

[0012] S5. Build an FPGA simulation environment, perform simulated FPGA reasoning, and optimize the selection of quantization factors.

[0013] A further preferred technical solution of the present invention is that, in step S1, the ResNet model is loaded and the model structure is adjusted, specifically by:

[0014] Load the ResNet18 model through TorchVision, including its learnable parameters and network structure;

[0015] Extract the network structure source code from the package to achieve visual code display;

[0016] After loading the model, the BN layer is fused through a custom fusion function, and the relevant parameters of the BN layer are reconstructed into the weights and biases of the convolutional layer through equations.

[0017] Preferably, the method for preprocessing the image in step S2 includes operations of cropping, converting and normalizing the image.

[0018] Preferably, the input and output captured during the forward reasoning process in step S2 are used for the subsequent calculation of quantization factors and evaluation of model performance. The specific operation is: use the register_forward_hook method of PyTorch to register the hook function in each convolutional layer, and when the forward propagation reaches the layer, the input and output data of the layer are automatically captured and stored.

[0019] Preferably, in step S3, the quantization factor calculation is completed layer by layer based on the captured data, and the specific method is:

[0020] Based on the input and output data of each convolutional layer, the mean square error of the weights, inputs, biases, and outputs of the convolutional layer is calculated in the integer range [-20, 20], and the value with the smallest mean square error is selected as the initial value of the quantization factor. In order to meet hardware limitations, the quantization factors of each parameter are truncated: for outputs and biases, the factors are truncated to the range [-16, 15]; for inputs and weights, they are truncated to the range [-8, 7].

[0021] Then, the output quantization factor of the current convolutional layer is passed as the input quantization factor of the next layer. Combined with the layer structure parameters of the model, by traversing the module_name in the model dictionary, the model information of each layer is loaded, and combined with the input and output data obtained by the hook function, the quantization factor calculation of the convolutional layer input, output, weight and bias is completed layer by layer.

[0022] Finally, by sorting out the parent-child node relationship of the model residual network, the overall output quantization factor is adjusted. Specifically, the output quantization factors of the parent node and the child node are added and averaged, and the result is rounded off to ensure that it is an integer, so as to obtain the optimized output quantization factor, and use it to adjust the input quantization factor.

[0023] Preferably, step S3 further includes:

[0024] Combined with the weight quantization factor S of each layer w , bias quantization factor S b And the output quantization factor S ofm , initialize the displacement quantization factor S from weight to bias w2b and the offset quantization factor S to the output b2o Then, according to the storage unit capacity and computing resource limit of FPGA, the displacement quantization factor is calculated and adjusted to S w2b Truncated in the integer interval [-5,3], S b2o Truncated to the integer interval [5,8].

[0025] Preferably, in step S4, a reverse search is performed to adjust the quantization of the convolution operation and the activation function to optimize the selection of the quantization factor of each layer. The specific method is:

[0026] The search starts from the last layer of the ResNet model and proceeds layer by layer. For each convolutional layer searched, the layer is quantized in sequence by traversing the quantization factor combination generated by the Cartesian product, while the other layers maintain full precision inference. During the traversal process, the mean squared error of each combination is calculated using torch.mean_squared_error(), and the combination with the smallest error is selected as the optimal quantization factor for the layer.

[0027] After the optimal quantization factor of the current layer is determined, continue to search the previous layer. During the search process of the previous layer, the current layer is fixed as the optimal quantization factor, and the Cartesian product combination is traversed again to find the optimal quantization factor of the previous layer;

[0028] Repeat the above steps, reverse search layer by layer, and finally obtain the optimal quantization factor configuration of the entire model.

[0029] Preferably, in step S5, FPGA simulation reasoning is performed to optimize the quantization factor. The specific method is as follows:

[0030] Use PyTorch's register_forward_hook() to register a forward hook function at each convolutional layer to monitor and obtain the input, output, and module name of each layer of the model; then, sort the module names in reverse order and load each convolutional layer and its quantization factor in reverse order according to the module_name in the model dictionary;

[0031] Based on the value of the initial quantization factor of each layer, the search range is determined to be an integer interval of ±5. Within this range, itertools.product() is called to generate all possible combinations of quantization factors of each layer, ensuring that the search for quantization factors focuses on the initial value.

[0032] Perform hardware adaptability screening on the generated quantization factor combinations and filter out combinations that exceed the FPGA quantization support range;

[0033] For each quantization combination that meets the conditions, perform simulation inference; calculate the mean square error between the simulation results and the saved full-precision model output; compare the mean square error results of all combinations, and select the combination with the smallest mean square error as the optimized quantization factor configuration.

[0034] Preferably, in step S5, an FPGA simulation environment is constructed, PyTorch is used for FPGA simulation, and the convolution calculation process in PyTorch is modified. The specific method is as follows:

[0035] When processing the convolutional layers of the ResNet18 model, if the number of input or output channels is less than a multiple of 64, use the pad() function of the torch.nn.functional module to perform zero padding to expand the number of channels to a multiple of 64;

[0036] Extract and store the bias of the convolutional layer separately, and set the bias parameter of the convolutional layer to False to turn off the automatic calculation of the bias;

[0037] Convert the convolution layer to a grouped convolution with a convolution kernel size of 1, and set the number of groups to the product of the number of input channels and the number of output channels of the original convolution; and fill the convolution kernel weights of each group into the corresponding grouped convolution kernel one by one according to the order of the output channels and input channels of the convolution layer;

[0038] According to the group number requirement of the group convolution, the number of channels of the input tensor is expanded to match the channel configuration of the group convolution;

[0039] Apply group convolution to the expanded input tensor and perform convolution calculations. For the calculated output, more than 2 7 The elements are truncated;

[0040] Add the previously saved bias to the convolution output, and truncate any elements in the result that exceed 215.

[0041] As a preference, during the inference process, only the convolution calculation method is adjusted, keeping the basic block structure of ResNet18 unchanged; when inferring to the classifier module, the torch.nn.AvgPool2d() called by the original model is replaced with a custom integer pooling function.

[0042] Beneficial effects: The ResNet network quantization and simulation inference method for FPGA deployment proposed in this invention has the following significant technical effects compared with the existing technology:

[0043] 1. Improve model accuracy: Through precise quantization factor calculation and optimization, the quantization model accuracy of the present invention on validation sets such as ImageNet reaches 70.86%, which is a significant improvement compared to the 58.72% of the traditional post-training quantization method. The average accuracy improvement is more than 10%, demonstrating its robustness and reliability in various scenarios.

[0044] 2. Improved Inference Efficiency: Our optimization technology reduces inference time per image to approximately 150ms, essentially meeting the requirements of real-time FPGA inference. Compared to the average inference time of 200ms to 250ms achieved by traditional quantization methods, this technology demonstrates significant efficiency advantages.

[0045] 3. Enhanced hardware adaptability: By precisely adjusting and optimizing the quantization factors, the present invention ensures the stable operation of the quantization model on the FPGA, avoids the problems of data overflow and reduced computing speed, and improves hardware adaptability.

[0046] 4. Reduce resource consumption: This invention reduces the resource consumption of model deployment and reduces development costs by reducing the dependence on calibration datasets.

[0047] 5. Reduce development cycle: By completing simulation reasoning on the GPU, the present invention significantly reduces the cost and time of hardware verification, shortens the development cycle by about 30%, and significantly improves development efficiency.

[0048] 6. Improve model applicability: The flexible deployment method of the present invention improves the applicability of the model, provides a more competitive solution for embedded intelligent applications, and has broad market prospects.

[0049] 7. Optimize storage and computing efficiency: Through specific FPGA simulation modifications, such as zero padding, independent management of bias terms, and grouped convolution, the present invention optimizes storage and computing efficiency and ensures aligned data transmission on the FPGA.

[0050] 8. Enhance the practicality and competitiveness of the model: In embedded and edge computing scenarios, the optimization technology of the present invention effectively meets the needs of high-concurrency applications, making it more practical and competitive in practical applications.

[0051] 9. Achievement of Technology Maturity: This invention has achieved a high level of technology maturity, implementing and verifying the complete process of loading, fusion, quantization, and simulation inference of the ResNet model. It has high operability and adaptability, and is easy to integrate into different systems or hardware platforms.

[0052] Through the description of the above technical effects, it is shown that the present invention not only theoretically proposes a new ResNet network quantization and simulation inference method for FPGA deployment, but also verifies its effectiveness in practice, providing a new solution for the efficient deployment of deep learning models on FPGAs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flowchart of the ResNet network quantization and simulation inference method for FPGA deployment in the present invention.

[0054] Figure 2 This is the ResNet18 network structure after BN layer fusion in an embodiment of the present invention.

[0055] Figure 3 This is a structural diagram of the classifier module in an embodiment of the present invention.

[0056] Figure 4 This is a flowchart of the average processing of the quantization factors output by the parent and child nodes in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0058] Embodiment: This embodiment aims to solve the problems of precision loss, low computational efficiency, and poor hardware adaptability faced by the existing ResNet model in FPGA hardware deployment. To address these problems, this embodiment proposes a ResNet quantization and simulation inference method specifically for FPGA deployment, which can effectively alleviate the numerical overflow and precision degradation problems caused by direct rounding in traditional quantization methods. Figures 1-4 The solution of this embodiment is described in detail.

[0059] This embodiment has three key parts, namely, quantization factor initialization, quantization inference process simulation, and quantization factor optimization. Figure 1 The specific implementation is as follows:

[0060] The quantization factor initialization part requires a ResNet model. In this embodiment, the ResNet18 model is loaded through TorchVision, including its learnable parameters and network structure. In order to facilitate the analysis and control of the inference process, the network structure source code is extracted from the package to achieve visual code display. After loading the model, the BN layer is fused through a custom fusion function, and the relevant parameters of the BN layer are reconstructed into the weights and biases of the convolution layer through equations. The network structure of the ResNet18 model after the BN layer is fused is as follows: Figure 2 Then, the mean square error method is used to calculate the quantization factor layer by layer, including the convolution weight S of each layer. w , bias S b , weight to bias shift S w2b , bias to output displacement S b2o , enter S ifm , output S ofm Quantization factor. The specific steps and calculation method of the quantization factor are as follows:

[0061] Read a full-precision image from the specified path and preprocess it using TorchVision's built-in Transforms library. This includes cropping it to 224x224 pixels, converting it to a Tensor data type, and normalizing the pixel values ​​to the [0,1] range.

[0062] The preprocessed image is fed into the BN-fused model for full-precision forward inference. The input and output of each convolutional layer are recorded by registering a hook function at each layer. This is done using PyTorch's register_forward_hook method. When forward propagation reaches that layer, the layer's input and output data are automatically captured and stored for subsequent analysis and calculation of quantization factors.

[0063] The quantization factor of each layer of convolution operation is calculated using the mean square error method. The mean square errors of the weights, inputs, biases and outputs of the convolution layer are calculated in the integer range [-20, 20], and the value with the minimum mean square error is selected as the initial value of the quantization factor. In order to meet the hardware limitation, the quantization factor of each parameter is subjected to a truncation operation: for the output and bias, the factor is truncated in the range [-16, 15]; for the input and weight, it is truncated in the range [-8, 7] to ensure that the quantization factor can operate stably under the accuracy requirement of FPGA.

[0064] The output quantization factor of the current convolution layer is passed as the input quantization factor of the next layer to ensure the consistency of the quantization factor in the inter-layer transmission and improve the inference accuracy and performance stability of the quantized model.

[0065] Combined with the layer structure parameters of the model, the model information of each layer is loaded by traversing the module_name in the model dictionary, and the input and output data obtained by the hook function are used to calculate the quantization factor of the convolution layer input, output, weight and bias layer by layer.

[0066] As shown in Figure 4 , the overall output quantization factor is adjusted by analyzing the parent-child node relationship of the model residual network. Specifically, the output quantization factors of the parent node and the child node are added and averaged, and the result is ensured to be an integer by rounding, so as to obtain the optimized output quantization factor. Then, the input quantization factor is adjusted to ensure the transmission and consistency of the quantization factor in the residual network, and the balance between accuracy and performance is realized.

[0067] Combined with the weight quantization factor S w , the bias quantization factor S b and the output quantization factor S ofm of each layer, the weight-to-bias displacement quantization factor S w2b and the bias-to-output displacement quantization factor S b2o are initialized. Then, according to the storage unit capacity and calculation resource limitation of FPGA, the displacement quantization factor is calculated and adjusted to ensure that S w2b is in the integer interval [-5, 3], and S b2o meets the integer interval requirement of [5, 8] to optimize the hardware adaptability.

[0068] Through the above mean square error processing and adjustment, the quantization factor meeting the deployment requirements of FPGA can be obtained in the initialization stage, which ensures the quantization accuracy of the model and the adaptability of the FPGA hardware, and lays a foundation for subsequent model inference and deployment.

[0069] Perform reverse search and adjust the quantization of convolution operations and activation functions to optimize the selection of quantization factors for each layer. During the reverse search process, adjust the quantization of convolution operations and activation functions to optimize the selection of quantization factors for each layer. The specific process is as follows:

[0070] The search begins at the last layer of the ResNet model and proceeds layer by layer. For each convolutional layer searched, the layer is sequentially quantized by traversing the quantization factor combinations generated by the Cartesian product, while the remaining layers maintain full-precision inference. During the traversal process, the mean squared error of each combination is calculated using torch.mean_squared_error(), and the combination with the smallest error is selected as the optimal quantization factor for that layer.

[0071] After determining the optimal quantization factor for the current layer, the search continues to the previous layer. The previous layer's search process fixes the optimal quantization factor for the current layer, and the Cartesian product combination is traversed again to find the optimal quantization factor for the previous layer. This reverse search continues layer by layer, ultimately obtaining the optimal quantization factor configuration for the entire model, ensuring the best quantization effect.

[0072] After completing the quantization factor initialization and reverse retrieval, the process enters the simulation reasoning and optimization stage of the quantization factor.

[0073] Factor simulation inference is an independent process that supports FPGA simulation and quantization factor optimization. Since FPGA implements convolution calculation by programming and configuring internal circuits, which is different from the nn.Conv2d convolution calculation method in PyTorch, this embodiment makes specific modifications to the convolution calculation process in PyTorch to accurately simulate the FPGA calculation process. The specific steps are as follows:

[0074] FPGA designs use 64-bit-wide data buses and memory units. Therefore, when processing the convolutional layers of the ResNet18 model, if the number of input or output channels is less than a multiple of 64, the torch.nn.functional module's pad() function must be used to perform zero padding to expand the number of channels to a multiple of 64. This padding ensures aligned data transmission on the FPGA, optimizing storage and computational efficiency.

[0075] The bias terms of the convolutional layer are extracted and stored separately, and the bias parameter of the convolutional layer is set to False to disable automatic bias calculation. This approach ensures independent management of bias data and simplifies the calculation process of the convolutional layer.

[0076] Convert the convolution layer to a grouped convolution with a kernel size of 1. Set the number of groups to the product of the number of input channels and the number of output channels of the original convolution. Fill the corresponding grouped convolution kernel with the convolution kernel weights of each group, following the order of the convolution layer's output and input channels, to adapt to the FPGA's grouped convolution computation structure.

[0077] According to the group number requirement of group convolution, the number of channels of the input tensor is expanded to match the channel configuration of group convolution to ensure data consistency during the convolution calculation process.

[0078] Apply group convolution to the expanded input tensor and perform convolution calculation, and truncate the calculated output. Specifically, if the output has more than 2 7 The elements of the data are truncated to limit the output value to the specified range, adapt to the hardware limitations of the FPGA and prevent data overflow.

[0079] Add the previously saved bias to the convolution output. If the value of the element in the result exceeds 2 15 , it is truncated to ensure that the output is within the FPGA hardware limit and avoid data overflow.

[0080] After the initial quantization factor is set, a search range is determined for each quantization factor (set to an integer interval of ±5 of the initial quantization factor). PyTorch's itertools.product() function is used to generate all possible quantization factor combinations, forming a Cartesian product. For each combination, convolutional inference is performed on a simulated FPGA, and the result is compared with the full-precision model inference result by mean squared error (MSE). The combination with the lowest MSE is selected as the optimized quantization factor combination, thereby minimizing the impact of the quantization process on model accuracy.

[0081] The main operation process of the simulation reasoning and optimization stage of the quantization factor is as follows:

[0082] Use PyTorch's register_forward_hook() to register a forward hook function at each convolutional layer to monitor and obtain the input, output, and module name of each layer. Then, sort the module names in reverse order and load each convolutional layer and its quantization factor in reverse order according to the module_name in the model dictionary to support subsequent optimization operations.

[0083] Based on the initial quantization factor values ​​for each layer, the search range is determined to be an integer interval of ±5. Within this range, itertools.product() is called to generate all possible combinations of quantization factors for each layer, ensuring that the search for quantization factors is focused near the initial values, thereby improving the efficiency of quantization optimization.

[0084] Perform hardware adaptability screening on the generated quantization factor combinations and filter out combinations that exceed the FPGA quantization support range. For example, ensure that all quantization factor combinations comply with the FPGA's integer range restrictions (such as S w2b In the interval [-5,3], S b2o In the range [5,8]), only hardware-supported combinations are retained to ensure that the quantized model runs stably on the FPGA.

[0085] For each quantization combination that meets the requirements, perform simulation inference. Calculate the mean square error (MSE) between the simulation results and the saved full-precision model output, and compare the results using torch.mean_squared_error(). A smaller MSE indicates a smaller error between the quantized model and the full-precision model.

[0086] Compare the mean square error (MSE) results of all combinations and select the combination with the smallest MSE as the optimized quantization factor configuration. This combination ensures the best balance between model accuracy and performance in FPGA deployment.

[0087] During the inference process, only the convolution calculation method is adjusted, and the basic block structure of ResNet18 remains unchanged. Figure 3 When using the classifier module shown in the example, the torch.nn.AvgPool2d() called by the original model is replaced with a custom integer pooling function to ensure that the pooled output strictly conforms to the fixed-point format requirements of the FPGA, thereby supporting accurate reasoning in the FPGA hardware environment.

[0088] To simulate the quantized inference process, you first need to install and configure the necessary tools for the deep learning model in a GPU environment, including libraries such as PyTorch and TorchVision. Ensure these libraries are correctly installed using pip or conda to support model loading and forward inference. A script for the validation dataset is written in Python, and the quantized simulation process is implemented in the main program. By logically connecting these various code components, a complete quantized inference simulation framework is formed. Users can use command-line arguments (using the argparse library) to select whether to verify the accuracy of the quantization factors or obtain the optimized quantization factors for the current model.

[0089] After the program starts, the model loads the validation dataset from the specified path and performs inference batch by batch on the GPU. During this process, the console outputs the processing time and Top-1 and Top-5 accuracy of each image in real time, allowing you to observe the performance of the quantized model during inference. Inference time can be measured using the PyTorch torch.cuda.Event method, while accuracy can be calculated by calling the torch.topk() function to obtain the Top-1 and Top-5 accuracy performance of each image.

[0090] During runtime, the program can optionally generate a log file containing TraceIDs to record detailed inference results for each quantization factor combination. This includes the mean square error (MSE) of each quantization factor combination compared to the full-precision model inference result, which serves as an indicator of quantization error; the optimal quantization factor for the current layer; and the final set of optimal quantization factors for each layer, which is used for subsequent FPGA deployment optimization. After the log file is generated, the program automatically saves the quantization factor configuration to a local directory in .txt format and displays the file path in the console.

[0091] A large number of tests have verified the significant advantages of this embodiment in terms of quantization factor accuracy and efficiency. As shown in Table 1, the experimental data under different validation sets show that the accuracy of the quantization factors generated by this embodiment reached 70.86% on the ImageNet validation set, which is a significant improvement compared to traditional post-training quantization methods (usually with an accuracy between 50% and 60%). This improvement makes the performance of the present invention better than traditional methods on different test sets, with an average accuracy improvement of more than 10%, demonstrating its robustness and reliability in various scenarios.

[0092] Table 1 Comparison of model accuracy under different validation sets

[0093] Validation set Full-precision model accuracy The quantization accuracy of the present invention Traditional post-training quantization method ImageNet-validation set 75.50％ 70.86％ 58.72％ CIFAR-10 validation set 92.10％ 89.30％ 80.50％ CIFAR-100-validation set 68.50％ 65.00％ 59.80％ Tiny ImageNet-Validation Set 60.80％ 57.10％ 50.60％

[0094] In terms of inference efficiency, the optimization technology of this embodiment reduces the inference time for each image to approximately 150ms, which basically meets the requirements of FPGA real-time inference. Compared with the average inference time of 200ms to 250ms of traditional quantization methods, this embodiment demonstrates outstanding efficiency advantages. Under the inference load of multiple images, the throughput performance of this embodiment remains stable, effectively meeting the needs of high-concurrency applications, making it highly practical and competitive in embedded and edge computing scenarios.

[0095] In addition, this embodiment has achieved a high level of technical maturity, realizing and verifying the complete process of loading, fusion, quantization and simulation reasoning of the ResNet model. The key links of this process have been verified to be stable in actual applications, with high operability and adaptability, and are easy to integrate into different systems or hardware platforms. In particular, through the quantitative simulation reasoning method of this embodiment, the ResNet model can be simulated on the GPU before hardware verification, which greatly reduces the cost and time of hardware verification, shortens the development cycle by about 30%, and significantly improves development efficiency.

[0096] Aiming at the demands of FPGAs in embedded and edge computing, this implementation demonstrates unique advantages in addressing the limitations of traditional neural network quantization methods. Its flexible deployment enhances the model's applicability, providing a more competitive solution for embedded intelligent applications and promising broad market prospects.

[0097] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A ResNet network quantization and simulation inference method for FPGA deployment, characterized by: The following steps are involved: S1. Load the ResNet model and adjust the model structure, including the operation of integrating the BN layer, to adapt to FPGA deployment requirements; S2, read the full-precision image and perform preprocessing; Use the adjusted model to perform forward inference on the preprocessed image, and use hook functions to capture the input and output data of each layer; S3. Based on the captured data, the quantization factor calculation is completed layer by layer as the initial value of the model quantization factor; Specifically: Based on the input and output data of each convolutional layer, the mean square error of the weight, input, bias, and output of the convolutional layer is calculated in the integer range [-20, 20], and the value with the smallest mean square error is selected as the initial value of the quantization factor; To meet hardware limitations, the quantization factors of each parameter are truncated: for output and bias, the factors are truncated to the range of [-16, 15]; for input and weight, they are truncated to the range of [-8, 7]; Then, the output quantization factor of the current convolutional layer is passed as the input quantization factor of the next layer. Combined with the layer structure parameters of the model, by traversing the module_name in the model dictionary, the model information of each layer is loaded, and combined with the input and output data obtained by the hook function, the quantization factor calculation of the convolutional layer input, output, weight and bias is completed layer by layer. Finally, by sorting out the parent-child node relationship of the model residual network, the overall output quantization factor is adjusted. Specifically, the output quantization factors of the parent node and the child node are added and averaged, and the result is rounded to an integer to obtain the optimized output quantization factor, which is then used to adjust the input quantization factor. S4. Perform reverse search and adjust the quantization of convolution operations and activation functions to optimize the selection of quantization factors for each layer; specifically: The search starts from the last layer of the ResNet model and proceeds layer by layer. For each convolutional layer searched, the layer is quantized in sequence by traversing the quantization factor combination generated by the Cartesian product, while the other layers maintain full precision inference. During the traversal process, the mean squared error of each combination is calculated using torch.mean_squared_error(), and the combination with the smallest error is selected as the optimal quantization factor for the layer. After the optimal quantization factor of the current layer is determined, continue to search the previous layer. During the search process of the previous layer, the current layer is fixed as the optimal quantization factor, and the Cartesian product combination is traversed again to find the optimal quantization factor of the previous layer; Repeat the above steps, reverse search layer by layer, and finally obtain the optimal quantization factor configuration of the entire model; S5. Build an FPGA simulation environment, perform FPGA simulation reasoning, and optimize the quantization factor. Specifically: Use PyTorch's register_forward_hook() to register a forward hook function at each convolutional layer to monitor and obtain the input, output, and module name of each layer of the model; then, sort the module names in reverse order and load each convolutional layer and its quantization factor in reverse order according to the module_name in the model dictionary; Based on the value of the initial quantization factor of each layer, the search range is determined to be an integer interval of ±5. Within this range, itertools.product() is called to generate all possible combinations of quantization factors of each layer, ensuring that the search for quantization factors focuses on the initial value. Perform hardware adaptability screening on the generated quantization factor combinations and filter out combinations that exceed the FPGA quantization support range; For each quantitative combination that meets the conditions, perform simulation reasoning; Calculate the mean square error between the simulation results and the saved full-precision model output; Compare the mean square error results of all combinations and select the combination with the smallest mean square error as the optimized quantization factor configuration.

2. The ResNet network quantization and simulation inference method for FPGA deployment according to claim 1 is characterized in that: In step S1, load the ResNet model and adjust the model structure. The specific method is as follows: Load the ResNet18 model through TorchVision, including its learnable parameters and network structure; Extract the network structure source code from the package to achieve visual code display; After loading the model, the BN layer is fused through a custom fusion function, and the relevant parameters of the BN layer are reconstructed into the weights and biases of the convolutional layer through equations.

3. The ResNet network quantization and simulation inference method for FPGA deployment according to claim 1 is characterized in that: The method for preprocessing the image in step S2 includes operations of cropping, converting and normalizing the image.

4. The ResNet network quantization and simulation inference method for FPGA deployment according to claim 1 is characterized in that The inputs and outputs captured during the forward inference process in step S2 are used for the subsequent calculation of quantization factors and evaluation of model performance. The specific operation is: use PyTorch's register_forward_hook method to register a hook function at each convolutional layer. When forward propagation reaches this layer, the input and output data of this layer are automatically captured and stored.

5. The ResNet network quantization and simulation inference method for FPGA deployment according to claim 1 is characterized in that: Step S3 further includes: Combined with the weight quantization factor S of each layer w , bias quantization factor S b And the output quantization factor S ofm , initialize the displacement quantization factor S from weight to bias w2b and the offset quantization factor S to the output b2o Then, according to the storage unit capacity and computing resource limit of FPGA, the displacement quantization factor is calculated and adjusted to S w2b Truncated in the integer interval [-5,3], S b2o Truncated to the integer interval [5,8].

6. The ResNet network quantization and simulation inference method for FPGA deployment according to claim 1 is characterized in that: In step S5, an FPGA simulation environment is constructed, PyTorch is used for FPGA simulation, and the convolution calculation process in PyTorch is modified. The specific method is as follows: When processing the convolutional layers of the ResNet18 model, if the number of input or output channels is less than a multiple of 64, use the pad() function of the torch.nn.functional module to perform zero padding to expand the number of channels to a multiple of 64; Extract and store the bias of the convolutional layer separately, and set the bias parameter of the convolutional layer to False to turn off the automatic calculation of the bias; Convert the convolution layer to a grouped convolution with a convolution kernel size of 1, and set the number of groups to the product of the number of input channels and the number of output channels of the original convolution; and fill the convolution kernel weights of each group into the corresponding grouped convolution kernel one by one according to the order of the output channels and input channels of the convolution layer; According to the group number requirement of the group convolution, the number of channels of the input tensor is expanded to match the channel configuration of the group convolution; Apply group convolution to the expanded input tensor and perform convolution calculations. For the calculated output, more than 2 7 The elements are truncated; Add the previously saved bias to the convolution output, and truncate any elements in the result that exceed 215.

7. The ResNet network quantization and simulation inference method for FPGA deployment according to claim 6, characterized in that: During inference, only the convolution calculation method is adjusted, keeping the basic block structure of ResNet18 unchanged; when inferring to the classifier module, the torch.nn.AvgPool2d() called by the original model is replaced with a custom integer pooling function.

Images