ResNet network quantification and simulation reasoning method oriented to FPGA deployment

Abstract

The invention discloses a ResNet network quantification and simulation reasoning method oriented to FPGA deployment. The ResNet network quantification and simulation reasoning method is used for effectively relieving the problems of numerical value overflow and precision reduction caused by direct rounding-off in a traditional quantification method. According to the method, the weight and the activation value of the ResNet model are subjected to quantitative conversion of the adaptive FPGA, so that the model keeps higher precision under the condition of limited FPGA resources. Besides, the invention designs a simulation reasoning method without hardware verification, the performance and precision of the quantitative model can be tested in advance in an FPGA simulation environment, and complex steps of hardware debugging and verification are reduced, so that the development efficiency is greatly improved, and the resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence hardware acceleration technology, and in particular to a ResNet network quantization and simulation reasoning method for FPGA deployment. Background Art

[0002] With the rapid development of deep learning and neural network technology, the residual network structure (ResNet) has significantly improved the performance of deep neural networks in the field of deep learning. The introduction of the residual module effectively alleviates the gradient vanishing and gradient exploding problems in deep networks, thereby enhancing the stability of the training process. At the same time, residual network structures such as ResNet18, with their small number of parameters and excellent classification performance, perform well on hardware with rich computing resources such as GPUs, and have been widely used in various computing tasks.

[0003] However, due to the limitations of FPGA in computing and storage resources, it is challenging to deploy ResNet directly on FPGA to achieve efficient inference. Traditional neural network quantization technology reduces the consumption of computing and storage resources by converting the weights and biases of the model from floating point numbers to low-precision integers. Neural network quantization is mainly divided into quantization aware training (QAT) and post-training quantization (PTQ). In the post-training quantization method, the quantization range is usually determined by least squares or histogram to minimize the loss of accuracy during the quantization process. Although post-training quantization technology can effectively reduce the computing and storage requirements of neural networks, direct rounding of floating point numbers may cause model weight overflow. When deployed on resource-constrained hardware, this overflow may reduce the operation speed and energy efficiency, thereby affecting the real-time processing capability of the device.

[0004] Many studies have explored and optimized the post-training quantization method of the ResNet model. For example, a semi-supervised pedestrian re-identification model, method and device disclosed in publication number CN113657267A proposes to introduce an average teacher model for mutual learning and supervision in the pre-training stage of the ResNet model to improve the model accuracy and make it have better performance before loading into the FPGA. Although this method alleviates the overfitting problem of the ResNet model to a certain extent, its optimization focuses on the distillation and pseudo-label generation of the full-precision model, and does not fully consider the resource limitations of the FPGA. An FPGA-based intelligent capture system and method disclosed in publication number CN1162302A uses the ResNet18 model for training and uses TensorRT for quantization and compilation to generate a quantized model suitable for FPGA edge device deployment. TensorRT quantization quantizes weights and activations into INT8 format by calibrating the activation range of the data statistical model, significantly improving the inference speed and reducing memory requirements. However, if the calibration data set is insufficient or unrepresentative, TensorRT quantization may result in a large loss of accuracy.

[0005] In addition, the current model deployment process for FPGAs usually directly loads the quantized model onto the hardware for verification to evaluate its performance in actual applications. This method is not only time-consuming, but also due to the lack of a comprehensive evaluation of the model's operating environment, it may lead to large differences in the inference efficiency and energy efficiency of the quantized model on different hardware platforms. Summary of the invention

[0006] The purpose of the present invention is to provide a ResNet network quantization and simulation inference method for FPGA deployment, aiming to optimize the model performance while fully considering the resource limitations of FPGA hardware and reducing the dependence on calibration data sets, so as to achieve a more efficient and stable model deployment effect.

[0007] In order to achieve the above-mentioned purpose, the present invention provides a ResNet network quantization and simulation reasoning method for FPGA deployment, comprising the following steps:

[0008] S1. Load the ResNet model and adjust the model structure, including the operation of integrating the BN layer, to adapt to the FPGA deployment requirements;

[0009] S2, read the full-precision image and perform preprocessing; use the adjusted model to perform forward inference on the preprocessed image, and use the hook function to capture the input and output data of each layer;

[0010] S3. Based on the captured data, the quantitative factor calculation is completed layer by layer as the initial value of the model quantitative factor;

[0011] S4, perform reverse retrieval and adjust the quantization of the convolution operation and activation function to optimize the selection of quantization factors of each layer;

[0012] S5. Build an FPGA simulation environment, perform simulation FPGA reasoning, and optimize the selection of quantization factors.

[0013] A further preferred technical solution of the present invention is that in step S1, a ResNet model is loaded and the model structure is adjusted, and the specific method is as follows:

[0014] Load the ResNet18 model through TorchVision, including its learnable parameters and network structure;

[0015] Extract the network structure source code from the package to achieve visual code display;

[0016] After loading the model, the BN layer is fused through a custom fusion function, and the relevant parameters of the BN layer are reconstructed into the weights and biases of the convolutional layer through equations.

[0017] Preferably, the method for preprocessing the image in step S2 includes operations of cropping, converting and normalizing the image.

[0018] Preferably, the input and output captured during the forward reasoning process in step S2 are used for the subsequent calculation of quantization factors and the evaluation of model performance. The specific operation is: use the register_forward_hook method of PyTorch to register the hook function in each convolutional layer, and when the forward propagation reaches the layer, the input and output data of the layer are automatically captured and stored.

[0019] Preferably, in step S3, the quantization factor calculation is completed layer by layer based on the captured data, and the specific method is:

[0020] Based on the input and output data of each convolutional layer, the mean square error of the weight, input, bias and output of the convolutional layer is calculated in the integer range [-20, 20], and the value with the smallest mean square error is selected as the initial value of the quantization factor; and in order to meet the hardware limitations, the quantization factor of each parameter is truncated: for output and bias, the factor is truncated in the range of [-16, 15]; for input and weight, it is truncated in the range of [-8, 7];

[0021] Then, the output quantization factor of the current convolutional layer is passed as the input quantization factor of the next layer. Combined with the layer structure parameters of the model, the model information of each layer is loaded by traversing the module_name in the model dictionary, and combined with the input and output data obtained by the hook function, the quantization factor calculation of the convolutional layer input, output, weight and bias is completed layer by layer.

[0022] Finally, by sorting out the parent-child node relationship of the model residual network, the overall output quantization factor is adjusted. Specifically, the output quantization factors of the parent node and the child node are added and averaged, and the result is rounded to an integer to obtain the optimized output quantization factor, which is then used to adjust the input quantization factor.

[0023] Preferably, step S3 further comprises:

[0024] Combine the weight quantization factor S of each layer w , bias quantization factor S b And the output quantization factor S ofm , initialize the displacement quantization factor S from weight to bias w2b and the offset quantization factor S to the output b2o Then, according to the storage unit capacity and computing resource limit of FPGA, the displacement quantization factor is calculated and adjusted to S w2b Truncated to the integer interval [-5,3], S b2o Truncated to the integer interval [5,8].

[0025] Preferably, in step S4, a reverse search is performed to adjust the quantization of the convolution operation and the activation function to optimize the selection of the quantization factor of each layer. The specific method is:

[0026] The retrieval starts from the last layer of the ResNet model and proceeds layer by layer. For each convolutional layer retrieved, the layer is quantized in turn by traversing the quantization factor combination generated by the Cartesian product, and the other layers maintain full-precision reasoning. During the traversal process, the mean square error of each combination is calculated by torch.mean_squared_error(), and the combination with the smallest error is selected as the optimal quantization factor for the layer.

[0027] After the optimal quantization factor of the current layer is determined, continue to search for the previous layer. During the search process of the previous layer, the current layer is fixed as the optimal quantization factor, and the Cartesian product combination is traversed again to find the optimal quantization factor of the previous layer;

[0028] Repeat the above steps, perform reverse retrieval layer by layer, and finally obtain the optimal quantization factor configuration of the entire model.

[0029] Preferably, in step S5, FPGA simulation reasoning is performed to optimize the quantization factor, and the specific method is as follows:

[0030] Use PyTorch's register_forward_hook() to register the forward hook function at each convolutional layer to monitor and obtain the input, output, and module name of each layer of the model; then, sort the module names in reverse order and load each convolutional layer and its quantization factor in reverse order according to the module_name in the model dictionary;

[0031] According to the value of the initial quantization factor of each layer, the search range is determined to be an integer interval of ±5. In this range, itertools.product() is called to generate all possible combinations of quantization factors of each layer to ensure that the search of quantization factors is focused near the initial value.

[0032] Perform hardware adaptability screening on the generated quantization factor combinations and filter out combinations that exceed the FPGA quantization support range;

[0033] For each quantization combination that meets the conditions, perform simulation reasoning; calculate the mean square error between the simulation results and the saved full-precision model output; compare the mean square error results of all combinations, and select the combination with the smallest mean square error as the optimized quantization factor configuration.

[0034] Preferably, in step S5, an FPGA simulation environment is constructed, PyTorch is used for FPGA simulation, and the convolution calculation process in PyTorch is modified. The specific method is as follows:

[0035] When processing the convolutional layer of the ResNet18 model, if the number of input or output channels is less than a multiple of 64, use the pad() function of the torch.nn.functional module to perform zero padding to expand the number of channels to a multiple of 64;

[0036] Extract and store the bias item of the convolutional layer separately, and set the bias parameter of the convolutional layer to False to turn off the automatic calculation of the bias;

[0037] Convert the convolution layer to a grouped convolution with a convolution kernel size of 1, and set the number of groups to the product of the number of input channels and the number of output channels of the original convolution; and fill the convolution kernel weights of each group into the corresponding grouped convolution kernel one by one according to the order of the output channels and input channels of the convolution layer;

[0038] According to the group number requirement of the group convolution, the number of channels of the input tensor is expanded to match the channel configuration of the group convolution;

[0039] Apply grouped convolution to the expanded input tensor and perform convolution calculations. For more than 2 7 The elements of are truncated;

[0040] Add the previously saved bias to the convolution output, and truncate the result if the element value exceeds 215.

[0041] Preferably, during the inference process, only the convolution calculation method is adjusted to keep the basic block structure of ResNet18 unchanged; when inferring to the classifier module, the torch.nn.AvgPool2d() called by the original model is replaced with a custom integer pooling function.

[0042] Beneficial effects: Compared with the prior art, the ResNet network quantization and simulation inference method for FPGA deployment proposed in the present invention has the following significant technical effects:

[0043] 1. Improve model accuracy: Through precise calculation and optimization of quantization factors, the quantization model accuracy of the present invention on validation sets such as ImageNet has reached 70.86%, which is significantly improved compared to 58.72% of the traditional post-training quantization method. The average accuracy improvement is more than 10%, showing robustness and reliability in a variety of scenarios.

[0044] 2. Improve inference efficiency: The optimization technology of the present invention shortens the inference time of each image to about 150ms, which basically meets the requirements of FPGA real-time inference. Compared with the average inference time of 200ms to 250ms of the traditional quantization method, the present invention shows an excellent efficiency advantage.

[0045] 3. Enhanced hardware adaptability: By accurately adjusting and optimizing the quantization factors, the present invention ensures the stable operation of the quantization model on the FPGA, avoids the problems of data overflow and reduced computing speed, and improves hardware adaptability.

[0046] 4. Reduce resource consumption: The present invention reduces the resource consumption of model deployment and reduces development costs by reducing the dependence on the calibration data set.

[0047] 5. Reduce the development cycle: By completing simulation reasoning on the GPU, the present invention greatly reduces the cost and time of hardware verification, shortens the development cycle by about 30%, and significantly improves development efficiency.

[0048] 6. Improve the applicability of the model: The flexible deployment method of the present invention improves the applicability of the model, provides a more competitive solution for embedded intelligent applications, and has broad market prospects.

[0049] 7. Optimize storage and computing efficiency: Through specific FPGA simulation modifications, such as zero padding, independent management of bias terms, grouped convolution, etc., the present invention optimizes storage and computing efficiency and ensures aligned transmission of data on FPGA.

[0050] 8. Enhance the practicality and competitiveness of the model: In embedded and edge computing scenarios, the optimization technology of the present invention effectively meets the needs of high-concurrency applications, making it more practical and competitive in practical applications.

[0051] 9. Achieve technology maturity: The present invention has achieved a high level of technology maturity, realizing and verifying the complete process of loading, fusion, quantization and simulation reasoning of the ResNet model. It has high operability and adaptability, and is easy to integrate into different systems or hardware platforms.

[0052] Through the description of the above technical effects, it is shown that the present invention not only theoretically proposes a new ResNet network quantization and simulation inference method for FPGA deployment, but also verifies its effectiveness in practice, providing a new solution for the efficient deployment of deep learning models on FPGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the ResNet network quantization and simulation reasoning method for FPGA deployment in the present invention.

[0054] Figure 2 This is the ResNet18 network structure after BN layer fusion in an embodiment of the present invention.

[0055] Figure 3 Schematic diagram of the classifier module in an embodiment of the present invention.

[0056] Figure 4 The figure is a flowchart of averaging the quantization factors output by the parent and child nodes in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0058] Embodiment: This embodiment aims to solve the problems of precision loss, low computational efficiency, and poor hardware adaptability faced by the existing ResNet model in FPGA hardware deployment. To address these problems, this embodiment proposes a ResNet quantization and simulation inference method specifically for FPGA deployment, so as to effectively alleviate the numerical overflow and precision loss problems caused by direct rounding in traditional quantization methods. Figure 1-Figure 4 The solution of this embodiment is described in detail.

[0059] There are three key parts in this embodiment, namely, quantization factor initialization, quantization inference process simulation and quantization factor optimization. Figure 1 The specific implementation is as follows:

[0060] The quantization factor initialization part requires a ResNet model. In this embodiment, the ResNet18 model is loaded through TorchVision, including its learnable parameters and network structure. In order to facilitate the analysis and control of the inference process, the network structure source code is extracted from the package to achieve visual code display. After loading the model, the BN layer is fused through a custom fusion function, and the relevant parameters of the BN layer are reconstructed into the weights and biases of the convolution layer through equations. The network structure of the ResNet18 model after the BN layer fusion is as follows: Figure 2 Next, the mean square error method is used to calculate the quantization factor layer by layer, including the convolution weight S of each layer. w , bias S b , weight to bias shift S w2b , bias to output displacement S b2o , enter S ifm , output S ofm Quantization factor. The specific steps and calculation method of the quantization factor are as follows:

[0061] Read a full-precision image from the specified path and preprocess it using TorchVision's built-in Transforms library, including cropping to 224x224 pixels, converting to the Tensor data type, and normalizing the pixel values ​​to the [0,1] range.

[0062] The preprocessed image is input into the BN fused model for full-precision forward inference. The input and output of each layer are recorded by registering a hook function on each convolutional layer. The specific operation is to use the register_forward_hook method of PyTorch to register a hook function on each convolutional layer. When the forward propagation reaches the layer, the input and output data of the layer are automatically captured and stored for subsequent analysis and calculation of quantization factors.

[0063] The mean square error method is used to calculate the quantization factor of each convolution operation. The mean square error of the weight, input, bias, and output of the convolution layer is calculated in the integer range [-20, 20], and the value with the smallest mean square error is selected as the initial value of the quantization factor. To meet the hardware limitations, the quantization factor of each parameter is truncated: for output and bias, the factor is truncated in the range of [-16, 15]; for input and weight, it is truncated in the range of [-8, 7] to ensure that the quantization factor can run stably under the accuracy requirements of the FPGA.

[0064] The output quantization factor of the current convolutional layer is passed as the input quantization factor of the next layer to ensure the consistency of the quantization factor transmission between layers and improve the reasoning accuracy and performance stability of the model after quantization.

[0065] Combined with the layer structure parameters of the model, by traversing the module_name in the model dictionary, loading the model information of each layer, and combining the input and output data obtained by the hook function, the quantization factor calculation of the convolution layer input, output, weight and bias is completed layer by layer.

[0066] like Figure 4 As shown in the figure, by combing the parent-child node relationship of the model residual network, the overall output quantization factor is adjusted. Specifically, the output quantization factors of the parent node and the child node are added and averaged, and the result is rounded to an integer to obtain the optimized output quantization factor. Then, the input quantization factor is adjusted to ensure the transmission and consistency of the quantization factor in the residual network, achieving a balance between accuracy and performance.

[0067] Combine the weight quantization factor S of each layer w , set the quantization factor S b And the output quantization factor S ofm , initialize the displacement quantization factor S from weight to bias w2b and the offset quantization factor S to the output b2o Then, according to the storage unit capacity and computing resource limit of FPGA, the displacement quantization factor is calculated and adjusted to ensure S w2b In the integer interval [-5,3], and S b2o Then the integer interval requirement of [5,8] is satisfied to optimize hardware adaptability.

[0068] Through the above-mentioned mean square error processing and adjustment, the present invention can obtain quantization factors that meet the FPGA deployment requirements in the initialization stage, ensure the quantization accuracy of the model and the adaptability of the FPGA hardware, and lay the foundation for subsequent model reasoning and deployment.

[0069] Perform reverse retrieval to adjust the quantization of convolution operations and activation functions to optimize the selection of quantization factors for each layer. During the reverse retrieval process, adjust the quantization of convolution operations and activation functions to optimize the selection of quantization factors for each layer. The specific process is as follows:

[0070] The retrieval starts from the last layer of the ResNet model and proceeds layer by layer. For each retrieved convolutional layer, the layer is quantized in turn by traversing the quantization factor combination generated by the Cartesian product, and the other layers are kept in full precision inference. During the traversal process, the mean square error of each combination is calculated by torch.mean_squared_error(), and the combination with the smallest error is selected as the optimal quantization factor for the layer.

[0071] After the optimal quantization factor of the current layer is determined, continue to search for the previous layer. The search process of the previous layer fixes the current layer as the optimal quantization factor, and traverses the Cartesian product combination again to find the optimal quantization factor of the previous layer. In this way, reverse search layer by layer is performed to finally obtain the optimal quantization factor configuration of the entire model to ensure the best quantization effect.

[0072] After completing the quantization factor initialization and reverse retrieval, the process enters the simulation reasoning and optimization stage of the quantization factor.

[0073] The factor simulation reasoning is an independent process that provides support for FPGA simulation and quantization factor optimization. Since FPGA implements convolution calculation by programming and configuring internal circuits, which is different from the nn.Conv2d convolution calculation method in PyTorch, in order to accurately simulate the FPGA calculation process, this embodiment makes specific modifications to the convolution calculation process in PyTorch. The specific steps are as follows:

[0074] The FPGA design uses a 64-bit wide data bus and storage unit, so when processing the convolutional layer of the ResNet18 model, if the number of input or output channels is less than a multiple of 64, the pad() function of the torch.nn.functional module needs to be used for zero padding to expand the number of channels to a multiple of 64. This padding ensures aligned transmission of data on the FPGA, optimizing storage and computing efficiency.

[0075] The bias items of the convolutional layer are extracted and stored separately, and the bias parameter of the convolutional layer is set to False to turn off the automatic calculation of the bias. This processing method ensures independent management of bias data and simplifies the calculation process of the convolutional layer.

[0076] Convert the convolution layer to a grouped convolution with a convolution kernel size of 1, and set the number of groups to the product of the number of input channels and the number of output channels of the original convolution. According to the order of the output channels and input channels of the convolution layer, fill the convolution kernel weights of each group into the corresponding grouped convolution kernel one by one to adapt to the grouped convolution calculation structure of the FPGA.

[0077] According to the group number requirement of group convolution, the number of channels of the input tensor is expanded to match the channel configuration of group convolution to ensure data consistency during the convolution calculation process.

[0078] Apply grouped convolution to the expanded input tensor and perform convolution calculations, and truncate the calculated output results. Specifically, if more than 2 7 The elements of the output are truncated to limit the output value to the specified range, adapt to the hardware limitations of the FPGA and prevent data overflow.

[0079] Add the previously saved bias to the convolution output. If the value of the element in the result exceeds 2 15 , it is truncated to ensure that the output is within the FPGA hardware limit and avoid data overflow.

[0080] After the initial setting of the quantization factor is completed, the search range is determined for each quantization factor (set to the integer interval of ±5 of the initial quantization factor), and PyTorch's itertools.product() is used to generate all possible quantization factor combinations to form a Cartesian product. For each combination, the convolution inference of the simulated FPGA is performed, and its result is compared with the full-precision model inference result by mean square error (MSE) to select the combination with the smallest mean square error as the optimized quantization factor combination, thereby minimizing the impact of the quantization process on the model accuracy.

[0081] The main operation process of the simulation reasoning and optimization stage of the quantization factor is as follows:

[0082] Use PyTorch's register_forward_hook() to register the forward hook function at each convolution layer to monitor and obtain the input, output, and module name of each layer of the model. Then, sort the module names in reverse order and load each convolution layer and its quantization factor in reverse order according to the module_name in the model dictionary to support subsequent optimization operations.

[0083] According to the value of the initial quantization factor of each layer, the search range is determined to be an integer interval of ±5. In this range, itertools.product() is called to generate all possible combinations of quantization factors of each layer to ensure that the search of quantization factors is focused near the initial value to improve the efficiency of quantization optimization.

[0084] Perform hardware adaptability screening on the generated quantization factor combinations and filter out combinations that exceed the FPGA quantization support range. For example, ensure that all quantization factor combinations meet the integer range restrictions of the FPGA (such as S w2b In the interval [-5,3], S b2o In the interval [5,8]), only combinations supported by hardware are retained to ensure that the quantized model runs stably on the FPGA.

[0085] For each quantization combination that meets the conditions, perform simulation inference. Calculate the mean square error (MSE) between the simulation results and the saved full-precision model output, and use torch.mean_squared_error() to compare the results. The smaller the mean square error, the smaller the error between the quantized model and the full-precision model.

[0086] Compare the mean square error results of all combinations and select the combination with the smallest mean square error as the optimized quantization factor configuration. This combination ensures the best balance between model accuracy and performance in FPGA deployment.

[0087] During the inference process, only the convolution calculation method is adjusted, and the basic block structure of ResNet18 remains unchanged. Figure 3 When using the classifier module shown in the figure, the torch.nn.AvgPool2d() called by the original model is replaced with a custom integer pooling function to ensure that the output after pooling strictly conforms to the fixed-point format requirements of the FPGA, thereby supporting precise reasoning in the FPGA hardware environment.

[0088] When performing a simulation of the quantitative reasoning process, you first need to install and configure the necessary tools for the deep learning model in the GPU environment, including libraries such as PyTorch and TorchVision. Make sure these libraries are correctly installed using pip or conda to support model loading and forward reasoning. Use Python to write a script for the validation dataset and implement the quantitative simulation process in the main program. By connecting the logic of each part of the code in series, a complete quantitative reasoning simulation framework is formed. Users can choose to verify the accuracy of the quantization factor or obtain the optimized quantization factor of the current model through command line parameters (using the argparse library).

[0089] After the program is started, the model will load the validation dataset from the specified path and perform inference on the GPU batch by batch. During this process, the console outputs the processing time and Top-1 and Top-5 accuracy of each image in real time to observe the performance of the quantized model during inference. The inference time can be measured through the torch.cuda.Event method of PyTorch, and the accuracy can be calculated by calling the torch.topk() function to obtain the accuracy performance of each image in the Top-1 and Top-5 indicators.

[0090] When the program is running, you can choose to generate a log file containing TraceID to record the detailed inference results of each quantization factor combination, including: the mean square error (MSE) of each quantization factor combination and the full-precision model inference result to evaluate the quantization error; the optimal quantization factor of the current layer; the optimal quantization factor set finally selected for each layer for subsequent FPGA deployment optimization. After the log file is generated, the program automatically saves the quantization factor configuration to the local path in the .txt file format and displays the path of the file in the console.

[0091] A large number of tests have verified the significant advantages of this embodiment in terms of quantization factor accuracy and efficiency. As shown in Table 1, the experimental data under different validation sets show that the accuracy of the quantization factor generated by this embodiment reaches 70.86% on the ImageNet validation set, which is significantly improved compared to the traditional post-training quantization method (usually with an accuracy of 50% to 60%). This improvement makes the performance of the present invention better than the traditional method on different test sets, with an average accuracy improvement of more than 10%, showing its robustness and reliability in a variety of scenarios.

[0092] Table 1 Comparison of model accuracy under different validation sets

[0093] Validation set Full precision model accuracy The quantification accuracy of the present invention Traditional post-training quantization method ImageNet-Validation Set 75.50％ 70.86％ 58.72％ CIFAR-10 - Validation Set 92.10％ 89.30％ 80.50％ CIFAR-100 - Validation Set 68.50％ 65.00％ 59.80％ Tiny ImageNet-Validation Set 60.80％ 57.10％ 50.60％

[0094] In terms of inference efficiency, the optimization technology of this embodiment shortens the inference time of each image to about 150ms, which basically meets the requirements of FPGA real-time inference. Compared with the average inference time of 200ms to 250ms of the traditional quantization method, this embodiment shows an excellent efficiency advantage. Under the inference load of multiple images, the throughput performance of this embodiment remains stable, effectively meeting the needs of high-concurrency applications, making it highly practical and competitive in embedded and edge computing scenarios.

[0095] In addition, this embodiment has reached a high level of technical maturity, and has realized and verified the complete process of loading, fusion, quantization and simulation reasoning of the ResNet model. The key links of this process have verified its stability in practical applications, and have high operability and adaptability, which is easy to integrate into different systems or hardware platforms. In particular, through the quantitative simulation reasoning method of this embodiment, the ResNet model can be simulated on the GPU before hardware verification, which greatly reduces the cost and time of hardware verification, shortens the development cycle by about 30%, and significantly improves development efficiency.

[0096] In response to the needs of FPGA in embedded and edge computing, this embodiment shows unique advantages in solving the problem of limited application of traditional neural network quantization methods. Its flexible deployment method improves the applicability of the model, provides a more competitive solution for embedded intelligent applications, and has broad market prospects.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ResNet network quantization and simulation inference method for FPGA deployment, characterized in that: The following steps are involved: S1. Load the ResNet model and adjust the model structure, including the operation of integrating the BN layer, to adapt to the FPGA deployment requirements; S2, read the full-precision image and perform preprocessing; Use the adjusted model to perform forward inference on the preprocessed image, and use hook functions to capture the input and output data of each layer; S3. Based on the captured data, the quantitative factor calculation is completed layer by layer as the initial value of the model quantitative factor; S4, perform reverse retrieval and adjust the quantization of the convolution operation and activation function to optimize the selection of quantization factors of each layer; S5. Build an FPGA simulation environment, perform simulation FPGA reasoning, and optimize the selection of quantization factors.

2. The ResNet network quantization and simulation reasoning method for FPGA deployment according to claim 1 is characterized in that: In step S1, the ResNet model is loaded and the model structure is adjusted. The specific method is as follows: Load the ResNet18 model through TorchVision, including its learnable parameters and network structure; Extract the network structure source code from the package to achieve visual code display; After loading the model, the BN layer is fused through a custom fusion function, and the relevant parameters of the BN layer are reconstructed into the weights and biases of the convolutional layer through equations.

3. The ResNet network quantization and simulation reasoning method for FPGA deployment according to claim 1, characterized in that: The method for preprocessing the image in step S2 includes operations of cropping, converting and normalizing the image.

4. The ResNet network quantization and simulation reasoning method for FPGA deployment according to claim 1 is characterized in that: The input and output captured during the forward reasoning in step S2 are used for the subsequent calculation of quantization factors and evaluation of model performance. The specific operation is: use the register_forward_hook method of PyTorch to register the hook function in each convolutional layer. When the forward propagation reaches the layer, the input and output data of the layer are automatically captured and stored.

5. The ResNet network quantization and simulation reasoning method for FPGA deployment according to claim 4 is characterized in that: In step S3, based on the captured data, the quantization factor calculation is completed layer by layer. The specific method is: Based on the input and output data of each convolutional layer, the mean square error of the weight, input, bias and output of the convolutional layer is calculated in the integer range [-20, 20], and the value with the smallest mean square error is selected as the initial value of the quantization factor; To meet hardware limitations, the quantization factors of each parameter are truncated: for output and bias, the factors are truncated to the range [-16, 15]; for input and weight, they are truncated to the range [-8, 7]; Then, the output quantization factor of the current convolutional layer is passed as the input quantization factor of the next layer. Combined with the layer structure parameters of the model, the model information of each layer is loaded by traversing the module_name in the model dictionary, and combined with the input and output data obtained by the hook function, the quantization factor calculation of the convolutional layer input, output, weight and bias is completed layer by layer. Finally, by sorting out the parent-child node relationship of the model residual network, the overall output quantization factor is adjusted. Specifically, the output quantization factors of the parent node and the child node are added and averaged, and the result is rounded to an integer to obtain the optimized output quantization factor, which is then used to adjust the input quantization factor.

6. The ResNet network quantization and simulation reasoning method for FPGA deployment according to claim 5, characterized in that: Step S3 also includes: Combine the weight quantization factor S of each layer w , bias quantization factor S b And the output quantization factor S ofm , initialize the displacement quantization factor S from weight to bias w2b and the offset quantization factor S to the output b2o Then, according to the storage unit capacity and computing resource limit of FPGA, the displacement quantization factor is calculated and adjusted to S w2b Truncated to the integer interval [-5,3], S b2o Truncated to the integer interval [5,8].

7. The ResNet network quantization and simulation reasoning method for FPGA deployment according to claim 1, characterized in that: In step S4, reverse retrieval is performed to adjust the quantization of the convolution operation and the activation function to optimize the selection of the quantization factor of each layer. The specific method is as follows: The retrieval starts from the last layer of the ResNet model and proceeds layer by layer. For each convolutional layer retrieved, the layer is quantized in turn by traversing the quantization factor combination generated by the Cartesian product, and the other layers maintain full-precision reasoning. During the traversal process, the mean square error of each combination is calculated by torch.mean_squared_error(), and the combination with the smallest error is selected as the optimal quantization factor for the layer. After the optimal quantization factor of the current layer is determined, continue to search for the previous layer. During the search process of the previous layer, the current layer is fixed as the optimal quantization factor, and the Cartesian product combination is traversed again to find the optimal quantization factor of the previous layer; Repeat the above steps, perform reverse retrieval layer by layer, and finally obtain the optimal quantization factor configuration of the entire model.

8. The ResNet network quantization and simulation reasoning method for FPGA deployment according to claim 1, characterized in that: In step S5, FPGA simulation reasoning is performed to optimize the quantization factor. The specific method is as follows: Use PyTorch's register_forward_hook() to register the forward hook function at each convolutional layer to monitor and obtain the input, output, and module name of each layer of the model; then, sort the module names in reverse order and load each convolutional layer and its quantization factor in reverse order according to the module_name in the model dictionary; According to the value of the initial quantization factor of each layer, the search range is determined to be an integer interval of ±5. In this range, itertools.product() is called to generate all possible combinations of quantization factors of each layer to ensure that the search of quantization factors is focused near the initial value. Perform hardware adaptability screening on the generated quantization factor combinations and filter out combinations that exceed the FPGA quantization support range; For each quantitative combination that meets the conditions, perform simulation reasoning; Calculate the mean square error between the simulation results and the saved full-precision model output; Compare the mean square error results of all combinations and select the combination with the smallest mean square error as the optimized quantization factor configuration.

9. The ResNet network quantization and simulation reasoning method for FPGA deployment according to claim 1, characterized in that: In step S5, an FPGA simulation environment is constructed, PyTorch is used for FPGA simulation, and the convolution calculation process in PyTorch is modified. The specific method is as follows: When processing the convolutional layer of the ResNet18 model, if the number of input or output channels is less than a multiple of 64, use the pad() function of the torch.nn.functional module to perform zero padding to expand the number of channels to a multiple of 64; Extract and store the bias item of the convolutional layer separately, and set the bias parameter of the convolutional layer to False to turn off the automatic calculation of the bias; Convert the convolution layer to a grouped convolution with a convolution kernel size of 1, and set the number of groups to the product of the number of input channels and the number of output channels of the original convolution; and fill the convolution kernel weights of each group into the corresponding grouped convolution kernel one by one according to the order of the output channels and input channels of the convolution layer; According to the group number requirement of the group convolution, the number of channels of the input tensor is expanded to match the channel configuration of the group convolution; Apply grouped convolution to the expanded input tensor and perform convolution calculations. For more than 2 7 The elements of are truncated; Add the previously saved bias to the convolution output, and truncate the result if the element value exceeds 215.

10. The ResNet network quantization and simulation reasoning method for FPGA deployment according to claim 9, characterized in that: During the inference process, only the convolution calculation method is adjusted to keep the basic block structure of ResNet18 unchanged; when inferring to the classifier module, the torch.nn.AvgPool2d() called by the original model is replaced with a custom integer pooling function.

Images