An improved neural network hardware acceleration method and device based on FPGA

Abstract

The application relates to the technical field of image processing, in particular to an improved neural network hardware acceleration method and device based on FPGA, which comprises the following steps: training an SSD_MobilenetV1 network through transfer learning, data enhancement, multi-scale training and cosine annealing; performing structural pruning on the trained SSD_MobilenetV1 network, taking a convolution kernel or each network layer as a basic unit for pruning; adopting a QAT algorithm, introducing a pseudo-quantization operation for training, and using the pseudo-quantization operation to simulate the error of a quantization process; and converting the quantized SSD_MobilenetV1 network into a calculation graph. The application simultaneously uses an FPGA and an ARM processor to perform reasoning on a model, executes time-consuming convolution operators in the convolution network model in the FPGA, and executes other operators in the ARM processor, so that fast reasoning of the network model can be realized, power consumption is low, and the network model is beneficial to deployment to a terminal.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an improved neural network hardware acceleration method and apparatus based on FPGA. Background Technology

[0002] Neural networks have been widely used in image processing for target recognition, but image processing models based on deep convolutional neural networks often have a large number of parameters and computational demands. Traditional general-purpose processors, such as central processing units (CPUs), have rich control logic capable of various operations, thus supporting the operation of various models. However, they have only a small number of logic computing units and limited computing resources, resulting in low efficiency and slow speed when used for network model inference. Parallel processors, such as graphics processing units (GPUs), can perform parallel and rapid computation of data, accelerating model inference. However, they are large, consume a lot of power, and are expensive, making them difficult to apply in terminal applications. Specialized application-specific integrated circuits (ASICs) can allocate various resources autonomously, achieving faster computing speeds and lower power consumption, significantly accelerating inference time. However, due to their long development cycles, they are difficult to apply new technologies and networks.

[0003] FPGA (Field Programmable Array) can achieve high inference speed, low power consumption and low cost, but it often requires redesigning the structure when changing the network model, and its adaptability to network models is relatively weak. Summary of the Invention

[0004] To address the shortcomings of existing algorithms, this invention uses both FPGA and ARM processors for model inference. Time-consuming convolution operators in the convolutional network model are executed on the FPGA, while other operators are executed on the ARM processor. This enables fast inference of the network model with low power consumption, making it easy to deploy to terminals. It also provides good support for other convolutional neural network models other than SSD_MobilenetV1.

[0005] The technical solution adopted in this invention is: an improved neural network hardware acceleration method based on FPGA, comprising the following steps:

[0006] Step 1: Train the SSD_MobilenetV1 network using transfer learning, data augmentation, multi-scale training, and cosine annealing.

[0007] Furthermore, the learning rate formula for cosine annealing is:

[0008]

[0009] in, and T represents the range of the learning rate. cur T represents the number of training rounds since the last restart. i This indicates the number of training rounds required for the i-th restart.

[0010] Step 2: Perform structured pruning on the trained SSD_MobilenetV1 network, pruning based on convolutional kernels or each network layer as the basic unit;

[0011] Furthermore, this includes: pruning m from the i-th convolutional layer. i Each convolutional kernel, the process is as follows:

[0012] S21. For each convolution kernel F i,j Calculate the sum of the absolute values ​​of the weights of the convolution kernel:

[0013]

[0014] Among them, K l The nth convolutional kernel represents the l-th layer. i The number of convolutional kernel layers;

[0015] S22, according to s j Sort;

[0016] S23, m i The convolutional kernel with the smallest sum of absolute weights and its corresponding feature layer are pruned, and the convolutional kernels associated with the pruned feature layer in the next convolutional layer are removed.

[0017] S24. Create new weight matrices for the i-th and i+1-th layers;

[0018] S25. When pruning multiple layers simultaneously, a greedy strategy is adopted. After pruning, the model structure changes, and training is performed again. Alternating pruning and training are used, and pruning is performed layer by layer or convolution kernel by kernel.

[0019] Step 3: Use the QAT algorithm and introduce pseudo-quantization operations for training to simulate the errors introduced by the quantization process;

[0020] Furthermore, specifically including:

[0021] During backpropagation, the weights are quantized before the input convolution. If there is a Batch Normalization layer, the Batch Normalization layer is incorporated into the weights. The activation values ​​are quantized after the activation function has finished executing.

[0022] The conversion between floating-point numbers and 8-bit fixed-point numbers is accomplished through affine mapping, as shown in the following formula:

[0023] r=S(qZ) (3)

[0024]

[0025]

[0026]

[0027] Where r is the real number to be quantized; q is the quantized integer; n is the quantization bit width; S and Z are the quantization scale and zero point, respectively; array max and array min These are the maximum and minimum values ​​in a series of real numbers to be quantized.

[0028] Step 4: Convert the quantized SSD_MobilenetV1 network into a computation graph;

[0029] An apparatus for an improved neural network hardware acceleration method based on FPGA includes: an AIX bus interface module, a pattern configurator module, a data scheduling module, a convolution calculation module, and a data buffer. The AIX bus interface module is responsible for exchanging data between the FPGA accelerator and the HPS, including receiving data from the HPS, transmitting the configuration information of the FPGA accelerator to the pattern configurator module, configuring and preparing various registers for calculation; reading the feature map and convolution parameters for convolution calculation from the DDR and transmitting them to the calculation data buffer; and receiving the convolution calculation results and storing them in the DDR.

[0030] The mode configurator module parses the configuration data passed in by HPS and configures the various modules of the FPGA accelerator.

[0031] Furthermore, the configuration includes the type of convolution computation, the size of the feature map, the input feature map parameters and the addresses of the input and output data, and whether batch normalization and activation processing are performed after the convolution computation;

[0032] The data scheduling module controls the data flow of the FPGA accelerator during the computation process. The data scheduling module loads data into the computation data buffer according to the address of the input feature map and convolution parameters. After the convolution calculation is completed, the module controls the computation result buffer to receive the convolution calculation result.

[0033] Furthermore, the data cache includes a computation data cache and a computation result cache. The computation data cache adopts a ping-pong operation design. When a certain cache has finished receiving data and starts to perform the next convolution calculation, the data scheduling module will control another computation data cache to receive the next convolution data.

[0034] The calculation result buffer adopts a ping-pong operation design. When a certain calculation result buffer stores the result of the previous convolution, the data scheduling module will control another calculation result buffer to receive the calculation result of the next convolution.

[0035] The convolution calculation module performs calculations for convolution, depthwise convolution, batch normalization, and activation operators. Convolution and depthwise convolution group the convolution kernels according to the number of channels based on hardware resources, with each group containing 16 channels. The next group is calculated only after the previous group has been completed. Batch normalization and activation operators are performed in a pipelined manner, with the number of channels set to 16.

[0036] The beneficial effects of this invention are:

[0037] 1. Compared with traditional ARM processor-based model inference methods, this method can fully utilize FPGA computing resources and improve network inference speed while ensuring the accuracy of the network model.

[0038] 2. The SSD_MobilenetV1 network was trained using training strategies such as transfer learning, data augmentation, multi-scale training, and cosine annealing; structured pruning and training were performed based on the sensitivity of each layer in the network model; the model was compressed to 8 bits using the Quantization Aware Training method; operation fusion, prior_box operator offline computation, and operator optimization were performed on the network at the computation graph level; an accelerator was designed to accelerate convolution operations using the parallel computing characteristics of FPGA; and FPGA+ARM hybrid inference of the network was performed based on the PaddleLite framework. Attached Figure Description

[0039] Figure 1 This is a flowchart of the improved neural network hardware acceleration method based on FPGA of the present invention;

[0040] Figure 2 This is a pseudo-quantization diagram of the present invention;

[0041] Figure 3 This is a schematic diagram of the prior_box structure of the present invention;

[0042] Figure 4 This is a schematic diagram of the device structure of the FPGA-based improved neural network hardware acceleration method of the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0044] like Figure 1As shown, an improved neural network hardware acceleration method based on FPGA includes the following steps:

[0045] Step 1: Train the SSD_MobilenetV1 network using transfer learning, data augmentation, multi-scale training, and cosine annealing.

[0046] The feature pyramid concept is used to obtain the feature information of the six convolutional layers of the SSD_MobilenetV1 network for multi-scale and multi-target target detection.

[0047] The SSD_MobilenetV1 network was trained on the VOC (The PASCAL Visual Object Classes) dataset, which contains a total of 16,551 images and 40,058 objects in the training and validation sets.

[0048] During training, model accuracy is improved through methods such as transfer learning, data augmentation, multi-scale training, and cosine annealing.

[0049] Transfer learning: The MS COCO (Microsoft Common Objects in Context) dataset was selected for transfer learning. The COCO dataset is similar to the VOC dataset in terms of scenario. The model obtained using this dataset was used as a pre-trained model, which enabled specific low-level features such as edges, shapes, corners, and brightness to be shared between tasks.

[0050] Data augmentation: Data augmentation is performed through methods including random flipping, rotation, cropping, deformation scaling, adding noise, and color perturbation. This allows us to create as much value as possible from limited data, obtain new datasets from existing images, greatly increase the amount of data, artificially introduce prior knowledge of human vision, improve the model's generalization ability and robustness, and avoid imbalanced samples to some extent.

[0051] Multi-scale training: The SSD_MobilenetV1 network is trained on image datasets of different scales to learn the features of images at different resolutions. The datasets used have multiple scales for the same target to improve the model's accuracy at each scale.

[0052] Cosine annealing: This method uses a cosine function to reduce the learning rate. During training, the learning rate initially decreases slowly, then accelerates, then decelerates. Once the learning rate reaches a predetermined value, it immediately increases back to its initial value, repeating this process. This aims to break through local optima and reach the global optimum. The learning rate η is... t The specific changes are as follows:

[0053]

[0054] in, and T represents the range of the learning rate. cur T represents the number of training rounds since the last restart. i This indicates the total number of training rounds required for the i-th restart.

[0055] Step 2: Perform structured pruning on the trained SSD_MobilenetV1 network, pruning based on convolutional kernels or each network layer as the basic unit;

[0056] A sensitivity-based structured pruning strategy is adopted. First, sensitivity analysis is performed to determine the sensitivity of each layer in the SSD_MobilenetV1 network to pruning. Each layer is pruned independently, and the pruned network is evaluated on the validation set to observe the change in model accuracy with the pruning rate. Layers with gentler slopes are more sensitive to pruning. The pruning rate is determined according to the sensitivity of each layer. For layers with low sensitivity, a larger pruning ratio is set, while for layers with high sensitivity, a very small pruning ratio is set, or even no pruning is performed.

[0057] After determining the pruning rate for each layer, pruning is performed, and the number of channels m to be subtracted from the i-th layer is calculated based on the pruning rate. i ; use F i,j This represents the j-th convolutional kernel in the i-th layer, obtained through ∑|F i,j |(The sum of all absolute values ​​within a convolutional kernel – L1 regularization term) represents the importance of the convolutional kernel in each layer, and layers with lower importance are pruned.

[0058] For cutting m from the i-th convolutional layer i The process of creating a convolutional kernel is as follows:

[0059] 1. For each convolution kernel F i,j Calculate the sum of the absolute values ​​of its weights:

[0060]

[0061] Among them, K l This represents the l-th layer of the convolution kernel;

[0062] 2. According to s j Sort;

[0063] 3. Place m i The convolutional kernel with the smallest sum of absolute weights and its corresponding feature layer are pruned, and the convolutional kernels associated with the pruned feature layer are removed in the next convolutional layer.

[0064] 4. A new weight matrix is ​​created for the i-th and i+1-th layers, and the remaining weight parameters are copied into the new model.

[0065] When pruning multiple layers simultaneously, a greedy strategy is adopted to avoid the influence between layers. That is, when pruning a new layer, the removed convolutional kernels are not included in the calculation. After pruning, the model structure changes and needs to be retrained to compensate for the accuracy loss caused by pruning. An alternating pruning and training method is adopted, pruning layer by layer or convolutional kernel by convolution, and then training, repeating multiple times.

[0066] Step 3: Use the QAT algorithm and introduce pseudo-quantization operations for training to simulate the errors introduced by the quantization process;

[0067] QAT (Quantization Aware Training) is used, which introduces pseudo-quantization operations for training to simulate the errors introduced during the quantization process; for example... Figure 2 As shown, during backpropagation, the weights are quantized before the input convolution. If there is a Batch Normalization layer, the Batch Normalization layer is incorporated into the weights. The activation values ​​are quantized after the activation function has finished executing.

[0068] The conversion between floating-point numbers and 8-bit fixed-point numbers is accomplished through affine mapping, as shown in the following formula:

[0069] r=S(qZ) (3)

[0070]

[0071]

[0072]

[0073] Where r is the real number to be quantized; q is the quantized integer; n is the quantization bit width, with a value of 8; S and Z are the quantization scale and zero point, respectively, both of which are quantization parameters; array max and array min These are the maximum and minimum values ​​in a series of real numbers to be quantized.

[0074] Step 4: Convert the quantized SSD_MobilenetV1 network into a computation graph;

[0075] In the SSD_MobilenetV1 network, each operator or variable corresponds to a node in the computation graph; where operator nodes and variable nodes are adjacent; based on the computation graph, the following model optimization operations are performed, including operation fusion, prior_box operator offline computation, and operator selection;

[0076] After obtaining the computation graph, multiple consecutive operators are merged into one operator through operation fusion, thereby saving the storage of intermediate variables and accelerating model inference. This is represented in the computation graph by replacing a series of consecutive operator nodes and variable nodes with a new computation node. Specifically, the following two types of computation graphs are replaced: the first is a convolutional graph (conv2d) + batch normalization layer (batch_norm) + activation layer (relu6); the second is a depthwise convolutional graph (depthwise_conv2d) + batch normalization layer (batch_norm) + activation layer (relu6). The three computation nodes of the first type of computation graph are merged together to form a new computation node, as shown in the following formula:

[0077]

[0078]

[0079]

[0080] ReLU6 = min(6, max(0, BN) γ,β (x i ))) (10)

[0081] Where w is a weight parameter of the convolution kernel; x is a data point in the input feature map; m is the number of data points in a batch; ∈ represents the minimum value; and μ B and These are the mean and variance of the data in a batch, respectively; where γ and β are learnable parameters obtained during network training; the second computational graph is fused using the same method.

[0082] In the computation graph Figure 3 The prior_box operator structure shown is optimized by traversing all operators in the computation graph. When the prior_box operator is found, its result is calculated based on the characteristic that it is only related to the number of channels in the feature map and does not depend on the specific numerical value of the feature map. The results of the subsequently connected flatten2 and concat operators can also be calculated. The obtained results are then embedded into the parameters of the box_coder operator. The prior_box operator, along with the subsequently connected flatten2 and concat operators, are then removed, leaving only the box_coder operator. During model inference, the relevant operators do not need to be calculated again, reducing computational load and thus accelerating model inference.

[0083] During inference, this invention employs a hybrid inference method using an ARM CPU processor and an FPGA accelerator. Convolution operators can be executed on both the processor and FPGA accelerator sides. Operator optimization determines the execution device for operators based on factors such as operator type, input / output data type, and computational scale in the computation graph. Operators executed on the FPGA accelerator should simultaneously meet the following conditions:

[0084] (1) The operator type is either a convolution operator or a depthwise convolution operator;

[0085] (2) The input and output data types of the operators are both int8 quantized type. These operators that meet the conditions are marked, and the continuous operators that can be executed on the accelerator are fused and divided into a subgraph containing multiple operator nodes.

[0086] An apparatus for an improved neural network hardware acceleration method based on FPGA, comprising:

[0087] In the SSD_MobilenetV1 network, the most computationally intensive and time-consuming operations are convolution and depthwise convolution, and most convolutional and depthwise convolutional layers are followed by batch normalization and activation layers, accounting for over 90% of the computation. To address these characteristics, the FPGA accelerator designed in this invention implements convolution, depthwise convolution, batch normalization, and activation operators; the FPGA accelerator's structural design is as follows... Figure 4 As shown, it mainly includes an AIX bus interface module, a mode configurator module, a data scheduling module, a convolution calculation module, and a data buffer. The FPGA accelerator does not include HPS and DDR. HPS (Hard Processor System) is an ARM processor and its peripherals.

[0088] The AXI bus interface module is responsible for exchanging data between the FPGA accelerator and the external environment. It is a high-performance, low-latency, high-bandwidth on-chip bus with independent address and data channels. Each channel can be optimized, and timing channels can be controlled as needed to increase clock frequency and reduce latency. The functions of the AXI bus interface module include receiving data from the HPS, transmitting the FPGA accelerator's configuration information to the mode configurator module, configuring and preparing various registers for calculation (where each register is an internal storage unit of the FPGA accelerator used to store configuration information during calculation), reading the feature map and convolution parameter data required for convolution calculation from DDR, transmitting them to the calculation data buffer, and finally receiving the convolution calculation results and storing them in DDR.

[0089] The mode configurator module parses the configuration data passed in by HPS and configures the various modules of the FPGA accelerator. The configuration content includes the type of convolution calculation, the size of the feature map, the input feature map parameters and the addresses of the input and output data, and whether to perform batch normalization and activation processing after the convolution calculation. In order to enhance the versatility and reliability of the FPGA accelerator, the parameters of the above configuration content are combined into a parameter set and transmitted to the mode configurator module at once before the convolution calculation begins. The mode configurator module then configures the relevant convolution calculation module and data scheduling module.

[0090] The data scheduling module controls the data flow of the FPGA accelerator during the computation process. After the mode configurator module completes the configuration, the HPS sends a computation start signal. Upon receiving the signal, the data scheduling module begins the computation of the FPGA accelerator. First, the data scheduling module loads data into the computation data buffer according to the address of the input feature map and convolution parameters. The computation data buffer adopts a ping-pong operation design. For example, when computation data buffer 1 finishes receiving data and starts the next convolution calculation, the data scheduling module controls computation data buffer 2 to start receiving the next convolution data. When the convolution calculation is completed, the data scheduling module controls the computation result buffer to receive the calculation result. The computation result buffer also adopts a ping-pong operation design. For example, when computation result buffer 1 stores the previous convolution result, it simultaneously controls computation result buffer 2 to receive the calculation result of the next convolution.

[0091] The data buffer includes a computation data buffer and a computation result buffer. Due to the ping-pong operation design, each buffer contains two identical storage blocks. The size of the storage blocks is calculated based on the computational load of convolution calculations in the SSD_MobilenetV1 network to ensure that no data is lost during each convolution calculation. The computation data buffer needs to receive external feature values ​​and convolutional weight parameters transmitted via the AXI bus and also needs to send them to the convolution calculation module. The computation result buffer needs to receive data sent by the convolution calculation module and also needs to store it in DDR via the AXI bus. Therefore, both the computation data buffer and the computation result buffer are implemented using asynchronous dual-port RAM.

[0092] The convolution computation module is the core of the FPGA accelerator, including the computation of multiple operators such as convolution, depthwise convolution, batch normalization, and activation. For convolution and depthwise convolution, in order to make full use of the FPGA's computing resources, a parallel computing approach is adopted. First, the convolution kernels are grouped according to the number of channels based on hardware resources, with each group containing a fixed number of channels, set to 16. The next group is computed only after one group is completed. For batch normalization and activation operators, the computational load is relatively less than that of convolution computation, and it does not involve multi-dimensional large-scale data processing, so a pipeline approach is adopted. In order to improve processing efficiency, the parallelism of batch normalization and activation computation is kept consistent with that of convolution computation, and the number of channels is also set to 16.

[0093] FPGA+ARM hybrid inference based on PaddleLite;

[0094] Inference in the SSD_MobilenetV1 network is jointly performed by an FPGA accelerator and an ARM processor. The FPGA accelerator performs convolution and depthwise convolution calculations, while the ARM processor performs calculations of operators such as calib, transpose2, and flatten2, as well as overall scheduling. The ARM processor is deployed through PaddleLite, an inference engine developed by Baidu that supports multiple hardware components for inference computation. This invention completes the PaddleLite backend integration and develops the corresponding SDK (Software Development Kit) and driver.

[0095] The designed FPGA accelerator is connected to the hardware backend via subgraph access. A subgraph detection optimizer for the FPGA accelerator is added to the PaddleLite framework. Through operator optimization, the corresponding convolution and depthwise convolution operators are mapped onto the FPGA accelerator to form a subgraph. The subgraph Op of the FPGA accelerator is registered. After the FPGA subgraph is detected, it is reduced to a hardware graph IR that is easy to deploy, including rearranging the weight data so that the accelerator can receive it. Then, space is allocated to the input and output nodes of the subgraph.

[0096] The SDK mainly performs input data rearrangement, driver calls, and output rearrangement. Input-output rearrangement rearranges the PaddleLite data format into a data format acceptable to the FPGA accelerator. Driver calls are made after the data rearrangement is completed, passing the rearranged input data and weight data to the driver for further processing. Once the calculation is complete, the driver returns the calculated output data.

[0097] The main function of the driver is to interact with the FPGA accelerator, move the required data between user space and kernel space, and pass the data to the FPGA accelerator at the address corresponding to the DDR. This includes driver initialization, memory space allocation, and passing convolution parameters. For the computation of multiple nodes in the subgraph, a double buffering method is used. While the current node is being computed, the parameters of the next node are being transmitted simultaneously, using computation time to mask parameter transmission time and speeding up inference.

[0098] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. An improved neural network hardware acceleration method based on FPGA, characterized in that, Includes the following steps: Step 1: Train the SSD_MobilenetV1 network using transfer learning, data augmentation, multi-scale training, and cosine annealing; the training set is images. Step 2: Perform structured pruning on the trained SSD_MobilenetV1 network, pruning based on convolutional kernels or each network layer as the basic unit; Step 3: Use the QAT algorithm and introduce pseudo-quantization to train the system to simulate the error of the quantization process. During backpropagation, the weights are quantized before the input convolution. If there is a Batch Normalization layer, it is incorporated into the weights. The activation values ​​are quantized after the activation function is executed. Step 4: Convert the quantized SSD_MobilenetV1 network into a computation graph; Each operator or variable in SSD_MobilenetV1 corresponds to a node in the computation graph; operator nodes and variable nodes are adjacent; operation fusion and prior_box operator offline computation are performed based on the computation graph. Operation fusion includes: after obtaining the computation graph, merging multiple consecutive operators into one operator through operation fusion; replacing two types of computation graphs: the first is conv2d+batch_norm+relu6; the second is depthwise_conv2d+batch_norm+relu6; and merging three computation nodes to form a new computation node, as shown in the following formula: （7） （8） （9） （10） in, w This is a weight parameter of the convolution kernel; x i This is a data point from the input feature map; m ϵ represents the number of data in a batch; ϵ is a very small value. and These are the mean and variance of the data in a batch, respectively; γ and β are learnable parameters. Offline computation of the prior_box operator includes: optimizing the prior_box operator structure in the computation graph, traversing all operators in the computation graph, and when the prior_box operator is retrieved, calculating the prior_box operator result based on the characteristic that the prior_box operator is only related to the number of channels in the feature map and does not depend on the specific value of the feature map. At the same time, the results of the flatten2 operator and concat operator connected to the prior_box are calculated and fixed as the parameters of the box_coder operator, and only the box_coder operator is retained.

2. The improved neural network hardware acceleration method based on FPGA according to claim 1, characterized in that, The learning rate formula for cosine annealing is: （1） in, and Indicates the range of the learning rate. This indicates the number of training rounds since the last restart. Indicates the first i The number of training rounds required for each restart.

3. The improved neural network hardware acceleration method based on FPGA according to claim 1, characterized in that, Step two specifically includes: from the first i Cut off the convolutional layer m i Each convolutional kernel, the process is as follows: S21. For each convolution kernel F i,j Calculate the sum of the absolute values ​​of the weights of the convolution kernel: （2） in, K l The convolution kernel is represented by the first... l layer, n i The number of convolutional kernel layers; S22, according to Sort; S23, will m i The convolutional kernel with the smallest sum of absolute weights and its corresponding feature layer are pruned, and the convolutional kernels associated with the pruned feature layer in the next convolutional layer are removed. S24, Creating the first i Layer and first i+1 The new weight matrix of the layer; S25. When pruning multiple layers simultaneously, a greedy strategy is adopted. After pruning, the model structure changes, and training is performed again. Alternating pruning and training are used, and pruning is performed layer by layer or convolution kernel by kernel.

4. The improved neural network hardware acceleration method based on FPGA according to claim 1, characterized in that, Step three also includes: the conversion between floating-point numbers and 8-bit fixed-point numbers is accomplished through affine mapping, as shown in the following formula: （3） （4） （5） （6） in, r For the real number to be quantified; q The quantized integer; n This is for quantization bit width; S and Z These are the quantization scale and the zero point, respectively. array max and array min These are the maximum and minimum values ​​in a series of real numbers to be quantized.

5. An apparatus employing the FPGA-based improved neural network hardware acceleration method according to any one of claims 1-4, characterized in that, include: The AIX bus interface module consists of a mode configurator module, a data scheduling module, a convolution calculation module, and a data buffer. The AIX bus interface module is used for data exchange between the FPGA accelerator and the HPS, including receiving data from the HPS, transmitting the FPGA accelerator's configuration information to the mode configurator module, configuring and preparing calculations for each register, reading the feature map and convolution parameters from the DDR and transmitting them to the data buffer, and receiving the convolution calculation results and storing them in the DDR. The mode configurator module parses the configuration data passed in by HPS and configures the modules of the FPGA accelerator. The data scheduling module controls the data flow of the FPGA accelerator during the computation process. The data scheduling module loads data into the computation data buffer according to the address of the input feature map and convolution parameters; after the convolution calculation is completed, it controls the calculation result buffer to receive the convolution calculation result. The convolution calculation module performs calculations for convolution, depthwise convolution, batch normalization, and activation operators. Convolution and depthwise convolution group the convolution kernels according to the number of channels based on hardware resources. Each group contains multiple channels, and the next group is calculated only after the previous group has been calculated. Batch normalization and activation operators are performed in a pipelined manner, with multiple channels set.

6. The apparatus for the FPGA-based improved neural network hardware acceleration method according to claim 5, characterized in that, The data cache includes a computation data cache and a computation result cache. The computation data cache adopts a ping-pong operation design. When a certain cache has finished receiving data and starts to perform the next convolution calculation, the data scheduling module will control another computation data cache to receive the next convolution data. The calculation result buffer adopts a ping-pong operation design. When a certain calculation result buffer stores the result of the previous convolution, the data scheduling module will control another calculation result buffer to receive the calculation result of the next convolution.

Images