FPGA-based neural network acceleration method and system

Abstract

The invention discloses an FPGA-based neural network acceleration method and system, and the method specifically comprises the steps: S1, carrying out the adaptive processing of input data, and converting original data into standardized feature data; s2, performing neural network model hardware mapping, and generating configuration parameters of the FPGA reconfigurable computing array; s3, parallel computing scheduling is carried out; s4, performing multi-level storage optimization; s5, integrating and outputting results, summarizing operation results of all the calculation units, executing interlayer data standardization processing, generating a current network layer output feature map, repeating the steps S3-S4 until all network layer operations are completed, and finally outputting a neural network reasoning result; dynamic energy efficiency adjustment is further included, and overall power consumption is reduced; the method has the advantages of multi-model adaptation, high calculation efficiency, low storage cost, high energy efficiency ratio and flexible precision, can be popularized on a large scale in scenes of edge calculation, automatic driving, intelligent security and protection and the like, and has practical application value and potential of nationwide multi-scene large-scale popularization.

Description

Technical Field

[0001] This invention belongs to the field of deep learning hardware acceleration technology, specifically relating to a neural network acceleration method and system based on FPGA. Background Technology

[0002] With the widespread application of deep learning in fields such as computer vision and natural language processing, the complexity of neural network models continues to increase, placing higher demands on the computing power, latency, and energy efficiency of inference hardware. Existing acceleration solutions suffer from the following core technical shortcomings, hindering practical applications:

[0003] 1. Poor model adaptability: Traditional FPGA acceleration solutions require manual writing of hardware logic for specific neural networks, and cannot dynamically switch models. If it is necessary to switch from a ResNet classification model to a YOLO detection model, the hardware circuit needs to be redesigned, the development cycle is as long as several weeks, and the adaptation cost is high.

[0004] 2. Computational efficiency bottleneck: No specific optimization has been made for convolution operations, and general computing logic is still used. The data reuse rate is low, resulting in frequent access to external memory and high latency. The parallelism of computing units is fixed and cannot be dynamically adjusted according to the amount of input data. This leads to insufficient computing power under high load and waste of resources under low load.

[0005] 3. High storage access cost: The neural network weights and feature maps have a large amount of data, and existing solutions mostly rely on external storage, resulting in high data loading latency; moreover, the weights are not compressed and optimized, leading to excessive storage bandwidth usage, which further exacerbates the latency.

[0006] 4. Energy efficiency imbalance: The FPGA computing unit is always running at full power. Even if the load rate is less than 30%, it still maintains full clock and full voltage power supply, resulting in high dynamic power consumption waste, which does not meet the low power consumption requirements of edge devices.

[0007] 5. Insufficient precision and flexibility: Most solutions use fixed computational precision, which cannot be adjusted according to the type of task.

[0008] Therefore, developing a neural network acceleration method that can overcome the shortcomings of difficult model adaptation, low computational efficiency, high storage cost, poor energy efficiency, and inflexible accuracy has become the key to breaking through the bottleneck of complex application scenarios. Summary of the Invention

[0009] The technical problem this invention aims to solve is to overcome the aforementioned technical deficiencies and provide a neural network acceleration method that features multi-model adaptation, high computational efficiency, low storage cost, high energy efficiency, and flexible accuracy. To solve the above technical problem, this invention provides a neural network acceleration method and system based on FPGA, specifically including the following steps:

[0010] Step S1. Input data adaptation processing: Receive external neural network input data, perform data format conversion based on the FPGA hardware interface, convert the original data into a fixed-point or floating-point format that can be directly processed by the FPGA, and perform data dimension alignment according to the input dimension requirements of the neural network layer;

[0011] Step S2. Neural Network Model Hardware Mapping: Parse the model structure file of the pre-trained neural network, extract the operation types and parameters of each layer, and generate the configuration parameters of the FPGA reconfigurable computing array. The configuration parameters include the number of computing units, data path width, and operation precision.

[0012] Step S3. Parallel computing scheduling:

[0013] S31 divides the input feature map of the current network layer into blocks according to the number of FPGA computing units, with each computing unit divided into...

[0014] Each sub-feature map data block is configured;

[0015] Each computing unit of S32 executes its corresponding operation in parallel based on the configuration parameters, with the convolution operation using a sliding window block.

[0016] The reuse strategy employs matrix block multiplication and addition optimization for fully connected operations;

[0017] S33 enables intermediate data exchange between computing units through the FPGA on-chip bus, allowing data transmission and computation to proceed in parallel.

[0018] Step S4. Multi-level storage optimization: On-chip cache is used to store the weight parameters and feature map data required for the current calculation. The ping-pong caching mechanism is used to realize the pipeline operation of data loading and calculation. At the same time, data not in the current layer is stored in the FPGA external memory.

[0019] Step S5. Result Integration and Output: Summarize the operation results of each computing unit, perform inter-layer data standardization processing, generate the output feature map of the current network layer, repeat steps S3-S4 until all network layer operations are completed, and finally output the neural network inference result.

[0020] Furthermore, the FPGA-based neural network acceleration method also includes dynamic energy efficiency adjustment: real-time monitoring of the computational load of the FPGA computing units, and when the load rate is <30%, shutting down the clock signals of some computing units and reducing the power supply voltage of the corresponding area to reduce the overall power consumption of the FPGA.

[0021] Furthermore, in step S1, the data format conversion supports dynamic switching between 8-bit fixed-point numbers and 32-bit floating-point numbers. When the neural network is performing a classification task, it uses the 8-bit fixed-point format, and when it is performing a regression task, it uses the 32-bit floating-point format.

[0022] Furthermore, in step S2, the model structure file is in ONNX or TensorFlow Lite format, and the FPGA automatically extracts the operation parameters through its built-in model parser, without the need for manual writing of hardware acceleration logic; the configuration parameters of the FPGA reconfigurable computing array are generated into a configuration bitstream through a hardware description language, supporting dynamic switching between different neural network models such as ResNet, YOLO, and Transformer on the same FPGA hardware platform.

[0023] Furthermore, in step S32, when processing a network layer with a convolution kernel size of 1×1, the system automatically switches to a dedicated 1×1 convolution calculation mode to improve computational efficiency by reorganizing the input and output channel dimensions.

[0024] Furthermore, in step S32, the convolution operation adopts weighted reuse optimization, preloading the convolution kernel parameters into the internal register of the computing unit, and reducing the number of external memory accesses by reusing local data in the sliding window of the input feature map.

[0025] Furthermore, in step S33, the on-chip bus adopts a hierarchical cross switch structure, and the data transmission bandwidth increases linearly with the number of computing units.

[0026] Furthermore, in step S4, the multi-level storage optimization also includes weight compression, which employs a weight pruning algorithm based on Taylor expansion to retain the weight parameters that have the greatest impact on the output, and uses an FPGA hardware accelerator to achieve fast decompression of the compressed weights.

[0027] Furthermore, in step S5, the inter-layer data standardization processing supports parallel computation of batch normalization, layer normalization, and instance normalization.

[0028] Furthermore, an FPGA-based neural network acceleration system is characterized by comprising a data adaptation and preprocessing module, a model parsing and configuration module, a reconfigurable computing array module, a multi-level storage management module, a result integration and standardization module, and an energy efficiency management module.

[0029] The data adaptation preprocessing module is used for the adaptation processing of input data, including original data format conversion, dimension alignment and invalid data pruning, and outputs standardized feature data that can be directly processed by the FPGA.

[0030] The model parsing and configuration module is used to parse model files in ONNX / TensorFlow Lite format, extract the operation types and parameters of each layer, generate a configuration bitstream of the reconfigurable computing array, and realize dynamic model switching.

[0031] The reconfigurable computing array module is a two-dimensional grid structure composed of two or more computing units. The reconfigurable computing array module is used to perform parallel computing. The computing units support convolution and fully connected operations and adopt a weighted reuse and sliding window block strategy.

[0032] The multi-level storage management module includes: an on-chip cache, an external storage interface, a ping-pong cache controller, and a weight compression accelerator; the on-chip cache is used to store the currently computed weights and feature map data; the external storage interface is used to store non-current layer data and model parameters; the ping-pong cache controller is used to implement pipelined operations for data loading and computation; the weight compression accelerator is used to execute a Taylor expansion-based pruning algorithm to complete weight compression and decompression.

[0033] The result integration and standardization module is used to summarize the intermediate results of each computing unit, perform inter-layer standardization processing of batch normalization, layer normalization and instance normalization, generate the final inference result and output it through the interface;

[0034] The energy efficiency management module is used to realize dynamic energy efficiency adjustment, monitor the load rate of the computing unit in real time, and automatically shut down the clock of idle units and reduce the power supply voltage when the load rate is <30%.

[0035] The advantages of this invention compared to the prior art are:

[0036] This invention reduces adaptation costs and development cycles by automatically parsing ONNX / TensorFlow Lite format models, thus improving model adaptability. It improves computational efficiency by designing a dedicated 1×1 convolution mode and reduces external storage access through a right-reuse strategy. The invention utilizes a multi-level storage architecture to significantly reduce data loading latency. Dynamic energy efficiency adjustment reduces FPGA power consumption under low load, extending the battery life of edge devices. This invention supports dynamic switching between 8-bit fixed-point and 32-bit floating-point, balancing different needs for precision and resources. Attached Figure Description

[0037] Fig. 1 This is the overall flowchart of a neural network acceleration method and system based on FPGA according to the present invention.

[0038] Fig. 2 This is a system module block diagram of a neural network acceleration method and system based on FPGA according to the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The present invention will be further described in detail below with reference to the accompanying drawings, providing a method and system for accelerating neural networks based on FPGA.

[0040] Combined with appendix Figs. 1-2 A neural network acceleration method and system based on FPGA, comprising the following steps:

[0041] Step S1. Input data adaptation processing: Receive external neural network input data, perform data format conversion based on the FPGA hardware interface, convert the original data into a fixed-point or floating-point format that the FPGA can directly process, eliminate the data format and dimension incompatibility problem, and perform data dimension alignment according to the input dimension requirements of the neural network layer;

[0042] Step S2. Neural Network Model Hardware Mapping: Parse the model structure file of the pre-trained neural network, extract the operation types and parameters of each layer, and generate the configuration parameters of the FPGA reconfigurable computing array. The configuration parameters include the number of computing units, data path width, and operation precision. The number of computing units is 2-16, which can be adjusted according to the model complexity. The data path width is 128 bits or 256 bits, matching the storage bandwidth. The operation precision is 8-bit fixed-point or 32-bit floating-point, matching the task type.

[0043] Step S3. Parallel computing scheduling:

[0044] S31 divides the input feature map of the current network layer into blocks according to the number of FPGA computing units, and assigns an independent sub-feature map data block to each computing unit;

[0045] Each computing unit of S32 executes corresponding operations in parallel based on configuration parameters. The convolution operation adopts a sliding window block reuse strategy, and the fully connected operation adopts matrix block multiplication and addition optimization, which splits the large-size matrix into small sub-matrices. Each computing unit executes sub-matrix multiplication and addition in parallel to avoid data overflow and resource waste.

[0046] S33 enables intermediate data exchange between computing units through the FPGA on-chip bus, allowing data transmission and computation to proceed in parallel.

[0047] Step S4. Multi-level storage optimization: On-chip high-speed cache is used to store the weight parameters and feature map data required for the current calculation. The pipeline operation of data loading and calculation is realized through the ping-pong cache mechanism. At the same time, non-current layer data is stored in the FPGA external memory. The ping-pong cache scheduling adopts a dual-buffer alternating read and write mechanism. When buffer A is used for the current layer data calculation, buffer B synchronously loads the next layer data without waiting delay.

[0048] Step S5. Result Integration and Output: Summarize the operation results of each computing unit, perform inter-layer data standardization processing, generate the output feature map of the current network layer, use the output feature map of the current layer as the input of the next layer, repeat steps S3-S4 until all network layer operations are completed, and finally output the neural network inference result.

[0049] In a specific embodiment, a neural network acceleration method based on FPGA also includes dynamic energy efficiency adjustment: real-time monitoring of the computing load of FPGA computing units; when the load rate is <30%, automatically shutting down the clock signal of idle computing units and reducing the power supply voltage of the corresponding area to reduce the overall power consumption of FPGA; when the load rate rises to ≥70%, restoring the clock and voltage to ensure that the computing power meets the requirements.

[0050] In a specific embodiment, in step S1, the data format conversion supports dynamic switching between 8-bit fixed-point numbers and 32-bit floating-point numbers. When the neural network is performing a classification task such as ImageNet classification, it uses the 8-bit fixed-point format, and when it is performing a regression task such as object detection box coordinate regression, it uses the 32-bit floating-point format.

[0051] In a specific embodiment, in step S2, the model structure file is in ONNX or TensorFlow Lite format. The FPGA automatically extracts the operation parameters through its built-in model parser, eliminating the need for manual writing of hardware acceleration logic. The operation parameters include the convolution kernel size, stride, and number of output channels. The configuration parameters of the FPGA reconfigurable computing array are generated into a configuration bitstream using the Verilog hardware description language, supporting dynamic switching between different neural network models such as ResNet, YOLO, and Transformer on the same FPGA hardware platform. The configuration process does not require restarting the FPGA, resulting in a short response time.

[0052] In a specific embodiment, in step S32, when processing a network layer with a convolution kernel size of 1×1, the system automatically switches to a dedicated 1×1 convolution calculation mode to improve computational efficiency by reorganizing the input and output channel dimensions.

[0053] In a specific embodiment, in step S32, the convolution operation adopts weighted reuse optimization, preloading the convolution kernel parameters into the internal register of the computing unit, and reducing the number of external memory accesses by reusing local data in the sliding window of the input feature map.

[0054] In a specific embodiment, in step S33, the on-chip bus adopts a hierarchical cross switch structure, and the data transmission bandwidth increases linearly with the number of computing units.

[0055] In a specific embodiment, in step S4, multi-level storage optimization also includes weight compression. A weight pruning algorithm based on Taylor expansion is used to retain the weight parameters that have the greatest impact on the output. The compressed weights are then rapidly decompressed using an FPGA hardware accelerator, further reducing storage bandwidth usage.

[0056] In a specific embodiment, in step S5, the inter-layer data normalization processing supports parallel computation of batch normalization, layer normalization, and instance normalization.

[0057] In a specific embodiment, an FPGA-based neural network acceleration system includes a data adaptation and preprocessing module, a model parsing and configuration module, a reconfigurable computing array module, a multi-level storage management module, a result integration and standardization module, and an energy efficiency management module.

[0058] The data adaptation preprocessing module is used for the adaptation processing of input data, including raw data format conversion, dimension alignment and invalid data pruning, and outputs standardized feature data that can be directly processed by the FPGA;

[0059] The model parsing and configuration module is used to parse model files in ONNX / TensorFlow Lite format, extract the operation types and parameters of each layer, generate a configuration bitstream for the reconfigurable computing array, and realize dynamic model switching.

[0060] The reconfigurable computing array module is a two-dimensional mesh structure composed of two or more computing units. The reconfigurable computing array module is used to perform parallel computing. The computing units support convolution and fully connected operations, and adopt weight reuse and sliding window block strategy.

[0061] The multi-level storage management module includes: on-chip cache, external storage interface, ping-pong cache controller, and weight compression accelerator; the on-chip cache is used to store the currently computed weights and feature map data; the external storage interface is used to store non-current layer data and model parameters; the ping-pong cache controller is used to implement pipelined operations for data loading and computation; the weight compression accelerator is used to execute a Taylor expansion-based pruning algorithm to complete weight compression and decompression.

[0062] The results integration and standardization module is used to summarize the intermediate results of each computing unit, perform inter-layer standardization processing such as batch normalization, layer normalization and instance normalization, generate the final inference results and output them through the interface;

[0063] The energy efficiency management module is used to achieve dynamic energy efficiency adjustment and monitor the load rate of the computing unit in real time. When the load rate is less than 30%, it automatically shuts down the clock of the idle unit and reduces the power supply voltage.

[0064] In an embodiment of the YOLOv5s target detection model accelerated by Xilinx Zynq Ultra Scale+FPGA, the FPGA-based neural network acceleration system receives a 640×640 RGB image. The data adaptation preprocessing module converts the image format from H×W×3 to the N×3×640×640 format required by YOLOv5. Using an 8-bit fixed-point format, the pixel values ​​are normalized from [0,255] to the [-1,1] range. Invalid black borders are cropped from the image edges, and standardized feature data is output.

[0065] The model parsing and configuration module parses the ONNX file of YOLOv5s and extracts the parameters of each layer: 20 1×1 convolutional layers, 2 fully connected layers, and 5 max pooling layers; the number of computation units is set to 16, the data path width is 256 bits, and the computation precision is 8-bit fixed-point; the configuration bitstream is loaded onto the FPGA through the PCIe 4.0 interface, the configuration response time is 3.2ms, and the hardware mapping of the YOLOv5s model is completed.

[0066] The reconfigurable computing array module divides the 640×640 input feature map into 16 computing units, with each unit processing 40×40 sub-blocks; at the same time, it automatically switches to dedicated mode, recombining input channel 32 and output channel 64 into an 8×8 sub-matrix to improve computing efficiency.

[0067] The on-chip cache allocates 2MB to store the current convolutional layer weights and feature map data; non-current layer data is stored in DDR4; when cache A calculates the current layer, cache B loads the weights of the next layer; the Taylor expansion pruning algorithm is used to retain the 40% of weight parameters that have the greatest impact on the output, and the YOLOv5s weights are compressed from 14MB to 5.6MB, with a weight compression accelerator decompression latency of 8μs.

[0068] The results integration and standardization module collects the results of 40×40 sub-blocks from 16 computing units, reorganizes them into a complete 640×640 feature map, and summarizes the results.

[0069] Repeat the above steps to complete 20 1×1 convolutional layers and 2 fully connected layers, and finally output the target detection result, which is transmitted to the host through the PCIe interface. The total inference latency is 14.8ms.

[0070] The energy efficiency management module monitors the load rate of 16 computing units in real time. During the intermittent period, the energy efficiency management module shuts down the clock signals of 8 idle computing units, reduces the power supply voltage from 1.0V to 0.8V, and reduces the dynamic power consumption of the FPGA from 12W to 7.2W, greatly reducing power consumption.

[0071] In summary, the FPGA-based neural network acceleration method and system proposed in this invention have the advantages of multi-model adaptation, high computational efficiency, low storage cost, high energy efficiency, and flexible accuracy. It can be widely promoted in scenarios such as edge computing, autonomous driving, and intelligent security, and has practical application value and potential for large-scale promotion in multiple scenarios across the country.

[0072] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A neural network acceleration method based on FPGA, characterized in that, Includes the following steps: Step S1. Input data adaptation processing: Receive external neural network input data, perform data format conversion based on the FPGA hardware interface, convert the original data into a fixed-point or floating-point format that can be directly processed by the FPGA, and perform data dimension alignment according to the input dimension requirements of the neural network layer; Step S2. Neural Network Model Hardware Mapping: Parse the model structure file of the pre-trained neural network, extract the operation types and parameters of each layer, and generate the configuration parameters of the FPGA reconfigurable computing array. The configuration parameters include the number of computing units, data path width, and operation precision. Step S3. Parallel computing scheduling: S31 divides the input feature map of the current network layer into blocks according to the number of FPGA computing units, and assigns an independent sub-feature map data block to each computing unit; Each computing unit of S32 executes corresponding operations in parallel based on configuration parameters. The convolution operation adopts a sliding window block reuse strategy, and the fully connected operation adopts matrix block multiplication and addition optimization. S33 enables intermediate data exchange between computing units through the FPGA on-chip bus, allowing data transmission and computation to proceed in parallel. Step S4. Multi-level storage optimization: On-chip cache is used to store the weight parameters and feature map data required for the current calculation. The ping-pong caching mechanism is used to realize the pipeline operation of data loading and calculation. At the same time, data not in the current layer is stored in the FPGA external memory. Step S5. Result Integration and Output: Summarize the operation results of each computing unit, perform inter-layer data standardization processing, generate the output feature map of the current network layer, repeat steps S3-S4 until all network layer operations are completed, and finally output the neural network inference result.

2. The FPGA-based neural network acceleration method according to claim 1, characterized in that: It also includes dynamic energy efficiency adjustment: real-time monitoring of the computing load of FPGA computing units, when the load rate is <30%, shutting down the clock signal of some computing units and reducing the power supply voltage of the corresponding area to reduce the overall power consumption of the FPGA.

3. The FPGA-based neural network acceleration method according to claim 1, characterized in that: In step S1, the data format conversion supports dynamic switching between 8-bit fixed-point numbers and 32-bit floating-point numbers. When the neural network is for a classification task, it uses the 8-bit fixed-point format, and when it is for a regression task, it uses the 32-bit floating-point format.

4. The FPGA-based neural network acceleration method according to claim 3, characterized in that: In step S2, the model structure file is in ONNX or TensorFlow Lite format. The FPGA automatically extracts the computation parameters through its built-in model parser, eliminating the need for manual writing of hardware acceleration logic. The configuration parameters of the FPGA reconfigurable computing array are generated into a configuration bitstream through a hardware description language, supporting dynamic switching between different neural network models such as ResNet, YOLO, and Transformer on the same FPGA hardware platform.

5. The FPGA-based neural network acceleration method according to claim 1, characterized in that: In step S32, when processing a network layer with a convolution kernel size of 1×1, the system automatically switches to a dedicated 1×1 convolution calculation mode to improve computational efficiency by reorganizing the input and output channel dimensions.

6. The FPGA-based neural network acceleration method according to claim 5, characterized in that: In step S32, the convolution operation adopts weighted reuse optimization, preloading the convolution kernel parameters into the internal register of the computing unit, and reducing the number of external memory accesses by reusing local data in the sliding window of the input feature map.

7. The FPGA-based neural network acceleration method according to claim 5, characterized in that: In step S33, the on-chip bus adopts a hierarchical cross switch structure, and the data transmission bandwidth increases linearly with the number of computing units.

8. The FPGA-based neural network acceleration method according to claim 1, characterized in that: In step S4, the multi-level storage optimization also includes weight compression, which uses a weight pruning algorithm based on Taylor expansion to retain the weight parameters that have the greatest impact on the output, and uses an FPGA hardware accelerator to achieve fast decompression of the compressed weights.

9. The FPGA-based neural network acceleration method according to claim 1, characterized in that: In step S5, the inter-layer data standardization processing supports parallel computation of batch normalization, layer normalization, and instance normalization.

10. An FPGA-based neural network acceleration system for implementing the FPGA-based neural network acceleration method as described in any one of claims 1-9, characterized in that: It includes a data adaptation and preprocessing module, a model parsing and configuration module, a reconfigurable computing array module, a multi-level storage management module, a result integration and standardization module, and an energy efficiency management module; The data adaptation preprocessing module is used for the adaptation processing of input data, including original data format conversion, dimension alignment and invalid data pruning, and outputs standardized feature data that can be directly processed by the FPGA. The model parsing and configuration module is used to parse model files in ONNX / TensorFlow Lite format, extract the operation types and parameters of each layer, generate a configuration bitstream of the reconfigurable computing array, and realize dynamic model switching. The reconfigurable computing array module is a two-dimensional grid structure composed of two or more computing units. The reconfigurable computing array module is used to perform parallel computing. The computing units support convolution and fully connected operations and adopt a weighted reuse and sliding window block strategy. The multi-level storage management module includes: an on-chip cache, an external storage interface, a ping-pong cache controller, and a weight compression accelerator; the on-chip cache is used to store the currently computed weights and feature map data; the external storage interface is used to store non-current layer data and model parameters; the ping-pong cache controller is used to implement pipelined operations for data loading and computation; the weight compression accelerator is used to execute a Taylor expansion-based pruning algorithm to complete weight compression and decompression. The result integration and standardization module is used to summarize the intermediate results of each computing unit, perform inter-layer standardization processing of batch normalization, layer normalization and instance normalization, generate the final inference result and output it through the interface; The energy efficiency management module is used to realize dynamic energy efficiency adjustment, monitor the load rate of the computing unit in real time, and automatically shut down the clock of idle units and reduce the power supply voltage when the load rate is <30%.

Images