Convolutional neural network FPGA acceleration system and method based on configurable IP core

Abstract

The invention relates to a convolutional neural network FPGA acceleration system and method based on a configurable IP core, belongs to the technical field of artificial intelligence hardware acceleration, and aims to solve the problems of insufficient flexibility, low resource utilization rate, off-chip storage bandwidth bottleneck and the like in an existing CNN acceleration scheme. According to the system, an SoC framework is adopted, a configurable convolution batch normalization activation IP core and a pooling IP core are integrated in an FPGA programmable logic part, parameters are dynamically configured by a processing system according to a network layer sequence, and time division multiplexing of hardware resources is achieved. The convolution and pooling modules are directly connected through AXI4-Stream, intermediate data write-back and rereading are reduced, and delay and bandwidth pressure are reduced. According to the method, by means of software and hardware cooperative scheduling, universality and expansibility are taken into account while the calculation throughput rate is increased, and the method is suitable for efficient reasoning acceleration of various CNN models.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence hardware acceleration technology, and particularly relates to a method and system for accelerating convolutional neural network (CNN) inference based on field programmable gate array (FPGA), and specifically relates to a high-efficiency and flexible hardware acceleration scheme realized by using a dynamically configurable intellectual property core (IP core). BACKGROUND

[0002] In recent years, with the rapid development of deep learning theory and application, convolutional neural network (CNN) has become the core algorithm of computer vision tasks and has achieved great success in image recognition, target detection, semantic segmentation and other fields. In order to meet the growing demand for computing power, the deployment of CNN usually relies on high-performance computing platforms.

[0003] The current mainstream CNN acceleration scheme is mainly based on a graphics processing unit (GPU) or an application-specific integrated circuit (ASIC). The GPU has a large number of parallel computing units and can provide strong computing power, but its power consumption is high and its volume is large, making it difficult to deploy in embedded devices or edge computing scenarios with limited power consumption and space. The ASIC is a chip customized for a specific algorithm and can achieve a very high energy efficiency ratio, but its design and manufacturing cost is extremely high, the development cycle is long, and once it is finalized, it cannot be modified, lacking flexibility and being difficult to adapt to the rapid iteration of CNN models.

[0004] Field programmable gate array (FPGA) has become an extremely attractive CNN inference acceleration platform due to its inherent hardware parallel processing capability, reconfigurability, and lower development cost and shorter time-to-market period compared with ASIC. However, there are still several problems to be solved in the existing FPGA acceleration scheme. One common method is to design fixed hardware computing modules for each layer of a specific CNN model, or to use fixed architecture IPs provided by FPGA manufacturers (such as deep learning processors DPU). Although this type of scheme can improve the operation speed of a specific network, its limitations are also very obvious:

[0005] Lack of flexibility and versatility: Different CNN models, or even different layers of the same model, have different parameters (such as convolution kernel size, input / output channel number, stride, etc.). It is difficult to efficiently adapt to all cases by designing special hardware for each layer or using fixed architecture IPs, resulting in poor hardware versatility.

[0006] Memory bandwidth bottleneck: In the calculation process of a multi-layer network, the intermediate feature maps generated by each layer usually need to be written back to the off-chip dynamic random access memory (such as DDR) and then read by the next layer. This frequent off-chip memory access (memory access) consumes a large amount of bus bandwidth, becomes a bottleneck of system performance, and brings additional power consumption and delay.

[0007] Low hardware resource utilization: If independent hardware acceleration modules are instantiated separately for each layer in the network, it will sharply consume the precious on-chip resources of FPGA, such as logic cells (LUTs), digital signal processing units (DSPs) and block random access memories (BRAMs). This not only limits the size and complexity of the deployable network model, but also causes great waste of hardware resources, because only the hardware of one layer is working at any moment.

[0008] Therefore, how to design a CNN acceleration architecture on FPGA that can efficiently perform calculations, has high flexibility and versatility, maximizes the reuse of hardware resources, and reduces the dependence on off-chip memory bandwidth, is a technical problem to be solved in the field. SUMMARY

[0009] The present application aims to solve the problems of poor flexibility, low hardware resource reuse efficiency and off-chip memory bandwidth bottleneck in the prior art FPGA acceleration CNN scheme. Specifically, the core technical problems to be solved by the present application include:

[0010] How to design a unified configurable computing unit to adapt to the computing needs of different CNN layers and avoid designing dedicated hardware for each layer.

[0011] How to efficiently reuse hardware resources among different network layers to reduce FPGA resource occupation and support larger-scale networks.

[0012] How to optimize the data flow path to reduce the read and write times of intermediate calculation results to off-chip DDR, to alleviate the storage bandwidth bottleneck and reduce the delay.

[0013] How to build an acceleration system with high performance and versatility, so that it can flexibly support different CNN network structures through software configuration, rather than relying on specific FPGA chips or custom hardware.

[0014] To solve the above problems, the application provides a convolutional neural network FPGA acceleration system based on a configurable IP core. The system adopts a system on chip (SoC) architecture and comprises a processing system (PS) and a programmable logic (PL). The PL internally integrates a set of configurable computing units, which at least include a configurable convolution batch normalization activation IP core and a configurable pooling IP core. The PS is responsible for scheduling and controlling the entire acceleration process. According to the network structure of the CNN model, the PS dynamically configures the operation parameters (such as the convolution kernel size, the stride, the number of channels, etc.) of the same set of IP cores in the PL in the order of layers through an AXI bus, and coordinates the DMA controller to complete the transfer of feature maps, weights and other data between the shared DDR and the PL. In this way, a set of hardware IP cores can be time-multiplexed by different network layers in time, realizing efficient use of hardware resources.

[0015] In addition, in order to reduce DDR access, the convolution IP core and the pooling IP core are directly connected through an AXI4-Stream stream interface, so that the output data of the convolution layer can be directly and seamlessly used as the input of the pooling layer, forming an on-chip hardware pipeline, avoiding the rewriting and re-reading of the intermediate feature maps, and significantly reducing the delay and bandwidth pressure.

[0016] The application also provides a corresponding CNN acceleration method. The method uses the aforementioned system and comprises the following steps: first, initialization, preloading network weights and other parameters to the DDR. Then, the PS performs cyclic processing in the order of network layers: for each layer, the PS first configures the parameters of the IP core, then starts the DMA to transfer the input data from the DDR to the PL, triggers the IP core to perform pipeline calculation, and after the calculation is completed, the DMA writes the result back to the DDR. The process is repeated until all layers of the network are calculated. While the PL is calculating, the PS can prepare the data and parameters of the next layer in parallel, further improving the efficiency of the system.

[0017] The application has the following advantages:

[0018] (1) The application provides a unified convolution and pooling IP core, which can support multiple CNN layers with different sizes of convolution kernels, different numbers of channels and different strides through runtime parameter configuration, so that a single hardware design can be flexibly adapted to different network models, greatly improving the universality and reusability of the hardware.

[0019] (2) The functions of convolution, batch normalization (BN) and activation function (such as LeakyReLU) are deeply integrated in the same IP core and executed in a hardware pipeline manner. The calculation results are directly transferred within the IP core without the need for off-chip storage and data transfer between functional units, significantly reducing the processing delay and improving the calculation efficiency of a single layer.

[0020] (3) By time division multiplexing the same set of convolution / pooling hardware between multiple layers, the present application greatly reduces the occupation of FPGA logic resources, DSP and BRAM. This makes it possible to deploy deeper and more complex CNN models on small and medium-sized FPGAs with limited resources, effectively saving hardware costs.

[0021] (4) By adopting AXI4-Stream direct connection between convolution kernel and pooling kernel and optimized data scheduling strategy, a large amount of intermediate feature map data is directly processed in a streaming manner on-chip, greatly reducing the access frequency to DDR. This effectively alleviates the constraints of storage bandwidth bottleneck on system performance, while reducing the power consumption caused by data transfer.

[0022] (5) The IP core internally adopts a deep pipeline and parallel computing architecture design, which can achieve extremely high computing throughput (such as outputting one pixel result per clock cycle). Combined with hardware pipeline operation, the system achieves orders of magnitude acceleration effect, and is very suitable for video monitoring, automatic driving and other application scenarios that require real-time or near real-time processing.

[0023] (6) The present scheme adopts high-level synthesis (HLS) and standard AXI interface for design, making the IP core easy to modify, maintain and transplant to different FPGA platforms. Developers can easily adjust the data bit width (such as fixed-point quantization) or extend new functional modules, with good engineering practical value and scalability.

[0024] Other advantages, objects, and features of the present application will be apparent to those skilled in the art from the following specification, in conjunction with the accompanying drawings. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the specification as follows. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred detailed description of the present application will be given below in conjunction with the drawings, in which:

[0026] Figure 1 The overall structure block diagram of the FPGA acceleration system of the present application is shown in the figure;

[0027] Figure 2 The internal structure schematic diagram of the configurable convolution batch normalization activation IP core of the present application is shown in the figure;

[0028] Figure 3 The internal structure schematic diagram of the configurable pooling IP core of the present application is shown in the figure;

[0029] Figure 4 The flowchart of the multi-layer network hardware acceleration method of the present application is shown in the figure. Detailed Implementation

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0031] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0032] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0033] The following will be combined with the appendix Figures 1 to 4 The preferred embodiments of the present invention will be described in detail so that those skilled in the art can better understand the present invention.

[0034] I. System Overall Architecture

[0035] like Figure 1 As shown, the FPGA acceleration system in this embodiment is built on a SoC FPGA platform, such as the Xilinx Zynq-7000 series chip. The system mainly consists of two parts: a processing system (PS) and programmable logic (PL).

[0036] Processing System PS: usually a high-performance embedded ARM multi-core processor (such as Cortex-A9). PS is responsible for running Linux or bare-metal applications, undertaking the "brain" function of the entire system, including: parsing CNN model files, managing and loading weights and parameters of each layer, dynamically configuring each IP core in the PL through the AXI4-Lite bus 106, scheduling DMA for data transfer, and monitoring the status of the entire inference process.

[0037] Programmable Logic PL: the core part of the FPGA, used to implement customized parallel computing hardware. In this embodiment, the core acceleration unit of the present application is deployed in the PL, including a configurable convolution batch normalization activation IP core and a configurable pooling IP core. In addition, a DMA / VDMA controller is integrated in the PL for efficient data transmission between the PL and the DDR memory.

[0038] DDR memory: as the global data buffer of the system, shared by PS and PL through AXI interconnection bus. It is used to store the original input image, the weights and BN parameters of each layer of CNN, the intermediate feature map, and the final inference result.

[0039] Data path: the core path of data flow is: input data (such as from a camera or a storage device) is stored in DDR by VDMA / DMA; when inference starts, PS configures DMA to transfer input feature maps from DDR to convolution IP core through AXI4-Stream bus a; the output of the convolution IP core flows directly to the pooling IP core through the on-chip AXI4-Stream bus b; the final output of the pooling IP core (or the convolution IP core when there is no pooling layer) is transmitted to DMA through AXI4-Stream bus c, and then written back to the specified location of DDR by DMA.

[0040] II. Configurable IP core design

[0041] 1. Configurable convolution batch normalization activation IP core (see Figure 2 )

[0042] This IP core integrates convolution, batch normalization and activation three serial operations into a deep pipelined hardware module, and its internal structure is shown in Figure 2 .

[0043] Input and configuration interface: data enters through the input AXI4-Stream interface, and the processing result is sent out through the output AXI4-Stream interface 207. PS configures its working parameters through the AXI4-Lite interface 208 at runtime, such as convolution kernel size K, stride S, feature map width and height W / H, input / output channel number, etc.

[0044] Row buffer: composed of on-chip BRAM, used to buffer input stream data. When processing KxK convolution, it can buffer at least K-1 rows of pixels, ensuring that K-1 rows of data above the processing pixel are available, thus supporting sliding window operation.

[0045] Sliding window generator: linked with row buffer, extracts data from input stream and row buffer, generates a KxK pixel window in real time, and sends it to the subsequent calculation unit.

[0046] Convolution calculation module (MAC array): this is the core of the calculation. The module contains a large number of parallel MAC units. For each output channel, it performs multiply-accumulate operation on the pixels in the sliding window with the corresponding weights loaded from DDR (via AXI master interface, not detailed in the figure), and adds the bias. Through HLS optimization techniques such as loop unrolling, parallel calculation on multiple output channels can be realized, greatly improving the throughput.

[0047] Batch normalization (BN) module: immediately after the MAC array, the convolution result is processed by BN. It uses the mean, variance inverse, gamma and beta parameters configured by PS to perform linear transformation on each data: "subtract mean, multiply variance inverse, multiply gamma, add beta".

[0048] Activation module: performs nonlinear activation function on the data after BN processing.

[0049] For example, LeakyReLU function (output = max(0, x) + 0.1 x min(0, x)).

[0050] The entire IP core is optimized through instructions such as #pragma HLS PIPELINE to ensure that data can flow continuously through each processing unit like a pipeline, achieving II = 1 performance, i.e. processing a new data every clock cycle.

[0051] 2. Configurable pooling IP core (see Figure 3 )

[0052] This IP core is used to perform downsampling operations such as Max Pooling, and its structure is similar to the convolution core but with simpler calculation logic.

[0053] Input and configuration interface: input and output through AXI4-Stream interface, and configure the pooling window size K and stride S through AXI4-Lite interface.

[0054] Row buffer and sliding window generator: similar to the corresponding module in the convolution core, used to generate KxK pooling window from input stream.

[0055] Sliding window comparison unit: This is the core of the pooling kernel. It compares all pixel values within the KxK window, finds the maximum (for max pooling) and outputs it. The process is done in parallel for all channels to match the high throughput of the upstream convolution kernel.

[0056] III. Acceleration flow of multi-layer network (see Figure 4 )

[0057] The core idea of the present application is to use the software flexibility of the PS and the hardware parallelism of the PL to accelerate the entire CNN network by time-division multiplexing a set of hardware resources. The specific flow is as shown in Figure 4

[0058] S401: Initialization. After the system is powered on, the PS runs the control program to load all the weights, BN parameters, etc. of all layers required by the entire CNN network from the non-volatile storage (such as SD card) to the predetermined area of the DDR memory.

[0059] S402: Prepare input data. The PS controls the VDMA / DMA to move the original input image (such as a 224x224 RGB image) to be inferred from an external interface or a certain location in the DDR to the designated input buffer.

[0060] S403: Loop processing by layer. The PS schedules the calculation of each layer in turn according to the network topology, starting from the first layer.

[0061] S404: Configure IP kernel parameters. For the current layer (for example, Layer N), the PS writes the specific parameters of the layer to the convolution IP kernel and the pooling IP kernel through the AXI4-Lite bus 106. For example, configure the convolution kernel size to be 3x3, the stride to be 1, the input to be 64 channels, the output to be 128 channels, and inform the start address of the convolution kernel weight in the DDR. If there is pooling after this layer, configure the window of the pooling kernel to be 2x2 and the stride to be 2.

[0062] S405: Start data transmission and calculation. The PS configures and starts the DMA to stream the input feature map of Layer N (the original image for the first layer) from the DDR to the input end of the convolution IP kernel. Once the data starts to flow in, the hardware pipeline in the PL automatically starts the calculation.

[0063] S406: Pipeline calculation and result backwrite. The convolution IP kernel performs convolution, BN and activation calculation, and its output is streamed into the pooling IP kernel (if configured to be enabled). After the pooling IP kernel completes the calculation, its output stream is sent to the DMA, which writes it back to the output feature map area reserved for Layer N+1 in the DDR. During the entire calculation process, the PS does not need to intervene, realizing the decoupling of software and hardware. ​

[0064] S407: Determine whether it is the last layer. After the current layer is calculated, the PS determines whether all layers have been calculated by querying the DMA status or interrupt.

[0065] If not, return to S404, the PS updates the pointer to point to the parameters and data of the next layer, and repeats the configuration and start process to calculate the next layer using the same set of hardware IP cores.

[0066] If yes, go to S408: output the result. The entire network inference is completed, and the final result (such as classification confidence or target detection box coordinates) is stored in the DDR, and the PS can read the result for subsequent processing or send it to the user.

[0067] It is worth mentioning that during the "long" hardware time of the PL performing S406 to calculate, the PS is relatively idle. The present application takes advantage of this to realize the parallel and cooperative work of the PS and the PL. For example, when the PL calculates the Nth layer, the PS can start another DMA channel in advance to pre-fetch the weights required by the N+1th layer from the DDR main memory to the on-chip cache (if there is a weight cache inside the IP core), or parse the parameters of the N+1th layer in advance, thereby effectively hiding the overhead of software configuration and data preparation, realizing seamless connection between layers, and further improving the overall system performance.

[0068] In summary, the present application successfully constructs an efficient, flexible and resource-saving CNN acceleration system by a set of configurable and reusable IP cores, combined with the dynamic scheduling of the PS and the optimized data flow path, effectively solving many pain points in the prior art.

[0069] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, and they should be included in the scope of the claims of the present application.

Claims

1. A convolutional neural network (CNN) FPGA acceleration system based on a configurable IP core, characterized in that: The system comprises: a processing system PS, which is an on-chip processor; programmable logic PL, which is a programmable logic part of the FPGA; a double-rate synchronous dynamic random access memory DDR, which is shared by the PS and the PL for access; wherein the PL is integrated with a set of configurable computing units, which comprises: at least one configurable convolution batch normalization activation IP core, and at least one configurable pooling IP core; the PS is configured to: according to the parameters of a plurality of network layers in a CNN model to be accelerated, in the order of layers, time-division multiplex the set of computing units by writing configuration parameters to the configurable convolution batch normalization activation IP core and the configurable pooling IP core, to perform the calculation of the plurality of network layers. 2.The configurable IP core based convolutional neural network FPGA acceleration system of claim 1, wherein: The internal of the configurable convolution batch normalization activation IP core comprises: a row buffer memory for caching pixel data of input feature maps; a sliding window generator for reorganizing convolution windows according to the configured convolution kernel size; a parallel multiply-accumulate MAC array for performing convolution operations; a batch normalization BN unit for batch normalization processing of the convolution operation results; and an activation function unit for nonlinear activation processing of the batch normalization processed results. The batch normalization unit is configured to perform the following operation on the input data x: 3.The configurable IP core based convolutional neural network FPGA acceleration system of claim 2, wherein: The output port of the configurable convolution batch normalization activation IP core and the input port of the configurable pooling IP core are directly connected through the streaming protocol AXI4-Stream bus of the advanced extensible interface AXI, so as to realize that the convolution calculation results can be streamed to the pooling IP core for processing without being transferred through the DDR when a network layer containing convolution and pooling is executed. where y is the normalized output, μ is the mean, σ 2 is the variance, γ is the scaling factor, β is the translation factor, and ε is a small constant to prevent division by zero; μ, σ 2 , γ, and β are all parameters dynamically configured by the PS according to the current network layer. 4.The configurable IP core based convolutional neural network FPGA acceleration system of claim 1, wherein: The PL is also integrated with a direct memory access DMA control module, and the PS configures the DMA control module to realize the data transfer between the input feature maps, network weights and intermediate results in the DDR and the IP cores in the PL. 5.The configurable IP core based convolutional neural network FPGA acceleration system of claim 1, wherein: The configurable convolution batch normalization activation IP core and the configurable pooling IP core are both generated by high-level synthesis HLS technology, and receive configuration parameters from the PS through the AXI4-Lite interface of the AXI. 6.The configurable IP core based convolutional neural network FPGA acceleration system of claim 1, wherein: The method is applied to the configurable IP core-based convolutional neural network FPGA acceleration system according to claim 1, and comprises the following steps: 7.A configurable IP based convolutional neural network FPGA acceleration method, characterized in that: S1: initialization step: loading the weights and batch normalization parameters of each layer of the CNN model into the DDR; S2: layer-by-layer calculation step: the PS cyclically executes the following operations according to the order of network layers until all network layers are calculated: S21: parameter configuration: the PS configures the same set of configurable convolution batch normalization activation IP core and configurable pooling IP core in the PL according to the parameters of the current layer to be calculated through the bus interface; S22: data input: the PS controls the data transfer module to transfer the input feature maps of the current layer from the DDR to the PL; S23: acceleration calculation: the configured IP core performs calculation on the input feature map data; ​ S24: result write-back: the PS control data transfer module carries the output feature map of the current layer from the PL to the DDR. 8.The configurable IP-based convolutional neural network FPGA acceleration method of claim 7, wherein: In the S23, when processing a network structure immediately adjacent to a convolutional layer and a pooling layer, the calculation result of the configurable convolutional batch normalization activation IP core is directly transmitted to the configurable pooling IP core through an on-chip streaming bus, realizing pipeline parallel processing of convolution and pooling operations. 9.The configurable IP-based convolutional neural network FPGA acceleration method of claim 7, wherein: The method further includes a step of PS and PL parallel cooperation: when the IP core in the PL performs accelerated calculation of the current layer, the PS asynchronously prefetches and prepares the weight or feature map data required for the next layer calculation into the DDR.

10. The configurable IP based convolutional neural network FPGA acceleration method of claim 7, wherein: The S21 includes: the PS writes the convolution kernel size, stride, input and output channel number, feature map size, and storage address of the weight and batch normalization parameter in the DDR to the configurable convolutional batch normalization activation IP core through the AXI4-Lite interface of the AXI, and writes the pooling window size and stride to the configurable pooling IP core.

Images