Convolutional neural network hardware acceleration system based on Zynq

Abstract

The invention discloses a Zynq-based convolutional neural network hardware acceleration system, which is realized based on a Zynq heterogeneous chip and comprises a PL end module, a PS end module and a storage module. The PL end module comprises a convolution acceleration module, a full connection layer acceleration module and a Softmax activation module and is used for completing convolution calculation, full connection operation and activation function processing; the PS end module comprises an input data processing module and a post-processing module and is responsible for loading and preprocessing input data and sorting and screening detection results. The storage module is composed of an on-chip memory and an off-chip memory, wherein the on-chip memory stores an intermediate processing result of input data, a convolution kernel and intermediate feature map data; and the off-chip memory stores the original image data, the weight parameter of the full connection layer and the final detection result. All the modules are connected through a 64-bit AXI bus, the PS end module is responsible for coordinating data flow, and the PL end module executes efficient parallel computing. The system makes full use of the high-performance characteristic of the Zynq heterogeneous architecture, is suitable for an efficient reasoning acceleration scene of the convolutional neural network, and particularly has a wide application prospect in embedded AI equipment.

Description

Technical Field

[0001] The present invention relates to the field of embedded artificial intelligence hardware acceleration, and in particular to a convolutional neural network hardware acceleration system based on Zynq. Background Art

[0002] With the rapid development of artificial intelligence, convolutional neural networks have been widely used in image classification, target detection and other fields. However, CNN models usually have the characteristics of large computational complexity and high memory bandwidth requirements. When running on traditional general-purpose processors, they face performance bottlenecks and high power consumption problems. This makes how to design efficient hardware accelerators a research focus. In recent years, hardware accelerators based on programmable logic devices have gradually attracted attention. FPGA has the advantages of high performance, low power consumption, and strong flexibility. It is particularly suitable for implementing core operations of CNN such as convolution calculations, fully connected operations, and activation functions. As an integrated architecture of FPGA and ARM processor, Zynq heterogeneous chip not only has powerful parallel computing capabilities, but also can achieve software and hardware collaborative optimization through efficient collaboration between on-chip processing system and programmable logic area. Although there have been some accelerator studies based on Zynq architecture, how to further improve the efficiency of CNN reasoning and optimize data flow and storage structure in resource-constrained embedded environments is still a current technical difficulty. In view of the above problems, the present invention designs a convolutional neural network hardware acceleration system based on Zynq. By reasonably allocating the functions of the PS and PL modules and combining on-chip memory and off-chip memory to optimize data management, the reasoning speed and resource utilization efficiency are significantly improved, providing an efficient solution for embedded AI applications. Summary of the invention

[0003] In view of the shortcomings of the above-mentioned technology, the purpose of the present invention is to provide a convolutional neural network hardware acceleration system based on Zynq, which solves the problems of complex data flow management, low storage utilization and insufficient computing efficiency caused by limited embedded resources in the prior art through collaborative optimization design of software and hardware.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows: A convolutional neural network hardware acceleration system based on Zynq includes a PL end module, a PS end module and a storage module. The PL end module includes a convolution acceleration module, a fully connected layer acceleration module and a Softmax activation module; the PS end module includes an input data processing module and a post-processing module; the storage module includes an on-chip memory and an off-chip memory, the on-chip memory connects the PS end module and the PL end module through a 64-bit AXI bus, and is used to store intermediate data, convolution kernels and feature maps, the off-chip memory is used to store input images, model parameters and final detection results, and is connected to the PS end module and the PL end module through a 64-bit AXI bus, and the final result is stored in the off-chip memory.

[0005] It is further defined that the hardware acceleration system is characterized in that the PL-end module is located in the programmable logic area of ​​the Zynq heterogeneous chip; the PS-end module is located in the processing system area of ​​the Zynq heterogeneous chip; the storage module is located in the onboard DDR3 chip of the Zynq heterogeneous chip, which is beneficial in that by deploying the PL-end module in the programmable logic area, the parallel computing capability of the FPGA is fully utilized to accelerate convolution calculations, fully connected calculations and activation function processing; the PS-end module is deployed in the processing system area, so that it has flexible control and data processing capabilities, can efficiently load input data and perform result screening; the storage module relies on the onboard DDR3 chip, and is closely connected to the PL-end and PS-end modules through a 64-bit AXI bus to achieve high-speed data access and exchange.

[0006] It is further defined that the hardware acceleration system is characterized in that the convolution acceleration module includes: MAC unit array, consisting of 64 MAC units arranged in an 8×8 matrix, with the MAC units connected to the pipeline registers through hard wiring; The data input unit includes four FIFO buffers, each with a capacity of 4KB, connected to the storage module via a 64-bit AXI bus; The data transmission unit transmits the convolution calculation results to the fully connected layer acceleration module through the 64-bit AXI bus. The benefit lies in that each MAC unit in the MAC unit array is connected through a parallel processing structure and a pipeline register. The pipeline register keeps the input and output synchronized in each calculation cycle to reduce the calculation delay. The four FIFO caches of the data input unit perform data scheduling through a grouping mechanism. Each FIFO cache corresponds to a specific channel of the input data. A block transmission protocol is used between the cache and the 64-bit AXI bus to ensure the continuity of data transmission. The data transmission unit uses a pre-allocated AXI address space to write the convolution calculation results directly into the target storage area or pass them to the input interface of the fully connected layer acceleration module, avoiding multiple intermediate storage operations and improving the transmission efficiency of the calculation results and the collaborative performance between modules.

[0007] It is further defined that the hardware acceleration system is characterized in that the fully connected layer acceleration module includes: Matrix operation unit, consisting of 16 parallel matrix multiplication units, each of which is hardwired to the calculation register; The data input unit includes two FIFO buffers, each with a capacity of 8KB, which receives feature map data from the convolution acceleration module through a 64-bit AXI bus; The data output unit is connected to the Softmax activation module through a 64-bit AXI bus to transmit the matrix operation results to the Softmax activation module. The benefit lies in that the matrix operation unit realizes parallel operation of the input feature map and the fully connected layer weights through 16 parallel matrix multiplication units. Each matrix multiplication unit is hard-wired to a dedicated register group, which is responsible for storing the intermediate results of the operation and reducing the access to the on-chip memory; the data input unit uses two 8KB FIFO caches to receive the feature map data of the convolution acceleration module according to the data block allocation mechanism. The FIFO cache manages the transmission queue through the AXI protocol to ensure efficient data transmission; the data output unit directly transmits the matrix operation results to the Softmax activation module input interface through the 64-bit AXI bus. The data transmission adopts the streaming transmission mode to avoid repeated access to the intermediate data and improve the real-time and efficiency of data exchange between modules.

[0008] It is further defined that the Softmax activation module comprises: A lookup table operation unit, comprising a lookup table memory and an address controller, wherein the lookup table memory stores 256 lookup table data and is connected to the parallel computing logic through the address controller; Activate the computing unit, including 4-way parallel computing logic, connected to the lookup table computing unit; The interface unit is connected to the external storage module through a 64-bit AXI bus and transmits the Softmax calculation results to the external storage module. The benefit is that the lookup table operation unit quickly locates the exponent value and normalization parameter through the address controller to avoid complex real-time calculations, and activates the calculation unit to process batch data through a 4-way parallel logic pipeline. The interface unit uses a segmented transmission mechanism to optimize bandwidth utilization and reduce latency. The overall design improves the calculation speed and data transmission efficiency.

[0009] It is further defined that the input data processing module comprises: A data loading unit, loading an input image from a storage module via a DMA interface; The data preprocessing unit includes a normalization unit and a quantization unit, which are used to normalize and quantize the input image. The preprocessed data is stored in the on-chip storage module. The benefit lies in that the data loading unit realizes efficient data transmission through the DMA interface, reduces CPU occupancy and bus contention, the normalization unit uses a fixed-point processing method to scale the pixel values ​​of the input image to match the model requirements, and the quantization unit converts floating-point data into a fixed-point format to reduce computational complexity and storage requirements. The preprocessed data is directly stored in the on-chip storage module to increase access speed and reduce access latency of off-chip storage.

[0010] It is further defined that the post-processing module comprises: The data receiving unit receives the detection results from the PL-side module through the 64-bit AXI bus, including the category label, bounding box coordinates and confidence level; A sorting unit, connected to the data receiving unit via a 64-bit AXI bus, is used to sort the bounding boxes in the detection results according to the confidence level; The IOU calculation unit is connected to the sorting unit and is used to calculate the IOU intersection-over-union ratio between candidate bounding boxes; The box screening unit is connected to the IOU calculation unit and is used to screen the bounding boxes according to the preset threshold and output the final detection result. Its benefits lie in that the data receiving unit realizes the fast transmission of multi-channel detection results through the high-bandwidth AXI bus, the sorting unit uses the hardware priority queue to arrange the bounding boxes in descending order of confidence, the IOU calculation unit uses the pipeline structure to calculate the intersection-union ratio of the candidate boxes frame by frame, and the box screening unit eliminates redundant bounding boxes based on the IOU threshold through hard-wired logic. The entire post-processing module optimizes the screening speed of the detection results and reduces the system latency.

[0011] It is further defined that the storage module comprises: The on-chip memory uses a combination of distributed RAM and block RAM architecture, where the distributed RAM stores intermediate processing results, and the block RAM partitions store convolution kernels and intermediate feature map data. All data is quickly accessed through 64-bit alignment; The off-chip memory is based on the onboard DDR3 memory chip. The raw image data is loaded through the DMA interface. The weight parameters and bias parameters of the fully connected layer are written in batches through the 64-bit AXI bus burst mode. The detection results including category labels, bounding box coordinates and confidence are stored in fixed-size blocks. The on-chip memory and the off-chip memory are respectively connected to the PS-end module and the PL-end module through an independent 64-bit AXI bus. The benefit lies in that the on-chip memory adopts a multi-channel parallel architecture to achieve efficient access to intermediate data, convolution kernels and feature maps, while reducing access conflicts. The off-chip memory supports burst mode and segmented transmission through the 64-bit AXI bus to optimize the loading speed of large-scale parameters. The collaborative work of the on-chip and off-chip memories reduces data transmission delays through address mapping and blocking mechanisms, thereby improving the data processing efficiency and real-time performance of the entire system.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: through the division of labor design between on-chip memory and off-chip memory, the on-chip memory supports up to 256KB of intermediate data cache, and uses distributed RAM to achieve concurrent access to 32-bit data per cycle. The off-chip memory supports up to 1GB / s burst transmission bandwidth through a 64-bit AXI bus, realizing efficient loading of model parameters and input data. The PL end module uses an 8×8 matrix array composed of 64 MAC units, which can complete 64 multiplication and addition operations per cycle; the fully connected operation uses 16 parallel matrix multiplication units, and the operation speed is about 25% higher than that of the traditional FPGA accelerator. The PS end module can process 5,000 bounding box data per second through the hardware sorting unit, and the IOU calculation and bounding box screening unit completes the screening task with a delay of about 100μs. The present invention significantly improves the reasoning performance of the convolutional neural network by optimizing storage, calculation and data flow, and achieves the ability to process 26 images per second in the target detection task, which is suitable for resource-constrained embedded AI applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0014] Figure 1 This is a diagram of the hardware accelerator architecture.

[0015] Figure 2 This is the schematic diagram of the double buffering mechanism. DETAILED DESCRIPTION

[0016] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.

[0017] The present invention provides Figure 1 The Zynq-based convolutional neural network hardware acceleration system shown in the figure consists of a PS module, a PL module, and a storage module. The modules are connected through a 64-bit AXI bus to collaboratively complete the entire process from input data loading to final detection result output, aiming to optimize the performance of convolutional neural network reasoning, and is particularly suitable for resource-constrained embedded scenarios.

[0018] The PS module in the system includes an input data processing module and a post-processing module. The input data processing module loads the original image data from the storage module through the DMA interface and normalizes and quantizes it. The normalization operation maps the pixel value to the numerical range adapted to the model, and the quantization operation converts the floating-point data into an 8-bit integer format to meet the computing requirements of the hardware accelerator and reduce the overhead of data storage and transmission. The processed data is stored in the on-chip memory to provide subsequent computing support for the PL module. The post-processing module receives the detection results output by the PL module, including category labels, bounding box coordinates and confidence, and prioritizes the bounding boxes according to the confidence through the sorting unit, calculates the overlap rate between the candidate boxes using the IOU calculation unit, and then eliminates the redundant bounding boxes through the screening logic. The filtered results are stored in the external memory in a fixed format.

[0019] The PL module is deployed in the programmable logic area of ​​the Zynq chip, mainly including the convolution acceleration module, the fully connected layer acceleration module and the Softmax activation module. The convolution acceleration module realizes feature extraction through an 8×8 MAC unit array. Each MAC unit is responsible for performing a multiplication and addition operation and is connected to the upstream and downstream units through pipeline registers to realize continuous calculation of the data stream. The module adopts a double buffer mechanism. While calculating the convolution result, it preloads the next set of data into the backup buffer to cover the data loading delay. To further improve efficiency, the batch normalization operation of the convolution layer is integrated into the convolution kernel weight and bias, avoiding additional computing overhead and storage access. The fully connected layer acceleration module uses 16 parallel matrix multiplication units to complete the multiplication and addition operation of the weight matrix and the feature map. Combined with the pipeline structure, the calculation and data loading are carried out synchronously, and the calculation results are transmitted to the Softmax activation module through the AXI bus. The Softmax activation module uses a lookup table to store 256 index values ​​and normalization parameters. The address controller quickly accesses the corresponding data through the index, and completes the activation value calculation through a 4-way parallel logic pipeline, and the result is transmitted to the storage module.

[0020] Double buffer mechanism principle combination Figure 2 It is shown that this mechanism realizes the alternation of data loading (Load), calculation (Compute) and storage (Save) by introducing two buffers (Buffer0 and Buffer1) in the hardware accelerator, thereby significantly improving the hardware resource utilization and data processing efficiency.

[0021] In the traditional single buffer mode, each convolution operation needs to wait for the input feature map (ifm) to be loaded before calculation can begin, and after the calculation is completed, the new input feature map must be loaded after the result is stored. In this mode, time is divided into multiple discontinuous stages, and loading, calculation, and storage cannot be performed in parallel. For example, Time0 is used to load ifm, Time1 is used for convolution calculation, and Time2 is used for result storage. The waiting time for loading and storing wastes hardware resources and limits system performance.

[0022] In contrast, the double buffer mechanism adopts a Ping-pong structure, which allows loading, calculation, and storage to be performed simultaneously by alternating two buffers. At Time0, Buffer0 loads the input feature map ifm0; at Time1, while ifm0 of Buffer0 enters the calculation phase, Buffer1 starts to load ifm1; at Time2, the calculation result (ofm0) of Buffer0 is stored, ifm1 of Buffer1 enters the calculation phase, and Buffer0 starts to load the new ifm2. Through this alternating operation, the loading time and the calculation time overlap each other, and system resources can be fully utilized.

[0023] The implementation of this mechanism depends on the precise scheduling of the hardware control logic. In the present invention, the DMA interface connected through the AXI bus is responsible for loading data in burst mode, and the control logic monitors the state of the buffer in real time and switches the tasks of loading and calculating. Each buffer is marked as "load", "calculate" or "store" at a specific time. The system dynamically adjusts the data flow direction and processing order according to the buffer state to ensure the continuity of data flow between different buffers.

[0024] In addition, in order to further improve performance, the present invention combines data slicing technology in the double buffer mechanism to process larger input feature maps in blocks. Each block of data is loaded into the buffer independently, avoiding the bandwidth bottleneck caused by loading large batches of data at a time. At the same time, the pipeline structure is used in conjunction with the double buffer mechanism, and the MAC unit array maintains the continuity of calculation through the pipeline register, eliminating hardware idle time.

[0025] pass Figure 2 It can be clearly seen that the introduction of the double buffering mechanism significantly improves the throughput. In the traditional single buffering mode, the time slices of loading, computing, and storing can only be performed sequentially, and the system throughput is limited. Under the double buffering mechanism, since the time slices of loading and computing overlap, the hardware resources can be fully utilized in each computing cycle, thereby improving the reasoning performance of the system. Actual tests show that when processing high-resolution images or large batches of data, the double buffering mechanism can increase the throughput of convolution calculations by more than 30% and effectively reduce data transmission delays.

[0026] The storage module consists of on-chip memory and off-chip memory. The on-chip memory combines distributed RAM and block RAM design to cache intermediate feature maps, convolution kernels, and bias data, supporting multi-channel parallel read and write operations to improve data access speed. The off-chip memory achieves high-capacity data storage through the onboard DDR3 memory chip. The original image, fully connected layer weight parameters, and final detection results are stored here, and are loaded and written through the burst mode DMA interface, which optimizes the transmission efficiency of large-scale data. The on-chip memory and the off-chip memory collaborate through address mapping and block scheduling, forming an efficient data flow between data loading and calculation.

[0027] The system architecture of the present invention realizes efficient collaboration and pipeline optimization of each module through software and hardware collaborative design. In terms of data flow, through the double buffer mechanism and pipeline register design, loading, calculation and storage operations can be performed in parallel, thereby covering the delay of data loading and significantly improving the system throughput. In terms of computing performance, through the MAC array and pipeline structure of the convolutional layer, combined with Batch Normalization fusion technology and 8-bit quantization technology, the storage requirements and computational complexity are reduced, while ensuring the effectiveness of the model reasoning accuracy. The overall system performs well in convolutional neural network reasoning tasks, is suitable for embedded AI application scenarios such as target detection and image classification, and significantly reduces resource consumption and delay.

Claims

1. A Zynq-based convolutional neural network hardware acceleration system, characterized in that The device is implemented based on a Zynq heterogeneous chip; The convolutional neural network hardware acceleration system includes a PL end module, a PS end module and a storage module; The PL end modules include convolution acceleration module, fully connected layer acceleration module and Softmax activation module; The PS end module includes an input data processing module and a post-processing module; The storage module includes an on-chip memory and an off-chip memory; Among them, the PS end module is connected to the PL end module, the PS end module is connected to the storage module, and the storage module is connected to the PL end module.

2. The hardware acceleration system according to claim 1, characterized in that: The PL end module is located in the programmable logic area of ​​the Zynq heterogeneous chip; The PS end module is located in the processing system area of ​​the Zynq heterogeneous chip; The storage module is located in the DDR3 chip onboard the Zynq heterogeneous chip.

3. The hardware acceleration system according to claim 1, characterized in that: The convolution acceleration module includes: MAC unit array, consisting of 64 MAC units arranged in an 8×8 matrix, with the MAC units connected to the pipeline registers through hard wiring; The data input unit includes four FIFO buffers, each with a capacity of 4KB, connected to the storage module via a 64-bit AXI bus; The data transmission unit transmits the convolution calculation results to the fully connected layer acceleration module through the 64-bit AXI bus.

4. The hardware acceleration system according to claim 1, characterized in that: The fully connected layer acceleration module includes: Matrix operation unit, consisting of 16 parallel matrix multiplication units, each of which is hardwired to the calculation register; The data input unit includes two FIFO buffers, each with a capacity of 8KB, which receives feature map data from the convolution acceleration module through a 64-bit AXI bus; The data output unit is connected to the Softmax activation module through a 64-bit AXI bus and transmits the matrix operation results to the Softmax activation module.

5. The hardware acceleration system according to claim 1, characterized in that: The Softmax activation module includes: A lookup table operation unit, comprising a lookup table memory and an address controller, wherein the lookup table memory stores 256 lookup table data and is connected to the parallel computing logic through the address controller; Activate the computing unit, including 4-way parallel computing logic, connected to the lookup table computing unit; The interface unit is connected to the external storage module through a 64-bit AXI bus and transmits the Softmax calculation result to the external storage module.

6. The hardware acceleration system according to claim 1, characterized in that: The input data processing module comprises: A data loading unit, loading an input image from a storage module via a DMA interface; The data preprocessing unit includes a normalization unit and a quantization unit, which are used to perform normalization and quantization processing on the input image, and the preprocessed data is stored in the on-chip memory.

7. The hardware acceleration system according to claim 1, characterized in that: The post-processing module comprises: The data receiving unit receives the detection results from the PL-side module through the 64-bit AXI bus, including the category label, bounding box coordinates and confidence level; A sorting unit, connected to the data receiving unit via a 64-bit AXI bus, is used to sort the bounding boxes in the detection results according to the confidence level; The IOU calculation unit is connected to the sorting unit and is used to calculate the IOU intersection-over-union ratio between candidate bounding boxes; The box screening unit is connected to the IOU calculation unit and is used to screen the bounding box according to the preset threshold and output the final detection result.

8. The hardware acceleration system according to claim 1, characterized in that: The storage module comprises: On-chip memory, which stores intermediate processing results of input data, caches convolution kernels and bias data, and stores intermediate feature map data; Off-chip memory, which stores raw image data, weight parameters and bias parameters of the fully connected layer, and detection results, including category labels, bounding box coordinates, and confidence levels; The on-chip memory and the off-chip memory are respectively connected to the PS end module and the PL end module via a 64-bit AXI bus.

Images