Universal convolutional neural network hardware accelerator implementation method based on heterogeneous FPGA

Abstract

The invention discloses a universal convolutional neural network accelerator implementation method based on a heterogeneous FPGA (Field Programmable Gate Array), which is suitable for a Xilinx ZYNQ series platform and adopts a collaborative architecture of a processor system and a programmable logic system. The PS end is responsible for model parameter loading, data pre / post-processing, task scheduling and configuration of calculation parameters for the PL end through an AXI bus; a cache module, a convolution calculation module, a pooling calculation module, a residual calculation module and a linear layer calculation module are integrated at the PL end, and operations such as convolution, pooling, residual connection and full connection are supported. Wherein the cache module adopts a ping-pong structure to improve data throughput, and the convolution calculation module realizes multi-channel parallel multiplication and addition operation based on a 7 * 7 DSP (Digital Signal Processor) array and supports a convolution integral block to be matched with a dynamic core in size; the pooling module is compatible with maximum and average pooling; according to the method, through heterogeneous cooperation, ping-pong caching, multi-channel parallelism, block processing, deep assembly line and other technologies, the reasoning efficiency is remarkably improved, the method can flexibly adapt to various mainstream convolutional neural network models, and the method is suitable for edge end low-delay and high-energy-efficiency intelligent reasoning scenes.

Description

Technical Field

[0001] This invention relates to the fields of convolutional neural networks and edge computing machine vision, and in particular to a method for implementing a general convolutional neural network hardware accelerator based on heterogeneous FPGA. Background Technology

[0002] Currently, with the widespread application of convolutional neural network (CNN) technology in machine vision fields such as object recognition and detection, and the continuous improvement in the detection accuracy of CNNs, their model complexity and computational load are also constantly increasing, posing challenges to the deployment of CNNs on computing platforms. Currently, common neural network hardware accelerators mainly include four hardware implementation schemes: CPU, GPU, ASIC, and FPGA. When deployed for edge computing platforms, embedded CPUs are limited by their performance and cannot perform fast inference on high-performance CNN models; GPUs are limited by power consumption and are not suitable for use in edge computing devices that require low power consumption; while ASICs have the advantages of high performance and high energy efficiency, their development cycle is long and their design cost is high. In contrast, FPGAs have advantages such as low power consumption, high flexibility, and reconfigurability, making them an ideal deployment platform for CNNs in edge computing.

[0003] While there has been some research on FPGA-based convolutional neural network accelerators, traditional FPGA-based convolutional neural network acceleration design methods primarily involve hardware-based design for specific neural networks, mapping the convolutional neural network using hardware description languages. For example, Chinese patent CN113762491A discloses an FPGA-based convolutional neural network accelerator. Although this approach allows for hardware optimization design for specific networks, it lacks compatibility with other convolutional neural networks, significantly reducing its versatility. If the network structure changes, most of the hardware circuitry needs to be redesigned, leading to a substantial increase in workload. Summary of the Invention

[0004] Based on the above analysis, it is necessary to propose a general convolutional neural network hardware accelerator implementation method based on heterogeneous FPGAs. Its characteristic is heterogeneous computing, which allows for flexible deployment. This heterogeneous computing refers to the PS-PL dual-end collaborative approach used in this invention to design a general convolutional neural network accelerator architecture. Control algorithms are designed on the PS side, and computational acceleration units are designed on the PL side for intensive computation acceleration. Acceleration of different convolutional neural network models can be achieved through parameter configuration.

[0005] The technical solution of this invention:

[0006] A method for implementing a general convolutional neural network accelerator based on heterogeneous FPGAs includes two parts: a processor system (PS) and a programmable logic system (PL). The hardware platform uses Xilinx's ZYNQ series heterogeneous FPGAs. The overall method includes the following steps:

[0007] Step 1: Design and train a convolutional neural network model based on the PyTorch framework; after training, divide the parameters of each layer of the network into blocks according to the preset computational parallelism, and store the weight parameters of the blocks in binary format to a file according to the processing order; then, deploy the binary file to the SD card or the eMMC storage medium of the ZYNQ development board.

[0008] Step 2: The ZYNQ's PS end reads the weight data to be calculated and the image data to be processed from the SD card or eMMC, and loads them into the PS end's DDR memory;

[0009] Step 3: The PS terminal writes the current layer's calculation configuration parameters to the PL terminal's control register via the AXI bus to specify the specific calculation mode and operation type;

[0010] Step 4: The PS end transmits the feature map data and weight data corresponding to the current calculation block to the PL end via the AXI_DMA interface; after receiving the data, the PL end caches it in the data cache module based on the ping-pong structure design.

[0011] Step 5: The control module at the PL end schedules each computing unit to perform convolution operations according to the configuration parameters in the control register; while the current computing block is being processed, the system preloads the weights and feature data required for the next computing block in parallel.

[0012] Step 6: The PL continuously performs data loading and calculation operations, and during the calculation process, it continuously merges the partial results of the current output channel until all calculation tasks of the current output channel are completed; then, the complete output channel result of the current block is transferred back to the DDR memory of the PS through AXI_DMA;

[0013] Step 7: Repeat steps 4-6 to process all computation blocks of the current network layer in sequence until all computation tasks of the layer are completed.

[0014] Step 8: Repeat steps 3-7 to complete the inference calculation of the entire neural network layer by layer, and finally upload the network output results to the host computer via serial port.

[0015] In the above technical solution, the PS end includes an ARM processor core, DDR memory, DMA data transfer module, AXI bus, and processor control algorithm; the processor control algorithm includes data read control, DMA data transfer control, and network layer calculation parameter control; the data read control is used to read the image data to be processed and weight data into the DDR memory and store the data into their respective arrays; the DMA data transfer control is used to send data to the PL end for accelerated calculation, and it sends the data according to the memory address corresponding to the data; the network layer calculation parameter control is used to control the specific parameters when the PL end calculates each layer, and sends control instructions to the PL end control register through the AXI_Lite protocol bus; in addition, the PS end also includes pre- and post-processing algorithms for convolutional neural networks, used to process the image data to be processed and the final network result.

[0016] The PL-end convolutional neural network accelerator includes a control module, a cache module, a convolution calculation module, a pooling calculation module, a residual calculation module, and a linear layer calculation module.

[0017] The PL-side control module mainly includes a control register and a status register. Before the current layer inference begins, the PL receives computation control instructions from the PS via the AXI_Lite interface, parses the instructions, and writes them into the control register. The parameters stored in the control register include computation type, kernel size, padding size, stride, number of data blocks to be processed in the current layer, whether pooling is enabled, and pooling type, etc., used to configure and drive each computation unit to complete the computation task of the current layer. The status register manages the overall computation status of the PL and also monitors and coordinates the operating status of each functional module. Furthermore, the contents of the status register can be fed back to the PS in real time via the AXI_Lite interface, enabling the PS to dynamically grasp the PL's work progress and operating status, and perform task scheduling and process control accordingly.

[0018] The caching module includes an input feature map cache, an input weight cache, an input bias cache, and an output buffer module. The input feature map cache, input weight cache, and input bias cache all use ping-pong caching to improve loading efficiency. The output buffer module uses FIFO to buffer the calculation result data to be output to DDR to avoid data loss.

[0019] The convolution calculation module includes a padding module, a matrix generation module, a DSP array module, a channel merging module, and a quantization module. The padding module fills the feature map to control the size of the final output feature map. Before convolution and pooling layer calculations, it pads the image edges with zeros. The padding module uses row and column counters to control the zero-padding positions. When the counter reaches the corresponding position, it pulls the read enable signal low and outputs 0; at other positions, it outputs the corresponding feature map data. The matrix generation module generates the convolution window, which consists of multiple synchronous FIFOs and control circuits. The first and second ends of the multiple FIFOs are connected together, and data sequentially enters the multiple FIFOs. Each FIFO (First FIFO) generates a data matrix by arranging the input data with the output data of the previous n FIFOs. The DSP array module is used to multiply and accumulate the input feature data and weight data. Each DSP array consists of 7*7 DSP modules, which are split and used according to the size of the current layer convolution kernel given by the calculation instruction during actual calculation. The channel merging module is used to accumulate and merge the results calculated by the DSP array, and obtain the final calculation result of the current output channel through multiple iterations. The quantization module is used to requantize the final calculation result of the current channel, requantizing the calculation result into an 8-bit result.

[0020] The pooling calculation module is used to calculate the pooling layer operations in the neural network. It includes a control module, a max pooling calculation module, and an average pooling calculation module. The control module is used to dynamically select the pooling calculation method to be executed according to the pooling type parameter written in the configuration register. The max pooling calculation module is used to implement the operation of taking the maximum value within the window. The average pooling calculation module is used to implement the operation of taking the average value within the window. The pooling module is placed after the convolution calculation module and can perform downsampling processing on the convolution output results according to the actual needs of the network structure.

[0021] The residual calculation module is used to implement residual connection calculation in the convolutional neural network. It includes an addition-requantization module, wherein the addition module is used to add and merge the two sets of input feature maps of the main path output and the skip connection path element by element, and the requantization module is used to requantize the result after addition in order to unify the data bit width and maintain quantization consistency.

[0022] The linear layer calculation module is used to calculate the fully connected linear transformation layer in the convolutional neural network, and completes the mapping from high-dimensional features to categories through matrix multiplication and bias superposition.

[0023] Compared with the prior art, the advantages of the present invention are as follows: The present invention is based on heterogeneous FPGA and designs a general convolutional neural network accelerator implementation method. The general convolutional neural network accelerator designed based on this implementation method achieves accelerated computation through PS-PL co-processing, ping-pong buffering, multi-channel parallelism, convolution block processing, pipeline architecture and other methods. It can achieve effective inference acceleration for mainstream convolutional neural networks. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a general convolutional neural network hardware accelerator based on heterogeneous FPGA.

[0025] Figure 2 This is a schematic diagram of the hardware accelerator structure at the PL end.

[0026] Figure 3 This is a schematic diagram of convolution calculation using a DSP array. Detailed Implementation

[0027] To more clearly illustrate the technical solutions in the implementation of this invention, the technical solutions of this invention will be fully described below in conjunction with the accompanying drawings.

[0028] First, the convolutional neural network to be accelerated needs to be trained and quantized. Specifically, the target convolutional neural network is trained using the PyTorch framework to obtain the weight and bias data for each layer. Then, these parameters are quantized using 8-bit fixed-point quantization: the weight data uses symmetric quantization, mapping to the integer range [-128, 127]; the feature map data uses asymmetric quantization, mapping to [0, 255]; and the bias data is converted to a 32-bit fixed-point format to retain sufficient dynamic range and computational precision. After quantization, the quantized model parameters and related data, such as zeros and quantization factors, are structured and saved. Scalar parameters such as zeros and quantization factors are directly written to the corresponding binary configuration file; tensor data such as weights and biases are stored in blocks according to the accelerator's parallel architecture to optimize the efficiency of subsequent data loading from external storage to the PL. In this embodiment, the parallelism of both the input and output channels of the convolutional neural network accelerator is set to 4. Therefore, for the convolutional weight tensor of dimension (k_in, k_out, n, n), its shape is reshaped to (k_in / 4, k_out / 4, n*n, 4, 4). When transferring data from DDR to PL for computation, the weight data block to be read for each computation has a shape of (n*n, 4, 4), where n is the convolutional kernel size, which perfectly matches the parallel computing unit structure of the accelerator, thereby achieving efficient and continuous parallel loading and computation. Following the above block-based strategy, the weight data of each layer of the network is saved layer by layer as an independent binary file; the bias data also uses a similar block-based and aligned method and is saved to the corresponding binary file. Finally, all processed model files are uniformly stored in a non-volatile storage medium such as an SD card or onboard eMMC, for loading by the PS end and scheduling to the PL end accelerator for inference tasks after system startup. For the input feature map, this embodiment also stores it in the SD card, and loads it into DDR storage during inference to await inference.

[0029] In this embodiment, the general convolutional neural network accelerator is implemented based on a heterogeneous FPGA, and the specific architecture is as follows: Figure 1 As shown, it completes functions such as data loading, data transmission, layer-by-layer parameter control, data preprocessing, and network post-processing through the PS end; the PL end is mainly used to implement computationally intensive tasks, and it implements the main computations in the convolutional neural network based on programmable logic units.

[0030] In this embodiment, the PS terminal mainly undertakes core functions such as data loading, parameter configuration, task scheduling, data preprocessing and postprocessing, and collaborative control with the PL terminal. Before the accelerated computation starts, the PS terminal reads the various parameters required for inference, including quantized weight data, bias data, zero points, quantization factors, etc., from non-volatile storage media such as SD cards or onboard eMMC, and loads them into external DDR memory. The weight and bias data are stored layer by layer into the parameter buffer of the corresponding layer according to the rearrangement format optimized for parallel architecture. At the same time, it reads the input image data to be inferred and performs preprocessing operations such as normalization, channel rearrangement, and quantization to meet the requirements. The input requirements of the accelerator are met. The inference process is initiated by the PS end sending a computation start command to the PL end via the AXI_Lite bus. After receiving the ready confirmation signal from the PL end, the layer-by-layer inference phase officially begins. For each network layer, the PS end first configures the control parameters of the current layer to the PL end via the AXI_Lite bus, including computation type, kernel size, padding size, stride, number of data blocks to be processed in the current layer, whether to enable pooling operation and pooling type, etc., to dynamically adapt to the structural requirements of different network layers. Then, the AXI_DMA high-speed data transfer module transfers the corresponding data of the current layer according to the preset block strategy. Feature map data, weight data, and bias data are sequentially transmitted to the on-chip cache of the PL (Programmer) in blocks. While the PL performs calculations on the current block, it simultaneously preloads the data for the next block, achieving pipelined overlap between computation and data transmission to improve overall throughput efficiency. When the spatial size of the input feature map is too large to be loaded completely in one go, the PS (Programmer) can further divide it into multiple spatial blocks for batch processing based on its dimensions. In this embodiment, the parallelism of both the input and output channels is set to 4. If spatial block division is required for a single-channel feature map, the number of blocks is dynamically selected as 4 or 16 based on its height and width dimensions to balance computational efficiency and on-chip storage. Resource utilization; After each block is calculated, the PS reads the calculation result from the PL through the AXI_DMA module and temporarily stores it in the corresponding result buffer array in DDR for use by subsequent layers. After the data transmission is completed, it receives the current block calculation completion signal returned by the PL and then starts the processing flow of the next calculation block. After all blocks of the current network layer have been processed, the PL sends the current layer calculation completion signal. After receiving the signal, the PS starts the configuration and starts the calculation flow of the next layer. When all network layers are calculated, the PS performs post-processing operations such as dequantization and classification on the final result output by the PL to generate the final inference result.

[0031] In this embodiment, the PL terminal is mainly used to accelerate computation for computationally intensive tasks. The hardware architecture of the PL terminal is as follows: Figure 2As shown, the PL terminal includes a control module, a cache module, and a calculation module. The control module controls the specific parameters of each layer's calculation, the PL terminal acceleration system, and the specific state control of individual calculation modules. The cache module mainly receives the feature map data to be processed and calculation parameters such as weights, biases, quantization factors, and quantization zeros transmitted from the PS terminal. It also buffers the calculation results after the calculation is completed to avoid data loss during data transmission to the PS terminal. The calculation module mainly implements various calculations in mainstream convolutional neural networks, including convolution calculation modules, pooling calculation modules, linear layer calculation modules, and residual calculation modules. It is scheduled according to the control register parameters in the PL terminal control module and loads data from the cache module to complete the corresponding calculations.

[0032] The PL-side control module mainly includes a control register and a status register. Before the current layer inference begins, the PL receives computation control instructions from the PS via the AXI_Lite interface, parses the instructions, and writes them into the control register. The parameters stored in the control register include computation type, kernel size, padding size, stride, number of data blocks to be processed in the current layer, whether pooling is enabled, and pooling type, etc., used to configure and drive each computation unit to complete the computation task of the current layer. The status register manages the overall computation status of the PL and also monitors and coordinates the operating status of each functional module. Furthermore, the contents of the status register can be fed back to the PS in real time via the AXI_Lite interface, enabling the PS to dynamically grasp the PL's work progress and operating status, and perform task scheduling and process control accordingly.

[0033] The PL-end cache module is mainly used to cache the feature data to be processed and the calculation parameters such as weights and biases transmitted from the PS end to the PL end via AXI_DMA. It mainly includes feature cache, weight cache, bias cache and output buffer module. The feature cache is used to receive feature data to be processed from the PS end. In this embodiment, a 4-channel parallel input structure is adopted and the feature data is in 8-bit quantization format. Therefore, the feature cache adopts a 32-bit ping-pong cache structure. By synchronously preloading the next data block during the calculation process, the data loading efficiency is effectively improved. The weight cache is used to store the weight data required for the current calculation block. It is also implemented using a 32-bit ping-pong cache. According to the block storage method of the weight data, the current block weight data with the shape (n*n, 4, 4) is read in parallel each time it is loaded. The bias cache is used to store the bias parameters in 32-bit fixed-point format. It adopts a 32-bit conventional cache structure. The corresponding bias value is loaded one by one during the calculation process. The output buffer module is used to temporarily buffer the calculation results sent back from the PL end to the PS end to prevent data loss due to bus congestion or timing mismatch, and ensure that the result data is transmitted to the PS end completely and reliably.

[0034] The PL-side computing module is mainly used to implement the calculations of each layer in the convolutional neural network, including convolutional layers, pooling layers, residual calculations, and linear layers. The computing module controls the current calculation by reading the calculation control parameters in the control register. During calculation, the specific computing module is controlled by the status register. When performing calculations, the corresponding computing module first reads the calculation control parameters from the control register, then reads the required feature data and weight data from the cache module, performs calculations according to the control parameters, and finally transmits the calculation results to the output buffer to wait for transmission to the PS-side DDR storage.

[0035] The convolutional layer computation module is mainly used to compute the convolutional layers in the convolutional neural network. It consists of a padding module, a matrix generation module, a DSP array module, a channel merging module, and a quantization module. The padding module fills the feature map to control the size of the final output feature map. Before the convolution and pooling layer computations, it pads the image edges with zeros. The padding module uses row and column counters to control the zero-padding positions. When the counter reaches the corresponding position, it pulls the read enable signal low and outputs 0; at other positions, it outputs the corresponding feature map data. The matrix generation module generates the convolutional window, which consists of multiple synchronous FIFOs and control circuitry. The first and second ends of the multiple FIFOs are connected together, and data enters the multiple FIFOs sequentially. By arranging the input data with the output data of the first n FIFOs, a data matrix is ​​generated for each FIFO. These two modules then generate the convolutional window from the feature data for subsequent multiplication and addition calculations.

[0036] The DSP array is the core computing unit of the convolution computation module, mainly used to efficiently perform convolution operations through the DSP multiply-accumulate computation unit. In this embodiment, due to the use of an architecture with a parallelism of 4 for both the input and output channels, 16 independent convolution computations need to be completed in parallel simultaneously. To balance versatility and broad compatibility with mainstream convolutional neural networks, a single DSP subarray is organized in a 7*7 scale, and the overall DSP array contains a total of 7*7*16, or 784 DSP units. For layers with a convolution kernel size smaller than 7*7, effective computing resources can be flexibly reused in the 7*7 DSP array by controlling the data flow path, achieving compatible computation without hardware reconstruction. During the computation process, the DSP array first reads the weight data of the current block from the weight cache and loads it into the register inside the corresponding DSP unit; then it receives the constructed convolution window data from the matrix generation module and performs parallel multiply-accumulate operations; the intermediate results are initially aggregated according to the output channel dimension and then sent to the channel merging module for further processing.

[0037] The channel merging module and the quantization module jointly handle the fusion and post-processing of the intermediate results output by the DSP array. Since this embodiment employs a design with a parallelism of 4 for both input and output channels, a single-layer convolution needs to be performed multiple times, resulting in a total of k_in*k_out / 16 calculations. Each calculation only yields a partial accumulated result for the corresponding output channel; therefore, the channel merging module must accumulate all block results channel by channel to obtain the complete output channel activation value. The quantization module then performs subsequent processing on the final result of each output channel after channel merging: first, it adds the result to the corresponding channel's bias data; then, it performs a requantization operation on the added result, mapping it to an 8-bit unsigned integer within the range of 0 to 255 to meet the data format requirements of subsequent layers and reduce transmission and storage overhead.

[0038] The pooling computation module is used to execute pooling layer operations in the neural network, realizing spatial downsampling of the feature map. It includes a control module, a max-pooling computation module, and an average-pooling computation module. The control module dynamically selects the pooling method to be executed based on the pooling type parameter written in the configuration register. The max-pooling computation module compares the values ​​of each element within a preset pooling window and outputs the maximum value to retain the most significant feature response. The average-pooling computation module calculates the arithmetic mean of all elements within the window to achieve a smoothing and noise reduction effect. Deployed after the convolution computation module, the pooling computation module can flexibly downsample the feature map output by convolution according to the actual needs of different network structures, thereby reducing subsequent computation and enhancing the translation invariance of the model.

[0039] The residual calculation module is used to implement residual connection calculation in the convolutional neural network, and its core is the addition-requantization module. The addition module receives the convolution output from the main path and the identity mapping input from the skip connection path, and adds the two sets of feature maps element-wise to complete residual fusion. The requantization module then requantizes the addition result, converting it into an 8-bit fixed-point format consistent with other parts of the network. This ensures data bit width alignment and maintains quantization consistency throughout the entire link, avoiding the accumulation of inference errors due to precision mismatch.

[0040] The linear layer computation module is used to implement the fully connected linear transformation layer in the convolutional neural network, mainly responsible for the mapping function from the high-dimensional feature space to the final class logits. This module multiplies the input feature vector with the weight matrix through large-scale matrix multiplication operations, and adds the corresponding bias terms to output the inactive classification score. To adapt to the overall accelerator's dataflow architecture, the linear layer computation can reuse the DSP array resources in the convolutional computation module, and combine dedicated scheduling logic to optimize memory access efficiency.

[0041] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described in detail. However, as long as the combination of these technical features is not logically contradictory and can jointly achieve the technical objective of the present invention, it should be considered as the content recorded in this specification and falls within the protection scope of the present invention.

[0042] The embodiments described above are merely illustrative of the technical solutions of the present invention, and while their descriptions are specific and detailed, they should not be construed as limiting the scope of protection of this invention. It should be noted that those skilled in the art can make various modifications, substitutions, or improvements to the above embodiments without departing from the core concept of the present invention, such as adjusting parallelism configuration, optimizing cache structure, and expanding supported network layer types. These derivative solutions should all be included within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims and their equivalent meanings and scope.

Claims

1. A method for implementing a general-purpose convolutional neural network hardware accelerator based on heterogeneous FPGA, characterized in that, It consists of two parts: a processor system (PS) and a programmable logic unit (PL). The hardware platform uses Xilinx's ZYNQ series heterogeneous FPGA chips. On the PS side, a system control algorithm is designed to control the overall computation process of the convolutional neural network. On the PL side, a computation acceleration unit is designed to accelerate computation for computationally intensive tasks. The processor system includes a Cortex-M0 processor core, an AXI_DMA data transfer module, DDR memory, an AXI bus, and a processor operation control algorithm. The processor control algorithm includes data read control, DMA data transfer control, and network layer calculation parameter control. The data read control is used to read the image data to be processed and the weight data into the DDR memory and store the data into their respective arrays. The DMA data transfer control is used to send data to the PL end for accelerated calculation, and it sends the data according to the memory address corresponding to the data; the network layer calculation parameter control is used to control the specific parameters when the PL end calculates each layer, and sends control instructions to the PL end control register through the AXI_Lite protocol bus; in addition, the PS end also includes pre- and post-processing algorithms for convolutional neural networks, which are used to process the image data to be processed and the final network result; The PL-side computation acceleration unit includes a control module, a cache module, and a computation module. The computation module includes a convolution computation module, a pooling computation module, a residual computation module, and a linear layer computation module, which are used to implement various computations commonly used in convolutional neural networks.

2. The method for implementing a general convolutional neural network hardware accelerator based on heterogeneous FPGA according to claim 1, characterized in that: The PL-side control module mainly includes a control register and a status register. Before the current layer inference begins, the PL receives computation control instructions from the PS via the AXI_Lite interface, parses the instructions, and writes them into the control register. The parameters stored in the control register include computation type, kernel size, padding size, stride, number of data blocks to be processed in the current layer, whether pooling is enabled, and pooling type, etc., used to configure and drive each computation unit to complete the computation task of the current layer. The status register manages the overall computation status of the PL and also monitors and coordinates the operating status of each functional module. Furthermore, the contents of the status register can be fed back to the PS in real time via the AXI_Lite interface, enabling the PS to dynamically grasp the PL's work progress and operating status, and perform task scheduling and process control accordingly. The PL-end cache module includes an input feature map cache, an input weight cache, an input bias cache, and an output buffer module. The input feature map cache, input weight cache, and input bias cache all use ping-pong caching to improve loading efficiency. The output buffer module uses FIFO to buffer the calculation result data to be output to DDR to avoid data loss. The PL-side computing module includes a convolution computing module, a pooling computing module, a residual computing module, and a linear layer computing module, which are used to implement various accelerated computing methods in the convolutional neural network.

3. The method for implementing a general-purpose convolutional neural network hardware accelerator based on heterogeneous FPGA as described in claim 2, further comprising: The PL-side cache module includes a feature cache, a weight cache, a bias cache, and an output buffer module. The feature cache and weight cache adopt a 32-bit ping-pong cache structure; the bias cache adopts a 32-bit conventional cache structure; and the output buffer module is used to temporarily cache the calculation results sent back from the PL end to the PS end to prevent data loss due to bus congestion or timing mismatch.

4. The method for implementing a general convolutional neural network hardware accelerator based on heterogeneous FPGA as described in claim 2, Its features also include: The convolutional layer calculation module is mainly used to calculate the convolutional layers in the convolutional neural network. It consists of a padding module, a matrix generation module, a DSP array module, a channel merging module, and a quantization module. The padding module is used to fill the feature map to control the size of the final output feature map. Before the convolution and pooling layer calculations, zeros are padded at the image edges. The padding module uses a row and column counter to control the position of the zeros. When the counter counts to the corresponding position, it pulls the read enable signal low and outputs 0. At other positions, it outputs the corresponding feature map data. The matrix generation module is used to generate the convolution window. It consists of multiple synchronous FIFOs and control circuits. The first and second ends of the multiple FIFOs are connected together. Data enters the multiple FIFOs sequentially. By arranging the input data with the output data of the first n FIFOs, a data matrix can be generated at each FIFO. After these two modules, the feature data is used to generate a convolution window for subsequent multiplication and addition calculations; the DSP array module uses 784 DSP units (7*7*16) to complete convolution calculations with a parallelism of 4 for both the input and output channels, and can be split to be compatible with all convolution calculations of size less than or equal to 7*7; the channel merging module and the quantization module are jointly responsible for fusing and post-processing the intermediate results output by the DSP array. The pooling calculation module is used to execute pooling layer operations in the neural network to achieve spatial downsampling of the feature map. It includes a control module, a max-pooling calculation module, and an average-pooling calculation module. The control module dynamically selects the pooling method to be executed based on the pooling type parameter written in the configuration register. The max-pooling calculation module compares the values ​​of each element within a preset pooling window and outputs the maximum value to retain the most significant feature response. The average-pooling calculation module calculates the arithmetic mean of all elements within the window to achieve a smoothing and noise reduction effect. Deployed after the convolution calculation module, the pooling calculation module can flexibly downsample the feature map output by convolution according to the actual needs of different network structures, thereby reducing subsequent computation and enhancing the model's translation invariance. The residual calculation module is used to implement residual connection calculation in the convolutional neural network, and its core is the addition-requantization module. The addition module receives the convolution output from the main path and the identity mapping input from the skip connection path, and adds the two sets of feature maps element-wise to complete residual fusion. The requantization module then requantizes the addition result, converting it into an 8-bit fixed-point format consistent with other parts of the network. This ensures data bit width alignment and maintains quantization consistency throughout the entire link, avoiding the accumulation of inference errors due to precision mismatch. The linear layer computation module is used to implement the fully connected linear transformation layer in the convolutional neural network, mainly responsible for the mapping function from the high-dimensional feature space to the final class logits. This module multiplies the input feature vector with the weight matrix through large-scale matrix multiplication operations, and adds the corresponding bias terms to output the inactive classification score. To adapt to the overall accelerator's dataflow architecture, the linear layer computation can reuse the DSP array resources in the convolutional computation module, and combine dedicated scheduling logic to optimize memory access efficiency.

5. A method for implementing a general-purpose convolutional neural network hardware accelerator based on heterogeneous FPGA, characterized in that the method for implementing the general-purpose convolutional neural network hardware accelerator is as follows: Step 1: Design and train a convolutional neural network model based on the PyTorch framework; after training, divide the parameters of each layer of the network into blocks according to the preset computational parallelism, and store the weight parameters of the blocks in binary format to a file according to the processing order; then, deploy the binary file to the SD card or the eMMC storage medium of the ZYNQ development board. Step 2: The ZYNQ's PS end reads the weight data to be calculated and the image data to be processed from the SD card or eMMC, and loads them into the PS end's DDR memory; Step 3: The PS terminal writes the current layer's calculation configuration parameters to the PL terminal's control register via the AXI bus to specify the specific calculation mode and operation type; Step 4: The PS end transmits the feature map data and weight data corresponding to the current calculation block to the PL end via the AXI_DMA interface; After receiving the data, the PL end caches it in a data cache module based on a ping-pong structure design; Step 5: The control module at the PL end schedules each computing unit to perform convolution operations according to the configuration parameters in the control register; while the current computing block is being processed, the system preloads the weights and feature data required for the next computing block in parallel. Step 6: The PL continuously performs data loading and calculation operations, and during the calculation process, it continuously merges the partial results of the current output channel until all calculation tasks of the current output channel are completed; then, the complete output channel result of the current block is transferred back to the DDR memory of the PS through AXI_DMA; Step 7: Repeat steps 4-6 to process all computation blocks of the current network layer in sequence until all computation tasks of the layer are completed; Step 8: Repeat steps 3-7 to complete the inference calculation of the entire neural network layer by layer, and finally upload the network output results to the host computer via serial port.

Images