An FPGA neural network data flow architecture implementation method based on computing-storage collaborative optimization

Abstract

The application discloses an FPGA neural network data stream architecture implementation method based on computing-storage collaborative optimization. The method comprises the following steps: firstly, constructing resource mapping logic, independently configuring the cluster size of a processing unit and the resource capacity of an on-chip block memory for each layer; secondly, constructing a line buffer data stream controller and a static weight storage array, loading all weight parameters of the neural network to the static storage area at one time; thirdly, constructing a deep pipeline interconnection structure, forming an end-to-end direct data path between layers; fourthly, constructing a dynamic power consumption management unit, dynamically managing idle processing units; and finally, integrating signals and converging time sequences of each unit to generate an FPGA neural network accelerator. The application realizes accurate matching of neural network computing-storage resources, reduces data transmission overhead, improves data throughput, realizes fine power consumption control, and improves the energy efficiency ratio of the neural network.

Description

Technical Field

[0001] This invention relates to the field of hardware acceleration technology for deep learning models, and in particular to an FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization. Background Technology

[0002] With the widespread application of deep learning models in mobile devices, edge devices, and real-time systems, inference efficiency directly determines user experience and system feasibility. FPGA, as a reconfigurable hardware platform, demonstrates unique advantages in accelerating neural network inference.

[0003] However, existing FPGA neural network accelerator designs face the following key challenges: (1) mismatch between computing units and storage resources, resulting in low utilization of computing units and storage bandwidth bottlenecks; (2) lack of flexibility in data flow scheduling, making it difficult to adapt to different network structures and data reuse modes; (3) weight loading usually adopts a dynamic loading mechanism, which leads to frequent access to external storage during inference, increasing latency and reducing energy efficiency; (4) the power management mechanism is crude and cannot achieve fine power control while maintaining performance.

[0004] Therefore, there is an urgent need to invent a neural network accelerator design method that can achieve dynamic matching of computing units and storage resources during neural network inference, so as to solve the problem of uneven resource allocation in the existing technology and improve the energy efficiency and resource utilization of hardware accelerators. Summary of the Invention

[0005] The purpose of this invention is to provide a method for implementing an FPGA neural network dataflow architecture based on computation-storage collaborative optimization in FPGA neural network accelerator design, which features high accuracy in matching computation and storage resources, low data transmission overhead, high data throughput, precise power consumption control, and high energy efficiency.

[0006] The technical solution to achieve the purpose of this invention is: a method for implementing an FPGA neural network dataflow architecture based on computation-storage co-optimization, comprising the following steps:

[0007] Step 1: Construct resource mapping logic, analyze the parameters and structure of the target neural network, and configure the cluster size of the processing unit and the on-chip block memory resource capacity for each layer independently.

[0008] Step 2: Construct a row buffer data flow controller. Through the row buffer data flow controller, plan the transmission path, dwell time and multiplexing sequence of feature maps and intermediate data between the on-chip buffer and processing unit. Construct a static weight storage array to load all weight parameters of the neural network into the pre-allocated on-chip static storage area at once.

[0009] Step 3: Construct a deep pipeline interconnection structure, hard-wire the processing unit clusters of each layer in the forward propagation order, so that the output of the upper layer is directly used as the input of the lower layer after the inter-layer buffer, forming an end-to-end inter-layer direct data path.

[0010] Step 4: Construct a dynamic power management unit to continuously sample the combination state of the valid input data signal and the computing enable signal of each processing unit in the inter-layer direct data path, and dynamically cut off the local clock supply for processing units that continuously meet the idle judgment condition.

[0011] Step 5: Integrate the resource mapping logic, row buffer data flow controller, static weight storage array, deep pipeline interconnect structure and dynamic power management unit to generate an FPGA neural network accelerator.

[0012] Furthermore, the resource mapping logic described in step 1, which parses the parameters and structure of the target neural network, and independently configures the processing unit cluster size and on-chip block memory resource capacity for each layer, is as follows:

[0013] Step 1.1: Construct resource mapping logic and analyze the topological information of each layer of the target neural network, including convolution kernel size, input / output feature map dimension, stride, padding method, and computation graph connectivity.

[0014] Step 1.2: Based on the computational load and data access characteristics of each layer of the target neural network, configure the processing unit cluster size independently for each layer;

[0015] Step 1.3: Based on the computational load and data access characteristics of each layer of the target neural network, configure the on-chip block memory resource capacity independently for each layer.

[0016] Furthermore, in step 1.2, based on the computational load and data access characteristics of each layer of the target neural network, the processing unit cluster size is configured independently for each layer, as detailed below:

[0017] For neural networks Layer, calculate its theoretical computational workload Combined with target reasoning delay constraints Determine the number of processing units required. The constraints are satisfied:

[0018]

[0019] in, η is the system operating clock frequency; η is the effective computational efficiency factor of a single processing unit under typical workload, with a value of η = 0.7~0.95; This represents the maximum allowable computational latency for this layer.

[0020] Furthermore, step 1.3, which involves independently configuring on-chip block memory resource capacity for each layer based on its computational load and data access characteristics, is as follows:

[0021] Step 1.3.1, Memory area partitioning, during the FPGA placement phase... Divided into a dedicated storage area for weights With dynamic data buffer pool Satisfying capacity constraints ;

[0022] Step 1.3.2, weight storage area configuration, will The capacity is set to Its address decoding logic, read / write control signals, and timing are exclusively managed by the static weight loading module for persistent storage. All weight parameters of the layer;

[0023] Step 1.3.3, Dynamic Buffer Pool Management: This involves managing the dynamic data buffer pool. The capacity is set to The line buffer is dynamically divided into an input line buffer and a partial sum line buffer by the line buffer controller;

[0024] Step 1.3.4, the first On-chip storage capacity Constraints must be met:

[0025]

[0026] in, The minimum storage capacity required to store all weight parameters of this layer. The storage capacity required to cache a single row or a single data block of input feature map or intermediate portion and data.

[0027] Furthermore, step 2 involves constructing a row buffer data flow controller. This controller plans the transmission path, dwell time, and multiplexing sequence of feature maps and intermediate data between the on-chip buffer and processing unit. A static weight storage array is then constructed, loading all weight parameters of the neural network into the pre-allocated on-chip static storage area at once, as detailed below:

[0028] Step 2.1: Construct a row buffered data flow controller with a double buffering mechanism, and plan the transmission path, dwell time and multiplexing sequence of the input feature map data and the intermediate part and data between the on-chip buffer unit and the processing unit;

[0029] Step 2.2: Construct a static weight storage array. During the hardware initialization phase after the FPGA configuration is completed, load all weight parameters of the neural network into the pre-allocated on-chip static storage area at once.

[0030] Furthermore, the construction of the row-buffered data flow controller with a double-buffering mechanism described in step 2.1, which plans the transmission path, dwell time, and multiplexing sequence of the input feature map data and the intermediate data between the on-chip buffer unit and the processing unit, is as follows:

[0031] Step 2.1.1: Define data reuse rate evaluation indicators The calculation formula is:

[0032]

[0033] in This represents the total number of multiplication and addition operations in a single reasoning operation. This is the total number of times the input feature map is read from the on-chip buffer and the intermediate portion is summed.

[0034] Step 2.1.2: Construct a row-buffered data flow controller with a double-buffering mechanism, plan the transmission path, dwell time and multiplexing sequence of input feature map data and intermediate data between the on-chip buffer unit and the processing unit, cache the input feature map to the input row buffer unit according to the preset row / block, and cache the convolution intermediate data to the partial data row buffer unit.

[0035] Furthermore, in step 2.2, the construction of the static weight storage array involves loading all weight parameters of the neural network into the pre-allocated on-chip static storage area during the hardware initialization phase after FPGA configuration, as detailed below:

[0036] Step 2.2.1: During the initialization window after FPGA configuration is complete, transfer the weight parameter set via PCIe DMA or the configuration memory interface. Write multiple on-chip BRAM memory banks in a hierarchical order to construct a static weighted memory array;

[0037] Step 2.2.2: During the inference phase, the data flow controller generates an address sequence in real time. The computing unit is arranged according to Read the weight values, where For BRAM index, This represents the word offset address within the memory.

[0038] Furthermore, the deep pipeline interconnection structure described in step 3 involves hard-connecting the processing unit clusters of each layer in the forward propagation order, so that the output of the upper layer is directly used as the input of the lower layer after being buffered between layers, forming an end-to-end interlayer direct data path, as detailed below:

[0039] Step 3.1: Set the number of pipeline segments Equal to the number of effective computational layers in the neural network, the th The segment corresponds to the first The single-frame data processing time of a layer is denoted as By adjusting the number of processing units in each segment And internal parallelism, so that the processing time of all pipeline stages meets the balance constraint:

[0040]

[0041] in , , Based on the system clock cycle The set allowable time deviation threshold ranges from 1 to 3 times. After the pipeline enters steady state, the interval between adjacent output frames in steady state is... equal ;

[0042] Step 3.2: Adjacent layer processing unit clusters connect data via an asynchronous FIFO. This FIFO has an independent read / write clock domain, configurable depth, and full / empty status flags; connecting the... Layer and First FIFO depth of the layer Must meet ,in For the first The maximum time, in seconds, for a layer to write a complete feature map data block to the FIFO in continuous working mode; The system clock cycle is in seconds; deep configuration is used to prevent FIFO overflow or underflow during burst writes or consumption delays, maintaining pipeline data continuity.

[0043] Furthermore, the dynamic power management unit described in step 4 continuously samples the combination state of the valid input data signal and the computation enable signal of each processing unit in the inter-layer direct data path, and dynamically cuts off the local clock supply for processing units that continuously meet the idle determination condition, as follows:

[0044] Step 4.1: Design a processing unit-level clock gating system, configuring an independent idle counter for each processing unit (PE). At the rising edge of the system clock, if the input data is valid... With calculation enable signal If the result of the logical AND operation is low, then... Increment by 1, otherwise clear to zero; The count value exceeds the preset threshold At this time, the power management module generates a low-level active clock gating signal. The clock input of the PE is cut off by a gating unit inserted through the clock tree, where a preset threshold is set. Represents the number of consecutive idle clock cycles; clock supply is automatically restored when a valid compute request is detected.

[0045] Step 4.2: Design a global performance-power regulator to periodically collect the processing time of each pipeline segment. and the corresponding idle counter value When judging a certain segment The value is higher than the threshold for M consecutive monitoring periods and When this occurs, mark the segment as a performance redundancy segment; among which The preset scaling factor has a value of 0.6 to 0.8; the global performance-power consumption regulator can execute any of the following optimization strategies:

[0046] Strategy A: Send a backpressure signal to the upstream data flow controller of this segment to reduce the data injection rate and decrease the activation frequency of the PE in this segment;

[0047] Strategy B: After meeting the adjusted processing time... Constraint on the current maximum time segment Below, the operating voltage or clock frequency of the PE segment is reduced through the Dynamic Voltage Frequency Scaling (DVFS) interface, so that... Appropriately increase and compress the idle time window;

[0048] Step 4.3: Design a collaborative mechanism for two-level power management based on processing unit-level clock gating and a global performance-power regulator, working collaboratively in the following manner:

[0049] (1) Hierarchical positioning: PE-level clock gating serves as the first level of fine-grained power consumption optimization, enabling rapid response to tiny, instantaneous idle cycles; the global regulator serves as the second level of coarse-grained optimization, actively adjusting for continuous pipeline segment-level load imbalances.

[0050] (2) Coordination rule: When the global regulator executes strategy B for a certain pipeline segment, the operating frequency of all PEs in that pipeline segment will change. In order to avoid the idle counter being falsely triggered or the control loop oscillating due to the frequency change, the global regulator sends a coordination control signal to the power management module of all PEs in that segment before executing the DVFS operation, temporarily disabling the clock gating function of the idle counter; after the DVFS adjustment is completed and the system enters a steady state, the normal judgment function of the idle counter is restored; when the global regulator executes strategy A, the idle period of the relevant PEs may increase due to the reduced data injection rate. At this time, the PE-level clock gating will respond automatically without additional intervention.

[0051] Furthermore, step 5 involves signal integration and timing closure of the resource mapping logic, row buffer data flow controller, static weighted storage array, deep pipeline interconnect structure, and dynamic power management unit to generate an FPGA neural network accelerator, as detailed below:

[0052] The resource mapping logic, row buffered dataflow controller, static weighted storage array, deep pipelined interconnect structure, and dynamic power management unit are integrated and time-paced at the hardware level to generate a synthesizable and place-routeable neural network inference accelerator hardware circuit. This circuit is implemented as a parameterized configurable IP core, with the top-level module exposing a structured configuration interface to receive configuration vectors. ,in This represents the total number of layers in the neural network. Configure an array for the number of PEs in each layer. Configure an array for the total capacity of on-chip BRAM at each layer, in bits; Configure an array for the inter-layer FIFO depth, in units of data blocks;

[0053] IP core integration phase based on Automatically instantiate a specified number of PE clusters, by The BRAM primitives are partitioned and instantiated to form a memory array, according to... Configure the asynchronous FIFO depth parameters between each layer, and generate data flow scheduling logic, address generation circuits, and global interconnection network that strictly match the configuration.

[0054] Compared with the prior art, the present invention has the following significant advantages: (1) It adopts a resource mapping method based on layer features, dynamically configures the cluster size and storage capacity of processing units for different layers’ computing load and data access characteristics, and achieves accurate matching of computing and storage resources; (2) It adopts a dual-buffered row data flow controller, which maximizes data reuse rate and reduces storage access frequency by accurately planning data transmission path, residence time and reuse sequence; (3) It adopts a weight static loading mechanism, which loads all weight parameters at once during the initialization phase, avoids dynamic weight loading during inference, reduces latency and improves energy efficiency; (4) It constructs a deep pipeline, which connects the cluster of processing units through hard-wired connections to achieve data pass-through between layers, reduces data transmission overhead and improves data throughput; (5) It adopts a dynamic power management mechanism, which dynamically cuts off the clock supply according to the idle state of the processing unit, achieves fine power control and improves energy efficiency ratio. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating an FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to the present invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 1 As shown, the present invention provides a method for implementing an FPGA neural network dataflow architecture based on computation-storage co-optimization, comprising the following steps:

[0058] Step 1: Construct resource mapping logic, analyze the parameters and structure of the target neural network, and independently configure the processing unit cluster size and on-chip block memory resource capacity for each layer, as follows:

[0059] Step 1.1: Construct resource mapping logic and analyze the topological information of each layer of the target neural network, including convolution kernel size, input / output feature map dimension, stride, padding method, and computation graph connectivity.

[0060] Step 1.2: Based on the computing load and data access characteristics of each layer, configure the processing unit cluster size independently for each layer, as follows:

[0061] For neural networks Layer, calculate its theoretical computational workload Combined with target reasoning delay constraints The unit is seconds, determining the number of processing units required. The constraints are satisfied:

[0062]

[0063] in, η is the system operating clock frequency; η is the effective computational efficiency factor of a single processing unit under typical workload, with a value of η = 0.7~0.95; This represents the maximum allowable computational latency for this layer.

[0064] Step 1.3: Based on the computational load and data access characteristics of each layer, independently configure the on-chip block memory resource capacity for each layer, as follows:

[0065] Step 1.3.1, Memory area partitioning, during the FPGA placement phase... Divided into a dedicated storage area for weights With dynamic data buffer pool Satisfying capacity constraints ;

[0066] Step 1.3.2, weight storage area configuration, will The capacity is set to Its address decoding logic, read / write control signals and timing are exclusively managed by the static weight loading module to persistently store all weight parameters of this layer;

[0067] Step 1.3.3, Dynamic Buffer Pool Management: This involves managing the dynamic data buffer pool. The capacity is set to The data flow controller dynamically divides the data into an input line buffer and a partial sum line buffer. The capacity allocation ratio of the two buffers is determined by the data reuse mode based on the current layer convolution stride, padding mode, and feature map size. The division boundary is adjusted in real time through the configuration register.

[0068] Step 1.3.4, the first On-chip storage capacity Constraints must be met:

[0069]

[0070] in, The minimum storage capacity required to store all weight parameters of this layer. This specifies the storage capacity required to cache a single row or block of input feature map or intermediate portion and data. This capacity allocation lays the foundation for subsequent double buffering mechanisms and static weight storage.

[0071] Step 2: Construct a row-buffered dataflow controller. This controller plans the transmission path, dwell time, and multiplexing sequence of feature maps and intermediate data between the on-chip buffer and processing unit. A static weight storage array is then constructed to load all weight parameters of the neural network into the pre-allocated on-chip static storage area at once, as detailed below:

[0072] Step 2.1: Construct a row-buffered data flow controller with a double-buffering mechanism to accurately plan the transmission path, dwell time, and multiplexing sequence of the input feature map data and intermediate data between the on-chip buffer unit and the processing unit, as detailed below:

[0073] Step 2.1.1: Define data reuse rate evaluation indicators The calculation formula is:

[0074]

[0075] in This represents the total number of multiplication and addition operations in a single reasoning operation. The total number of times the input feature map and intermediate parts are read from the on-chip buffer is high. Value means more efficient data reuse;

[0076] Step 2.1.2: Construct a row-buffered data flow controller with a dual-buffering mechanism. Accurately plan the transmission path, dwell time, and multiplexing timing of input feature map data and intermediate data between the on-chip buffer unit and the processing unit. Buffer the input feature map into the input row buffer unit according to preset rows / blocks, and buffer the convolution intermediate data into the partial and row buffer units. Generate a data supply timing control table based on the current layer's convolution kernel size, stride, and padding parameters. Dynamically adjust the start trigger time, duration of validity, and buffer switching cycle of the input row buffer unit and the partial and row buffer units outputting data to the PE cluster to achieve precise timing scheduling.

[0077] Step 2.2: Construct a static weight storage array. During the hardware initialization phase after FPGA configuration, all weight parameters of the neural network are loaded into the pre-allocated on-chip static storage area at once, as follows:

[0078] Step 2.2.1: During the initialization window after FPGA configuration is complete, transfer the weight parameter set via PCIe DMA or the configuration memory interface. Write multiple on-chip BRAM memory banks in a hierarchical order to construct a static weighted memory array;

[0079] Step 2.2.2: During the inference phase, the data flow controller generates an address sequence in real time. The computing unit is arranged according to Read the weight values, where For BRAM index, This represents the word offset address within the memory.

[0080] This invention employs static weight loading to reduce inference power consumption, but to support multi-model switching, it reserves some reconfigurability. Specifically, the static weight storage area is divided into multiple memory banks, each corresponding to the weights of a model. When a model switch is required, simply updating the weight base address register through the configuration interface to point to the weight storage bank of the new model allows the model switch to be completed within a few clock cycles without reconfiguring the FPGA bitstream.

[0081] Step 3: Construct a deep pipeline interconnect structure, hard-wire the processing unit clusters of each layer in the forward propagation order, so that the output of the upper layer is directly used as the input of the lower layer after being buffered between layers, forming an end-to-end interlayer direct data path, as follows:

[0082] Step 3.1: Set the number of pipeline segments Equal to the number of effective computational layers in the neural network, the th The segment corresponds to the first The single-frame data processing time of a layer is denoted as By adjusting the number of processing units in each segment And internal parallelism, so that the processing time of all pipeline stages meets the balance constraint:

[0083]

[0084] in , , Based on the system clock cycle The set allowable time deviation threshold ranges from 1 to 3 times. After the pipeline enters steady state, the interval between adjacent output frames in steady state is... equal ;

[0085] Step 3.2: Adjacent layer processing unit clusters connect data via an asynchronous FIFO. This FIFO has an independent read / write clock domain, configurable depth, and full / empty status flags; connecting the... Layer and First FIFO depth of the layer Must meet ,in For the first The maximum time, in seconds, for a layer to write a complete feature map data block to the FIFO in continuous working mode; The system clock cycle is in seconds; this depth configuration is used to prevent FIFO overflow or underflow during burst writes or consumption delays, maintaining pipeline data continuity.

[0086] Step 1's resource mapping provides a lower limit on the number of processing units required to meet the latency requirements of each layer, but the final pipeline processing time... Factors such as inter-layer data transfer and buffering also need to be considered. When constructing a deep pipeline, if the processing time of the m-th layer... The processing time of the m-th layer is significantly larger than that of other layers and can be adjusted in two ways: 1) If there are sufficient FPGA resources, add additional processing units to the m-th layer that exceed the lower limit calculated in step 1 to shorten its processing time; 2) If resources are exhausted, accept... As a bottleneck in the pipeline, and to ensure that the interlayer FIFO depth is sufficient to absorb back pressure caused by velocity differences, the core of this invention lies in providing a configurable architecture that allows the above adjustments to be made by modifying the configuration vector. The NPE parameter can be quickly implemented.

[0087] Step 4: Construct a dynamic power management unit to continuously sample the combination state of the valid input data signal and the computation enable signal of each processing unit in the inter-layer direct data path, and dynamically cut off the local clock supply of the processing unit that continuously meets the idle determination condition, as follows:

[0088] Step 4.1: Design a processing unit-level clock gating system, configuring an independent idle counter for each processing unit (PE). At the rising edge of the system clock, if the input data is valid... With calculation enable signal If the result of the logical AND operation is low, then... Increment by 1, otherwise clear to zero; The count value exceeds the preset threshold At this time, the power management module generates a low-level active clock gating signal. The clock input of the PE is cut off by a gating unit inserted through the clock tree, where a preset threshold is set. Represents the number of consecutive idle clock cycles; clock supply is automatically restored when a valid compute request is detected.

[0089] The preset threshold The following method was used to determine the idle cycle distribution of each processing unit cluster under typical workloads: First, in a system-level simulation environment, multi-frame inference of the target neural network was run to statistically analyze the idle cycle distribution of each processing unit cluster under typical workloads; then, using total power consumption as the objective function, different... Choose the value that minimizes the total power consumption. Value. For the ResNet-50 model, experiments show... It can achieve the best energy efficiency ratio;

[0090] Step 4.2: Design a global performance-power regulator to periodically collect the processing time of each pipeline segment. and the corresponding idle counter value When judging a certain segment The value is higher than the threshold for M consecutive monitoring periods and When this occurs, mark the segment as a performance redundancy segment; among which The preset scaling factor has a value of 0.6 to 0.8; the global performance-power consumption regulator can execute any of the following optimization strategies:

[0091] Strategy A: Send a backpressure signal to the upstream data flow controller of this segment to reduce the data injection rate and decrease the activation frequency of the PE in this segment;

[0092] Strategy B: After meeting the adjusted processing time... Constraint on the current maximum time segment Below, the operating voltage or clock frequency of the PE segment is reduced through the Dynamic Voltage Frequency Scaling (DVFS) interface, so that... Appropriately increase and compress the idle time window;

[0093] Step 4.3: Design a collaborative mechanism for two-level power management based on processing unit-level clock gating and a global performance-power regulator, working collaboratively in the following manner:

[0094] (1) Hierarchical positioning: PE-level clock gating serves as the first level of fine-grained power consumption optimization, enabling rapid response to tiny, instantaneous idle cycles; the global regulator serves as the second level of coarse-grained optimization, actively adjusting for continuous pipeline segment-level load imbalances.

[0095] (2) Coordination rule: When the global regulator executes strategy B for a certain pipeline segment, the operating frequency of all PEs in that pipeline segment will change. In order to avoid the idle counter being falsely triggered or the control loop oscillating due to the frequency change, the global regulator sends a coordination control signal to the power management module of all PEs in that segment before executing the DVFS operation, temporarily disabling the clock gating function of the idle counter; after the DVFS adjustment is completed and the system enters a steady state, the normal judgment function of the idle counter is restored; when the global regulator executes strategy A, the idle period of the relevant PEs may increase due to the reduced data injection rate. At this time, the PE-level clock gating will respond automatically without additional intervention.

[0096] Through the aforementioned hierarchical collaborative control architecture, this invention achieves comprehensive power consumption optimization from fine-grained to coarse-grained levels while maintaining stable pipeline operation.

[0097] Step 5: Integrate the resource mapping logic, row buffer data flow controller, static weighted storage array, deep pipeline interconnect structure, and dynamic power management unit to generate an FPGA neural network accelerator, as detailed below:

[0098] The resource mapping logic, row buffered dataflow controller, static weighted storage array, deep pipelined interconnect structure, and dynamic power management unit are integrated and time-paced at the hardware level to generate a synthesizable and place-routeable neural network inference accelerator hardware circuit. This circuit is implemented as a parameterized configurable IP core, with its top-level module exposing a structured configuration interface to receive configuration vectors. ,in This represents the total number of layers in the neural network. Configure an array for the number of PEs in each layer. Configure an array for the total capacity of on-chip BRAM at each layer (in bits). Configure an array for the inter-layer FIFO depth, in units of data blocks;

[0099] IP core integration phase based on Automatically instantiate a specified number of PE clusters, by The BRAM primitives are partitioned and instantiated to form a memory array, according to... Configure the asynchronous FIFO depth parameters between each layer, and generate data flow scheduling logic, address generation circuits, and global interconnection network that strictly match the configuration.

[0100] Example

[0101] To verify the effectiveness of the FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization proposed in this invention, practical tests were conducted on two mainstream FPGA platforms: Xilinx VCK190 (Versal ACAP) and ZCU102 (UltraScale+), and a horizontal comparison was made with NVIDIA V100 GPU, Intel i9 CPU, and the existing FPGA accelerator AutoViTAcc. The experiments used INT8 quantization precision, batch size = 1, and clock frequencies as shown in Table 1. The technical advantages of this invention are presented below from four dimensions: performance, energy efficiency, resource utilization, and design efficiency. Experimental results are shown in Tables 1 and 2:

[0102] Table 1 Comparison of inference performance and energy efficiency on different hardware platforms

[0103]

[0104] Table 2 Performance Comparison of Different Methods

[0105]

[0106] Low latency and high throughput: Deploying ResNet-50 on the VCK190 platform, this invention achieves an end-to-end latency of 1.2 ms, corresponding to a throughput of 833 FPS. Compared to the NVIDIA V100 GPU's 3.4 ms and 294 FPS, latency is reduced by 65%, and throughput is increased by 2.83 times; compared to the Intel i9 CPU's 18.5 ms and 54 FPS, throughput is increased by 15.4 times. Compared to the ZCU102 baseline design on the same platform (5.8 ms and 172 FPS), latency is compressed by 45%, and throughput is increased by 1.8 times. This advantage stems from precise resource mapping and row-buffered data flow control: based on... Configure the number of PEs at each layer to eliminate computational bottlenecks; improve data reuse rate through a double buffering mechanism. Increased to 92%, significantly reducing off-chip memory access.

[0107] This invention achieves 21.9 FPS / W on the VCK190, a 18.6-fold improvement over the V100 GPU's 1.18 FPS / W and a 2.3-fold improvement over the ZCU102 baseline's 9.56 FPS / W. The energy efficiency advantage primarily stems from static weight storage and dynamic power management. Weight fixation eliminates dynamic loading power consumption during inference; local clock gating cuts off the clock after the PE has been idle for more than 8 consecutive cycles, reducing dynamic power consumption by 28%–42%; global linkage control further improves energy efficiency by 23%–31% through identifying performance redundancy segments and sending reverse voltage or DVFS adjustment.

[0108] By precisely mapping resources and independently configuring the number of PEs and BRAM capacity for each layer, a DSP utilization rate of 86% was achieved when deploying ResNet-50 on the ZCU102, far exceeding the 57% of the traditional uniform allocation scheme. BRAM usage was only 19.4 Mb, a 21% reduction compared to the traditional 24.5 Mb, demonstrating that dynamic buffer pool management effectively avoids storage idleness. On the VCK190 platform, due to the larger total resource volume, the DSP utilization rate was 84%, and the BRAM usage was 22.1 Mb, still maintaining high efficiency. For the ViT-B model, this invention achieved a 79% DSP utilization rate and a BRAM usage of 23.8 Mb. Compared to AutoViTAcc's 76% DSP utilization rate and 22.0 Mb BRAM, the DSP utilization rate is higher and the BRAM usage is slightly higher, but the energy efficiency advantage is significant.

[0109] The parameterized IP core automatically generates hardware based on the configuration vector C = [L, N^PE, M^BRAM, D] during the synthesis stage. Its resource prediction error is less than 5%, which is consistent with the conclusion of R²>0.94 of the recent high-precision prediction model. It can accurately predict resource usage before synthesis and avoid repeated iterations.

[0110] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for implementing an FPGA neural network dataflow architecture based on computation-storage co-optimization, characterized in that, Includes the following steps: Step 1: Construct resource mapping logic, analyze the parameters and structure of the target neural network, and independently configure the cluster size of the processing unit and the on-chip block memory resource capacity for each layer. Step 2: Construct a row buffer data flow controller. Through the row buffer data flow controller, plan the transmission path, dwell time and multiplexing sequence of feature maps and intermediate data between the on-chip buffer and processing unit. Construct a static weight storage array to load all weight parameters of the neural network into the pre-allocated on-chip static storage area at once. Step 3: Construct a deep pipeline interconnection structure, hard-wire the processing unit clusters of each layer in the forward propagation order, so that the output of the upper layer is directly used as the input of the lower layer after the inter-layer buffer, forming an end-to-end inter-layer direct data path. Step 4: Construct a dynamic power management unit to continuously sample the combination state of the valid input data signal and the computing enable signal of each processing unit in the inter-layer direct data path, and dynamically cut off the local clock supply for processing units that continuously meet the idle judgment condition. Step 5: Integrate the resource mapping logic, row buffer data flow controller, static weight storage array, deep pipeline interconnect structure and dynamic power management unit to generate an FPGA neural network accelerator.

2. The FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to claim 1, characterized in that, The resource mapping logic described in step 1 involves parsing the parameters and structure of the target neural network, and configuring the processing unit cluster size and on-chip block memory resource capacity for each layer independently, as detailed below: Step 1.1: Construct resource mapping logic and analyze the topological information of each layer of the target neural network, including convolution kernel size, input / output feature map dimension, stride, padding method, and computation graph connectivity. Step 1.2: Based on the computational load and data access characteristics of each layer of the target neural network, configure the processing unit cluster size independently for each layer; Step 1.3: Based on the computational load and data access characteristics of each layer of the target neural network, configure the on-chip block memory resource capacity independently for each layer.

3. The FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to claim 2, characterized in that, Step 1.2 describes configuring the processing unit cluster size independently for each layer of the target neural network based on the computational load and data access characteristics of each layer, as follows: For neural networks Layer, calculate its theoretical computational workload Combined with target reasoning delay constraints Determine the number of processing units required. The constraints are satisfied: in, η is the system operating clock frequency; η is the effective computational efficiency factor of a single processing unit under typical workload, with a value of η = 0.7~0.95; This represents the maximum allowable computational latency for this layer.

4. The FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to claim 2, characterized in that, Step 1.3 describes the allocation of on-chip block memory resource capacity for each layer of the target neural network based on its computational load and data access characteristics, as follows: Step 1.3.1, Memory area partitioning, during the FPGA placement phase... Divided into a dedicated storage area for weights With dynamic data buffer pool Satisfying capacity constraints ; Step 1.3.2, weight storage area configuration, will The capacity is set to Its address decoding logic, read / write control signals, and timing are exclusively managed by the static weight loading module for persistent storage. All weight parameters of the layer; Step 1.3.3, Dynamic Buffer Pool Management: This involves managing the dynamic data buffer pool. The capacity is set to The line buffer is dynamically divided into an input line buffer and a partial sum line buffer by the line buffer controller; Step 1.3.4, the first On-chip storage resource capacity Constraints must be met: in, The minimum storage capacity required to store all weight parameters of this layer. The storage capacity required to cache a single row or a single data block of input feature map or intermediate portion and data.

5. The FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to claim 1, characterized in that, Step 2 describes the construction of a row buffer data flow controller. This controller plans the transmission path, dwell time, and multiplexing sequence of feature maps and intermediate data between the on-chip buffer and processing unit. A static weight storage array is then constructed to load all weight parameters of the neural network into the pre-allocated on-chip static storage area at once, as detailed below: Step 2.1: Construct a row buffered data flow controller with a double buffering mechanism, and plan the transmission path, dwell time and multiplexing sequence of the input feature map data and the intermediate part and data between the on-chip buffer unit and the processing unit; Step 2.2: Construct a static weight storage array. During the hardware initialization phase after the FPGA configuration is completed, load all weight parameters of the neural network into the pre-allocated on-chip static storage area at once.

6. The FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to claim 5, characterized in that, Step 2.1 describes the construction of a row-buffered data flow controller with a double-buffering mechanism, planning the transmission path, dwell time, and multiplexing sequence of the input feature map data and intermediate data between the on-chip buffer unit and the processing unit, as follows: Step 2.1.1: Define data reuse rate evaluation indicators The calculation formula is: in This represents the total number of multiplication and addition operations in a single reasoning operation. This is the total number of times the input feature map is read from the on-chip buffer and the intermediate portion is summed. Step 2.1.2: Construct a row-buffered data flow controller with a double-buffering mechanism, plan the transmission path, dwell time and multiplexing sequence of input feature map data and intermediate data between the on-chip buffer unit and the processing unit, cache the input feature map to the input row buffer unit according to the preset row / block, and cache the convolution intermediate data to the partial data row buffer unit.

7. The FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to claim 5, characterized in that, Step 2.2 describes the construction of a static weight storage array. During the hardware initialization phase after FPGA configuration, all weight parameters of the neural network are loaded into the pre-allocated on-chip static storage area at once, as follows: Step 2.2.1: During the initialization window after FPGA configuration is complete, transfer the weight parameter set via PCIe DMA or the configuration memory interface. Write multiple on-chip BRAM memory banks in a hierarchical order to construct a static weighted memory array; Step 2.2.2: During the inference phase, the data flow controller generates the address sequence in real time. The computing unit is arranged according to Read the weight values, where For BRAM index, This represents the word offset address within the memory.

8. The FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to claim 1, characterized in that, Step 3 describes constructing a deep pipeline interconnect structure by hard-wired the processing unit clusters of each layer in the forward propagation order. This allows the upper-layer output to be directly used as the lower-layer input after being buffered between layers, forming an end-to-end interlayer data path. Specifically, the following is a detailed description: Step 3.1: Set the number of pipeline segments Equal to the number of effective computational layers in the neural network, the th The segment corresponds to the first The single-frame data processing time of a layer is denoted as By adjusting the number of processing units in each segment And internal parallelism, so that the processing time of all pipeline stages meets the balance constraint: in , , Based on the system clock cycle The set allowable time deviation threshold ranges from 1 to 3 times. After the pipeline enters steady state, the interval between adjacent output frames in steady state is... equal ; Step 3.2: Adjacent layer processing unit clusters connect data via an asynchronous FIFO. This FIFO has an independent read / write clock domain, configurable depth, and full / empty status flags; connecting the... Layer and First FIFO depth of the layer Must meet ,in For the first The maximum time, in seconds, for a layer to write a complete feature map data block to the FIFO in continuous working mode; The system clock cycle is expressed in seconds. Deep configuration is used to prevent FIFO overflow or underflow during burst writes or consumption delays, maintaining pipeline data continuity.

9. The FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to claim 1, characterized in that, Step 4 describes the construction of a dynamic power management unit that continuously samples the combination state of the valid input data signal and the computation enable signal of each processing unit in the inter-layer direct data path. It then dynamically cuts off the local clock supply to processing units that continuously meet the idle condition, as detailed below: Step 4.1: Design a processing unit-level clock gating system, configuring an independent idle counter for each processing unit (PE). At the rising edge of the system clock, if the input data is valid... With calculation enable signal If the result of the logical AND operation is low, then... Increment by 1, otherwise clear to zero; The count value exceeds the preset threshold At this time, the power management module generates a low-level active clock gating signal. The clock input of the PE is cut off by a gating unit inserted through the clock tree, where a preset threshold is set. Represents the number of consecutive idle clock cycles; clock supply is automatically restored when a valid compute request is detected. Step 4.2: Design a global performance-power regulator to periodically collect the processing time of each pipeline segment. and the corresponding idle counter value When judging a certain segment The value is higher than the threshold for M consecutive monitoring periods and When this occurs, mark the segment as a performance redundancy segment; among which The preset scaling factor has a value of 0.6 to 0.8; the global performance-power consumption regulator can execute any of the following optimization strategies: Strategy A: Send a backpressure signal to the upstream data flow controller of this segment to reduce the data injection rate and decrease the activation frequency of the PE in this segment; Strategy B: After meeting the adjusted processing time... Constraint on the current maximum time interval Below, the operating voltage or clock frequency of the PE segment is reduced through the Dynamic Voltage Frequency Scaling (DVFS) interface, so that... Appropriately increase and compress the idle time window; Step 4.3: Design a collaborative mechanism for two-level power management based on processing unit-level clock gating and a global performance-power regulator, working collaboratively in the following manner: (1) Hierarchical positioning: PE-level clock gating serves as the first level of fine-grained power consumption optimization, enabling rapid response to tiny, instantaneous idle cycles; the global regulator serves as the second level of coarse-grained optimization, actively adjusting for continuous pipeline segment-level load imbalances. (2) Coordination rule: When the global regulator executes strategy B for a certain pipeline segment, the operating frequency of all PEs in that pipeline segment will change. In order to avoid the idle counter being falsely triggered or the control loop oscillating due to the frequency change, the global regulator sends a coordination control signal to the power management module of all PEs in that segment before performing the DVFS operation, temporarily disabling the clock gating function of the idle counter; after the DVFS adjustment is completed and the system enters a steady state, the normal judgment function of the idle counter is restored. When the global regulator executes strategy A, the idle period of the relevant PE may increase due to the reduced data injection rate. At this time, the PE-level clock gating will respond automatically without additional intervention.

10. The FPGA neural network dataflow architecture implementation method based on computation-storage co-optimization according to claim 1, characterized in that, Step 5 involves signal integration and timing closure of the resource mapping logic, row buffer data flow controller, static weight storage array, deep pipeline interconnect structure, and dynamic power management unit to generate an FPGA neural network accelerator, as detailed below: The resource mapping logic, row buffered dataflow controller, static weighted storage array, deep pipelined interconnect structure, and dynamic power management unit are integrated and time-paced at the hardware level to generate a synthesizable and place-routeable neural network inference accelerator hardware circuit. This circuit is implemented as a parameterized configurable IP core, with the top-level module exposing a structured configuration interface to receive configuration vectors. ,in This represents the total number of layers in the neural network. Configure an array for the number of PEs in each layer. Configure an array for the total capacity of on-chip BRAM at each layer, in bits; Configure an array for the inter-layer FIFO depth, in units of data blocks; IP core integration phase based on Automatically instantiate a specified number of PE clusters, by The BRAM primitives are partitioned and instantiated to form a memory array, according to... Configure the asynchronous FIFO depth parameters between each layer, and generate data flow scheduling logic, address generation circuits, and global interconnection network that strictly match the configuration.

Images