Field programmable gate array (FPGA)-based row-column double-flow water type universal pooling implementation method
By adopting a row-column dual-pipeline architecture design, the problems of insufficient versatility and computational speed of FPGA pooling implementation methods are solved, enabling flexible support for commonly used CNN algorithms and efficient data processing, thereby improving resource utilization.
Patent Information
- Application Number
- CN202511860575.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-10
- Publication Date
- 2026-03-20
AI Technical Summary
Existing FPGA-based pooling implementations suffer from poor versatility, insufficient data throughput and computation speed, and weak module coordination, making it difficult to meet the real-time processing requirements of high-resolution images and complex CNN models.
It adopts a row-column dual-pipeline architecture design, combining a pooling layer control center module, a row-column pipelined data scheduling module, a row-column dual-pipeline buffer output module, and a pooling operator control module to achieve full coverage of the pooling requirements of common CNN algorithms, thereby improving data throughput and computing speed.
It achieves flexible support for commonly used CNN algorithms such as ResNet and YOLO, improving data throughput and computation speed, and increasing resource utilization.
Smart Images
Figure CN121706856A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of convolutional neural network technology, specifically relating to a generalized pooling implementation method based on FPGA with row and column dual pipelines. It is applicable to scenarios that require high-speed, high-throughput computation of pooling layers in CNNs, such as artificial intelligence applications like image recognition, object detection, and speech processing. Background Technology
[0002] In the inference and training process of convolutional neural networks, pooling layers are a key component, mainly playing the roles of reducing feature map dimensionality, reducing computational cost, preserving key features, and preventing overfitting. Currently, commonly used pooling operations include max pooling, average pooling, and summation pooling, and different CNN algorithms (such as ResNet, YOLO, VGG, etc.) have different parameter requirements for pooling layers, such as kernel size (2×2, 3×3, etc.), stride (1, 2, etc.), and padding method (VALID, SAME), which may all be different.
[0003] Existing FPGA-based pooling implementations still suffer from the following drawbacks: Poor versatility: Most solutions are designed only for specific types of pooling operations (e.g., supporting only max pooling) or specific pooling parameters (e.g., a fixed 2×2 pooling kernel with a stride of 2). When adapting to the pooling requirements of different CNN algorithms, hardware logic redesign and verification are necessary, resulting in long development cycles and low flexibility. Insufficient data throughput and computation speed: Existing solutions often employ single-pipelined or row-column separated non-pipelined architectures for data processing and computation. This leads to latency during data transmission and low utilization of computational units, making it difficult to fully leverage the parallel computing advantages of FPGAs. Consequently, the overall data throughput and pooling computation speed cannot meet the real-time processing requirements of high-resolution images and complex CNN models. Weak module coordination: In some solutions, the interface designs of various functional modules (e.g., data scheduling module, computation module, cache module) are inconsistent, resulting in complex data interaction logic. Therefore, a generalized pooling implementation method that can solve the above problems and achieve high speed, covers the computational requirements of commonly used CNN algorithms, and provides high data throughput is urgently needed. Summary of the Invention
[0004] Based on the above analysis, this invention aims to disclose a generalized pooling implementation method based on FPGA with dual row and column pipelines. The purpose is to overcome the shortcomings of existing FPGA-based pooling implementation methods, such as poor versatility, insufficient data throughput and computing speed, and weak module coordination. This invention provides a generalized pooling implementation method based on FPGA with dual row and column pipelines. By constructing an efficient module architecture and dual pipeline data processing logic, it achieves full coverage of the pooling requirements of commonly used CNN algorithms, while improving data throughput and computing speed to meet the needs of high real-time application scenarios.
[0005] This invention discloses a generalized pooling implementation method based on FPGA with row and column dual pipelines, including a pooling layer control center module, a row and column pipeline data scheduling module, a row and column dual pipeline buffer output module, a pooling operator control module, and a pooling calculation module;
[0006] First, the pooling layer control center module configures parameters, including feature map width W, feature map height H, feature map channel size CH1, pooling kernel size N, pooling kernel step S, pooling kernel mode M, and pooling kernel fill count P. Based on the parsed configuration parameters, this module sends corresponding control signals to the row / column pipelined data scheduling module, the pooling operator control module, and the row / column dual-pipeline buffer output module, enabling collaborative operation among these modules. For example, based on the pooling kernel size and step size, it sends timing control signals for data reading and scheduling to the row / column pipelined data scheduling module; based on the pooling type, it sends operator selection signals to the pooling operator control module; and based on the number of channels in the input feature map and output requirements, it sends buffer read / write and data output control signals to the row / column dual-pipeline buffer output module. In addition, the module also has a status monitoring function, which collects the working status of each module in real time (such as data ready status, calculation completed status, cache full / empty status). When an abnormal status occurs (such as data transmission error, cache overflow), it promptly sends an alarm signal to the external system and triggers the corresponding abnormal handling logic (such as pausing data transmission, resetting the module).
[0007] Furthermore, the pooling layer control center module sends a configuration command to the row and column pipeline data scheduling module and receives its configuration completion signal, thus completing the configuration of the row and column pipeline data scheduling module;
[0008] Furthermore, after receiving the configuration command, the row and column data scheduling module has a maximum row data write address of W*CH1. The data and valid signals are written into the BRAM cores sequentially according to the channel order. When the current row data write address reaches the maximum, the BRAM selection register is incremented by one, and the IP core being written is moved. When the Nth BRAM is full, the BRAM selection register is reset to 1, and the data in the first BRAM is overwritten, thus completing the step-by-step storage of the row buffer module.
[0009] Furthermore, when the data is cached in the NP-th BRAM and the row data write address reaches (NP)*CH1, after the starting condition for the pooling kernel to start calculation is met, the BRAM read register is pulled high, and the row and column pipelined data scheduling module starts to generate the read address and read valid signal.
[0010] Furthermore, when generating the pooling kernel window, the first coordinate (addr1) of the non-filled data in the first row of the pooling kernel window is determined first. The output addresses of the filled data and other data in this row are then determined. The address of the data to the left of the first coordinate data position is addr1-n*CH1, and the address to the right of the first coordinate data position is addrl+n*CHl. The specific amount of movement depends on the pooling window. The data to be filled in the row direction does not output a valid data address or a valid enable signal, thus completing the data filling in the row direction.
[0011] Furthermore, there are N data output addresses and N corresponding valid read signals. When data output begins, only NP BRAMs contain data, so the row number needs to be determined. When data filling is required, no data address or valid enable signal is sent to the BRAM, thus completing the data filling in the column direction.
[0012] Furthermore, the pooling layer control center module has N valid addresses and N valid enable signals. The addresses and valid signals are output as N data in the column direction, and this output continues for the next N cycles. Within the N cycles, an N*N pooling kernel window is output.
[0013] Secondly, the row-column dual-pipeline buffer module receives the data write address, data write valid signal, data read address, and data read valid signal from the pooling control center module. In one time cycle, only one BRAM is writing data. Only after the BRAM finishes writing a row of data does the next BRAM begin writing, completing the row-direction buffer module function. After data writing reaches the condition that the read start signal goes high, N BRAMs simultaneously receive the data read address and data read valid signal, outputting N data points, corresponding to the column-direction data in the pooling core window, completing the column-direction concatenation module function. Throughout the scheduling process, the row-direction and column-direction pipeline processing logics are executed in parallel. That is, while the row-direction scheduling unit processes the next row of data, the column-direction scheduling unit processes the current row of data, achieving "wait-free" pipelined data transmission and significantly improving data scheduling efficiency.
[0014] Then, the pooling operator control module receives the pooling kernel window size N*N and pooling mode M parameters from the pooling control center module, configures the pooling calculation module, and assigns the data receiving part to the pooling kernel window N. The pooling operator control module pre-integrates the pooling operator logic required by commonly used CNN algorithms, including max pooling operator, average pooling operator, summation pooling operator, etc. When the pooling layer control center module determines that the current pooling type is max pooling, the pooling operator control module sends a "max pooling enable" signal to the pooling calculation module and configures the comparison logic unit (such as selector and comparator) inside the calculation module to perform calculations according to the max pooling rule (i.e., selecting the maximum value from the pooling window data as the pooling result). When the pooling type is average pooling, an "average pooling enable" signal is sent, and the adder and divider (or shifter, used to implement fast division operations) inside the calculation module are configured to sum the pooling window data and divide by the window size to obtain the average pooling result. When the pooling type is summation pooling, a "summation pooling enable" signal is sent, and the adder logic is configured to perform summation operations on the pooling window data to obtain the result. Finally, the pooling calculation module receives the column data kernel validity signal from the row and column dual pipelined buffer output module, inputs the column data and validity signal into the pooling operator, and obtains the pooling result of N data. In the following N cycles, a total of N columns of pooling results are obtained. These N pooling results are rearranged and input into the pooling operator to obtain the pooling result of the N*N pooling window.
[0015] Compared with the prior art, the present invention has the following beneficial effects:
[0016] High versatility: Through the flexible configuration of the pooling layer control center module and the dynamic operator selection of the pooling operator control module, it can support common pooling types such as max pooling, average pooling, and summation pooling, as well as pooling requirements with different pooling kernel sizes (2×2 to 4×4), strides (1 to 3), and padding. It can cover the pooling computation requirements of common CNN algorithms such as ResNet, YOLO, and VGG without redesigning hardware logic, offering high flexibility and a short development cycle.
[0017] High data throughput and computing speed: It adopts a row-column dual pipeline architecture design, in which the data processing logic of row pipeline and column pipeline is executed in parallel. Combined with the cache synchronization mechanism of the row-column dual pipeline cache output module, it realizes "wait-free" data transmission. At the same time, the pooling computing module adopts a multi-channel parallel computing unit, giving full play to the parallel computing advantages of FPGA and significantly improving data throughput and computing speed.
[0018] High resource utilization: Through data format conversion, precision control, and dynamic adaptation of parallel computing units, the hardware resource consumption of FPGA is effectively reduced and the resource utilization is improved while meeting the requirements of computing accuracy and performance. Attached Figure Description
[0019] Figure 1 This is a block diagram of a generalized pooling implementation method based on FPGA with row and column dual pipelines, provided by an example of the present invention.
[0020] Figure 2 This is a schematic diagram of the pooling operator provided in an embodiment of the present invention.
[0021] Figure 3 This is a schematic diagram of the pooling module process of a convolutional neural network model provided in an embodiment of the present invention.
[0022] Figure 4 This is a diagram of a row-column bidirectional pipelined buffer structure provided in an embodiment of the present invention.
[0023] Figure 5 This is a schematic diagram of the sliding window extraction and row update process provided in an embodiment of the present invention.
[0024] Figure 6 A schematic diagram illustrating the process of pooling the feature map provided in an embodiment of the present invention. Detailed Implementation
[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] In this embodiment, as Figure 3 As shown, the FPGA-based row-column dual-pipelined generalized pooling scheme is applied to the 5×5 pooling operation in the SPPf (Spatial Pyramid Pooling Fast) structure of the YOLOv5s algorithm. Let the input feature map size be width W = 20, height H = 15, and the number of channels CH. i =512, pooling kernel size N=5, step size S=1, fill P=2. This embodiment details the workflow and data processing of each module under the above parameter conditions.
[0027] First, such as Figure 1As shown, the pooling layer control center module configures the parameters of the entire pooling layer. Specifically, the pooling layer control center module sets the feature map width W=20, height H=15, number of input channels CH1=512, pooling kernel size N=5, stride S=1, padding P=2, and pooling type M (e.g., max pooling mode in this embodiment). Based on the parsed parameters, the pooling layer control center module sends corresponding control signals to other modules. For example, based on the pooling kernel size N=5 and stride S=1, it sends timing control signals for data reading and scheduling to the row-column dual-pipeline buffer module; based on the pooling type (max pooling), it sends operator selection signals to the pooling operator control module; and based on the number of input feature map channels 512 and output requirements, it sends buffer read / write and data output control signals to the row-column dual-pipeline buffer module. In addition, the pooling layer control center module monitors the status of each module in real time (such as data ready, calculation completed, cache full / empty, etc.). When an anomaly occurs (such as data transmission error, cache overflow), it sends an alarm signal and triggers the corresponding anomaly handling logic (such as pausing data transmission, resetting the module, etc.) to ensure reliable system operation.
[0028] Upon receiving the configuration command from the pooling layer control center module, the row-column dual-pipeline buffer module begins writing the input feature map data row by row into the row buffer (using BRAM storage units). For the first row (Row0) of the input feature map, since the width W = 20 and the number of channels CH1 = 512, each row contains 20 × 512 = 10240 data points. The data scheduling module writes the data to the current BRAM unit sequentially according to the channel order: the write address increments from 0 to 10239, and after writing is complete, it indicates that the current BRAM has buffered one row of data. Figure 5 As shown, when a row of data is written, the BRAM select register is incremented by 1, switching to the next BRAM to write the next row of data. Simultaneously, the writing of each row of data is accompanied by a valid write signal and a corresponding data signal, which are synchronously sent to the BRAM. The module is configured with five BRAM circular buffers (corresponding to N=5) to implement row-level multi-level buffering. Specifically, when the fifth BRAM (i.e., N=5) is filled with a row of data, the BRAM select register is reset to 1, and subsequent rows of data continue to overwrite the first BRAM, achieving circular overwrite buffering. This multi-level row buffer design supports continuous reading and writing and cyclical use of the input feature map in the vertical direction. Figure 4 As shown, during the writing process from the first to the third row, when the third BRAM is written (i.e., the write address reaches 3 × 512 = 1536, which is an example threshold used to determine that 3 rows of data have been cached), the starting condition for the pooling kernel to begin calculation is met. At this time, the row and column pipelined data scheduling module will pull the BRAM read register high and start generating the read address and read valid signal, preparing for the subsequent pooling window data output.
[0029] After the initial conditions for pooling computation are met, the row-column dual-pipeline buffer module generates the addresses and enable functions for each window position step by step according to the 5×5 pooling window rule. First, based on the address (denoted as addr1) of the first data in the non-filled portion of the first row of the current pooling window to be calculated in the input feature map, the addresses of the remaining positions in the first row of the window, as well as the output control for the filled region, are determined sequentially. The specific processing flow is as follows:
[0030] Determine the first coordinate address of the first row: Let addr1 correspond to the leftmost valid data coordinate of the first row of the current window. Since the input in this example has already been written to BRAM in the first three rows, addr1 can be read from this data.
[0031] Row direction filling: For the upper region of the pooling window that needs to be filled (since P=2, there are 2 rows of zero filling above), no valid output is generated at these positions, and no address or enable signal is sent; for the first row of actual data area, starting from addr1, the actual data position is read sequentially to the right according to the channel offset, and its address value is addr1, addr1+CH1, addr1+2·CH1, ... (here CH1=512 is the cross column offset).
[0032] Column offset: For other rows in the window, the vertical offset relative to addr1 is calculated as follows: the address of the next row is addr2 = addr1 + W·CH1 (moving down 1 row), the next row is addr1 + 2·W·CH1, and so on. For the blank fill in the upper left corner of the window in this example, no valid signal is sent to the top two rows; similarly, when the first row of the window is positioned at the right edge of the image, no valid signal is sent to the lower right fill position.
[0033] Output Address and Valid Signal Generation: The control center module outputs 5 addresses and 5 read enable signals (N=5) sequentially according to column order. Within each clock cycle, it outputs data for one column of pooled windows, completing the data output for the entire 5×5 window over 5 consecutive cycles. At the start of actual output, since only 3 BRAMs (NP=3) have valid data buffered, no valid output is generated for the corresponding row / column positions that need filling. The row / column pipeline scheduling module invalidates the corresponding address and enable signal, completing zero-filling processing.
[0034] Through the above steps, the row-column dual-pipelined buffer module can generate a complete data address sequence and valid signal for each 5×5 window, providing them to the next-level module for calculation. The row-column dual-pipelined buffer module is responsible for row and column-direction buffering and output. During the data writing phase, this module receives the write address and write valid signal from the pooling layer control center module, writing data to only one BRAM block per clock cycle. When the current BRAM is full (10240 data points), the module automatically switches to the next BRAM to write the next row of data, achieving continuous buffering in the row direction. After the write operation is completed and the read conditions are met, all five BRAM blocks simultaneously receive the read address and read enable signal, outputting the column data corresponding to each BRAM column-wise. Within one clock cycle, each BRAM outputs one data point (a total of five data points), corresponding to one column of data in the pooling window; thus, a complete 5×5 window of data can be obtained after five consecutive cycles.
[0035] Throughout the process, the data processing logic of row pipeline and column pipeline is executed in parallel: while the row pipeline scheduling unit continues to process the next row of data writing, the column pipeline scheduling unit performs column-wise reading and output of the current window without waiting for the former to complete, thereby realizing "wait-free" pipeline transmission of data and significantly improving data scheduling efficiency and throughput.
[0036] Pooling operations are completed collaboratively by the pooling operator control module and the pooling computation module. For example... Figure 6 As shown, the pooling operator control module first receives the pooling kernel size N×N and the pooling mode M=max pooling signal from the control center. The module pre-integrates commonly used pooling operator logic units, including max pooling, average pooling, and summation pooling. When M is detected as max pooling, the control module sends a "max pooling enable" signal and configures internal comparison logic units (selector, comparator, etc.) to make the pooling calculation module work according to the max pooling rule: that is, selecting the maximum value of the data within each 5×5 window; if M is average pooling, it sends an "average pooling enable" signal and configures adders and dividers (or shifters) to perform cumulative summation and then divide by 25; if M is summation pooling, it configures adder logic to directly sum the window data.
[0037] The pooling computation module receives column-oriented data and valid signals from the row-column dual-pipeline buffer output module. For example... Figure 2As shown, taking max pooling as an example, the module internally configures a 5×5 comparison structure in parallel for each input channel: First, each column of data (a total of 5 data points) is input to the corresponding comparator, and the maximum value of that column is output in parallel. This process continues for 5 clock cycles, obtaining one column maximum value in each cycle, for a total of 5 column maximum values. Subsequently, these 5 column maximum values are input to the final comparator group to calculate the global maximum value of the 5×5 window, i.e., the pooling result. Throughout the process, the data from 512 channels can undergo the above operations in parallel, greatly improving the calculation speed.
[0038] The example uses the 5×5 pooling of the SPPf module of the YOLOv5s algorithm to fully verify the universality, efficiency and practicality of the proposed FPGA-based row and column dual pipelined generalized pooling design.
Claims
1. A generalized pooling implementation method based on FPGA with row and column dual pipelines, characterized in that, The pooling module receives configuration parameters and input feature map data from the host computer. The pooling control center module configures the row and column pipeline data scheduling module and the pooling calculation control module through the configurable parameter module. The feature map data is processed by the row and column dual pipeline data caching module and the pooling calculation module to obtain the pooling calculation result. First, the pooling layer control center module receives the configuration parameters transmitted from the host computer and then transmits them to the configurable parameter register module via a parallel bus. The configurable parameter register module then transmits the parameters in register format to the row and column pipeline data scheduling module and the pooling operator control module, respectively. Secondly, after receiving the parameter register, the row and column pipelined data scheduling module modifies the number and size of the feature map buffer, the padding, and the step size S in the row and column pipelined data scheduling module. After receiving the parameter register, the pooling operator control module modifies the pooling method M and the pooling kernel size N of the pooling calculation module. Then, when the feature map data enters the pooling module, the row-column dual-pipeline caching module splits the feature map data and caches it row by row and channel by channel through the row-direction caching module and the column-direction concatenation module. Subsequently, the feature map data is output according to the size of the pooling kernel window module. Finally, after receiving the pooling window data, the pooling calculation module inputs the pooling operator module according to the pooling window input array and the corresponding pooling mode, and finally obtains the pooling result data.
2. The generalized pooling implementation method based on FPGA with row and column dual pipelines as described in claim 1, characterized in that, The pooling layer control center sends internal module parameters to the configurable parameter register module via the internal AXI bus. The configurable parameter register group configures parameters such as pooling window size N, pooling step size S, input feature map size W, pooling method M, and padding in the row-column dual-pipeline buffer module and the pooling calculation control module in the pooling layer module. At the same time, it configures the specific configuration of the feature map image data buffer BRAM and comparators (multiply-accumulators) in the module, such as the BRAM buffer size and the number of comparators required for pooling calculation.
3. The generalized pooling implementation method based on FPGA with row and column dual pipelines as described in claim 1, characterized in that: After the feature map input data enters the row-column dual-pipeline buffer module, the serially input data is sliced and reorganized in both row and column directions, and finally the data is input into the pooling calculation module according to the requirements of the pooling window. The row-column dual-pipeline buffer module further includes a row-direction buffer structure and a column-direction splicing array. The row-direction buffer module structure realizes horizontal pipeline (row pipeline). Data is written to the input buffer in rows. The row scheduler controls the row window to move and outputs the window rows with a step size S. The column-oriented splicing module implements vertical pipelined flow (column pipelined flow). Each group of window rows, after passing through the column-oriented buffer, is spliced into a complete pooled window and sent to the pooling unit. Bidirectional pipeline interleaving: the row and column schedulers operate asynchronously or semi-synchronously, enabling continuous data supply and output in both directions. This structure forms a two-dimensional pipeline scheduling mechanism, avoiding the pauses and gaps caused by window row or column swapping in traditional pipelined structures. The pooling kernel window control module controls the window size, pooling step size S, and edge padding of the bidirectional pipeline interleaving, and finally outputs pooling window data that meets the requirements according to the configuration instructions.
4. The generalized pooling implementation method based on FPGA with row and column dual pipelines as described in claim 1, characterized in that: After receiving the pooling window data from the row-column dual-pipeline buffer module, the pooling calculation module arranges and stores the data according to the requirements of the window input array module, and then passes the data to the pooling operator module according to the requirements of the pooling mode to obtain the pooling result. The pooling computation module consists of three sub-modules: Window input array module: Arranges the pooled window data output by the row and column dual pipeline buffer module according to the corresponding pooling method. Pooling mode configuration module: Receives instructions from the pooling operator control module to configure the pooling mode, which can be configured as max pooling, average pooling, or hybrid pooling. Pooling operator module: After the pooling window and pooling mode are determined, the data is input into the pooling operator, parallel calculation is performed based on the window data, and the pooling result is output. The pooling mode configures the pooling operator to the appropriate structure, schedules the data from the window input array module for pooling calculation, and finally outputs the pooling result.
5. The generalized pooling implementation method based on FPGA with row and column dual pipelines as described in claim 1, characterized in that: The pooling computation module matches the input data format of the convolution acceleration module. This pooling structure can be directly connected to the convolution acceleration module, supports continuous pipelined processing, and does not require additional data reassembly or intermediate buffering during the pipeline process.
6. The generalized pooling implementation method based on FPGA with row and column dual pipelines as described in claim 1, characterized in that: The pooling layers employ a bidirectional pipeline architecture based on rows and columns. The computation results of the pooling layer are directly fed into the on-chip BRAM cache and input to the next pooling layer, without the need for other data exchange channels. Furthermore, the BRAM only needs to cache row feature map data, not the entire feature map data. After the row data is prepared and entered into the BRAM cache in the previous layer, the next layer can start the computation without waiting for the entire image to be computed.