A multi-stage pooling method based on FPGA
By dividing the dynamic random access memory (DRAM) into regions A and B in the FPGA and switching its function according to the pooling level, and using M×M block storage units for pooling processing, the problems of high resource consumption and processing difficulty in multi-level pooling are solved, achieving efficient resource utilization and real-time performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING HUAHANG RADIO MEASUREMENT & RES INST
- Filing Date
- 2022-12-05
- Publication Date
- 2026-06-23
AI Technical Summary
In deep learning algorithms, FPGA-based multi-level pooling operations suffer from high resource consumption and processing difficulty.
The dynamic random access memory is divided into region A and region B, and matching pooling instructions are generated according to the pooling level. Region A and region B are switched between the partitioned region and the pooling result storage area. Pooling is performed through M×M block storage units to optimize resource utilization and real-time performance.
It achieves optimal resource utilization during multi-level pooling, reduces processing difficulty, ensures high real-time performance, and reduces storage resource consumption.
Smart Images
Figure CN118196463B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning intelligent algorithm technology, and in particular to a multi-level pooling method based on FPGA. Background Technology
[0002] In the implementation of deep learning intelligent algorithms, if the image size is too large, pooling can be introduced to reduce the size of the parameter matrix, thereby reducing the number of parameters in the final fully connected layer and preventing overfitting during model training. Furthermore, in the field of image recognition, pooling operations can also provide translation and rotation invariance, essentially acting as a downsampling process.
[0003] When deploying deep learning algorithms on an FPGA, it is usually necessary to repeatedly read and write intermediate computation results to DDR to complete the entire operation process. To implement the pooling layer method, it is usually necessary to allocate a BRAM on the FPGA chip, read each feature layer into the chip, and then perform the pooling operation. This implementation method consumes a lot of on-chip storage resources and cannot guarantee real-time performance.
[0004] In addition, during the actual implementation process, there may be multi-level pooling, which further increases the resource consumption and processing difficulty of FPGA-based multi-level pooling. Summary of the Invention
[0005] This invention provides a multi-level pooling method based on FPGA, which can solve the problems of high resource consumption and high processing difficulty in the existing FPGA-based multi-level pooling technology.
[0006] This invention provides a multi-level pooling method based on FPGA, comprising:
[0007] Based on the above solution, the present invention also makes the following improvements:
[0008] The dynamic random access memory is divided into region A and region B, and the image to be processed corresponding to the first pooling level is sent to region A.
[0009] For each pooling level, the following pooling process is executed:
[0010] Generate a pooling instruction matching the current pooling level; the pooling instruction is used to: when the pooling level is odd, control area A as a partition area and area B as a pooling result storage area; when the pooling level is even, control area A as a pooling result storage area and area B as a partition area;
[0011] Dynamic random access memory configures regions A and B according to the pooling instructions;
[0012] The image to be processed corresponding to the current pooling level is segmented by row using the segmentation region, and the segmented sub-images are stored sequentially by row; starting from the first row, every M rows of sub-images are treated as a group of sub-images to be processed; the size of the pooling kernel is M×M;
[0013] Each group of sub-images to be processed is received and pooled to obtain the pooling result;
[0014] The pooling results of each group of sub-images to be processed are stored in the pooling result storage area and used as the images to be processed for the next pooling level.
[0015] Jump to the next pooling stage to perform the pooling process, until the last pooling stage.
[0016] Based on further improvements to the above method, each group of sub-images to be processed is received and pooled to obtain the pooling results, including:
[0017] For each set of sub-images to be processed, the M rows of sub-images in the set are synchronously input into a block storage unit of size M×M in the FPGA chip. When the block storage unit is full of M×M data, pooling is performed and the pooling result is output.
[0018] Further improvements to the above method include receiving and pooling each group of sub-images to be processed to obtain the pooling result, and also include:
[0019] After the block storage unit has processed M×M data in a pool, the block storage unit is cleared.
[0020] Then, the block storage unit continues to receive the remaining column data of the M rows of sub-images in the current group of sub-images to be processed.
[0021] Further improvements to the above method include receiving and pooling each group of sub-images to be processed to obtain the pooling result, and also include:
[0022] Real-time monitoring of the status of block storage units:
[0023] When the block storage unit is detected to be full, the reception of the sub-image to be processed is stopped;
[0024] When the state of the block storage unit is detected as empty, the remaining column data of the M rows of sub-images in the previous group of sub-images to be processed continue to be received.
[0025] Further improvements to the above method include receiving and pooling each group of sub-images to be processed to obtain the pooling result, and also include:
[0026] Each group of sub-images to be processed is simultaneously received and pooled.
[0027] Further improvements to the above method include receiving and pooling each group of sub-images to be processed to obtain the pooling result, and also include:
[0028] The sub-images to be processed in each group are received and pooled sequentially according to the group number from smallest to largest.
[0029] Based on a further improvement to the above method, storing the pooling results output from each group of sub-images to be processed in the pooling result storage area includes:
[0030] The pooling results output after processing the same group of sub-images are stored row-wise in the order of output.
[0031] The pooling results output after processing different groups of sub-images are stored in column-directed order according to the group number from smallest to largest; the first M rows of sub-images starting from the first row are taken as the first group of sub-images to be processed, and the group numbers of the subsequent groups of sub-images to be processed are sequentially increased.
[0032] The array formed by storing all rows and columns is used as the pooling result of the image to be processed and stored in the pooling result storage area.
[0033] Based on the further improvement of the above method, after the pooling process corresponding to each pooling level is completed, the partition corresponding to the current pooling level is left empty.
[0034] Based on the further improvement of the above method, the size of the image to be processed corresponding to the first pooling level is (N1×M)×(N2×M); N1, N2, and M are all positive integers, N1 and N2 are both integer multiples of M, and M is greater than 1.
[0035] Based on the further improvement of the above method, the length and width of the image to be processed corresponding to each pooling level are both integer multiples of M.
[0036] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0037] This invention provides a multi-level pooling method based on FPGA. By dividing the dynamic random access memory (DRAM) into regions A and B, and generating pooling instructions matching the current pooling level, the methods define the functions of region A and region B as segmentation areas and pooling result storage areas. In practice, when region A is a segmentation area and region B is a pooling result storage area, pooling processing is performed on the image to be processed in region A. After processing, the pooling level is incremented by 1. At this point, the pooling result in region B is stored as a segmentation area, and the vacant segmentation area in region A is converted into a pooling result storage area. Then, region B (the pooling result of the previous level, also in image form) is further segmented, and the pooling result is stored in the vacant region A, thus consuming minimal resources, achieving multi-level pooling, and reducing processing difficulty. Meanwhile, in the pooling module, pooling operations can be achieved using only M×M block storage units, maximizing the saving of on-chip storage resources and providing high real-time performance. During the data reading process, calculations are performed as soon as M×M data are acquired, without waiting for all data to be read before execution, which greatly saves the resources of the block storage units and ensures the real-time performance of the calculations.
[0038] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description
[0039] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.
[0040] Figure 1 A flowchart of a multi-level pooling method based on FPGA provided in an embodiment of the present invention;
[0041] Figure 2 This is a schematic diagram of a multi-level pooling system structure based on FPGA provided in an embodiment of the present invention. Detailed Implementation
[0042] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0043] This invention provides a multi-level pooling method based on FPGA, the flowchart of which is shown below. Figure 1 As shown, it includes the following steps:
[0044] Step S1: Divide the dynamic random access memory into region A and region B, and send the image to be processed corresponding to the first pooling level to region A;
[0045] Step S2: For each pooling level, perform the following pooling process:
[0046] Step S21: Generate a pooling instruction matching the current pooling level; the pooling instruction is used to: when the pooling level is odd, control area A as a partition area and area B as a pooling result storage area; when the pooling level is even, control area A as a pooling result storage area and area B as a partition area;
[0047] Step S22: The dynamic random access memory configures region A and region B according to the pooling instruction;
[0048] Step S23: Divide the image to be processed corresponding to the current pooling level into rows using the segmentation region, and store the segmented sub-images in row order; starting from the first row, every M rows of sub-images are treated as a group of sub-images to be processed; the size of the pooling kernel is M×M;
[0049] Step S24: Receive and pool each group of sub-images to be processed to obtain the pooling result;
[0050] Specifically, including:
[0051] Step S241: For each set of sub-images to be processed received, the M rows of sub-images in the set are synchronously input into the block storage unit of size M×M in the FPGA chip in columns; when the block storage unit is full of M×M data, pooling processing is performed and the pooling result is output.
[0052] in,
[0053] Step S2411: After the block storage unit has finished pooling and processing M×M data, the block storage unit is cleared;
[0054] Step S2412: Then, the block storage unit continues to receive the remaining column data of the M rows of sub-images in the current group of sub-images to be processed.
[0055] During this process, the status of the block storage unit is monitored in real time: when the block storage unit is detected to be full, the reception of the sub-image to be processed is stopped; when the block storage unit is detected to be empty, the remaining column data of the M rows of sub-images in the previous group of sub-images to be processed continues to be received.
[0056] In step S24, each group of sub-images to be processed can be received and pooled synchronously; or, each group of sub-images to be processed can be received and pooled sequentially in ascending order of group number.
[0057] Step S25: Store the pooling results of each group of sub-images to be processed in the pooling result storage area, and use them as the images to be processed for the next pooling level; specifically:
[0058] Step S251: The pooling results output after processing the same group of sub-images are stored row-wise in the output order;
[0059] Step S252: The pooling results output after processing different groups of sub-images are stored in column-directed order according to the group number from smallest to largest; wherein, the first M rows of sub-images starting from the first row are taken as the first group of sub-images to be processed, and the group number of each subsequent group of sub-images to be processed increases sequentially.
[0060] Step S253: The array formed by storing all rows and columns is used as the pooling result of the image to be processed and stored in the pooling result storage area.
[0061] After the pooling process corresponding to each pooling level is completed, the partition corresponding to the current pooling level is left empty.
[0062] Step S26: Jump to the step corresponding to the next pooling level, until the last pooling level.
[0063] This invention also discloses an FPGA-based multi-level pooling system, which can be used in conjunction with the above method. A schematic diagram of the system is shown below. Figure 2 As shown, it includes:
[0064] The pooling control module is used to generate pooling instructions pointing to the dynamic random access memory based on the pooling level.
[0065] The dynamic random access memory (DRAM) is divided into region A and region B. Based on the pooling instruction, the DRAM switches between region A and region B between the segmentation area and the pooling result storage area. The segmentation area is used to segment the image to be processed corresponding to the current pooling level by row, and to store the segmented sub-images sequentially by row. Starting from the first row, every M rows of sub-images form a group of sub-images to be processed. The pooling kernel size is M×M. The pooling result storage area is used to store the pooling results.
[0066] The pooling module includes M input interfaces, an M×M block storage unit, and an output interface. It inputs M rows of sub-images from each group of sub-images to be processed in the segmented region into the block storage unit through the M corresponding input interfaces. When the block storage unit receives M columns of data, it performs pooling processing on the M×M data in the block storage unit and outputs the pooling result to the pooling result storage area through the output interface.
[0067] Preferably, for each group of sub-images to be processed, after the block storage unit has completed the pooling processing of M×M data, the block storage unit is cleared; then, the block storage unit continues to receive the remaining column data of the M rows of sub-images in the current group of sub-images to be processed. After the block storage unit receives M columns of data again, the pooling processing and pooling result output are repeated.
[0068] Preferably, the number of pooling modules is set to 1 or M. Specifically:
[0069] When the number of pooling modules is 1, in the FPGA chip, after each group of sub-images to be processed is pooled, the process jumps to the pooling process of the next group of sub-images to be processed.
[0070] When the number of pooling modules equals the total number of sub-image groups to be processed, each sub-image group to be processed corresponds to one pooling module, and the pooling module only performs pooling processing on the corresponding sub-image group to be processed.
[0071] Because data input needs to be paused when the block storage unit is full (i.e., filled with M×M data), and needs to continue when it is empty, this embodiment also includes a detection control unit. The detection control unit is used to disable the input interface when the block storage unit is detected to be full, at which point the input interface stops receiving sub-images; and to enable the input interface when the block storage unit is detected to be empty, at which point the data interface continues receiving sub-images.
[0072] In the pooling result storage area, the pooling results output after processing the same group of sub-images are stored row-wise in the output order; simultaneously, the pooling results output after processing different groups of sub-images are stored column-wise in ascending order of group number; wherein, the first M rows of sub-images starting from the first row are regarded as the first group of sub-images to be processed, and the group numbers of the subsequent groups of sub-images to be processed are sequentially increased. In this way, after the pooling of the image to be processed corresponding to the current pooling level is completed, the array formed by the row-wise and column-wise storage in the output dynamic random access memory is used as the pooling result of the image to be processed corresponding to the current pooling level; at the same time, the pooling result of the image to be processed corresponding to the current pooling level is used as the image to be processed corresponding to the next pooling level.
[0073] In this embodiment, after each pooling level is completed, the pooling result storage area stores the pooling result of the image to be processed corresponding to the pooling level. At this time, the pooling control module updates the pooling level and generates a pooling instruction pointing to the dynamic random access memory (DRAM) based on the updated pooling level. Exemplarily, the DRAM, according to the pooling instruction, enables the switching between region A and region B between the segmentation region and the pooling result storage area, specifically including:
[0074] When the number of pooling levels is odd, control area A as the partitioning area and area B as the pooling result storage area;
[0075] When the pooling level is even, control area A is the pooling result storage area and area B is the partitioning area.
[0076] To facilitate better implementation of this solution by those skilled in the art, the following explanation is provided regarding the changes in pooling levels and their corresponding pooling instructions and operations:
[0077] When performing the first-level pooling, the pooling level is 1. The original image to be processed is stored in region A as the image to be processed in the first-level pooling. After being processed by the pooling module, the pooling result of the first-level pooling is stored in region B, while region A is left empty.
[0078] Then, the pooling control module updates the pooling level to 2, and simultaneously updates the pooling instructions, controlling area A as the pooling result storage area and area B as the partitioning area. At this time,
[0079] The pooling result of the first-level pooling stored in area B is used as the image to be processed for the second-level pooling. After being processed by the pooling module, the pooling result of the second-level pooling is stored in area A, while area B is left empty.
[0080] The processing steps for subsequent series are similar.
[0081] It should be noted that, considering the image size requirements of pooling processing, in this embodiment, the size of the image to be processed corresponding to the first pooling level is (N1×M)×(N2×M); N1, N2, and M are all positive integers, N1 and N2 are integer multiples of M, and M is greater than 1. The size of the image to be processed directly determines the maximum number of pooling operations that can be performed, and the length and width of the image to be processed corresponding to each pooling level are integer multiples of M.
[0082] Preferably, in this embodiment, a double-rate synchronous dynamic random access memory is selected to effectively improve the real-time performance of pooling processing.
[0083] In this embodiment, the pooling control module, the pooling module, and the detection control unit are all housed within the FPGA chip.
[0084] In summary, the FPGA-based multi-level pooling method and approach provided in this embodiment divides the dynamic random access memory (DRAM) into regions A and B, and limits their functions. This ensures that when region A is a segmentation region and region B is a pooling result storage region, pooling processing is performed on the image to be processed in region A. After processing, the pooling level is incremented by 1. At this point, the pooling result in region B is stored as a segmentation region, and the vacant segmentation region in region A is converted into a pooling result storage region. Then, region B (the pooling result of the previous level, which is also in image form) is further segmented, and the pooling result is stored in the vacant region A, thus consuming minimal resources to achieve multi-level pooling and reducing processing difficulty. Furthermore, the pooling module can implement pooling operations using only M×M block storage units, maximizing the saving of on-chip storage resources and providing high real-time performance. During data reading, calculations are performed as soon as M×M data are acquired, without waiting for all data to be read before execution, greatly saving block storage unit resources and ensuring real-time operation.
[0085] Those skilled in the art will understand that all or part of the processes of the system described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0086] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A multi-level pooling method based on FPGA, characterized in that, include: The dynamic random access memory is divided into region A and region B, and the image to be processed corresponding to the first pooling level is sent to region A. For each pooling level, the following pooling process is executed: Generate a pooling instruction matching the current pooling level; the pooling instruction is used to: when the pooling level is odd, control area A as a partition area and area B as a pooling result storage area; when the pooling level is even, control area A as a pooling result storage area and area B as a partition area; Dynamic random access memory configures regions A and B according to the pooling instructions; The image to be processed corresponding to the current pooling level is segmented by row using the segmentation region, and the segmented sub-images are stored sequentially in row order; Starting from the first row, each M rows of sub-images are treated as a group of sub-images to be processed; The size of the pooling core is M×M; Each group of sub-images to be processed is received and pooled to obtain the pooling result; The pooling results of each group of sub-images to be processed are stored in the pooling result storage area and used as the images to be processed for the next pooling level. Jump to the next pooling stage to perform the pooling process, until the last pooling stage.
2. The FPGA-based multi-level pooling method according to claim 1, characterized in that, For each group of sub-images to be processed, receive and perform pooling processing to obtain the pooling results, including: For each set of sub-images to be processed, the M rows of sub-images in the set are synchronously input into a block storage unit of size M×M in the FPGA chip. When the block storage unit is full of M×M data, pooling is performed and the pooling result is output.
3. The FPGA-based multi-level pooling method according to claim 2, characterized in that, For each group of sub-images to be processed, the receiving and pooling processes are performed to obtain the pooling results, which also include: After the block storage unit has processed M×M data in a pool, the block storage unit is cleared. Then, the block storage unit continues to receive the remaining column data of the M rows of sub-images in the current group of sub-images to be processed.
4. The FPGA-based multi-level pooling method according to claim 3, characterized in that, For each group of sub-images to be processed, the receiving and pooling processes are performed to obtain the pooling results, which also include: Real-time monitoring of the status of block storage units: When the block storage unit is detected to be full, the reception of the sub-image to be processed is stopped; When the state of the block storage unit is detected as empty, the remaining column data of the M rows of sub-images in the previous group of sub-images to be processed continue to be received.
5. The FPGA-based multi-level pooling method according to claim 4, characterized in that, For each group of sub-images to be processed, the receiving and pooling processes are performed to obtain the pooling results, which also include: Each group of sub-images to be processed is simultaneously received and pooled.
6. The FPGA-based multi-level pooling method according to claim 4, characterized in that, For each group of sub-images to be processed, the receiving and pooling processes are performed to obtain the pooling results, which also include: The sub-images to be processed in each group are received and pooled sequentially according to the group number from smallest to largest.
7. The FPGA-based multi-level pooling method according to any one of claims 1-6, characterized in that, The step of storing the pooling results output from each group of sub-images to be processed in the pooling result storage area includes: The pooling results output after processing the same group of sub-images are stored row-wise in the order of output. The pooling results output after processing different groups of sub-images are stored in column-directed order according to the group number from smallest to largest; the first M rows of sub-images starting from the first row are taken as the first group of sub-images to be processed, and the group numbers of the subsequent groups of sub-images to be processed are sequentially increased. The array formed by storing all rows and columns is used as the pooling result of the image to be processed and stored in the pooling result storage area.
8. The FPGA-based multi-level pooling method according to claim 7, characterized in that, After the pooling process corresponding to each pooling level is completed, the partition corresponding to the current pooling level is left empty.
9. The FPGA-based multi-level pooling method according to claim 7, characterized in that, The size of the image to be processed corresponding to the first pooling level is (N1×M)×(N2×M); N1, N2, and M are all positive integers, N1 and N2 are integer multiples of M, and M is greater than 1.
10. The FPGA-based multi-level pooling method according to claim 9, characterized in that, The length and width of the image to be processed for each pooling level are integer multiples of M.