AI accelerator design technology based on maximum multiplexing of storage resources
By designing a multi-level storage architecture and data block caching, the memory access bandwidth bottleneck in CNN accelerators was solved, achieving efficient utilization of storage resources and reduced energy consumption, thereby improving computational efficiency and versatility.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING AEROSPACE AUTOMATIC CONTROL RES INST
- Filing Date
- 2025-12-31
- Publication Date
- 2026-05-01
AI Technical Summary
As existing CNN accelerators improve their computing power, memory access bandwidth becomes an efficiency bottleneck, and DDR memory access is the main source of energy consumption. Therefore, it is necessary to optimize the utilization of storage resources to reduce memory access volume and energy consumption.
The design incorporates a multi-layered storage architecture, including input feature cache, weight data cache, and output feature cache. Through data partitioning and on-chip caching, efficient data flow and reuse between different storage layers are achieved. Combining the characteristics of FPGA storage resources, the cache size is dynamically adjusted to adapt to the computational needs of different deep learning algorithms.
It improves the utilization of storage resources and bandwidth, reduces data transfer overhead, enhances neural network processing efficiency, and has strong versatility and computational efficiency.
Smart Images
Figure CN121958153A_ABST
Abstract
Description
An AI accelerator design technique based on maximizing the reuse of storage resources Technical Field
[0001] This invention belongs to the field of dedicated algorithm hardware circuit design, specifically relating to an AI accelerator design technology based on maximizing the reuse of storage resources. Background Technology
[0002] In recent years, target detection technology based on deep learning has developed rapidly and has been widely used in military and civilian fields, far surpassing traditional algorithms in detection accuracy. Due to the parallel nature of CNN computation, the CPU, as a serial scalar computing engine, cannot meet the computational requirements of training and inference. Dedicated hardware architectures are needed to improve computational efficiency. The computationally intensive backbone network typically uses dedicated AI acceleration chips or FPGA-based CNN acceleration circuits to accelerate deep learning algorithms, thereby improving the computational efficiency on the edge.
[0003] Computational power and memory access capability are the two major factors limiting the performance of CNN accelerators. Intuitively, the number of arithmetic units on a chip increases proportionally to its area, while memory bandwidth increases proportionally to its perimeter. Their growth rates are mismatched; that is, as computational power increases, memory access gradually becomes a new bottleneck for computational efficiency. On the other hand, DDR memory access is a major source of power consumption for CNN accelerators. To reduce memory bandwidth requirements, improve the utilization of computing resources, and reduce power consumption, we need to analyze the memory access characteristics of CNN algorithms and set appropriate on-chip caches to minimize memory access by maximizing data reuse.
[0004] This invention addresses the need for efficient data flow management in the design of neural network accelerators. It proposes an AI accelerator design technology based on maximizing the reuse of storage resources. Combining the characteristics of neural networks and FPGA storage resources, the accelerator architecture and on-chip cache design are carried out. By making full use of the locality of data access, the number of memory accesses is reduced by maximizing the number of data reuses, thereby improving the utilization rate of storage resources and bandwidth, and further improving the accelerator's processing efficiency for neural networks. Summary of the Invention
[0005] This invention relates to an AI accelerator design technology based on maximizing the reuse of storage resources, belonging to the field of dedicated algorithm hardware circuit design, and is used for efficient data management and reuse in hardware acceleration of deep learning algorithms, reducing data transfer overhead.
[0006] The accelerator in this invention comprises four parts: a control unit, a PE processing unit, a cache unit, and a data interconnect bus, as shown in Figure 1. Due to limited on-chip storage resources and the massive amount of data involved in the computation of each layer of the CNN algorithm, data partitioning must be considered, loading feature data and weight data into blocks onto the chip for computation. Therefore, the accelerator's workflow consists of the following steps:
[0007] (1) The main control unit actively obtains the instruction sequence from the external DDR, and after parsing, it dispatches the information in the instruction to each functional module through the on-chip control bus.
[0008] (2) The CACHE module reads the data blocks (Tensors) required for PE unit calculation from off-chip according to the instruction information, or writes the final result of PE unit calculation to a specified location in DDR.
[0009] (3) The PE unit performs operations such as convolution, activation, and pooling on the data block according to the instruction information. The entire calculation process is autonomously completed by the main control unit, which schedules and coordinates the various functional modules within the chip according to the information in the instruction sequence.
[0010] For a single forward inference operation of a CNN algorithm, the main control processor only needs to provide the CNN acceleration unit with a start signal and relevant initial information. The entire subsequent computation process is then completed autonomously by the CNN acceleration unit.
[0011] The beneficial effects of this invention are as follows:
[0012] (1) To address the efficiency bottleneck caused by chip memory access bandwidth in improving computing power, a multi-level storage architecture was designed using on-chip cache, which enabled efficient flow and reuse of data between different storage levels such as off-chip storage, on-chip cache, and computing units, reducing data transfer overhead.
[0013] (2) It has strong versatility and can dynamically adjust the cache size according to different types of deep learning algorithms to meet the data flow management needs during algorithm calculation. Attached Figure Description
[0014] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0015] Figure 1 shows the overall architecture of an AI accelerator design based on maximizing the reuse of storage resources.
[0016] Figure 2 is a schematic diagram of the on-chip cache structure for input features;
[0017] Figure 3 is a schematic diagram of the on-chip cache structure for weighted data;
[0018] Figure 4 is a schematic diagram of the on-chip cache structure for output features. Detailed Implementation
[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0020] To meet the data reuse requirements and minimize data interaction between memory accesses, this invention sets up three types of cache units: input feature cache (Fmin_buf), weight data cache (W_buf), and output feature cache (Fmout_buf). The memory access processes of input features, weight data, and output features are completely independent, thereby ensuring that each type of data can be reused multiple times during the calculation process.
[0021] (1) Input feature cache structure design
[0022] As shown in Figure 2, Fmin_buf is used to cache the input feature tiles read from the off-chip DDR and provide the data required for computation to the PE unit. To simplify the control logic and ensure that the input feature caching process and the PE unit's computation process occur simultaneously without interfering with each other, we set up two identical on-chip storage blocks, Ping and Pong, both dual-port RAMs implemented using BRAM, with a capacity of V. i_buf ×F_BIT, where F_BIT is the bit width of the feature data.
[0023] Port A transmits data to DDR via Datamover, with a bit width of 512 bits, meaning the Fmin_buf_wb bus has a bit width of 512 bits. Port B supplies feature data to PE, requiring a PD supply per clock cycle. i Since there are 10 characteristic data points, the bit width of the Fmin_buf_rb bus should be F_BIT × PD. i bit 。 In addition to the two data buses, Fmin_buf also has two chip select signals, Fmin_buf_wsel and Fmin_buf_rsel, used to select the Ping and Pang memory banks.
[0024] The depth of port A can be determined by the capacity V of Fmin_buf. i_buf The port width is simply determined by: V i_bufThe depth of port B is determined by ×F_BIT / 512. To fully utilize BRAM resources, the depth of port A can be adjusted to an integer multiple of 1024.
[0025] (2) Weighted data on-chip cache structure design
[0026] As shown in Figure 3, W_buf is used to cache the weight data tiles read from off-chip DDR and provide the data required for computation to the computing unit PE. It is a dual-port RAM implemented by BRAM with a total capacity of V. w_buf ×W_BIT, where W_BIT is the bit width of the weight data.
[0027] Port A transmits data to DDR via Datamover, with a bit width of 512 bits, meaning the W_buf_wb bus has a bit width of 512 bits. Port B supplies weight data to PE, with a bit width of PD. i ×K conv ×K conv ×W_BIT bit 。 In PD i ×PD o With the parallelism setting, each computation requires PD i ×K conv ×K conv ×PD o Each weighted data point, but the B port of W_buf can supply a maximum of PD to PE per clock cycle. i ×K conv ×K conv The number of reusable data sets is A. This is because the amount of weight data required for each PE calculation is too large. We adjust its reuse distance so that it is reused multiple times before the next batch is replaced. Under the constraint of data partitioning, the maximum number of times each batch of weight data can be reused is A. o_tile Generally, the following relationship holds:
[0028] A o_tile >PD o
[0029] Therefore, we set up two sets of weight data registers (W_reg). While one set of W_reg is used for calculation, data is buffered into the other set of W_reg, thus parallelizing the calculation with the W_reg buffering. In this case, the calculation time A o_tile The time PD greater than the buffer W_reg o Therefore, the bandwidth requirement for W_reg can be significantly reduced without slowing down the computation. When the above relationship is not satisfied, the PE unit will enter an idle state.
[0030] The depth of port A can be determined by the capacity V of W_buf. w_buf The port width is simply determined by: V w_buf The depth of port B is determined by ×W_BIT / 512. To fully utilize BRAM resources, the depth of port A can be adjusted to an integer multiple of 1024.
[0031] (3) Output Feature On-Chip Cache Structure Design
[0032] As shown in Figure 4, Fmout_buf consists of Conv_buf and Dump_fifo. Conv_buf is used to cache intermediate results of PE output, and Dump_fifo is used to cache the final results (output characteristics) of PE output, so that they can be written back to DDR in batches through Datamover.
[0033] Conv_buf is a dual-port RAM implemented using BRAM, with a total capacity of V. o_buf ×SUM_BIT, where SUM_BIT is the bit width of the intermediate result. Port A is used to read the intermediate result, and port B is used to write the intermediate result. Each PE calculation requires reading and writing PD from Conv_buf. o This is an intermediate result, therefore the bit width of ports A and B is PD. o ×SUM_BIT bit 。 Then the depth of ports A and B is V. o_buf / PD o To fully utilize BRAM resources, the depth can be adjusted to...
[0034] Dump_fifo is a PD implemented using BRAM. o / PD i It consists of several FIFOs. The image above shows the PD. o / PD i =2. Each FIFO buffer PD i The output characteristics of each channel, as can be seen from the arrangement of Fmout in the DDR, are that each PD... i The output characteristics of each channel are stored consecutively in DDR; therefore, the data in each FIFO of the Dump_fifo is written to DDR sequentially. The total capacity of the Dump_fifo is V. d_fifo ×F_BIT, where F_BIT is the bit width of the feature data.
[0035] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An AI accelerator design technology based on maximizing the reuse of storage resources, characterized in that, The accelerator comprises four parts: a control unit, a PE processing unit, a cache unit, and a data interconnect bus. The accelerator's workflow consists of the following steps: (1) The control unit actively obtains the instruction sequence from the external DDR, parses it, and then dispatches the information in the instruction to each functional module through the on-chip control bus; (2) The cache unit reads the data block (Tensor) required by the PE unit from outside the chip according to the instruction information, or writes the final result calculated by the PE unit to the specified location of the DDR; (3) The PE processing unit performs convolution, activation, pooling, and other operations on the data block according to the instruction information.
2. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 1, characterized in that, The entire calculation process is autonomously completed by the main control unit, which schedules and coordinates the various functional modules within the chip based on the information in the instruction sequence.
3. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 2, characterized in that, For a single forward inference operation of a CNN algorithm, the main control processor only needs to provide the CNN acceleration unit with a start signal and relevant initial information. The entire subsequent computation process is then completed autonomously by the CNN acceleration unit.
4. An AI accelerator design technology based on maximizing the reuse of storage resources according to any one of claims 1-3, characterized in that, Three types of cache units were set up: input feature cache (Fmin_buf), weight data cache (W_buf), and output feature cache (Fmout_buf).
5. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 4, characterized in that, The memory access processes for the input features, the weight data, and the output features are completely independent.
6. An AI accelerator design technique based on maximizing the reuse of storage resources according to claim 4 or 5, characterized in that, The Fmin_buf is used to cache the input feature tiles read from the off-chip DDR and to provide the data required for computation to the computing unit PE.
7. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 6, characterized in that, Two identical on-chip storage devices, Ping and Pong, are both dual-port RAMs implemented using BRAM, and each has a capacity of V. i_buf ×F_BIT, where F_BIT is the bit width of the feature data.
8. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 7, characterized in that, Port A transmits data to DDR via Datamover, with a bit width of 512 bits, meaning the Fmin_buf_wb bus has a bit width of 512 bits; Port B supplies feature data to PE, requiring a PD supply per clock cycle. i Since there are 10 characteristic data points, the bit width of the Fmin_buf_rb bus should be F_BIT × PD. i bit; In addition to the two data buses, Fmin_buf also has two chip select signals, Fmin_buf_wsel and Fmin_buf_rsel, for selecting Ping and Pang memory banks.
9. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 8, characterized in that, The depth of port A can be determined by the capacity V of Fmin_buf. i_buf The port width is simply determined by: V i_buf The depth of port B is determined by ×F_BIT / 512; to fully utilize BRAM resources, the depth of port A can be adjusted to an integer multiple of 1024.
10. An AI accelerator design technique based on maximizing the reuse of storage resources according to claim 4 or 5, characterized in that, The W_buf is used to cache the weight data Tile read from the off-chip DDR and to provide the data required for calculation to the computing unit PE.
11. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 10, characterized in that, A dual-port RAM implemented using BRAM, with a total capacity of V. w_buf ×W_BIT, where W_BIT is the bit width of the weight data.
12. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 11, characterized in that, Port A transmits data to DDR via Datamover, with a bit width of 512 bits, meaning the W_buf_wb bus has a bit width of 512 bits; Port B supplies weight data to PE, with a bit width of PD. i ×K conv ×K conv ×W_BIT bit; in PD i ×PD o With the parallelism setting, each computation requires PD. i ×K conv ×K conv ×PD o Each weighted data point, but the B port of W_buf can supply a maximum of PD_i×K_conv×K_conv to PE per clock cycle.
13. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 12, characterized in that, Under the constraint of data partitioning, the maximum number of times each batch of weighted data can be reused is A. o_tile Generally, the following relationship holds: A o_tile >PD o .
14. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 13, characterized in that, Two sets of weight data registers (W_reg) are set up. While one set of W_reg is used for calculation, data is buffered to the other set of W_reg, thus parallelizing the calculation with the W_reg buffering. In this case, the calculation time A o_tile The time PD greater than the buffer W_reg o Therefore, the bandwidth requirement for W_reg can be significantly reduced without slowing down the calculation; when the above relationship is not satisfied, the PE unit will be in an empty waiting state.
15. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 14, characterized in that, The depth of port A can be determined by the capacity V of W_buf. w_buf The port width is simply determined by: V w_buf The depth of port B is determined by ×W_BIT / 512. To fully utilize BRAM resources, the depth of port A can be adjusted to an integer multiple of 1024.
16. An AI accelerator design technique based on maximizing the reuse of storage resources according to claim 4 or 5, characterized in that, The Fmout_buf consists of Conv_buf and Dump_fifo. Conv_buf is used to cache the intermediate results of the PE output, and Dump_fifo is used to cache the final results of the PE output, so as to prepare for batch writing back to DDR through Datamover.
17. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 16, characterized in that, The output result is the output feature.
18. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 17, characterized in that, The Conv_buf is a dual-port RAM implemented by the BRAM, with a total capacity of V. o_buf ×SUM_BIT, where SUM_BIT is the bit width of the intermediate result; port A is used to read the intermediate result, and port B is used to write the intermediate result; each calculation of PE requires reading and writing PD from the Conv_buf. o This is an intermediate result, therefore the bit width of ports A and B is PD. o ×SUM_BITbit, where the depth of ports A and B is V. o_buf / PD o To fully utilize BRAM resources, the depth can be adjusted to 19. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 18, characterized in that, The Dump_fifo is a PD implemented by BRAM. o / PD i It consists of FIFOs; each FIFO caches PDs. i The output characteristics of each channel, as can be seen from the arrangement of Fmout in the DDR, are that each PD... i The output characteristics of each channel are stored in a continuous set in DDR. Therefore, the data of each FIFO in Dump_fifo is written to DDR sequentially.
20. The AI accelerator design technology based on maximizing the reuse of storage resources according to claim 19, characterized in that, The total capacity of the Dump_fifo is V. d_fifo ×F_BIT, where F_BIT is the bit width of the feature data.