A lightweight neural network processor storage architecture co-optimization method
By optimizing the storage architecture of neural network processors through differentiated single-port storage, semi-pulsating arrays, and fused ALU architecture, and combining timing analysis with dynamic adjustment strategies, the area and power consumption issues of storage architecture in μNPU scenarios were resolved, achieving efficient storage system optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-03-05
- Publication Date
- 2026-07-21
AI Technical Summary
The existing storage architecture of high-performance neural network processors is difficult to meet the strict constraints of area and power consumption in micro machine learning environments, and traditional designs cannot be effectively optimized in the μNPU scenario.
It adopts differentiated single-port storage settings, half-pulse array and converged ALU computing architecture, combined with dynamic adjustment strategy of timing analysis, to optimize storage structure to reduce bandwidth and bit width requirements, and handle access contention through ping-pong buffer and hardware arbitration.
While ensuring computing performance, it significantly improves the area efficiency and access efficiency of the storage system, achieving coordinated optimization of bandwidth, area and power consumption.
Smart Images
Figure CN121809563B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural network inference hardware technology, and in particular to a method for co-optimization of lightweight neural network processor storage architecture. Background Technology
[0002] With the rapid development of the Internet of Things (IoT) and edge intelligence, TinyML has become a key paradigm for achieving real-time intelligent decision-making in resource-constrained environments. Its goal is to perform neural network inference directly on microcontrollers (MCUs) with milliwatt-level power consumption and only a few hundred KB of storage space, thereby ensuring data privacy, reducing cloud dependence, and achieving low-latency response. However, the limited computing power and storage resources of traditional MCUs cannot meet the extremely high demands of modern deep neural networks for computing power and storage. This contradiction has directly driven the development of microcontroller-level neural network processors (μNPUs). μNPUs, through customized hardware architectures, can achieve inference efficiency orders of magnitude higher than general-purpose MCUs while maintaining low power consumption. Among these, the storage architecture is crucial in determining area cost and computational energy efficiency: on the one hand, on-chip storage access is frequent, and the power consumption of a single read / write operation is significantly higher than that of the computing unit; on the other hand, to reduce access to external DRAM, μNPUs typically integrate large amounts of on-chip storage to accommodate high-frequency data, resulting in a storage area ratio often exceeding 60%. Considering that MCUs are extremely sensitive to performance, power consumption, and area cost metrics, collaborative optimization research on the storage architecture of μNPUs is essential.
[0003] The development of storage architecture in neural network processors exhibits clear stages. Early solutions, such as those based on Cadence DSPs or early Cambricon processors, generally employed a shared memory architecture. This architecture places various types of data, including inputs, weights, and outputs, in a uniformly addressed memory space. While this helps improve the overall utilization of memory space, it requires the design of complex multi-port read / write arbitration units to handle parallel access conflicts generated by multiple computing units, which limits further improvements in system energy efficiency. In pursuit of ultimate computational efficiency and energy efficiency, modern designs have shifted to dedicated storage architectures deeply customized for specific data flow patterns, resulting in two main technological paths. The systolic array architecture, represented by Google TPU, achieves deep reuse of weights by allowing data (such as weights and activation values) to flow and be computed between processing units (PEs) according to a fixed rhythm, thereby significantly reducing reliance on high-bandwidth external storage. However, this efficiency is built upon strict spatiotemporal synchronization control of data flow, requiring a large number of registers to construct data systolic paths and exhibiting high pipeline fill latency. Another technological path, represented by NVDLA and VTA, employs a computational model that combines multicast and additive trees. This architecture is naturally suited for convolutional operations, its programming model is relatively simple, and it does not require the introduction of IM2COL data reorganization operations that would cause memory bloat. However, its significant drawback is its extremely high demand for storage bandwidth, especially for weighted data, requiring the storage system to provide a dense data supply for the highly parallel processing unit array within a single cycle.
[0004] However, the pulsating array of TPUs and the multicast-addition tree architecture of NVDLA / VTA are both designed for cloud-side or high-performance edge computing scenarios with huge computing power requirements (usually no less than 256 MACs) and relatively abundant storage resources (often reaching hundreds of KB levels). The core optimization goal of these designs is to pursue peak computing power and overall energy efficiency, rather than extreme area cost efficiency. This is fundamentally in conflict with the strict constraints on area and power consumption in the μNPU scenario, and cannot be adapted through simple direct miniaturization. Taking a typical lightweight NPU for the end market as an example, its typical configuration is 32 to 256 MACs of computing power and 18 to 50KB of internal storage. If the fine-grained, multi-block design scheme adopted by NVDLA / VTA to meet high bandwidth is used, it will inevitably lead to too many storage blocks and too small physical capacity per block. Since the fixed area overhead of SRAM peripheral circuits increases sharply as the capacity decreases, the effective bit area efficiency of the storage cell is significantly reduced, and the total area cost is thus out of control. If the TPU systolic array approach is used to integrate storage to reduce bandwidth, the number of storage blocks can be reduced, but the large number of registers required to build the data systolic network will introduce new and significant area overhead, which is also difficult to bear under the strict area budget of μNPU.
[0005] In summary, existing architectures designed for high-performance scenarios cannot directly meet the stringent area and cost requirements of μNPU. Summary of the Invention
[0006] To address the aforementioned technical problems in the existing technology, this invention proposes a lightweight neural network processor memory architecture collaborative optimization method, the specific technical solution of which is as follows:
[0007] A method for collaborative optimization of the storage architecture of a lightweight neural network processor includes:
[0008] Step 1: Analyze the differences in bandwidth requirements and access patterns of on-chip data, and set differentiated single-port storage settings for the buffer storage of each data in the neural network processor;
[0009] Step 2: The GEMM module of the neural network processor is set up based on a half-systolic array, and an ALU operation fusion processing module is used to reduce the data storage bandwidth and bit width;
[0010] Step 3: Optimize the storage structure by introducing a dynamic adjustment strategy based on time series analysis, thereby outputting an optimized neural network processor.
[0011] Furthermore, in step 1, for high-bandwidth, access-intensive input data and weight data, a ping-pong buffer-based dual-block single-port storage is used for alternating access; for high-bandwidth, access-intensive partial data, a distributed register cache is used; for low-bandwidth, access-sparse bias parameters and output data, a single-block single-port storage is used, and access contention is handled through a hardware arbitration mechanism.
[0012] Furthermore, the number of times the weight data is reused is less than or equal to the number of entries in the register.
[0013] Furthermore, the hardware arbitration mechanism for the output data is fixed to write-first.
[0014] Furthermore, in step 2, the semi-pulsating array is a two-dimensional processing array that broadcasts along the parallel direction K and pulsates along the accumulation direction C, used to reduce the read bandwidth of the weights while maintaining the vectorized storage and access of the input data along the C direction.
[0015] Furthermore, each computing unit of the two-dimensional processing array has two sets of weight registers, one set for the current calculation and the other set for preloading the next calculation; each computing unit also has a set of partial sum registers for accumulation along the vertical direction.
[0016] Furthermore, since the first row of computing units in the two-dimensional processing array does not require partial sum input data, the bit width of the adder and partial sum register of each row of computing units is directly determined by the input and weight bit width; the partial sum register of the last row of computing units in the two-dimensional processing array is accumulated with the external partial sum register.
[0017] Furthermore, in step 2, the ALU operation fusion processing module is used for multi-cycle bias parameter reading, so that the bit width of the bias storage is reduced to the bus bit width.
[0018] Furthermore, in step 3, the dynamic adjustment strategy based on timing analysis specifically involves: performing timing checks on the buffer storage generated under the target process through logic synthesis; when the timing meets the requirements, directly outputting the optimized neural network processor architecture; when the large local storage capacity leads to increased latency and timing violations, performing a storage splitting operation; splitting the local storage into multiple small-capacity storage blocks that can be accessed independently and in parallel, keeping the width unchanged and reducing the depth to shorten read and write access latency; repeating the storage splitting operation until the timing meets the requirements; then adjusting the original hardware arbitration-dependent storage access to a software-scheduled ping-pong access mode; utilizing the additional bandwidth obtained after splitting the bias parameter storage to shorten the number of cycles required to load data into the fused ALU register.
[0019] Furthermore, during the repeated execution of the storage splitting operation, if there is still a timing violation after one split, the splitting continues until the minimum capacity configuration that the target process can generate for this bit width is reached. If there is still a timing violation under the minimum capacity configuration, an error is reported: the timing violation cannot be resolved by storage splitting, and it is recommended to adjust the content outside the storage to ensure timing.
[0020] Beneficial effects: While ensuring computing performance, this invention significantly improves the area efficiency and access efficiency of the storage system, and achieves synergistic optimization of bandwidth, area and power consumption. Attached Figure Description
[0021] Figure 1 This is a schematic diagram of a typical neural network processor architecture based on dedicated storage in this embodiment;
[0022] Figure 2 This is a schematic diagram of the neural network processor architecture based on differentiated single-port storage in this embodiment;
[0023] Figure 3 This is an architecture diagram of a semi-pulsating array according to this embodiment;
[0024] Figure 4 This is a diagram of the neural network processor architecture after memory bit width co-optimization in this embodiment;
[0025] Figure 5This is a flowchart of the dynamic adjustment strategy for the storage structure in this embodiment. Detailed Implementation
[0026] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0027] like Figure 1 The diagram illustrates a typical neural network processor architecture based on dedicated storage. The Load module is responsible for moving input and weight data from external Dynamic Random Access Memory (DRAM) to the internal input buffer INP, weight buffer WGT, and bias buffer BIAS. The GEMM module performs general matrix multiplication calculations, assuming its PE array size is nP x nR, where nP is the number of parallel operations and nR is the number of accumulations. The nP dimension uses multicast, and the nR dimension uses an addition tree. The data types for input, weights, partial sums, biases, and outputs are INT8, INT8, INT32, INT32, and INT8, respectively. Without register data multiplexing, the GEMM module needs to read 1 x nR x INT8 of input data, nP x nR x INT8 of weight data, 1 x nP x INT32 of partial sum input data, and write 1 x nP x INT32 of partial sum output data per cycle. Because the GEMM and ALU modules access the partial sum and buffer ACC simultaneously, a two-read, two-write access pattern occurs, resulting in high storage interface requirements and significant hardware overhead for direct implementation. Therefore, a serial execution method for GEMM and ALU, such as VTA, is typically chosen. The ALU module is responsible for performing post-processing operations such as biasing and quantization. Each cycle, it needs to read 1xnPxINT32 of partial sum and input data, at most 1xnPxINT32 of bias or quantization parameters, and write 1xnPxINT32 of partial sum and output data. Since these are typically in-place processing operations, the partial sum and read / write operations become the performance bottleneck for the ALU. The final output of the ALU is written to the output buffer OUT. After the entire output block has been calculated, the Store module is responsible for writing the output block back to the external DRAM.
[0028] This invention provides a collaborative optimization method for storage architecture targeting lightweight neural network processors, with area cost as the optimization objective, as detailed below:
[0029] The peak computing power of the neural network processor depends on the number of MAC (Multiply-Accumulate) units in the GEMM (Generative Multiplication and Accumulation) module. It needs to acquire input data, weight data, and partial sum data each cycle to maintain sufficient MAC utilization. Access to this data is intensive and bandwidth-intensive. After the GEMM completes all accumulation operations, the ALU begins the corresponding post-processing operations. It needs to acquire partial sum data (ACC) and bias (BIAS) each cycle, and after processing, write the final result to the output buffer (OUT). Therefore, reading BIAS and writing OUT are sparse, triggered only after the partial sum data accumulation is complete.
[0030] Therefore, this invention designs a differentiated single-port storage implementation method based on data access characteristics. First, by analyzing the differences in bandwidth requirements and access modes of on-chip data, the more expensive pseudo-dual-port storage is replaced with various single-port storage methods that are more area-efficient.
[0031] like Figure 2 The diagram illustrates the application of the differentiated single-port storage implementation method of this invention. Specifically, for high-bandwidth, access-intensive input data and weighted data, a dual-block single-port storage based on ping-pong buffers is adopted. Alternating access is achieved through software scheduling; for example, the Load module writes to INP0, and the GEMM module reads from INP1. At the hardware level, INP0 and INP1 are the low and high address spaces of the local input storage, respectively. At the software level, the two input blocks executed in ping-pong are mapped to INP0 and INP1 respectively through address mapping. GEMM also accesses the required input block through address. That is, the selectors at the Load and GEMM ends of the input buffers INP0 and INP1 determine which block to access based on address, and software avoids conflicts from accessing the same storage array.
[0032] For high-bandwidth, intensively accessed partial sums and data, the demand for parallel read / write operations is high. Leveraging the limited number of accumulations in TinyML scenarios, the centralized partial sum buffer can be directly replaced with distributed registers. As long as the accessed register entries are different, multi-path parallel read / write access can be supported. However, storing partial sums and data in registers requires the coordination of multiplexed output data streams, and the number of times the weights are reused must be less than or equal to the number of register entries to prevent partial sum data overflow during accumulation. The data stream can be implemented through software scheduling.
[0033] For low-bandwidth, sparsely accessed bias parameters and output data, single-block, single-port storage is used, and access contention is handled through a hardware arbitration mechanism. Bias parameters are only read during post-processing operations on the accumulated portion of the data. Furthermore, bias parameters are reused spatially; when segmenting data along the spatial dimension, the bias parameters only need to be loaded once, resulting in sparse reads and writes. The hardware arbitration mechanism for bias parameters can employ simple polling, or, considering multiple read requests from the Load module, a write-first approach can reduce congestion on the bus read channel. Quantization parameters are processed similarly to bias parameters and are also stored in the bias buffer BIAS. Similarly, after the ALU completes post-processing operations, it writes the output results sporadically to the output buffer OUT. Once the entire output block is complete, the Store performs continuous read operations, writing the output to external storage. In the entire neural network processor architecture, only the Store module issues write requests to the bus; therefore, a write-first hardware arbitration mechanism for OUT is recommended to avoid impacting the Store module's performance.
[0034] This method achieves overall simplification and area optimization of the storage system by finely differentiating the storage architecture, while ensuring the performance of critical data paths.
[0035] Secondly, a storage bit width co-optimization method based on a semi-pulsating array and a fused ALU computing architecture is designed to reduce the bandwidth and bit width requirements of critical storage, thereby optimizing the storage area.
[0036] Among them, such as Figure 3 As shown, this invention designs a semi-systolic array architecture. The horizontal direction is the parallel direction K, implemented using a broadcast method, and the vertical direction is the accumulation direction C, implemented using a systolic method. This architecture significantly reduces the read bandwidth of the weights while maintaining the vectorized storage and access of the input data along the C direction. Specifically, the input data INP requires a set of registers to organize the systolic data of the semi-systolic array. An input data vector is written diagonally to these registers, and the input data required by the semi-systolic array at the same time is read vertically. Both reading and writing are implemented using selectors. The weight data WGT needs to be vectorized offline at 1xnP, and can be directly read during the operation of the semi-systolic array.
[0037] Each computational unit in a semi-systolic array has two sets of weight registers (WReg): one for the current computation and one for loading the next computation. Each unit also has a partial sum register (AReg) for vertical accumulation. Notably, the computational units in the top row do not require partial sum input data; therefore, the bit width of the adders and partial sum registers in each row is directly determined by the input and weight bit widths. The partial sum register (AReg) of the bottom row of computational units is accumulated with the external partial sum register (ACC). The workflow of the semi-systolic array is as follows: first, weights are loaded into the computational unit's weight register (WReg); then, input data is loaded to calculate the systolic partial sum; finally, the systolic partial sum is updated in the external partial sum register stack. During computation, the next weight can be preloaded into another weight register to hide the weight loading delay.
[0038] Meanwhile, this invention adopts existing fused ALU architecture and fused GEMM-ALU architecture, such as the ALU operation fusion processing module suitable for neural networks with Chinese patent application number 2025112423467, which supports micro-operation instructions with multiple execution cycles. In addition to Load, Store and instructions with immediate values, other micro-operation instructions are divided into two forms: register hit and register miss. The former has the same execution cycle as Load, Store and instructions with immediate values, while the latter has a longer execution cycle to support loading parameters such as bias and quantization in multiple cycles.
[0039] like Figure 4 The diagram illustrates the neural network processor architecture with optimized storage bit width according to the present invention. Figure 2 The differences in the architecture are: 1) The GEMM module of the primitive semi-systolic array reduces the read bandwidth of the weight storage to the write bandwidth, while requiring multiple sets of partial and register units, with a minimum of max(nP, nR), to avoid idle multiplication-accumulation unit (MAC) units; 2) The ALU operation fusion processing module allows the post-processing operation sequence to read the partial and register units only once, and the bias parameters between different outputs to support multi-cycle bias parameter readings, reducing the bit width of the bias storage (BIAS) to the bus bit width. This module loads the bias parameters into the ALU internal registers and combines data flow scheduling to achieve spatial reuse of the bias. This collaborative design achieves significant optimization of the area efficiency of weight and bias storage while maintaining computational performance.
[0040] Finally, a dynamic adjustment strategy for the storage structure based on critical path timing analysis is introduced to ensure timing convergence and further simplify control. For example... Figure 5As shown. Specifically, the input is the original neural network processor architecture, and the output is the neural network processor architecture optimized by the storage system with area cost as the objective. When dedicated storage is single-ported and bit-width optimized, dedicated storage with the same data achieves maximum aggregation, resulting in a deep and narrow storage shape, which increases read and write access latency. Under the target process, dedicated storage is generated, and timing analysis is performed through logic synthesis. When the timing meets the requirements, the optimized architecture is directly output. When the increased latency due to the large local storage capacity leads to timing violations, it is split into multiple small-capacity storage blocks that can be accessed independently and in parallel, keeping the width unchanged and reducing the depth to shorten the read and write access latency. If timing violations still occur after one split, the splitting continues until the minimum capacity configuration that the target process can generate with that bit width is reached. If timing violations still occur under the minimum capacity configuration, an error is reported: Timing violations cannot be resolved by storage splitting; it is recommended to adjust the content outside the storage to ensure timing. If the timing is satisfactory, a further dedicated storage port setup and bit-width optimization are performed, with minor adjustments to the existing storage architecture: storage access that previously relied on hardware arbitration can be adjusted to a software-scheduled ping-pong access mode; bias and other parameter storage can utilize the additional bandwidth gained from the splitting process to shorten the number of cycles required to load data into the fused ALU registers. This strategy eliminates timing bottlenecks while improving storage access efficiency and simplifying hardware control logic.
[0041] Through the above embodiments, the technical solution of the present invention can significantly improve the area efficiency and access efficiency of the storage system while ensuring computing performance, and achieve synergistic optimization of bandwidth, area and power consumption.
[0042] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for collaborative optimization of storage architecture in a lightweight neural network processor, characterized in that, include: Step 1: Analyze the differences in bandwidth requirements and access patterns of on-chip data, and set differentiated single-port storage settings for the buffer storage of each data in the neural network processor; For input data and weight data, a dual-block single-port storage based on ping-pong buffer is used for alternating access; For partial and data sets, a distributed register cache is used; For bias parameters and output data, single-block single-port storage is used, and access contention is handled through a hardware arbitration mechanism; Step 2: The GEMM module of the neural network processor is set up based on a half-systolic array, and an ALU operation fusion processing module is used to reduce the data storage bandwidth and bit width; The semi-pulsating array is a two-dimensional processing array that broadcasts along the parallel direction K and pulsates along the accumulation direction C, used to reduce the read bandwidth of the weights while maintaining the vectorized storage and access of the input data along the C direction. Step 3: Optimize the storage structure by introducing a dynamic adjustment strategy based on time-series analysis, thereby outputting an optimized neural network processor; The dynamic adjustment strategy based on timing analysis is as follows: Timing checks are performed on the buffer storage generated under the target process through logic synthesis. When the timing meets the requirements, an optimized neural network processor architecture is directly output. When a timing violation occurs due to increased latency caused by large local storage capacity, a storage splitting operation is performed. This involves splitting the local storage into multiple small-capacity storage blocks that can be accessed independently and in parallel, keeping the width constant while reducing the depth to shorten read / write access latency. This storage splitting operation is repeated until the timing meets the requirements. Then, the original hardware arbitration-dependent storage access is adjusted to a software-scheduled ping-pong access mode. The additional bandwidth obtained after splitting the bias parameter storage is utilized to shorten the number of cycles required to load data into the fused ALU register.
2. The method as described in claim 1, characterized in that, The number of times the weight data is reused is less than or equal to the number of entries in the register.
3. The method as described in claim 1, characterized in that, The hardware arbitration mechanism for the output data is fixed to write-first.
4. The method as described in claim 1, characterized in that, Each computing unit of the two-dimensional processing array has two sets of weight registers, one set for the current calculation and the other set for preloading the next calculation; each computing unit also has a set of partial sum registers for accumulation along the vertical direction.
5. The method as described in claim 4, characterized in that, In the two-dimensional processing array, the first row of computing units does not require partial sum input data, so the bit width of the adder and partial sum register of each row of computing units is directly determined by the input and weight bit width; the partial sum register of the last row of computing units in the two-dimensional processing array is accumulated with the external partial sum register.
6. The method as described in claim 1, characterized in that, In step 2, the ALU operation fusion processing module is used for multi-cycle bias parameter reading, so that the bit width of the bias storage is reduced to the bus bit width.
7. The method as described in claim 1, characterized in that, During the repeated execution of storage splitting operations, if timing violations still occur after one split, the splitting continues until the minimum capacity configuration that the target process bit width can generate is reached. If timing violations still occur under the minimum capacity configuration, it means that timing violations cannot be resolved by storage splitting, and the content outside the storage needs to be adjusted to ensure timing.