Lightweight neural network processor storage architecture collaborative optimization method

Abstract

The invention relates to the technical field of neural network reasoning hardware, in particular to a lightweight neural network processor storage architecture collaborative optimization method, which comprises the following steps of: 1, analyzing the difference between bandwidth requirements and access modes of on-chip data, and carrying out differentiated single-port storage setting on buffer storage of each data in a neural network processor; 2, setting a GEMM module of a neural network processor based on a semi-systolic array, and reducing the storage bandwidth and bit width of data by adopting an ALU operation fusion processing module; and step 3, optimizing a storage structure by introducing a dynamic adjustment strategy based on time sequence analysis so as to output an optimized neural network processor. According to the method, the area efficiency and the access efficiency of the storage system are remarkably improved while the computing performance is guaranteed, and collaborative optimization of the bandwidth, the area and the power consumption is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural network inference hardware, and particularly relates to a lightweight neural network processor storage architecture cooperative optimization method. BACKGROUND

[0002] With the rapid development of Internet of Things and edge intelligence, Tiny Machine Learning (TinyML) has become a key paradigm to realize real-time intelligent decision-making in resource-constrained environments. The goal is to directly complete neural network inference on microcontrollers (MCU) with only a few hundred KB of storage space and milliwatt-level power consumption, thereby ensuring data privacy, reducing cloud dependence, and achieving low-latency response. However, the limited computing power and storage resources of traditional MCUs cannot meet the high demand of modern deep neural networks for computing power and storage. This contradiction directly promotes the development of microcontroller-level neural network processors (μNPU). Through a customized hardware architecture, μNPU can maintain low power consumption while achieving orders of magnitude improvement in inference efficiency compared to general-purpose MCUs. Among them, the storage architecture is the key to determining the area cost and computing efficiency: on the one hand, on-chip storage is frequently accessed, and the power consumption of a single read-write operation is significantly higher than that of the computing unit; on the other hand, to reduce access to external DRAM, μNPU usually integrates a large on-chip storage to reside high-frequency data, resulting in a storage area ratio often exceeding 60%. Considering that MCUs are extremely sensitive to performance, power consumption, and area cost indicators, it is crucial to conduct cooperative optimization research on the storage architecture of μNPU.

[0003] The development of the memory architecture of neural network processors presents clear stage characteristics. Early schemes, such as designs based on Cadence DSP or Cambrian processors, generally use a shared memory architecture. This architecture places various types of data such as input, weight, and output in a unified addressable memory space, which helps to improve the overall utilization of the memory space, but requires the design of a complex multi-port read-write arbitration unit to handle the parallel access conflicts generated by multiple computing units, which limits the further improvement of system energy efficiency. In order to pursue the ultimate computing efficiency and energy efficiency, modern designs have shifted to specialized memory architectures that are deeply customized for specific data flow patterns, and two major technical paths have thus been formed. The Google TPU, as a representative of the systolic array architecture, achieves deep reuse of weights by making data (such as weights and activation values) flow and compute among processing elements (PEs) at a fixed rhythm, thereby greatly reducing the dependence on high-bandwidth external storage. However, this efficiency is built on strict spatiotemporal synchronization control of data flow, which requires a large number of registers to construct data systolic paths and has a high computing pipeline fill delay. Another technical path is represented by NVDLA and VTA, which use a combination of multicast and addition trees for computing. This architecture naturally fits convolution operations, its programming model is relatively simple, and it does not require the introduction of IM2COL data restructuring operations that would cause memory expansion, but its significant drawback is the high demand for storage bandwidth, especially for weight data, which requires the storage system to provide dense data supply to the highly parallel processing unit array within a single cycle.

[0004] However, the systolic array of TPU and the multicast-addition tree architecture of NVDLA / VTA are both designed for cloud-side or high-performance edge computing scenarios with huge computing power requirements (usually not less than 256 MAC) and relatively abundant storage resources (often up to hundreds of KB). The core optimization goal of such designs is to pursue peak computing power and overall energy efficiency, rather than extreme area cost efficiency, which is fundamentally in conflict with the strict constraints on area and power consumption in the μNPU scenario, and cannot be adapted by simple direct micro-transplantation. Taking a typical lightweight NPU for the terminal market as an example, its typical configuration is 32 to 256 MAC computing power and 18 to 50 KB internal storage. If the fine-grained, multi-storage block design adopted by NVDLA / VTA to meet high bandwidth is followed, it will inevitably lead to excessive number of storage blocks and too small physical capacity of single block. Since the fixed area overhead of SRAM peripheral circuitry increases dramatically with the decrease of capacity, the effective bit area efficiency of the storage unit is significantly reduced, and the total area cost is out of control. If the systolic array idea of TPU is used to integrate storage to reduce bandwidth, although the number of storage blocks can be reduced, the large number of registers required to build the data systolic network will introduce new significant area overhead, which is also difficult to bear under the strict area budget of μNPU.

[0005] In summary, existing architectures designed for high-performance scenarios cannot directly meet the stringent area and cost requirements of μNPU. Summary of the Invention

[0006] To address the aforementioned technical problems in the existing technology, this invention proposes a lightweight neural network processor memory architecture collaborative optimization method, the specific technical solution of which is as follows: A method for collaborative optimization of the storage architecture of a lightweight neural network processor includes: Step 1: Analyze the differences in bandwidth requirements and access patterns of on-chip data, and configure differentiated single-port storage for buffer storage of each data in the neural network processor; Step 2: The GEMM module of the neural network processor is set up based on a half-pulse array, and an ALU operation fusion processing module is used to reduce the data storage bandwidth and bit width; Step 3: Optimize the storage structure by introducing a dynamic adjustment strategy based on time series analysis, thereby outputting an optimized neural network processor.

[0007] Furthermore, in step 1, for high-bandwidth, access-intensive input data and weight data, a ping-pong buffer-based dual-block single-port storage is used for alternating access; for high-bandwidth, access-intensive partial data, a distributed register cache is used; for low-bandwidth, access-sparse bias parameters and output data, a single-block single-port storage is used, and access contention is handled through a hardware arbitration mechanism.

[0008] Furthermore, the number of times the weight data is reused is less than or equal to the number of entries in the register.

[0009] Furthermore, the hardware arbitration mechanism for the output data is fixed to write-first.

[0010] Furthermore, in step 2, the semi-pulsating array is a two-dimensional processing array that broadcasts along the parallel direction K and pulsates along the accumulation direction C, used to reduce the read bandwidth of the weights while maintaining the vectorized storage and access of the input data along the C direction.

[0011] Furthermore, each computing unit of the two-dimensional processing array has two sets of weight registers, one set for the current calculation and the other set for preloading the next calculation; each computing unit also has a set of partial sum registers for accumulation along the vertical direction.

[0012] Furthermore, since the first row of computing units in the two-dimensional processing array does not require partial sum input data, the bit width of the adder and partial sum register of each row of computing units is directly determined by the input and weight bit width; the partial sum register of the last row of computing units in the two-dimensional processing array is accumulated with the external partial sum register.

[0013] Furthermore, in step 2, the ALU operation fusion processing module is used for multi-cycle bias parameter reading, so that the bit width of the bias storage is reduced to the bus bit width.

[0014] Furthermore, in step 3, the dynamic adjustment strategy based on timing analysis specifically involves: performing timing checks on the buffer storage generated under the target process through logic synthesis; when the timing meets the requirements, directly outputting the optimized neural network processor architecture; when the large local storage capacity leads to increased latency and timing violations, performing a storage splitting operation; splitting the local storage into multiple small-capacity storage blocks that can be accessed independently and in parallel, keeping the width unchanged and reducing the depth to shorten read and write access latency; repeating the storage splitting operation until the timing meets the requirements; then adjusting the original hardware arbitration-dependent storage access to a software-scheduled ping-pong access mode; utilizing the additional bandwidth obtained after splitting the bias parameter storage to shorten the number of cycles required to load data into the fused ALU register.

[0015] Furthermore, during the repeated execution of the storage splitting operation, if there is still a timing violation after one split, the splitting continues until the minimum capacity configuration that the target process can generate for this bit width is reached. If there is still a timing violation under the minimum capacity configuration, an error is reported: the timing violation cannot be resolved by storage splitting, and it is recommended to adjust the content outside the storage to ensure timing.

[0016] Beneficial effects: While ensuring computing performance, this invention significantly improves the area efficiency and access efficiency of the storage system, and achieves synergistic optimization of bandwidth, area and power consumption. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a typical neural network processor architecture based on dedicated storage in this embodiment; Figure 2 This is a schematic diagram of the neural network processor architecture based on differentiated single-port storage in this embodiment; Figure 3 This is an architecture diagram of a semi-pulsating array according to this embodiment; Figure 4 This is a diagram of the neural network processor architecture after memory bit width co-optimization in this embodiment; Figure 5 This is a flowchart of the dynamic adjustment strategy for the storage structure in this embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0019] like Figure 1The diagram illustrates a typical neural network processor architecture based on dedicated storage. The Load module is responsible for moving input and weight data from external Dynamic Random Access Memory (DRAM) to the internal input buffer INP, weight buffer WGT, and bias buffer BIAS. The GEMM module performs general matrix multiplication calculations, assuming its PE array size is nP x nR, where nP is the number of parallel operations and nR is the number of accumulations. The nP dimension uses multicast, and the nR dimension uses an addition tree. The data types for input, weights, partial sums, biases, and outputs are INT8, INT8, INT32, INT32, and INT8, respectively. Without register data multiplexing, the GEMM module needs to read 1 x nR x INT8 of input data, nP x nR x INT8 of weight data, 1 x nP x INT32 of partial sum input data, and write 1 x nP x INT32 of partial sum output data per cycle. Because the GEMM and ALU modules access the partial sum and buffer ACC simultaneously, a two-read, two-write access pattern occurs, resulting in high storage interface requirements and significant hardware overhead for direct implementation. Therefore, a serial execution method for GEMM and ALU, such as VTA, is typically chosen. The ALU module is responsible for performing post-processing operations such as biasing and quantization. Each cycle, it needs to read 1xnPxINT32 of partial sum and input data, at most 1xnPxINT32 of bias or quantization parameters, and write 1xnPxINT32 of partial sum and output data. Since these are typically in-place processing operations, the partial sum and read / write operations become the performance bottleneck for the ALU. The final output of the ALU is written to the output buffer OUT. After the entire output block has been calculated, the Store module is responsible for writing the output block back to the external DRAM.

[0020] This invention provides a collaborative optimization method for storage architecture targeting lightweight neural network processors, with area cost as the optimization objective, as detailed below: The peak computing power of the neural network processor depends on the number of MAC (Multiply-Accumulate) units in the GEMM (Generative Multiplication and Accumulation) module. It needs to acquire input data, weight data, and partial sum data each cycle to maintain sufficient MAC utilization. Access to this data is intensive and bandwidth-intensive. After the GEMM completes all accumulation operations, the ALU begins the corresponding post-processing operations. It needs to acquire partial sum data (ACC) and bias (BIAS) each cycle, and after processing, write the final result to the output buffer (OUT). Therefore, reading BIAS and writing OUT are sparse, triggered only after the partial sum data accumulation is complete.

[0021] Therefore, this invention designs a differentiated single-port storage implementation method based on data access characteristics. First, by analyzing the differences in bandwidth requirements and access modes of on-chip data, the more expensive pseudo-dual-port storage is replaced with various single-port storage methods that are more area-efficient.

[0022] like Figure 2 The diagram illustrates the application of the differentiated single-port storage implementation method of this invention. Specifically, for high-bandwidth, access-intensive input data and weighted data, a dual-block single-port storage based on ping-pong buffers is adopted. Alternating access is achieved through software scheduling; for example, the Load module writes to INP0, and the GEMM module reads from INP1. At the hardware level, INP0 and INP1 are the low and high address spaces of the local input storage, respectively. At the software level, the two input blocks executed in ping-pong are mapped to INP0 and INP1 respectively through address mapping. GEMM also accesses the required input block through address. That is, the selectors at the Load and GEMM ends of the input buffers INP0 and INP1 determine which block to access based on address, and software avoids conflicts from accessing the same storage array.

[0023] For high-bandwidth, intensively accessed partial sums and data, the demand for parallel read / write operations is high. Leveraging the limited number of accumulations in TinyML scenarios, the centralized partial sum buffer can be directly replaced with distributed registers. As long as the accessed register entries are different, multi-path parallel read / write access can be supported. However, storing partial sums and data in registers requires the coordination of multiplexed output data streams, and the number of times the weights are reused must be less than or equal to the number of register entries to prevent partial sum data overflow during accumulation. The data stream can be implemented through software scheduling.

[0024] For low-bandwidth, sparsely accessed bias parameters and output data, single-block, single-port storage is used, and access contention is handled through a hardware arbitration mechanism. Bias parameters are only read during post-processing operations on the accumulated portion of the data. Furthermore, bias parameters are reused spatially; when segmenting data along the spatial dimension, the bias parameters only need to be loaded once, resulting in sparse reads and writes. The hardware arbitration mechanism for bias parameters can employ simple polling, or, considering multiple read requests from the Load module, a write-first approach can reduce congestion on the bus read channel. Quantization parameters are processed similarly to bias parameters and are also stored in the bias buffer BIAS. Similarly, after the ALU completes post-processing operations, it writes the output results sporadically to the output buffer OUT. Once the entire output block is complete, the Store performs continuous read operations, writing the output to external storage. In the entire neural network processor architecture, only the Store module issues write requests to the bus; therefore, a write-first hardware arbitration mechanism for OUT is recommended to avoid impacting the Store module's performance.

[0025] This method achieves overall simplification and area optimization of the storage system by finely differentiating the storage architecture, while ensuring the performance of critical data paths.

[0026] Secondly, a storage bit width co-optimization method based on a semi-pulsating array and a fused ALU computing architecture is designed to reduce the bandwidth and bit width requirements of critical storage, thereby optimizing the storage area.

[0027] Among them, such as Figure 3 As shown, this invention designs a semi-systolic array architecture. The horizontal direction is the parallel direction K, implemented using a broadcast method, and the vertical direction is the accumulation direction C, implemented using a systolic method. This architecture significantly reduces the read bandwidth of the weights while maintaining the vectorized storage and access of the input data along the C direction. Specifically, the input data INP requires a set of registers to organize the systolic data of the semi-systolic array. An input data vector is written diagonally to these registers, and the input data required by the semi-systolic array at the same time is read vertically. Both reading and writing are implemented using selectors. The weight data WGT needs to be vectorized offline at 1xnP, and can be directly read during the operation of the semi-systolic array.

[0028] Each computational unit in a semi-systolic array has two sets of weight registers (WReg): one for the current computation and one for loading the next computation. Each unit also has a partial sum register (AReg) for vertical accumulation. Notably, the computational units in the top row do not require partial sum input data; therefore, the bit width of the adders and partial sum registers in each row is directly determined by the input and weight bit widths. The partial sum register (AReg) of the bottom row of computational units is accumulated with the external partial sum register (ACC). The workflow of the semi-systolic array is as follows: first, weights are loaded into the computational unit's weight register (WReg); then, input data is loaded to calculate the systolic partial sum; finally, the systolic partial sum is updated in the external partial sum register stack. During computation, the next weight can be preloaded into another weight register to hide the weight loading delay.

[0029] Meanwhile, this invention adopts existing fused ALU architecture and fused GEMM-ALU architecture, such as the ALU operation fusion processing module suitable for neural networks with Chinese patent application number 2025112423467, which supports micro-operation instructions with multiple execution cycles. In addition to Load, Store and instructions with immediate values, other micro-operation instructions are divided into two forms: register hit and register miss. The former has the same execution cycle as Load, Store and instructions with immediate values, while the latter has a longer execution cycle to support loading parameters such as bias and quantization in multiple cycles.

[0030] like Figure 4The diagram illustrates the neural network processor architecture with optimized storage bit width according to the present invention. Figure 2 The differences in the architecture are: 1) The GEMM module of the primitive semi-systolic array reduces the read bandwidth of the weight storage to the write bandwidth, while requiring multiple sets of partial and register units, with a minimum of max(nP, nR), to avoid idle multiplication-accumulation unit (MAC) units; 2) The ALU operation fusion processing module allows the post-processing operation sequence to read the partial and register units only once, and the bias parameters between different outputs to support multi-cycle bias parameter readings, reducing the bit width of the bias storage (BIAS) to the bus bit width. This module loads the bias parameters into the ALU internal registers and combines data flow scheduling to achieve spatial reuse of the bias. This collaborative design achieves significant optimization of the area efficiency of weight and bias storage while maintaining computational performance.

[0031] Finally, a dynamic adjustment strategy for the storage structure based on critical path timing analysis is introduced to ensure timing convergence and further simplify control. For example... Figure 5 As shown. Specifically, the input is the original neural network processor architecture, and the output is the neural network processor architecture optimized by the storage system with area cost as the objective. When dedicated storage is single-ported and bit-width optimized, dedicated storage with the same data achieves maximum aggregation, resulting in a deep and narrow storage shape, which increases read and write access latency. Under the target process, dedicated storage is generated, and timing analysis is performed through logic synthesis. When the timing meets the requirements, the optimized architecture is directly output. When the increased latency due to the large local storage capacity leads to timing violations, it is split into multiple small-capacity storage blocks that can be accessed independently and in parallel, keeping the width unchanged and reducing the depth to shorten the read and write access latency. If timing violations still occur after one split, the splitting continues until the minimum capacity configuration that the target process can generate with that bit width is reached. If timing violations still occur under the minimum capacity configuration, an error is reported: Timing violations cannot be resolved by storage splitting; it is recommended to adjust the content outside the storage to ensure timing. If the timing is satisfactory, a further dedicated storage port setup and bit-width optimization are performed, with minor adjustments to the existing storage architecture: storage access that previously relied on hardware arbitration can be adjusted to a software-scheduled ping-pong access mode; bias and other parameter storage can utilize the additional bandwidth gained from the splitting process to shorten the number of cycles required to load data into the fused ALU registers. This strategy eliminates timing bottlenecks while improving storage access efficiency and simplifying hardware control logic.

[0032] Through the above embodiments, the technical solution of the present invention can significantly improve the area efficiency and access efficiency of the storage system while ensuring computing performance, and achieve synergistic optimization of bandwidth, area and power consumption.

[0033] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for collaborative optimization of storage architecture in a lightweight neural network processor, characterized in that, include: Step 1: Analyze the differences in bandwidth requirements and access patterns of on-chip data, and set differentiated single-port storage settings for the buffer storage of each data in the neural network processor; Step 2: The GEMM module of the neural network processor is set up based on a half-systolic array, and an ALU operation fusion processing module is used to reduce the data storage bandwidth and bit width; Step 3: Optimize the storage structure by introducing a dynamic adjustment strategy based on time series analysis, thereby outputting an optimized neural network processor.

2. The method as described in claim 1, characterized in that, In step 1, For high-bandwidth, densely accessed input and weighted data, a dual-block single-port storage based on ping-pong buffer is used for alternating access. For high-bandwidth, intensively accessed data, a distributed register cache is used. For low-bandwidth, sparsely accessed bias parameters and output data, single-block single-port storage is used, and access contention is handled through a hardware arbitration mechanism.

3. The method as described in claim 2, characterized in that, The number of times the weight data is reused is less than or equal to the number of entries in the register.

4. The method as described in claim 2, characterized in that, The hardware arbitration mechanism for the output data is fixed to write-first.

5. The method as described in claim 1, characterized in that, In step 2, the semi-pulsating array is a two-dimensional processing array that broadcasts along the parallel direction K and pulsates along the accumulation direction C, used to reduce the read bandwidth of the weights while maintaining the vectorized storage and access of the input data along the C direction.

6. The method as described in claim 5, characterized in that, Each computing unit of the two-dimensional processing array has two sets of weight registers, one set for the current calculation and the other set for preloading the next calculation; each computing unit also has a set of partial sum registers for accumulation along the vertical direction.

7. The method as described in claim 6, characterized in that, In the two-dimensional processing array, the first row of computing units does not require partial sum input data, so the bit width of the adder and partial sum register of each row of computing units is directly determined by the input and weight bit width; the partial sum register of the last row of computing units in the two-dimensional processing array is accumulated with the external partial sum register.

8. The method as described in claim 1, characterized in that, In step 2, the ALU operation fusion processing module is used for multi-cycle bias parameter reading, so that the bit width of the bias storage is reduced to the bus bit width.

9. The method as described in claim 1, characterized in that, In step 3, the dynamic adjustment strategy based on timing analysis is specifically as follows: Timing checks are performed on the buffer storage generated under the target process through logic synthesis. When the timing meets the requirements, the optimized neural network processor architecture is directly output. When a timing violation occurs due to increased latency caused by large local storage capacity, a storage splitting operation is performed. This involves splitting the local storage into multiple small-capacity storage blocks that can be accessed independently and in parallel, keeping the width constant while reducing the depth to shorten read / write access latency. The storage splitting operation is repeated until the timing meets the requirements. Then, the original hardware arbitration-dependent storage access is adjusted to a software-scheduled ping-pong access mode. The additional bandwidth obtained after splitting the bias parameter storage is utilized to shorten the number of cycles required to load data into the fused ALU register.

10. The method as described in claim 9, characterized in that, During the repeated execution of storage splitting operations, if timing violations still occur after one split, the splitting continues until the minimum capacity configuration that the target process can generate for this bit width is reached. If timing violations still occur under the minimum capacity configuration, an error will be reported: Timing violations cannot be resolved by storage splitting. It is recommended to adjust the content outside the storage to ensure timing.

Images