A double-layer cache end-to-end neural network acceleration system based on in-memory computing of DRAM

Abstract

This invention discloses a dual-layer cache end-to-end neural network acceleration system based on DRAM in-memory computing, relating to DRAM technology and addressing the problems in existing technologies. The system includes a data and command scheduling module for end-to-end task decomposition into commands, execution timing, and synchronization management, coordinating command broadcasting and multi-bank data interaction; a global cache module for storing intermediate feature maps and partial output results; a local cache module, located at each bank level, for local data reuse; a near-memory computing core module, an embedded MAC core module within the DRAM bank, for performing computational operations; and a channel-level computing core module, located at the channel level, for cross-bank aggregation, post-processing, and output write-back operations. The advantages include a dual-layer cache end-to-end architecture, enabling local execution of various operators such as convolution, normalization, activation, and pooling on the storage side, reducing host interaction overhead, and improving overall energy efficiency and computing power utilization.

Description

Technical Field

[0001] This invention relates to the field of DRAM technology, and in particular to a dual-layer cache end-to-end neural network acceleration system based on DRAM in-memory computing. Background Technology

[0002] While DRAM in-memory computing technology has become one of the mainstream in-memory computing solutions due to its advantages of high bandwidth, mature manufacturing processes, and good compatibility with existing memory architectures, the current architecture still has several limitations. Although it has the ability to perform multiply-accumulate operations in memory arrays, it is difficult to achieve complete end-to-end deep neural network inference due to limitations in computational completeness, buffer hierarchy and data reuse, and architectural flexibility, facing significant challenges in terms of energy efficiency and resource utilization.

[0003] Regarding computational completeness, existing DRAM in-memory computing typically only supports multiply-accumulate operations, with other operators still requiring host processing. Frequent data interaction between the host and in-memory computing units leads to a significant increase in communication latency and power consumption, creating a "computation-communication imbalance." In terms of buffer hierarchy and data reuse, typical DRAM in-memory computing architectures such as SK Hynix GDDR6-AiM have a limited global buffer capacity of only about 2KB per channel, making it difficult to simultaneously cache intermediate feature map data required for multi-layer convolutional inference. Intermediate results need to be frequently migrated between different banks, resulting in a significant increase in access latency and power consumption, while also limiting reuse efficiency and leading to a reduction in overall energy efficiency. Regarding architectural flexibility, existing GDDR6-AiM systems have a shallow architecture hierarchy, with highly coupled command and data flow paths, lacking flexible hierarchical scheduling and data flow organization mechanisms. This fixed structure is difficult to adapt to the multi-layered and complex operator combinations with varying memory access patterns in mainstream deep networks, resulting in communication bottlenecks and resource utilization imbalances in end-to-end inference tasks. Regarding system software and hardware collaboration, existing DRAM in-memory computing technologies largely rely on fixed operator mappings and static command stream generation methods, lacking dynamic scheduling mechanisms for different network structures and cache configurations. When cache resources are limited, the compiler and control end cannot flexibly adjust data reuse strategies based on inter-layer feature mappings, resulting in decreased computing power utilization and energy efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a dual-layer cache end-to-end neural network acceleration system based on DRAM in-memory computing, so as to solve the problems existing in the prior art.

[0005] The dual-layer cache end-to-end neural network acceleration system based on DRAM in-memory computing described in this invention includes:

[0006] The data and command scheduling module is used for end-to-end task decomposition into commands, execution timing and synchronization management, command broadcast coordination and multi-bank data interaction;

[0007] The global cache module is used to store intermediate feature maps and some output results; it is a channel-level shared cache.

[0008] The local caching module, located at each Bank level, is used for local data reuse: temporarily caching local convolutional kernels and input features;

[0009] The near-memory computing core module is a MAC core module embedded inside the DRAM bank, used to perform computational operations;

[0010] The Channel-level computing core module, located at the Channel end, is used for cross-bank aggregation, post-processing, and output write-back operations.

[0011] The dual-layer cache end-to-end neural network acceleration system based on DRAM in-memory computing described in this invention has the advantage of having a dual-layer cache end-to-end architecture compared to existing single-layer cache DRAM in-memory computing architectures (such as GDDR6-AiM). It enables the local execution of multiple operators such as convolution, normalization, activation, and pooling on the storage side, reducing host interaction overhead and improving overall energy efficiency and computing power utilization. Meanwhile, a feasible solution is proposed to address the three major challenges of DRAM in-memory computing technology: Regarding computational completeness, an operator fusion and heterogeneous collaboration mechanism is introduced on the storage side, enabling the in-memory computing core to directly support operator operations such as convolution, batch normalization, activation, pooling, and residual addition, significantly improving computational completeness; Regarding buffer hierarchy and data reuse, a local cache module at the Bank level and a global cache module at the Channel level form a two-level data reuse mechanism, reducing redundant memory accesses and cross-Bank data migration, significantly improving bandwidth utilization and overall energy efficiency; Regarding architectural flexibility, the two-layer buffer hierarchy brings flexible data flow organization, custom in-memory computing commands form an extensible instruction set, and the simulation framework Ramulator2 achieves modular integration, possessing reconfigurability and engineering compatibility at the architectural level. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the structure of the dual-layer cache end-to-end neural network acceleration system described in this invention.

[0013] Figure 2 This is a schematic diagram of the near-memory computing core module described in this invention.

[0014] Figure 3 This is a schematic diagram illustrating the interaction between the Channel-level computing core module and adjacent modules as described in this invention.

[0015] Figure 4 This is a schematic diagram of the custom instruction execution path of the dual-layer cache end-to-end acceleration system described in this invention.

[0016] Figure 5 This is a schematic diagram of the execution flow of custom commands in the dual-layer cache end-to-end neural network acceleration system described in this invention.

[0017] Figure 6 This is a schematic diagram of the computation process of the dual-layer cache end-to-end neural network acceleration system described in this invention. Detailed Implementation

[0018] The dual-layer cache end-to-end neural network acceleration system based on DRAM in-memory computing described in this invention is as follows: Figure 1 The system includes: a data and command scheduling module, which receives convolution / activation / pooling / normalization task requests from the host, generates corresponding DRAM in-memory computation command sequences based on the task type, and schedules each module to execute, completing operator parsing, command decoding, timing and synchronization management, and task allocation; a global cache module, located at the Channel level, which stores intermediate feature and weight data shared across banks, supports broadcast reuse mechanisms, and coordinates data transmission and result aggregation between different banks; a local cache module, located at the Bank level, which stores and retrieves local convolution weights and input features, using commands such as PIM_BK2LBUF and PIM_LBUF2BK to achieve local loading, reuse, and write-back, reducing repeated memory accesses; and a near-memory computation core module, which performs input multiplication by weight operations within the Bank and supports BN / ReLU fusion instructions to implement convolutional layer and post-processing layer computations. The Channel-level computing core module is used for cross-bank result aggregation and non-convolutional operator execution. It can work in conjunction with the near-memory computing core module according to scheduling commands to achieve feature output and inter-layer data flow.

[0019] The near-memory computing core module is as follows: Figure 2 As shown, the system includes a multiply-accumulate array, adders, registers, batch normalization (BN), and ReLU activation units. Each local cache is connected to its corresponding near-memory computation core module via an independent read / write interface to support parallel data access and computation. The capacity of the local cache module is configurable. The operation of the near-memory computation core module is as follows: the input activation data is broadcast from the global cache module to the local cache modules of each Bank, and the local cache modules load and reuse the convolution weights. Subsequently, the multiply-accumulate array performs multiplication and addition operations on the input and weights, and the results are summarized by an accumulator and written to a register. The data output from the register is sequentially processed by batch normalization and activation to form an intermediate result. This result can be written back to the local cache module or transmitted to the upper-level module via the global cache module, enabling the continuous execution of operators such as convolution, normalization, and activation.

[0020] The Channel-level computing core module, such as Figure 3 As shown, the system includes residual addition and activation units, and pooling units. During operation, it interacts with the data and command scheduling module, the host interface, and the storage bank interface. The specific workflow is as follows: At the start of computation, the host loads input data or writes back inference results through the interface. The data and command scheduling module receives tasks from the host, parses and allocates commands, and maps feature data to the global cache module. The global cache module is used for cross-bank broadcasting and data reuse. The near-memory computation core modules in each bank perform convolution, normalization, and other calculations, then converge the results to the channel-level computation core module. The channel-level computation core module performs residual addition, activation, and pooling operations, and the results are updated by the global cache module or transmitted back to the host. The channel-level computation core module implements multi-channel collaborative computation and storage-side inference processes through bidirectional data flow.

[0021] To facilitate a clearer understanding of the custom instruction set of the dual-layer cache end-to-end neural network acceleration system described in this invention, the functions of all in-memory computation-type Compute instructions and data-moving-type Data Move instructions supported by this invention are further explained in conjunction with the following table.

[0022]

[0023] In the table, this invention classifies custom instructions into three categories based on different types of operators and different positions of operation execution:

[0024] The first category consists of near-memory computation instructions, used to perform fusion operator operations such as convolution, batch normalization, and activation within the near-memory computation core module of the Bank. These include: PIM_CONV_BN_RELU: performs a three-operator fusion operation of convolution, batch normalization, and ReLU activation; and PIM_CONV_BN: performs a two-operator fusion operation of convolution and batch normalization.

[0025] The second category consists of Channel-level computation instructions, used to perform cross-bank data aggregation or post-processing operations within the Channel-level computation core module. These include: PIM_POOL: performs pooling operations; PIM_ADD_RELU: performs a fusion operation of residual addition and ReLU activation.

[0026] The third category: Data migration instructions, used to coordinate data migration and reuse between the global cache module, local cache module, and storage bank. These include: PIM_BK2GBUF: Write-back of data from the bank to the global cache module; PIM_BK2LBUF: Load of data from the bank to the local cache module; PIM_GBUF2BK: Distribute of data from the global cache module to the bank; PIM_LBUF2BK: Return of data from the local cache module to the bank.

[0027] When performing in-memory computation tasks, the data and command scheduling module first determines the execution path of the corresponding operator based on the instruction set shown in the table. For convolutional operators, the scheduling module triggers the PIM_BK2LBUF instruction to load the convolution kernel into the local cache module, and triggers the PIM_CONV_BN or PIM_CONV_BN_RELU instruction on the PIMcore side to complete near-memory convolution, batch normalization, and activation logic. When the network layer contains residual branches, cross-bank aggregation, or pooling operations, the scheduling module selects the PIM_POOL or PIM_ADD_RELU instruction on the Channel side to complete the corresponding operation. If intermediate results need to be moved between different storage levels, the PIM_BK2GBUF, PIM_GBUF2BK, PIM_BK2LBUF, or PIM_LBUF2BK instructions are scheduled according to the data flow to achieve global sharing, local reuse, and cross-layer transfer of intermediate features. The above instructions, through unified semantics and composable execution, enable the dual-layer cache structure of the present invention to maintain a continuous and reconfigurable data flow path between different network operators, thereby achieving complete storage-side inference execution.

[0028] By combining the above instruction set, this invention can dynamically construct an end-to-end data flow path between the near-memory computing core module and the Channel-level computing core module according to the operator types and data reuse characteristics of different network layers, achieving efficient parallel execution of various operators such as convolution, normalization, activation, residual addition, and pooling on the memory side. The above instruction set also serves as a basis for subsequent... Figure 4 , Figure 5 The custom command execution flow shown provides a unified semantic basis.

[0029] Figure 4 This diagram illustrates the custom instruction execution path of the dual-layer cache end-to-end neural network acceleration system described in this invention. It further explains the data flow and execution relationships of various custom instructions within the global cache module, local cache module, storage bank, and different computational core modules. When executing different types of operators, the data and command scheduling module triggers corresponding data movement and computation instructions based on the defined instruction semantics, thereby constructing a complete and continuous storage-side execution path.

[0030] The PIM_BK2GBUF and PIM_GBUF2BK instructions are used to write back and distribute data between the storage bank and the global cache module, respectively, supporting the sharing and global reuse of input feature maps across banks. The PIM_BK2LBUF and PIM_LBUF2BK instructions are used to load convolution weights and intermediate results between the storage bank and the local cache module, enabling the local cache module to provide resident input data and weights to the near-memory computation core module. After receiving the PIM_BK2LBUF instruction, the local cache module can load the corresponding convolution kernel into its local cache, which can then be used by the near-memory computation core module to perform convolution, batch normalization, and activation fusion operations by executing computational instructions such as PIM_CONV_BN or PIM_CONV_BN_RELU.

[0031] When executing non-convolutional operators such as cross-bank aggregation or pooling, the data and command scheduling module further triggers the PIM_POOL or PIM_ADD_RELU instruction, which is then used by the Channel-level computation core module to perform the corresponding computation based on shared intermediate data from the global cache module. The computation results can be written back to the global cache module via PIM_BK2GBUF or other data transfer instructions, according to the scheduling strategy, for continuous execution by the next layer of operators. This invention... Figure 4 The instruction execution path shown enables collaborative work between the near-memory computing core module and the Channel-level computing core module, allowing operators such as convolution, normalization, activation, residual addition, and pooling to form a reconfigurable end-to-end execution flow on the storage side.

[0032] The end-to-end custom command execution flow is as follows: Figure 5 As shown, the system first receives in-memory computation commands from the data and command scheduling module, classifying them into either computation (Compute) or data movement (Data Move) commands. When a command is a computation command, the scheduling module determines the operator type: if it's a convolution operation, it executes fusion computations such as PIM_CONV_BN or PIM_CONV_BN_RELU within the near-memory computation core module; if it's a non-convolution operator, it executes PIM_POOL or PIM_ADD_RELU instructions in the Channel-level computation core module. When a command is a data movement command, it triggers corresponding data transfer commands based on the data flow direction, including data read / write and multiplexing between the Bank-global cache module and the Bank-local cache module. After completing all commands, the system proceeds to the next round of execution, achieving coordinated control and automatic pipelined processing of computation and data transmission.

[0033] In another specific embodiment, to facilitate a clearer understanding by those skilled in the art of the operation mode of the dual-layer cache end-to-end neural network acceleration system under different storage configurations, a system operation mode with configurable storage levels and in-memory computing processes is further provided. This embodiment... Figures 1 to 5 Based on the system architecture shown, adjustable cache level parameters are set to adapt to the feature mapping size and weight reuse requirements of different neural network layers. The capacity of the global cache module can be selected from 0KB to 8KB to handle the broadcasting, sharing, and cross-bank reuse of input data and intermediate features at the Channel level. The capacity of the local cache module can be configured from 0B to 256B to cache local convolutional kernels, input features, and some intermediate results within a Bank, reducing repeated memory accesses and improving the local data reuse efficiency of the near-memory computing core module. This embodiment combines the capacities of the two cache modules. To represent different cache combination methods, this embodiment uses the configuration representation method GmKB_LnB, where GmKB represents the global cache module capacity of mKB and LnB represents the local cache module capacity of nB. This can form G2KB_L0B (i.e., global cache module capacity 2KB, local cache module capacity 0B), G2KB_L64B, G2KB_L128B, G2KB_L256B, G... The system features multiple operating modes, including 4KB_L0B, G4KB_L64B, G4KB_L128B, G4KB_L256B, G6KB_L0B, G6KB_L64B, G6KB_L128B, G6KB_L256B, G8KB_L0B, G8KB_L64B, G8KB_L128B, and G8KB_L256B, enabling it to flexibly organize data flow and adaptively reuse local data in network structures of varying depths.

[0034] In this embodiment, an end-to-end deep neural network inference task is used as a typical application, with convolutional layers, batch normalization layers, activation layers, and residual addition layers as the main operator types. During operation, the data and command scheduling module dynamically constructs an adapted data path based on the operator type, input feature map size, and buffer configuration of the current network layer. According to the operator type and data reuse requirements, the scheduling module selects to schedule according to the appropriate in-memory computation command. For example, when intermediate data needs to be reused globally, the intermediate data is loaded into the global cache module, which triggers cross-bank broadcasting. When data needs to be transferred to the local cache module for further computation such as convolution multiplication and addition or batch normalization, the scheduling module triggers instructions such as PIM_BK2LBUF to load weights into the local cache module for repeated access by the near-memory computation core module. During the convolutional layer execution phase, the near-memory computation core module performs convolution multiplication and addition, batch normalization, and activation logic based on the weights and input features provided by the local cache module. It selectively writes intermediate results back to the local cache module or pushes them to the global cache module according to the scheduling strategy, enabling immediate reuse of local computation results in subsequent layers. In network structures involving cross-bank aggregation or residual paths, the channel-level computation core module performs addition aggregation, pooling, and necessary post-processing operations according to commands issued by the scheduling module. It then writes the processed output data back to the global cache module or returns it to the host-side interface, forming a continuous inter-layer data stream.

[0035] Through the aforementioned configurable caching structure, this embodiment can select the appropriate combination mode of Channel-level computing core modules and near-memory computing core modules according to operator type and data reuse requirements. Furthermore, depending on the experimental objective, different sizes of global cache modules and local cache modules can be selected and matched to maintain high data reuse efficiency and near-memory computing efficiency under various neural network operator requirements, reduce cross-bank data movement, and improve overall storage-side execution throughput. This embodiment further demonstrates that the dual-layer cache end-to-end neural network acceleration system proposed in this invention can not only support large-scale different operator requirements but also provide efficient weight and input reuse in locally computationally intensive scenarios, enabling end-to-end inference to present a stable and continuous execution flow on the storage side.

[0036] The computational flow of the dual-layer cache end-to-end neural network acceleration system described in this invention is as follows: Figure 6 As shown, it includes the following steps:

[0037] Step S1: The host or upper-layer computing module writes the input feature map and control instructions of the current layer into the system. The command scheduling module generates the corresponding DRAM in-memory computing command sequence according to the task type and allocates the computing tasks to each module.

[0038] Step S2: The input feature map is loaded into the global cache module. The cross-bank broadcast mechanism is completed within the global cache module to realize data sharing and reuse among banks for subsequent parallel computing.

[0039] Step S3: Each participating bank reads its corresponding input features from the global cache module and allocates them to the local cache module. The local cache module triggers data loading and local reuse preparation according to the scheduling command.

[0040] Step S4: The local cache module loads the convolution weights from the Bank by executing the PIM_BK2LBUF instruction, and then reuses them in subsequent matrix multiplication and addition operations.

[0041] Step S5: Perform the multiplication and addition operation between the input and weights within the near-memory computation core module. This module simultaneously supports batch normalization (BN) and activation ReLU fusion commands, enabling integrated execution of convolution and post-processing operations.

[0042] Step S6: The adder and register accumulate the partial product results output by the near-memory computation core module and store the intermediate results to form a complete output feature map array.

[0043] Step S7: The scheduling module triggers the PIM_BK2GBUF command to write the calculated output feature map back to the global cache module. The written-back data can be used as input for the next layer's calculation, realizing data transfer and feature reuse between layers.

[0044] Step S8: The system determines whether there are any unfinished computation tasks; if so, it returns to step S3 to continue loading and computation; if all tasks are completed, the final result is output to the host, completing the end-to-end inference pipeline.

[0045] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A two-tier cache end-to-end neural network acceleration system based on DRAM in-memory computation, characterized in that, include: The data and command scheduling module is used for end-to-end task decomposition into commands, execution timing and synchronization management, command broadcast coordination and multi-bank data interaction; The global cache module is used to store intermediate feature maps and some output results; it is a channel-level shared cache. The global cache module includes a channel-level shared storage area to store intermediate input data. It synchronously distributes input data during broadcast control logic and supports two access methods: bank-level index and channel-level index. The capacity of the global cache module is configurable. Each local cache module corresponds one-to-one with each near-memory computing core module: each local cache is connected to its corresponding near-memory computing core module through an independent read / write interface to support parallel data access and computation; the capacity of the local cache module is configurable. The global cache module and the local cache module together form a two-layer cache structure. The local caching module, located at each Bank level, is used for local data reuse: temporarily caching local convolutional kernels and input features; The near-memory computing core module is a MAC core module embedded inside the DRAM bank, used to perform computational operations; The Channel-level computing core module, located at the Channel end, is used for cross-bank aggregation, post-processing, and output write-back operations.

2. The DRAM on-chip computing based two-level cache end-to-end neural network acceleration system of claim 1, wherein, The near-memory computing core module includes: a multiply-accumulate array, adders and accumulators, register units, batch normalization (BN), and activation units (ReLU); the channel-level computing core module includes: residual addition and activation units, and pooling units. 3.The DRAM-based in-memory-computing end-to-end neural network acceleration system with two-level cache according to claim 2, wherein, The data and command scheduling module receives convolution / activation / pooling / normalization task requests from the upper-layer host or compiler, parses the operation type, and generates the corresponding DRAM in-memory computation command stream scheduler.

4. The DRAM on-chip computing based two-level cache end-to-end neural network acceleration system of claim 3, wherein, The DRAM in-memory computing command flow scheduling includes: controlling the issuance and priority of custom DRAM in-memory computing instructions, managing the command parallelism between different banks, and triggering the next layer of tasks after the task is completed.

5. The DRAM on-chip computing based two-level cache end-to-end neural network acceleration system of claim 4, wherein, The custom DRAM in-memory calculation instructions include: PIM_BK2GB, PIM_GB2BK, PIM_BK2LB, PIM_LB2BK, PIM_POOL, PIM_ADD_RELU, PIM_CONV_BN_RELU, and PIM_CONV_BN.

6. The DRAM on-chip computing based two-level cache end-to-end neural network acceleration system of claim 5, wherein, The near-memory computing core module performs computational operations including convolution, matrix multiplication, ReLU, and BatchNorm. The input of the near-memory computing core module is data from the global cache module or the local cache module. The output of the module is directly written back to the local cache module or passed to the global cache module. It has an extensible command interface for receiving and parsing in-DRAM computation instructions to realize computation and data transmission functions.

7. The DRAM on-chip computing based two-level cache end-to-end neural network acceleration system of claim 6, wherein, The operation process of the Channel-level computing core module is as follows: It aggregates the outputs of the near-memory computing core module from several banks; Perform cross-bank aggregation operations; Post-processing after activation or batch normalization; The control results are written back to the global cache module or host interface to provide input for the next layer of network or output.

Images