Three-dimensional stacked heterogeneous in-memory computing system and data interaction processing method thereof

Abstract

The invention belongs to the technical field of chip design, and discloses a three-dimensional stacked heterogeneous in-memory computing system and a data interaction processing method thereof.The three-dimensional stacked heterogeneous in-memory computing system comprises a storage chip layer and an AI processing chip layer located below the storage chip layer, the storage chip layer is electrically connected with the AI processing chip layer, and the AI processing chip layer is electrically connected with the storage chip layer. The AI processing chip layer is of a heterogeneous storage and calculation integrated structure integrating an RRAM storage and calculation array and an SRAM storage and calculation array, and the RRAM storage and calculation array is used for storing system configuration data; and the SRAM memory array is used for writing read-write data of the memory chip layer and completing data processing in the array. According to the three-dimensional stacking heterogeneous storage and calculation integrated device, through heterogeneous cooperation of the RRAM and the SRAM storage and calculation integrated array, key problems existing in the prior art are fully balanced and considered, finally, the optimal design of a system architecture level is achieved, and core obstacles are cleared away for large-scale application of the three-dimensional stacking technology in the field of AI chips.

Description

Technical Field

[0001] This invention belongs to the field of chip design technology, and in particular relates to a three-dimensional stacked heterogeneous in-memory computing system and its data interaction processing method. Background Technology

[0002] As artificial intelligence technology enters the stage of large-scale deployment, large-scale models based on the Transformer architecture (such as large language models and multimodal models) continue to push the boundaries of parameter scale and task complexity, placing increasingly stringent demands on the computational efficiency of underlying AI chips. These demands primarily focus on two core dimensions: on-chip computing power and off-chip data transmission bandwidth. On-chip computing power directly determines the chip's efficiency in processing prepared data. Currently, with the continuous upgrading of semiconductor manufacturing processes (such as the evolution from 7nm to 3nm and more advanced nodes) and the reasonable expansion of chip physical area, on-chip computing power has achieved synchronous improvement with model requirements, effectively supporting the parallel computing needs of large models. However, off-chip data transmission bandwidth has gradually become a key bottleneck restricting the performance release of AI chips. Limited by the development pace of memory technology (such as DRAM storage density and read / write speed improvement lagging behind computing needs) and high-speed interface technology (such as the bandwidth expansion speed of interfaces such as PCIe and CXL being difficult to match data throughput needs), off-chip data transmission bandwidth not only cannot keep up with the pace of model parameter growth and computing power improvement, but also leads to a significant increase in data transmission latency in the "off-chip storage-on-chip computing" link, directly affecting the overall operating efficiency of AI chips, and becoming a core technical obstacle that urgently needs to be overcome in the current large-scale model application.

[0003] To address the off-chip data transmission bandwidth bottleneck faced by AI chips, the 3D stacking technology, with its innovative optimization of chip integration architecture, has become a highly feasible and reasonable solution at the current stage. Although 3D stacking technology provides an effective path to overcome the "memory wall" of AI chips, this architecture still faces two key challenges at its current development stage, limiting its further realization of performance potential.

[0004] The first challenge stems from the inherent limitations of DRAM memory. In a 3D stacked architecture, the memory cells integrated with the AI ​​processing chip mostly use DRAM technology. As a typical volatile memory, DRAM's data retention depends on continuous power supply. This means that after each system power failure and restart, massive amounts of data, such as large model parameters and training samples, need to be reloaded from external data sources. This not only prolongs system startup and task initialization time but also increases energy consumption and error risks during data transmission. At the same time, DRAM's storage density is significantly lower than that of non-volatile memories such as NAND Flash. Within the limited space of a 3D stack, its storage capacity is relatively limited. When facing ultra-large AI models with hundreds of billions or even trillions of parameters, additional external storage modules may still be required, indirectly weakening the bandwidth improvement advantage of the 3D stacked architecture.

[0005] The second challenge stems from the inherent flaws of the traditional von Neumann architecture. Most current 3D stacked AI processing chips still utilize the von Neumann architecture with its "storage-computation separation" principle. Even with 3D integration optimizing off-chip data transfer efficiency, on-chip data still needs frequent movement between caches (such as SRAM) and computing units (such as AI computing cores). This "data movement" process generates significant energy waste—industry statistics show that the energy consumed by moving data between different modules on-chip is often several to tens of times greater than the energy consumed by the computing unit performing computations. As the computational demands of AI models increase, the frequency of on-chip data movement increases accordingly, directly leading to a significant rise in overall system power consumption. This not only increases the energy cost of hardware operation but also places more stringent demands on chip heat dissipation design, further limiting the application of 3D stacked architectures in high-power-sensitive scenarios (such as edge AI devices). Summary of the Invention

[0006] The purpose of this invention is to provide a three-dimensional stacked heterogeneous in-memory computing system and its data interaction processing method to solve the four key problems existing in the prior art: insufficient off-chip data transmission bandwidth, high power consumption for on-chip data transfer, easy loss of data when power is off, and mismatch between on-chip computing capabilities.

[0007] To achieve the above objectives, the device of this invention adopts a "die-level vertical stacking" scheme: the upper layer is a DRAM Die (memory chip layer), and the lower layer is a Logic Die (AI processing chip layer, integrating the aforementioned heterogeneous in-memory array of RRAM and SRAM). Electrical connection and data interaction between the two dies are achieved through TSV technology or Hybrid Bonding technology. TSV technology creates direct signal and power channels between dies by etching vertical vias inside the silicon wafer and filling them with conductive material, increasing interconnect density by 1-2 orders of magnitude compared to traditional wire bonding. Hybrid Bonding technology replaces traditional bump connections with direct metal-to-metal bonding (such as Cu-Cu bonding), reducing interconnect spacing to below the micrometer level, further improving bandwidth density and stacking stability while reducing signal transmission loss. The device of this invention is compatible with both of these stacking technologies.

[0008] Optionally, storage and computation functions can be integrated into the RRAM array: Dedicated computation units (such as multipliers, adders, and other basic arithmetic modules) can be embedded into the RRAM array initially used to store critical configuration data. This design eliminates the need to transfer critical data stored in the RRAM (such as computing power scheduling parameters and precision control thresholds) to separate on-chip computing units. Instead, related computational tasks (such as pre-calculation of computing power allocation based on configuration parameters and precision adaptation calculations) can be performed directly within the RRAM array. This deeply integrated storage-computing architecture not only eliminates the transmission of critical data between the RRAM and computing units but also avoids signal attenuation and latency during transmission, significantly improving the real-time performance of on-chip data processing.

[0009] Optionally, the working principle of an RRAM array can be simplified into three steps: First, data that needs to be used for a long time and is not frequently updated (such as configuration parameters of AI chips and fixed weights of models) is written into the RRAM array. The core advantage of RRAM is its "non-volatility". Once the data is written, it will not be lost even if the power is off. Unlike DRAM, it does not need to be reloaded every time it is powered on. It can be directly retained as the "basic data" for subsequent calculations.

[0010] Then, in-situ calculations are performed: calculations are performed directly without moving the data: for the RRAM of SLC, its resistance value can correspond to the data (for example, a high resistance state represents "1" and a low resistance state represents "0"). During calculation, there is no need to read the data from the RRAM and send it to an independent calculation unit. By applying a specific voltage to the RRAM array, the calculation is completed directly inside the RRAM array through circuit operation using the current superposition effect of different RRAM cells, and the calculation result is obtained.

[0011] Finally, the results are output. After the calculation is completed, there is no need for complex data backhaul. The results can be read directly from the output of the RRAM array and quickly used for subsequent AI tasks (such as model inference and parameter adaptation).

[0012] Optionally, for the integrated storage and computing of on-chip SRAM arrays, this invention also embeds suitable computing units into the on-chip SRAM array of traditional AI processor chips. These computing units can directly handle the read and write data from the DRAM die and complete data preprocessing (such as data format conversion and feature extraction) within the SRAM without having to move the raw data to other computing modules. On the other hand, they can participate in other core AI operations (such as intermediate layer calculations in model inference and parallel data operations) in addition to the calculation of key data in the RRAM, further reducing the on-chip computing pressure.

[0013] Optionally, the SRAM-CIM integrated array of the device of the present invention supports both SRAM and CIM modes. The SRAM mode is used to preload various data, and the CIM mode performs AI calculation operations. Each SRAM-CIMMacro contains 256 input activations, 64 partial sum outputs, and a 256×64 8-bit weight array. Each sub-CIM unit integrates a 256×8-bit bitwise multiplier, a parallel adder tree, and a partial sum accumulator, which can realize large-scale parallel MAC operations. It has high programmability, with input activations supporting 1-8 bit programmable width (signed / unsigned), and weights supporting 8 and 16 bits. Multiple macros can be configured to adapt to neural networks with different topologies through parallel, serial, or 2D array configurations. It can also perform weight updates and MAC operations concurrently. Moreover, the all-digital design avoids the accuracy loss of analog CIM, and the accumulator output bit width meets the requirements of subsequent batch normalization, pooling, and other vector calculations.

[0014] Optionally, through the aforementioned heterogeneous in-memory computing structure of "RRAM computing array + SRAM computing array," this device achieves dual optimization of functionality and performance: the RRAM computing array ensures the security of critical data due to its non-volatile characteristics, while reducing on-chip transmission of core configuration data through in-situ computation; the SRAM computing array, relying on its fast read / write response capabilities, optimizes the on-chip transmission efficiency of DRAM data interaction and general AI computing while maintaining compatibility with existing system interfaces. Figure 5 As shown, the synergistic effect of the two enables the device to have both rapid task response efficiency (instant access to critical data and in-situ computation) and efficient AI computing capabilities (rapid processing of general data and rational allocation of computing power), further enhancing the overall performance advantages of the system architecture from the perspective of on-chip data transmission and computational collaboration.

[0015] Optionally, all communication data between the DRAM Die and the AI ​​processor chip (Logic Die) should be written to the SRAM-CIM computing array only through the 3D integrated interface, rather than establishing a data channel directly with the RRAM computing array.

[0016] Optionally, to further optimize the memory access efficiency between the NPU (Neural Processing Unit) and the DRAM Die: according to the interface design, each NPU is typically connected to multiple DRAM Banks through a three-dimensional integrated interface (such as an SDR interface) to form a "one NPU, multiple banks" hardware configuration.

[0017] The technical effects of this invention are as follows:

[0018] Addressing the two core challenges currently faced by 3D stacked architectures—"DRAM volatility and capacity limitations" and "power consumption for on-chip data transfer in von Neumann architecture"—this invention proposes an advanced 3D stacked heterogeneous in-memory computing device. Building upon the advantages of 3D stacking technology in solving off-chip data transmission bandwidth issues, this invention achieves a synergistic breakthrough in addressing multiple pain points through the integration and innovation of heterogeneous storage and in-memory computing architecture.

[0019] The core design of this device lies in the hierarchical deployment of a "heterogeneous in-memory computing array" on the Logic layer of a three-dimensional stacked architecture. On the one hand, it uses an in-memory computing array based on RRAM (resistive random access memory) to store core data. As a typical non-volatile memory, RRAM can retain data without continuous power supply, fundamentally solving the problem in the original architecture where all data is lost after a power outage and all data needs to be reloaded after power-on. This significantly shortens the system restart and task initialization time, while reducing the energy consumption and risk of repeated data transmission. RRAM has a significantly higher storage density than DRAM, and can carry a larger capacity of data within the limited space of a three-dimensional stack. It can better adapt to the storage needs of ultra-large AI models with hundreds of billions or trillions of parameters, avoid dependence on additional external storage modules, and further enhance bandwidth advantages.

[0020] On the other hand, the on-chip computing process is optimized by using an in-memory computing array based on SRAM (Static Random Access Memory). SRAM has ultra-high read and write speeds, and through the "in-memory computing" architecture design, data storage and computing units are deeply integrated, allowing some computing tasks to be completed directly within the storage array. This completely breaks the limitations of the traditional von Neumann architecture's "storage-computation separation" and significantly reduces the frequent movement of on-chip data between the cache and computing units. This design not only reduces the ineffective energy consumption caused by "data movement" (avoiding energy losses several to tens of times greater than the computation itself in traditional architectures), but also improves computing density through "proximity computing," effectively bridging the gap between on-chip computing power and storage efficiency.

[0021] In summary, the three-dimensional stacked heterogeneous in-memory computing device of the present invention achieves a full balance of consideration for the four key problems of chip "insufficient off-chip data transmission bandwidth, high power consumption for on-chip data transfer, easy loss of data when power is off, and mismatch of on-chip computing capabilities" through the heterogeneous collaboration of RRAM and SRAM in-memory computing arrays. Ultimately, it achieves the optimal design at the system architecture level, clearing the core obstacles for the large-scale application of three-dimensional stacking technology in the field of AI chips. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 This is a schematic diagram of the structural design of the three-dimensional stacked heterogeneous in-memory computing system in an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the specific implementation structure of RRAM-CIM in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the SRAM in-memory computing array structure in an embodiment of the present invention;

[0027] Figure 4 This is a diagram of the integrated heterogeneous in-memory computing structure in an embodiment of the present invention;

[0028] Figure 5 This is a performance comparison chart between the present invention and traditional solutions. Detailed Implementation

[0029] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0030] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0031] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0032] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] like Figure 1 - Figure 5 As shown, this embodiment provides a three-dimensional stacked heterogeneous in-memory computing system, including a memory chip layer and an AI processing chip layer located below the memory chip layer. The memory chip layer and the AI ​​processing chip layer are electrically connected. The AI ​​processing chip layer is a heterogeneous in-memory computing structure integrating an RRAM in-memory array and an SRAM in-memory array. The RRAM in-memory array is used to store system configuration data; the SRAM in-memory array is used to write read and write data to the memory chip layer and complete data processing within the array.

[0035] The three-dimensional stacked heterogeneous in-memory computing system performs business data transmission processing, configuration data update processing, and memory access interaction processing between the neural network processing unit and the DRAM die.

[0036] Addressing the two core challenges currently faced by 3D stacked architectures—"DRAM volatility and capacity limitations" and "power consumption for on-chip data transfer in von Neumann architecture"—this embodiment proposes an advanced 3D stacked heterogeneous in-memory computing device. Building upon the advantages of 3D stacking technology in solving off-chip data transmission bandwidth issues, it achieves a synergistic breakthrough in addressing multiple pain points through the integration and innovation of heterogeneous storage and in-memory computing architecture.

[0037] The specific implementation process of this embodiment includes:

[0038] 1. Three-dimensional stacking technology solution:

[0039] The three-dimensional stacked heterogeneous in-memory computing device proposed in this embodiment relies on advanced packaging technologies such as TSV (Through Silicon Via), Hybrid Bonding, and Flip Chip to realize its core stacking architecture. The core value of these technologies lies in breaking the limitations of the planar connection between traditional chips and PCBs (Printed Circuit Boards). Through high-density interconnection in the vertical or near-vertical direction, the originally dispersed chip components are stacked in three dimensions, providing a hardware foundation for improving the integration density and transmission efficiency of the device.

[0040] Specifically, regarding the structural design of the device in this embodiment (such as...) Figure 1 As shown, the device employs a "Die-level vertical stacking" scheme: the upper layer is a DRAM Die (memory chip layer), and the lower layer is a Logic Die (AI processing chip layer, integrating the aforementioned heterogeneous in-memory array of RRAM and SRAM). Electrical connections and data exchange between the two dies are achieved through TSV technology or Hybrid Bonding technology. TSV technology creates direct signal and power channels between dies by etching vertical vias inside the silicon wafer and filling them with conductive material, increasing interconnect density by 1-2 orders of magnitude compared to traditional wire bonding. Hybrid Bonding technology replaces traditional bump connections with direct metal-to-metal bonding (such as Cu-Cu bonding), reducing interconnect spacing to below the micrometer level, further improving bandwidth density and stacking stability while reducing signal transmission loss. The device in this embodiment is compatible with both of these stacking technologies.

[0041] This vertical stacking design of "DRAM Die + Logic Die" brings two key advantages: First, it significantly improves device integration density. By utilizing vertical space instead of planar layout, more functional units can be accommodated within the same package size, especially providing space support for the large-scale deployment of heterogeneous in-memory computing arrays on Logic Dies. Second, it further alleviates the off-chip bandwidth bottleneck of AI processing chips. DRAM Die and Logic Die achieve "near-chip storage" through TSV / Hybrid Bonding. The data transmission path is shortened by more than 90% compared to the traditional "chip-PCB-storage" mode, reducing bandwidth loss and effectively matching the high-frequency data requirements of AI computing units on Logic Dies. This lays the transmission foundation for heterogeneous in-memory computing arrays to achieve "storage-computing synergy" performance.

[0042] 2. Heterogeneous In-Memory Computing Solution:

[0043] In traditional devices that employ 3D stacking technology and whose memory chips are based on DRAM, AI processor chips will face a severe performance bottleneck due to "double volatility" if they follow conventional design schemes. DRAM Die itself is volatile memory, while the on-chip cache of Logic Die (AI processing chip layer) relies on SRAM (also a volatile memory). Under this architecture, once the system loses power, not only will the massive sample data and model parameters used for AI calculations be lost, but more importantly, the core data on the Logic Die used to configure the working state of the AI ​​processor (such as computing power scheduling parameters, precision control thresholds, computing unit collaboration rules, task priority configuration, etc.) will also be completely cleared.

[0044] These configuration data are crucial to the performance of AI processors: for example, computing power scheduling parameters directly determine the load distribution efficiency of multiple computing cores, and precision control thresholds affect the balance between the accuracy and speed of model inference. If these data need to be reloaded from external storage (such as hard drives or external SSDs) every time the system is powered on, it will not only occupy the already tight off-chip transmission bandwidth, but also generate significant loading latency. In traditional solutions, the loading process of configuration data alone may take hundreds of milliseconds to several seconds, causing the AI ​​processor to be unable to enter an efficient working state in time. Especially in scenarios that are sensitive to response speed, such as edge AI and real-time inference, this latency will directly restrict the overall system performance.

[0045] To address this pain point, the heterogeneous in-memory computing solution in this embodiment features a key innovation at the Logic Die layer: by integrating a non-volatile RRAM (Resistive Random Access Memory) array specifically for storing critical configuration data strongly correlated with the performance of the AI ​​processor. Leveraging RRAM's core characteristic of "retaining data without continuous power," this configuration data remains stably stored within the Logic Die even during system power outages. When the system is powered on again, the AI ​​processor can directly access the configuration data stored in the RRAM to initialize its operating state, eliminating the need for external loading. This design completely avoids the "configuration data loss-reloading" process of traditional solutions, saving not only the time and bandwidth costs of the loading process but also enabling the AI ​​processor to quickly enter a high-efficiency computing state after power-on, significantly reducing task startup latency.

[0046] In terms of actual performance, this design not only solves the fundamental problem of "data loss when power is off" but also breaks through the performance bottleneck of "slow loading when power is on". It enables the heterogeneous storage and computing capabilities of Logic Die to work synergistically with the bandwidth advantages of three-dimensional stacking. During AI computing, not only can massive amounts of business data be quickly transmitted through DRAM Die, but core configuration data can also be instantly accessed through RRAM. This comprehensively improves the efficiency of AI computing from the perspective of data storage and scheduling, further solidifying the optimality of the device in this embodiment at the architectural level.

[0047] Building upon the solution to the problem of data volatility during system power outages using RRAM, this embodiment further leverages in-memory computing technology to optimize on-chip data transmission efficiency of the AI ​​processor chip at the architectural level. The core idea is to replace the traditional "data transport-computation" process with "in-situ data computation," thereby significantly reducing the movement of on-chip data between storage and computing units, thus lowering transmission latency and energy consumption.

[0048] First, storage and computation functions are integrated into the RRAM array: Dedicated computation units (such as multipliers, adders, and other basic arithmetic modules) are embedded within the RRAM array initially used to store critical configuration data. This design eliminates the need to transfer critical data stored in the RRAM to separate on-chip computation units; instead, related computational tasks (such as pre-calculation of computing power allocation based on configuration parameters and precision adaptation calculations) can be performed directly within the RRAM array. This deeply integrated "storage-computation" architecture not only eliminates the transmission of critical data between the RRAM and computation units but also avoids signal attenuation and latency during transmission, significantly improving the real-time performance of on-chip data processing.

[0049] RRAM-CIM (Resistive Random Access Memory-Computing) is a technical solution that integrates data storage and computation within the same space. Its core principle relies on the characteristics of RRAM to avoid data being moved back and forth between storage and computation units, thereby reducing latency and energy consumption while addressing data volatility. Its specific implementation structure is as follows: Figure 2 As shown.

[0050] Its working principle can be simplified into three steps:

[0051] First, data that needs to be used for a long time and is not frequently updated (such as the configuration parameters of AI chips and fixed weights of models) is written into the RRAM array. The core advantage of RRAM is its "non-volatility". Once the data is written, it will not be lost even if the power is off. Unlike DRAM, it does not need to be reloaded every time it is powered on. It can be directly stored as the "basic data" for subsequent calculations.

[0052] Then, in-situ calculations are performed: calculations are performed directly without moving the data: for the RRAM of SLC, its resistance value can correspond to the data (for example, a high resistance state represents "1" and a low resistance state represents "0"). During calculation, there is no need to read the data from the RRAM and send it to an independent calculation unit. By applying a specific voltage to the RRAM array, the calculation is completed directly inside the RRAM array through circuit operation using the current superposition effect of different RRAM cells, and the calculation result is obtained.

[0053] Finally, the results are output. After the calculation is completed, there is no need for complex data backhaul. The results can be read directly from the output of the RRAM array and quickly used for subsequent AI tasks (such as model inference and parameter adaptation).

[0054] Secondly, regarding the in-memory computing of the on-chip SRAM array, this embodiment focuses on compatibility with existing system interfaces: traditional AI processor chips generally rely on on-chip SRAM as a high-speed cache, serving as a buffer for DRAM chip data read / write and temporary storage for high-frequency data. To avoid significant modifications to existing system interfaces and data interaction logic, this embodiment also embeds compatible computing units within the on-chip SRAM array of the traditional AI processor chip. These computing units can directly handle the read / write data from the DRAM die, completing data preprocessing (such as data format conversion and feature extraction) within the SRAM without transferring the raw data to other computing modules. Furthermore, they can participate in other core AI operations besides critical RRAM data computation (such as intermediate layer computation in model inference and parallel data computation), further reducing the on-chip computing load.

[0055] like Figure 3 As shown, the SRAM-CIM integrated array of this embodiment supports both SRAM and CIM modes. The SRAM mode is used to preload various data, while the CIM mode performs AI calculation operations. Each SRAM-CIMMacro contains 256 input activations, 64 partial sum outputs, and a 256×64 8-bit weight array. Each sub-CIM unit integrates a 256×8-bit bitwise multiplier, a parallel adder tree, and a partial sum accumulator, enabling large-scale parallel MAC operations. It has high programmability, with input activations supporting 1-8 bit programmable widths (signed / unsigned), and weights supporting 8 and 16 bits. Multiple macros can be configured to adapt to different topologies of neural networks through parallel, serial, or 2D array configurations. It can also perform weight updates and MAC operations concurrently. Furthermore, the all-digital design avoids the accuracy loss of analog CIM, and the accumulator output bit width meets the requirements of subsequent batch normalization, pooling, and other vector calculations.

[0056] Integrated heterogeneous in-memory computing architecture, such as Figure 4 As shown. Through the aforementioned heterogeneous in-memory computing structure of "RRAM computing array + SRAM computing array", this device achieves dual optimization of functionality and performance: the RRAM computing array ensures the security of critical data with its non-volatile characteristics, while reducing on-chip transmission of core configuration data through in-situ computation; the SRAM computing array, relying on its fast read / write response capabilities, optimizes the on-chip transmission efficiency of DRAM data interaction and general AI computing while maintaining compatibility with existing system interfaces. Figure 5As shown, the synergistic effect of the two enables the device to have both rapid task response efficiency (instant access to critical data and in-situ computation) and efficient AI computing capabilities (rapid processing of general data and rational allocation of computing power), further enhancing the overall performance advantages of the system architecture from the perspective of on-chip data transmission and computational collaboration.

[0057] 3. Integration of 3D stacking and heterogeneous in-memory computing: In terms of the deep integration of 3D integration technology and heterogeneous in-memory computing architecture, this embodiment of the device innovatively proposes the "SRAM-CIM single in-memory computing array interface technology solution". By clarifying the data interaction path and interface adaptation design, the simplicity of the system architecture and the stability of data transmission are further optimized.

[0058] The design logic of this solution is closely integrated with the data storage characteristics of heterogeneous in-memory arrays: As mentioned above, RRAM in-memory arrays mainly store "configuration data and AI parameter data that do not require frequent updates" (such as computing power scheduling parameters, model fixed weights, etc.). The update frequency of this type of data is much lower than that of the dynamically flowing business data in the DRAM Die (such as real-time training samples and inference input data). Based on this, to avoid interface redundancy and data interaction chaos, this embodiment stipulates that all communication data between the DRAM Die and the AI ​​processor chip (Logic Die) must be written to the SRAM-CIM in-memory array only through the three-dimensional integration interface, rather than directly establishing a data channel with the RRAM in-memory array. This is the core meaning of "single" in "single in-memory array interface", that is, the DRAM only forms a direct data interaction with the SRAM in-memory array, without the need to design a separate interface for the RRAM with an extremely low update frequency, which greatly simplifies the interface architecture at the three-dimensional integration level.

[0059] The specific data interaction process is divided into two scenarios: First, in the regular business data transmission scenario, real-time data used for AI computing in the DRAM die (such as image data to be inferred and text features to be trained) is directly stored into the SRAM-CIM storage array through the 3D integration interface. Then, the SRAM can rely on its own storage and computing functions to directly preprocess or perform core operations on this data without additional data transfer, which is in line with the design goal of "optimizing on-chip data transmission" mentioned above. Second, in the RRAM data update scenario, when the configuration data or AI parameters stored in the RRAM need to be updated (such as weight adjustment after model iteration and optimization of computing power rules), there is no need for the DRAM to directly transmit data to the RRAM. Instead, the DRAM first writes the updated data into the SRAM-CIM storage array. After the data has been verified or adapted in the SRAM, it is then read from the SRAM and written to the RRAM. This "DRAM→SRAM→RRAM" relay update path not only utilizes the high-speed read and write capabilities of SRAM to ensure data transmission efficiency, but also avoids the stability risks of RRAM caused by frequent interface interactions (because RRAM itself is not suitable for high-frequency data writing). At the same time, through the buffering effect of SRAM, it reduces the timing mismatch problems that may occur when DRAM and RRAM communicate directly.

[0060] In terms of interface technology selection, the three-dimensional integrated interface between the DRAM Die and the AI ​​processor chip (Logic Die) adopts a "single data rate (SDR) interface". The core reason is to "match the interface behavior of SRAM memory": As a high-speed static memory, SRAM has a fast data read and write response speed, and its interface timing characteristics are more suitable for the "single clock edge data transmission" mode of SDR. Compared with the dual clock edge transmission of the double data rate (DDR) interface, the control logic of the SDR interface is simpler and can better synchronize with the read and write cycle of SRAM, avoiding data transmission delays or errors caused by incompatible interface timing. At the same time, the SDR interface has relatively lower power consumption, which is highly consistent with the requirements of SRAM memory array for "high-frequency data interaction but energy consumption control", further ensuring the high efficiency and low power consumption of on-chip data transmission.

[0061] In summary, the SRAM-CIM single-path in-memory computing array interface solution achieves efficient integration of three-dimensional integration and heterogeneous in-memory computing through a single-path design of "DRAM only interfacing with SRAM", logical optimization of "RRAM updates via SRAM", and technical selection of "SDR interface adapting to SRAM behavior". This simplifies the interface architecture at the three-dimensional integration level, and relies on the bridging role of SRAM to balance the rapid transmission of regular business data and the stable updating of RRAM data. At the same time, interface adaptation ensures the stability of data interaction, further enhancing the optimality of the device in this embodiment at the system architecture level.

[0062] In the process of implementing the three-dimensional stacked architecture and heterogeneous in-memory computing solution, it is also necessary to take into account the hardware physical constraints and the in-depth optimization of memory access efficiency. These two aspects of design, together with the SRAM-CIM interface solution mentioned above, will further improve the overall performance of the system.

[0063] On the one hand, addressing the physical constraint that "Logic Die area is typically limited in 3D stacked chips (not exceeding DRAM Die area)," this embodiment achieves a balance between computing power and area through a "high-density in-memory computing architecture." As mentioned earlier, the Logic Die integrates both RRAM and SRAM in-memory computing arrays. Compared to the traditional "independent storage + independent computing" architecture (which requires separate areas for storage and computing units), the in-memory computing architecture deeply integrates data storage and computing functions, simultaneously realizing both "storage" and "computing" functions within the same physical space, significantly reducing the area occupied by functional modules. This design allows for the integration of larger-scale computing units within the limited area of ​​the Logic Die without sacrificing storage capacity, effectively alleviating the problem of "insufficient computing power due to area constraints," achieving the goal of "supporting higher computing power in a limited area," perfectly adapting to the area ratio constraints of Logic Die and DRAM Die in 3D stacking, and avoiding increased packaging difficulty or cost due to exceeding area limits.

[0064] Based on a three-dimensional stacked architecture, this device integrates a heterogeneous in-memory computing structure of RRAM-CIM and SRAM-CIM in the Logic Die for AI computing. Given the non-volatility of RRAM, RRAM-CIM is used to store and compute configuration parameters and the first few layers of AI model parameters that require almost no updates, thereby improving response speed. Given the faster read and write speed of SRAM-CIM, SRAM-CIM is used to load other AI parameters from DRAM for subsequent calculations.

[0065] When RRAM-CIM performs AI processor configuration and calculations for the first few layers of the AI ​​model, SRAM-CIM can quickly load weights from DRAM to form a pipeline operation with RRAM-CIM. SRAM-CIM performs subsequent calculations immediately after RRAM-CIM completes its calculations. This heterogeneous collaboration greatly improves the response and computational efficiency of AI tasks.

[0066] On the other hand, based on the three-dimensional integrated interface scheme established above, this embodiment further optimizes the memory access efficiency between the NPU (Neural Processing Unit) and the DRAM Die: According to the interface design, each NPU typically establishes a connection with multiple DRAM Banks (memory banks) through a three-dimensional integrated interface (such as an SDR interface), forming a "one NPU, multiple banks" hardware configuration. When the NPU needs to access DRAM data (such as calling real-time training samples or intermediate model data), Figure 1 The controller shown triggers an "interleaving" mechanism through "reasonable memory access address settings". Specifically, the controller breaks down the NPU's continuous memory access addresses into discrete address segments corresponding to multiple DRAM banks, so that multiple data segments that originally needed to be accessed serially can be responded to in parallel by different DRAM banks. For example, addresses 0, 2, and 4 are assigned to Bank1, and addresses 1, 3, and 5 are assigned to Bank2. When the NPU initiates continuous address access, Bank1 and Bank2 can read the corresponding data simultaneously and transmit it to the SRAM-CIM array through the three-dimensional integrated interface.

[0067] The core value of this interleaved access design lies in two aspects: First, it breaks through the bandwidth limitations of a single DRAM bank, significantly improving overall data throughput through parallel access to multiple banks and preventing the NPU from being idle due to waiting for data from a single bank. Second, it reduces memory access latency by utilizing address splitting and parallel transmission (especially when processing massive amounts of continuous data, the latency can be reduced to near the response time of a single bank), matching the high-speed data processing capabilities of the SRAM-CIM array. Simultaneously, this mechanism is entirely implemented using existing 3D integrated interfaces, requiring no additional hardware interfaces. This ensures compatibility with the previously mentioned interface solutions while further exploring the hardware potential of multiple DRAM banks, optimizing the data flow path efficiency of "SRAM-CIM-controller-DRAM Bank".

[0068] The high computing power density in-memory computing design of Logic Die solves the "computing power pressure under area constraints", while the interleaved access mechanism of NPU and DRAM Bank optimizes the "data reading efficiency". The two, from the two dimensions of "computing power supply" and "data input", work together with the SRAM-CIM interface solution and heterogeneous in-memory computing array to build a complete system architecture of "physical constraint adaptation, simple interface, efficient data and sufficient computing power", further consolidating the technical advantages of the device in this embodiment.

[0069] In summary, the device in this embodiment, through the deep integration of "three-dimensional stacking technology + heterogeneous in-memory computing architecture," closely addresses the four key issues of AI chips: insufficient off-chip data transmission bandwidth, high power consumption for on-chip data transfer, data loss due to power failure, and mismatch between on-chip computing capabilities. It systematically solves the four core pain points of modern AI chips from six dimensions: architecture, transmission, energy consumption, efficiency, physical adaptation, and compatibility. Ultimately, it achieves comprehensive advantages of "low latency, low power consumption, high bandwidth, high computing density, and high compatibility," providing crucial hardware support for the efficient deployment of ultra-large AI models at the edge and on terminals. It also possesses strong technological innovation and industrial application potential.

[0070] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A three-dimensional stacked heterogeneous in-memory computing system, characterized in that, It includes a storage chip layer and an AI processing chip layer located below the storage chip layer. The storage chip layer and the AI ​​processing chip layer are electrically connected. The AI ​​processing chip layer is a heterogeneous in-memory computing structure that integrates an RRAM in-memory array and an SRAM in-memory array. The RRAM in-memory array is used to store system configuration data. The SRAM in-memory array is used to write read and write data to the storage chip layer and complete data processing within the array.

2. The three-dimensional stacked heterogeneous in-memory computing system according to claim 1, characterized in that, The storage chip layer and the AI ​​processing chip layer are electrically connected and interact with each other through TSV technology or Hybrid Bonding technology.

3. The three-dimensional stacked heterogeneous in-memory computing system according to claim 1, characterized in that, The memory chip layer includes two DRAM Dies arranged in sequence. Each DRAM Die includes multiple DRAM Banks. The two DRAM Dies are electrically connected and interact with each other through TSV technology or Hybrid Bonding technology.

4. The three-dimensional stacked heterogeneous in-memory computing system according to claim 3, characterized in that, The communication data between the storage chip layer and the AI ​​processing chip layer is written to the SRAM storage array through a three-dimensional integrated interface.

5. The three-dimensional stacked heterogeneous in-memory computing system according to claim 4, characterized in that, The AI ​​processing chip layer includes several neural network processing units, each of which is connected to multiple parallel DRAM banks.

6. The three-dimensional stacked heterogeneous in-memory computing system according to claim 1, characterized in that, Both the RRAM and SRAM memory arrays have corresponding computing units embedded in them.

7. A data interaction processing method for a three-dimensional stacked heterogeneous in-memory computing system, applied to the three-dimensional stacked heterogeneous in-memory computing system according to any one of claims 1-6, characterized in that, include: The three-dimensional stacked heterogeneous in-memory computing system performs business data transmission processing, configuration data update processing, and memory access interaction processing between the neural network processing unit and the DRAM die.

8. The data interaction processing method according to claim 7, characterized in that, The specific process of the business data transmission processing includes: acquiring real-time data for AI calculation in the DRAM die, the real-time data including image data to be inferred and text feature data to be trained; storing the real-time data into the SRAM storage array through the three-dimensional integration interface and performing corresponding data processing based on the preset task.

9. The data interaction processing method according to claim 7, characterized in that, The specific process of the configuration data update includes: when the configuration data or AI parameters stored in the RRAM storage array need to be updated, the DRAM Die writes the updated data into the SRAM storage array for verification or format adaptation, and after the SRAM storage array completes the processing, it writes the data into the RRAM storage array.

10. The data interaction processing method according to claim 7, characterized in that, The specific process of the memory access interaction includes: When the neural network processing unit needs to access data in the DRAM Die, the continuous memory access address of the neural network processing unit is decomposed into discrete address segments corresponding to multiple DRAM Banks, and different DRAM Banks respond and process in parallel. Each DRAM Bank reads the corresponding data at the same time and transmits it to the SRAM storage array through the three-dimensional integrated interface.

Images