Monolithic 3d integrated heterogeneous memory media in-memory processing neural network processor

Abstract

The application relates to a memory-computing integrated neural network processor based on a three-dimensional integrated heterogeneous storage medium and a control method thereof, which comprises an off-chip volatile storage chip connected with a logic chip and used for information interaction with the logic chip, an off-chip non-volatile storage chip connected with the logic chip and used for information interaction with the logic chip, and the logic chip configured to realize a combination of any of the following functions: executing a computing task, performing data caching and realizing information interaction; the off-chip volatile storage chip, the logic chip and the off-chip non-volatile storage chip are integrated by using a three-dimensional stacking technology, the off-chip volatile storage chip is located at the top in the vertical direction and is connected with an external heat dissipation structure; so that the off-chip volatile storage chip with high heat generation can be better dissipated through the heat dissipation structure connected therewith, and the heat influence on the surrounding modules is reduced as much as possible.

Description

Technical Field

[0001] This application relates to the field of semiconductor chip technology, specifically to a three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor and its control method. Background Technology

[0002] Large Language Models (LLMs), as a core evolutionary direction of Natural Language Processing (NLP) technology in the field of Artificial Intelligence (AI), have shown an exponential growth trend in recent years. Their technological maturity and application breadth have continued to break through, demonstrating irreplaceable advantages in several key areas.

[0003] Meanwhile, the development of LLMs has gradually revealed a core principle widely validated by academia and industry—the scaling law. The core meaning of this principle is that, with sufficient high-quality training data and matching computational resources, the core performance indicators of LLMs (such as perplexity in language generation and accuracy in downstream tasks) show a significant positive correlation with the scale of model parameters. For example... Figure 1 As shown, the model performance score continuously improves with the exponential increase in the number of parameters.

[0004] Traditional computer systems rely on "storage system architecture" (such as...) Figure 2 As shown, the process of registers → on-chip cache → main memory → local disk → remote storage achieves a trade-off between capacity, speed, and cost. Figure 3 As shown, each tier exhibits significant differences in relative cost, storage density, bandwidth, access latency, and power consumption.

[0005] However, while the three-dimensional stacked structure brings high performance, it also raises serious challenges in thermal management: In existing 3D integration solutions, the stacking order is usually determined primarily by the convenience of interconnect wiring or electrical performance, often neglecting the differences in thermal characteristics of chips with different properties. Volatile memory chips (such as DRAM) not only generate a lot of heat and have long thermal cycles during frequent read and write operations, but are also extremely sensitive to temperature; while logic chips often have localized high heat flux density points, and heat dissipation bottlenecks can cause the chips to be forced to throttling due to overheating, affecting computing efficiency.

[0006] Therefore, it is necessary to improve existing neural network processors. Summary of the Invention

[0007] To address the above problems, this application provides a three-dimensional integrated heterogeneous storage medium in-memory neural network processor, comprising: an off-chip volatile memory chip connected to a logic chip for information interaction with the logic chip; an off-chip non-volatile memory chip connected to the logic chip for information interaction with the logic chip; and the logic chip configured to perform any combination of the following functions: performing computational tasks, performing data caching, and performing information interaction; the off-chip volatile memory chip, the logic chip, and the off-chip non-volatile memory chip are integrated using three-dimensional stacking technology, with the off-chip volatile memory chip located at the top in the vertical direction and capable of being connected to an external heat dissipation structure.

[0008] This application places off-chip volatile memory chips (such as DRAM chips) on the top layer of a three-dimensional stacked structure. The advantages of this layout are twofold: First, compared to logic chips with high heat density when undertaking heavy computational tasks, and off-chip non-volatile memory chips (such as Flash) with high storage density but only localized heat generation, off-chip volatile memory chips generate more heat and have longer thermal cycles. Placing them on the top layer reduces the thermal impact of high-heat-generating chips on surrounding modules. Second, the top layer position allows for more direct contact with external heat dissipation structures (such as heat sinks and thermal pads), resulting in a shorter and more efficient heat conduction path. This allows for rapid dissipation of heat from the off-chip volatile memory chips and lower-layer chips, avoiding the problem of "heat accumulating upwards from the lower layer and hindering heat dissipation on the top layer" in traditional stacking. This layout ensures stable operation of the off-chip volatile memory chips in low-temperature environments (avoiding increased access latency or performance degradation due to high temperatures) and provides a more unobstructed heat dissipation channel for lower-layer logic chips and off-chip non-volatile memory chips, ultimately improving the thermal stability and long-term operational reliability of the entire system.

[0009] Optionally, the off-chip volatile memory chip, the logic chip, and the off-chip non-volatile memory chip are integrated sequentially in the vertical direction using a three-dimensional stacking technique.

[0010] This application places the logic chip between an off-chip volatile memory chip and an off-chip non-volatile memory chip. This design offers several advantages. First, because the off-chip volatile memory chip and logic chip are positioned close together, they do not need to be penetrated by the three-dimensional vias of the off-chip non-volatile memory chip. This allows for a larger storage capacity of the off-chip non-volatile memory chip within the same area, thus improving computational efficiency. Second, the close proximity of the off-chip non-volatile memory chip and logic chip, due to the higher storage density of the off-chip non-volatile memory chip, provides additional advantages for specific large-scale language modeling (LLM) tasks, especially the storage requirements of key-value cache (KV Cache) data in the prefill stage and long-sequence inference. Specifically, it provides ample storage space for tasks with extremely large storage requirements like LLM, overcoming the challenge of limited traditional DRAM capacity. Furthermore, in long-sequence inference, the KV Cache continuously grows; the high capacity of the off-chip non-volatile memory chip can avoid frequent KV cache deletion. The cache is swapped between off-chip volatile memory chips and off-chip non-volatile memory chips, reducing complexity and additional latency.

[0011] Optionally, the off-chip volatile memory chip and the logic chip are bonded together in a front-to-front bonding manner, and the off-chip non-volatile memory chip is bonded to the logic chip with either the front or back side.

[0012] The off-chip volatile memory chip and the logic chip are bonded together in a face-to-face bonding manner. Considering that the memory access latency of the off-chip volatile memory chip is on the order of nanoseconds (typically tens of nanoseconds), its memory access efficiency is greatly affected by the bit width and transmission rate of the external I / O interface. When the two are bonded in a face-to-face (F2F) manner, F2F bonding supports extremely high-density I / O interconnects, significantly increasing the interface bit width and supporting high-bandwidth data interaction. Compared with bonding schemes that require penetrating the silicon substrate and introducing parasitic capacitance and resistance, face-to-face bonding significantly reduces link loss, ensures the signal integrity of the off-chip volatile memory under high-speed transmission, and reduces the error correction burden. In addition, the core memory access characteristics of the off-chip non-volatile memory chip determine its adaptation requirements for interconnect bonding methods: firstly, the memory access latency is relatively high, typically on the order of 30-50μs microseconds; secondly, the amount of data read in a single operation is fixed, generally 4-16KB. From a performance bottleneck perspective, the memory access efficiency of off-chip non-volatile memory chips (such as Flash) is primarily constrained by their internal operating mechanisms, rather than the performance of external I / O interconnect links. This conclusion can be fully verified from two key dimensions: I / O density and I / O speed. From the perspective of I / O density, the core value of front-to-front bonding lies in improving I / O concurrent interaction capabilities to meet high-frequency data transmission demands. However, Flash itself does not require high-frequency, high-concurrency I / O interaction. Even with high-density F2F bonding, its memory access efficiency cannot be linearly improved; instead, it leads to ineffective redundancy and waste of I / O resources. From the perspective of I / O speed, compared to back-side bonding, it requires through-silicon vias (TSVs) to penetrate the silicon substrate. TSVs inevitably introduce parasitic capacitance and resistance, which affects high-speed I / O links, leading to reduced transmission rates and, in severe cases, even threatening data security.

[0013] Therefore, on the one hand, the front-to-front bonding method between the off-chip volatile memory chip and the logic chip increases the interface bit width, supports high-bandwidth data interaction, and meets the high-speed memory access and high-bandwidth data interaction requirements of the off-chip volatile memory chip. On the other hand, the off-chip non-volatile memory chip, due to its memory access latency reaching the microsecond level, has extremely low sensitivity to I / O rate and is not negatively affected by the front-to-front bonding between the off-chip volatile memory chip and the logic chip.

[0014] Optionally, the off-chip volatile memory chip and the logic chip are bonded together in a back-to-back bonding manner, and the off-chip non-volatile memory chip is bonded to the logic chip on either the front or back side.

[0015] Since both the off-chip volatile memory chip and the logic chip generate a lot of heat during operation, and the back side of the chip is mainly composed of a silicon substrate with high thermal insulation performance, the back-to-back bonding method can avoid the heat transfer between the two and thus avoid affecting the computing efficiency due to heat generation. At the same time, the back-to-back bonding allows the front side of the off-chip volatile memory chip to conduct heat directly to the outside, thereby achieving a better heat dissipation effect.

[0016] Optionally, a global controller is integrated on the logic chip; the global controller is connected to the off-chip volatile memory chip and the off-chip non-volatile memory chip respectively, and is configured to: perform dynamic data scheduling based on the bonding method between the off-chip volatile memory chip and the off-chip non-volatile memory chip and the logic chip, the task characteristics, and the storage medium attributes of the off-chip volatile memory chip and the off-chip non-volatile memory chip.

[0017] This application breaks through the limitations of single storage management by comprehensively sensing the physical interconnect architecture (bonding method), computing load characteristics (task characteristics) and physical media attributes (storage media), realizing dynamic data flow scheduling of software and hardware collaboration, and maximizing the overall processing efficiency of heterogeneous stacked systems.

[0018] Optionally, the global controller is configured to store high-bandwidth-demand task data to a memory chip that is bonded face-to-face with the logic chip.

[0019] This application utilizes the highest interconnect density and lowest parasitic loss characteristics of front-to-front (F2F) bonding to accurately allocate high-bandwidth-demand tasks to this physical path, ensuring that high-speed computing tasks (such as neural network weight loading) are not constrained by interface bandwidth, and significantly improving computing power density.

[0020] Optionally, the global controller is configured to store thermally sensitive task data to a memory chip bonded to the logic chip in a front-to-back or back-to-back manner.

[0021] This application leverages the physical advantages of back-to-front (B2F) or back-to-back (B2B) bonding on the physical layer spacing or substrate thermal insulation / heat dissipation characteristics to reduce the impact of high-heat areas in the logic layer on heat-sensitive data through spatial isolation, effectively extending chip life and reducing the signal transmission bit error rate caused by temperature fluctuations.

[0022] Optionally, the off-chip volatile memory chip, the off-chip non-volatile memory chip, and the logic chip are integrated sequentially in the vertical direction using three-dimensional stacking technology.

[0023] Optionally, the logic chip is a logic chip that performs computing tasks, and the logic chip integrates a digital in-memory computing unit.

[0024] Optionally, the logic chip is configured as a data cache chip for data caching.

[0025] Optionally, the logic chip is configured as an interface chip for implementing information interaction.

[0026] Optionally, the logic chip integrates at least one off-chip non-volatile memory chip controller, at least one off-chip volatile memory chip controller, and a global controller; the off-chip non-volatile memory chip controller is used to access the off-chip non-volatile memory chip, and the off-chip volatile memory chip controller is used to access the off-chip volatile memory chip; the global controller is used to manage the off-chip volatile memory chip controller, the off-chip non-volatile memory chip controller, and the digital in-memory computing unit, to realize dynamic adaptation of data storage scheme and computing data flow, so as to globally optimize storage capacity, data bandwidth, access latency, and computing efficiency.

[0027] Optionally, the off-chip volatile memory chip is a DRAM (Dynamic Random Access Memory) chip.

[0028] Optionally, in a large language model computation scenario, the off-chip volatile memory chip is used to store key-value cache data, activation values, and intermediate computation results.

[0029] Optionally, the off-chip non-volatile memory chip is Flash memory.

[0030] Optionally, the off-chip non-volatile memory chip is used to store Q-generated weights, K-generated weights, V-generated weights, Attention weights, and FFN weights.

[0031] Optionally, the off-chip volatile memory chip is composed of one off-chip volatile memory module, or is integrated by at least two off-chip volatile memory modules using three-dimensional stacking technology. The off-chip volatile memory modules are integrated face to face. The side of the structure formed by the three-dimensional stacking of the at least two off-chip volatile memory modules closer to the logic control layer is defined as the front side of the off-chip volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip volatile memory chip.

[0032] Optionally, the off-chip non-volatile memory chip is composed of one off-chip non-volatile memory module, or is integrated by at least two off-chip non-volatile memory modules using three-dimensional stacking technology; wherein, each of the off-chip non-volatile memory modules is integrated face to face, and the side of the structure formed by the three-dimensional stacking of the at least two off-chip non-volatile memory modules closer to the logic control layer is defined as the front side of the off-chip non-volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip non-volatile memory chip.

[0033] Optionally, the logic chip consists of a single logic module or is integrated from at least two logic modules using three-dimensional stacking technology.

[0034] Optionally, the three-dimensional stacking technology is one or more of through-silicon via (TSV) technology, hybrid bonding technology, or flip-chip technology.

[0035] Optionally, the digital in-memory computing unit is integrated with an on-chip volatile memory chip and a computing unit. The on-chip volatile memory chip is one or more of the following: SRAM (Static Random Access Memory), eDRAM (Embedded DRAM), DRAM (Dynamic Random Access Memory), Flash, MRAM (Magnetoresistive RAM), and ReRAM (Resistive RAM).

[0036] Optionally, the global controller is configured to perform fine-grained dynamic scheduling of data based on the interconnect characteristics of the bonding scheme of the three-dimensional stacking technology and the inherent properties of each storage medium.

[0037] Optionally, the global controller is configured to support the configuration of data storage mapping rules through upper-layer software.

[0038] Optionally, the off-chip volatile memory chip includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory chip controller, enabling the neural network processor to be accessed by external devices as a standard DRAM memory chip; the interface protocol satisfies one or more of LPDDR (Low Power DDR) 5, LPDDR6, HBM (High Bandwidth Memory) 2, HBM3, HBM3e, HBM4, GDDR (Graphics DDR) 5, GDDR6, GDDR7, DDR (Double Data Rate SDRAM) 5, and DDR6.

[0039] Optionally, the Flash includes an interface conforming to JEDEC or an industry-standard memory interface, and the Flash of the neural network processor can be accessed as a standard Flash memory chip. The protocol of the interface satisfies one or more of the following: SPI (Serial Peripheral Interface), QSPI (Quad SPI), Octal SPI (OSPI), eMMC (embedded Multi-Media Card), UFS (Universal Flash Storage), NVMe (Non-Volatile Memory express), and Parallel Flash.

[0040] To achieve the above-mentioned objectives, this application provides a control method for a three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor, which applies the three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor described above, and includes the following steps. Detect whether the attention module computation is triggered; and When the attention module is detected to be triggered to perform calculations, the off-chip non-volatile memory chip is scheduled to start data loading and the logic chip is scheduled to start preprocessing while performing a read operation on the off-chip non-volatile memory chip.

[0041] The technical advantage of this control method lies in its ability to successfully "hide" the physical access bottleneck of Flash memory during the computational preprocessing process through parallel scheduling across media, thus overcoming the performance bottleneck caused by media speed mismatch in three-dimensional stacked heterogeneous systems. Attached Figure Description

[0042] Figure 1A graph showing the relationship between the core performance metrics of a large language model and the scale of model parameters; Figure 2 A diagram of the storage system architecture of a traditional computer system; Figure 3 The performance and cost characteristics of each layer in the storage system architecture of traditional computer systems; Figure 4 This is a schematic diagram of a storage architecture based on a traditional computer system. Figure 5 A schematic diagram of the parameter transfer process for large language model computation based on traditional computer systems; Figure 6 A schematic diagram of the parameter transport process for large language model computation based on data offloading technology; Figure 7 This is a schematic diagram of a storage packaging structure based on Chiplet and Flip Chip technologies; Figure 8 This is a schematic diagram of a storage packaging structure based on three-dimensional integration technology; Figure 9 This is a schematic diagram of a storage packaging structure based on three-dimensional integration technology; Figure 10 A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention; Figure 11 A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention; Figure 12 This is a schematic diagram of one type of three-dimensional integrated bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of one type of three-dimensional integrated bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of one type of three-dimensional integrated bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 15 This is a schematic diagram of one type of three-dimensional integrated bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 16 This is a schematic diagram comparing the implementation of the present invention with the storage architecture of the prior art; Figure 17 This is a schematic diagram of the structure of the digital in-memory computing unit in an embodiment of the present invention; Figure 18 A schematic diagram illustrating the features of 3D integrated bonding for different interface positional relationships; Figure 19This is a schematic diagram of one of the storage schemes for computing large language models provided in the embodiments of the present invention; Figure 20 This is a flowchart illustrating a computation scheduling scheme for a global controller of a neural network processor provided in an embodiment of the present invention. Figure 21 This is a timing diagram of the parallel pipeline coordination mechanism of the global controller of the neural network processor provided in an embodiment of the present invention; Figure 22 This is a schematic diagram illustrating the steps of the control method for a neural network processor provided in an embodiment of the present invention. Figure 23 This is a schematic diagram of the structure of the off-chip volatile memory chip extension of the neural network processor provided in an embodiment of the present invention; Figure 24 This is a schematic diagram of the structure of the off-chip non-volatile memory chip extension of the neural network processor provided in an embodiment of the present invention; Figure 25 This is a schematic diagram of the logic chip extension of the neural network processor provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] Due to differences in process compatibility between non-volatile memory chips, volatile memory chips, and computing logic (such as the incompatibility between advanced processes in computing logic and mature processes in memory chips, and the high integration difficulty caused by different material systems), considering factors such as area utilization, chip yield, and process feasibility, adopting an integrated solution would lead to problems such as excessive chip area redundancy, a significant decrease in yield, and insufficient process adaptability. Therefore, if... Figure 4 As shown, traditional computer systems break it down into multiple chips for independent implementation.

[0045] Based on traditional computer system storage architecture, storage systems can be divided into four categories: off-chip non-volatile memory chips (such as remote storage, SSD, NAND Flash, ReRAM, etc.), off-chip volatile memory chips (such as DRAM), on-chip volatile memory chips (such as SRAM), and registers; for example... Figure 5As shown, when deploying large language model computation in an existing system, the process is usually as follows: After the AI ​​task is started, the main control chip first loads as much large model parameter data as possible from the non-volatile memory chip to the off-chip volatile memory chip, then moves the data to be computed from the off-chip volatile memory chip to the on-chip volatile memory chip, then selects the data that needs to be computed immediately and sends it to the register, and finally completes the computation in the computing unit.

[0046] However, as analyzed above, the inference process of LLM (Large Language Model) under traditional architecture still faces severe capacity and bandwidth bottlenecks. To effectively alleviate this problem, the industry has successively explored several technical solutions, as follows: First, data offloading technology. For example... Figure 6 As shown, its core logic is "trading communication for capacity": compared to the fixed data loading path of the traditional deployment process in Figure 5, this technology unloads data that is not currently needed in the computing stage from high-speed on-chip storage (such as SRAM) and stores it back to off-chip volatile memory chips (such as DDR) or off-chip non-volatile memory chips (such as SSD); when subsequent computing requires the data, it is retrieved from the corresponding storage medium and reloaded onto the chip. This approach can avoid irrelevant data occupying scarce on-chip storage resources for a long time, thereby directly alleviating the storage capacity bottleneck, and is especially suitable for LLM tasks that process long sequence inputs (such as long document understanding and multi-turn dialogue).

[0047] However, the core limitation of data offloading technology lies in the fact that it does not change the traditional "compute-storage separation" architecture of memory chips in computing systems. While this technology can alleviate storage capacity bottlenecks by "trading communication for capacity" and even relieve bandwidth pressure to some extent, this process also comes at a significant cost: frequent data offloading and retrieval generate a large number of on-chip and off-chip communication needs, which in turn keeps the system continuously subject to severe bandwidth constraints. Even if the system can therefore support neural networks with a larger number of parameters, the frequent transfer of large amounts of data between on-chip and off-chip will seriously drag down computational efficiency, ultimately resulting in significant limitations on the overall inference efficiency of large models.

[0048] Second, high-density packaging technology. For example... Figure 7 As shown, this technology, through processes such as Chiplet and Flip Chip, breaks the physical limitations of traditional planar packaging, integrating multiple functional chips (such as logic chips and cache chips) into a single package in a more compact manner. Its core function is to shorten the physical transmission distance across memory levels (such as on-chip cache and off-chip memory, or cache between different computing units), reducing data transmission latency and improving overall bandwidth through parallelized transmission paths, thereby alleviating the problem of low "data moving" efficiency.

[0049] High-density packaging technology is effective in alleviating the bandwidth bottleneck of traditional memory chip architectures, but it has two key limitations. First, it does not break through the underlying design of traditional memory chip architectures, but only improves data transmission efficiency by optimizing the packaging form, so the core problem of limited storage capacity is not solved. Second, its performance improvement has a clear ceiling, and it will continue to be limited by the speed limit of the IO interface, making it difficult to achieve unlimited bandwidth breakthroughs.

[0050] Third, three-dimensional integration technology. For example... Figure 8 , Figure 9 As shown, this technology overcomes the spatial limitations of two-dimensional packaging through key processes such as TSV (Through Silicon Via) and Hybrid Bonding, enabling vertical stacking integration of logic chips and memory chips. For example, the widely used HBM (High Bandwidth Memory) technology uses TSV to vertically stack multiple DRAM chips next to the logic chip, forming a tightly coupled "compute-memory" structure. Compared to traditional DDR memory chips, HBM's bandwidth can be increased several times (e.g., HBM3 has a bandwidth of 819GB / s), while significantly reducing the area occupied by the planar circuit board, fundamentally optimizing the "memory-compute" interaction efficiency required for LLM inference.

[0051] 3D integration technology, with its 3D stacking structure and high-density interconnect design, can effectively address the capacity shortage and bandwidth limitations faced by traditional storage architectures. However, existing implementation solutions for this technology still have significant limitations. Current mainstream solutions (such as HBM high-bandwidth memory chips and NAND Flash memory) are essentially still limited to the stacking and interconnection of memory chips of the same type and level—they have neither broken through the underlying framework of traditional memory chip architectures nor achieved a synergistic breakthrough in capacity and bandwidth through a unified storage structure.

[0052] Meanwhile, current optimization solutions for storage architecture mostly focus on simply expanding capacity and increasing bandwidth, neglecting the core characteristic of computer systems: the dynamic transfer of bottlenecks. When capacity and bandwidth bottlenecks are alleviated through technical means, the processing power of computing units often becomes a new performance constraint. For example, in large model inference scenarios, even if high-speed data storage and transmission are achieved through technologies such as HBM and Chiplet, if the computing power of logic chips such as GPUs / TPUs cannot match the data throughput, resource idleness will still occur, resulting in "data waiting for computation," making it difficult to improve the overall system efficiency. More importantly, existing solutions generally adopt a "fragmented optimization" approach, failing to build a unified architecture that coordinates computing power, storage efficiency, and bandwidth performance. This fails to break the cycle of "solving old bottlenecks and creating new bottlenecks" at the system level, making it difficult to achieve synergistic breakthroughs in the performance of various components.

[0053] Therefore, existing solutions have failed to break free from the constraints of the underlying framework of traditional memory chip architecture. The core problem lies in the inability to achieve coordinated optimization of capacity, bandwidth, and computing power based on a unified storage structure. This structural shortcoming directly results in significant performance limitations and severe implementation challenges for large language models when deployed for inference in systems built on existing solutions.

[0054] If the applicant is aware of the aforementioned technical issues, Figure 10 and Figure 11 As shown, this embodiment provides a neural network processor based on in-memory computing and three-dimensional stacking technology, including: an off-chip volatile memory chip 100 connected to a logic chip 300 for information interaction with the logic chip; an off-chip non-volatile memory chip 200 connected to the logic chip 300 for information interaction with the logic chip; and a logic chip 300 configured to perform any combination of the following functions: performing computational tasks, performing data caching, and performing information interaction; the off-chip volatile memory chip 100, the logic chip 300, and the off-chip non-volatile memory chip 200 are integrated using three-dimensional stacking technology, with the off-chip volatile memory chip 100 located at the top in the vertical direction and capable of being connected to an external heat dissipation structure (not shown).

[0055] In this embodiment, the off-chip volatile memory chip 100 (e.g., DRAM chip) is placed on the top layer of the three-dimensional stacked structure. The advantages of this layout are mainly twofold: First, compared to the logic chip 300, which has a high heat density when undertaking a large number of computing tasks, and the off-chip non-volatile memory chip 200 (e.g., Flash), which has a high storage density but only generates local heat, the off-chip volatile memory chip 100 has a higher heat generation and a longer thermal cycle. Placing it on the top layer can reduce the thermal impact of the high-heat chip on the surrounding modules. Second, the top layer position can more directly contact the heat dissipation structure outside the package (e.g., heat sink, thermal pad, etc.), resulting in a shorter and more efficient heat conduction path. This allows the heat from the off-chip volatile memory chip 100 and the lower-layer chips to be quickly dissipated from the top layer, avoiding the problem of "heat from the lower layer accumulating upwards and heat dissipation being blocked on the top layer" in traditional stacking. This layout ensures that the off-chip volatile memory chip 100 operates stably in low-temperature environments (avoiding increased access latency or performance degradation caused by high temperatures), and also provides a smoother heat dissipation channel for the lower-level logic chip 300 and the off-chip non-volatile memory chip 200, ultimately improving the thermal stability and long-term operational reliability of the entire system.

[0056] Optionally, the off-chip volatile memory chip 100, logic chip 300 and off-chip non-volatile memory chip 200 are integrated in the vertical direction using three-dimensional stacking technology.

[0057] In this embodiment, the logic chip 300 is placed between the off-chip volatile memory chip 100 and the off-chip non-volatile memory chip 200. This design has two advantages: First, because the off-chip volatile memory chip 100 and the logic chip 300 are positioned adjacent to each other, they do not need to be penetrated by the three-dimensional vias of the off-chip non-volatile memory chip 200. This allows for a larger storage capacity of the off-chip non-volatile memory chip 200 within the same area, thus improving computational efficiency. Second, because the off-chip non-volatile memory chip 200 and the logic chip 300 are positioned adjacent to each other, and because the off-chip non-volatile memory chip 200 has a higher storage density, the logic chip 300 (Logic...)... The physical close integration of the Die (a type of memory chip) with the off-chip non-volatile memory chip 200 offers additional advantages for certain large-scale language model (LLM) tasks, especially the storage requirements of key-value cache (KVCache) data in the prefill stage and long sequence inference. Specifically, it provides ample storage space for LLM tasks with extremely large storage capacity requirements, thereby overcoming the challenge of limited capacity of traditional DRAM. In addition, in long sequence inference, the KV Cache is continuously growing, and the high capacity of the off-chip non-volatile memory chip 200 can avoid frequent swapping of the KV Cache between the off-chip volatile memory chip 100 and the off-chip non-volatile memory chip 200, reducing complexity and additional latency.

[0058] Optionally, such as Figure 12 and Figure 13 As shown, the off-chip volatile memory chip 100 and the logic chip 300 are bonded together in a front-to-front bonding manner, and the off-chip non-volatile memory chip 200 is bonded to the logic chip 300 in either the front or back position.

[0059] The off-chip volatile memory chip 100 and the logic chip 300 are bonded together in a face-to-face bonding manner. Considering that the memory access latency of the off-chip volatile memory chip 100 is on the order of nanoseconds (typically tens of nanoseconds), its memory access efficiency is greatly affected by the bit width and transmission rate of the external I / O interface. When the two are bonded in a face-to-face (F2F) manner, F2F bonding supports extremely high-density I / O interconnects, significantly increasing the interface bit width and supporting high-bandwidth data interaction. Compared with bonding schemes that require penetrating the silicon substrate and introducing parasitic capacitance and resistance, face-to-face bonding significantly reduces link loss, ensures the signal integrity of the off-chip volatile memory under high-speed transmission, and reduces the error correction burden. In addition, the core memory access characteristics of the off-chip non-volatile memory chip determine its adaptation requirements for interconnect bonding methods: firstly, the memory access latency is relatively high, typically on the order of 30-50μs microseconds; secondly, the amount of data read in a single operation is fixed, generally 4-16KB. From a performance bottleneck perspective, the memory access efficiency of off-chip non-volatile memory chips (such as Flash) is primarily constrained by their internal operating mechanisms, rather than the performance of external I / O interconnect links. This conclusion can be fully verified from two key dimensions: I / O density and I / O speed. From the perspective of I / O density, the core value of front-to-front bonding lies in improving I / O concurrent interaction capabilities to meet high-frequency data transmission demands. However, Flash itself does not require high-frequency, high-concurrency I / O interaction. Even with high-density F2F bonding, its memory access efficiency cannot be linearly improved; instead, it leads to ineffective redundancy and waste of I / O resources. From the perspective of I / O speed, compared to back-side bonding, it requires through-silicon vias (TSVs) to penetrate the silicon substrate. TSVs inevitably introduce parasitic capacitance and resistance, which affects high-speed I / O links, leading to reduced transmission rates and, in severe cases, even threatening data security.

[0060] Therefore, on the one hand, the face-to-face bonding between the off-chip volatile memory chip 100 and the logic chip 300 increases the interface bit width, supports high-bandwidth data interaction, and meets the high-speed memory access and high-bandwidth data interaction requirements of the off-chip volatile memory chip 100. On the other hand, the off-chip non-volatile memory chip 200, due to its memory access latency reaching the microsecond level, has extremely low sensitivity to I / O rate and is not negatively affected by the face-to-face bonding between the off-chip volatile memory chip 100 and the logic chip 300.

[0061] Optionally, such as Figure 14 and Figure 15 As shown, the off-chip volatile memory chip 100 and the logic chip 300 are bonded together in a back-to-back bonding manner, and the off-chip non-volatile memory chip 200 is bonded to the logic chip 300 in either the front or back position.

[0062] Since both off-chip volatile memory chips and logic chips generate significant heat during operation, and the back side of the chips is mainly composed of silicon substrates with high thermal insulation properties, back-to-back bonding can prevent heat transfer between the two, thus avoiding the impact on computing efficiency due to heat generation. At the same time, back-to-back bonding allows the front side of the off-chip volatile memory chip to conduct heat directly to the outside, achieving a better heat dissipation effect.

[0063] Optionally, refer to Figure 11 A global controller 340 is integrated on the logic chip 300. The global controller 340 is connected to the off-chip volatile memory chip 100 and the off-chip non-volatile memory chip 200 respectively, and is configured to perform dynamic data scheduling based on the bonding method, task characteristics and storage medium attributes of the off-chip volatile memory chip 100 and the off-chip non-volatile memory chip 200 with the logic chip 300.

[0064] This implementation breaks through the limitations of single storage management by comprehensively sensing the physical interconnect architecture (bonding method), computing load characteristics (task characteristics), and physical media attributes (storage media), realizing dynamic data flow scheduling through software and hardware collaboration, and maximizing the overall processing efficiency of heterogeneous stacked systems.

[0065] Optionally, the global controller 340 is configured to store high-bandwidth-demand task data to a memory chip that is bonded face-to-face with the logic chip 300.

[0066] This implementation utilizes the highest interconnect density and lowest parasitic loss characteristics of front-to-front (F2F) bonding to accurately allocate high-bandwidth-demand tasks to this physical path, ensuring that high-speed computing tasks (such as neural network weight loading) are not constrained by interface bandwidth, and significantly improving computing power density.

[0067] Optionally, the global controller 340 is configured to store thermally sensitive task data to a memory chip bonded to the logic chip 300 using a front-to-back or back-to-back bonding method.

[0068] This embodiment utilizes the physical advantages of back-to-front (B2F) or back-to-back (B2B) bonding on the physical layer spacing or substrate thermal insulation / heat dissipation characteristics. Through spatial isolation, it reduces the impact of heat-sensitive data on the high-heat-generating areas of the logic layer, effectively extending chip life and reducing the signal transmission bit error rate caused by temperature fluctuations.

[0069] Optionally, in some embodiments, the off-chip volatile memory chip 100, the off-chip volatile memory chip 100 and the logic chip 300 may be integrated in the vertical direction using three-dimensional stacking technology. All of the above embodiments are within the protection scope of the present invention.

[0070] Optionally, the off-chip volatile memory chip 100 is a DRAM chip. For ease of explanation, a DRAM chip will be used as an example of the off-chip volatile memory chip 100 in the following text.

[0071] Optionally, the off-chip non-volatile memory chip 200 is Flash. For ease of explanation, Flash is used as an example of the off-chip non-volatile memory chip 200 in the following text.

[0072] The neural network processor provided in this embodiment does not require the DRAM chip to be penetrated by the three-dimensional vias of the Flash memory because the DRAM chip and the logic chip are arranged adjacent to each other. This design allows the DRAM chip to have a larger storage capacity in the same area, thus improving the computing efficiency.

[0073] The neural network processor provided in this embodiment has additional advantages for specific large language model (LLM) tasks, especially the storage requirements of KV cache in the prefill stage and long sequence inference, due to the adjacent arrangement of Flash and logic chips and the high storage density of Flash. Specifically, it provides sufficient storage space for LLM tasks with extremely large storage capacity requirements, thereby overcoming the challenge of limited capacity of traditional DRAM. In addition, in long sequence inference, KV cache is continuously growing, and the high capacity of Flash can avoid frequent swapping of KV cache between DRAM and Flash, reducing complexity and additional latency.

[0074] Optionally, the logic chip 300 is a computing chip that performs computing tasks. The logic chip integrates a digital in-memory computing unit 310 for performing computing tasks. In the following text and figures, for ease of explanation, the logic chip 300 is configured as a computing chip as an example. Based on the spirit of the present invention, it can be understood that the logic chip 300 can also be configured as a data cache chip and interface chip as described below to achieve functions adapted to the application scenario.

[0075] Optionally, the logic chip is a data cache chip used for data caching, which temporarily stores data between computing and storage, improving access efficiency and reducing latency and power consumption caused by frequent access to external storage.

[0076] Optionally, the logic chip is an interface chip used to realize information interaction, and is used to realize information interaction and protocol conversion between different chips or systems.

[0077] Optionally, in a neural network processor, the logic chip 300 can be configured as multiple, such as three, which are respectively configured as a logic chip, an interface chip, and a data cache chip. Multiple chips can be integrated with each other using three-dimensional stacking technology to achieve flexible and diverse functions.

[0078] It should be noted that the off-chip volatile memory chip 100 and the off-chip non-volatile memory chip 200 can also be implemented using other storage media / memory chips that meet the requirements, and this embodiment does not limit them.

[0079] The neural network processor provided in this embodiment can achieve at least the following technical effects: First, it can effectively solve the problem of heat accumulation in traditional three-dimensional stacking technology by rationally planning the order of chip stacking. In this embodiment, the off-chip volatile memory chip 100 is configured on the top layer of the three-dimensional stacking structure. The advantages of this layout are twofold: First, compared with the logic chip 300, which has a high heat density when undertaking a large number of computing tasks, and the off-chip non-volatile memory chip 200, which has a high storage density but only generates local heat, the off-chip volatile memory chip 100 has a higher heat generation and a longer thermal cycle. Placing it on the top layer can reduce the thermal impact of the high-heat chip on the surrounding modules. Second, the top layer position can more directly contact the heat dissipation structure outside the package (such as heat sink, thermal pad, etc.), and the heat conduction path is shorter and more efficient. It can quickly dissipate the heat conducted from the off-chip volatile memory chip 100 and the lower layer chips to the top layer, avoiding the problem of "heat accumulation on the lower layer and obstruction of heat dissipation on the top layer" in traditional stacking. This layout ensures that the off-chip volatile memory chip 100 operates stably in low-temperature environments (avoiding increased access latency or performance degradation caused by high temperatures), and also provides a smoother heat dissipation channel for the lower-level logic chip 300 and the off-chip non-volatile memory chip 200, ultimately improving the thermal stability and long-term operational reliability of the entire system.

[0080] Second, such as Figure 16 As shown, this implementation constructs a disruptive and innovative storage architecture, completely breaking through the inherent framework of traditional storage chips' "layer-by-layer cascading and unidirectional data transport"—traditional storage systems rely on a fixed hierarchy of "on-chip cache → off-chip storage → external storage," requiring data to be transmitted level by level to meet computing demands. This not only generates a large amount of redundant data transport overhead but also leads to bandwidth limitations and latency accumulation due to the large physical distance between levels, becoming a core efficiency bottleneck for computationally intensive tasks such as large language models. The architecture provided by this implementation achieves end-to-end optimization through the synergistic innovation of three-dimensional stacking and in-memory computing technology: on the one hand, three-dimensional integration technology directly stacks off-chip storage chips (DRAM / Flash) and logic chips in three dimensions, shortening the physical distance between them to the micrometer level. This improves off-chip storage access bandwidth through high-density bonding and reduces cross-chip transmission losses to lower latency; on the other hand, in-memory computing technology integrates on-chip storage chips (such as DCIM based on 6T SRAM) into a single layer. The deep integration of the array with the computing unit enables operations such as multiplication and accumulation to be performed directly within the on-chip storage chip, fundamentally eliminating the latency and energy consumption of on-chip data transfer. Ultimately, this builds a "low-latency, high-bandwidth, low-power" storage-computing collaborative system, completely breaking free from the performance constraints of traditional hierarchical architectures.

[0081] Third, in terms of capacity, the storage capacity of traditional architectures relies on planar expansion (increasing chip area or adding independent storage modules), which is easily limited by packaging space and power consumption budget. Moreover, after capacity is increased, performance is easily wasted due to bandwidth mismatch. This implementation achieves a leapfrog, bottleneck-free capacity expansion through "three-dimensional vertical stacking + high-density media selection": using NAND Flash as the core high-density storage medium, the number of three-dimensional stacked Flash chips can be increased through bonding technology—for example, an 8-layer stacked Flash module / unit can achieve a single module capacity of 60-128GB in the same planar area as traditional 2D packaging (traditional 2D Flash in the same area is only 10-20GB); if further expansion is needed, the number of Flash stacking layers can be increased in the three-dimensional architecture (current technology supports 200+ layers of 3D NAND), or multiple sets of Flash-three-dimensional stacked units can be connected in parallel, and the expansion process does not occupy additional planar space.

[0082] In terms of bandwidth, the bandwidth of traditional 2D architectures depends on interface protocol upgrades (such as DDR5→DDR6), with limited single-channel bandwidth improvement (approximately 30%-50% per generation), and multi-channel expansion is easily limited by motherboard wiring density. This implementation achieves efficient and scalable bandwidth expansion through "three-dimensional interconnect optimization + multi-chip parallelism," such as increasing the number of three-dimensional stacks of high-bandwidth media (DRAM / SRAM), for example, vertically integrating two groups of DRAM chips with logic chips via F2F bonding. Each group of DRAM chips provides 200-300GB / s bandwidth, and the two groups in parallel can achieve a total bandwidth of 400-600GB / s. At the same time, the micron-level wiring capability of the silicon interposer can be used to construct a parallel interconnect channel of "multi-memory chip-logic chip," avoiding the wiring congestion problem of traditional 2D architectures. In addition, this implementation can prioritize F2F hybrid bonding for high-frequency interaction links (such as Logic Die and DRAM), reducing interconnect latency to sub-nanosecond levels, further releasing bandwidth potential.

[0083] With the storage and bandwidth issues resolved through the aforementioned solutions, computing power has become a new bottleneck in the systemic problem of storage-bandwidth-computing power. The expansion of computing power in traditional architectures relies on increasing the area of ​​logic chips or upgrading process nodes, which is costly and easily limited by heat dissipation (increased area leads to increased heat density). At the same time, after the computing power is increased, it is easy to cause "computing power idle" due to insufficient storage bandwidth. This implementation method, while retaining the process node improvement, uses Compute-In-Memory (CIM) technology as its core to achieve high-density, low-power expansion of computing power. On the one hand, the number of CIM macrocells can be increased modularly, with a DCIM array based on 6T SRAM as the basic unit. Each unit can provide 100-150 TOPS of computing power. By increasing the number of CIM unit layers through three-dimensional stacking (such as 4-layer CIM stacking), a total computing power of 400-600 TOPS can be achieved, and the computing power density per unit area is 2-3 times that of traditional GPUs. Upgrading the logic chip process node: upgrading the Logic Die from 14nm to 3nm, and coordinating with the process optimization of CIM units, the computing power per unit area can be further increased by 50%-80%. Specifically, in this implementation method, the computing unit is integrated into the on-chip volatile memory chip on the logic chip 300, so that the calculation data does not need to be transferred to the register through the on-chip volatile memory chip, but can be calculated directly inside the on-chip volatile memory chip. The on-chip volatile memory chip with integrated computing units is designed as a Digital Computing-In-Memory (DCIM) unit. For example... Figure 17 As shown, the structure of the digital in-memory computing unit 310 mainly includes a DCIM array based on 6T SRAM, a word line decoder (WL DEC), a word line buffer (WL DRV), read / write circuitry, and a controller. The DCIM array consists of C DCIM columns, each containing R subarrays and an adder tree. Each subarray integrates 6T SRAM bit cells and a Local Read-Out and Compute Unit (LRCC), storing N-bit weights per row. Functionally, the DCIM macro can perform matrix-vector multiplication (MVM) multiplication-accumulation (MAC) calculations: the operating mode is switched via the M signal; when M is low, the N-bit weights in the subarray are multiplied by a 1-bit input, read and latched by the LRCC, accumulated by the adder tree, and then sent to a shift-and-accumulator to extend the time-dimensional input precision, ultimately completing a full MVM operation with K-bit input and N-bit weights, while also supporting the storage and efficient computation of multi-weight matrices.

[0084] In summary, the neural network processor provided in this embodiment achieves direct contact between the high-heat chip and the external heat dissipation structure by placing the off-chip volatile memory chip (DRAM) on the top layer of the stack, thus shortening the heat conduction path; effectively reducing the temperature rise of the logic chip and the non-volatile memory chip, avoiding heat accumulation, and improving the thermal stability and long-term reliability of the system.

[0085] The storage architecture is reconstructed (end-to-end bandwidth optimization), breaking the traditional "cache-external storage-external storage" layered model. Through three-dimensional bonding, DRAM / Flash and logic chips are directly vertically interconnected, shortening the interconnection distance to the micrometer level. At the same time, combined with in-memory computing (CIM) technology, data can be multiplied and accumulated within the storage array, reducing data transfer latency and energy consumption, and building a low-latency, high-bandwidth data path.

[0086] Storage capacity expansion and density improvement are achieved by using "three-dimensional vertical stacking + high-density NAND Flash" to achieve three-dimensional expansion of storage capacity, increasing capacity by 3 to 6 times without occupying additional planar area; it supports hundreds of layers of Flash stacking or multiple units in parallel, breaking through the limitations of packaging space.

[0087] Bandwidth enhancement and interconnect optimization are achieved through F2F bonding and parallel interconnection of multiple DRAM chips to achieve linear bandwidth superposition (200–300 GB / s per channel, total bandwidth up to 400–600 GB / s); with the help of the micron-level wiring capability of silicon interposer, I / O density is significantly improved and interconnect latency is reduced to sub-nanosecond level.

[0088] The improved computing power density and energy efficiency are achieved by embedding DCIM (Digital In-Memory Computing) units based on 6T SRAM into the logic chip, realizing "storage as computing". By increasing the number of CIM macrocell layers through three-dimensional stacking and upgrading advanced process nodes (14nm→3nm), the computing power density per unit area is increased to 2 to 3 times that of traditional GPUs, and the energy efficiency ratio is improved by about 50% to 80%.

[0089] The neural network processor provided in this embodiment forms a collaborative closed loop in a three-dimensional architecture, integrating storage, bandwidth, and computing power: high-density storage provides sufficient data; high-bandwidth pathways ensure real-time supply; and the in-memory computing unit efficiently executes computations, achieving a performance balance of "low latency, high computing power, and low power consumption" at the system level.

[0090] Through a systematic design of "thermal structure optimization + three-dimensional interconnection + in-memory computing integration", a comprehensive breakthrough has been achieved in four dimensions of neural network processors: thermal management, data path, storage capacity and computing power density. This results in a new computing architecture that is highly integrated, high-bandwidth, scalable and low-power, and suitable for computing-intensive scenarios such as large language models, visual computing and edge inference.

[0091] In the field of 3D integration technology, each chip (die / core) includes: Face: This refers to the side where the metal interconnect layer is located, including the top layer wiring, solder joints (micro-bumps), or redistribution layer (RDL).

[0092] Back side: This is the silicon substrate side, usually the bare silicon surface, which is electrically connected to the front side through thinning, opening, or TSV (through silicon via).

[0093] like Figure 18 As shown, the characteristics of 3D integration bonding are demonstrated when different interface positional relationships are used. The interface positional relationships in 3D integration are mainly of three types: face to face (F2F), face to back (F2B), and back to back (B2B).

[0094] In front-to-front mode, interconnect density and bandwidth are high, RDL rerouting is optional, cost (yield) is low, signal integrity is high, but thermal management is difficult. In front-to-back mode, interconnect density and bandwidth are low, RDL rerouting is necessary, cost (yield) and signal integrity are both moderate, and thermal management is moderate. In back-to-back mode, interconnect density and bandwidth are low, RDL rerouting is necessary, cost (yield) is high, signal integrity is low, but thermal management is relatively easy.

[0095] Despite the superior overall performance of F2F, in a three-layer stacking scenario, if two layers are to be bonded using F2F, the third layer will inevitably form an F2B structure with one of the layers. Therefore, three-layer stacking requires comprehensive consideration of hardware factors such as interconnect density, cost, signal integrity, and thermal management to determine the specific arrangement of F2F, F2B, and B2B, resulting in higher design complexity compared to two-layer stacking.

[0096] Specifically, in response to the performance requirements of large language model inference tasks (such as long sequence generation and high-concurrency query), the global controller in this embodiment can combine the interconnection characteristics of the bonding scheme (such as bandwidth and thermal efficiency) with the inherent properties of the storage medium (such as capacity and latency) to achieve fine-grained dynamic management of the data storage scheme. Specifically, targeted scheduling will be implemented based on the different hardware resource requirements of each task: high-bandwidth task data (such as key-value pair cache in the attention mechanism) will be stored in a storage chip bonded to the logic chip 300 using an F2F connection. This type of storage chip can match the high-frequency data interaction requirements due to its high interconnect bandwidth characteristics; heat-sensitive task data (such as intermediate results of large-scale matrix operations) will be stored in a storage chip bonded to the logic chip 300 using an F2B or B2B connection. This type of bonding scheme has better thermal management capabilities and can avoid local overheating from affecting data stability; at the same time, for the differentiated requirements of storage capacity and latency, non-real-time data requiring large-capacity storage (such as pre-trained weight backups) will be stored in an off-chip non-volatile storage chip, while latency-sensitive real-time computing data (such as the embedding vector of the input sequence) will be stored in an off-chip volatile storage medium, forming a storage architecture that precisely matches "features and requirements".

[0097] In this implementation, computing power and storage / bandwidth work in tandem. When expanding computing power, the bonding methods between different dies are flexibly adjusted according to actual needs, ensuring the bonding method matches the application scenario. For example, F2F bonding increases the bandwidth of off-chip volatile memory chips, while B2B bonding increases the capacity of off-chip non-volatile memory chips, ensuring a match between "computing power enhancement and data supply" and avoiding bottlenecks. By using F2F bonding between high-performance chips and F2B / B2B bonding between low-speed or peripheral chips, optimal allocation of signal paths and bandwidth resources is achieved. Flexible design of different bonding surface combinations (such as F2B and B2B) allows high-power chips to directly expose their back surfaces to enhance heat dissipation, thereby reducing overall thermal resistance. Design flexibility and compatibility are enhanced, allowing for flexible adjustment of bonding order and interface orientation according to system architecture (such as Flash–Logic–DRAM), compatibility with different process nodes and technology platforms, and support for heterogeneous chip integration. Balancing yield and cost, the hybrid bonding structure effectively reduces manufacturing complexity and yield loss, achieving an optimal trade-off between performance and cost. Signal integrity optimization is achieved by concentrating high-frequency interconnects at the F2F interface and transmitting low-speed interconnects via F2B or TSV, shortening signal paths, reducing crosstalk and delay, and improving overall system signal quality. The neural network processor provided in this embodiment breaks through the interconnect limitations of traditional two-layer stacking, offering a configurable three-dimensional integration framework for multi-layer heterogeneous chips. It achieves a system-level balance with adjustable overall performance, significantly improving the performance density and manufacturability of three-dimensional integrated systems.

[0098] Optionally, continue to refer to Figure 11 The logic chip 300 integrates at least one off-chip non-volatile memory chip controller 330, at least one off-chip volatile memory chip controller 320, and a global controller 340. The off-chip non-volatile memory chip controller 330 is used to access the off-chip non-volatile memory chip 200, and the off-chip volatile memory chip controller 320 is used to access the off-chip volatile memory chip 100. The global controller 340 is used to manage the digital in-memory computing unit 310, the off-chip non-volatile memory chip controller 330, and the off-chip volatile memory chip controller 320 of the logic chip 300, so as to realize the dynamic adaptation of data storage scheme and computing data flow, so as to globally optimize storage capacity, data bandwidth, access latency, and computing efficiency.

[0099] Furthermore, the global controller 340 can be configured to be software programmable, supporting storage strategies defined through upper-layer software. Regardless of the aforementioned bonding structure used in this implementation, different data storage mapping rules (such as task-media binding relationships and data migration priorities) can be configured through software, and performance tests can be conducted in conjunction with actual inference scenarios (such as bandwidth utilization, latency fluctuations, and thermal loss monitoring). Ultimately, the optimal storage solution suitable for the current task type is adaptively selected, ensuring that large language model inference can achieve maximum storage-computation collaborative efficiency under different hardware configurations.

[0100] Specifically, the global controller 340, the off-chip non-volatile memory chip controller 330, and the off-chip volatile memory chip controller 320 can be implemented by a CPU or other existing control modules or combinations of control modules capable of implementing the functions mentioned in this embodiment. This embodiment does not limit this.

[0101] Figure 19 illustrates a configurable storage scheme for large language model computation. Its core is a precise hierarchical matching based on data characteristics and storage medium capabilities, using an off-chip volatile memory chip 100 as a DRAM chip and an off-chip non-volatile memory chip 200 as Flash as an example. For the highly dynamic KV Cache (requiring high-frequency updates and random access) and the activation values ​​and intermediate calculation results with high real-time requirements in large language model inference, this implementation utilizes the low latency and high random access throughput of DRAM to store them in DRAM, ensuring efficient read / write and update efficiency for dynamic data. However, the Q-generation weights, K-generation weights, V-generation weights, Attention weights, and FFN weights, with their large number of parameters, have high storage capacity requirements and relatively low access frequency. Therefore, this invention can store them in Flash, leveraging the large capacity and non-volatile characteristics of Flash to meet storage needs. This solution, by differentiating the attributes of "dynamic calculation data - static weight data" and adapting the media, ensures both the response speed of the dynamic calculation process and meets the storage capacity requirements of large-scale weights, thus achieving efficient utilization of storage resources and cost balance in large language model inference scenarios.

[0102] During the large language model calculation process, the global controller 340 provided in this embodiment is configured to dynamically allocate the off-chip volatile memory chip controller 320, the off-chip non-volatile memory chip controller 330, and the digital in-memory computing unit 310 with DCIM as the core, based on the current three-dimensional integrated bonding structure (such as F2F / F2B / B2B) and storage medium distribution, so as to achieve precise adaptation of hardware resources and computing tasks.

[0103] Optionally, the global controller 340 is configured to support the configuration of data storage mapping rules via upper-layer software.

[0104] Figure 20 shows a typical example of this scheduling logic: The neural network processor provided in this embodiment places static weights (or cold weights) in Flash memory (red box) with optimal capacity but high latency, places dynamic activations / intermediate states in low-latency DRAM chips (green box), and places the actual execution operators (matrix multiplication, Softmax, weighted sum, projection, FFN) in low-latency, high-computing-power logic chips (black box). The global controller 340 schedules the three to work together, minimizing cross-chip transmission and latency through caching / reuse and parallelization.

[0105] Since the generation of Q and K requires access to weight data stored in Flash (such as Q / K generation weights), the global controller 340 schedules the logic chip and Flash to work together to calculate Q and K. The generation of V involves more dynamic data interaction (such as combining with activation values), so it is executed by the logic chip in conjunction with the DRAM chip, leveraging the low latency of DRAM to ensure the real-time nature of V generation. In the Attention calculation phase, the logic chip first calculates similarity based on Q and K, then performs non-linear normalization operations such as Softmax, and finally performs a weighted fusion with the previously generated V, completing the core computational flow of the Attention mechanism.

[0106] Further integration Figure 21 As can be seen, the scheduling scheme of this implementation can deeply explore hardware characteristics and build an efficient parallel pipeline coordination mechanism: In view of the hardware attribute of Flash having a fixed read latency, the global controller will simultaneously start two key operations when triggering Flash to read Q / K weight data - first, scheduling the DRAM controller to perform the dynamic data (such as activation value, intermediate variable) required for V generation to access, and second, starting the logic chip 300 to carry out preliminary basic operations (such as activation value normalization, data format conversion), so as to avoid hardware resource idleness through "parallel startup".

[0107] Specifically, when the global controller 340 triggers an Attention Block calculation, the system needs to read the weight parameters of Q / K from Flash.

[0108] Flash access has a fixed latency window (tens to hundreds of nanoseconds). Traditional architectures will only start subsequent operations after the Flash is read, resulting in wasted time waiting.

[0109] This implementation uses a time-parallel scheduling mechanism to start two tasks in parallel while initiating a Flash read operation: DRAM controller startup → Preloads the dynamic data (activation value, intermediate state) required to generate V.

[0110] The logic chip initiates preprocessing → activates existing inputs to perform normalization (Norm) and data format conversion (Quant / Dequant).

[0111] With this design, the Flash, DRAM, and logic chips work in parallel, and the Flash delay window is fully filled.

[0112] Since the sum of the access latency (t_DRAM) of the DRAM chip and the computing latency (t_logic) of the logic chip < Flash latency (t_Flash).

[0113] Therefore, before the Flash weights are fully read, all pre - tasks of the neural network processing can be completed. When the Flash data arrives, it immediately enters the core matrix calculation stage without idle cycles.

[0114] Thus, the Flash latency is completely hidden; the operation and memory access are fully overlapped; and an efficient scheduling of "continuous pipelining" is achieved.

[0115] Correspondingly, this embodiment provides a control method for a neural network processor, as Figure 22 shown, including the steps: Detect whether to trigger the calculation of the attention module; and When it is detected that the calculation of the attention module is triggered, while performing the read operation of the off - chip non - volatile storage chip, schedule the off - chip volatile storage chip to start data loading and the logic chip to start pre - processing.

[0116] Particularly crucial is that this embodiment uses the compute - in - memory (CIM) unit as the core computing execution unit. Its characteristic of "tight coupling of storage and computing" significantly shortens the data transfer path and operation time. Through the coordinated scheduling and timing optimization of the CIM unit, DRAM chip, and logic chip, it can be achieved that the sum of "DRAM parameter transfer latency + logic chip computing latency" is less than "Flash parameter transfer latency". This core breakthrough can completely hide the data transfer time of DRAM and the operation time of the logic chip within the latency window of Flash reading weights, completely eliminating the efficiency bottleneck of "hardware waiting for data" in the traditional architecture.

[0117] This design is essentially different from the traditional storage architecture: in the traditional architecture, due to the separation of storage and computing, data needs to be frequently transferred between different hardware, and operation and memory access cannot be deeply parallelized, making it difficult to completely hide the latency. However, in this invention, with the characteristics of the CIM unit and the optimized pipeline timing, the access of the DRAM chip, the calculation of the logic chip, and the Flash memory access form a gapless parallel pipeline, which not only avoids the loss caused by a single hardware waiting but also maximizes the performance of each hardware module. Finally, it provides an operation guarantee of "low latency and high throughput" for large - language model inference, significantly improving the utilization rate of hardware resources and the overall computing efficiency of the system.

[0118] Optionally, the off - chip volatile storage chip 100 can be composed of an off - chip volatile storage module 10, as Figure 23As shown, the off-chip volatile memory chip 100 can also be integrated by at least two off-chip volatile memory modules 10 using three-dimensional stacking technology.

[0119] Optionally, the off-chip non-volatile memory chip 200 can be composed of an off-chip non-volatile memory module 20, such as... Figure 24 As shown, the off-chip non-volatile memory chip 200 can also be integrated by at least two off-chip non-volatile memory modules 20 using three-dimensional stacking technology.

[0120] Optionally, the logic chip 300 can be composed of a logic module 30, such as Figure 25 As shown, the logic chip 300 can also be integrated by at least two logic modules 30 using three-dimensional stacking technology.

[0121] In this embodiment, each chip can be flexibly expanded by three-dimensional stacking to increase computing power and storage.

[0122] It should be noted that when multiple modules are stacked, their three-dimensional integration bonding method can be based on... Figures 12-15 The bonding methods shown can be combined in an adaptive manner based on the needs of the scenario, and all of the above implementation methods are within the protection scope of this application.

[0123] It should be noted that when the off-chip volatile memory module 10, the off-chip non-volatile memory module 20, and the logic module 30 are configured individually, they can independently implement the functions of the off-chip volatile memory chip 100, the off-chip non-volatile memory chip 200, and the logic chip 300.

[0124] It should be noted that each off-chip volatile memory module 10 is integrated face to face. The side of the structure formed by three-dimensional stacking and integration of at least two off-chip volatile memory modules 10, the side closer to the logic control layer is defined as the front side of the off-chip volatile memory chip 100, and the side closer to the memory array layer is defined as the back side of the off-chip volatile memory chip 100.

[0125] Taking two off-chip volatile memory modules 10 as an example, with them integrated face to face, the off-chip volatile memory chip 100 will show the back of the off-chip volatile memory module 10. The reason for defining the side closer to the logic control layer as the front is that the control circuits of the off-chip volatile memory module 10 are all on the logic control layer (CMOS Die), while the memory array layer (Array Die) cannot work independently. Therefore, it is more convenient to bond the logic control layer as the front side to the front side of the logic chip 300, which facilitates the transmission of signals and data.

[0126] Similarly, it should be noted that each off-chip non-volatile memory module 20 is integrated face to face. The side of the structure formed by three-dimensional stacking and integration of at least two off-chip non-volatile memory modules 20 is defined as the front side of the off-chip non-volatile memory chip 200, and the side is defined as the back side of the off-chip non-volatile memory chip 200.

[0127] Optionally, the three-dimensional stacking technology is one or more of TSV, hybrid bonding, or Filp Chip.

[0128] Optionally, the on-chip volatile memory chip is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

[0129] Optionally, the global controller 340 is configured to perform fine-grained dynamic scheduling of data based on the bonding scheme interconnect characteristics of the three-dimensional stacking technology and the inherent properties of each storage medium.

[0130] Optionally, the off-chip volatile memory chip 100 includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory chip controller integrated on the logic chip 300, enabling the off-chip volatile memory chip 100 of the neural network processor to be accessed by external devices or systems as a standard DRAM memory chip. The interface protocol meets one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6. In an architecture where the off-chip volatile memory chip 100 and the Logic Die are vertically stacked using three-dimensional integration technology (such as TSV and Hybrid Bonding), this standard interface can still ensure DRAM compatibility with external devices. At the same time, by leveraging the low latency and high bandwidth interconnect characteristics of three-dimensional integration, the data transmission efficiency during standard interface access is improved, avoiding the performance bottleneck of the standard interface under traditional 2D connection.

[0131] Optionally, the off-chip non-volatile memory chip 200 includes a standard interface (compliant with JEDEC or industry-standard memory interface standards). Taking Flash as an example, the Flash of the neural network processor can be accessed as a standard Flash memory chip. The protocol of this standard interface can be configured to meet one or more of the following: SPI (Serial Peripheral Interface), QSPI (Quad SPI), Octal SPI (OSPI), eMMC (embedded MultiMediaCard, compliant with JEDEC JESD216 standard), UFS (Universal Flash Storage, compliant with JEDEC JESD220 / JESD223 standard), NVMe (Non-Volatile Memory Express), and Parallel Flash.

[0132] The technical solution of the present invention has now been described in conjunction with the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to the specific embodiments described above. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor, characterized in that, include: An off-chip volatile memory chip is connected to the logic chip and is used to exchange information with the logic chip; An off-chip non-volatile memory chip, connected to the logic chip, is used for information interaction with the logic chip; and The logic chip is configured to perform any combination of the following functions: Perform computational tasks, cache data, and facilitate information exchange; The off-chip volatile memory chip, the logic chip, and the off-chip non-volatile memory chip are integrated using three-dimensional stacking technology. In the vertical direction, the off-chip volatile memory chip is located at the top and can be connected to an external heat dissipation structure. The logic chip integrates a digital in-memory computing unit and a global controller. The digital in-memory computing unit is used to implement data storage and execute computing tasks within the logic chip. The global controller is configured as follows: High-bandwidth task data is stored in a memory chip that is bonded face-to-face with the logic chip, and... Thermally sensitive task data is stored in a memory chip bonded to the logic chip using a front-to-back or back-to-back bonding method. To achieve dynamic data scheduling based on the bonding method, task characteristics, and storage medium attributes of the off-chip volatile memory chip, the off-chip non-volatile memory chip, and the logic chip; The global controller is also used for: While performing the read operation of the off-chip non-volatile memory chip, the off-chip volatile memory chip is scheduled to start data loading and the logic chip is scheduled to start preprocessing, so that the sum of the access latency of the off-chip volatile memory chip and the computation latency of the logic chip is less than the latency of the read operation of the off-chip non-volatile memory chip.

2. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 1, characterized in that, The off-chip volatile memory chip, the logic chip, and the off-chip non-volatile memory chip are integrated sequentially in the vertical direction using three-dimensional stacking technology.

3. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 2, characterized in that, The off-chip volatile memory chip and the logic chip are bonded together in a front-to-front bonding manner, and the off-chip non-volatile memory chip is bonded to the logic chip in either the front or back position.

4. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 2, characterized in that, The off-chip volatile memory chip and the logic chip are bonded together in a back-to-back bonding manner, and the off-chip non-volatile memory chip is bonded to the logic chip in either a front or back manner.

5. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 1, characterized in that, The off-chip volatile memory chip, the off-chip non-volatile memory chip, and the logic chip are integrated sequentially in the vertical direction using three-dimensional stacking technology.

6. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The logic chip is a computing chip that performs computing tasks.

7. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The logic chip is a data cache chip used for data caching.

8. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The logic chip is an interface chip used to realize information interaction.

9. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The logic chip integrates at least one off-chip non-volatile memory chip controller and at least one off-chip volatile memory chip controller; wherein, The off-chip non-volatile memory chip controller is used to access the off-chip non-volatile memory chip, and the off-chip volatile memory chip controller is used to access the off-chip volatile memory chip. The global controller is used to manage the off-chip volatile memory chip controller, the off-chip non-volatile memory chip controller, and the digital in-memory computing unit, so as to realize the dynamic adaptation of data storage scheme and computing data flow, and to globally optimize storage capacity, data bandwidth, access latency and computing efficiency.

10. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The off-chip volatile memory chip is a DRAM chip.

11. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 10, characterized in that, In large language model computing scenarios, the off-chip volatile memory chip is used to store key-value cache data, activation values, and intermediate calculation results.

12. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The off-chip non-volatile memory chip is Flash.

13. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 12, characterized in that, In large language model computation scenarios, the off-chip non-volatile memory chip is used to store Q-generated weights, K-generated weights, V-generated weights, Attention weights, and FFN weights.

14. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The off-chip volatile memory chip is composed of an off-chip volatile memory module; or, The off-chip volatile memory chip is integrated by at least two off-chip volatile memory modules using three-dimensional stacking technology. Each off-chip volatile memory module is integrated face to face. The side of the structure formed by the three-dimensional stacking of the at least two off-chip volatile memory modules closer to the logic control layer is defined as the front side of the off-chip volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip volatile memory chip.

15. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The off-chip non-volatile memory chip is composed of an off-chip non-volatile memory module; or, The off-chip non-volatile memory chip is integrated by at least two off-chip non-volatile memory modules using three-dimensional stacking technology; wherein, each of the off-chip non-volatile memory modules is integrated face to face, and the side of the structure formed by the three-dimensional stacking of the at least two off-chip non-volatile memory modules closer to the logic control layer is defined as the front side of the off-chip non-volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip non-volatile memory chip.

16. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The logic chip consists of a logic module; or, The logic chip is integrated from at least two logic modules using three-dimensional stacking technology.

17. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-4, characterized in that, The three-dimensional stacking technology is one or more of through-silicon via (TSV) technology, hybrid bonding technology, or flip-chip technology.

18. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 1, characterized in that, The digital in-memory computing unit is integrated with an on-chip volatile memory chip and a computing unit. The on-chip volatile memory chip is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

19. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 9, characterized in that, The global controller is configured to support the configuration of data storage mapping rules through upper-layer software.

20. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 9, characterized in that, The off-chip volatile memory chip includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory chip controller, enabling the neural network processor to be accessed by external devices as a standard DRAM memory chip; the interface protocol satisfies one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6.

21. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 12, characterized in that, The Flash includes an interface that conforms to JEDEC or an industry-standard storage interface, and the Flash of the neural network processor can be accessed as a standard Flash storage chip. The protocol of the interface meets one or more of SPI, QSPI, Octal SPI, eMMC, UFS, NVMe, and Parallel Flash.

22. A control method for a three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor, employing the three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor according to any one of claims 1-21, characterized in that, Including the following steps: Detect whether the attention module calculation has been triggered; as well as When the attention module is detected to be triggered to perform calculations, while performing the read operation of the off-chip non-volatile memory chip, the off-chip volatile memory chip is scheduled to start data loading and the logic chip is scheduled to start preprocessing, so that the sum of the access latency of the off-chip volatile memory chip and the calculation latency of the logic chip is less than the latency of the read operation of the off-chip non-volatile memory chip.

Images