Storage and calculation integrated neural network processor of three-dimensional integrated heterogeneous storage medium

Abstract

The invention relates to a storage and calculation integrated neural network processor based on a three-dimensional integrated heterogeneous storage medium and a control method thereof, and the processor comprises an off-chip nonvolatile storage chip which is connected with a logic chip and is used for carrying out information interaction with the logic chip; the off-chip volatile storage chip is connected with the logic chip and is used for carrying out information interaction with the logic chip; the logic chip is configured to realize any combination of the following functions: executing a calculation task, performing data caching and realizing information interaction; the off-chip volatile storage chip, the logic chip and the off-chip nonvolatile storage chip are integrated by adopting a three-dimensional stacking technology, the off-chip volatile storage chip is positioned between the off-chip nonvolatile storage chip and the logic chip, and the off-chip volatile storage chip is used as a high-efficiency cache layer; the high delay of the off-chip nonvolatile memory chip is hidden in a parallel window for reading and real-time calculation, so that the efficiency loss caused by the delay of the off-chip nonvolatile memory chip is eliminated.

Description

Technical Field

[0001] This application relates to the field of semiconductor chip technology, specifically to a three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor and its control method. Background Technology

[0002] Large Language Models (LLMs), as a core evolutionary direction of Natural Language Processing (NLP) technology in the field of Artificial Intelligence (AI), have shown an exponential growth trend in recent years. Their technological maturity and application breadth have continued to break through, demonstrating irreplaceable advantages in several key areas.

[0003] Meanwhile, the development of LLMs has gradually revealed a core principle widely validated by academia and industry—the scaling law. The core meaning of this principle is that, with sufficient high-quality training data and matching computational resources, the core performance indicators of LLMs (such as perplexity in language generation and accuracy in downstream tasks) show a significant positive correlation with the scale of model parameters. For example... Figure 1 As shown, the model performance score continuously improves with the exponential increase in the number of parameters.

[0004] Traditional computer systems rely on "storage system architecture" (such as...) Figure 2 As shown, the process of registers → on-chip cache → main memory → local disk → remote storage achieves a trade-off between capacity, speed, and cost. Figure 3 As shown, each tier exhibits significant differences in relative cost, storage density, bandwidth, access latency, and power consumption.

[0005] In existing hardware architectures, off-chip non-volatile memory chips (such as Flash memory) are typically used as the primary storage medium to meet the demands for large-capacity and non-volatile storage. However, compared to the operating speed of logic computing units, the read and write performance of off-chip non-volatile memory chips suffers from a significant bottleneck. Specifically, the single-read latency of non-volatile memory is relatively high, often involving complex low-level operations (such as block erasure and page read), making it difficult for its data throughput to keep up with the demands of high-performance logic chips.

[0006] Meanwhile, with the application of computing-in-memory (CIMM) technology, the computing density of logic chips has been significantly improved, enabling core operations such as multiplication and accumulation to be performed in parallel within memory units, greatly reducing computation time. This contradiction between "fast computation" and "slow data retrieval" has become increasingly prominent: during large model inference, logic chips are often forced into an idle state while waiting for data to be loaded from off-chip non-volatile memory chips, preventing efficient pipelined operations. This computational blockage caused by data supply delays severely restricts the overall energy efficiency and inference speed of neural network processors. Furthermore, traditional planar interconnects or non-targeted stacking architectures have failed to effectively shorten the distance and complexity of data transport in the physical path, further exacerbating the complexity and cost of system design.

[0007] Therefore, it is necessary to improve existing neural network processors. Summary of the Invention

[0008] To address the above problems, this application provides a three-dimensional integrated heterogeneous storage medium in-memory neural network processor, comprising: an off-chip non-volatile memory chip connected to a logic chip for information interaction with the logic chip; an off-chip volatile memory chip connected to the logic chip for information interaction with the logic chip; and a logic chip configured to perform any combination of the following functions: performing computational tasks, performing data caching, and performing information interaction; the off-chip volatile memory chip, the logic chip, and the off-chip non-volatile memory chip are integrated using three-dimensional stacking technology, and the off-chip volatile memory chip is located between the off-chip non-volatile memory chip and the logic chip in the vertical direction; wherein, the off-chip non-volatile memory chip is connected to the off-chip volatile memory chip, and the off-chip volatile memory chip can temporarily store data stored in the off-chip non-volatile memory chip for use by the logic chip.

[0009] Considering the relatively high single-read latency of off-chip non-volatile memory chips (requiring block erasure, page read, and other operations), and the faster random access capability of off-chip volatile memory chips, the three-dimensional integrated architecture provided in this application configures off-chip volatile memory chips between off-chip non-volatile chips and logic chips. Through dynamic scheduling by the global controller, off-chip volatile memory chips are used as a high-efficiency cache layer for off-chip non-volatile memory chips. With this design, during large model inference, data flows along the path of "loading off-chip non-volatile memory chips → temporary storage on off-chip volatile memory chips → calling on logic chips": first, the model weights (such as Q / K / V generation weights) stored in the off-chip non-volatile memory chips are preloaded into the off-chip volatile memory chips, and then the off-chip volatile memory chips quickly feed data to the logic chips.

[0010] Crucially, leveraging the high computing density of the logic chip's in-memory computing (capable of performing core operations like multiplication and addition in parallel within the storage unit, significantly reducing computation time), the total time spent on data reading and real-time computation from the off-chip volatile memory chip to the logic chip can be less than the loading latency from the off-chip non-volatile memory chip to the off-chip volatile memory chip. This timing matching completely hides the loading process of the off-chip non-volatile memory chip within the parallel window of off-chip volatile memory chip reading and logic chip computation, forming a non-blocking pipeline of "off-chip non-volatile memory chip preloading - off-chip volatile memory chip buffering - logic chip real-time computation." This completely eliminates the efficiency loss of "waiting for off-chip non-volatile memory chip data" in traditional architectures, significantly improving the overall computational efficiency of large language model inference.

[0011] The physical architecture of placing off-chip volatile memory chips between off-chip non-volatile chips and logic chips is more compatible with the pipeline's operating logic, further improving computational efficiency and reducing design complexity and cost.

[0012] Optionally, the off-chip non-volatile memory chip, the off-chip volatile memory chip, and the logic chip are sequentially integrated using three-dimensional stacking technology in the vertical direction.

[0013] Optionally, the logic chip, the off-chip volatile memory chip, and the off-chip non-volatile memory chip are sequentially integrated using three-dimensional stacking technology in the vertical direction.

[0014] In some application scenarios, logic chips may have high requirements for heat dissipation due to computing tasks or data interaction. In addition, when a neural network processor is integrated into an electronic device, the logic chip usually needs to be connected to the main control chip of the electronic device. Therefore, this application places the logic chip at the top layer of the neural network processor to meet the heat dissipation requirements and the connection requirements with the main control chip, providing more diverse architecture options that fit the application scenarios.

[0015] Optionally, the logic chip is a computing chip that performs computing tasks, and the logic chip integrates a digital in-memory computing unit.

[0016] Optionally, the logic chip is a data cache chip for data caching.

[0017] Optionally, the logic chip is an interface chip used to realize information interaction.

[0018] Optionally, the logic chip integrates at least one off-chip volatile memory chip controller, at least one off-chip non-volatile memory chip controller, and a global controller; the off-chip volatile memory chip controller is used to access the off-chip volatile memory chip, and the off-chip non-volatile memory chip controller is used to access the off-chip non-volatile memory chip; the global controller is used to manage the off-chip non-volatile memory chip controller, the off-chip volatile memory chip controller, and the digital in-memory computing unit, to achieve dynamic adaptation of data storage schemes and computing data flows, so as to globally optimize storage capacity, data bandwidth, access latency, and computing efficiency.

[0019] Optionally, the off-chip volatile memory chip is a DRAM chip.

[0020] Optionally, in a large language model computation scenario, the off-chip volatile memory chip is used to store key-value cache data, activation values, and intermediate computation results.

[0021] Optionally, the off-chip non-volatile memory chip is Flash.

[0022] Optionally, in a large language model computation scenario, the off-chip non-volatile memory chip is used to store Q-generated weights, K-generated weights, V-generated weights, Attention weights, and FFN weights.

[0023] Optionally, the off-chip volatile memory chip is composed of one off-chip volatile memory module, or is integrated by at least two off-chip volatile memory modules using three-dimensional stacking technology. The off-chip volatile memory modules are integrated face to face. The side of the structure formed by the three-dimensional stacking of the at least two off-chip volatile memory modules closer to the logic control layer is defined as the front side of the off-chip volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip volatile memory chip.

[0024] Optionally, the off-chip non-volatile memory chip is composed of one off-chip non-volatile memory module, or is integrated by at least two off-chip non-volatile memory modules using three-dimensional stacking technology; wherein, each of the off-chip non-volatile memory modules is integrated face to face, and the side of the structure formed by the three-dimensional stacking of the at least two off-chip non-volatile memory modules closer to the logic control layer is defined as the front side of the off-chip non-volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip non-volatile memory chip.

[0025] Optionally, the logic chip consists of a single logic module or is integrated from at least two logic modules using three-dimensional stacking technology.

[0026] Optionally, the three-dimensional stacking technology is one or more of through-silicon via (TSV) technology, hybrid bonding technology, or flip-chip technology.

[0027] Optionally, the digital in-memory computing unit is integrated with an on-chip volatile memory chip and a computing unit, wherein the on-chip volatile memory chip is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

[0028] Optionally, the global controller is configured to perform fine-grained dynamic scheduling of data based on the interconnect characteristics of the bonding scheme of the three-dimensional stacking technology and the inherent properties of each storage medium.

[0029] Optionally, the global controller is configured to support the configuration of data storage mapping rules through upper-layer software.

[0030] Optionally, the off-chip volatile memory chip includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory chip controller, enabling the neural network processor to be accessed by external devices as a standard DRAM memory chip; the interface protocol satisfies one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6.

[0031] Optionally, the Flash includes an interface that conforms to JEDEC or an industry-standard storage interface, and the Flash of the neural network processor can be accessed as a standard Flash memory chip. The protocol of the interface meets one or more of SPI, QSPI, Octal SPI, eMMC, UFS, NVMe, and Parallel Flash.

[0032] Optionally, the off-chip volatile memory chip and the logic chip are bonded together in a three-dimensional stacked manner, face to face.

[0033] Optionally, the off-chip volatile memory chip and the logic chip are bonded in a three-dimensional stacked manner with their backs to the back.

[0034] Given that the memory access latency of off-chip volatile memory chips is on the order of nanoseconds (typically tens of nanoseconds), their memory access efficiency is significantly affected by the bit width and transmission rate of the external I / O interface. When the two are bonded in a face-to-face (F2F) manner, F2F bonding supports extremely high-density I / O interconnects, greatly increasing the interface bit width and supporting high-bandwidth data interaction. Compared with bonding schemes that require penetrating the silicon substrate and introducing parasitic capacitance and resistance, face-to-face bonding significantly reduces link loss, ensures the signal integrity of off-chip volatile memory under high-speed transmission, and reduces the error correction burden. In addition, the core memory access characteristics of off-chip non-volatile memory chips determine their adaptation requirements for interconnect bonding methods: firstly, the memory access latency is relatively high, typically on the order of 30-50μs microseconds; secondly, the amount of data read in a single operation is fixed, generally 4-16KB. From a performance bottleneck perspective, the memory access efficiency of off-chip non-volatile memory chips (such as Flash) is primarily constrained by their internal operating mechanisms, rather than the performance of external I / O interconnect links. This conclusion can be fully verified from two key dimensions: I / O density and I / O speed. From the perspective of I / O density, the core value of front-to-front bonding lies in improving I / O concurrent interaction capabilities to meet high-frequency data transmission demands. However, Flash itself does not require high-frequency, high-concurrency I / O interaction. Even with high-density F2F bonding, its memory access efficiency cannot be linearly improved; instead, it leads to ineffective redundancy and waste of I / O resources. From the perspective of I / O speed, compared to back-side bonding, it requires through-silicon vias (TSVs) to penetrate the silicon substrate. TSVs inevitably introduce parasitic capacitance and resistance, which affects high-speed I / O links, leading to reduced transmission rates and, in severe cases, even threatening data security.

[0035] Therefore, on the one hand, the front-to-front bonding method between the off-chip volatile memory chip and the logic chip increases the interface bit width, supports high-bandwidth data interaction, and meets the high-speed memory access and high-bandwidth data interaction requirements of the off-chip volatile memory chip. On the other hand, the off-chip non-volatile memory chip, due to its memory access latency reaching the microsecond level, has extremely low sensitivity to I / O rate and is not negatively affected by the front-to-front bonding between the off-chip volatile memory chip and the logic chip.

[0036] Since both the off-chip volatile memory chip and the logic chip generate a lot of heat during operation, and the back side of the chip is mainly composed of a silicon substrate with high thermal insulation performance, the back-to-back bonding method can avoid the heat transfer between the two and thus avoid affecting the computing efficiency due to heat generation. At the same time, the back-to-back bonding allows the front sides of the two chips to conduct heat to the outside through the substrate or directly to achieve a better heat dissipation effect.

[0037] To achieve the above-mentioned objectives, this application provides a control method for a three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor, which applies the three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor described above, and includes the following steps. Detect whether the attention module computation is triggered; and When the attention calculation is triggered, the off-chip non-volatile memory chip is scheduled to start data loading and the logic chip is scheduled to start calculation preprocessing while the off-chip non-volatile memory chip read operation is being performed.

[0038] The technical advantage of this control method lies in its ability to successfully "hide" the physical access bottleneck of Flash memory during the computational preprocessing process through parallel scheduling across media, thus overcoming the performance bottleneck caused by media speed mismatch in three-dimensional stacked heterogeneous systems. Attached Figure Description

[0039] Figure 1 A graph showing the relationship between the core performance metrics of a large language model and the scale of model parameters; Figure 2 A diagram of the storage system architecture of a traditional computer system; Figure 3 The performance and cost characteristics of each layer in the storage system architecture of traditional computer systems; Figure 4 This is a schematic diagram of a storage architecture based on a traditional computer system. Figure 5 A schematic diagram of the parameter transfer process for large language model computation based on traditional computer systems; Figure 6 A schematic diagram of the parameter transport process for large language model computation based on data offloading technology; Figure 7 This is a schematic diagram of a storage packaging structure based on Chiplet and Flip Chip technologies; Figure 8 This is a schematic diagram of a storage packaging structure based on three-dimensional integration technology; Figure 9 This is a schematic diagram of a storage packaging structure based on three-dimensional integration technology; Figure 10 A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention; Figure 11 A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention; Figure 12 A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention; Figure 13This is a schematic diagram of one type of three-dimensional integrated bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of one type of three-dimensional integrated bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 15 This is a schematic diagram of one type of three-dimensional integrated bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 16 This is a schematic diagram of one type of three-dimensional integrated bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 17 This is a schematic diagram comparing the implementation of the present invention with the storage architecture of the prior art; Figure 18 This is a schematic diagram of the structure of the digital in-memory computing unit in an embodiment of the present invention; Figure 19 A schematic diagram illustrating the features of 3D integrated bonding for different interface positional relationships; Figure 20 This is a schematic diagram of one of the storage schemes for computing large language models provided in the embodiments of the present invention; Figure 21 This is a flowchart illustrating a computation scheduling scheme for a global controller of a neural network processor provided in an embodiment of the present invention. Figure 22 This is a timing diagram of the parallel pipeline coordination mechanism of the global controller of the neural network processor provided in an embodiment of the present invention; Figure 23 This is a schematic diagram illustrating the steps of the control method for a neural network processor provided in an embodiment of the present invention. Figure 24 This is a schematic diagram of the structure of the off-chip volatile memory chip extension of the neural network processor provided in an embodiment of the present invention; Figure 25 This is a schematic diagram of the structure of the off-chip non-volatile memory chip extension of the neural network processor provided in an embodiment of the present invention; Figure 26 This is a schematic diagram of the logic chip extension of the neural network processor provided in an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] Due to differences in process compatibility between volatile and non-volatile memory chips and computing logic (such as incompatibility between advanced computing logic processes and mature memory chip processes, and high integration difficulty caused by different material systems), considering factors such as area utilization, chip yield, and process feasibility, adopting an integrated solution would lead to problems such as excessive chip area redundancy, significantly reduced yield, and insufficient process adaptability. Therefore, if... Figure 4 As shown, traditional computer systems break it down into multiple chips for independent implementation.

[0042] Based on traditional computer system storage architecture, storage systems can be divided into four categories: off-chip non-volatile memory chips (such as remote storage, SSD, NAND Flash, ReRAM, etc.), off-chip volatile memory chips (such as DRAM), on-chip volatile memory chips (such as SRAM), and registers; for example... Figure 5 As shown, when deploying large language model computation in an existing system, the process is usually as follows: After the AI ​​task is started, the main control chip first loads as much large model parameter data as possible from the non-volatile memory chip to the off-chip volatile memory chip, then moves the data to be computed from the off-chip volatile memory chip to the on-chip volatile memory chip, then selects the data that needs to be computed immediately and sends it to the register, and finally completes the computation in the computing unit.

[0043] However, as analyzed above, the inference process of LLM (Large Language Model) under traditional architecture still faces severe capacity and bandwidth bottlenecks. To effectively alleviate this problem, the industry has successively explored several technical solutions, as follows: First, data offloading technology. For example... Figure 6 As shown, its core logic is "trading communication for capacity": compared to the fixed data loading path of the traditional deployment process in Figure 5, this technology unloads data that is not currently needed in the computing stage from high-speed on-chip storage (such as SRAM) and stores it back to off-chip volatile memory chips (such as DDR) or off-chip non-volatile memory chips (such as SSD); when subsequent computing requires the data, it is retrieved from the corresponding storage medium and reloaded onto the chip. This approach can avoid irrelevant data occupying scarce on-chip storage resources for a long time, thereby directly alleviating the storage capacity bottleneck, and is especially suitable for LLM tasks that process long sequence inputs (such as long document understanding and multi-turn dialogue).

[0044] However, the core limitation of data offloading technology lies in the fact that it does not change the traditional "compute-storage separation" architecture of memory chips in computing systems. While this technology can alleviate storage capacity bottlenecks by "trading communication for capacity" and even relieve bandwidth pressure to some extent, this process also comes at a significant cost: frequent data offloading and retrieval generate a large number of on-chip and off-chip communication needs, which in turn keeps the system continuously subject to severe bandwidth constraints. Even if the system can therefore support neural networks with a larger number of parameters, the frequent transfer of large amounts of data between on-chip and off-chip will seriously drag down computational efficiency, ultimately resulting in significant limitations on the overall inference efficiency of large models.

[0045] Second, high-density packaging technology. This technology, through processes such as chiplets and flip chips, breaks the physical limitations of traditional planar packaging, integrating multiple functional chips (such as logic chips and cache chips) into a single package in a more compact manner. Its core function is to shorten the physical transmission distance across memory levels (such as on-chip cache and off-chip memory, and cache between different computing units), which can reduce data transmission latency and improve overall bandwidth through parallel transmission paths, thereby alleviating the problem of low "data moving" efficiency.

[0046] High-density packaging technology is effective in alleviating the bandwidth bottleneck of traditional memory chip architectures, but it has two key limitations. First, it does not break through the underlying design of traditional memory chip architectures, but only improves data transmission efficiency by optimizing the packaging form, so the core problem of limited storage capacity is not solved. Second, its performance improvement has a clear ceiling, and it will continue to be limited by the speed limit of the IO interface, making it difficult to achieve unlimited bandwidth breakthroughs.

[0047] Third, three-dimensional integration technology. For example... Figure 8 , Figure 9 As shown, this technology overcomes the spatial limitations of two-dimensional packaging through key processes such as TSV (Through Silicon Via) and Hybrid Bonding, enabling vertical stacking integration of logic chips and memory chips. For example, the widely used HBM (High Bandwidth Memory) technology uses TSV to vertically stack multiple DRAM chips next to the logic chip, forming a tightly coupled "compute-memory" structure. Compared to traditional DDR memory chips, HBM's bandwidth can be increased several times (e.g., HBM3 has a bandwidth of 819GB / s), while significantly reducing the area occupied by the planar circuit board, fundamentally optimizing the "memory-compute" interaction efficiency required for LLM inference.

[0048] 3D integration technology, with its 3D stacking structure and high-density interconnect design, can effectively address the capacity shortage and bandwidth limitations faced by traditional storage architectures. However, existing implementation solutions for this technology still have significant limitations. Current mainstream solutions (such as HBM high-bandwidth memory chips and NAND Flash memory) are essentially still limited to the stacking and interconnection of memory chips of the same type and level—they have neither broken through the underlying framework of traditional memory chip architectures nor achieved a synergistic breakthrough in capacity and bandwidth through a unified storage structure.

[0049] Meanwhile, current optimization solutions for storage architecture mostly focus on simply expanding capacity and increasing bandwidth, neglecting the core characteristic of computer systems: the dynamic transfer of bottlenecks. When capacity and bandwidth bottlenecks are alleviated through technical means, the processing power of computing units often becomes a new performance constraint. For example, in large model inference scenarios, even if high-speed data storage and transmission are achieved through technologies such as HBM and Chiplet, if the computing power of logic chips such as GPUs / TPUs cannot match the data throughput, resource idleness will still occur, resulting in "data waiting for computation," making it difficult to improve the overall system efficiency. More importantly, existing solutions generally adopt a "fragmented optimization" approach, failing to build a unified architecture that coordinates computing power, storage efficiency, and bandwidth performance. This fails to break the cycle of "solving old bottlenecks and creating new bottlenecks" at the system level, making it difficult to achieve synergistic breakthroughs in the performance of various components.

[0050] Therefore, existing solutions have failed to break free from the constraints of the underlying framework of traditional memory chip architecture. The core problem lies in the inability to achieve coordinated optimization of capacity, bandwidth, and computing power based on a unified storage structure. This structural shortcoming directly results in significant performance limitations and severe implementation challenges for large language models when deployed for inference in systems built on existing solutions.

[0051] If the applicant is aware of the aforementioned technical issues, Figure 10 and Figure 11As shown, this embodiment provides a neural network processor based on in-memory computing and three-dimensional stacking technology, including: an off-chip non-volatile memory chip 100 connected to a logic chip 300 for information interaction with the logic chip; an off-chip volatile memory chip 200 connected to the logic chip 300 for information interaction with the logic chip; and a logic chip 300 configured to perform any combination of the following functions: performing computational tasks, performing data caching, and performing information interaction; the off-chip volatile memory chip 200, the logic chip 300, and the off-chip non-volatile memory chip 100 are integrated using three-dimensional stacking technology, and the off-chip volatile memory chip 200 is located between the off-chip non-volatile memory chip 100 and the logic chip 300 in the vertical direction; wherein, the off-chip non-volatile memory chip 100 is connected to the off-chip volatile memory chip 200, and the off-chip volatile memory chip 200 can temporarily store data stored in the off-chip non-volatile memory chip 100 for use by the logic chip 300.

[0052] Considering the relatively high single-read latency of off-chip non-volatile memory chips (requiring block erasure, page read, and other operations), and the faster random access capability of off-chip volatile memory chips, the three-dimensional integrated architecture provided in this application configures off-chip volatile memory chips between off-chip non-volatile chips and logic chips. Through dynamic scheduling by the global controller, off-chip volatile memory chips are used as a high-efficiency cache layer for off-chip non-volatile memory chips. With this design, during large model inference, data flows along the path of "loading off-chip non-volatile memory chips → temporary storage on off-chip volatile memory chips → calling on logic chips": first, the model weights (such as Q / K / V generation weights) stored in the off-chip non-volatile memory chips are preloaded into the off-chip volatile memory chips, and then the off-chip volatile memory chips quickly feed data to the logic chips.

[0053] Crucially, leveraging the high computing density of the logic chip's in-memory computing (capable of performing core operations like multiplication and addition in parallel within the storage unit, significantly reducing computation time), the total time spent on data reading and real-time computation from the off-chip volatile memory chip to the logic chip can be less than the loading latency from the off-chip non-volatile memory chip to the off-chip volatile memory chip. This timing matching completely hides the loading process of the off-chip non-volatile memory chip within the parallel window of off-chip volatile memory chip reading and logic chip computation, forming a non-blocking pipeline of "off-chip non-volatile memory chip preloading - off-chip volatile memory chip buffering - logic chip real-time computation." This completely eliminates the efficiency loss of "waiting for off-chip non-volatile memory chip data" in traditional architectures, significantly improving the overall computational efficiency of large language model inference.

[0054] The physical architecture of placing off-chip volatile memory chips between off-chip non-volatile chips and logic chips is more compatible with the pipeline's operating logic, further improving computational efficiency and reducing design complexity and cost.

[0055] Alternatively, in some implementations, such as Figure 12 As shown, the logic chip 300, the off-chip non-volatile memory chip 100, and the off-chip volatile memory chip 200 are integrated in sequence using three-dimensional stacking technology in the vertical direction. The technical solutions formed by this embodiment in combination with other embodiments of the present invention are all within the protection scope of the present invention. In this embodiment, since the logic chip 300 is set on the top layer of the three-dimensional stacking structure, the heat dissipation requirements of the logic chip 300 are taken into account, and the computing power of the neural network is improved.

[0056] Optionally, such as Figure 13 and Figure 15 As shown, in the vertical direction, the off-chip non-volatile memory chip 100, the off-chip volatile memory chip 200 and the logic chip 300 are integrated sequentially using three-dimensional stacking technology, and the off-chip volatile memory chip 200 and the logic chip 300 are bonded together in a three-dimensional stacking manner with the front facing the front or the back facing the back.

[0057] Optionally, such as Figure 14 and Figure 16 As shown, in the vertical direction, the logic chip 300, the off-chip volatile memory chip 200 and the off-chip non-volatile memory chip 100 are integrated sequentially using three-dimensional stacking technology, and the off-chip volatile memory chip 200 and the logic chip 300 are bonded together in a three-dimensional stacking manner with the front facing the front or the back facing the back.

[0058] In this embodiment, considering that the memory access latency of off-chip volatile memory chips is on the order of nanoseconds (typically tens of nanoseconds), their memory access efficiency is greatly affected by the bit width and transmission rate of the external I / O interface. When the two are bonded in a face-to-face (F2F) manner, F2F bonding supports extremely high-density I / O interconnects, significantly increasing the interface bit width and supporting high-bandwidth data interaction. Compared with bonding schemes that require penetrating the silicon substrate and introducing parasitic capacitance and resistance, face-to-face bonding significantly reduces link loss, ensures the signal integrity of off-chip volatile memory under high-speed transmission, and reduces the error correction burden. In addition, the core memory access characteristics of off-chip non-volatile memory chips determine their adaptation requirements for interconnect bonding methods: firstly, the memory access latency is relatively high, typically on the order of 30-50μs microseconds; secondly, the amount of data read in a single operation is fixed, generally 4-16KB. From a performance bottleneck perspective, the memory access efficiency of off-chip non-volatile memory chips (such as Flash) is primarily constrained by their internal operating mechanisms, rather than the performance of external I / O interconnect links. This conclusion can be fully verified from two key dimensions: I / O density and I / O speed. From the perspective of I / O density, the core value of front-to-front bonding lies in improving I / O concurrent interaction capabilities to meet high-frequency data transmission demands. However, Flash itself does not require high-frequency, high-concurrency I / O interaction. Even with high-density F2F bonding, its memory access efficiency cannot be linearly improved; instead, it leads to ineffective redundancy and waste of I / O resources. From the perspective of I / O speed, compared to back-side bonding, it requires through-silicon vias (TSVs) to penetrate the silicon substrate. TSVs inevitably introduce parasitic capacitance and resistance, which affects high-speed I / O links, leading to reduced transmission rates and, in severe cases, even threatening data security.

[0059] Therefore, in this embodiment, on the one hand, the front-to-front bonding method between the off-chip volatile memory chip 200 and the logic chip 300 increases the interface bit width, supports high-bandwidth data interaction, and meets the high-speed memory access and high-bandwidth data interaction requirements of the off-chip volatile memory chip; on the other hand, the off-chip non-volatile memory chip 100 has a memory access latency in the microsecond range and is extremely insensitive to I / O rate, so it is not negatively affected by the front-to-front bonding method between the off-chip volatile memory chip 200 and the logic chip 300.

[0060] In addition, since both the off-chip volatile memory chip and the logic chip generate a lot of heat during operation, and the back side of the chip is mainly composed of a silicon substrate with high thermal insulation performance, the back-to-back bonding method can avoid the heat transfer between the two and thus avoid affecting the computing efficiency due to heat generation. At the same time, the back-to-back bonding allows the front sides of the two chips to conduct heat to the outside through the substrate or directly to achieve a better heat dissipation effect.

[0061] Optionally, the off-chip non-volatile memory chip 100 is Flash. For ease of explanation, Flash is used as an example of the off-chip non-volatile memory chip 100 in the following text.

[0062] Optionally, the off-chip volatile memory chip 200 is a DRAM chip. For ease of explanation, a DRAM chip will be used as an example of the off-chip volatile memory chip 200 in the following text.

[0063] In 3D integrated circuit architectures, Flash memory often becomes a system performance bottleneck due to its high access latency, while DRAM chips, with their low latency, are well-suited for the role of high-speed cache. When DRAM chips are located in the middle layer of a 3D stack, their data transfer direction to the underlying logic die is consistent with that of Flash, thus enabling the construction of a hierarchical storage structure with DRAM chips serving as Flash cache. By rationally planning the 3D integration density, the data supply efficiency of Flash can be significantly improved, allowing its high-latency access to be coordinated with the fast response of DRAM chips and the efficient computation of logic chips. Based on this, the system can achieve parallel pipelined operations between Flash access, DRAM chip access, and logic computation, effectively hiding Flash access latency and improving overall throughput and energy efficiency.

[0064] Optionally, the logic chip 300 is a computing chip for performing computing tasks. The logic chip 300 integrates a digital in-memory computing unit 310 for performing computing tasks. In the following text and figures, for ease of explanation, the logic chip 300 is used as an example of a computing chip. Based on the spirit of the present invention, it can be understood that the logic chip 300 can also be configured as a data cache chip and an interface chip as described below to achieve functions adapted to the application scenario.

[0065] Optionally, the logic chip 300 is a data cache chip used for data caching, which temporarily stores data between computing and storage, improving access efficiency and reducing latency and power consumption caused by frequent access to external storage.

[0066] Optionally, the logic chip 300 is an interface chip used to realize information interaction, and is used to realize information interaction and protocol conversion between different chips or systems.

[0067] Optionally, in a neural network processor, the logic chip 300 can be configured as multiple, such as three, which are respectively configured as a logic chip, an interface chip, and a data cache chip. Multiple chips can be integrated with each other using three-dimensional stacking technology to achieve flexible and diverse functions.

[0068] It should be noted that the off-chip non-volatile memory chip 100 and the off-chip volatile memory chip 200 can also be implemented using other storage media / memory chips that meet the requirements, and this embodiment does not limit them.

[0069] The neural network processor provided in this embodiment can achieve at least the following technical effects: like Figure 17 As shown, this implementation constructs a disruptive and innovative storage architecture, completely breaking through the inherent framework of traditional storage chips' "layer-by-layer cascading and unidirectional data transport"—traditional storage systems rely on a fixed hierarchy of "on-chip cache → off-chip storage → external storage," requiring data to be transferred level by level to meet computing demands. This not only generates a large amount of redundant data transport overhead but also leads to bandwidth limitations and latency accumulation due to the large physical distance between levels, becoming a core efficiency bottleneck for computationally intensive tasks such as large language models. The architecture provided by this implementation achieves end-to-end optimization through the synergistic innovation of three-dimensional stacking and in-memory computing technology: on the one hand, three-dimensional integration technology directly stacks off-chip storage chips (DRAM / Flash) and logic chips in three dimensions, shortening the physical distance between them to the micrometer level. This improves off-chip storage access bandwidth through high-density bonding and reduces cross-chip transmission losses to lower latency; on the other hand, in-memory computing technology integrates on-chip memory (such as DCIM based on 6T SRAM) with on-chip memory (such as DCIM based on 6T SRAM). The deep integration of the array with the computing unit enables operations such as multiplication and accumulation to be performed directly in the on-chip memory, fundamentally eliminating the latency and energy consumption of on-chip data transfer. Ultimately, it builds a storage-computing collaborative system with "low latency, high bandwidth, and low power consumption", completely breaking free from the performance constraints of traditional hierarchical architectures.

[0070] In terms of capacity, traditional architectures rely on planar expansion (increasing chip area or adding independent storage modules), which is easily limited by packaging space and power consumption budgets. Furthermore, capacity increases are prone to performance waste due to bandwidth mismatch. This implementation achieves a leapfrog, bottleneck-free capacity expansion through "three-dimensional vertical stacking + high-density media selection": using NAND Flash as the core high-density storage medium, bonding technology can increase the number of three-dimensionally stacked Flash chips—for example, an 8-layer stacked Flash module / unit. Within the same planar area as traditional 2D packaging, a single module can achieve a capacity of 60-128GB (traditional 2D Flash only 10-20GB in the same area). For further capacity expansion, the number of Flash stacking layers can be increased in the three-dimensional architecture (current processes support 200+ layers of 3D NAND), or multiple sets of Flash-three-dimensional stacked units can be connected in parallel, and the expansion process does not occupy additional planar space.

[0071] In terms of bandwidth, the bandwidth of traditional 2D architectures depends on interface protocol upgrades (such as DDR5→DDR6), with limited single-channel bandwidth improvement (approximately 30%-50% per generation), and multi-channel expansion is easily limited by motherboard wiring density. This implementation achieves efficient and scalable bandwidth expansion through "three-dimensional interconnect optimization + multi-chip parallelism," such as increasing the number of three-dimensional stacks of high-bandwidth media (DRAM / SRAM), for example, vertically integrating two groups of DRAM chips with logic chips via F2F bonding. Each group of DRAM chips provides 200-300GB / s bandwidth, and the two groups in parallel can achieve a total bandwidth of 400-600GB / s. At the same time, the micron-level wiring capability of the silicon interposer can be used to construct a parallel interconnect channel of "multi-memory chip-logic chip," avoiding the wiring congestion problem of traditional 2D architectures. In addition, this implementation can prioritize F2F hybrid bonding for high-frequency interaction links (such as Logic Die and DRAM), reducing interconnect latency to sub-nanosecond levels, further releasing bandwidth potential.

[0072] With the storage and bandwidth issues resolved through the aforementioned solutions, computing power has become a new bottleneck in the systemic problem of storage-bandwidth-computing power. The expansion of computing power in traditional architectures relies on increasing the area of ​​logic chips or upgrading process nodes, which is costly and easily limited by heat dissipation (increased area leads to increased heat density). At the same time, after the computing power is increased, it is easy to cause "computing power idle" due to insufficient storage bandwidth. This implementation method, while retaining the process node improvement, uses Compute-In-Memory (CIM) technology as its core to achieve high-density, low-power expansion of computing power. On the one hand, the number of CIM macrocells can be increased modularly, with a DCIM array based on 6T SRAM as the basic unit. Each unit can provide 100-150 TOPS of computing power. By increasing the number of CIM unit layers through three-dimensional stacking (such as 4-layer CIM stacking), a total computing power of 400-600 TOPS can be achieved, and the computing power density per unit area is 2-3 times that of traditional GPUs. Upgrading the logic chip process node: upgrading the Logic Die from 14nm to 3nm, and coordinating with the process optimization of CIM units, the computing power per unit area can be further improved by 50%-80%. Specifically, in this implementation method, the computing unit is integrated into the on-chip volatile memory chip on the logic chip 300, so that the calculation data does not need to be transferred to the register through the on-chip non-volatile memory chip, but can be calculated directly inside the on-chip volatile memory chip. The on-chip volatile memory chip with integrated computing units is designed as a Digital Computing-In-Memory (DCIM) unit. For example... Figure 18 As shown, the structure of the digital in-memory computing unit 310 mainly includes a DCIM array based on 6T SRAM, a word line decoder (WL DEC), a word line buffer (WL DRV), read / write circuitry, and a controller. The DCIM array consists of C DCIM columns, each containing R subarrays and an adder tree. Each subarray integrates 6T SRAM bit cells and a Local Read-Out and Compute Unit (LRCC), storing N-bit weights per row. Functionally, the DCIM macro can perform matrix-vector multiplication (MVM) multiplication-accumulation (MAC) calculations: the operating mode is switched via the MEM_EN signal. When MEM_EN is low, the N-bit weights in the subarray are multiplied by a 1-bit input, read and latched by the LRCC, accumulated by the adder tree, and then sent to a shift-and-accumulator to extend the input precision in the time dimension, ultimately completing a full MVM operation with K-bit input and N-bit weights, while also supporting the storage and efficient computation of multi-weight matrices.

[0073] In summary, the neural network processor provided in this embodiment features a reconstructed storage architecture (end-to-end bandwidth optimization), breaking the traditional "cache-external storage-external memory" hierarchical model. Through three-dimensional bonding, DRAM / Flash and logic chips are directly vertically interconnected, shortening the interconnection distance to the micrometer level. At the same time, combined with in-memory computing (CIM) technology, data can be multiplied and accumulated within the storage array, reducing data transfer latency and energy consumption, and constructing a low-latency, high-bandwidth data path.

[0074] Storage capacity expansion and density improvement are achieved by using "three-dimensional vertical stacking + high-density NAND Flash" to achieve three-dimensional expansion of storage capacity, increasing capacity by 3 to 6 times without occupying additional planar area; it supports hundreds of layers of Flash stacking or multiple units in parallel, breaking through the limitations of packaging space.

[0075] Bandwidth enhancement and interconnect optimization are achieved through F2F bonding and parallel interconnection of multiple DRAM chips to achieve linear bandwidth superposition (200–300 GB / s per channel, total bandwidth up to 400–600 GB / s); with the help of the micron-level wiring capability of silicon interposer, I / O density is significantly improved and interconnect latency is reduced to sub-nanosecond level.

[0076] The improved computing power density and energy efficiency are achieved by embedding DCIM (Digital In-Memory Computing) units based on 6T SRAM into the logic chip, realizing "storage as computing". By increasing the number of CIM macrocell layers through three-dimensional stacking and upgrading advanced process nodes (14nm→3nm), the computing power density per unit area is increased to 2 to 3 times that of traditional GPUs, and the energy efficiency ratio is improved by about 50% to 80%.

[0077] The neural network processor provided in this embodiment forms a collaborative closed loop in a three-dimensional architecture, integrating storage, bandwidth, and computing power: high-density storage provides sufficient data; high-bandwidth pathways ensure real-time supply; and the in-memory computing unit efficiently executes computations, achieving a performance balance of "low latency, high computing power, and low power consumption" at the system level.

[0078] Through a systematic design of "three-dimensional interconnection + in-memory computing integration", the neural network processor has achieved a comprehensive breakthrough in three dimensions: data path, storage capacity and computing power density, forming a new computing architecture with high integration, high bandwidth, scalability and low power consumption, which is suitable for computing power intensive scenarios such as large language models, visual computing, and edge inference.

[0079] In the field of 3D integration technology, each chip (die / core) includes: Face: This refers to the side where the metal interconnect layer is located, including the top layer wiring, solder joints (micro-bumps), or redistribution layer (RDL).

[0080] Back side: This is the silicon substrate side, usually the bare silicon surface, which is electrically connected to the front side through thinning, opening, or TSV (through silicon via).

[0081] like Figure 19 As shown, the characteristics of 3D integration bonding are demonstrated when different interface positional relationships are used. The interface positional relationships in 3D integration are mainly of three types: face to face (F2F), face to back (F2B), and back to back (B2B). In front-to-front mode, interconnect density and bandwidth are high, RDL rerouting is optional, cost (yield) is low, signal integrity is high, but thermal management is difficult; in front-to-back mode, interconnect density and bandwidth are low, RDL rerouting is necessary, cost (yield) and signal integrity are both moderate, and thermal management is moderate; in back-to-back mode, interconnect density and bandwidth are low, RDL rerouting is necessary, cost (yield) is high, signal integrity is low, but thermal management is relatively easy.

[0082] Despite the superior overall performance of F2F, in a three-layer stacking scenario, if two layers are to be bonded using F2F, the third layer will inevitably form an F2B structure with one of the layers. Therefore, three-layer stacking requires comprehensive consideration of hardware factors such as interconnect density, cost, signal integrity, and thermal management to determine the specific arrangement of F2F, F2B, and B2B, resulting in higher design complexity compared to two-layer stacking.

[0083] Specifically, the three-dimensional integrated structure proposed in this invention can flexibly adjust the bonding method between different dies according to actual needs, and supports flexible combinations of F2F, F2B or B2B bonding.

[0084] Specifically, in response to the performance requirements of large language model inference tasks (such as long sequence generation and high-concurrency query), the global controller in this embodiment can combine the interconnection characteristics of the bonding scheme (such as bandwidth and thermal efficiency) with the inherent properties of the storage medium (such as capacity and latency) to achieve fine-grained dynamic management of the data storage scheme. Specifically, targeted scheduling will be implemented based on the different hardware resource requirements of each task: high-bandwidth task data (such as key-value pair cache in the attention mechanism) will be stored in a storage chip bonded to the logic chip 300 using an F2F connection. This type of storage chip can match the high-frequency data interaction requirements due to its high interconnect bandwidth characteristics; heat-sensitive task data (such as intermediate results of large-scale matrix operations) will be stored in a storage chip bonded to the logic chip 300 using an F2B or B2B connection. This type of bonding scheme has better thermal management capabilities and can avoid local overheating from affecting data stability; at the same time, for the differentiated requirements of storage capacity and latency, non-real-time data requiring large-capacity storage (such as pre-trained weight backups) will be stored in an off-chip non-volatile storage chip, while latency-sensitive real-time computing data (such as the embedding vector of the input sequence) will be stored in an off-chip volatile storage medium, forming a storage architecture that precisely matches "features and requirements".

[0085] In this implementation, computing power and storage / bandwidth work in tandem. When expanding computing power, the bonding methods between different dies are flexibly adjusted according to actual needs, ensuring the bonding method matches the application scenario. For example, F2F bonding increases the bandwidth of off-chip volatile memory chips, while B2B bonding increases the capacity of off-chip non-volatile memory chips, ensuring a match between "computing power enhancement and data supply" and avoiding bottlenecks. By using F2F bonding between high-performance chips and F2B / B2B bonding between low-speed or peripheral chips, optimal allocation of signal paths and bandwidth resources is achieved. Flexible design of different bonding surface combinations (such as F2B and B2B) allows high-power chips to directly expose their back surfaces to enhance heat dissipation, thereby reducing overall thermal resistance. Design flexibility and compatibility are enhanced, allowing for flexible adjustment of bonding order and interface orientation according to system architecture (such as Flash–Logic–DRAM), compatibility with different process nodes and technology platforms, and support for heterogeneous chip integration. Balancing yield and cost, the hybrid bonding structure effectively reduces manufacturing complexity and yield loss, achieving an optimal trade-off between performance and cost. Signal integrity optimization is achieved by concentrating high-frequency interconnects at the F2F interface and transmitting low-speed interconnects via F2B or TSV, shortening signal paths, reducing crosstalk and delay, and improving overall system signal quality. The neural network processor provided in this embodiment breaks through the interconnect limitations of traditional two-layer stacking, offering a configurable three-dimensional integration framework for multi-layer heterogeneous chips. It achieves a system-level balance with adjustable overall performance, significantly improving the performance density and manufacturability of three-dimensional integrated systems.

[0086] Optionally, continue to refer to Figure 11 The logic chip 300 integrates at least one off-chip volatile memory chip controller 330, at least one off-chip non-volatile memory chip controller 320, and a global controller 340. The off-chip volatile memory chip controller 330 is used to access the off-chip volatile memory chip 200, and the off-chip non-volatile memory chip controller 320 is used to access the off-chip non-volatile memory chip 100. The global controller 340 is used to manage the digital in-memory computing unit 310, the off-chip volatile memory chip controller 330, and the off-chip non-volatile memory chip controller 320 of the logic chip 300, so as to realize the dynamic adaptation of data storage scheme and computing data flow, so as to globally optimize storage capacity, data bandwidth, access latency, and computing efficiency.

[0087] Furthermore, the global controller 340 can be configured to be software programmable, supporting storage strategies defined through upper-layer software. Regardless of the aforementioned bonding structure used in this implementation, different data storage mapping rules (such as task-media binding relationships and data migration priorities) can be configured through software, and performance tests can be conducted in conjunction with actual inference scenarios (such as bandwidth utilization, latency fluctuations, and thermal loss monitoring). Ultimately, the optimal storage solution suitable for the current task type is adaptively selected, ensuring that large language model inference can achieve maximum storage-computation collaborative efficiency under different hardware configurations.

[0088] Specifically, the global controller 340, the off-chip volatile memory chip controller 330, and the off-chip non-volatile memory chip controller 320 can be implemented by a CPU or other existing control modules or combinations of control modules capable of implementing the functions mentioned in this embodiment. This embodiment does not limit this.

[0089] Figure 20 illustrates a configurable storage scheme for large language model computation. Its core is a precise hierarchical matching based on data characteristics and storage medium capabilities, using off-chip non-volatile memory chip 100 as Flash and off-chip volatile memory chip 200 as DRAM as an example. For the highly dynamic KV Cache (requiring high-frequency updates and random access) and the activation values ​​and intermediate calculation results with high real-time requirements in large language model inference, this implementation utilizes the low latency and high random access throughput of DRAM to store them in DRAM, ensuring efficient read / write and update efficiency for dynamic data. However, the Q, K, V generation weights, Attention weights, and FFN weights, which have a large number of parameters, require high storage capacity and have relatively low access frequency. Therefore, this invention can store them in Flash, leveraging the large capacity and volatile characteristics of Flash to meet storage requirements. This solution, by differentiating the attributes of "dynamic calculation data - static weight data" and adapting the media, ensures both the response speed of the dynamic calculation process and meets the storage capacity requirements of large-scale weights, thus achieving efficient utilization of storage resources and cost balance in large language model inference scenarios.

[0090] During the large language model calculation process, the global controller 340 provided in this embodiment is configured to dynamically allocate the off-chip non-volatile memory chip controller 320, the off-chip volatile memory chip controller 330, and the digital in-memory computing unit 310 with DCIM as the core, based on the current three-dimensional integrated bonding structure (such as F2F / F2B / B2B) and the distribution of storage media, so as to achieve precise adaptation of hardware resources and computing tasks.

[0091] Optionally, the global controller 340 is configured to support the configuration of data storage mapping rules via upper-layer software.

[0092] Figure 21 shows a typical example of this scheduling logic: The neural network processor provided in this embodiment places static weights (or cold weights) in Flash memory (red box) with optimal capacity but high latency, places dynamic activations / intermediate states in low-latency DRAM chips (green box), and places the actual execution operators (matrix multiplication, Softmax, weighted sum, projection, FFN) in low-latency, high-computing-power logic chips (black box). The global controller 340 schedules the three to work together, minimizing cross-chip transmission and latency through caching / reuse and parallelization.

[0093] Since the generation of Q and K requires access to weight data stored in Flash (such as Q / K generation weights), the global controller 340 schedules the logic chip and Flash to work together to calculate Q and K. The generation of V involves more dynamic data interaction (such as combining with activation values), so it is executed by the logic chip in conjunction with the DRAM chip, leveraging the low latency of DRAM to ensure the real-time nature of V generation. In the Attention calculation phase, the logic chip first calculates similarity based on Q and K, then performs non-linear normalization operations such as Softmax, and finally performs a weighted fusion with the previously generated V, completing the core computational flow of the Attention mechanism.

[0094] Further, referring to Figure 22, it can be seen that the scheduling scheme of this embodiment can deeply explore hardware characteristics and build an efficient parallel pipeline coordination mechanism: In view of the hardware attribute of Flash having a fixed read latency, when the global controller triggers the Flash to read Q / K weight data, it will simultaneously start two key operations: First, it schedules the DRAM controller to perform the access to the dynamic data (such as activation value, intermediate variable) required for V generation; second, it starts the logic chip 300 to carry out preliminary basic operations (such as activation value normalization, data format conversion). By "parallel startup", hardware resources are avoided from being idle.

[0095] Specifically, when the global controller 340 triggers an Attention Block calculation, the system needs to read the weight parameters of Q / K from Flash.

[0096] Flash access has a fixed latency window (tens to hundreds of nanoseconds). Traditional architectures will only start subsequent operations after the Flash is read, resulting in wasted time waiting.

[0097] This implementation uses a time-parallel scheduling mechanism to start two tasks in parallel while initiating a Flash read operation: DRAM controller startup → Preloads the dynamic data (activation value, intermediate state) required to generate V.

[0098] The logic chip initiates preprocessing → activates existing inputs to perform normalization (Norm) and data format conversion (Quant / Dequant).

[0099] With this design, the Flash, DRAM, and logic chips work in parallel, and the Flash delay window is fully filled.

[0100] Since the sum of the access latency (t_DRAM) of the DRAM chip and the computing latency (t_logic) of the logic chip < the Flash latency (t_Flash).

[0101] Therefore, the neural network processing can complete all pre - tasks before the Flash weights are fully read. When the Flash data arrives, it immediately enters the core matrix calculation stage without idle cycles.

[0102] Thus, the Flash latency is completely hidden; the operation and memory access are fully overlapped; and an efficient scheduling of "continuous pipelining" is achieved.

[0103] Correspondingly, this embodiment provides a control method for a neural network processor, as Figure 23 shown, including the steps: Detect whether to trigger the calculation of the attention module; and When it is detected that the calculation of the attention module is triggered, while performing the read operation of the off - chip non - volatile storage chip, schedule the off - chip volatile storage chip to start data loading and the logic chip to start pre - processing.

[0104] Particularly crucial is that this embodiment uses the compute - in - memory (CIM) unit as the core computing execution unit. Its characteristic of "tight coupling of storage and computing" significantly shortens the data transfer path and operation time. Through the coordinated scheduling and timing optimization of the CIM unit, DRAM chip, and logic chip, it can be achieved that the sum of "DRAM parameter transfer latency + logic chip computing latency" < "Flash parameter transfer latency". This core breakthrough can completely hide the data transfer time of DRAM and the operation time of the logic chip within the latency window of Flash weight reading, completely eliminating the efficiency bottleneck of "hardware waiting for data" in the traditional architecture.

[0105] This design forms an essential difference from the traditional storage architecture: In the traditional architecture, due to the separation of storage and computing, data needs to be frequently transferred between different hardware, and operation and memory access cannot be deeply parallelized, making it difficult to completely hide the latency latency. While this invention, through the characteristic加持 of the CIM unit and pipeline timing optimization, enables the access of the DRAM chip, the calculation of the logic chip, and the Flash memory access to form a gapless parallel pipeline, which not only avoids the losses caused by single - hardware waiting but also maximizes the performance of each hardware module. Finally, it provides a "low - latency, high - throughput" operation guarantee for large - language model inference, significantly improving the utilization rate of hardware resources and the overall computing efficiency of the system.

[0106] Optionally, the off - chip non - volatile storage chip 100 can be composed of an off - chip non - volatile storage module 10, as Figure 24As shown, the off-chip non-volatile memory chip 100 can also be integrated by at least two off-chip non-volatile memory modules 10 using three-dimensional stacking technology.

[0107] Optionally, the off-chip volatile memory chip 200 can be composed of an off-chip volatile memory module 20, such as... Figure 25 As shown, the off-chip volatile memory chip 200 can also be integrated by at least two off-chip volatile memory modules 20 using three-dimensional stacking technology.

[0108] Optionally, the logic chip 300 can be composed of a logic module 30, such as Figure 26 As shown, the logic chip 300 can also be integrated by at least two logic modules 30 using three-dimensional stacking technology.

[0109] In this embodiment, each chip can be flexibly expanded by three-dimensional stacking to increase computing power and storage.

[0110] It should be noted that when multiple modules are stacked, their three-dimensional integration bonding method can be based on... Figures 13-16 The bonding methods shown can be combined in an adaptive manner based on the needs of the scenario, and all of the above implementation methods are within the protection scope of this application.

[0111] It should be noted that when the off-chip non-volatile memory module 10, the off-chip volatile memory module 20, and the logic module 30 are configured individually, they can independently implement the functions of the off-chip non-volatile memory chip 100, the off-chip volatile memory chip 200, and the logic chip 300.

[0112] It should be noted that each off-chip volatile memory module 20 is integrated face to face. The side of the structure formed by three-dimensional stacking and integration of at least two off-chip volatile memory modules 20 is defined as the front side of the off-chip volatile memory chip 200 near the logic control layer, and the side is defined as the back side of the off-chip volatile memory chip 200 near the memory array layer.

[0113] Taking two off-chip volatile memory modules 20 as an example, with them integrated face to face, the off-chip volatile memory chip 200 will show the back of the off-chip volatile memory module 20. The reason for defining the side closer to the logic control layer as the front is that the control circuits of the off-chip volatile memory module 20 are all on the logic control layer (CMOS Die), while the memory array layer (Array Die) cannot work independently. Therefore, it is more convenient to bond the logic control layer as the front side to the front side of the logic chip 300, which facilitates the transmission of signals and data.

[0114] Similarly, it should be noted that each off-chip non-volatile memory module 10 is integrated face to face. The side of the structure formed by three-dimensional stacking and integration of at least two off-chip non-volatile memory modules 10 is defined as the front side of the off-chip non-volatile memory chip 100 near the logic control layer, and the side is defined as the back side of the off-chip non-volatile memory chip 100 near the memory array layer.

[0115] Optionally, the three-dimensional stacking technology is one or more of TSV, hybrid bonding, or Filp Chip.

[0116] Optionally, the on-chip volatile memory chip is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

[0117] Optionally, the global controller 340 is configured to perform fine-grained dynamic scheduling of data based on the bonding scheme interconnect characteristics of the three-dimensional stacking technology and the inherent properties of each storage medium.

[0118] Optionally, the off-chip volatile memory chip 200 includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory chip controller integrated on the logic chip 300, enabling the off-chip volatile memory chip 200 of the neural network processor to be accessed by external devices or systems as a standard DRAM memory chip. The interface protocol meets one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6. In an architecture where the off-chip volatile memory chip 200 and the Logic Die are vertically stacked using three-dimensional integration technology (such as TSV and Hybrid Bonding), this standard interface can still ensure DRAM compatibility with external devices. At the same time, by leveraging the low latency and high bandwidth interconnect characteristics of three-dimensional integration, the data transmission efficiency during standard interface access is improved, avoiding the performance bottleneck of the standard interface under traditional 2D connection.

[0119] Optionally, the off-chip non-volatile memory chip 100 includes a standard interface (compliant with JEDEC or industry-standard memory interface standards). Taking Flash as an example, the Flash of the neural network processor can be accessed as a standard Flash memory chip. The protocol of this standard interface can be configured to meet one or more of the following: SPI (Serial Peripheral Interface), QSPI (Quad SPI), Octal SPI (OSPI), eMMC (embedded MultiMediaCard, compliant with JEDEC JESD216 standard), UFS (Universal Flash Storage, compliant with JEDEC JESD220 / JESD223 standard), NVMe (Non-Volatile Memory Express), and Parallel Flash.

[0120] The technical solution of the present invention has now been described in conjunction with the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to the specific embodiments described above. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor, characterized in that, include: An off-chip non-volatile memory chip is connected to the logic chip and is used to exchange information with the logic chip; An off-chip volatile memory chip, connected to the logic chip, is used for information exchange with the logic chip; and The logic chip is configured to perform any combination of the following functions: Perform computational tasks, cache data, and facilitate information exchange; The off-chip volatile memory chip, the logic chip, and the off-chip non-volatile memory chip are integrated using three-dimensional stacking technology, with the off-chip volatile memory chip located between the off-chip non-volatile memory chip and the logic chip in the vertical direction; wherein, the off-chip non-volatile memory chip is connected to the off-chip volatile memory chip, and the off-chip volatile memory chip can temporarily store the data stored in the off-chip non-volatile memory chip for the logic chip to access.

2. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 1, characterized in that, The off-chip non-volatile memory chip, the off-chip volatile memory chip, and the logic chip are integrated from top to bottom using three-dimensional stacking technology in the vertical direction.

3. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 1, characterized in that, The logic chip, the off-chip volatile memory chip, and the off-chip non-volatile memory chip are integrated from top to bottom using three-dimensional stacking technology in the vertical direction.

4. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-3, characterized in that, The logic chip is a computing chip that performs computing tasks, and the logic chip integrates a digital in-memory computing unit.

5. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-3, characterized in that, The logic chip is a data cache chip used for data caching.

6. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-3, characterized in that, The logic chip is an interface chip used to realize information interaction.

7. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 4, characterized in that, The logic chip integrates at least one off-chip volatile memory chip controller, at least one off-chip non-volatile memory chip controller, and a global controller; wherein: The off-chip volatile memory chip controller is used to access the off-chip volatile memory chip; The off-chip non-volatile memory chip controller is used to access the off-chip non-volatile memory chip; The global controller is used to manage the off-chip non-volatile memory chip controller, the off-chip volatile memory chip controller, and the digital in-memory computing unit, so as to realize the dynamic adaptation of data storage scheme and computing data flow, and to globally optimize storage capacity, data bandwidth, access latency and computing efficiency.

8. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-3, characterized in that, The off-chip volatile memory chip is a DRAM chip.

9. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 8, characterized in that, In large language model computing scenarios, the off-chip volatile memory chip is used to store key-value cache data, activation values, and intermediate calculation results.

10. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-3, characterized in that, The off-chip non-volatile memory chip is Flash.

11. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 8, characterized in that, In large language model computation scenarios, the off-chip non-volatile memory chip is used to store Q-generated weights, K-generated weights, V-generated weights, Attention weights, and FFN weights.

12. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-3, characterized in that, The off-chip volatile memory chip is composed of an off-chip volatile memory module; or, The chip is integrated using a three-dimensional stacking technique with at least two off-chip volatile memory modules, wherein each off-chip volatile memory module is integrated face to face. The side of the structure formed by the three-dimensional stacking of the at least two off-chip volatile memory modules closer to the logic control layer is defined as the front side of the off-chip volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip volatile memory chip.

13. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to any one of claims 1-3, characterized in that, The off-chip non-volatile memory chip is composed of an off-chip non-volatile memory module; or, The chip is integrated using three-dimensional stacking technology with at least two off-chip non-volatile memory modules. Each off-chip non-volatile memory module is integrated face to face. The side of the structure formed by the three-dimensional stacking of the at least two off-chip non-volatile memory modules closer to the logic control layer is defined as the front side of the off-chip non-volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip non-volatile memory chip.

14. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 4, characterized in that, The logic chip consists of a logic module; or, It is integrated by applying three-dimensional stacking technology to at least two logical modules.

15. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 1, characterized in that, The three-dimensional stacking technology is one or more of through-silicon via (TSV) technology, hybrid bonding technology, or flip-chip technology.

16. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 4, characterized in that, The digital in-memory computing unit is integrated with an on-chip volatile memory chip and a computing unit. The on-chip volatile memory chip is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

17. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 7, characterized in that, The global controller is used to perform fine-grained dynamic scheduling of data based on the interconnect characteristics of the bonding scheme of the three-dimensional stacking technology and the inherent properties of each storage medium.

18. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 7, characterized in that, The global controller is used to support the configuration of data storage mapping rules through upper-layer software.

19. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 7, characterized in that, The off-chip volatile memory chip includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory chip controller, enabling the neural network processor to be accessed by external devices as a standard DRAM memory chip; the interface protocol satisfies one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6.

20. The in-memory computing neural network processor with three-dimensional integrated heterogeneous storage medium according to claim 10, characterized in that, The Flash includes an interface that conforms to JEDEC or an industry-standard storage interface. The Flash of the neural network processor can be accessed as a standard Flash memory chip. The protocol of the interface meets one or more of SPI, QSPI, Octal SPI, eMMC, UFS, NVMe, and Parallel Flash.

21. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to claim 2 or 3, characterized in that, The off-chip volatile memory chip and the logic chip are bonded together in a three-dimensional stacked configuration, face to face.

22. The in-memory computing neural network processor with a three-dimensional integrated heterogeneous storage medium according to claim 2 or 3, characterized in that, The off-chip volatile memory chip and the logic chip are bonded together in a three-dimensional stacked manner, back to back.

23. A control method for a three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor, employing the three-dimensional integrated heterogeneous storage medium in-memory computing neural network processor according to any one of claims 1-22, characterized in that, Includes steps; Detect whether the attention module calculation has been triggered; as well as When the attention calculation is triggered, the off-chip non-volatile memory chip is scheduled to start data loading and the logic chip is scheduled to start calculation preprocessing while the off-chip non-volatile memory chip read operation is being performed.

Images