Three-dimensional integrated and substrate interconnected in-memory neural network processor

Abstract

The application relates to a memory-compute integrated neural network processor based on three-dimensional integration and substrate interconnection heterogeneous storage media and a control method thereof, which comprises an off-chip volatile storage chip connected with a logic chip and used for information interaction with the logic chip, an off-chip non-volatile storage chip connected with the logic chip and used for information interaction with the logic chip, and the logic chip configured to realize a combination of any of the following functions: executing a computing task, performing data caching and realizing information interaction; the off-chip volatile storage chip and the logic chip are integrated into a three-dimensional integrated module in a vertical direction by using a three-dimensional stacking technology; the three-dimensional integrated module and the off-chip non-volatile storage chip are integrated by using a substrate interconnection technology; in the three-dimensional integrated module, the off-chip volatile storage chip and the logic chip are bonded in a flip-chip bonding mode; and the high-speed memory access and high-bandwidth data interaction requirements of the off-chip volatile storage chip are met.

Description

Technical Field

[0001] This application relates to the field of semiconductor chip technology, and more specifically to a three-dimensional integrated and substrate interconnect heterogeneous storage medium in-memory neural network processor and its control method. Background Technology

[0002] Large Language Models (LLMs), as a core evolutionary direction of Natural Language Processing (NLP) technology in the field of Artificial Intelligence (AI), have shown an exponential growth trend in recent years. Their technological maturity and application breadth have continued to break through, demonstrating irreplaceable advantages in several key areas.

[0003] Meanwhile, the development of LLMs has gradually revealed a core principle widely validated by academia and industry—the scaling law. The core meaning of this principle is that, with sufficient high-quality training data and matching computational resources, the core performance indicators of LLMs (such as perplexity in language generation and accuracy in downstream tasks) show a significant positive correlation with the scale of model parameters. For example... Figure 1 As shown, the model performance score continuously improves with the exponential increase in the number of parameters.

[0004] Traditional computer systems rely on "storage system architecture" (such as...) Figure 2 As shown, the process of registers → on-chip cache → main memory → local disk → remote storage achieves a trade-off between capacity, speed, and cost. Figure 3 As shown, each tier exhibits significant differences in relative cost, storage density, bandwidth, access latency, and power consumption.

[0005] However, existing 3D integration solutions often lack fine-grained matching between device physical characteristics and task requirements when dealing with the integration and interconnection of heterogeneous storage media (volatile memory vs. non-volatile memory), and mainly suffer from the following technical pain points: Existing integration solutions often employ a uniform stacking or interconnect strategy for different types of memory, ignoring the significant differences in their memory access characteristics. The memory access latency of volatile memory chips (such as DRAM) is typically on the order of nanoseconds (ns), and their performance is highly dependent on the bit width and signal integrity of the I / O interface; while the latency of non-volatile memory chips (such as Flash) is on the order of microseconds (μs), limited by internal mechanisms rather than link bandwidth. In existing technologies, blindly incorporating non-volatile memory into high-density three-dimensional stacks not only fails to linearly improve performance by increasing I / O density but also results in redundant waste of expensive vertical interconnect resources (such as TSVs). Furthermore, in the three-dimensional stacked structure of logic chips and volatile memory chips, since both are high-power, high-heat-generating core devices, their tight three-dimensional stacking introduces thermal coupling problems, easily leading to heat accumulation between chips or performance throttling due to obstructed heat dissipation paths.

[0006] Therefore, it is necessary to improve existing neural network processors. Summary of the Invention

[0007] To address the above problems, this application provides a three-dimensional integrated in-memory neural network processor with substrate interconnect, comprising: an off-chip volatile memory chip connected to a logic chip for information interaction with the logic chip; an off-chip non-volatile memory chip connected to the logic chip for information interaction with the logic chip; and a logic chip configured to perform any combination of the following functions: performing computational tasks, performing data caching, and performing information interaction; the off-chip volatile memory chip and the logic chip are integrated using three-dimensional stacking technology to form a three-dimensional integrated module, and the three-dimensional integrated module is integrated with the off-chip non-volatile memory chip through substrate interconnect technology; wherein, in the three-dimensional integrated module, the off-chip volatile memory chip and the logic chip are bonded using a same-surface bonding method.

[0008] The off-chip volatile memory chip and the logic chip described in this application are bonded together using a same-side bonding method. Since the same-side bonding uses the same material, it is easier to bond and reduces the process cost.

[0009] Furthermore, considering that the memory access latency of off-chip volatile memory chips is on the order of nanoseconds (typically tens of nanoseconds), their memory access efficiency is greatly affected by the bit width and transmission rate of the external I / O interface. When the two are bonded in a face-to-face (F2F) manner, F2F bonding supports extremely high-density I / O interconnects, significantly increasing the interface bit width and supporting high-bandwidth data interaction. Compared with bonding schemes that require penetrating the silicon substrate and introducing parasitic capacitance and resistance, face-to-face bonding significantly reduces link loss, ensures the signal integrity of off-chip volatile memory under high-speed transmission, and reduces the error correction burden. In addition, the core memory access characteristics of off-chip non-volatile memory chips determine their adaptation requirements for interconnect bonding methods: firstly, the memory access latency is relatively high, typically on the order of 30-50μs microseconds; secondly, the amount of data read in a single operation is fixed, generally 4-16KB. From a performance bottleneck perspective, the memory access efficiency of off-chip non-volatile memory chips (such as Flash) is primarily constrained by their internal operating mechanisms, rather than the performance of external I / O interconnect links. This conclusion can be fully verified from two key dimensions: I / O density and I / O speed. From the perspective of I / O density, the core value of front-to-front bonding lies in improving I / O concurrent interaction capabilities to meet high-frequency data transmission demands. However, Flash itself does not require high-frequency, high-concurrency I / O interaction. Even with high-density F2F bonding, its memory access efficiency cannot be linearly improved; instead, it leads to ineffective redundancy and waste of I / O resources. From the perspective of I / O speed, compared to back-side bonding, it requires through-silicon vias (TSVs) to penetrate the silicon substrate. TSVs inevitably introduce parasitic capacitance and resistance, which affects high-speed I / O links, leading to reduced transmission rates and, in severe cases, even threatening data security.

[0010] Therefore, on the one hand, the front-to-front bonding method between the off-chip volatile memory chip and the logic chip increases the interface bit width, supports high-bandwidth data interaction, and meets the high-speed memory access and high-bandwidth data interaction requirements of the off-chip volatile memory chip. On the other hand, the off-chip non-volatile memory chip, due to its memory access latency reaching the microsecond level, has extremely low sensitivity to I / O rate and is not negatively affected by the front-to-front bonding between the off-chip volatile memory chip and the logic chip.

[0011] Since both the off-chip volatile memory chip and the logic chip generate a lot of heat during operation, and the back side of the chip is mainly composed of a silicon substrate with high thermal insulation performance, the back-to-back bonding method can avoid the heat transfer between the two and thus avoid affecting the computing efficiency due to heat generation. At the same time, the back-to-back bonding allows the front sides of the two chips to conduct heat to the outside through the substrate or directly to achieve a better heat dissipation effect.

[0012] Optionally, the logic chip is a computing chip that performs computing tasks, and the logic chip integrates a digital in-memory computing unit.

[0013] Optionally, the logic chip is a data cache chip for data caching.

[0014] Optionally, the logic chip is an interface chip used to realize information interaction.

[0015] Optionally, the logic chip integrates at least one off-chip non-volatile memory chip controller, at least one off-chip volatile memory chip controller, and a global controller; the off-chip non-volatile memory chip controller is used to access the off-chip non-volatile memory, and the off-chip volatile memory chip controller is used to access the off-chip volatile memory; the global controller is used to manage the off-chip volatile memory chip controller, the off-chip non-volatile memory chip controller, and the digital in-memory computing unit, to realize dynamic adaptation of data storage scheme and computing data flow, so as to globally optimize storage capacity, data bandwidth, access latency, and computing efficiency.

[0016] Optionally, the off-chip volatile memory chip is a DRAM (Dynamic Random Access Memory) chip.

[0017] Optionally, in a large language model computation scenario, the off-chip volatile memory chip is used to store key-value cache data, activation values, and intermediate computation results.

[0018] Optionally, the off-chip non-volatile memory chip is Flash memory.

[0019] Optionally, in the large language model computation scenario, the off-chip non-volatile memory chip is used to store Q-generated weights, K-generated weights, V-generated weights, Attention weights, and FFN weights.

[0020] Optionally, the off-chip volatile memory chip is composed of one off-chip volatile memory module, or is integrated by at least two off-chip volatile memory modules using three-dimensional stacking technology. The off-chip volatile memory modules are integrated face to face. The side of the structure formed by the three-dimensional stacking of the at least two off-chip volatile memory modules closer to the logic control layer is defined as the front side of the off-chip volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip volatile memory chip.

[0021] Optionally, the off-chip non-volatile memory chip is composed of one off-chip non-volatile memory module, or is integrated by at least two off-chip non-volatile memory modules using three-dimensional stacking technology; wherein, each of the off-chip non-volatile memory modules is integrated face to face, and the side of the structure formed by the three-dimensional stacking of the at least two off-chip non-volatile memory modules closer to the logic control layer is defined as the front side of the off-chip non-volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip non-volatile memory chip.

[0022] Optionally, the logic chip consists of a single logic module or is integrated from at least two logic modules using three-dimensional stacking technology.

[0023] Optionally, the three-dimensional stacking technology is one or more of through-silicon via (TSV) technology, hybrid bonding technology, or flip-chip technology.

[0024] Optionally, the digital in-memory computing unit is integrated with an on-chip volatile memory and a computing unit. The on-chip volatile memory is one or more of the following: SRAM (Static Random Access Memory), eDRAM (Embedded DRAM), DRAM (Dynamic Random Access Memory), Flash, MRAM (Magnetoresistive RAM), and ReRAM (Resistive RAM).

[0025] Optionally, the global controller is configured to perform fine-grained dynamic scheduling of data based on the interconnect characteristics of the bonding scheme of the three-dimensional stacking technology and the inherent properties of each storage medium.

[0026] Optionally, the global controller is configured to support the configuration of data storage mapping rules through upper-layer software.

[0027] Optionally, the off-chip volatile memory chip includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory controller integrated on the logic chip, enabling the neural network processor to be accessed by external devices as a standard DRAM memory; the interface protocol satisfies one or more of LPDDR (Low Power DDR) 5, LPDDR6, HBM (High Bandwidth Memory) 2, HBM3, HBM3e, HBM4, GDDR (Graphics DDR) 5, GDDR6, GDDR7, DDR (Double Data Rate SDRAM) 5, and DDR6.

[0028] Optionally, the Flash includes an interface conforming to JEDEC or an industry-standard memory interface, and the Flash of the neural network processor can be accessed as a standard Flash memory. The protocol of the interface satisfies one or more of the following: SPI (Serial Peripheral Interface), QSPI (Quad SPI), Octal SPI (OSPI), eMMC (embedded Multi-Media Card), UFS (Universal Flash Storage), NVMe (Non-Volatile Memory express), and Parallel Flash.

[0029] Optionally, the off-chip volatile memory chip or the logic chip, and the off-chip non-volatile memory chip are interconnected with the substrate through a silicon interposer.

[0030] Optionally, the off-chip volatile memory chip is stacked on top of the logic chip, with the front side of the off-chip volatile memory chip being three-dimensionally integrated with the front side of the logic chip, and the back side of the logic chip being interconnected with the substrate.

[0031] Optionally, the off-chip volatile memory chip is stacked on top of the logic chip, the back side of the off-chip volatile memory chip is three-dimensionally integrated with the back side of the logic chip, and the front side of the logic chip is interconnected with the substrate.

[0032] Optionally, the off-chip volatile memory chip is stacked below the logic chip, with the front side of the off-chip volatile memory chip being three-dimensionally integrated with the front side of the logic chip, and the back side of the off-chip volatile memory chip being interconnected with the substrate.

[0033] Optionally, the off-chip volatile memory chip is stacked below the logic chip, the back side of the off-chip volatile memory chip is three-dimensionally integrated with the back side of the logic chip, and the front side of the off-chip volatile memory chip is interconnected with the substrate.

[0034] To achieve the above-mentioned objectives, this application provides a control method for a three-dimensional integrated and substrate interconnected in-memory neural network processor, which utilizes the three-dimensional integrated and substrate interconnected heterogeneous storage medium described above, and includes the following steps. Detect whether the attention module computation is triggered; and When the attention module is triggered to perform calculations, it schedules the external volatile memory chip to start data loading and the logic chip to start preprocessing while performing a read operation on the external non-volatile memory chip.

[0035] The technical advantage of this control method lies in its ability to successfully "hide" the physical access bottleneck of Flash memory during the computational preprocessing process through parallel scheduling across media, thus overcoming the performance bottleneck caused by media speed mismatch in three-dimensional stacked heterogeneous systems. Attached Figure Description

[0036] Figure 1 A graph showing the relationship between the core performance metrics of a large language model and the scale of model parameters; Figure 2 A diagram of the storage system architecture of a traditional computer system; Figure 3 The performance and cost characteristics of each layer in the storage system architecture of traditional computer systems; Figure 4 This is a schematic diagram of a storage architecture based on a traditional computer system. Figure 5 A schematic diagram of the parameter transfer process for large language model computation based on traditional computer systems; Figure 6 A schematic diagram of the parameter transport process for large language model computation based on data offloading technology; Figure 7 This is a schematic diagram of a storage packaging structure based on Chiplet and Flip Chip technologies; Figure 8 This is a schematic diagram of a storage packaging structure based on three-dimensional integration technology; Figure 9 This is a schematic diagram of a storage packaging structure based on three-dimensional integration technology; Figure 10 A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention; Figure 11A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention; Figure 12 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 15 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 16 A schematic diagram illustrating the features of 3D integrated bonding for different interface positional relationships; Figure 17 This is a schematic diagram comparing the implementation of the present invention with the storage architecture of the prior art; Figure 18 This is a schematic diagram of the structure of the digital in-memory computing unit in an embodiment of the present invention; Figure 19 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 20 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 21 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 22 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 23 This is a schematic diagram of one of the storage schemes for computing large language models provided in the embodiments of the present invention; Figure 24 This is a flowchart illustrating a computation scheduling scheme for a global controller of a neural network processor provided in an embodiment of the present invention. Figure 25 This is a timing diagram of the parallel pipeline coordination mechanism of the global controller of the neural network processor provided in an embodiment of the present invention; Figure 26 This is a schematic diagram illustrating the steps of the control method for a neural network processor provided in an embodiment of the present invention. Figure 27 This is a schematic diagram of the structure of the off-chip volatile memory chip extension of the neural network processor provided in an embodiment of the present invention; Figure 28 This is a schematic diagram of the structure of the off-chip non-volatile memory chip extension of the neural network processor provided in an embodiment of the present invention; Figure 29 This is a schematic diagram of the logic chip extension of the neural network processor provided in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] Due to differences in process compatibility between non-volatile memory, volatile memory, and computing logic (such as the mismatch between advanced processes for computing logic and mature processes for memory chips, and the high integration difficulty caused by different material systems), considering factors such as area utilization, chip yield, and process feasibility, adopting an integrated solution would lead to problems such as excessive chip area redundancy, a significant decrease in yield, and insufficient process adaptability. Therefore, if... Figure 4 As shown, traditional computer systems break it down into multiple chips for independent implementation.

[0039] Based on traditional computer system storage architecture, storage systems can be divided into four categories: off-chip non-volatile memory (such as remote storage, SSD, NAND Flash, ReRAM, etc.), off-chip volatile memory (such as DRAM), on-chip volatile memory (such as SRAM), and registers; for example Figure 5 As shown, when deploying large language model computation in an existing system, the process is usually as follows: After the AI ​​task is started, the main control chip first loads as much large model parameter data as possible from the non-volatile memory to the off-chip volatile memory, then moves the data to be computed from the off-chip volatile memory to the on-chip volatile memory, then selects the data that needs to be computed immediately and sends it to the register, and finally completes the computation in the computing unit.

[0040] However, as analyzed above, the inference process of LLM (Large Language Model) under traditional architecture still faces severe capacity and bandwidth bottlenecks. To effectively alleviate this problem, the industry has successively explored several technical solutions, as follows: First, data offloading technology. For example... Figure 6As shown, its core logic is "trading communication for capacity": compared to the fixed data loading path of the traditional deployment process in Figure 5, this technology unloads data that is not currently needed in the computing stage from high-speed on-chip storage (such as SRAM) and stores it back to off-chip volatile memory (such as DDR) or off-chip non-volatile memory (such as SSD); when subsequent computing requires the data, it is retrieved from the corresponding storage medium and reloaded onto the chip. This approach can avoid irrelevant data occupying scarce on-chip storage resources for a long time, thereby directly alleviating the storage capacity bottleneck, and is especially suitable for LLM tasks that process long sequence inputs (such as long document understanding and multi-turn dialogue).

[0041] However, the core limitation of data offloading technology lies in the fact that it does not change the traditional "compute-storage separation" memory architecture of computing systems. While this technology can alleviate storage capacity bottlenecks by "trading communication for capacity" and even relieve bandwidth pressure to some extent, this process also comes at a significant cost: frequent data offloading and retrieval generate a large number of on-chip and off-chip communication needs, which in turn keeps the system continuously subject to severe bandwidth constraints. Even if the system can therefore support neural networks with a larger number of parameters, the frequent transfer of large amounts of data between on-chip and off-chip environments will severely drag down computational efficiency, ultimately resulting in significant limitations on the overall inference efficiency of large models.

[0042] Second, high-density packaging technology. For example... Figure 7 As shown, this technology, through processes such as Chiplet and Flip Chip, breaks the physical limitations of traditional planar packaging, integrating multiple functional chips (such as logic chips and cache chips) into a single package in a more compact manner. Its core function is to shorten the physical transmission distance across memory levels (such as on-chip cache and off-chip memory, or cache between different computing units), reducing data transmission latency and improving overall bandwidth through parallelized transmission paths, thereby alleviating the problem of low "data moving" efficiency.

[0043] High-density packaging technology is effective in alleviating the bandwidth bottleneck of traditional memory architectures, but it has two key limitations. First, it does not break through the underlying design of traditional memory architectures, but only improves data transmission efficiency by optimizing the packaging form, so the core problem of limited storage capacity is not solved. Second, its performance improvement has a clear ceiling, and it will continue to be limited by the speed limit of the I / O interface, making it difficult to achieve unlimited bandwidth breakthroughs.

[0044] Third, three-dimensional integration and substrate interconnection technologies. For example... Figure 8 , Figure 9As shown, this technology overcomes the spatial limitations of two-dimensional packaging through key processes such as TSV (Through Silicon Via) and Hybrid Bonding, enabling vertical stacking integration of logic chips and memory. For example, the widely used HBM (High Bandwidth Memory) technology uses TSV to vertically stack multiple DRAM chips next to the logic chip, forming a tightly coupled "compute-storage" structure. Compared to traditional DDR memory, HBM's bandwidth can be increased several times (e.g., HBM3 has a bandwidth of 819 GB / s), while significantly reducing the area occupied by the planar circuit board, fundamentally optimizing the "storage-compute" interaction efficiency required for LLM inference.

[0045] 3D integration and substrate interconnect technologies, with their 3D stacking structure and high-density interconnect design, can effectively address the capacity shortage and bandwidth limitations faced by traditional memory architectures. However, existing implementation solutions for this technology still have significant limitations. Current mainstream solutions (such as HBM high-bandwidth memory and NAND Flash memory) are essentially still limited to the stacking and interconnection of similar types and levels of memory—they have neither broken through the underlying framework of traditional memory architectures nor achieved a synergistic breakthrough in capacity and bandwidth through a unified memory structure.

[0046] Meanwhile, current optimization solutions for storage architecture mostly focus on simply expanding capacity and increasing bandwidth, neglecting the core characteristic of computer systems: the dynamic transfer of bottlenecks. When capacity and bandwidth bottlenecks are alleviated through technical means, the processing power of computing units often becomes a new performance constraint. For example, in large model inference scenarios, even if high-speed data storage and transmission are achieved through technologies such as HBM and Chiplet, if the computing power of logic chips such as GPUs / TPUs cannot match the data throughput, resource idleness will still occur, resulting in "data waiting for computation," making it difficult to improve the overall system efficiency. More importantly, existing solutions generally adopt a "fragmented optimization" approach, failing to build a unified architecture that coordinates computing power, storage efficiency, and bandwidth performance. This fails to break the cycle of "solving old bottlenecks and creating new bottlenecks" at the system level, making it difficult to achieve synergistic breakthroughs in the performance of various components.

[0047] Therefore, existing solutions have failed to break free from the constraints of the underlying framework of traditional memory architectures. The core problem lies in the inability to achieve coordinated optimization of capacity, bandwidth, and computation based on a unified storage structure. This structural shortcoming directly results in significant performance limitations and severe implementation challenges for large language models when deployed for inference in systems built on existing solutions.

[0048] If the applicant is aware of the aforementioned technical issues, Figures 10-15As shown, this embodiment provides a neural network processor based on in-memory computing and three-dimensional stacking technology, including: an off-chip volatile memory chip 100 connected to a logic chip 300 for information interaction; an off-chip non-volatile memory chip 200 connected to the logic chip 300 for information interaction; and a logic chip 300 configured to perform any combination of the following functions: performing computational tasks, performing data caching, and performing information interaction; the off-chip volatile memory chip 100 and the logic chip 300 are integrated vertically in a three-dimensional integrated module using three-dimensional stacking technology, and the three-dimensional integrated module is integrated with the off-chip non-volatile memory chip 200 through substrate interconnect technology; wherein, in the three-dimensional integrated module, the off-chip volatile memory chip 100 and the logic chip 300 are bonded using a same-surface bonding method.

[0049] In the field of 3D integration technology, each chip (die / core) includes: Face: This refers to the side where the metal interconnect layer is located, including the top layer wiring, solder joints (micro-bumps), or redistribution layer (RDL).

[0050] Back side: This is the silicon substrate side, usually the bare silicon surface, which is electrically connected to the front side through thinning, opening, or TSV (through silicon via).

[0051] like Figure 16 The diagram illustrates the characteristics of 3D integration and substrate interconnect bonding under different interface positional relationships. The interface positional relationships for 3D integration and substrate interconnection mainly fall into three types: face-to-face (F2F), face-to-back (F2B), and back-to-back (B2B).

[0052] In front-to-front mode, interconnect density and bandwidth are high, RDL rerouting is optional, cost (yield) is low, signal integrity is high, but thermal management is difficult; in front-to-back mode, interconnect density and bandwidth are low, RDL rerouting is necessary, cost (yield) and signal integrity are both moderate, and thermal management is moderate; in back-to-back mode, interconnect density and bandwidth are low, RDL rerouting is necessary, cost (yield) is high, signal integrity is low, but thermal management is relatively easy.

[0053] Despite the superior overall performance of F2F, in a three-layer stacking scenario, if two layers are to be bonded using F2F, the third layer will inevitably form an F2B structure with one of the layers. Therefore, three-layer stacking requires comprehensive consideration of hardware factors such as interconnect density, cost, signal integrity, and thermal management to determine the specific arrangement of F2F, F2B, and B2B, resulting in higher design complexity compared to two-layer stacking.

[0054] In this embodiment, considering that the memory access latency of off-chip volatile memory chips is on the order of nanoseconds (typically tens of nanoseconds), their memory access efficiency is greatly affected by the bit width and transmission rate of the external I / O interface. When the two are bonded in a face-to-face (F2F) manner, F2F bonding supports extremely high-density I / O interconnects, significantly increasing the interface bit width and supporting high-bandwidth data interaction. Compared with bonding schemes that require penetrating the silicon substrate and introducing parasitic capacitance and resistance, face-to-face bonding significantly reduces link loss, ensures the signal integrity of off-chip volatile memory under high-speed transmission, and reduces the error correction burden. In addition, the core memory access characteristics of off-chip non-volatile memory chips determine their adaptation requirements for interconnect bonding methods: firstly, the memory access latency is relatively high, typically on the order of 30-50μs microseconds; secondly, the amount of data read in a single operation is fixed, generally 4-16KB. From a performance bottleneck perspective, the memory access efficiency of off-chip non-volatile memory chips (such as Flash) is primarily constrained by their internal operating mechanisms, rather than the performance of external I / O interconnect links. This conclusion can be fully verified from two key dimensions: I / O density and I / O speed. From the perspective of I / O density, the core value of front-to-front bonding lies in improving I / O concurrent interaction capabilities to meet high-frequency data transmission demands. However, Flash itself does not require high-frequency, high-concurrency I / O interaction. Even with high-density F2F bonding, its memory access efficiency cannot be linearly improved; instead, it leads to ineffective redundancy and waste of I / O resources. From the perspective of I / O speed, compared to back-side bonding, it requires through-silicon vias (TSVs) to penetrate the silicon substrate. TSVs inevitably introduce parasitic capacitance and resistance, which affects high-speed I / O links, leading to reduced transmission rates and, in severe cases, even threatening data security.

[0055] Therefore, in this embodiment, on the one hand, the front-to-front bonding method between the off-chip volatile memory chip 100 and the logic chip 300 increases the interface bit width, supports high-bandwidth data interaction, and meets the high-speed memory access and high-bandwidth data interaction requirements of the off-chip volatile memory chip; on the other hand, the off-chip non-volatile memory chip 200 has a memory access latency in the microsecond range and is extremely insensitive to I / O rate. Therefore, it is not negatively affected by the front-to-front bonding method between the off-chip volatile memory chip 100 and the logic chip 300.

[0056] In addition, since both the off-chip volatile memory chip and the logic chip generate a lot of heat during operation, and the back side of the chip is mainly composed of a silicon substrate with high thermal insulation performance, the back-to-back bonding method can avoid the heat transfer between the two and thus avoid affecting the computing efficiency due to heat generation. At the same time, the back-to-back bonding allows the front sides of the two chips to conduct heat to the outside through the substrate or directly to achieve a better heat dissipation effect.

[0057] Optionally, the off-chip volatile memory chip 100 is a DRAM chip. For ease of explanation, a DRAM chip will be used as an example of the off-chip volatile memory chip 100 in the following text.

[0058] Optionally, the off-chip non-volatile memory chip 200 is Flash. For ease of explanation, Flash is used as an example of the off-chip non-volatile memory chip 200 in the following text.

[0059] Optionally, the logic chip is a computing chip for performing computing tasks. The logic chip 300 integrates a digital in-memory computing unit 310 for performing computing tasks. In the following text and figures, for ease of explanation, the logic chip 300 is used as an example of a computing chip. Based on the spirit of the present invention, it can be understood that the logic chip can also be configured as a data cache chip and an interface chip as described below to achieve functions adapted to the application scenario.

[0060] Optionally, the logic chip is a data cache chip used for data caching, which temporarily stores data between computing and storage, improving access efficiency and reducing latency and power consumption caused by frequent access to external storage.

[0061] Optionally, the logic chip is an interface chip used to realize information interaction, and is used to realize information interaction and protocol conversion between different chips or systems.

[0062] Optionally, a neural network processor may contain multiple logic chips, such as three, which may be configured as a computing chip, an interface chip, and a data cache chip, respectively. These multiple chips may be integrated using a three-dimensional stacking technique to achieve flexible and diverse functions.

[0063] It should be noted that the off-chip volatile memory chip 100 and the off-chip non-volatile memory chip 200 can also be implemented using other storage media / memory that meet the requirements, and this embodiment does not limit them.

[0064] The neural network processor provided in this embodiment can achieve at least the following technical effects: like Figure 17 As shown, this implementation constructs a disruptive and innovative storage architecture, completely breaking through the inherent framework of traditional memory's "layer-by-layer cascading and unidirectional data transport"—traditional storage systems rely on a fixed hierarchy of "on-chip cache → off-chip storage → external storage," requiring data to be transferred level by level to meet computational demands. This not only generates a large amount of redundant data transport overhead but also leads to bandwidth limitations and latency accumulation due to the large physical distance between levels, becoming a core efficiency bottleneck for computationally intensive tasks such as large language models. The architecture provided by this implementation achieves end-to-end optimization through the synergistic innovation of three-dimensional stacking and in-memory computing technology: on the one hand, three-dimensional integration and substrate interconnect technology directly stack off-chip DRAM chips and logic chips in three dimensions, shortening the physical distance between them to the micrometer level. This improves off-chip storage access bandwidth through high-density bonding and reduces cross-chip transmission losses to lower latency. On the other hand, in-memory computing technology deeply integrates on-chip memory (such as a DCIM array based on 6T SRAM) with computing units, enabling operations such as multiplication and accumulation to be performed directly within the on-chip memory, fundamentally eliminating latency and energy overhead from on-chip data transport, ultimately building a system with "low latency, high bandwidth, and low power consumption." The storage-computing collaborative system completely breaks free from the performance constraints of traditional hierarchical architectures.

[0065] In terms of capacity, traditional architectures rely on planar expansion (increasing chip area or adding independent storage modules), which is easily limited by packaging space and power consumption budgets. Furthermore, capacity increases are prone to performance waste due to bandwidth mismatch. This implementation achieves a leapfrog, bottleneck-free capacity expansion through "three-dimensional vertical stacking + high-density media selection": using NAND Flash as the core high-density storage medium, bonding technology can increase the number of three-dimensionally stacked Flash chips—for example, an 8-layer stacked Flash module / unit. Within the same planar area as traditional 2D packaging, a single module can achieve a capacity of 60-128GB (traditional 2D Flash only 10-20GB in the same area). For further capacity expansion, the number of Flash stacking layers can be increased in the three-dimensional architecture (current processes support 200+ layers of 3D NAND), or multiple sets of Flash-three-dimensional stacked units can be connected in parallel, and the expansion process does not occupy additional planar space.

[0066] In terms of bandwidth, the bandwidth of traditional 2D architecture depends on the upgrade of interface protocol (such as DDR5→DDR6), the single-channel bandwidth improvement is limited (about 30%-50% per generation), and multi-channel expansion is easily limited by the motherboard wiring density. This implementation achieves efficient and scalable bandwidth expansion through "3D stacking + substrate interconnection + multi-chip parallel processing". For example, by increasing the number of 3D stacks of high-bandwidth media (DRAM / SRAM), such as vertically integrating two groups of DRAM chips with logic chips via F2F bonding, each group of DRAM chips provides 200-300GB / s bandwidth, and the two groups in parallel can achieve a total bandwidth of 400-600GB / s. At the same time, the micron-level wiring capability of the silicon interposer (supporting I / O densities of more than 500 pins / mm²) can be used to construct a parallel interconnect channel of "multi-memory chip-logic chip", avoiding the wiring congestion problem of traditional 2D architecture. In addition, this implementation can preferentially use F2F hybrid bonding for high-frequency interaction links (such as Logic Die and DRAM), reducing interconnect latency to the sub-nanosecond level, further releasing bandwidth potential.

[0067] With the storage and bandwidth issues resolved through the aforementioned solutions, computing power has become a new bottleneck in the systemic problem of storage-bandwidth-computing power. The expansion of computing power in traditional architectures relies on increasing the area of ​​logic chips or upgrading process nodes, which is costly and easily limited by heat dissipation (increased area leads to increased heat density). At the same time, after the computing power is increased, it is easy to cause "computing power idle" due to insufficient storage bandwidth. This implementation method, while retaining the process node improvement, uses Compute-In-Memory (CIM) technology as its core to achieve high-density, low-power expansion of computing power. On the one hand, the number of CIM macrocells can be increased modularly, with a DCIM array based on 6T SRAM as the basic unit. Each unit can provide 100-150 TOPS of computing power. By increasing the number of CIM unit layers through three-dimensional stacking (such as 4-layer CIM stack), a total computing power of 400-600 TOPS can be achieved, and the computing power density per unit area is 2-3 times that of traditional GPUs. Upgrading the chip logic process node: upgrading the Logic Die from 14nm to 3nm, and coordinating with the process optimization of CIM units, the computing power per unit area can be further increased by 50%-80% (the computing power density of 3nm CIM units can reach 300 TOPS / mm²). Specifically, this implementation method integrates computing units in the on-chip volatile memory on the logic chip 300, so that the calculation data does not need to be transferred to registers through the on-chip volatile memory, but can be calculated directly inside the on-chip volatile memory. The on-chip volatile memory of the integrated computing unit is designed as a Digital Computing-In-Memory (DCIM) unit. For example... Figure 18 As shown, the structure of the digital in-memory computing unit 310 mainly includes a DCIM array based on 6T SRAM, a word line decoder (WL DEC), a word line buffer (WL DRV), read / write circuitry, and a controller. The DCIM array consists of C DCIM columns, each containing R subarrays and an adder tree. Each subarray integrates 6T SRAM bit cells and a Local Read-Out and Compute Unit (LRCC), storing N-bit weights per row. Functionally, the DCIM macro can perform matrix-vector multiplication (MVM) multiplication-accumulation (MAC) calculations: the operating mode is switched via the M signal; when M is low, the N-bit weights in the subarray are multiplied by a 1-bit input, read and latched by the LRCC, accumulated by the adder tree, and then sent to a shift-and-accumulator to extend the input precision in the time dimension, ultimately completing a full MVM operation with K-bit input and N-bit weights, while also supporting the storage and efficient computation of multi-weight matrices.

[0068] In summary, the neural network processor provided in this embodiment achieves direct contact between the high-heat chip and the external heat dissipation structure by placing the off-chip volatile memory chip (DRAM) on the top layer of the stack, thus shortening the heat conduction path; effectively reducing the temperature rise of the logic chip and the non-volatile memory chip, avoiding heat accumulation, and improving the thermal stability and long-term reliability of the system.

[0069] The storage architecture is reconstructed (end-to-end bandwidth optimization), breaking the traditional "cache-external storage-external storage" layered model. Through three-dimensional bonding, DRAM chips and logic chips are directly vertically interconnected, shortening the interconnection distance to the micrometer level. At the same time, combined with in-memory computing (CIM) technology, data can be multiplied and accumulated within the storage array, reducing data transfer latency and energy consumption, and building a low-latency, high-bandwidth data path.

[0070] Storage capacity expansion and density improvement are achieved by using "three-dimensional vertical stacking + substrate interconnect high-density NAND Flash" to realize three-dimensional expansion of storage capacity, increasing capacity by 3 to 6 times without occupying additional planar area; it supports hundreds of layers of Flash stacking or multiple units in parallel, breaking through the limitations of packaging space.

[0071] Bandwidth enhancement and interconnect optimization are achieved through F2F bonding and parallel interconnection of multiple DRAM chips to achieve linear bandwidth superposition (200–300 GB / s per channel, total bandwidth up to 400–600 GB / s); with the help of the micron-level wiring capability of silicon interposer, I / O density is significantly improved and interconnect latency is reduced to sub-nanosecond level.

[0072] The improved computing power density and energy efficiency are achieved by embedding DCIM (Digital In-Memory Computing) units based on 6T SRAM into the logic chip, realizing "storage as computing". By increasing the number of CIM macrocell layers through three-dimensional stacking and upgrading advanced process nodes (14nm→3nm), the computing power density per unit area is increased to 2 to 3 times that of traditional GPUs, and the energy efficiency ratio is improved by about 50% to 80%.

[0073] The neural network processor provided in this embodiment forms a collaborative closed loop in terms of storage, bandwidth, and computing power under a three-dimensional stacked and substrate interconnect architecture: high-density storage provides sufficient data; high-bandwidth paths ensure real-time supply; and the in-memory computing unit efficiently performs calculations; thus achieving a performance balance of "low latency, high computing power, and low power consumption" at the system level.

[0074] Through a systematic design of "three-dimensional stacking + substrate interconnection + in-memory computing integration", a comprehensive breakthrough has been achieved in the three dimensions of data path, storage capacity and computing power density of neural network processors, forming a new computing architecture with high integration, high bandwidth, scalability and low power consumption, which is suitable for computing power intensive scenarios such as large language models, visual computing, and edge inference.

[0075] Optionally, refer to Figure 12 An off-chip volatile memory chip 100 is stacked on top of a logic chip 300. The front side of the off-chip volatile memory chip 100 is three-dimensionally integrated with the front side of the logic chip 300, and the back side of the logic chip 300 is interconnected with the substrate 400.

[0076] Optionally, refer to Figure 13 An off-chip volatile memory chip 100 is stacked on top of a logic chip 300. The back side of the off-chip volatile memory chip 100 is three-dimensionally integrated with the back side of the logic chip 300, and the front side of the logic chip 300 is interconnected with the substrate 400.

[0077] Optionally, refer to Figure 14 An off-chip volatile memory chip 100 is stacked below the logic chip 300. The front side of the off-chip volatile memory chip 100 is three-dimensionally integrated with the front side of the logic chip 300, and the back side of the off-chip volatile memory chip 100 is interconnected with the substrate 400.

[0078] Optionally, refer to Figure 15 An off-chip volatile memory chip 100 is stacked below the logic chip 300. The back side of the off-chip volatile memory chip 100 is three-dimensionally integrated with the back side of the logic chip 300, and the front side of the off-chip volatile memory chip 100 is interconnected with the substrate 400.

[0079] Optionally, such as Figures 19-22 As shown, in application Figures 12-15Based on the eight bonding methods described above, in some embodiments, the neural network processor provided in this application interconnects each chip with the substrate 400 through a silicon interposer 500. As the core carrier of high-density interconnection, the silicon interposer, with its unique material and process characteristics, exhibits interconnection advantages far exceeding those of traditional organic substrates: First, relying on the high-precision photolithography process of silicon materials, it can achieve micron-level or even submicron-level wiring density, supporting parallel interconnection of tens of thousands to hundreds of thousands of I / O interfaces, perfectly adapting to the high-density signal interaction requirements of multiple chips (such as Logic Die, DRAM, Flash) in three-dimensional stacking; Second, the low dielectric loss characteristics of silicon significantly reduce parasitic capacitance and inductance in signal transmission, significantly improving the integrity of high-frequency signals and reducing data transmission error rate; Third, its thermal conductivity is higher than that of organic substrates, which can help disperse the heat of stacked chips, forming a synergy with the heat dissipation design of three-dimensional integration; In addition, the silicon interposer can be compatible with heterogeneous chips of different process nodes and sizes, simplifying the integration difficulty of multiple types of chips through a unified interconnection interface, providing underlying support for the flexibility of three-dimensional stacking.

[0080] Optionally, continue to refer to Figure 11 The logic chip 300 integrates at least one off-chip non-volatile memory chip controller 330, at least one off-chip volatile memory chip controller 320, and a global controller 340. The off-chip non-volatile memory chip controller 330 is used to access the off-chip non-volatile memory 200, and the off-chip volatile memory chip controller 320 is used to access the off-chip volatile memory 100. The global controller 340 is used to manage the digital in-memory computing unit 310, the off-chip non-volatile memory controller 330, and the off-chip volatile memory chip controller 320 of the logic chip 300, so as to realize the dynamic adaptation of data storage scheme and computing data flow, so as to globally optimize storage capacity, data bandwidth, access latency, and computing efficiency.

[0081] Optionally, to meet the performance requirements of large language model inference tasks (such as long sequence generation and high-concurrency query), the global controller 340 in this embodiment can combine the interconnection characteristics of the bonding scheme (such as bandwidth and thermal efficiency) with the inherent properties of the storage medium (such as capacity and latency) to achieve fine-grained dynamic management of the data storage scheme. Specifically, targeted scheduling will be implemented based on the different hardware resource requirements of each task: high-bandwidth task data (such as key-value pair cache in the attention mechanism) will be stored in memory chips bonded to logic chip 300 using F2F, as these types of memory can match the high-frequency data interaction requirements due to their high interconnect bandwidth characteristics; heat-sensitive task data (such as intermediate results of large-scale matrix operations) will be stored in memory chips bonded to logic chip 300 using F2B or B2B, as these bonding schemes have better thermal management capabilities and can avoid local overheating affecting data stability; at the same time, for the differentiated requirements of storage capacity and latency, non-real-time data requiring large-capacity storage (such as pre-trained weight backups) will be stored in off-chip non-volatile memory chips, while latency-sensitive real-time computing data (such as the embedding vector of the input sequence) will be stored in off-chip volatile memory media, forming a storage architecture that precisely matches "features and requirements".

[0082] In this embodiment, data can be scheduled to memory chips with different bonding methods based on data type, so as to achieve the adaptation of data characteristics and bonding characteristics.

[0083] In this implementation, computing power and storage / bandwidth work in tandem. When expanding computing power, the bonding methods between different chips are flexibly adjusted according to actual needs, ensuring the bonding method matches the application scenario. For example, F2F bonding increases the bandwidth of off-chip volatile memory, while B2B bonding increases the capacity of off-chip non-volatile memory, ensuring a match between "computing power enhancement and data supply" and avoiding bottlenecks. By using F2F bonding between high-performance chips and F2B / B2B bonding between low-speed or peripheral chips, optimal allocation of signal paths and bandwidth resources is achieved. Flexible design of different bonding surface combinations (such as F2B and B2B) allows high-power chips to directly expose their back surfaces to enhance heat dissipation, thereby reducing overall thermal resistance. Design flexibility and compatibility are enhanced; the bonding sequence and interface direction can be flexibly adjusted according to the system architecture (such as Logic-DRAM), ensuring compatibility with different process nodes and technology platforms, and supporting heterogeneous chip integration. A balance between yield and cost is achieved; the use of hybrid bonding structures effectively reduces manufacturing complexity and yield loss, achieving an optimal trade-off between performance and cost. Signal integrity optimization is achieved by concentrating high-frequency interconnects at the F2F interface and transmitting low-speed interconnects via F2B or TSV, thereby shortening signal paths, reducing crosstalk and delay, and improving the overall signal quality of the system. The neural network processor provided in this embodiment breaks through the interconnect limitations of traditional two-layer stacking, offering a configurable 3D integration and substrate interconnect framework for multi-layer heterogeneous chips. It achieves a system-level balance with adjustable overall performance, significantly improving the performance density and manufacturability of 3D integration and substrate interconnect systems.

[0084] Furthermore, the global controller 340 can be configured to be software programmable, supporting storage strategies defined through upper-layer software. Regardless of the aforementioned bonding structure used in this implementation, different data storage mapping rules (such as task-media binding relationships and data migration priorities) can be configured through software, and performance tests can be conducted in conjunction with actual inference scenarios (such as bandwidth utilization, latency fluctuations, and thermal loss monitoring). Ultimately, the optimal storage solution suitable for the current task type is adaptively selected, ensuring that large language model inference can achieve maximum storage-computation collaborative efficiency under different hardware configurations.

[0085] Specifically, the global controller 340, the off-chip non-volatile memory chip controller 330, and the off-chip volatile memory chip controller 320 can be implemented by a CPU or other existing control modules or combinations of control modules capable of implementing the functions mentioned in this embodiment. This embodiment does not limit this.

[0086] Figure 23This paper presents a configurable storage scheme for large language model computation. Its core is a precise hierarchical matching based on data characteristics and storage medium capabilities, using an off-chip volatile memory chip 100 as a DRAM chip and an off-chip non-volatile memory 200 as Flash as an example. For the highly dynamic KV Cache (requiring high-frequency updates and random access) and the activation values ​​and intermediate calculation results with high real-time requirements in large language model inference, this implementation can utilize the low latency and high random access throughput of DRAM to store them in DRAM, ensuring efficient read / write and update efficiency of dynamic data. However, the Q-generation weights, K-generation weights, V-generation weights, Attention weights, and FFN weights, which have a large number of parameters, have high storage capacity requirements and relatively low access frequency. Therefore, this invention can store them in Flash, leveraging the large capacity and non-volatile characteristics of Flash to meet storage needs. This solution, by differentiating the attributes of "dynamic calculation data - static weight data" and adapting the media, ensures both the response speed of the dynamic calculation process and meets the storage capacity requirements of large-scale weights, thus achieving efficient utilization of storage resources and cost balance in large language model inference scenarios.

[0087] During the large language model calculation process, the global controller 340 provided in this embodiment is configured to dynamically allocate the off-chip volatile memory controller 320, the off-chip non-volatile memory controller 330, and the digital in-memory computing unit 340 with DCIM as the core based on the current three-dimensional integration and substrate interconnection bonding structure (such as F2F / F2B / B2B) and storage medium distribution, so as to achieve precise adaptation of hardware resources and computing tasks.

[0088] Optionally, the global controller 340 is configured to support the configuration of data storage mapping rules via upper-layer software.

[0089] Figure 24 shows a typical example of this scheduling logic: The neural network memory provided in this embodiment places static weights (or cold weights) in Flash memory (red box) with optimal capacity but high latency, places dynamic activations / intermediate states in low-latency DRAM chips (green box), and places the actual execution operators (matrix multiplication, Softmax, weighted sum, projection, FFN) in low-latency, high-computing-power logic chips (black box). The global controller 340 schedules the three components to work together, minimizing cross-chip transmission and latency through caching / reuse and parallelization.

[0090] Since the generation of Q and K requires access to weight data stored in Flash (such as Q / K generation weights), the global controller 340 schedules the logic chip and Flash to work together to complete the calculation of Q and K. The generation of V involves more dynamic data interaction (such as combining with activation values), so the logic chip works in conjunction with the DRAM chip to ensure the real-time performance of V generation by leveraging the low latency of DRAM. In the Attention calculation phase, the logic chip first performs similarity calculation based on Q and K, then performs non-linear normalization operations such as Softmax, and finally performs a weighted fusion with the previously generated V, thus completing the core computational flow of the Attention mechanism.

[0091] Further, referring to Figure 25, it can be seen that the scheduling scheme of this embodiment can deeply explore hardware characteristics and build an efficient parallel pipeline coordination mechanism: In view of the hardware attribute of Flash having a fixed read latency, when the global controller triggers Flash to read Q / K to generate weight data, it will simultaneously start two key operations: First, it schedules the DRAM controller to execute the access of dynamic data (such as activation value, intermediate variable) required for V generation; second, it starts the logic chip 300 to carry out preliminary basic operations (such as activation value normalization, data format conversion). By "parallel startup", hardware resources are avoided from being idle.

[0092] Specifically, when the global controller 340 triggers an Attention Block calculation, the system needs to read the weight parameters of Q / K from Flash.

[0093] Flash access has a fixed latency window (tens to hundreds of nanoseconds). Traditional architectures will only start subsequent operations after the Flash is read, resulting in wasted time waiting.

[0094] This implementation uses a time-parallel scheduling mechanism to start two tasks in parallel while initiating a Flash read operation: DRAM controller startup → Preloads the dynamic data (activation value, intermediate state) required to generate V.

[0095] The logic chip initiates preprocessing → activates existing inputs to perform normalization (Norm) and data format conversion (Quant / Dequant).

[0096] With this design, the Flash, DRAM, and logic chips 300 work in parallel, and the Flash delay window is fully filled.

[0097] Since the sum of the access latency (t_DRAM) of the DRAM chip and the computing latency (t_logic) of the logic chip < Flash latency (t_Flash).

[0098] Therefore, the neural network processing can complete all pre - tasks before the Flash weights are fully read. When the Flash data arrives, it immediately enters the core matrix calculation stage without idle cycles.

[0099] Thus, the Flash latency is completely hidden; the operation and memory access are fully overlapped; and an efficient scheduling of "continuous pipelining" is achieved.

[0100] Correspondingly, this embodiment provides a control method for a neural network processor, as Figure 26 shown, including the steps: Detect whether to trigger the calculation of the attention module; and When it is detected that the calculation of the attention module is triggered, while performing the read operation of the off - chip non - volatile storage chip, schedule the off - chip volatile storage chip to start data loading and the logic chip to start pre - processing.

[0101] Particularly crucial is that this embodiment uses the compute - in - memory (CIM) unit as the core computing execution unit. Its characteristic of "tight coupling of storage and computing" significantly shortens the data transfer path and computing time. Through the coordinated scheduling and timing optimization of the CIM unit, DRAM chip, and logic chip, it can be achieved that the sum of "DRAM parameter transfer latency + logic chip computing latency" is less than "Flash parameter transfer latency". This core breakthrough can completely hide the data transfer time of DRAM and the computing time of the computing chip within the latency window of Flash weight reading, completely eliminating the efficiency bottleneck of "hardware waiting for data" in the traditional architecture.

[0102] This design is essentially different from the traditional storage architecture: In the traditional architecture, due to the separation of storage and computing, data needs to be frequently transferred between different hardware, and operation and memory access cannot be deeply parallelized, making it difficult to completely hide the latency. While in this invention, with the characteristics of the CIM unit and pipelining timing optimization, the access of the DRAM chip, the computing of the logic chip, and the Flash memory access form a gapless parallel pipeline, which not only avoids the loss caused by a single hardware waiting but also maximizes the performance of each hardware module. Finally, it provides an operation guarantee of "low latency and high throughput" for large - language model inference, significantly improving the utilization rate of hardware resources and the overall computing efficiency of the system.

[0103] Optionally, as Figure 27As shown, the off-chip volatile memory chip 100 is composed of one off-chip volatile memory module 10, or is integrated by at least two off-chip volatile memory modules 10 using three-dimensional stacking technology.

[0104] Optionally, such as Figure 28 As shown, the off-chip non-volatile memory chip 200 consists of one off-chip non-volatile memory module 20, or is integrated by at least two off-chip non-volatile memory modules 20 using three-dimensional stacking technology.

[0105] Optionally, such as Figure 29 As shown, the logic chip 300 consists of one logic module 30, or is integrated by at least two logic modules 30 using three-dimensional stacking technology.

[0106] In this embodiment, each chip can be flexibly expanded by three-dimensional stacking to increase computing power and storage.

[0107] It should be noted that when multiple modules are stacked, the bonding method for their 3D integration and substrate interconnection can be based on... Figures 12-15 , Figures 19-22 The bonding methods shown can be combined in an adaptive manner based on the needs of the scenario, and all of the above implementation methods are within the protection scope of this application.

[0108] It should be noted that when the off-chip volatile memory module 10, the off-chip non-volatile memory module 20, and the logic module 30 are configured individually, they can independently implement the functions of the off-chip volatile memory chip 100, the off-chip non-volatile memory chip 200, and the logic chip 300.

[0109] It should be noted that each off-chip volatile memory module 10 is integrated face to face. The side of the structure formed by three-dimensional stacking and integration of at least two off-chip volatile memory modules 10, the side closer to the logic control layer is defined as the front side of the off-chip volatile memory chip 100, and the side closer to the memory array layer is defined as the back side of the off-chip volatile memory chip 100.

[0110] Taking two off-chip volatile memory modules 10 as an example, with them integrated face to face, the off-chip volatile memory chip 100 will show the back of the off-chip volatile memory module 10. The reason for defining the side closer to the logic control layer as the front is that the control circuits of the off-chip volatile memory module 10 are all on the logic control layer (CMOS Die), while the memory array layer (Array Die) cannot work independently. Therefore, it is more convenient to bond the logic control layer as the front side to the front side of the logic chip 300, which facilitates the transmission of signals and data.

[0111] Similarly, it should be noted that each off-chip non-volatile memory module 20 is integrated face to face. The side of the structure formed by three-dimensional stacking and integration of at least two off-chip non-volatile memory modules 20 is defined as the front side of the off-chip non-volatile memory chip 200, and the side is defined as the back side of the off-chip non-volatile memory chip 200.

[0112] Alternatively, the three-dimensional stacking technology may be one or more of TSV, hybrid bonding, or Filp Chip.

[0113] Optionally, the on-chip volatile memory is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

[0114] Optionally, the global controller 340 is configured to perform fine-grained dynamic scheduling of data based on the bonding scheme interconnect characteristics of the three-dimensional stacking technology and the inherent properties of each storage medium.

[0115] Optionally, the off-chip volatile memory chip 100 includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory controller integrated on the logic chip 300, enabling the off-chip volatile memory chip 100 of the neural network processor to be accessed as a standard DRAM memory by external devices or systems. The interface protocol meets one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6. In an architecture where the off-chip volatile memory chip 100 and the Logic Die are vertically stacked through three-dimensional integration and substrate interconnect technologies (such as TSV and Hybrid Bonding), this standard interface can still ensure DRAM compatibility with external devices. At the same time, by leveraging the low latency and high bandwidth interconnect characteristics of three-dimensional integration and substrate interconnect, the data transmission efficiency during standard interface access is improved, avoiding the performance bottleneck of the standard interface under traditional 2D connection.

[0116] Optionally, the off-chip non-volatile memory chip 200 includes a standard interface (compliant with JEDEC or industry-standard memory interface standards). Taking Flash as an example, the Flash of the neural network processor can be accessed as a standard Flash memory. The protocol of this standard interface can be configured to meet one or more of the following: SPI (Serial Peripheral Interface), QSPI (Quad SPI), Octal SPI (OSPI), eMMC (embedded MultiMediaCard, compliant with JEDEC JESD216 standard), UFS (Universal Flash Storage, compliant with JEDEC JESD220 / JESD223 standard), NVMe (Non-Volatile Memory Express), and Parallel Flash.

[0117] The technical solution of the present invention has now been described in conjunction with the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to the specific embodiments described above. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A three-dimensional integrated and substrate interconnected in-memory neural network processor, characterized in that, include: An off-chip volatile memory chip is connected to the logic chip and is used to exchange information with the logic chip; An off-chip non-volatile memory chip, connected to the logic chip, is used for information interaction with the logic chip; and The logic chip is configured to perform any combination of the following functions: Perform computational tasks, cache data, and facilitate information exchange; The off-chip volatile memory chip and the logic chip are integrated using three-dimensional stacking technology to form a three-dimensional integrated module, and the three-dimensional integrated module is integrated with the off-chip non-volatile memory chip through substrate interconnect technology. In the three-dimensional integration module, the off-chip volatile memory chip and the logic chip are bonded together using a co-surface bonding method. The logic chip integrates a digital memory unit and a global controller. The digital memory unit is used to implement data storage and execute computational tasks within the logic chip. The global controller is configured as follows: Based on the interconnect characteristics of the bonding scheme using 3D stacking technology and the inherent properties of each storage medium, data storage mapping rules are configured through upper-layer software. When performing large model calculations, the off-chip non-volatile memory chip is triggered to read weight data, while the off-chip volatile memory chip accesses intermediate activation value data and the logic chip performs basic operations simultaneously. This achieves dynamic adaptation between the data storage scheme and the computation data flow, ensuring that the sum of the access latency of the off-chip volatile memory chip and the computation latency of the logic chip is less than the read operation latency of the off-chip non-volatile memory chip.

2. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The logic chip is a computing chip that performs computing tasks.

3. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The logic chip is a data cache chip used for data caching.

4. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The logic chip is an interface chip used to realize information interaction.

5. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 2, characterized in that, The logic chip integrates at least one off-chip non-volatile memory chip controller and at least one off-chip volatile memory chip controller; wherein, The off-chip non-volatile memory chip controller is used to access off-chip non-volatile memory; The off-chip volatile memory chip controller is used to access the off-chip volatile memory; The global controller is used to manage the off-chip volatile memory chip controller, the off-chip non-volatile memory chip controller, and the digital in-memory computing unit, so as to realize the dynamic adaptation of data storage scheme and computing data flow, and to globally optimize storage capacity, data bandwidth, access latency and computing efficiency.

6. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-5, characterized in that, The off-chip volatile memory chip is a DRAM chip.

7. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 6, characterized in that, In large language model computing scenarios, the off-chip volatile memory chip is used to store key-value cache data, activation values, and intermediate calculation results.

8. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-5, characterized in that, The off-chip non-volatile memory chip is Flash.

9. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 8, characterized in that, In large language model computation scenarios, the off-chip non-volatile memory chip is used to store Q-generated weights, K-generated weights, V-generated weights, Attention weights, and FFN weights.

10. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-5, characterized in that, The off-chip volatile memory chip is composed of an off-chip volatile memory module; or, The off-chip volatile memory chip is integrated by at least two off-chip volatile memory modules using three-dimensional stacking technology. Each off-chip volatile memory module is integrated face to face. The side of the structure formed by the three-dimensional stacking of the at least two off-chip volatile memory modules closer to the logic control layer is defined as the front side of the off-chip volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip volatile memory chip.

11. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-5, characterized in that, The off-chip non-volatile memory chip is composed of an off-chip non-volatile memory module; or, The off-chip non-volatile memory chip is integrated by at least two off-chip non-volatile memory modules using three-dimensional stacking technology; wherein, each of the off-chip non-volatile memory modules is integrated face to face, and the side of the structure formed by the three-dimensional stacking of the at least two off-chip non-volatile memory modules closer to the logic control layer is defined as the front side of the off-chip non-volatile memory chip, and the side closer to the memory array layer is defined as the back side of the off-chip non-volatile memory chip.

12. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 2, characterized in that, The logic chip consists of a logic module; or, The logic chip is integrated from at least two logic modules using three-dimensional stacking technology.

13. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The three-dimensional stacking technology is one or more of through-silicon via (TSV) technology, hybrid bonding technology, or flip-chip technology.

14. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 2, characterized in that, The digital in-memory computing unit is integrated with an on-chip volatile memory and a computing unit. The on-chip volatile memory is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

15. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 5, characterized in that, The off-chip volatile memory chip includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory controller, enabling the neural network processor to be accessed by external devices as a standard DRAM memory. The interface protocol satisfies one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6.

16. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 8, characterized in that, The Flash memory includes an interface compliant with JEDEC or industry-standard memory interfaces, allowing the neural network processor's Flash memory to be accessed as a standard Flash memory. The interface protocol satisfies one or more of SPI, QSPI, Octal SPI, eMMC, UFS, NVMe, and Parallel Flash.

17. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip or the logic chip, and the off-chip non-volatile memory chip are interconnected with the substrate through a silicon interposer.

18. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-5, 12-15, and 17, characterized in that, The off-chip volatile memory chip is stacked on top of the logic chip, with the front side of the off-chip volatile memory chip being three-dimensionally integrated with the front side of the logic chip, and the back side of the logic chip being interconnected with the substrate.

19. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-5, 12-15, and 17, characterized in that, The off-chip volatile memory chip is stacked on top of the logic chip, the back side of the off-chip volatile memory chip is three-dimensionally integrated with the back side of the logic chip, and the front side of the logic chip is interconnected with the substrate.

20. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-5, 12-15, and 17, characterized in that, The off-chip volatile memory chip is stacked below the logic chip, with the front side of the off-chip volatile memory chip being three-dimensionally integrated with the front side of the logic chip, and the back side of the off-chip volatile memory chip being interconnected with the substrate.

21. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-5, 12-15, and 17, characterized in that, The off-chip volatile memory chip is stacked below the logic chip, the back side of the off-chip volatile memory chip is three-dimensionally integrated with the back side of the logic chip, and the front side of the off-chip volatile memory chip is interconnected with the substrate.

22. A control method for a three-dimensional integrated and substrate interconnected in-memory neural network processor, employing the three-dimensional integrated and substrate interconnected in-memory neural network processor as described in any one of claims 1-21, characterized in that, Includes steps; Detect whether the attention module calculation has been triggered; as well as When the attention module is triggered to perform calculations, the external non-volatile memory chip is scheduled to start data loading and the logic chip is scheduled to start preprocessing while the external non-volatile memory chip is performing a read operation. This ensures that the sum of the access latency of the external non-volatile memory chip and the calculation latency of the logic chip is less than the latency of the read operation of the external non-volatile memory chip.

Images