Storage and calculation integrated neural network processor with three-dimensional integration and substrate interconnection

Abstract

The invention relates to a storage and calculation integrated neural network processor based on three-dimensional integration and substrate interconnection and a control method thereof, and the processor comprises an off-chip volatile storage chip which is connected with a logic chip and is used for carrying out information interaction with the logic chip; the off-chip nonvolatile storage chip is connected with the logic chip and is used for carrying out information interaction with the logic chip; the logic chip is configured to realize any combination of the following functions: executing a calculation task, performing data caching and realizing information interaction; the off-chip volatile memory chip and the off-chip nonvolatile memory chip are integrated in the vertical direction by applying a three-dimensional stacking technology to form a three-dimensional integrated module, and the three-dimensional integrated module and the logic chip are integrated through a substrate interconnection technology; therefore, the logic chip and the off-chip volatile memory chip which are high in heat dissipation are respectively cooled, so that heat management is facilitated, and the calculation efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of semiconductor chip technology, and more specifically to a three-dimensional integrated and substrate interconnect heterogeneous storage medium in-memory neural network processor and its control method. Background Technology

[0002] Large Language Models (LLMs), as a core evolutionary direction of Natural Language Processing (NLP) technology in the field of Artificial Intelligence (AI), have shown an exponential growth trend in recent years. Their technological maturity and application breadth have continued to break through, demonstrating irreplaceable advantages in several key areas.

[0003] Meanwhile, the development of LLMs has gradually revealed a core principle widely validated by academia and industry—the scaling law. The core meaning of this principle is that, with sufficient high-quality training data and matching computational resources, the core performance indicators of LLMs (such as perplexity in language generation and accuracy in downstream tasks) show a significant positive correlation with the scale of model parameters. For example... Figure 1 As shown, the model performance score continuously improves with the exponential increase in the number of parameters.

[0004] Traditional computer systems rely on "storage system architecture" (such as...) Figure 2 As shown, the process of registers → on-chip cache → main memory → local disk → remote storage achieves a trade-off between capacity, speed, and cost. Figure 3 As shown, each tier exhibits significant differences in relative cost, storage density, bandwidth, access latency, and power consumption.

[0005] However, existing neural network processor architectures face the following main technical challenges in the pursuit of high-performance integration: First, high-density integration presents a heat dissipation bottleneck. In high-performance computing scenarios, logic chips and volatile memory (such as DRAM) are the main heat sources. Existing 3D integration solutions often tend to vertically stack logic chips and memory chips directly, or tightly bond them through high-density packaging. While this layout shortens the interconnect path, it leads to severe thermal coupling, causing the heat from both to accumulate and be difficult to dissipate effectively. Especially when the logic chip is running at full speed, the accumulated heat significantly affects the stability of the temperature-sensitive memory chip, forcing the processor to reduce its frequency to maintain thermal balance, severely limiting overall performance.

[0006] Second, the limitations of a single storage medium. Traditional in-memory computing architectures typically focus on a single type of storage medium (either volatile or non-volatile storage) to handle all data. However, neural network computing tasks involve a wide variety of complex data types (such as frequently read and written intermediate feature data and long-term stored weight model data). A single storage medium cannot simultaneously meet the multiple requirements of high throughput, low latency, and data non-volatility, resulting in low utilization of storage resources and difficulty in flexibly matching them according to data characteristics, thereby reducing overall computing efficiency.

[0007] Therefore, it is necessary to improve existing neural network processors. Summary of the Invention

[0008] To address the above problems, this application provides a three-dimensional integrated in-memory neural network processor with substrate interconnect, comprising: an off-chip volatile memory chip connected to a logic chip for information interaction with the logic chip; an off-chip non-volatile memory chip connected to the logic chip for information interaction with the logic chip; and the logic chip configured to perform any combination of the following functions: performing computational tasks, performing data caching, and performing information interaction; the off-chip volatile memory chip and the off-chip non-volatile memory chip are integrated using three-dimensional stacking technology to form a three-dimensional integrated module, and the three-dimensional integrated module is integrated with the logic chip through substrate interconnect technology.

[0009] In applications requiring high computing power, the heat dissipation capability of neural network processors directly affects chip performance. For neural network processors, the heat generated by their packaging structure is mainly generated by off-chip volatile memory chips and logic chips. In this embodiment, the logic chips and off-chip volatile memory chips are isolated by substrate interconnect technology, so that their heat dissipation will not affect each other due to stacking. That is, the two can dissipate heat separately.

[0010] In addition, the combination of off-chip volatile and off-chip non-volatile memory chips breaks through the limitations of a single storage medium. It can match the corresponding storage medium based on the characteristics of data / tasks, thereby improving computing efficiency and storage space utilization.

[0011] Optionally, the off-chip volatile memory chip is located at the top in the vertical direction.

[0012] This design stacks the off-chip non-volatile memory chips below the off-chip volatile memory chips, further improving the heat dissipation capability of the off-chip volatile memory chips.

[0013] Optionally, the off-chip non-volatile memory chip is located at the top in the vertical direction.

[0014] Off-chip non-volatile memory (NSM) chips often become system performance bottlenecks due to their high access latency, while the low latency of off-chip volatile memory (OTM) chips makes them suitable for the role of cache. When the OTM chip is located at the top, it is equivalent to the OTM chip being located in the middle layer. This ensures that the data transfer direction from the OTM chip to the logic chip is consistent with that of the OTM chip, thus enabling the construction of a hierarchical storage structure where the OTM chip acts as a cache. By rationally planning the three-dimensional integration density and substrate interconnect density, the data supply efficiency of the OTM chip can be significantly improved, allowing its high-latency access to be coordinated with the fast response of the OTM chip and the efficient computation of the logic chip. Based on this, the system can achieve parallel pipelined operations between OTM chip access and logic computation, effectively hiding the OTM chip access latency and improving overall throughput and energy efficiency.

[0015] Optionally, the logic chip is a computing chip for performing computing tasks, and the logic chip integrates a digital in-memory computing unit.

[0016] Optionally, the logic chip is a data cache chip for data caching.

[0017] Optionally, the logic chip is an interface chip used to realize information interaction.

[0018] Optionally, the logic chip integrates at least one off-chip non-volatile memory chip controller, at least one off-chip volatile memory chip controller, and a global controller; the off-chip non-volatile memory chip controller is used to access the off-chip non-volatile memory chip, and the off-chip volatile memory chip controller is used to access the off-chip volatile memory chip; the global controller is used to manage the off-chip volatile memory chip controller, the off-chip non-volatile memory chip controller, and the digital in-memory computing unit, to realize dynamic adaptation of data storage scheme and computing data flow, so as to globally optimize storage capacity, data bandwidth, access latency, and computing efficiency.

[0019] Optionally, the off-chip volatile memory chip is a DRAM chip.

[0020] Optionally, in a large language model computation scenario, the off-chip volatile memory chip is used to store key-value cache data, activation values, and intermediate computation results.

[0021] Optionally, the off-chip non-volatile memory chip is Flash.

[0022] Optionally, in a large language model computation scenario, the off-chip non-volatile memory chip is used to store Q-generated weights, K-generated weights, V-generated weights, Attention weights, and FFN weights.

[0023] Optionally, the off-chip volatile memory chip is composed of one off-chip volatile memory module, or is integrated by at least two off-chip volatile memory modules using three-dimensional stacking technology.

[0024] Optionally, the off-chip non-volatile memory chip is composed of one off-chip non-volatile memory module, or is integrated by at least two off-chip non-volatile memory modules using three-dimensional stacking technology.

[0025] Optionally, the logic chip consists of a single logic module or is integrated from at least two logic modules using three-dimensional stacking technology.

[0026] Optionally, the three-dimensional stacking technology is one or more of through-silicon via (TSV) technology, hybrid bonding technology, or flip-chip technology.

[0027] Optionally, the digital in-memory computing unit is integrated with an on-chip volatile memory chip and a computing unit, wherein the on-chip volatile memory chip is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

[0028] Optionally, the global controller is configured to perform fine-grained dynamic scheduling of data based on the interconnect characteristics of the bonding scheme of the three-dimensional stacking technology and the inherent properties of each storage medium.

[0029] Optionally, the global controller is configured to support the configuration of data storage mapping rules through upper-layer software.

[0030] Optionally, the off-chip volatile memory chip includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory chip controller integrated on the logic chip, enabling the neural network processor to be accessed by external devices as a standard DRAM memory chip; the interface protocol satisfies one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6.

[0031] Optionally, the Flash includes an interface that conforms to JEDEC or an industry-standard storage interface, and the Flash of the neural network processor can be accessed as a standard Flash memory chip. The protocol of the interface meets one or more of SPI, QSPI, Octal SPI, eMMC, UFS, NVMe, and Parallel Flash.

[0032] Optionally, the off-chip volatile memory chip or the off-chip non-volatile memory chip and the logic chip are interconnected with the substrate through a silicon interposer.

[0033] Optionally, the off-chip volatile memory chip is stacked on top of the off-chip non-volatile memory chip, the back side of the off-chip volatile memory chip is three-dimensionally integrated with the front side of the off-chip non-volatile memory chip, and the back side of the off-chip non-volatile memory chip is interconnected with the substrate.

[0034] Optionally, the off-chip volatile memory chip is stacked on top of the off-chip non-volatile memory chip, the back side of the off-chip volatile memory chip is three-dimensionally integrated with the back side of the off-chip non-volatile memory chip, and the front side of the off-chip non-volatile memory chip is interconnected with the substrate.

[0035] Optionally, the off-chip volatile memory chip is stacked on top of the off-chip non-volatile memory chip, the front side of the off-chip volatile memory chip is three-dimensionally integrated with the front side of the off-chip non-volatile memory chip, and the back side of the off-chip non-volatile memory chip is interconnected with the substrate.

[0036] Optionally, the off-chip volatile memory chip is stacked on top of the off-chip non-volatile memory chip, the front side of the off-chip volatile memory chip and the back side of the off-chip non-volatile memory chip are three-dimensionally integrated, and the front side of the off-chip non-volatile memory chip is interconnected with the substrate.

[0037] Optionally, the off-chip volatile memory chip is stacked below the off-chip non-volatile memory chip, the front side of the off-chip non-volatile memory chip is three-dimensionally integrated with the front side of the off-chip volatile memory chip, and the back side of the off-chip volatile memory chip is interconnected with the substrate.

[0038] Optionally, the off-chip volatile memory chip is stacked below the off-chip non-volatile memory chip, the back side of the off-chip non-volatile memory chip is three-dimensionally integrated with the front side of the off-chip volatile memory chip, and the back side of the off-chip volatile memory chip is interconnected with the substrate.

[0039] Optionally, the off-chip volatile memory chip is stacked below the off-chip non-volatile memory chip, the back side of the off-chip non-volatile memory chip is three-dimensionally integrated with the back side of the off-chip volatile memory chip, and the front side of the off-chip volatile memory chip is interconnected with the substrate.

[0040] Optionally, the off-chip volatile memory chip is stacked below the off-chip non-volatile memory chip, the front side of the off-chip non-volatile memory chip is three-dimensionally integrated with the back side of the off-chip volatile memory chip, and the front side of the off-chip volatile memory chip is interconnected with the substrate.

[0041] To achieve the above-mentioned objectives, this application provides a control method for a three-dimensional integrated and substrate interconnected in-memory neural network processor, which applies the three-dimensional integrated and substrate interconnected in-memory neural network processor described above, and includes the following steps. Detect whether the attention module computation is triggered; and When the attention module is triggered to perform calculations, the off-chip non-volatile memory chip is scheduled to start data loading and the logic chip is scheduled to start preprocessing while the off-chip volatile memory chip is performing a read operation.

[0042] The technical advantage of this control method lies in its ability to successfully "hide" the physical access bottleneck of Flash memory during the computational preprocessing process through parallel scheduling across media, thus overcoming the performance bottleneck caused by media speed mismatch in three-dimensional stacked heterogeneous systems. Attached Figure Description

[0043] Figure 1 A graph showing the relationship between the core performance metrics of a large language model and the scale of model parameters; Figure 2 A diagram of the storage system architecture of a traditional computer system; Figure 3 The performance and cost characteristics of each layer in the storage system architecture of traditional computer systems; Figure 4 This is a schematic diagram of a storage architecture based on a traditional computer system. Figure 5 A schematic diagram of the parameter transfer process for large language model computation based on traditional computer systems; Figure 6 A schematic diagram of the parameter transport process for large language model computation based on data offloading technology; Figure 7 This is a schematic diagram of a storage packaging structure based on Chiplet and Flip Chip technologies; Figure 8 This is a schematic diagram of a storage packaging structure based on three-dimensional integration technology; Figure 9 This is a schematic diagram of a storage packaging structure based on three-dimensional integration technology; Figure 10 A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention; Figure 11 A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention; Figure 12 This is a schematic diagram comparing the implementation of the present invention with the storage architecture of the prior art; Figure 13 This is a schematic diagram of the structure of the digital in-memory computing unit in an embodiment of the present invention; Figure 14 A schematic diagram illustrating the features of 3D integrated bonding for different interface positional relationships; Figure 15 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 16 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 17 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 18 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 19 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 20 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 21 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 22 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 23 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 24 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 25 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 26 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 27This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 28 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 29 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 30 This is a schematic diagram of one type of three-dimensional integration and substrate interconnect bonding structure of a neural network processor provided in an embodiment of the present invention; Figure 31 This is a schematic diagram of one storage scheme for a neural network processor used in computation of a large language model, as provided in an embodiment of the present invention. Figure 32 This is a flowchart illustrating a computation scheduling scheme for a global controller of a neural network processor provided in an embodiment of the present invention. Figure 33 This is a timing diagram of the parallel pipeline coordination mechanism of the global controller of the neural network processor provided in an embodiment of the present invention; Figure 34 This is a schematic diagram illustrating the steps of the control method for a neural network processor provided in an embodiment of the present invention. Figure 35 This is a schematic diagram of the structure of the off-chip volatile memory chip extension of the neural network processor provided in an embodiment of the present invention; Figure 36 This is a schematic diagram of the structure of the off-chip non-volatile memory chip extension for the neural network processor provided in an embodiment of the present invention; Figure 37 This is a schematic diagram of the logic chip extension structure of the neural network processor provided in an embodiment of the present invention; Figure 38 A schematic diagram of the structure of a neural network processor provided for an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] Due to differences in process compatibility between non-volatile memory chips, volatile memory chips, and computing logic (such as the mismatch between advanced processes in computing logic and mature processes in memory chips, and the high integration difficulty caused by different material systems), considering factors such as area utilization, chip yield, and process feasibility, adopting an integrated solution would lead to problems such as excessive chip area redundancy, a significant decrease in yield, and insufficient process adaptability. Therefore, if... Figure 4 As shown, traditional computer systems break it down into multiple chips for independent implementation.

[0046] Based on traditional computer system storage architecture, storage systems can be divided into four categories: off-chip non-volatile memory chips (such as remote storage, SSD, NAND Flash, ReRAM, etc.), off-chip volatile memory chips (such as DRAM), on-chip volatile memory chips (such as SRAM), and registers; for example... Figure 5 As shown, when deploying large language model computation in an existing system, the process is usually as follows: After the AI ​​task is started, the main control chip first loads as much large model parameter data as possible from the non-volatile memory chip to the off-chip volatile memory chip, then moves the data to be computed from the off-chip volatile memory chip to the on-chip volatile memory chip, then selects the data that needs to be computed immediately and sends it to the register, and finally completes the computation in the computing unit.

[0047] However, as analyzed above, the inference process of LLM (Large Language Model) under traditional architecture still faces severe capacity and bandwidth bottlenecks. To effectively alleviate this problem, the industry has successively explored several technical solutions, as follows: First, data offloading technology. For example... Figure 6 As shown, its core logic is "trading communication for capacity": compared to the fixed data loading path of the traditional deployment process in Figure 5, this technology unloads data that is not currently needed in the computing stage from high-speed on-chip storage (such as SRAM) and stores it back to off-chip volatile memory chips (such as DDR) or off-chip non-volatile memory chips (such as SSD); when subsequent computing requires the data, it is retrieved from the corresponding storage medium and reloaded onto the chip. This approach can avoid irrelevant data occupying scarce on-chip storage resources for a long time, thereby directly alleviating the storage capacity bottleneck, and is especially suitable for LLM tasks that process long sequence inputs (such as long document understanding and multi-turn dialogue).

[0048] However, the core limitation of data offloading technology lies in the fact that it does not change the traditional "compute-storage separation" architecture of memory chips in computing systems. While this technology can alleviate storage capacity bottlenecks by "trading communication for capacity" and even relieve bandwidth pressure to some extent, this process also comes at a significant cost: frequent data offloading and retrieval generate a large number of on-chip and off-chip communication needs, which in turn keeps the system continuously subject to severe bandwidth constraints. Even if the system can therefore support neural networks with a larger number of parameters, the frequent transfer of large amounts of data between on-chip and off-chip will seriously drag down computational efficiency, ultimately resulting in significant limitations on the overall inference efficiency of large models.

[0049] Second, high-density packaging technology. This technology, through processes such as chiplets and flip chips, breaks the physical limitations of traditional planar packaging, integrating multiple functional chips (such as logic chips and cache chips) into a single package in a more compact manner. Its core function is to shorten the physical transmission distance across memory levels (such as on-chip cache and off-chip memory, and cache between different computing units), which can reduce data transmission latency and improve overall bandwidth through parallel transmission paths, thereby alleviating the problem of low "data moving" efficiency.

[0050] High-density packaging technology is effective in alleviating the bandwidth bottleneck of traditional memory chip architectures, but it has two key limitations. First, it does not break through the underlying design of traditional memory chip architectures, but only improves data transmission efficiency by optimizing the packaging form, so the core problem of limited storage capacity is not solved. Second, its performance improvement has a clear ceiling, and it will continue to be limited by the speed limit of the IO interface, making it difficult to achieve unlimited bandwidth breakthroughs.

[0051] Third, three-dimensional integration and substrate interconnection technologies. For example... Figure 8 , Figure 9 As shown, this technology overcomes the spatial limitations of two-dimensional packaging through key processes such as TSV (Through Silicon Via) and Hybrid Bonding, enabling vertical stacking integration of logic chips and memory chips. For example, the widely used HBM (High Bandwidth Memory) technology uses TSV to vertically stack multiple DRAM chips next to the logic chip, forming a tightly coupled "compute-storage" structure. Compared to traditional DDR memory chips, HBM's bandwidth can be increased several times (e.g., HBM3 has a bandwidth of 819GB / s), while significantly reducing the area occupied by the planar circuit board, fundamentally optimizing the "storage-compute" interaction efficiency required for LLM inference.

[0052] 3D integration and substrate interconnect technologies, with their 3D stacked structures and high-density interconnect designs, can effectively address the capacity shortages and bandwidth limitations faced by traditional memory architectures. However, existing implementations of this technology still have significant limitations. Current mainstream solutions (such as HBM high-bandwidth memory chips and NAND Flash memory) are essentially still limited to the stacking and interconnection of memory chips of the same type and level—they have neither broken through the underlying framework of traditional memory chip architectures nor achieved a synergistic breakthrough in capacity and bandwidth through a unified memory structure.

[0053] Meanwhile, current optimization solutions for storage architecture mostly focus on simply expanding capacity and increasing bandwidth, neglecting the core characteristic of computer systems: the dynamic transfer of bottlenecks. When capacity and bandwidth bottlenecks are alleviated through technical means, the processing power of computing units often becomes a new performance constraint. For example, in large model inference scenarios, even if high-speed data storage and transmission are achieved through technologies such as HBM and Chiplet, if the computing power of logic chips such as GPUs / TPUs cannot match the data throughput, resource idleness will still occur, resulting in "data waiting for computation," making it difficult to improve the overall system efficiency. More importantly, existing solutions generally adopt a "fragmented optimization" approach, failing to build a unified architecture that coordinates computing power, storage efficiency, and bandwidth performance. This fails to break the cycle of "solving old bottlenecks and creating new bottlenecks" at the system level, making it difficult to achieve synergistic breakthroughs in the performance of various components.

[0054] Therefore, existing solutions have failed to break free from the constraints of the underlying framework of traditional memory chip architecture. The core problem lies in the inability to achieve coordinated optimization of capacity, bandwidth, and computing power based on a unified storage structure. This structural shortcoming directly results in significant performance limitations and severe implementation challenges for large language models when deployed for inference in systems built on existing solutions.

[0055] If the applicant is aware of the aforementioned technical issues, Figure 10 and Figure 11 As shown, this embodiment provides a neural network processor based on in-memory computing and three-dimensional stacking technology, including: an off-chip volatile memory chip 100 connected to a logic chip for information interaction with the logic chip; an off-chip non-volatile memory chip 200 connected to the logic chip for information interaction with the logic chip; and a logic chip configured to perform any combination of the following functions: performing computational tasks, performing data caching, and performing information interaction; the off-chip volatile memory chip 100 and the off-chip non-volatile memory chip 200 are integrated vertically in the three-dimensional stacking technology to form a three-dimensional integrated module, and the three-dimensional integrated module is integrated with the logic chip through substrate interconnect technology.

[0056] Optionally, the off-chip volatile memory chip 100 is a DRAM chip. For ease of explanation, a DRAM chip will be used as an example of the off-chip volatile memory chip 100 in the following text.

[0057] Optionally, the off-chip non-volatile memory chip 200 is Flash. For ease of explanation, Flash is used as an example of the off-chip non-volatile memory chip 200 in the following text.

[0058] In applications requiring high computing power, the heat dissipation capability of a neural network processor directly affects the chip's performance. For a neural network processor, the heat generated by its package structure is mainly generated by the off-chip volatile memory chip 100 and the logic chip 300. In this embodiment, the logic chip 300 and the off-chip volatile memory chip 100 are isolated by substrate interconnect technology, so that their heat dissipation will not affect each other due to stacking. That is, the two can dissipate heat separately. Furthermore, in this embodiment, the off-chip non-volatile memory chip 200 is stacked below the off-chip volatile memory chip 100, which further improves the heat dissipation capability of the off-chip volatile memory chip 100.

[0059] In addition, the combination of off-chip volatile memory chip 100 and off-chip non-volatile memory chip 200 breaks through the limitations of a single storage medium. It can improve computing efficiency and storage space utilization by matching the corresponding storage medium based on the characteristics of data / tasks.

[0060] Optionally, such as Figure 38 As shown, the off-chip non-volatile memory chip 200 is located at the top in the vertical direction.

[0061] The off-chip non-volatile memory chip 200, due to its high access latency, often becomes a system performance bottleneck, while the low latency of the off-chip volatile memory chip 100 makes it suitable for a cache role. When the off-chip non-volatile memory chip 200 is located at the top, it is equivalent to the off-chip volatile memory chip 100 being positioned between the logic chip 300 and the off-chip non-volatile memory chip 200. The data transfer direction between the logic chip 300 and the off-chip non-volatile memory chip 200 is consistent, thus enabling the construction of a hierarchical storage structure where the off-chip volatile memory chip 100 serves as a cache for the off-chip non-volatile memory chip 200. By rationally planning the three-dimensional integration density and substrate interconnect density, the data supply efficiency of the off-chip non-volatile memory chip 200 can be significantly improved, ensuring that its high-latency access is coordinated with the fast response of the off-chip volatile memory chip 100 and the efficient computation of the logic chip 300. Based on this, the system can realize parallel pipelined operations between accessing the off-chip non-volatile memory chip 200 and accessing the off-chip volatile memory chip 100 and logic calculation 300, effectively hiding the access latency of the off-chip non-volatile memory chip 200 and improving the overall computing throughput and energy efficiency.

[0062] Optionally, the logic chip 300 is a computing chip for performing computing tasks. The logic chip integrates a digital in-memory computing unit 310 for performing computing tasks. In the following text and figures, for ease of explanation, the logic chip 300 is configured as a computing chip as an example. Based on the spirit of the present invention, it can be understood that the logic chip can also be configured as a data cache chip and interface chip as described below to achieve functions adapted to the application scenario.

[0063] Optionally, the logic chip 300 is a data cache chip used for data caching, which temporarily stores data between computing and storage, improving access efficiency and reducing latency and power consumption caused by frequent access to external storage.

[0064] Optionally, the logic chip 300 is an interface chip used to realize information interaction, and is used to realize information interaction and protocol conversion between different chips or systems.

[0065] Optionally, in a neural network processor, the logic chip 300 can be configured as multiple, such as three, which are respectively configured as a computing chip, an interface chip, and a data cache chip. Multiple chips can be integrated with each other using three-dimensional stacking technology to achieve flexible and diverse functions.

[0066] It should be noted that the off-chip volatile memory chip 100 and the off-chip non-volatile memory chip 200 can also be implemented using other storage media / memory chips that meet the requirements, and this embodiment does not limit them.

[0067] The neural network processor provided in this embodiment can achieve at least the following technical effects: like Figure 12As shown, this implementation constructs a disruptive and innovative storage architecture, completely breaking through the inherent framework of traditional storage chips' "layer-by-layer cascading and unidirectional data transport"—traditional storage systems rely on a fixed hierarchy of "on-chip cache → off-chip storage → external storage," requiring data to be transmitted level by level to meet computing demands. This not only generates a large amount of redundant data transport overhead but also leads to bandwidth limitations and latency accumulation due to the large physical distance between levels, becoming a core efficiency bottleneck for computationally intensive tasks such as large language models. The architecture provided by this implementation achieves end-to-end optimization through the synergistic innovation of three-dimensional stacking and in-memory computing technology: on the one hand, three-dimensional integration and substrate interconnect technology directly stack off-chip DRAM chips and logic chips in three dimensions, shortening the physical distance between them to the micrometer level. This improves off-chip storage access bandwidth through high-density bonding and reduces cross-chip transmission losses to lower latency; on the other hand, in-memory computing technology integrates on-chip storage chips (such as DCIM based on 6T SRAM) into a single layer. The deep integration of the array with the computing unit enables operations such as multiplication and accumulation to be performed directly within the on-chip storage chip, fundamentally eliminating the latency and energy consumption of on-chip data transfer. Ultimately, this creates a "low-latency, high-bandwidth, low-power" storage-computing collaborative system, completely breaking free from the performance constraints of traditional hierarchical architectures.

[0068] In terms of capacity, traditional architectures rely on planar expansion (increasing chip area or adding independent storage modules), which is easily limited by packaging space and power consumption budgets. Furthermore, capacity increases are prone to performance waste due to bandwidth mismatch. This implementation achieves a leapfrog, bottleneck-free capacity expansion through "three-dimensional vertical stacking + high-density media selection": using NAND Flash as the core high-density storage medium, bonding technology can increase the number of three-dimensionally stacked Flash chips—for example, an 8-layer stacked Flash module / unit. Within the same planar area as traditional 2D packaging, a single module can achieve a capacity of 60-128GB (traditional 2D Flash only 10-20GB in the same area). For further capacity expansion, the number of Flash stacking layers can be increased in the three-dimensional architecture (current processes support 200+ layers of 3D NAND), or multiple sets of Flash-three-dimensional stacked units can be connected in parallel, and the expansion process does not occupy additional planar space.

[0069] In terms of bandwidth, the bandwidth of traditional 2D architectures depends on interface protocol upgrades (such as DDR5→DDR6), with limited single-channel bandwidth improvement (approximately 30%-50% per generation), and multi-channel expansion is easily limited by motherboard wiring density. This implementation achieves efficient and scalable bandwidth expansion through "3D stacking + substrate interconnection + multi-chip parallelism," such as increasing the number of 3D stacks of high-bandwidth media (DRAM / SRAM), for example, vertically integrating two groups of DRAM chips with logic chips via F2F bonding. Each group of DRAM chips provides 200-300GB / s bandwidth, and the two groups in parallel can achieve a total bandwidth of 400-600GB / s. At the same time, the micron-level wiring capability of the silicon interposer can be used to construct a parallel interconnect channel of "multi-memory chip - logic chip," avoiding the wiring congestion problem of traditional 2D architectures. In addition, this implementation can prioritize F2F hybrid bonding for high-frequency interaction links (such as Logic Die and DRAM), reducing interconnect latency to sub-nanosecond levels, further releasing bandwidth potential.

[0070] With the storage and bandwidth issues resolved through the aforementioned solutions, computing power has become a new bottleneck in the systemic problem of storage-bandwidth-computing power. The expansion of computing power in traditional architectures relies on increasing the area of ​​logic chips or upgrading process nodes, which is costly and easily limited by heat dissipation (increased area leads to increased heat density). At the same time, after the computing power is increased, it is easy to cause "computing power idle" due to insufficient storage bandwidth. This implementation method, while retaining the process node improvement, uses Compute-In-Memory (CIM) technology as its core to achieve high-density, low-power expansion of computing power. On the one hand, the number of CIM macrocells can be increased modularly, with a DCIM array based on 6T SRAM as the basic unit. Each unit can provide 100-150 TOPS of computing power. By increasing the number of CIM unit layers through three-dimensional stacking (such as 4-layer CIM stacking), a total computing power of 400-600 TOPS can be achieved, and the computing power density per unit area is 2-3 times that of traditional GPUs. Upgrading the logic chip process node: upgrading the Logic Die from 14nm to 3nm, and coordinating with the process optimization of CIM units, the computing power per unit area can be further increased by 50%-80%. Specifically, in this implementation method, the computing unit is integrated into the on-chip volatile memory chip on the logic chip 300, so that the calculation data does not need to be transferred to the register through the on-chip volatile memory chip, but can be calculated directly inside the on-chip volatile memory chip. The on-chip volatile memory chip with integrated computing units is designed as a Digital Computing-In-Memory (DCIM) unit. For example... Figure 13As shown, the structure of the digital in-memory computing unit 310 mainly includes a DCIM array based on 6T SRAM, a word line decoder (WL DEC), a word line buffer (WL DRV), read / write circuitry, and a controller. The DCIM array consists of C DCIM columns, each containing R subarrays and an adder tree. Each subarray integrates 6T SRAM bit cells and a Local Read-Out and Compute Unit (LRCC), storing N-bit weights per row. Functionally, the DCIM macro can perform matrix-vector multiplication (MVM) multiplication-accumulation (MAC) calculations: the operating mode is switched via the MEM_EN signal. When MEM_EN is low, the N-bit weights in the subarray are multiplied by a 1-bit input, read and latched by the LRCC, accumulated by the adder tree, and then sent to a shift-and-accumulator to extend the input precision in the time dimension, ultimately completing a full MVM operation with K-bit input and N-bit weights, while also supporting the storage and efficient computation of multi-weight matrices.

[0071] In summary, the neural network processor provided in this embodiment achieves direct contact between the high-heat chip and the external heat dissipation structure by placing the off-chip volatile memory chip (DRAM) on the top layer of the stack, thus shortening the heat conduction path; effectively reducing the temperature rise of the logic chip and the non-volatile memory chip, avoiding heat accumulation, and improving the thermal stability and long-term reliability of the system.

[0072] The storage architecture is reconstructed (end-to-end bandwidth optimization), breaking the traditional "cache-external storage-external storage" layered model. Through three-dimensional bonding, DRAM chips and logic chips are directly vertically interconnected, shortening the interconnection distance to the micrometer level. At the same time, combined with in-memory computing (CIM) technology, data can be multiplied and accumulated within the storage array, reducing data transfer latency and energy consumption, and building a low-latency, high-bandwidth data path.

[0073] Storage capacity expansion and density improvement are achieved by using "three-dimensional vertical stacking + substrate interconnect high-density NAND Flash" to realize three-dimensional expansion of storage capacity, increasing capacity by 3 to 6 times without occupying additional planar area; it supports hundreds of layers of Flash stacking or multiple units in parallel, breaking through the limitations of packaging space.

[0074] Bandwidth enhancement and interconnect optimization are achieved through F2F bonding and parallel interconnection of multiple DRAM chips to achieve linear bandwidth superposition (200–300 GB / s per channel, total bandwidth up to 400–600 GB / s); with the help of the micron-level wiring capability of silicon interposer, I / O density is significantly improved and interconnect latency is reduced to sub-nanosecond level.

[0075] The improved computing power density and energy efficiency are achieved by embedding DCIM (Digital In-Memory Computing) units based on 6T SRAM into the logic chip, realizing "storage as computing". By increasing the number of CIM macrocell layers through three-dimensional stacking and upgrading advanced process nodes (14nm→3nm), the computing power density per unit area is increased to 2 to 3 times that of traditional GPUs, and the energy efficiency ratio is improved by about 50% to 80%.

[0076] The neural network processor provided in this embodiment forms a collaborative closed loop in terms of storage, bandwidth, and computing power under a three-dimensional stacked and substrate interconnect architecture: high-density storage provides sufficient data; high-bandwidth paths ensure real-time supply; and the in-memory computing unit efficiently performs calculations; thus achieving a performance balance of "low latency, high computing power, and low power consumption" at the system level.

[0077] Through a systematic design of "three-dimensional stacking + substrate interconnection + in-memory computing integration", a comprehensive breakthrough has been achieved in the three dimensions of data path, storage capacity and computing power density of neural network processors, forming a new computing architecture with high integration, high bandwidth, scalability and low power consumption, which is suitable for computing power intensive scenarios such as large language models, visual computing, and edge inference.

[0078] In the field of 3D integration technology, each chip (die / core) includes: Face: This refers to the side where the metal interconnect layer is located, including the top layer wiring, solder joints (micro-bumps), or redistribution layer (RDL).

[0079] Back side: This is the silicon substrate side, usually the bare silicon surface, which is electrically connected to the front side through thinning, opening, or TSV (through silicon via).

[0080] like Figure 14 The diagram illustrates the characteristics of 3D integration and substrate interconnect bonding under different interface positional relationships. The interface positional relationships for 3D integration and substrate interconnection mainly fall into three types: face-to-face (F2F), face-to-back (F2B), and back-to-back (B2B). In front-to-front mode, interconnect density and bandwidth are high, RDL rerouting is optional, cost (yield) is low, signal integrity is high, but thermal management is difficult; in front-to-back mode, interconnect density and bandwidth are low, RDL rerouting is necessary, cost (yield) and signal integrity are both moderate, and thermal management is moderate; in back-to-back mode, interconnect density and bandwidth are low, RDL rerouting is necessary, cost (yield) is high, signal integrity is low, but thermal management is relatively easy.

[0081] Despite the superior overall performance of F2F, in a three-layer stacking scenario, if two layers are to be bonded using F2F, the third layer will inevitably form an F2B structure with one of the layers. Therefore, three-layer stacking requires comprehensive consideration of hardware factors such as interconnect density, cost, signal integrity, and thermal management to determine the specific arrangement of F2F, F2B, and B2B, resulting in higher design complexity compared to two-layer stacking.

[0082] Specifically, the three-dimensional integration and substrate interconnection structure proposed in this invention can flexibly adjust the bonding methods between different dies according to actual needs, supporting flexible combinations of F2F, F2B, or B2B bonding. It has a total of 8 bonding surface structures, as follows: Figures 15-22 As shown, specifically, refer to Figure 15 An off-chip volatile memory chip 100 is stacked on top of an off-chip non-volatile memory chip 200. The back side of the off-chip volatile memory chip 100 is three-dimensionally integrated with the front side of the off-chip non-volatile memory chip 200, and the back side of the off-chip non-volatile memory chip 200 is interconnected with the substrate 400.

[0083] Optionally, such as Figure 16 As shown, an off-chip volatile memory chip 100 is stacked on top of an off-chip non-volatile memory chip 200. The back side of the off-chip volatile memory chip 100 and the back side of the off-chip non-volatile memory chip 200 are three-dimensionally integrated, and the front side of the off-chip non-volatile memory chip 200 is interconnected with the substrate 400.

[0084] Optionally, such as Figure 17 As shown, an off-chip volatile memory chip 100 is stacked on top of an off-chip non-volatile memory chip 200. The front side of the off-chip volatile memory chip 100 and the front side of the off-chip non-volatile memory chip 200 are three-dimensionally integrated, and the back side of the off-chip non-volatile memory chip 200 is interconnected with the substrate 400.

[0085] Optionally, such as Figure 18 As shown, an off-chip volatile memory chip 100 is stacked on top of an off-chip non-volatile memory chip 200. The front side of the off-chip volatile memory chip 100 and the back side of the off-chip non-volatile memory chip 200 are three-dimensionally integrated, and the front side of the off-chip non-volatile memory chip 200 is interconnected with the substrate 400.

[0086] Optionally, such as Figure 19 As shown, the off-chip volatile memory chip 100 is stacked below the off-chip non-volatile memory chip 200. The front side of the off-chip non-volatile memory chip 200 is three-dimensionally integrated with the front side of the off-chip non-volatile memory chip 100, and the back side of the off-chip volatile memory chip 100 is interconnected with the substrate 400.

[0087] Optionally, such as Figure 20As shown, an off-chip volatile memory chip 100 is stacked below an off-chip non-volatile memory chip 200. The back side of the off-chip non-volatile memory chip 200 is three-dimensionally integrated with the front side of the off-chip volatile memory chip 100, and the back side of the off-chip volatile memory chip 100 is interconnected with the substrate 400.

[0088] Optionally, such as Figure 21 As shown, the off-chip volatile memory chip 100 is stacked below the off-chip non-volatile memory chip 200. The back side of the off-chip non-volatile memory chip 200 is three-dimensionally integrated with the back side of the off-chip volatile memory chip 100, and the front side of the off-chip volatile memory chip 100 is interconnected with the substrate 400.

[0089] Optionally, such as Figure 22 As shown, the off-chip volatile memory chip 100 is stacked below the off-chip non-volatile memory chip 200. The front side of the off-chip non-volatile memory chip 200 and the back side of the off-chip volatile memory chip 100 are three-dimensionally integrated, and the front side of the off-chip volatile memory chip 100 is interconnected with the substrate 400.

[0090] Optionally, such as Figures 23-30 As shown, in application Figures 15-22 Based on the eight bonding methods described above, in some embodiments, the neural network processor provided in this application interconnects each chip with the substrate 400 through a silicon interposer 500. As the core carrier of high-density interconnection, the silicon interposer, with its unique material and process characteristics, exhibits interconnection advantages far exceeding those of traditional organic substrates: First, relying on the high-precision photolithography process of silicon materials, it can achieve micron-level or even submicron-level wiring density, supporting parallel interconnection of tens of thousands to hundreds of thousands of I / O interfaces, perfectly adapting to the high-density signal interaction requirements of multiple chips (such as Logic Die, DRAM, Flash) in three-dimensional stacking; Second, the low dielectric loss characteristics of silicon significantly reduce parasitic capacitance and inductance in signal transmission, significantly improving the integrity of high-frequency signals and reducing data transmission error rate; Third, its thermal conductivity is higher than that of organic substrates, which can help disperse the heat of stacked chips, forming a synergy with the heat dissipation design of three-dimensional integration; In addition, the silicon interposer can be compatible with heterogeneous chips of different process nodes and sizes, simplifying the integration difficulty of multiple types of chips through a unified interconnection interface, providing underlying support for the flexibility of three-dimensional stacking.

[0091] It is important to note that the three-dimensional bonding between the off-chip volatile memory chip 100 and the off-chip non-volatile memory chip 200 in the neural network processor provided in this embodiment is not a simple stacking, but a precise design based on the complementary characteristics of the two. Taking DRAM and Flash as examples, Flash, with its high storage density and non-volatility, can handle terabytes of massive data (such as weight parameters of large language models), solving the capacity bottleneck of traditional storage devices. DRAM, with its low latency and high random access bandwidth, is suitable for the high-frequency data interaction needs during computation (such as KV cache and temporary storage of intermediate calculation results). Through three-dimensional bonding (supporting flexible bonding methods such as F2F / F2B), the physical distance between the two is shortened to the micrometer level, forming a "large capacity - high speed" storage hierarchy—which avoids the latency shortcomings of Flash when used alone and makes up for the capacity limitations of DRAM when used alone, achieving dual optimization of storage capacity and access bandwidth.

[0092] Meanwhile, the substrate interconnect structure of the storage device further enhances its synergy with the logic chip: it includes high-precision interconnect solutions such as silicon interposers. With the advantages of micron-level wiring density, low signal loss, and high heat dissipation efficiency, the silicon interposer can build parallel interconnect channels for tens of thousands of I / O interfaces, which significantly reduces the signal transmission delay and significantly improves the bandwidth between the storage device (Flash+DRAM) and the logic chip. At the same time, it helps to disperse the heat of the stacked modules and ensure the stable operation of the overall system.

[0093] In summary, this new storage device, through its design of "Flash-DRAM 3D integration + high-precision substrate interconnection," not only breaks through the performance limitations of a single storage medium but also lays the underlying foundation for efficient collaboration with logic chips, making it fully adaptable to the storage needs of computationally intensive tasks such as large language model inference.

[0094] Specifically, in response to the performance requirements of large language model inference tasks (such as long sequence generation and high-concurrency query), the global controller in this embodiment can combine the interconnection characteristics of the bonding scheme (such as bandwidth and thermal efficiency) with the inherent properties of the storage medium (such as capacity and latency) to achieve fine-grained dynamic management of the data storage scheme. Specifically, targeted scheduling will be implemented based on the different hardware resource requirements of each task: high-bandwidth task data (such as key-value pair cache in the attention mechanism) will be stored in a storage chip bonded to the logic chip 300 using an F2F connection. This type of storage chip can match the high-frequency data interaction requirements due to its high interconnect bandwidth characteristics; heat-sensitive task data (such as intermediate results of large-scale matrix operations) will be stored in a storage chip bonded to the logic chip 300 using an F2B or B2B connection. This type of bonding scheme has better thermal management capabilities and can avoid local overheating from affecting data stability; at the same time, for the differentiated requirements of storage capacity and latency, non-real-time data requiring large-capacity storage (such as pre-trained weight backups) will be stored in an off-chip non-volatile storage chip, while latency-sensitive real-time computing data (such as the embedding vector of the input sequence) will be stored in an off-chip volatile storage medium, forming a storage architecture that precisely matches "features and requirements".

[0095] In this embodiment, computing power and storage / bandwidth work in tandem. When expanding computing power, the bonding methods between different dies are flexibly adjusted according to actual needs to match the bonding method with the application scenario. For example, F2F bonding increases the bandwidth of off-chip volatile memory chips, and B2B bonding increases the capacity of off-chip non-volatile memory chips, ensuring a match between "computing power enhancement and data supply" and avoiding bottlenecks. By using F2F bonding between high-performance chips and F2B / B2B bonding between low-speed or peripheral chips, optimal allocation of signal paths and bandwidth resources is achieved. Through flexible design of different bonding surface combinations (such as F2B, B2B), high-power chips can directly expose their back surfaces to enhance heat dissipation, thereby reducing overall thermal resistance. Design flexibility and compatibility are enhanced, and the bonding sequence and interface direction can be flexibly adjusted according to the system architecture (such as Logic-DRAM), compatible with different process nodes and process platforms, and supporting heterogeneous chip integration. Balancing yield and cost, the hybrid bonding structure effectively reduces manufacturing complexity and yield loss, achieving an optimal trade-off between performance and cost. Signal integrity optimization is achieved by concentrating high-frequency interconnects at the F2F interface and transmitting low-speed interconnects via F2B or TSV, shortening signal paths, reducing crosstalk and delay, and improving overall system signal quality. The neural network processor provided in this embodiment breaks through the interconnect limitations of traditional two-layer stacking, offering a configurable 3D integration and substrate interconnect framework for multi-layer heterogeneous chips. It achieves a system-level balance with adjustable overall performance, significantly improving the performance density and manufacturability of 3D integration and substrate interconnect systems.

[0096] Optionally, continue to refer to Figure 11 The logic chip 300 integrates at least one off-chip non-volatile memory chip controller 330, at least one off-chip volatile memory chip controller 320, and a global controller 340. The off-chip non-volatile memory chip controller 330 is used to access the off-chip non-volatile memory chip 200, and the off-chip volatile memory chip controller 320 is used to access the off-chip volatile memory chip 100. The global controller 340 is used to manage the digital in-memory computing unit 310, the off-chip non-volatile memory chip controller 330, and the off-chip volatile memory chip controller 320 of the logic chip 300, so as to realize the dynamic adaptation of data storage scheme and computing data flow, so as to globally optimize storage capacity, data bandwidth, access latency, and computing efficiency.

[0097] Furthermore, the global controller 340 can be configured to be software programmable, supporting storage strategies defined through upper-layer software. Regardless of the aforementioned bonding structure used in this implementation, different data storage mapping rules (such as task-media binding relationships and data migration priorities) can be configured through software, and performance tests can be conducted in conjunction with actual inference scenarios (such as bandwidth utilization, latency fluctuations, and thermal loss monitoring). Ultimately, the optimal storage solution suitable for the current task type is adaptively selected, ensuring that large language model inference can achieve maximum storage-computation collaborative efficiency under different hardware configurations.

[0098] Specifically, the global controller 340, the off-chip non-volatile memory chip controller 330, and the off-chip volatile memory chip controller 320 can be implemented by a CPU or other existing control modules or combinations of control modules capable of implementing the functions mentioned in this embodiment. This embodiment does not limit this.

[0099] Figure 31This paper presents a configurable storage scheme for large language model computation. Its core is a precise hierarchical matching based on data characteristics and storage medium capabilities, using an off-chip volatile memory chip 100 as a DRAM chip and an off-chip non-volatile memory chip 200 as Flash as an example. For the highly dynamic KV Cache (requiring high-frequency updates and random access) and the activation values ​​and intermediate calculation results with high real-time requirements in large language model inference, this implementation can utilize the low latency and high random access throughput of DRAM to store them in DRAM, ensuring efficient read / write and update efficiency of dynamic data. However, the Q-generation weights, K-generation weights, V-generation weights, Attention weights, and FFN weights, which have a large number of parameters, have high storage capacity requirements and relatively low access frequency. Therefore, this invention can store them in Flash, leveraging the large capacity and non-volatile characteristics of Flash to meet storage needs. This solution, by differentiating the attributes of "dynamic calculation data - static weight data" and adapting the media, ensures both the response speed of the dynamic calculation process and meets the storage capacity requirements of large-scale weights, thus achieving efficient utilization of storage resources and cost balance in large language model inference scenarios.

[0100] During the large language model calculation process, the global controller 340 provided in this embodiment is configured to dynamically allocate the off-chip volatile memory chip controller 320, the off-chip non-volatile memory chip controller 330, and the digital in-memory computing unit 310 with DCIM as the core based on the current three-dimensional integration and substrate interconnection bonding structure (such as F2F / F2B / B2B) and storage medium distribution, so as to achieve precise adaptation of hardware resources and computing tasks.

[0101] Optionally, the global controller 340 is configured to support the configuration of data storage mapping rules via upper-layer software.

[0102] Figure 32 shows a typical example of this scheduling logic: The neural network storage chip provided in this embodiment places static weights (or cold weights) in the Flash memory (red box) with optimal capacity but high latency, places dynamic activations / intermediate states in the low-latency DRAM chip (green box), and places the actual execution operators (matrix multiplication, Softmax, weighted sum, projection, FFN) in the low-latency, high-computing-power logic chip (black box). The global controller 340 schedules the three to work together, minimizing cross-chip transmission and latency through caching / reuse and parallelization.

[0103] Since the generation of Q and K requires access to weight data stored in Flash (such as Q / K generation weights), the global controller 340 schedules the logic chip and Flash to work together to calculate Q and K. The generation of V involves more dynamic data interaction (such as combining with activation values), so it is executed by the logic chip in conjunction with the DRAM chip, leveraging the low latency of DRAM to ensure the real-time nature of V generation. In the Attention calculation phase, the logic chip first calculates similarity based on Q and K, then performs non-linear normalization operations such as Softmax, and finally performs a weighted fusion with the previously generated V, completing the core computational flow of the Attention mechanism.

[0104] Further, referring to Figure 33, it can be seen that the scheduling scheme of this embodiment can deeply explore hardware characteristics and build an efficient parallel pipeline coordination mechanism: In view of the hardware attribute of Flash having a fixed read latency, when the global controller triggers the Flash to read Q / K weight data, it will simultaneously start two key operations: First, it schedules the DRAM controller to perform the access to the dynamic data (such as activation value, intermediate variable) required for V generation; second, it starts the logic chip 300 to carry out preliminary basic operations (such as activation value normalization, data format conversion). By "parallel startup", hardware resources are avoided from being idle.

[0105] Specifically, when the global controller 340 triggers an Attention Block calculation, the system needs to read the weight parameters of Q / K from Flash.

[0106] Flash access has a fixed latency window (tens to hundreds of nanoseconds). Traditional architectures will only start subsequent operations after the Flash is read, resulting in wasted time waiting.

[0107] This implementation uses a time-parallel scheduling mechanism to start two tasks in parallel while initiating a Flash read operation: DRAM controller startup → Preloads the dynamic data (activation value, intermediate state) required to generate V.

[0108] The logic chip initiates preprocessing → activates existing inputs to perform normalization (Norm) and data format conversion (Quant / Dequant).

[0109] With this design, the Flash, DRAM, and logic chips work in parallel, and the Flash delay window is fully filled.

[0110] Since the sum of the access latency (t_DRAM) of the DRAM chip and the computing latency (t_logic) of the logic chip < Flash latency (t_Flash).

[0111] Therefore, before the Flash weights are completely read, all pre - tasks can be completed in the neural network processing. When the Flash data arrives, it immediately enters the core matrix calculation stage, without idle cycles.

[0112] Thus, the Flash latency is completely hidden; the operation and memory access are fully overlapped; and an efficient scheduling of "continuous pipelining" is achieved.

[0113] Correspondingly, this embodiment provides a control method for a neural network processor, as Figure 34 shown, including the steps: Detect whether to trigger the calculation of the attention module; and When it is detected that the calculation of the attention module is triggered, while performing the read operation of the off - chip non - volatile storage chip, schedule the off - chip volatile storage chip to start data loading and the logic chip to start pre - processing.

[0114] Particularly crucial is that this embodiment uses the compute - in - memory (CIM) unit as the core computing execution unit. Its characteristic of "tight coupling of storage and computing" significantly shortens the data transfer path and computing time. Through the coordinated scheduling and timing optimization of the CIM unit, DRAM chip, and logic chip, it can be achieved that the sum of "DRAM parameter transfer latency + logic chip computing latency" is less than "Flash parameter transfer latency". This core breakthrough can completely hide the data transfer time of DRAM and the computing time of the logic chip within the latency window of Flash weight reading, completely eliminating the efficiency bottleneck of "hardware waiting for data" in the traditional architecture.

[0115] This design is essentially different from the traditional storage architecture: in the traditional architecture, due to the separation of storage and computing, data needs to be frequently transferred between different hardware, and operation and memory access cannot be deeply parallelized, making it difficult to completely hide the latency. However, in this invention, with the characteristics of the CIM unit and the optimized pipeline timing, the access of the DRAM chip, the computing of the logic chip, and the Flash memory access form a gapless parallel pipeline, which not only avoids the loss caused by a single hardware waiting, but also maximizes the performance of each hardware module. Finally, it provides an operation guarantee of "low latency, high throughput" for large - language model inference, significantly improving the utilization rate of hardware resources and the overall computing efficiency of the system.

[0116] Optionally, as Figure 35As shown, the off-chip volatile memory chip 100 is composed of one off-chip volatile memory module 10, or is integrated by at least two off-chip volatile memory modules 10 using three-dimensional stacking technology.

[0117] Optionally, such as Figure 36 As shown, the off-chip non-volatile memory chip 200 consists of one off-chip non-volatile memory module 20, or is integrated by at least two off-chip non-volatile memory modules 20 using three-dimensional stacking technology.

[0118] Optionally, such as Figure 37 As shown, the logic chip 300 consists of one logic module 30, or is integrated by at least two logic modules 30 using three-dimensional stacking technology.

[0119] In this embodiment, each chip can be flexibly expanded by three-dimensional stacking to increase computing power and storage.

[0120] It should be noted that when multiple modules are stacked, the bonding method for their 3D integration and substrate interconnection can be based on... Figures 15-22 , Figures 23-30 The bonding methods shown can be adapted to different scenarios, and all of the above implementation methods are within the scope of protection of this application.

[0121] It should be noted that when the off-chip volatile memory module 10, the off-chip non-volatile memory module 20, and the logic module 30 are configured individually, they can independently implement the functions of the off-chip volatile memory chip 100, the off-chip non-volatile memory chip 200, and the logic chip 300.

[0122] Optionally, the three-dimensional stacking technology is one or more of TSV, hybrid bonding, or Filp Chip.

[0123] Optionally, the on-chip volatile memory chip is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

[0124] Optionally, the global controller 340 is configured to perform fine-grained dynamic scheduling of data based on the bonding scheme interconnect characteristics of the three-dimensional stacking technology and the inherent properties of each storage medium.

[0125] Optionally, the off-chip volatile memory chip 100 includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory chip controller integrated on the logic chip 300, enabling the off-chip volatile memory chip 100 of the neural network processor to be accessed by external devices or systems as a standard DRAM memory chip. The interface protocol meets one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6. In an architecture where the off-chip volatile memory chip 100 and the Logic Die are vertically stacked through three-dimensional integration and substrate interconnect technologies (such as TSV and Hybrid Bonding), this standard interface can still ensure DRAM compatibility with external devices. At the same time, by leveraging the low latency and high bandwidth interconnect characteristics of three-dimensional integration and substrate interconnect, the data transmission efficiency during standard interface access is improved, avoiding the performance bottleneck of the standard interface under traditional 2D connection.

[0126] Optionally, the off-chip non-volatile memory chip 200 includes a standard interface (compliant with JEDEC or industry-standard memory interface standards). Taking Flash as an example, the Flash of the neural network processor can be accessed as a standard Flash memory chip. The protocol of this standard interface can be configured to meet one or more of the following: SPI (Serial Peripheral Interface), QSPI (Quad SPI), Octal SPI (OSPI), eMMC (embedded MultiMediaCard, compliant with JEDEC JESD216 standard), UFS (Universal Flash Storage, compliant with JEDEC JESD220 / JESD223 standard), NVMe (Non-Volatile Memory Express), and Parallel Flash.

[0127] The technical solution of the present invention has now been described in conjunction with the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to the specific embodiments described above. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A three-dimensional integrated and substrate interconnected in-memory neural network processor, characterized in that, include: An off-chip volatile memory chip is connected to the logic chip and is used to exchange information with the logic chip; An off-chip non-volatile memory chip, connected to the logic chip, is used for information interaction with the logic chip; and The logic chip is configured to perform any combination of the following functions: Perform computational tasks, cache data, and facilitate information exchange; The off-chip volatile memory chip and the off-chip non-volatile memory chip are integrated using three-dimensional stacking technology to form a three-dimensional integrated module. The three-dimensional integrated module is integrated with the logic chip through substrate interconnect technology.

2. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip is located at the top in the vertical direction.

3. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip non-volatile memory chip is located at the top in the vertical direction.

4. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The logic chip is a computing chip used to perform computing tasks, and the logic chip integrates a digital in-memory computing unit.

5. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The logic chip is a data cache chip used for data caching.

6. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The logic chip is an interface chip used to realize information interaction.

7. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 4, characterized in that, The logic chip integrates at least one off-chip non-volatile memory chip controller, at least one off-chip volatile memory chip controller, and a global controller. The off-chip non-volatile memory chip controller is used to access the off-chip non-volatile memory chip; The off-chip volatile memory chip controller is used to access the off-chip volatile memory chip; The global controller is used to manage the off-chip volatile memory chip controller, the off-chip non-volatile memory chip controller, and the digital in-memory computing unit, so as to realize the dynamic adaptation of data storage scheme and computing data flow, and to globally optimize storage capacity, data bandwidth, access latency and computing efficiency.

8. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The off-chip volatile memory chip is a DRAM chip.

9. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 8, characterized in that, In large language model computing scenarios, the off-chip volatile memory chip is used to store key-value cache data, activation values, and intermediate calculation results.

10. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The off-chip non-volatile memory chip is Flash.

11. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 10, characterized in that, In large language model computation scenarios, the off-chip non-volatile memory chip is used to store Q-generated weights, K-generated weights, V-generated weights, Attention weights, and FFN weights.

12. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The off-chip volatile memory chip is composed of an off-chip volatile memory module; or, It is integrated by using three-dimensional stacking technology with at least two off-chip volatile memory modules.

13. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The off-chip non-volatile memory chip is composed of an off-chip non-volatile memory module; or, It is integrated by using three-dimensional stacking technology with at least two off-chip non-volatile memory modules.

14. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The logic chip consists of a single logic module, or is integrated from at least two logic modules using three-dimensional stacking technology.

15. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The three-dimensional stacking technology is one or more of through-silicon via (TSV) technology, hybrid bonding technology, or flip-chip technology.

16. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 4, characterized in that, The digital in-memory computing unit is integrated with an on-chip volatile memory chip and a computing unit. The on-chip volatile memory chip is one or more of SRAM, eDRAM, DRAM, Flash, MRAM, and ReRAM.

17. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 7, characterized in that, The global controller is configured to perform fine-grained dynamic scheduling of data based on the interconnect characteristics of the bonding scheme using three-dimensional stacking technology and the inherent properties of each storage medium.

18. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 7, characterized in that, The global controller is configured to support the configuration of data storage mapping rules through upper-layer software.

19. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 7, characterized in that, The off-chip volatile memory chip includes a JEDEC-compliant interface that is adapted to the off-chip volatile memory chip controller integrated on the logic chip, enabling the off-chip volatile memory chip of the neural network processor to be accessed by external devices as a standard DRAM memory chip. The interface protocol satisfies one or more of LPDDR5, LPDDR6, HBM2, HBM3, HBM3e, HBM4, GDDR5, GDDR6, GDDR7, DDR5, and DDR6.

20. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 10, characterized in that, The Flash memory includes an interface that conforms to JEDEC or industry-standard memory interfaces, allowing the neural network processor's Flash memory to be accessed as a standard Flash storage chip. The interface protocol satisfies one or more of SPI, QSPI, Octal SPI, eMMC, UFS, NVMe, and Parallel Flash.

21. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to any one of claims 1-3, characterized in that, The off-chip volatile memory chip or the off-chip non-volatile memory chip and the logic chip are interconnected with the substrate through a silicon interposer.

22. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip is stacked on top of the off-chip non-volatile memory chip, the back side of the off-chip volatile memory chip is three-dimensionally integrated with the front side of the off-chip non-volatile memory chip, and the back side of the off-chip non-volatile memory chip is interconnected with the substrate.

23. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip is stacked on top of the off-chip non-volatile memory chip, the back side of the off-chip volatile memory chip is three-dimensionally integrated with the back side of the off-chip non-volatile memory chip, and the front side of the off-chip non-volatile memory chip is interconnected with the substrate.

24. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip is stacked on top of the off-chip non-volatile memory chip, the front side of the off-chip volatile memory chip is three-dimensionally integrated with the front side of the off-chip non-volatile memory chip, and the back side of the off-chip non-volatile memory chip is interconnected with the substrate.

25. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip is stacked on top of the off-chip non-volatile memory chip, the front side of the off-chip volatile memory chip and the back side of the off-chip non-volatile memory chip are three-dimensionally integrated, and the front side of the off-chip non-volatile memory chip is interconnected with the substrate.

26. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip is stacked below the off-chip non-volatile memory chip. The front side of the off-chip non-volatile memory chip is three-dimensionally integrated with the front side of the off-chip volatile memory chip, and the back side of the off-chip volatile memory chip is interconnected with the substrate.

27. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip is stacked below the off-chip non-volatile memory chip. The back side of the off-chip non-volatile memory chip is three-dimensionally integrated with the front side of the off-chip volatile memory chip, and the back side of the off-chip volatile memory chip is interconnected with the substrate.

28. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip is stacked below the off-chip non-volatile memory chip. The back side of the off-chip non-volatile memory chip is three-dimensionally integrated with the back side of the off-chip volatile memory chip, and the front side of the off-chip volatile memory chip is interconnected with the substrate.

29. The in-memory neural network processor with three-dimensional integration and substrate interconnection according to claim 1, characterized in that, The off-chip volatile memory chip is stacked below the off-chip non-volatile memory chip. The front side of the off-chip non-volatile memory chip is three-dimensionally integrated with the back side of the off-chip volatile memory chip, and the front side of the off-chip volatile memory chip is interconnected with the substrate.

30. A control method for a three-dimensional integrated and substrate interconnected in-memory neural network processor, employing the three-dimensional integrated and substrate interconnected in-memory neural network processor as described in any one of claims 1-29, characterized in that, Includes steps; Detect whether the attention module calculation has been triggered; as well as When the attention module is triggered to perform calculations, the off-chip non-volatile memory chip is scheduled to start data loading and the logic chip is scheduled to start preprocessing while the off-chip volatile memory chip is performing a read operation.

Images