A unified memory architecture suitable for distributed 3D-DRAM

Abstract

This application provides a unified memory architecture suitable for distributed 3D-DRAM, relating to the field of artificial intelligence technology, including: a storage layer, a buffer layer, and a logic layer; the storage layer is formed by three-dimensional stacking of multiple DRAM chips for storing data; the DRAM chips are interconnected with the buffer layer via through-silicon via (TSV) technology; the buffer layer, located between the storage layer and the logic layer, is used to preprocess data from the storage layer and perform data stream scheduling; the logic layer integrates multiple computing cores for parallel processing of the data streams scheduled by the buffer layer; the computing cores are interconnected with the buffer layer via TSV technology and correspond one-to-one with the DRAM chips. This application achieves unified and efficient access to local and remote memory in a distributed 3D-DRAM architecture, thereby reducing programming complexity and improving overall computing performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a unified memory architecture suitable for distributed 3D-DRAM. BACKGROUND

[0002] With the advent of the intelligent information era, the scale of data is growing explosively, especially in the frontier field of artificial intelligence, the training and inference process of large-scale deep learning models involves frequent access and complex operation of huge data. For example, the parameter scale of large language models is huge, and data needs to be constantly moved between storage units and computing units during the training process. However, due to the bandwidth bottleneck of traditional dynamic random access memory (DRAM), the storage system is difficult to meet the demand of high-performance computing, resulting in a serious "memory wall" problem, which greatly limits the improvement of computing efficiency.

[0003] In order to alleviate the contradiction of insufficient storage bandwidth, the existing technology generally adopts a high bandwidth memory (HBM) scheme to improve the memory access capability of AI chips. This scheme stacks multiple layers of DRAM particles to form a DRAM bare chip through through silicon via (TSV) and micro bump interconnection, and integrates it with a computing logic chip at the packaging level, so as to realize a higher memory bandwidth. However, due to the limitation of the number of interconnections between chips, the interconnection density between HBM and logic chips is limited, which is difficult to fully meet the demand of super large scale AI computing.

[0004] In order to further reduce the physical distance between the storage medium and the computing logic, and improve the interconnection density between the two, the industry has proposed a technology of vertically stacking DRAM and computing logic. This technology uses a hybrid bonding method to directly stack DRAM on the logic chip, and realizes high-density interconnection through TSV, thereby significantly improving the memory bandwidth. However, in actual application, since a large number of computing cores are usually integrated on the logic chip, TSV can only realize the direct interconnection of the computing core and the DRAM particle directly above it, and the DRAM particle not directly above it cannot be directly interconnected. In this way, the DRAM particles above different computing cores exhibit a distributed feature, and when a computing core needs to access the DRAM particle above other cores, the memory access efficiency will be greatly reduced.

[0005] The common solution in current technology is to use the DRAM chips stacked above each computing core as local DRAM for that core, with the computing core directly accessing the local memory space via the TSV channel. When the computing core needs to access non-local DRAM, data exchange is required through the on-chip network. This approach not only consumes on-chip network bandwidth but also results in a significant latency difference between accessing local and remote DRAM, creating an "unfriendly" memory access mode. This non-uniform memory access method is extremely unfriendly to software programming, increases development complexity, and limits the overall performance of the system.

[0006] Therefore, how to achieve a unified memory access abstraction in a distributed 3D-DRAM architecture, reduce programming complexity, and at the same time improve memory access efficiency and reduce remote access latency has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] To address the problems in the prior art, this application provides a unified memory architecture suitable for distributed 3D-DRAM, which can solve the problem of how to achieve a unified memory access abstraction in a distributed 3D-DRAM architecture, reduce programming complexity, and improve memory access efficiency and reduce remote access latency.

[0008] In a first aspect, this application provides a unified memory architecture suitable for distributed 3D-DRAM, including: a storage layer, a buffer layer, and a logic layer;

[0009] The storage layer is formed by stacking multiple DRAM chips in three dimensions and is used to store data; the DRAM chips are interconnected with the buffer layer through through-silicon via (TSV) technology.

[0010] The buffer layer, located between the storage layer and the logic layer, is used to preprocess data from the storage layer and perform data stream scheduling.

[0011] The logic layer integrates multiple computing cores for parallel processing of the data streams scheduled by the buffer layer; the computing cores are interconnected with the buffer layer through through-silicon via (TSV) technology and correspond one-to-one with the DRAM chips.

[0012] Furthermore, it also includes: multiple first data interfaces and multiple second data interfaces;

[0013] The DRAM chip interacts with the buffer layer through a corresponding first data interface.

[0014] The computing core interacts with the buffer layer through a corresponding second data interface.

[0015] Furthermore, the buffer layer includes a switching network and multiple first virtual interfaces;

[0016] The switching network is used for multi-channel parallel data exchange between the DRAM chips and the computing core through the first virtual interface.

[0017] Furthermore, the buffer layer also includes multiple local cache units and multiple global data flow management engines;

[0018] Each of the global data stream management engines is connected to a corresponding local cache unit;

[0019] The global data flow management engine and the local cache unit interact with the storage layer through the first data interface, interact with the logic layer through the second data interface, and interact with the switching network through the first virtual interface.

[0020] The global data flow management engine is used to perform cross-computing core data flow synchronization, data flow path planning, data aggregation-splitting, and data arrangement transformation operations.

[0021] The local cache unit is used to cache the received data.

[0022] Furthermore, it also includes: multiple remote DRAM storage pools and multiple third data interfaces;

[0023] Each of the remote DRAM storage pools interacts with the buffer layer via a corresponding third data interface.

[0024] Furthermore, the buffer layer also includes multiple second virtual interfaces;

[0025] The switching network is also used for multi-channel parallel data exchange between the remote DRAM storage pool and the computing core via the second virtual interface.

[0026] Furthermore, the buffer layer also includes multiple remote local cache units and multiple remote global data stream management engines;

[0027] Each of the aforementioned remote global data stream management engines is connected to a corresponding remote local cache unit;

[0028] The remote global data stream management engine and the remote local cache unit interact with the remote DRAM storage pool through the third data interface, and interact with the switching network through the second virtual interface;

[0029] The remote global data flow management engine is used to perform cross-computing core data flow synchronization, data flow path planning, data aggregation-splitting, and data arrangement transformation operations.

[0030] The remote local cache unit is used to cache the received data.

[0031] Furthermore, the DRAM chips and the remote DRAM memory pool are constructed into a single logical address space through unified addressing.

[0032] Secondly, this application provides a chip including the unified memory architecture for distributed 3D-DRAM described in any of the above embodiments.

[0033] Thirdly, this application provides an electronic device including the chip described in the above embodiments.

[0034] This application provides a unified memory architecture suitable for distributed 3D-DRAM, comprising: a storage layer, a buffer layer, and a logic layer; the storage layer is formed by three-dimensional stacking of multiple DRAM chips and is used to store data; the DRAM chips are interconnected with the buffer layer via through-silicon via (TSV) technology; the buffer layer, located between the storage layer and the logic layer, is used to preprocess data from the storage layer and perform data stream scheduling; the logic layer integrates multiple computing cores for parallel processing of the data streams scheduled by the buffer layer; the computing cores are interconnected with the buffer layer via TSV technology and correspond one-to-one with the DRAM chips. The unified memory architecture for distributed 3D-DRAM provided by this application achieves unified and efficient access to local and remote memory under a distributed 3D-DRAM architecture, thereby reducing programming complexity and improving overall computing performance.

[0035] This application enables unified abstraction and efficient management of local and remote memory in a distributed 3D-DRAM architecture. By combining "increased storage density" with "distributed resource aggregation," it constructs a low-latency, high-bandwidth, and scalable memory system, effectively reducing programming complexity, improving memory access efficiency, and significantly reducing remote access latency. At the same time, by leveraging unified addressing, high-speed interconnection, and hardware-level synchronization mechanisms, it fully integrates the high-density physical advantages of 3D-DRAM with the scalability of distributed computing, thereby breaking through the performance bottleneck of traditional memory systems and providing core support for ultra-large-scale data processing and next-generation artificial intelligence chips. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of a unified memory architecture for distributed 3D-DRAM provided in one embodiment of this application;

[0038] Figure 2 This is a schematic diagram of a unified memory architecture for distributed 3D-DRAM provided in one embodiment of this application;

[0039] Figure 3 This is a schematic diagram of a unified memory architecture for distributed 3D-DRAM provided in one embodiment of this application;

[0040] Figure 4 This is a schematic diagram of the data flow aggregation process in a unified memory architecture applicable to distributed 3D-DRAM provided in an embodiment of this application;

[0041] Figure 5 This is a schematic diagram of the data flow splitting process in a unified memory architecture applicable to distributed 3D-DRAM provided in an embodiment of this application;

[0042] Figure 6 This is a schematic diagram of the data flow splitting process in a unified memory architecture applicable to distributed 3D-DRAM provided in an embodiment of this application;

[0043] Figure 7 This is a schematic diagram of the data flow aggregation process in a unified memory architecture applicable to distributed 3D-DRAM provided in an embodiment of this application;

[0044] Figure 8 This is a schematic diagram of the data flow splitting process in a unified memory architecture applicable to distributed 3D-DRAM provided in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of the data flow aggregation process in a unified memory architecture applicable to distributed 3D-DRAM provided in an embodiment of this application;

[0046] Figure 10 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0048] This application applies to artificial intelligence (AI) chips, AI servers, and AI server clusters. The technology of this application can effectively reduce programming complexity, while improving memory access efficiency and reducing memory access latency. It can also effectively combine the physical advantages of 3D-DRAM with the scalability of distributed computing, breaking through the performance bottleneck of traditional memory systems.

[0049] Figure 1 This is a schematic diagram of a unified memory architecture for distributed 3D-DRAM provided in an embodiment of this application, as shown below. Figure 1 As shown, the unified memory architecture for distributed 3D-DRAM provided in this application includes: a storage layer, a buffer layer, and a logic layer;

[0050] The storage layer is formed by three-dimensional stacking of multiple DRAM chips (DRAM-1 to DRAM-x) for storing data; the DRAM chips are interconnected with the buffer layer through through-silicon via (TSV) technology.

[0051] The buffer layer, located between the storage layer and the logic layer, is used to preprocess data from the storage layer and perform data stream scheduling.

[0052] The logic layer integrates multiple computing cores (Tile-1 to Tile-x) for parallel processing of the data streams scheduled by the buffer layer; the computing cores are interconnected with the buffer layer through through-silicon via (TSV) technology and correspond one-to-one with the DRAM chips.

[0053] Specifically, the storage layer, buffer layer, and logic layer are interconnected via three-dimensional stacking in a top-down order to achieve high-bandwidth, low-latency data interaction.

[0054] The DRAM Die, the core storage area of ​​the memory, is composed of multiple DRAM chips. These chips are integrated in a three-dimensional stacking manner to form a high-density storage structure. Each DRAM chip serves as a data storage unit, used to store the large-scale data required for computing tasks. To achieve efficient interconnection, the DRAM chips are electrically connected to the buffer layer through through-silicon vias (TSVs), enabling the transmission of stored data to the buffer layer with a shorter data path and higher interconnect density.

[0055] The buffer layer is positioned between the storage layer and the logic layer, acting as a data bridge between the two. The buffer layer not only has the ability to cache and temporarily store data, but also performs format conversion and preliminary processing operations on data from the storage layer, reducing the overhead of direct interaction between the storage layer and the logic layer and improving data transmission efficiency.

[0056] For example, when data is transferred from DRAM chips to the buffer layer, the buffer layer can perform preliminary sorting and scheduling to generate a data stream that meets the processing requirements of the logic layer. This effectively reduces the latency and bandwidth pressure when the computing core directly accesses the underlying storage. The existence of the buffer layer allows data to be optimized and scheduled before entering the logic layer, ensuring the high efficiency of subsequent parallel processing.

[0057] The Logic Die, located at the bottom of the architecture, integrates multiple computing cores. Each computing core contains numerous arithmetic units for executing complex computational tasks in parallel. Each computing core is connected to the buffer layer via through-silicon vias (TSVs) and can receive data streams scheduled by the buffer layer.

[0058] To achieve efficient mapping between tasks and storage resources, each computing core is associated with a DRAM chip, meaning there is a DRAM chip directly above each computing core. This mapping significantly reduces latency for local data access, while the preprocessing and scheduling mechanisms of the buffer layer ensure the orderly and efficient flow of data across cores.

[0059] In summary, the unified memory architecture in this application embodiment, through the hierarchical structure of storage layer, buffer layer and logic layer, and the high-density interconnection achieved by through silicon vias, enables data to flow from the storage layer, through the preprocessing and scheduling of the buffer layer, to the computing core of the logic layer for parallel processing, thereby achieving higher memory access efficiency and computing performance in a distributed 3D-DRAM environment.

[0060] In one embodiment, the computing core can be a variety of computing units such as a CPU core, a GPU core, or an NPU core.

[0061] Figure 2 This is a schematic diagram of a unified memory architecture for distributed 3D-DRAM provided in an embodiment of this application, as shown below. Figure 2 As shown, the unified memory architecture for distributed 3D-DRAM provided in this application also includes: multiple first data interfaces (DIO-1~DIO-x) and multiple second data interfaces (TIO-1~TIO-x).

[0062] The DRAM chip interacts with the buffer layer through a corresponding first data interface.

[0063] The computing core interacts with the buffer layer through a corresponding second data interface.

[0064] Specifically, multiple first data interfaces (DIO) and multiple second data interfaces (VIO) are used to achieve efficient data interaction between the storage layer, buffer layer, and logic layer.

[0065] The first data interface establishes a data transmission channel between the storage layer and the buffer layer. It serves as the I / O interface between the DRAM chips and the buffer layer, implemented using TSV technology. Each DRAM chip connects to the buffer layer through its corresponding first data interface, allowing its internally stored data to be directly transmitted to the buffer layer. Through this interface, the buffer layer can acquire data resources from different DRAM chips at high speed and low latency, and subsequently cache, preprocess, or schedule them, effectively shortening the data transmission path.

[0066] The second data interface is used to establish a data transmission channel between the logic layer and the buffer layer. It serves as the I / O interface between the computing core and the buffer layer and is implemented using TSV technology. Each computing core connects to the buffer layer through its corresponding second data interface, enabling it to receive data streams transmitted from the buffer layer and input them into the computing process.

[0067] Simultaneously, the computation results generated by the computing core can also be sent back to the buffer layer through this interface for further data storage or distribution. Since each computing core has its own independent second data interface, it can support multiple computing cores to access the data resources of the buffer layer in parallel, effectively improving the overall computing throughput.

[0068] Through the establishment of the first and second data interfaces, a clear hierarchical communication relationship is formed between the storage layer, the buffer layer, and the logic layer. The first data interface ensures efficient data transfer from the storage layer to the buffer layer, while the second data interface guarantees high-speed interaction between the buffer layer and the logic layer's computing cores. Their synergistic effect enables the unified memory architecture proposed in this application to achieve smoother data flow in a distributed 3D-DRAM environment, providing a solid foundation for subsequent parallel processing and cross-core data access.

[0069] In one embodiment, such as Figure 2 As shown, the buffer layer includes a switching network and multiple first virtual interfaces (VIO-1~VIO-x).

[0070] The switching network is used for multi-channel parallel data exchange between the DRAM chips and the computing core through the first virtual interface.

[0071] Specifically, the first virtual interface is located inside the buffer layer, serving as a logical abstraction port, corresponding to the DRAM chips in the storage layer and the computing cores in the logic layer. The first virtual interface is used to associate the DRAM chips and computing cores of this group with those of other groups via a switching network.

[0072] The switching network connects to multiple primary virtual interfaces (PVAs) to forward data between different PVAs and establish data transmission channels between DRAM chips and computing cores based on task requirements. The switching network can support multiple parallel transmission paths, allowing multiple data streams to be transmitted simultaneously between different PVAs. This multi-channel parallel mechanism significantly improves the bandwidth of cross-core data interaction and reduces latency when accessing remote memory units.

[0073] By setting up a switching network and a first virtual interface in the buffer layer, this application establishes a flexible data exchange mechanism between the storage layer and the logic layer. On the one hand, the virtual interface realizes a one-to-one correspondence abstraction between computing cores and storage units, simplifying system addressing and scheduling; on the other hand, the switching network provides high-bandwidth, multi-channel data transmission capabilities, ensuring efficient data flow and parallel access under a distributed 3D-DRAM architecture. This design not only improves the scalability of the architecture but also provides excellent support for subsequent large-scale parallel computing tasks.

[0074] In one embodiment, the switching network is a high-speed interconnect network, such as a mesh or crossbar, which facilitates efficient data exchange between distributed DRAM memory access and computing cores, supports multi-channel parallelism, and meets the memory access requirements of AI parallel computing tasks.

[0075] In one embodiment, such as Figure 2 As shown, the buffer layer also includes multiple local cache units and multiple global data flow management engines;

[0076] Each of the global data stream management engines is connected to a corresponding local cache unit;

[0077] The global data flow management engine and the local cache unit interact with the storage layer through the first data interface, interact with the logic layer through the second data interface, and interact with the switching network through the first virtual interface.

[0078] The global data flow management engine is used to perform cross-computing core data flow synchronization, data flow path planning, data aggregation-splitting, and data layout transformation operations.

[0079] The local cache unit is used to cache the received data.

[0080] Specifically, each global data flow management engine (GS engine) corresponds to a local cache unit (Cache) to jointly complete data preprocessing and scheduling.

[0081] The global data flow management engine is primarily used to implement data flow control and management functions across computing cores. When multiple computing cores execute tasks in parallel, there are certain dependencies between data access between different cores. The global data flow management engine can perform unified timing coordination of these data within the buffer layer, thereby avoiding data inconsistency or processing delays.

[0082] The global data flow management engine possesses various data coordination and optimization functions. First, it performs cross-core data flow synchronization, coordinating the timing of input and output data when multiple cores are processing tasks in parallel to ensure data consistency and processing synchronization across cores. Second, the engine can perform data flow path planning, dynamically determining the optimal transmission path based on the current data access source, target core, and system load status. This allows data to flow efficiently across multiple interfaces and switching networks, thereby reducing communication conflicts and latency.

[0083] Furthermore, the global data flow management engine can perform aggregation and splitting operations on data. This involves aggregating data distributed across different storage units into a single location so that the computing cores in the logical layer can process it efficiently in parallel, or splitting large-scale data streams into multiple data sets as needed. In summary, the global data flow management engine coordinates parallel computing among multiple computing cores to ensure data consistency (e.g., supporting multiple communication protocols such as MPI to accelerate distributed task synchronization); and it performs data aggregation-scat operations to reduce DRAM access overhead.

[0084] Building on this, the global data flow management engine can also perform data arrangement transformation operations, that is, adjust the data format, layout or alignment in the cache or storage according to the characteristics of different computing tasks, so that the data is in the optimal arrangement state before being transmitted to the logical layer computing core, thereby further improving memory access bandwidth and processing efficiency.

[0085] Local cache units are connected to the global data flow management engine and are used to temporarily store data processed by the engine. Through this caching mechanism, local cache units effectively reduce the frequency of direct access to the underlying DRAM by the computing cores, thereby reducing memory access latency and alleviating storage bandwidth pressure. Simultaneously, local cache units also provide high-speed data read capabilities, enabling the computing cores in the logic layer to quickly obtain the necessary data when needed.

[0086] In terms of data interaction, the global data flow management engine and local cache units establish a connection with the storage layer through a first data interface to receive data from DRAM chips; they establish a connection with the logic layer through a second data interface to provide scheduled data flows to the computing cores; and they establish a connection with the switching network through a first virtual interface to support data exchange in cross-core or cross-node scenarios. In this way, data can form an efficient transmission channel between the storage layer, buffer layer, and logic layer, while synchronization, path planning, aggregation, and cache optimization are achieved within the buffer layer.

[0087] As can be seen, by introducing local cache units and a global data flow management engine into the buffer layer, this application can achieve efficient data preprocessing and global coordination in a distributed 3D-DRAM architecture, which not only ensures the correctness of cross-core access but also improves the overall memory access efficiency, providing strong support for large-scale parallel computing tasks.

[0088] In one embodiment, such as Figure 2 As shown, the unified memory architecture for distributed 3D-DRAM provided in this application also includes: multiple remote DRAM storage pools (Remote DRAM-1~Remote DRAM-y) and multiple third data interfaces (Interconnect Interface-1~Interconnect Interface-y).

[0089] Each of the remote DRAM storage pools interacts with the buffer layer via a corresponding third data interface.

[0090] Specifically, by introducing remote storage, this application can overcome the capacity limitations of single-chip stacked DRAM and achieve distributed memory expansion across nodes. The remote DRAM storage pool is a DRAM space abstracted from other chips or nodes in the system. Through chip-level interconnect interfaces (such as Die2Die and Chip2Chip) and network interconnect interfaces (such as ROCE, InfiniBand, etc.), the memory capacity is expanded to form a shared memory pool with unified addressing.

[0091] Remote DRAM consists of multiple DRAM resources located outside the local chip. These remote memory cells are connected to the local chip via chip interconnect technology and are logically abstracted into a unified memory pool. This design enables dynamic expansion of storage capacity without changing the physical scale of the local storage layer, thereby meeting the high demand for storage resources in large-scale computing scenarios.

[0092] To enable data interaction with the buffer layer, each remote DRAM memory pool is equipped with an independent third data interface (interconnect interface). Through this interface, the buffer layer can efficiently transfer data from the logic layer to the remote DRAM memory pool, and simultaneously retrieve data from the remote DRAM memory pool for further processing and distribution within the buffer layer. The third data interface provides a stable communication channel for remote memory access, ensuring that, from the perspective of the logic layer's computing core, there is no fundamental difference between local and remote storage.

[0093] By introducing a remote DRAM storage pool and a corresponding third data interface, the unified memory architecture proposed in this application can achieve cross-node expansion based on local stacked storage, which not only improves the scalability of the memory system but also provides more flexible storage resource management capabilities for large-scale parallel computing. In this way, the computing cores in the logical layer do not need to distinguish between local and remote storage when accessing data, significantly reducing programming complexity and maintaining high memory access efficiency in a distributed environment.

[0094] In one embodiment, the third data interface may refer to chip-level interconnect interfaces such as Die2Die and Chip2Chip, or network interconnect interfaces such as ROCE and InfiniBand.

[0095] In one embodiment, such as Figure 2 As shown, the buffer layer also includes multiple second virtual interfaces (Remote VIO-1~Remote VIO-y).

[0096] The switching network is also used for multi-channel parallel data exchange between the remote DRAM storage pool and the computing core via the second virtual interface.

[0097] Specifically, the buffer layer also includes multiple Remote VIOs (Second Virtual Interfaces) to support unified access and management of remote storage resources. Each remote VIO corresponds to a remote DRAM pool and serves as a logical abstraction port to integrate the remote DRAM pool into the communication path with the local computing core. This allows the remote DRAM pool to be accessed in a manner similar to local DRAM, simplifying the complexity of addressing and data scheduling.

[0098] The switching network is connected not only to the first virtual interface for data exchange between the local DRAM and the computing core, but also to the second virtual interface for establishing a high-speed communication channel between the remote DRAM storage pool and the computing core.

[0099] By introducing a second virtual interface, the switching network can uniformly schedule remote and local storage access in cross-chip or cross-node environments, ensuring that the computing cores of the logical layer can transparently access storage resources from different sources. Remote VIO and local VIO work together to manage cross-node memory access, ensuring the consistency and scalability of distributed memory (supporting dynamic node addition and elastic expansion of the memory pool).

[0100] More importantly, the switching network supports a multi-channel parallel data exchange mechanism through multiple secondary virtual interfaces. When computing cores access different remote DRAM memory pools simultaneously, data streams can be transmitted in parallel across multiple virtual interfaces, avoiding the bottleneck problem of single-channel bandwidth limitation. This design enables the maintenance of high memory access bandwidth and low latency in a distributed 3D-DRAM architecture, meeting the performance requirements of large-scale parallel computing tasks for remote memory access.

[0101] Therefore, by setting a second virtual interface in the buffer layer and combining it with the switching network, this application not only achieves seamless access to remote storage resources, but also significantly improves cross-node memory access efficiency through a multi-channel parallel mechanism, further enhancing the scalability and computing support capabilities of the unified memory architecture.

[0102] In one embodiment, such as Figure 2 As shown, the buffer layer also includes multiple remote local cache units and multiple remote global data stream management engines;

[0103] Each of the aforementioned remote global data stream management engines is connected to a corresponding remote local cache unit;

[0104] The remote global data stream management engine and the remote local cache unit interact with the remote DRAM storage pool through the third data interface, and interact with the switching network through the second virtual interface;

[0105] The remote global data flow management engine is used to perform cross-computing core data flow synchronization, data flow path planning, data aggregation-splitting, and data arrangement transformation operations.

[0106] The remote local cache unit is used to cache the received data.

[0107] Specifically, the buffer layer also includes multiple remote local cache units (Remote Cache) and multiple remote global data stream management engines (GS engines). Each remote global data stream management engine corresponds to one remote local cache unit, forming a functional pairing to achieve preprocessing and synchronization management during remote storage access. The number of remote global data stream management engines and remote local cache units is the same as the number of third data interfaces (interconnect interfaces), with each interconnect interface corresponding to a set of remote global data stream management engines and remote local cache units.

[0108] The remote global data flow management engine is used to perform cross-compute core data synchronization when accessing remote DRAM storage pools. When multiple compute cores request remote data simultaneously, the remote global data flow management engine can coordinate the access requests to ensure that the data acquired by different compute cores is consistent in time and order.

[0109] The remote global data flow management engine performs cross-core data flow synchronization, ensuring data consistency and timing coordination when multiple compute cores access the same remote storage resource. It also performs data flow path planning, dynamically selecting the optimal transmission path based on the source location of remote data and network topology to reduce cross-node access latency. This engine can also perform aggregation and splitting operations on remote data, aggregating write-back data from different compute cores and transmitting it uniformly to remote DRAM, or splitting remotely read data and distributing it to multiple compute cores. Simultaneously, it optimizes the data storage structure in the remote cache through data arrangement transformations, making it more suitable for the parallel access needs of compute cores.

[0110] Among them, the remote global data flow management engine performs data aggregation and splitting operations, which aggregates data from multiple computing cores or splits large-scale data tasks into small data streams suitable for parallel computing, thereby improving the efficiency and flexibility of remote memory access.

[0111] The remote local cache unit, corresponding to the remote global data flow management engine, is used to temporarily store data. When data from the remote DRAM storage pool is loaded through a third data interface, the remote local cache unit can cache this data in advance, thereby avoiding the high latency caused by frequent access to remote storage by the computing core.

[0112] Meanwhile, the remote local cache unit can also store data processed by the remote global data flow management engine, enabling the logic layer's computing cores to quickly read it and further shorten the response time for cross-node memory access. The remote local cache unit utilizes RDMA (Remote Direct Memory Access) or load-store memory access semantics to achieve unified access to local and remote DRAM, reducing latency.

[0113] In terms of data interaction, each remote global data flow management engine and remote local cache unit connects to the remote DRAM storage pool through a third data interface to acquire or write remote storage data. Simultaneously, they are also connected to the switching network through a second virtual interface to enable efficient data exchange with the computing core of the logic layer in cross-node access scenarios. Remote DRAM refers to DRAM on other chips, which can be accessed through the third data interface of the current chip. The third data interface on the current chip corresponds to a set of remote global data flow management engines and remote local cache units, used to manage and control data access to the remote DRAM. Through this structural design, this application introduces synchronization and caching mechanisms into the remote storage access link, effectively reducing latency differences caused by cross-chip memory access.

[0114] As can be seen, by setting up remote local cache units and remote global data flow management engines in the buffer layer, this application not only enhances the data processing and synchronization capabilities of the remote DRAM storage pool, but also reduces the latency of remote memory access through the caching mechanism, further ensuring the unified memory access performance under the distributed 3D-DRAM architecture.

[0115] In one embodiment, the DRAM chips and the remote DRAM memory pool are constructed into a single logical address space through unified addressing.

[0116] Specifically, the multiple DRAM chips in the local storage layer and the multiple DRAM storage pools in the remote node all logically adopt the same address allocation rules and addressing system, thus forming a continuous storage space at the logical level.

[0117] Through this unified addressing mechanism, the computational cores of the logical layer do not need to distinguish whether their target data is stored in local DRAM or a remote DRAM storage pool when accessing storage resources. The computational cores only need to access the data based on the unified logical address, and the data can be automatically located and retrieved in local or remote storage according to the address mapping relationship. This approach effectively shields the physical differences in storage location, reduces the dependence of upper-layer software on storage location during programming, and significantly simplifies the programming complexity in distributed architectures.

[0118] Furthermore, the construction of a single logical address space can improve overall memory access efficiency. When the scale of computing tasks expands and remote DRAM storage pools need to be accessed, the logical layer computing cores can directly use a unified logical address for transparent access, without introducing additional addressing operations or data management burdens due to the distribution of storage across nodes. At the same time, unified addressing ensures the scalability of the architecture; that is, when adding new remote storage resources, they only need to be included in the unified address space, and the logical layer computing cores do not need to undergo additional adaptation.

[0119] Therefore, by unifying the addressing of DRAM chips and remote DRAM memory pools to form a single logical address space, this application not only realizes the logical aggregation of cross-node storage resources, but also significantly reduces the complexity of software development and system management, and improves the overall performance of the distributed 3D-DRAM architecture while maintaining a user-friendly programming interface.

[0120] In one embodiment, such as Figure 3 As shown, taking a certain computing chip as an example, this chip integrates 8 computing core tiles, and stacks 8 DRAM chips on top of the computing cores. Each computing core corresponds one-to-one with a DRAM chip, thus forming a local storage and computing unit. To further expand storage capacity, this computing chip also has 2 interconnect interfaces for connecting to external remote DRAM storage pools.

[0121] In one embodiment, such as Figure 4 As shown, when the computing core Tile-1 needs to read a copy of data from each of the eight DRAM chips (including DRAM-1 to DRAM-8) and perform calculations locally, the data access process is as follows:

[0122] 1. Tile-1 initiates a read request through its corresponding TIO-1 interface, and the requested data is distributed in DRAM-1 to DRAM-8.

[0123] 2. After VIO-1 receives the request, the corresponding global data flow management engine (GS engine) in the coordination buffer layer extracts data from DRAM-1 to DRAM-8 and aggregates it.

[0124] Data from DRAM-1 is directly accessed by DIO-1, while data from DRAM-2 to DRAM-8 is transmitted in parallel to the GS engine via a high-bandwidth switching network.

[0125] 3. After aggregation, the data is returned to Tile-1 through the TIO-1 interface for processing by the computing core, thus achieving low-latency, high-bandwidth data reading.

[0126] In one embodiment, such as Figure 5 As shown, when Tile-1 completes its calculation and needs to write the result data back to the original address space, the data splitting process is as follows:

[0127] 1. Tile-1 transmits the output data to the corresponding GS engine through the TIO-1 interface. The GS engine splits the data into write-back portions for each DRAM chip according to the address mapping relationship.

[0128] 2. The write-back portion of DRAM-1 is retrieved from the corresponding local cache unit Cache by DIO-1 and written back to DRAM-1; the write-back portions of DRAM-2 to DRAM-8 are retrieved in parallel from the local cache unit Cache corresponding to Tile-1 and then distributed to DRAM-2 to DRAM-8 through the switching network.

[0129] With the help of a distributed 3D-DRAM architecture, the split data can be written in parallel to the DRAM corresponding to multiple computing cores, thereby significantly improving write bandwidth and reducing overall latency.

[0130] In one embodiment, such as Figure 6 As shown, when data in DRAM-1 needs to be distributed across 8 computing cores (including Tile-1 to Tile-8) for parallel processing, the data transfer process is as follows:

[0131] 1. Read requests from multiple computing cores are first aggregated within the buffer layer and then uniformly accessed by DRAM-1. After DRAM-1 reads the required data, it transmits it via DIO-1 to the corresponding global data flow management engine (GS engine) in the buffer layer. The GS engine then splits the data and stores the split data in the corresponding local cache unit.

[0132] 2. For the data required by the computing core Tile-1, TIO-1 retrieves the corresponding portion from the local cache unit Cache and transmits it to Tile-1 for computation. For the data required by the remaining 7 computing cores Tile-2 to Tile-8, VIO-1 retrieves the corresponding portion from the local cache unit Cache and sends it to Tile-2 to Tile-8 respectively through the switching network, realizing high-bandwidth data access across tiles.

[0133] In one embodiment, such as Figure 7 As shown, when the eight computing cores have completed data processing and need to write the results back to DRAM-1, the data aggregation process is as follows:

[0134] 1. Tile-1 writes the calculation results into the local cache unit Cache via TIO-1 for temporary storage; at the same time, the processing results generated by Tile-2 to Tile-8 are transmitted to VIO-1 via the switching network and written into the corresponding local cache unit Cache for temporary storage.

[0135] 2. The GS engine reads the result data from the 8 computing cores from the cache and aggregates it to obtain a complete data stream.

[0136] 3. After aggregation, the data is written directly back to the original address space of DRAM-1 via DIO-1 to achieve unified data updates.

[0137] Through the above process, the embodiments of this application can achieve efficient splitting and parallel processing of data in a single storage unit (DRAM-1) across multiple computing cores, and quickly aggregate and write back the results after the task is completed.

[0138] In one embodiment, such as Figure 8 As shown, when data needs to be read from Remote DRAM-1 and distributed across 8 computing cores for parallel processing, the execution flow is as follows:

[0139] 1. Multiple computing cores initiate read requests through their respective TIO and VIO, and after being aggregated via the on-chip switching network, they access Remote DRAM-1 through Interconnect Interface-1 to complete the remote reading of data.

[0140] 2. The data read from Remote DRAM-1 enters the corresponding Remote Global Data Flow Management Engine (GS Engine) in the buffer layer. The GS Engine splits the data into data slices adapted to each computing core and caches these data slices in the corresponding Remote Cache unit.

[0141] 3. Subsequently, the remote virtual interface Remote VIO-1 reads data slices from the remote cache and distributes them in parallel to the eight computing cores Tile-1 to Tile-8 through the switching network, realizing high-bandwidth data transmission across nodes.

[0142] In one embodiment, such as Figure 9 As shown, when the 8 computing cores have completed data processing and need to write the results back to Remote DRAM-1, the execution flow is as follows:

[0143] 1. Each computing core transmits the processed data to the Remote Cache unit via the corresponding TIO and VIO interfaces through the switching network for temporary storage.

[0144] 2. The corresponding remote global data flow management engine (GS engine) reads the write-back data from the 8 computing cores from the Remote Cache, aggregates it, and generates a complete data flow.

[0145] 3. The aggregated data is transmitted directly to the remote node through the interconnect interface and finally written into the corresponding address space of Remote DRAM-1.

[0146] As can be seen from this embodiment, the unified memory architecture proposed in this application not only supports the aggregation and splitting of local DRAM, but also enables efficient data reading, distribution, aggregation, and write-back in remote DRAM scenarios. Through the synergistic effect of remote caching, the GS engine, and remote virtual interfaces, it can maintain low latency and high bandwidth in a distributed environment, significantly improving the overall performance of cross-node large-scale parallel computing.

[0147] This application provides a unified memory architecture suitable for distributed 3D-DRAM, comprising: a storage layer, a buffer layer, and a logic layer; the storage layer is formed by three-dimensional stacking of multiple DRAM chips and is used to store data; the DRAM chips are interconnected with the buffer layer via through-silicon via (TSV) technology; the buffer layer, located between the storage layer and the logic layer, is used to preprocess data from the storage layer and perform data stream scheduling; the logic layer integrates multiple computing cores for parallel processing of the data streams scheduled by the buffer layer; the computing cores are interconnected with the buffer layer via TSV technology and correspond one-to-one with the DRAM chips. The unified memory architecture for distributed 3D-DRAM provided by this application achieves unified and efficient access to local and remote memory under a distributed 3D-DRAM architecture, thereby reducing programming complexity and improving overall computing performance.

[0148] This application enables unified abstraction and efficient management of local and remote memory in a distributed 3D-DRAM architecture. By combining "increased storage density" with "distributed resource aggregation," it constructs a low-latency, high-bandwidth, and scalable memory system, effectively reducing programming complexity, improving memory access efficiency, and significantly reducing remote access latency. At the same time, by leveraging unified addressing, high-speed interconnection, and hardware-level synchronization mechanisms, it fully integrates the high-density physical advantages of 3D-DRAM with the scalability of distributed computing, thereby breaking through the performance bottleneck of traditional memory systems and providing core support for ultra-large-scale data processing and next-generation artificial intelligence chips.

[0149] This application also provides a chip including the unified memory architecture for distributed 3D-DRAM described in any of the above embodiments.

[0150] This application also provides an electronic device, including the chip described in the above embodiments. The electronic device may be a terminal device or a server, and this application does not limit the specific type of electronic device.

[0151] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, the electronic device may be an AI server, a training and promotion integrated machine, etc.

[0152] Reference Figure 10The electronic device may include one or more of the following components: an AI-accelerated computing module, a CPU module, a power supply module, a hard drive module, and a fan module. Each module works in conjunction with the bus system through a standardized hardware interface, as detailed in the following architecture:

[0153] The AI-accelerated computing module comprises multiple AI accelerator cards deployed in parallel. Each AI accelerator card integrates at least one AI accelerator chip (such as an RPU chip, GPU chip, or CGRA chip). Data communication between the AI ​​accelerator cards is achieved through a high-speed card-to-card (C2C) interconnect structure, supporting low-latency, high-bandwidth horizontal scaling. The AI ​​accelerator chip is dedicated to performing AI computing tasks such as high-density matrix operations, neural network model training, and / or inference, providing the main computing power support.

[0154] The CPU module includes at least one CPU board, which houses a central processing unit (CPU) chip and associated CPU memory (such as DDR4 / DDR5, RAM). The CPU chip serves as the system control center, responsible for task scheduling, resource allocation, I / O management, and coordinating the parallel computing of the AI ​​acceleration computing module, while also handling non-accelerated general-purpose computing tasks.

[0155] The power module is equipped with redundant power supply units to provide stable power distribution and management for the AI ​​acceleration computing module, CPU module and other modules.

[0156] The hard drive module integrates a high-speed solid-state drive (SSD) and / or a large-capacity hard disk drive (HDD), connected to the system bus via a backplane. The hard drive stores the operating system, AI training datasets, model parameters, and computation results, providing high-throughput data read / write channels and supporting data preprocessing and persistence.

[0157] The fan module uses a multi-zone independent speed-controlled fan array, which is configured in key heat source areas (such as AI accelerator cards and CPU heat dissipation areas) to achieve system heat dissipation through forced air cooling and ensure the stable operation of high-efficiency computing components.

[0158] The CPU module is connected to the AI ​​acceleration computing module via the PCIe bus to enable task distribution, result collection, and memory coordination.

[0159] The CPU module manages the data access of the hard drive module through SATA / SAS / NVMe interfaces.

[0160] The power module provides tiered power to all functional modules through the power distribution backplane.

[0161] The fan module adjusts the fan speed based on temperature monitoring signals from the CPU board and AI accelerator card.

[0162] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0163] The terms "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. The order of steps involved in the various embodiments is used to illustrate the implementation of this application, and the order of steps is not limited and may be adjusted appropriately as needed.

[0164] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A unified memory architecture suitable for distributed 3D-DRAM, characterized in that, include: Storage layer, buffer layer, logic layer, multiple first data interfaces and multiple second data interfaces; The storage layer is formed by stacking multiple DRAM chips in three dimensions and is used to store data; the DRAM chips are interconnected with the buffer layer through through-silicon via (TSV) technology. The buffer layer, located between the storage layer and the logic layer, is used to preprocess data from the storage layer and perform data stream scheduling. The logic layer integrates multiple computing cores for parallel processing of the data streams scheduled by the buffer layer; the computing cores are interconnected with the buffer layer through through-silicon vias (TSVs) and correspond one-to-one with the DRAM chips. The DRAM chip interacts with the buffer layer through a corresponding first data interface. The computing core interacts with the buffer layer through a corresponding second data interface. The buffer layer includes a switching network, multiple first virtual interfaces, multiple local cache units, and multiple global data flow management engines; The switching network is used to perform multi-channel parallel data exchange between the DRAM chip and the computing core through the first virtual interface; Each of the global data stream management engines is connected to a corresponding local cache unit; The global data flow management engine and the local cache unit interact with the storage layer through the first data interface, interact with the logic layer through the second data interface, and interact with the switching network through the first virtual interface. The global data flow management engine is used to perform cross-computing core data flow synchronization, data flow path planning, data aggregation-splitting, and data arrangement transformation operations. The local cache unit is used to cache the received data.

2. The unified memory architecture for distributed 3D-DRAM according to claim 1, characterized in that, Also includes: Multiple remote DRAM storage pools and multiple third-party data interfaces; Each of the remote DRAM storage pools interacts with the buffer layer via a corresponding third data interface.

3. The unified memory architecture for distributed 3D-DRAM according to claim 2, characterized in that, The buffer layer also includes multiple second virtual interfaces; The switching network is also used for multi-channel parallel data exchange between the remote DRAM storage pool and the computing core via the second virtual interface.

4. The unified memory architecture for distributed 3D-DRAM according to claim 3, characterized in that, The buffer layer also includes multiple remote local cache units and multiple remote global data stream management engines; Each of the aforementioned remote global data stream management engines is connected to a corresponding remote local cache unit; The remote global data stream management engine and the remote local cache unit interact with the remote DRAM storage pool through the third data interface, and interact with the switching network through the second virtual interface; The remote global data flow management engine is used to perform cross-computing core data flow synchronization, data flow path planning, data aggregation-splitting, and data arrangement transformation operations. The remote local cache unit is used to cache the received data.

5. The unified memory architecture for distributed 3D-DRAM according to claim 2, characterized in that, The DRAM chips and the remote DRAM memory pool are constructed into a single logical address space through unified addressing.

6. A chip, characterized in that, Includes a unified memory architecture suitable for distributed 3D-DRAM as described in any one of claims 1-5.

7. An electronic device, characterized in that, Includes the chip described in claim 6.

Images