Dual-channel memory addressing and bus system based on 3D stacking

Abstract

The present invention belongs to the field of semiconductor technology, specifically to a 3D-stacked dual-channel memory addressing and bus system. The system includes at least one intelligent memory processing unit (IMU), which comprises a 3D-stacked DRAM storage layer and a logic computing layer. The memory within the IMU is divided into two independent memory channels, each of which includes several compute engine groups. Each compute engine group includes several compute engine blocks, each of which is connected to a DRAM storage unit, an instruction scheduling engine, and a memory controller. This dual-channel design is suitable for high-bandwidth, low-latency applications such as high-performance computing, real-time data analysis, and image processing.

Description

Technical Field

[0001] The present invention belongs to the field of semiconductor technology, and in particular relates to a dual-channel memory addressing and bus system based on 3D stacking. Background Art

[0002] Existing computing systems (such as GPU / DRAM architecture) have memory bandwidth bottlenecks. Especially in the processing of large AI models, data transmission efficiency is low, resulting in insufficient utilization of computing resources.

[0003] Traditional DRAM-PIM architectures, due to shared physical space between memory and computing units or customized circuit designs, limit computing flexibility and bandwidth. Specifically, traditional DRAM-PIM architectures lack flexible memory address allocation mechanisms, making it difficult to support multimodal or multitasking parallel computing. The single-channel memory access bandwidth of traditional DRAM-PIM architectures is insufficient to meet the high-concurrency data transmission requirements of AI models, for example. Summary of the Invention

[0004] The present invention addresses the technical problems of limited computing flexibility and bandwidth in traditional DRAM-PIM architecture, and aims to provide a dual-channel memory addressing and bus system based on 3D stacking.

[0005] In order to solve the aforementioned technical problems, the present invention provides a dual-channel memory addressing and bus system based on 3D stacking, wherein the dual-channel memory addressing and bus system based on 3D stacking includes at least one smart memory processing unit (SMPU), wherein the smart memory processing unit includes a 3D stacked DRAM storage layer and a logic computing layer, wherein the DRAM storage layer has a plurality of DRAM storage cells, and the logic computing layer has a plurality of computing engine groups, a plurality of instruction scheduling engines, and a plurality of memory controllers;

[0006] The memory within the intelligent memory processing unit is divided into two independent memory channels, each of the memory channels includes several groups of computing engine groups, a single group of computing engine groups includes several computing engine blocks, and a single computing engine block is correspondingly connected to one DRAM storage unit, one instruction scheduling engine, and one memory controller. The instruction scheduling engine is used to schedule computing tasks, and the memory controller is used to allocate the DRAM storage layer resources according to different preset working modes.

[0007] Optionally, in the aforementioned dual-channel memory addressing and bus system based on 3D stacking, copper interconnection is achieved between the DRAM storage layer and the logic computing layer through hybrid bonding.

[0008] Optionally, in the aforementioned dual-channel memory addressing and bus system based on 3D stacking, a single DRAM storage unit includes a plurality of DRAM banks, and preferably a single DRAM storage unit includes four 4MB DRAM banks;

[0009] The instruction scheduling engine is further configured to calculate a mapping relationship between the computing engine block and the DRAM bank.

[0010] Optionally, in the aforementioned dual-channel memory addressing and bus system based on 3D stacking, a single DRAM storage unit includes 4 DRAM bank groups, and a single DRAM bank group includes 32 DRAM banks;

[0011] The instruction scheduling engine is further configured to calculate a mapping relationship between the computing engine block and the DRAM bank.

[0012] Optionally, in the aforementioned dual-channel memory addressing and bus system based on 3D stacking, each of the memory channels includes four groups of computing engine groups, and a single group of the computing engine group includes two computing engine blocks.

[0013] Optionally, in the aforementioned dual-channel memory addressing and bus system based on 3D stacking, the number of the DRAM storage units is the same as the number of the computing engine blocks.

[0014] Optionally, in the aforementioned dual-channel memory addressing and bus system based on 3D stacking, each of the memory channels shares the same instruction scheduling engine, and each group of the computing engines shares the same memory controller.

[0015] Optionally, in the dual-channel memory addressing and bus system based on 3D stacking as described above, a single computing engine block implements global data sharing through an on-chip bus, so that a single computing engine block can access other DRAM storage units across channels.

[0016] Optionally, in the aforementioned dual-channel memory addressing and bus system based on 3D stacking, the preset operating modes include a single-mode data parallel mode, a single-mode operator parallel mode, and a dual-mode parallel mode;

[0017] When in the single-mode data parallel mode, the memory controller divides the input data set into a plurality of subsets and distributes the subsets in parallel to each of the computing engine blocks, and the DRAM bank in each of the DRAM storage units stores the same weight parameters and independently processes the corresponding subset;

[0018] When in the single-mode operator parallel mode, the memory controller sequentially distributes the input data set to all the computing engine blocks of the two memory channels in a pipeline manner, and the DRAM bank in each DRAM storage unit stores weight parameters divided by model layer and sequentially processes a portion of the set amount of data in the input data set;

[0019] When in the dual-mode parallel mode, the two memory channels independently run different AI models, and each instruction scheduling engine configures and controls the instruction flow to execute different AI models.

[0020] Optionally, in the dual-channel memory addressing and bus system based on 3D stacking as described above, when in the single-mode data parallel mode, during the training process, the model parameters are synchronized through gradient aggregation and parameter update; in the inference stage, the results of each of the computing engine blocks are merged to generate the final output.

[0021] Optionally, in the aforementioned dual-channel memory addressing and bus system based on 3D stacking, when in the dual-mode parallel mode, each of the memory channels adopts one of the single-mode data parallel mode or the single-mode operator parallel mode.

[0022] Optionally, in the aforementioned dual-channel memory addressing and bus system based on 3D stacking, when in the dual-mode parallel mode, the results obtained by the two memory channels are merged to generate a final output.

[0023] The positive progress effect of the present invention is:

[0024] 1. The present invention combines logic computing clusters with DRAM through 3D stacking technology to achieve high-bandwidth, low-latency data transmission and flexible memory management.

[0025] 2. Bus and data transmission optimization: The present invention supports dual-channel simultaneous access, doubling the theoretical bandwidth.

[0026] 3. Bandwidth improvement: The dual-channel design of the present invention can achieve 2TB / s intra-chip transmission, increasing the memory bandwidth to about 70 times that of traditional architectures, solving the memory wall problem of AI models.

[0027] 4. Flexibility: This invention supports multi-task parallelism (such as vision + text processing) and is suitable for complex multimodal application scenarios.

[0028] 5. Compatibility: The present invention complies with the DDR protocol and can be directly integrated into the existing system without modifying the host hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The disclosure of the present invention will become more apparent with reference to the accompanying drawings. It should be understood that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. In the drawings:

[0030] Figure 1 A structural diagram of the present invention;

[0031] Figure 2 A structural block diagram of the single-mode data parallel mode of the present invention;

[0032] Figure 3 A structural block diagram of the single-mode operator parallel mode of the present invention;

[0033] Figure 4 This is a structural block diagram of the dual-mode parallel mode of the present invention. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present invention through specific examples. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the details in this specification may be modified or altered based on different perspectives and applications without departing from the spirit of the present invention.

[0035] It should be noted that, unless there is any conflict, the following embodiments and features therein may be combined with each other.

[0036] In the description of the present invention, it should be noted that, for directional words, such as the terms "outside", "middle", "inside", "outside", etc., the directions and positional relationships indicated are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and cannot be understood as limiting the specific scope of protection of the present invention.

[0037] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features. Therefore, the terms "first" and "second" may explicitly or implicitly refer to one or more of these features. Throughout the description of the present invention, "several" and "a number" mean two or more, unless otherwise specifically defined.

[0038] An embodiment of the present invention provides a dual-channel memory addressing and bus system based on 3D stacking, which includes at least one smart memory processing unit (SMPU). The smart memory processing unit includes a 3D stacked DRAM storage layer and a logic computing layer. The DRAM storage layer has a plurality of DRAM storage units, and the logic computing layer has a plurality of computing engine groups, a plurality of instruction scheduling engines, and a plurality of memory controllers.

[0039] The memory within the intelligent memory processing unit is divided into two independent memory channels. Each memory channel includes several groups of computing engine groups. A single computing engine group includes several computing engine blocks. A single computing engine block is connected to a DRAM storage unit, an instruction scheduling engine, and a memory controller. The instruction scheduling engine is used to schedule computing tasks, and the memory controller is used to allocate DRAM storage layer resources according to different preset working modes.

[0040] The dual-channel design within each SMPU unit of this invention plays a key role in data access and model deployment. When the SMPU accesses data in dual-channel mode, the two channels collaborate in a manner similar to the dual-channel mode on a motherboard. The processors (i.e., each compute engine block) within the SMPU can simultaneously communicate with two independent memory channels, theoretically doubling the SMPU's total bandwidth and significantly improving data read and write efficiency. This parallel communication approach not only avoids data contention and bandwidth bottlenecks in single-channel mode but also optimizes system performance through load balancing. This makes it particularly suitable for high-bandwidth, low-latency applications such as high-performance computing, real-time data analysis, and image processing.

[0041] This invention divides memory into two independent memory channels, supporting simultaneous dual-channel access, significantly increasing memory bandwidth and resolving the memory wall issue in AI models. This system architecture complies with the DDR protocol and can be directly integrated into existing systems without modifying host hardware.

[0042] In some embodiments, copper interconnection is achieved between the DRAM storage layer and the logic computing layer through hybrid bonding (or HybridBonding) to achieve 2TB / s intra-chip transmission.

[0043] In some embodiments, a single DRAM storage unit includes several DRAM banks. The instruction scheduling engine is also used to calculate the mapping relationship between the computing engine block and the DRAM bank.

[0044] A single DRAM storage unit preferably includes four 4MB DRAM banks. In this case, the memory space of each memory channel is subdivided into several physical blocks, each of which contains four 4MB DRAM banks, supporting fine-grained address allocation, such as grouping four 4MB spaces.

[0045] In some embodiments, a single DRAM storage unit includes four DRAM banks, and a single DRAM bank group includes 32 DRAM banks. The instruction scheduling engine is further configured to calculate a mapping relationship between a computing engine block and a DRAM bank.

[0046] At this time, the memory space of each memory channel is subdivided into several physical blocks, each of which contains 32 DRAM banks, supporting fine-grained address allocation, such as dividing the space into four groups of 4MB.

[0047] In some embodiments, each memory channel includes four compute engine groups, and a single compute engine group includes two compute engine blocks.

[0048] At this time, the memory space of each memory channel is subdivided into 8 physical blocks to correspond to 8 computing engine blocks, and each physical block contains several DRAM banks.

[0049] In some embodiments, each memory channel includes four compute engine groups, and a single compute engine group includes two compute engine blocks.

[0050] When a single DRAM memory cell includes four 4MB DRAM banks, refer to Figure 1 The memory within each SMPU unit is divided into two independent memory channels, memory channel 0 and memory channel 1. Each memory channel includes four groups of eight compute engine blocks, each of which physically corresponds to a DRAM storage unit. Each DRAM storage unit is further divided into four different DRAM banks. Each DRAM bank is subdivided into four groups of 4MB of memory address space, enabling more flexible parameter deployment and memory allocation. From memory space allocation to model deployment, the two memory channels can operate independently or work together.

[0051] When a single DRAM storage unit includes 32 DRAM banks, each memory channel includes 8 computing engine blocks and corresponding 8 DRAM storage units. A single DRAM storage unit has 4 DRAM bank groups, and a single DRAM bank group includes 32 DRAM banks. Therefore, each memory channel includes 8×32 DRAM banks. Taking this as an example, at an operating frequency of 500MHz, the data IO of each DRAM bank is 128 bits, or 16B, and the bandwidth is 0.5GHz×16B×8×32, which equals 2TB / S. The traditional LPDDR4 bandwidth is 29.864GB / S, so 2TB / S divided by 29.864GB / S is approximately 70. Therefore, this design of the present invention can increase the memory bandwidth to approximately 70 times that of the traditional architecture.

[0052] In some embodiments, reference Figure 1 ,Each memory channel shares the same instruction scheduling engine, and each group of computing engines shares the same memory controller.

[0053] In other words, each SMPU has two memory channels, each with an instruction dispatch engine responsible for scheduling computing tasks for all compute engine blocks within that memory channel. The two compute engine blocks, as a group, share the same memory controller, dynamically allocating DRAM bank resources based on the selected mode (data / operator / dual mode).

[0054] In some embodiments, a single computing engine block implements global data sharing through an on-chip bus, so that the single computing engine block can access other DRAM storage units across channels.

[0055] In some embodiments, the preset operating modes include a single-modal model data parallelism mode, a single-modal model operator parallelism mode, and a dual-modal model parallelism mode.

[0056] The preset working mode is usually issued by the host chip, which sends a serial memory command to the SMPU unit. The serial memory command includes a mode selection. The memory controller of the SMPU unit supports dynamic switching of the working mode to single-mode data parallel mode, single-mode operator parallel mode or dual-mode parallel mode according to the serial memory command.

[0057] When in single-mode data parallel mode, the memory controller divides the input data set into several subsets and distributes them in parallel to each computing engine block. The DRAM bank in each DRAM storage unit stores the same weight parameters and processes its corresponding subset independently.

[0058] When in single-mode operator parallel mode, the memory controller distributes the input data set sequentially to all computing engine blocks of the two memory channels in a pipeline manner. The DRAM bank in each DRAM storage unit stores weight parameters divided by model layer and sequentially processes a portion of the set amount of data in the input data set.

[0059] When in dual-mode parallel mode, the two memory channels independently run different AI models, and each instruction scheduling engine configures and controls the instruction flow to execute different AI models.

[0060] In some embodiments, reference Figures 2 to 4 The two memory channels are memory channel 0 and memory channel 1. Each memory channel has 8 computing engine blocks, of which 4 computing engine blocks (computing engine block 0 to computing engine block 3) are directly involved in data processing. Therefore, the 4 computing engine blocks correspond to 4 DRAM storage units (BANK0 to BANK3), and each DRAM storage unit is further subdivided into 4 different DRAM banks (UNIT0 to UNIT3).

[0061] Reference Figure 2 In single-mode data parallel mode, the memory controller of the SMPU unit divides the input data set into several subsets and distributes them in parallel to all 32 DRAM banks. Each DRAM bank stores the same weight parameters and processes its corresponding subset independently.

[0062] In single-mode data parallel mode, during training, model parameters are synchronized through gradient aggregation and parameter updates. During inference, the results of each compute engine block are merged to generate the final output. Therefore, each memory channel also indirectly participates in data processing using another compute engine block (compute engine block 4 / compute engine block 5), merging the results of each compute engine block within the same memory channel.

[0063] Reference Figure 3In single-mode operator parallel mode, the SMPU unit's memory controller sequentially distributes the input data set to all 32 DRAM banks in the two memory channels. The input data set is processed in a pipeline manner, with each DRAM bank sequentially calculating a set amount of data in the input data set. The calculation results of each DRAM bank are passed to the next DRAM bank corresponding to the next model level for further calculation, forming a continuous calculation pipeline.

[0064] Reference Figure 4 , a dual-mode parallel mode, with two memory channels operating completely independently. Different memory channels can simultaneously handle different application scenarios for the same input dataset. For example, for image input, one memory channel (Memory Channel 0) can perform image recognition, while another memory channel (Memory Channel 1) can perform semantic analysis of the image and output a text description. Another example: one memory channel (Memory Channel 1) can handle speech-to-text conversion, while the other memory channel (Memory Channel 0) can perform semantic analysis on the generated text and output inference results.

[0065] In dual-mode parallel mode, each memory channel uses either single-mode data parallel mode or single-mode operator parallel mode. Figure 4 As shown in Figure 1, single-mode data parallelism is used when executing the CNN model in memory channel 0, and single-mode operator parallelism is used when executing the Transformer model in memory channel 1. At this time, each memory channel also uses compute engine block 4 and any one compute engine block 5 to merge the results obtained from the two memory channels to generate the final output.

[0066] The present invention has been described in detail above with reference to the embodiments of the accompanying drawings. A person skilled in the art can make various modifications to the present invention based on the above description. Therefore, certain details in the embodiments should not be construed as limiting the present invention. The scope of protection of the present invention shall be determined by the scope defined by the appended claims.

Claims

1. A dual-channel memory addressing and bus system based on 3D stacking, characterized in that: The 3D stacking-based dual-channel memory addressing and bus system includes at least one intelligent memory processing unit, the intelligent memory processing unit includes a 3D stacked DRAM storage layer and a logic computing layer, the DRAM storage layer has a plurality of DRAM storage units, the logic computing layer has a plurality of computing engine groups, a plurality of instruction scheduling engines and a plurality of memory controllers; The memory in the intelligent memory processing unit is divided into two independent memory channels, each memory channel includes several groups of computing engine groups, each computing engine group includes several computing engine blocks, and each computing engine block is connected to one DRAM storage unit, one instruction scheduling engine, and one memory controller. The instruction scheduling engine is used to schedule computing tasks, and the memory controller is used to allocate DRAM storage layer resources according to different preset working modes. The preset working modes include single-mode data parallel mode, single-mode operator parallel mode and dual-mode parallel mode; When in the single-mode data parallel mode, the memory controller divides the input data set into a plurality of subsets and distributes the subsets in parallel to each of the computing engine blocks, and the DRAM banks in each of the DRAM storage units store the same weight parameters and independently process the corresponding subsets; When in the single-mode operator parallel mode, the memory controller sequentially distributes the input data set to all the computing engine blocks of the two memory channels in a pipeline manner, and the DRAM bank in each DRAM storage unit stores weight parameters divided by model layer and sequentially processes a portion of the set amount of data in the input data set; When in the dual-mode parallel mode, the two memory channels independently run different AI models, and each instruction scheduling engine configures and controls the instruction flow to execute different AI models.

2. The dual-channel memory addressing and bus system based on 3D stacking according to claim 1, wherein: The DRAM storage layer and the logic computing layer are interconnected by copper through hybrid bonding.

3. The 3D stacking-based dual-channel memory addressing and bus system according to claim 1, wherein: A single DRAM storage unit includes a plurality of DRAM banks; The instruction scheduling engine is further configured to calculate a mapping relationship between the computing engine block and the DRAM bank.

4. The 3D stacking-based dual-channel memory addressing and bus system according to claim 3, wherein: A single DRAM storage unit includes four 4MB DRAM banks; Alternatively, a single DRAM storage unit includes 4 DRAM bank groups, and a single DRAM bank group includes 32 DRAM banks.

5. The dual-channel memory addressing and bus system based on 3D stacking according to claim 1, wherein: Each of the memory channels includes four computing engine groups, and a single computing engine group includes two computing engine blocks.

6. The 3D stacking-based dual-channel memory addressing and bus system according to claim 5, wherein: The number of the DRAM storage units is the same as the number of the computing engine blocks, and a single DRAM storage unit includes four 4MB DRAM banks; The instruction scheduling engine is further used to calculate the mapping relationship between the computing engine block and the DRAM bank; Or, the number of the DRAM storage units is the same as the number of the computing engine blocks, a single DRAM storage unit includes 4 DRAM bank groups, and a single DRAM bank group includes 32 DRAM banks; The instruction scheduling engine is further configured to calculate a mapping relationship between the computing engine block and the DRAM bank.

7. The dual-channel memory addressing and bus system based on 3D stacking according to claim 1, wherein: Each of the memory channels shares the same instruction scheduling engine, and each group of computing engines shares the same memory controller.

8. The dual-channel memory addressing and bus system based on 3D stacking according to claim 1, wherein: A single computing engine block implements global data sharing via an on-chip bus, so that the single computing engine block can access other DRAM storage units across channels.

9. The 3D stacking-based dual-channel memory addressing and bus system according to any one of claims 1 to 8, wherein: When in the single-mode data parallel mode, during the training process, the model parameters are synchronized through gradient aggregation and parameter update; In the inference phase, the results of each of the computational engine blocks are combined to generate the final output; and / or, when in the dual-mode parallel mode, each of the memory channels adopts one of the single-mode data parallel mode or the single-mode operator parallel mode; And / or, when in the dual-mode parallel mode, the results obtained by the two memory channels are combined to generate the final output.

Images