Two-channel memory addressing and bus system based on 3D stacking

Abstract

The invention belongs to the technical field of semiconductors, and particularly relates to a two-channel memory addressing and bus system based on 3D stacking, which comprises at least one intelligent memory processing unit, and the intelligent memory processing unit comprises a DRAM (Dynamic Random Access Memory) storage layer and a logic calculation layer of 3D stacking; a memory in the intelligent memory processing unit is divided into two independent memory channels, each memory channel comprises a plurality of calculation engine groups, each calculation engine group comprises a plurality of calculation engine blocks, and each calculation engine block is correspondingly connected with one DRAM storage unit, one instruction scheduling engine and one memory controller. The dual-channel design is suitable for high-bandwidth and low-delay application scenes such as high-performance calculation, real-time data analysis and image processing.

Description

Technical Field

[0001] The present invention belongs to the field of semiconductor technology, and particularly relates to a dual-channel memory addressing and bus system based on 3D stacking. Background Art

[0002] Existing computing systems (such as GPU / DRAM architectures) have memory bandwidth bottlenecks. Especially in the processing of large AI models, the data transmission efficiency is low, resulting in insufficient utilization of computing resources.

[0003] In traditional DRAM-PIM architectures, due to the sharing of physical space or customized circuit design between the memory and computing units, the computing flexibility and bandwidth are limited. Specifically, traditional DRAM-PIM architectures lack a flexible memory address allocation mechanism and are difficult to support multi-modal or multi-task parallel computing. The single-channel memory access bandwidth of traditional DRAM-PIM architectures is insufficient to meet the requirements of high-concurrency data transmission such as AI models. Summary of the Invention

[0004] Aiming at the technical problems of limited computing flexibility and bandwidth in traditional DRAM-PIM architectures, the purpose of the present invention is to provide a dual-channel memory addressing and bus system based on 3D stacking.

[0005] To solve the foregoing technical problems, the present invention provides a dual-channel memory addressing and bus system based on 3D stacking. The dual-channel memory addressing and bus system based on 3D stacking includes at least one Smart Memory Processing Unit (SMPU). The Smart Memory Processing Unit includes a 3D-stacked DRAM storage layer and a logic computing layer. The DRAM storage layer has a number of DRAM storage units, and the logic computing layer has a number of groups of computing engine groups, a number of instruction scheduling engines, and a number of memory controllers.

[0006] The memory within the Smart Memory Processing Unit is divided into two independent memory channels. Each memory channel includes a number of groups of the computing engine groups. A single group of the computing engine groups includes a number of computing engine blocks. Each single computing engine block is correspondingly connected to a DRAM storage unit, an instruction scheduling engine, and a memory controller. The instruction scheduling engine is used to schedule computing tasks, and the memory controller is used to allocate resources of the DRAM storage layer according to different preset working modes.

[0007] Optionally, in the dual-channel memory addressing and bus system based on 3D stacking as described above, copper interconnection is achieved between the DRAM storage layer and the logic computing layer through hybrid bonding.

[0008] Optionally, in the 3D stack - based dual - channel memory addressing and bus system as described above, a single DRAM storage unit includes several DRAM banks. Preferably, a single DRAM storage unit includes 4 DRAM banks of 4MB each;

[0009] The instruction scheduling engine is also used to calculate the mapping relationship between the computing engine blocks and the DRAM banks.

[0010] Optionally, in the 3D stack - based dual - channel memory addressing and bus system as described above, a single DRAM storage unit includes 4 DRAM bank groups, and each DRAM bank group includes 32 DRAM banks;

[0011] The instruction scheduling engine is also used to calculate the mapping relationship between the computing engine blocks and the DRAM banks.

[0012] Optionally, in the 3D stack - based dual - channel memory addressing and bus system as described above, each memory channel includes four groups of computing engine groups, and a single group of computing engine groups includes two computing engine blocks.

[0013] Optionally, in the 3D stack - based dual - channel memory addressing and bus system as described above, the number of DRAM storage units is the same as the number of computing engine blocks.

[0014] Optionally, in the 3D stack - based dual - channel memory addressing and bus system as described above, each memory channel shares the same instruction scheduling engine, and each group of computing engine groups shares the same memory controller.

[0015] Optionally, in the 3D stack - based dual - channel memory addressing and bus system as described above, a single computing engine block realizes global data sharing through an on - chip bus, so that a single computing engine block can access other DRAM storage units across channels.

[0016] Optionally, in the 3D stack - based dual - channel memory addressing and bus system as described above, the preset working modes include a single - mode data parallel mode, a single - mode operator parallel mode, and a dual - mode parallel mode;

[0017] When in the single - mode data parallel mode, the memory controller divides the input data set into several subsets, and distributes the several subsets in parallel to each computing engine block. Each DRAM bank in each DRAM storage unit stores the same weight parameters and independently processes its corresponding subset;

[0018] When in the single-mode operator parallel mode, the memory controller sequentially distributes the input data set to all the computing engine blocks of the two memory channels in a pipelined manner, and each DRAM bank in each DRAM storage unit stores weight parameters divided by model layers and sequentially processes a part of the set amount of data in the input data set;

[0019] When in the dual-mode parallel mode, the two memory channels operate different AI models independently, and each instruction scheduling engine configures and controls the instruction stream to execute different AI models.

[0020] Optionally, in the 3D stacked-based dual-channel memory addressing and bus system as described above, when in the single-mode data parallel mode, during the training process, the synchronization of model parameters is achieved through gradient aggregation and parameter update; in the inference phase, the results of each computing engine block are merged to generate the final output.

[0021] Optionally, in the 3D stacked-based dual-channel memory addressing and bus system as described above, when in the dual-mode parallel mode, each memory channel adopts one of the single-mode data parallel mode or the single-mode operator parallel mode.

[0022] Optionally, in the 3D stacked-based dual-channel memory addressing and bus system as described above, when in the dual-mode parallel mode, the results obtained by the two memory channels are merged to generate the final output.

[0023] The positive and progressive effects of the present invention are as follows:

[0024] 1. The present invention combines 3D stacking technology with a logic computing cluster and DRAM to achieve high-bandwidth, low-latency data transmission and flexible memory management.

[0025] 2. Bus and data transmission optimization: The present invention supports simultaneous dual-channel access, doubling the theoretical bandwidth.

[0026] 3. Bandwidth improvement: The dual-channel design of the present invention can achieve an on-chip transmission of 2TB / s, increasing the memory bandwidth to about 70 times that of the traditional architecture and solving the memory wall problem of AI models.

[0027] 4. Flexibility: The present invention supports multi-task parallelism (such as vision + text processing) and is suitable for complex multi-modal application scenarios.

[0028] 5. Compatibility: The present invention complies with the DDR protocol, does not require modification of the host hardware, and can be directly integrated into the existing system. Description of the Drawings

[0029] The disclosure of the present invention will become more apparent with reference to the accompanying drawings. It should be understood that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. In the drawings:

[0030] Figure 1 is a structural block diagram of the present invention;

[0031] Figure 2 is a structural block diagram of the single-mode data parallel mode of the present invention;

[0032] Figure 3 is a structural block diagram of the single-mode operator parallel mode of the present invention;

[0033] Figure 4 is a structural block diagram of the dual-mode parallel mode of the present invention. Detailed implementation manners

[0034] The following uses specific specific examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0035] It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0036] In the description of the present invention, it should be noted that for orientation terms, such as terms "outer side", "middle section", "inner", "outer", etc., indicating orientation and position relationships are based on the orientation or position relationships shown in the drawings. This is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and should not be construed as limiting the specific protection scope of the present invention.

[0037] In addition, such terms as "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, the meanings of "several" and "a number of" are two or more, unless otherwise clearly and specifically defined.

[0038] An embodiment of the present invention provides a dual-channel memory addressing and bus system based on 3D stacking. The dual-channel memory addressing and bus system based on 3D stacking includes at least one Smart Memory Processing Unit (SMPU). The Smart Memory Processing Unit includes a 3D-stacked DRAM storage layer and a logic computing layer. The DRAM storage layer has a number of DRAM storage units, and the logic computing layer has a number of groups of computing engine groups, a number of instruction scheduling engines, and a number of memory controllers.

[0039] The memory within the Smart Memory Processing Unit is divided into two independent memory channels. Each memory channel includes a number of groups of computing engine groups. A single group of computing engine groups includes a number of computing engine blocks. Each single computing engine block is correspondingly connected to a DRAM storage unit, an instruction scheduling engine, and a memory controller. The instruction scheduling engine is used to schedule computing tasks, and the memory controller is used to allocate resources of the DRAM storage layer according to different preset working modes.

[0040] The dual-channel design within each SMPU unit of the present invention plays a key role in data access and model deployment. When the SMPU unit accesses data in dual-channel mode, the cooperation mode between the two channels is similar to the dual-channel mode on the motherboard. The processors (i.e., each computing engine block) within the SMPU can communicate with two independent memory channels simultaneously, theoretically doubling the total bandwidth of the SMPU and significantly improving the data reading and writing efficiency. This parallel communication mode not only avoids data competition and bandwidth bottlenecks in the single-channel mode but also optimizes the system performance through load balancing. This is particularly applicable to high-bandwidth and low-latency application scenarios such as high-performance computing, real-time data analysis, and image processing.

[0041] By dividing the memory into two independent memory channels, the present invention supports simultaneous dual-channel access, greatly improving the memory bandwidth and solving the memory wall problem of AI models. The above system structure of the present invention conforms to the DDR protocol and can be directly integrated into the existing system without modifying the host hardware.

[0042] In some embodiments, copper interconnection is achieved between the DRAM storage layer and the logic computing layer through hybrid bonding (or HybridBonding) to achieve the purpose of on-chip transmission at 2TB / s.

[0043] In some embodiments, a single DRAM storage unit includes a number of DRAM banks. The instruction scheduling engine is also used to calculate the mapping relationship between the computing engine blocks and the DRAM banks.

[0044] A single DRAM storage unit preferably includes 4 DRAM banks of 4MB each. At this time, the memory space of each memory channel is subdivided into several physical blocks, each physical block containing 4 DRAM banks of 4MB each, supporting fine-grained address allocation, such as dividing the 4MB space into 4 groups by group.

[0045] In some embodiments, a single DRAM storage unit includes 4 groups of DRAM banks, and a single group of DRAM banks includes 32 DRAM banks. The instruction scheduling engine is also used to calculate the mapping relationship between the computing engine blocks and the DRAM banks.

[0046] At this time, the memory space of each memory channel is subdivided into several physical blocks, each physical block containing 32 DRAM banks, supporting fine-grained address allocation, such as dividing the 4MB space into 4 groups by group.

[0047] In some embodiments, each memory channel includes four groups of computing engine groups, and a single group of computing engine groups includes two computing engine blocks.

[0048] At this time, the memory space of each memory channel is subdivided into 8 physical blocks to correspond to 8 computing engine blocks, and each physical block contains several DRAM banks.

[0049] In some embodiments, each memory channel includes four groups of computing engine groups, and a single group of computing engine groups includes two computing engine blocks.

[0050] When a single DRAM storage unit includes 4 DRAM banks of 4MB each, referring to Figure 1 , the memory within each SMPU unit is divided into two independent memory channels, namely memory channel 0 and memory channel 1. Each memory channel includes a total of 8 computing engine blocks in four groups. Each computing engine block directly corresponds to a DRAM storage unit in terms of physical structure. Each DRAM storage unit is further subdivided into 4 different DRAM banks. Each DRAM bank is subdivided into 4 groups of 4MB memory address spaces, thus achieving more flexible parameter deployment and memory allocation. From memory space allocation to model deployment, the two memory channels can operate independently or work together.

[0051] When a single DRAM memory cell includes 32 DRAM banks, each memory channel includes 8 computing engine blocks and corresponding 8 DRAM memory cells. The single DRAM memory cell has 4 DRAM bank groups, and each single DRAM bank group includes 32 DRAM banks. Then each memory channel includes 8×32 DRAM banks. Taking this as an example, at a working frequency of 500 MHz, the data I / O of each DRAM bank is 128 bits, that is, 16B, and the bandwidth is 0.5 GHz×16B×8×32, which is equal to 2 TB / S. While the bandwidth of traditional LPDDR4 is 29.864 GB / S, then 2 TB / S divided by 29.864 GB / S is approximately equal to 70. Therefore, this design of the present invention can increase the memory bandwidth to about 70 times that of the traditional architecture.

[0052] In some embodiments, referring to Figure 1 , each memory channel shares the same instruction scheduling engine, and each group of computing engine groups shares the same memory controller.

[0053] That is to say, each SMPU unit has two memory channels, and each memory channel correspondingly has an instruction scheduling engine to be responsible for the scheduling and calculation tasks of all computing engine blocks within the memory channel. Two computing engine blocks as a group of computing engine groups share the same memory controller to dynamically allocate DRAM bank resources according to the mode selection (data / operator / dual mode).

[0054] In some embodiments, a single computing engine block realizes global data sharing through an on-chip bus, so that a single computing engine block can access other DRAM memory cells across channels.

[0055] In some embodiments, the preset working modes include a single-mode data parallel mode (Single-Modal Model Data Parallelism), a single-mode operator parallel mode (Single-Modal Model Operator Parallelism), and a dual-mode parallel mode (Dual-Modal Model Parallelism).

[0056] The preset working modes are usually issued by the host chip. The host chip sends a sequence memory command to the SMPU unit, and the sequence memory command contains a mode selection. The memory controller of the SMPU unit supports dynamically switching the working mode to the single-mode data parallel mode, the single-mode operator parallel mode, or the dual-mode parallel mode according to the sequence memory command.

[0057] When in the single-mode data parallel mode, the memory controller divides the input data set into several subsets and distributes the several subsets in parallel to each computing engine block. Each DRAM bank in each DRAM storage unit stores the same weight parameter and independently processes its corresponding subset.

[0058] When in the single-mode operator parallel mode, the memory controller sequentially distributes the input data set to all computing engine blocks of two memory channels in a pipelined manner. Each DRAM bank in each DRAM storage unit stores the weight parameters divided by model layer and sequentially processes a part of the set amount of data in the input data set.

[0059] When in the dual-mode parallel mode, the two memory channels operate different AI models independently, and each instruction scheduling engine configures and controls the instruction stream to execute different AI models.

[0060] In some embodiments, referring to Figures 2 to 4 , the two memory channels are memory channel 0 and memory channel 1 respectively. Each memory channel has 8 computing engine blocks, and 4 computing engine blocks (computing engine block 0 to computing engine block 3) directly participate in data processing. Therefore, the 4 computing engine blocks respectively correspond to 4 DRAM storage units (BANK0 to BANK3), and each DRAM storage unit is further divided into 4 different DRAM banks (UNIT0 to UNIT3).

[0061] Referring to Figure 2 , for the single-mode data parallel mode, the memory controller of the SMPU unit divides the input data set into several subsets and distributes the several subsets in parallel to all 32 DRAM banks. Each DRAM bank stores the same weight parameter and independently processes its corresponding subset.

[0062] In the single-mode data parallel mode, during the training process, the synchronization of model parameters is achieved through gradient aggregation and parameter update; in the inference phase, the results of each computing engine block are merged to generate the final output. Therefore, each memory channel also indirectly participates in data processing by using another computing engine block (computing engine block 4 / computing engine block 5), that is, merging the results obtained by each computing engine block within the same memory channel.

[0063] Referring to Figure 3, it is the single-mode operator parallel mode. The memory controller of the SMPU unit sequentially distributes the input data set to all 32 DRAM banks of the two memory channels. The input data set is processed in a pipelined manner. Each DRAM bank sequentially calculates a set amount of data in the input data set, and the calculation result of each DRAM bank is passed to the next DRAM bank corresponding to the next model level for further calculation, forming a continuous calculation pipeline.

[0064] Refer to Figure 4 , it is the dual-mode parallel mode, and the two memory channels operate completely independently. For the same input data set, different memory channels can process different application scenarios simultaneously. For example, for image input, one memory channel (memory channel 0) can perform image recognition, and the other memory channel (memory channel 1) can perform image semantic analysis and output a text description. Another example is that one memory channel (memory channel 1) can process speech-to-text conversion, while the other memory channel (memory channel 0) can perform semantic analysis on the generated text and output the inference result.

[0065] In the dual-mode parallel mode, each memory channel adopts one of the single-mode data parallel mode or the single-mode operator parallel mode. For example, as Figure 4 shown, the single-mode data parallel mode is adopted when executing the CNN model in memory channel 0, and the single-mode operator parallel mode is adopted when executing the Transformer model in memory channel 1. At this time, each memory channel also uses the calculation engine block 4 and any one of the calculation engine blocks 5 to merge the results obtained from the two memory channels to generate the final output.

[0066] The present invention has been described in detail in combination with the embodiments in the accompanying drawings. Those of ordinary skill in the art can make various variations to the present invention according to the above description. Therefore, certain details in the embodiments should not constitute a limitation to the present invention, and the present invention will take the scope defined by the appended claims as the protection scope.

Claims

1. A dual-channel memory addressing and bus system based on 3D stacking, characterized in that The 3D stack-based dual-channel memory addressing and bus system includes at least one intelligent memory processing unit, and the intelligent memory processing unit includes a 3D stacked DRAM storage layer and a logic computing layer. The DRAM storage layer has a number of DRAM storage units, and the logic computing layer has a number of groups of computing engine groups, a number of instruction scheduling engines, and a number of memory controllers; The memory in the intelligent memory processing unit is divided into two independent memory channels. Each memory channel includes a number of groups of the computing engine groups. A single group of the computing engine groups includes a number of computing engine blocks. Each single computing engine block is correspondingly connected to a DRAM storage unit, an instruction scheduling engine, and a memory controller. The instruction scheduling engine is used to schedule computing tasks, and the memory controller is used to allocate resources of the DRAM storage layer according to different preset working modes.

2. The dual-channel memory addressing and bus system based on 3D stacking according to claim 1, wherein Copper interconnection is achieved between the DRAM storage layer and the logic computing layer through hybrid bonding.

3. The dual-channel memory addressing and bus system based on 3D stacking according to claim 1, characterized in that, Each single DRAM storage unit includes a number of DRAM banks; The instruction scheduling engine is also used to calculate the mapping relationship between the computing engine block and the DRAM bank.

4. The dual-channel memory addressing and bus system based on 3D stacking according to claim 3, wherein Each single DRAM storage unit includes 4 DRAM banks of 4MB; Alternatively, each single DRAM storage unit includes 4 groups of DRAM banks, and each single group of DRAM banks includes 32 DRAM banks.

5. The dual-channel memory addressing and bus system based on 3D stacking according to claim 1, wherein Each memory channel includes four groups of the computing engine groups, and a single group of the computing engine groups includes two computing engine blocks.

6. The dual-channel memory addressing and bus system based on 3D stacking as claimed in claim 5, wherein The number of DRAM storage units is the same as the number of computing engine blocks. Each single DRAM storage unit includes 4 DRAM banks of 4MB; The instruction scheduling engine is also used to calculate the mapping relationship between the computing engine block and the DRAM bank; Alternatively, the number of DRAM storage units is the same as the number of computing engine blocks. Each single DRAM storage unit includes 4 groups of DRAM banks, and each single group of DRAM banks includes 32 DRAM banks; The instruction scheduling engine is also used to calculate the mapping relationship between the computing engine block and the DRAM bank.

7. The dual-channel memory addressing and bus system based on 3D stacking according to claim 1, characterized in that, Each memory channel shares the same instruction scheduling engine, and each group of the computing engine groups shares the same memory controller.

8. The dual-channel memory addressing and bus system based on 3D stacking according to claim 1, characterized in that, Each single computing engine block realizes global data sharing through an on-chip bus, so that each single computing engine block can access other DRAM storage units across channels.

9. The dual-channel memory addressing and bus system based on 3D stacking according to any one of claims 1 to 8, characterized in that The preset working modes include a single-mode data parallel mode, a single-mode operator parallel mode, and a dual-mode parallel mode; When in the single-mode data parallel mode, the memory controller divides the input data set into several subsets and distributes the several subsets in parallel to each computing engine block. The DRAM banks in each DRAM storage unit store the same weight parameters and independently process their corresponding subsets; When in the single-mode operator parallel mode, the memory controller sequentially distributes the input data set to all the computing engine blocks of the two memory channels in a pipelined manner, and each DRAM bank in each DRAM storage unit stores weight parameters divided by model layers and sequentially processes a part of the set amount of data in the input data set; When in the dual-mode parallel mode, the two memory channels operate different AI models independently, and each instruction scheduling engine configures and controls the instruction stream to execute different AI models.

10. The dual-channel memory addressing and bus system based on 3D stacking as claimed in claim 9, wherein, When in the single-mode data parallel mode, during the training process, the synchronization of model parameters is achieved through gradient aggregation and parameter update; In the inference phase, the results of all the computing engine blocks are merged to generate the final output; And / or, when in the dual-mode parallel mode, each memory channel adopts one of the single-mode data parallel mode or the single-mode operator parallel mode; And / or, when in the dual-mode parallel mode, the results obtained by the two memory channels are merged to generate the final output.

Images