Servicing large language models with 3D-DRAM chiplets

Abstract

The disclosed systems and methods herein use a plurality of compute-memory chiplets to provide high bandwidth processing. A compute package may be configured with a plurality of the compute-memory chiplets to perform processing operations associated with a large language model. The compute-memory chiplet may be configured to operate using a small low-power compute die that may operate efficiently for workloads with low arithmetic strength.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of U.S. patent application No. 18 / 210,846 filed on June 16, 2023, which claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 505,725 filed on June 2, 2023, and the disclosure of the U.S. Provisional Patent Application is hereby incorporated herein by reference. Background Art

[0003] Compute packages with multiple high bandwidth memory ("HBM") die can be used to perform high performance computing. However, these packages have been configured to operate applications involving high arithmetic intensity by using high power compute die. However, the high arithmetic intensity and high power design of current packages make them unsuitable for use as machine learning accelerators for large language models. Summary of the invention

[0004] What is needed is a high bandwidth memory package design that is configured to operate efficiently as a machine learning accelerator for applications with lower arithmetic intensity, such as for serving large language models (LLMs) or other models that are limited by memory bandwidth. Various aspects of the present disclosure are directed to a machine learning accelerator having a 3D memory die implemented with multiple chiplets to increase bandwidth. The machine learning accelerator is further designed for low arithmetic intensity to reduce the power required for operation of the package. In addition, various aspects of the present disclosure allow the machine learning accelerator to operate on a package that is limited in thermal constraints and in the amount of space available for compute and memory components.

[0005] According to aspects of the present disclosure, a computing package may include: a packaging substrate; and one or more computing clusters, wherein the one or more computing clusters are located on the packaging substrate; wherein each of the one or more computing clusters includes multiple computing-memory stacks that communicate with an input-output tube core; wherein each computing-memory stack may include multiple memory tube cores stacked together with a low-power computing tube core; and wherein the input-output tube core may be configured to transmit data of the multiple computing-memory stacks through one or more peripheral component interconnects.

[0006] In other aspects of the present disclosure, each low-power compute die of the plurality of compute-memory stacks may be configured to operate at a power supply of approximately 40 W or less.

[0007] In still other aspects, for a particular compute-memory stack among the plurality of compute-memory stacks, the low-power compute die occupies an area on the packaging substrate that is less than 30% larger than an area occupied by the plurality of memory dies.

[0008] In some other aspects of the present disclosure, the computing package may also include: multiple computing clusters, the multiple computing clusters are located on the packaging substrate, and the input-output tube core of each computing cluster is connected to one or more input-output tube cores of other computing clusters located on the packaging substrate.

[0009] In some other aspects of the present disclosure, the packaging substrate may include four computing clusters, and each computing cluster includes four or more computing-memory stacks. In addition, each computing cluster may include at least one inactive spare computing-memory stack.

[0010] In other aspects of the present disclosure, the packaging substrate may include two computing clusters, and each computing cluster includes eight or more computing-memory stacks. In addition, each computing cluster may include at least one inactive spare computing-memory stack.

[0011] In still other aspects of the present disclosure, the plurality of compute-memory stacks are stacked 3D-DRAM chiplets, and the compute package is configured to operate as a large model processing unit.

[0012] In still other aspects of the present disclosure, the input-output die may be further configured to communicate with at least one of an external DRAM or an external remote direct memory access (RDMA) to interconnect with other computing packages.

[0013] In other aspects of the present disclosure, a computing method may include: receiving processing commands at one or more computing clusters located on a packaging substrate; performing computing operations based on the processing commands using multiple computing-memory stacks that communicate with input-output tube cores, wherein each computing-memory stack includes multiple memory tube cores stacked together with a low-power computing tube core; and transmitting data from the input-output tube core through one or more peripheral component interconnects.

[0014] In yet another aspect of the disclosure, each low-power compute die of the plurality of compute-memory stacks may be configured to operate at a power supply of approximately 40 W or less.

[0015] In still other aspects of the present disclosure, for a particular compute-memory stack among the plurality of compute-memory stacks, an area occupied by the low-power compute die on the packaging substrate may be less than 30% larger than an area occupied by the plurality of memory dies.

[0016] In some other aspects of the present disclosure, performing the computing operation may also include: using multiple computing clusters located on the packaging substrate to perform the computing operation, and wherein the input-output tube core of each computing cluster is connected to one or more input-output tube cores of other computing clusters located on the packaging substrate.

[0017] In other aspects of the present disclosure, the input-output die may be further configured to communicate with at least one of external DRAM or external remote direct memory access (RDMA) to interconnect with other computing packages.

[0018] According to still other aspects of the present disclosure, a large model processing unit may include one or more computing packages connected by peripheral component interconnects, and each computing package may include: a packaging substrate; and one or more computing clusters, wherein the one or more computing clusters are located on the packaging substrate; wherein each of the one or more computing clusters includes multiple computing-memory stacks that communicate with input-output tube cores; wherein each computing-memory stack includes multiple memory tube cores stacked together with low-power computing tube cores; and wherein the input-output tube cores are configured to transmit data of the multiple computing-memory stacks through the one or more peripheral component interconnects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of a compute package with a compute-memory chiplet in accordance with aspects of the present disclosure.

[0020] Figure 2 is a block diagram of a partial side view of a compute package with a compute-memory chiplet in accordance with aspects of the present disclosure.

[0021] Figure 3 Block diagram of a compute package without a compute-memory chiplet.

[0022] Figure 4 is a block diagram of a compute package with a cluster of two compute-memory chiplets in accordance with aspects of the present disclosure.

[0023] Figure 5 is a block diagram of a compute package with a memory stack cluster in communication with compute and IO dies in accordance with aspects of the present disclosure.

[0024] Figure 6 is a flow chart of an example process for performing computing operations according to aspects of the present disclosure. DETAILED DESCRIPTION

[0025] The present technology generally relates to high bandwidth processing in low arithmetic intensity environments. For example, systems and methods for large model processing are disclosed, which may include using a machine learning accelerator to serve a large language model (LLM). Large model processing may refer to the ability of one or more machine learning models to achieve general understanding and / or generation by training based on large amounts of data. An example large model processing unit may include an LLM. A machine learning accelerator may include multiple chiplets, each having a 3D memory die. The machine learning accelerator may be designed to exploit the low arithmetic intensity associated with LLM processing by including multiple low-power and small computing dies in each of the multiple chiplets.

[0026] According to various implementations, the disclosed systems and methods use multiple compute-memory chiplets to provide high-bandwidth processing. According to one aspect, a compute package is configured with multiple compute-memory chiplets to perform processing operations associated with large language models. According to another aspect, the compute-memory chiplets are configured to operate using a small, low-power compute die that can operate efficiently for workloads with low arithmetic intensity.

[0027] Figure 1 1 is a block diagram 100 of a multi-chip package 101 operable as a machine learning accelerator in accordance with aspects of the present disclosure. The multi-chip package 101 may include a substrate 110 on which a plurality of computing clusters 102a-102d are located. Each of the clusters 102a-102d may include an input-output (IO) die 108a-108d and a plurality of computing-memory chiplets 104a-104e. Each chiplet 104 may include a 3D stack of a computing die 120 and a plurality of memory die 122. For example, as shown in the block diagram 100, a chiplet 104a of cluster 102a has a computing die 120 stacked with a plurality of memory die 122. Other chiplets 104b-104e of cluster 102a and chiplets 104a-104e of other clusters 102b-102d may include a similar set of memory die 122 stacked onto the computing die 120.

[0028] Each chiplet 104a-104e within a cluster 102 is connected to a corresponding IO die 108 within the cluster 102. For example, as shown in block diagram 100, chiplet 104a is connected to IO die 108a via connection 131. Similar connections exist between IO die 108a and each of the other chiplets 104b-104e of cluster 102a. In addition, clusters 102a-102d of multi-chip package 101 can be connected to each other via IO die 108a-108d. For example, each IO die 108a-108d can be connected to each other IO die 108a-108d via connection 130. For simplicity, only some of the connections 130 of block diagram 100 are identified with reference numerals. Each chiplet 104a-104e within cluster 102 can also be configured to communicate directly with each other via additional connections (not shown).

[0029] The IO die 108a-108d of each cluster 102a-102d may also be configured to communicate with components or devices external to the multi-chip package 101. For example, the connections 132a-132d may be configured so that the IO die 108a-108d may communicate with an external dynamic random access memory (DRAM) or an external inter-core interconnect (ICI). In addition, the IO die 108a-108d may each be configured to transmit data via a peripheral component interconnect, such as a peripheral component interconnect express (PCIe) connection 134a-134d. According to various implementations, the input-output die is configured to transmit data of multiple compute-memory stacks via one or more peripheral component interconnects, thereby allowing efficient data communication between multi-chip packages, so that a compute package including one or more multi-chip packages can operate efficiently as a machine learning accelerator. As an example of a peripheral component interconnect and as Figure 1 Each PCIe connection 134a-134d shown can be configured to transmit data for all chiplets 104a-104e within a given cluster. Thus, multi-chip package 101 can be configured to communicate with other multi-chip packages and to operate as a machine learning accelerator as part of a larger language model server.

[0030] Figure 22 is a block diagram 200 of a partial side view of a multi-chip package 101 showing a compute die 120 and a memory die 122 of a chiplet 104a-104c. Each chiplet 104a-104c includes a plurality of memory die 122 stacked onto the compute die 120. Each compute die 120 may be directly connected to a substrate 110, such as a circuit board, via an electrical connection 112. The plurality of memory die 122 may be a 3D stack of HBM die that have been stacked onto the compute die 120, and each memory die 122 may be electrically connected to an adjacent memory die 122 within the stack to allow HBM electrical signals to pass through the chiplet 104. As shown in the block diagram 200, the compute die 120 and the memory die 122 may be configured to include one or more internal through silicon vias (TSVs 210) that extend through the stack of chiplets within the chiplet 104. Electrical connections between the compute die 120 and the plurality of memory die may exist through the TSVs 210. Although each chiplet 104a-104c is shown as containing four memory dies 122, the number of memory dies 122 on a given chiplet can vary based on the capacity required for a given application and based on the total number of chiplets 104 included on the package 101. Specifically, when a large number of chiplets are used per package 101, fewer memory dies 122 may be required per chiplet. Thus, increasing the number of chiplets 104 can reduce the height of the package 101 and can reduce thermal constraints and cooling requirements that would occur when using larger stacks of memory dies 122.

[0031] Because each chiplet 104 contains its own compute die 120, the chiplet 104 can be designed to have a smaller compute die than would be required if only a single compute die were used for all memory dies 122 within a cluster or package 101. According to aspects of the present disclosure, the compute die 120 of each chiplet 104 can be designed to have the same or similar footprint as the memory die 122. For example, in Figure 200, the compute die 120 is shown as having the same footprint as the memory die 122 stacked therewith. Alternatively, the footprint of the compute die 120 on the package substrate may be less than 30% larger than the footprint of the memory die 122 stacked therewith. Thus, the package 101 may be able to accommodate a large number of small, low-power chiplets 104. In addition, the stacked configuration of the compute die 120 and the memory die 122 allows the chiplets 104 of the package 101 to be directly connected to the substrate 110 without the use of an interposer.

[0032] For example, Figure 3 Is not used combination Figure 1 and Figure 2Block diagram 300 of package 301 for a compute-memory chiplet 104 is depicted. Specifically, a compute die 320 and a memory stack 310 are each connected to an interposer 340, which is itself connected to a substrate 330 via electrical connections 352. The compute die 320 communicates with multiple memory dies 322 within the memory stack 310 via the interposer 340. Thus, the number, size, and placement of the compute dies 320 and memory dies 322 are limited by the interposer 340 used. Furthermore, the package 301 is configured such that a single compute die 320 performs all processing for the entire package 301, and a single compute die 320 needs to access all ten memory dies 322 within the memory stack 310. In this configuration, the compute die 320 has a footprint that is twice the size of the memory die 322.

[0033] Back to Figure 1 , package 101 may have dimensions that allow it to fit within other packages on a pre-existing substrate. The specific numbers described herein are exemplary only and are not intended to be limiting, however, by way of example only, the length and width of package 101 relative to the substrate may be approximately 60 mm x 60 mm, and computing die 120 may each have a width of approximately 100 mm. 2 The footprint of the IO die 108 may be approximately 150 mm each. 2 The memory die 122 may be a 3D stacked dynamic random access memory (DRAM) die with a bandwidth of about 800 GB / s, a capacity of about 4 GB, and a thermal design power (TDP) of about 10 W per chip. The computing die 120 may each have a thermal design power of about 40 W or less, a processing capability of about 32 TFLOPs, and an SRAM of about 32 MB. The IO die 108 a-108 d may have a thermal design power of about 35 W TDP.

[0034] The package 101 may enable high bandwidth communication between the IO die 108. As an example, the total IO bandwidth of each IO die 108 may be up to several TB / s, such as up to 4 TB / s. For example, 400 GB / s of bandwidth may be dedicated to each compute die 120 within the cluster 102, and 400 GB / s of bandwidth may be dedicated to each IO die 108 within the package 101. As another example, if high bandwidth IO die communication is not required, the total IO bandwidth of each IO die 108 may be lower, such as about 400 GB / s. For example, for each IO die 108, 160 GB / s of bandwidth may be dedicated to the compute die 120 within the cluster 102, 150 GB / s of bandwidth may be dedicated to other IO die 108 within the package 101, 16 GB / s of bandwidth may be dedicated to the host such as a host device supporting PCIe Gen5, 64 GB / s of bandwidth may be dedicated to external DRAM, and 50 GB / s of bandwidth may be dedicated to external RDMA. The IO die 108 may contain lightweight computations for intelligent routing such as aggregation of partial sums computed by a memory-compute stack connected to the IO die 108, and may be connected to external components for additional processing.

[0035] According to aspects of the present disclosure, each cluster 102 of the package 101 can be configured such that at least one chiplet 104 within the cluster 102 is at least initially designated as a cold spare. For example, of the five chiplets 104a-104e in the cluster 102a, four chiplets 104a-104d can be designated as active, while the fifth chiplet 104e is designated as an inactive spare. Thus, the IO die 108 will only communicate with the chiplets 104a-104d for processing operations of the machine learning accelerator. However, the IO die can also be configured to receive and transmit diagnostic information about the operation of each of the active chiplets 104a-104d, and replace any faulty chiplet 104a-104d with a spare chiplet 104e. For example, if it is determined that active chiplet 104a has experienced a failure or is otherwise not operating correctly, chiplet 104a may be redesignated from an active chiplet to an inactive chiplet, and spare chiplet 104e may be redesignated from an inactive spare chiplet to an active chiplet. Thus, package 101 may be designed for increased reliability because a failure in any one chiplet 104 will not compromise the operation of the entire machine learning accelerator. In this configuration, package 101 would have four spare chiplets and 16 active chiplets.

[0036] In one example where each of the 16 active compute dies 120 has a TDP of 40W, each of the 16 DRAM chips has a TDP of 10W, and each of the four IP chips has a TDP of 35W, the total TDP for the entire package 101 would be 620W. However, even in the case of using low power compute dies 120 (including compute dies 120 with a TDP of approximately 10-40W), the use of multiple compute-memory chiplets allows the package 101 to perform large language model processing with an increase in total bandwidth relative to a conventional HBM configuration. This bandwidth increase depends on the number of chiplets 104 used within the package, and the small size and low power consumption of the compute die 120 allows a larger number of chiplets 104 to be used per package without the low arithmetic intensity of the large language model processing preventing the small low power compute die 120 from operating efficiently.

[0037] For large language models, the arithmetic intensity of the processing may be an order of magnitude lower than other machine learning processing, and current architectures of machine learning accelerators may overprovision this low level of arithmetic intensity by a factor of 10 or more. However, in combination with package 101, the size of compute die 120 (100 mm 2 ) and thermal design power (10-40W) allow the use of a larger number of compute-memory chiplets 104. This large number of compute-memory chiplets 104 increases the available bandwidth for low arithmetic intensity applications without increasing operating costs relative to conventional machine learning accelerators.

[0038] According to aspects of the present disclosure, package 101 may be configured to operate in a manner that performs multiple levels of sharding when assigning processing to individual chiplets 104. For example, if a large language model is sharded to 16 GB in conjunction with a conventional machine learning accelerator, the presently disclosed system may be configured to perform the same sharding algorithm, where each compute-memory stack is a shard, except that the memory capacity of each shard (4 GB) is smaller than the shard size (16 GB) used by the conventional machine learning accelerator described above.

[0039] According to aspects of the present disclosure, a machine learning accelerator may utilize one or more packages having different sizes and having compute-memory chiplets and other components having different specifications than the above components. For example, Figure 4 4 is a block diagram 400 of another example package 401 that may be implemented as part of a machine learning accelerator for a large language model. Package 401 may be configured to operate in a similar manner to package 101 described above. However, as Figure 4 As shown, package 401 includes two computing clusters 402a-402b. Each cluster 402 includes an IO die 408 and nine chiplets 404a-404i.

[0040] Each chiplet 404a-404e within a cluster 402a-402b can be connected to a corresponding IO die 408a-408b within the cluster 402a-402b, respectively. For example, as shown in block diagram 400, chiplet 404a is connected to IO die 408a via connection 431. Similar connections exist between IO die 408a and each of the other chiplets 404b-404i of cluster 402a. In addition, clusters 402a-402b of multi-chip package 401 can be connected to each other via IO die 408a and 408b. For example, IO die 408a and 408b can be connected via connection 430. Each chiplet 404a-404i within cluster 402 can also be configured to communicate directly with each other via additional connections (not shown).

[0041] The IO die 408a and 408b of each cluster 402a and 402b can also be configured to communicate with components or devices external to the multi-chip package 401. For example, the connections 432a-432b can be configured so that the IO die 408a-408b can communicate with an external dynamic random access memory (DRAM) or an external inter-core interconnect (ICI). In addition, the IO die 408a-408b can each be configured to transmit data via a peripheral component interconnect express (PCIe) connection 434a-434b. Figure 4 Each PCIe connection 434a-434b shown can be configured to transmit data for all chiplets 104a-104i within a given cluster. Thus, multi-chip package 401 can be configured to communicate with other multi-chip packages and to operate as a machine learning accelerator as part of a larger language model server.

[0042] The architecture of package 401 may be such that it includes a different number of spares than the example described above for package 101, which provides four spare chiplets 104 (one per cluster) and 16 active chiplets 104 (four per cluster). For example, package 401 may include one spare for each cluster 402a and 402b. Thus, at any given moment, package 401 may operate using 16 active chiplets 404 and 4 inactive spare chiplets 404.

[0043] The specific numbers described herein are exemplary and not intended to be limiting, however, by way of example only, package 401 may include 3D stacked dynamic random access memory (DRAM) dies having a bandwidth of approximately 800 GB / s, a capacity of 4 GB, and a TDP of 10 W per DRAM stack. Package 401 may also include a 100 mm footprint per die. 2, a computing die with a TDP of 20W, a processing power of 32TFLOPs, and an SRAM of 32MB. The area occupied by the IO die 408a-408b may be approximately 150mm 2 , and the TDP of each die is 50W. The total IO bandwidth of each IO die 408 can be about 450GB / s, for example, 144GB / s for the compute die, 128GB / s for other IO die, 32GB / s for the host such as PCIe Gen5, 64GB / s for external DRAM, and / or 50GB / s for external RDMA. The overall TDP of package 401 can reach about 580W, where the compute die uses a total of about 320W, the memory die uses about 160W, and the IO die uses about 100W.

[0044] Figure 5 5 is a block diagram 500 of another package 501 that can operate as a machine learning accelerator. For package 501, clusters 502a-502d include memory stacks 504a-504e, which include multiple memory tube cores 522, but do not include a computing tube core. Instead, the computing and input-output operations are performed by the computing and IO tube cores 508a-508d. In addition to the substrate 510, the package 501 also includes an interposer 550a-550d for each of the clusters 502a-502d. Therefore, each of the clusters 502a-502d may include an interposer 550a-550d connected to the substrate 510. Then, the computing and IO tube core 508 and the multiple memory stacks 504a-504e are connected to one of the interposers 550a-550d.

[0045] Each memory stack 504a-504e within the cluster 502 is connected to a corresponding compute and IO die 508 within the cluster 502. For example, as shown in the block diagram 500, the memory stack 504a is connected to the IO die 508a via a connection 531. A similar connection exists between the compute and IO die 508a and each of the other memory stacks 504b-504e of the cluster 502a. In addition, the clusters 502a-502d of the package 501 can be connected to each other via the compute and IO die 508a-508d. For example, each compute and IO die 508a-508d can be connected to each other compute and IO die 508a-508d via a connection 530. For simplicity, only some of the connections 530 of the block diagram 500 are identified with reference numerals.

[0046] The compute IO die 508a-508d of each cluster 502a-502d may also be configured to communicate with components or devices external to the package 501. For example, the connections 532a-d may be configured such that the compute and IO die 508a-508d may communicate with external dynamic random access memory (DRAM) or external inter-core interconnect (ICI). In addition, the compute die and IO die 508a-508d may each be configured to communicate data via peripheral component interconnect express (PCIe) connections 534a-534d. Figure 5 Each PCIe connection 134a-134d shown may be configured to transfer data for all memory stacks 504a-504e within a given cluster 502. Thus, multi-chip package 501 may be configured to communicate with other multi-chip packages and to operate as a machine learning accelerator as part of a larger language model server.

[0047] The specific numbers described herein are exemplary only and are not intended to be limiting, however, by way of example only, the length and width of package 501 relative to the substrate may be approximately 80 mm x 80 mm, and the computing and IO dies 508a-508d may each have a width of approximately 200-300 mm. 2 The interposer layers 550a-550d may each have a footprint of approximately 1000 mm 2 The memory die 122 may be a 3D stacked dynamic random access memory (DRAM) die having a bandwidth of about 800 GB / s, a capacity of about 16 GB, and a TDP of about 35 W for each stack. The compute die and IO die 508 a-508 d may each have a TDP of about 150 W, a processing power of about 128 TFLOPs, and about 128 MB of SRAM. The IO die 108 a-108 d may have a thermal design power of about 35 W TDP. In addition, each cluster may have four active memory stacks and one inactive standby stack. The total TDP of the 16 active memory stacks 104 may be about 560 W, and the total TDP of the four compute and IO die may be about 600 W, while the interposer and packaging may require an additional 100 W. Therefore, in this example, the package 501 may have a total TDP of 1300 W.

[0048] Figure 6 600 is a flow chart of an example process 600 for performing computing operations. The example process 600 may be performed on a system of one or more processors in one or more locations, such as Figure 1 A multi-chip package 101 is depicted.

[0049] As shown in block 610, the multi-chip package 101 receives a processing command at one or more computing clusters located on the package substrate. The processing command may include any instructions for processing data to perform any of a variety of computing operations. For example, these computing operations may include loading data into a computing circuit, moving data to one or more processing elements of the computing circuit, processing data by one or more processing elements, and pushing data out of the computing circuit.

[0050] As shown in block 620, the multi-chip package 101 uses multiple compute-memory stacks that communicate with the input-output die to perform computing operations based on processing commands. Each compute-memory stack includes multiple memory dies stacked together with a low-power compute die. Each low-power compute die can be configured to operate at a power supply of about 40W or less. For a particular compute-memory stack, the low-power compute die may occupy less than 30% of the area on the package substrate than the area occupied by the multiple memory dies.

[0051] Performing a computing operation may also include performing the computing operation using multiple computing clusters located on the packaging substrate. The input-output tube core of each computing cluster can be connected to one or more input-output tube cores of other computing clusters located on the packaging substrate. For example, the packaging substrate may include four computing clusters, each of which includes four or more active computing-memory stacks. As another example, the packaging substrate may include two computing clusters, each of which includes eight or more active computing-memory stacks. Each computing cluster may include at least one inactive standby computing-memory stack. The multiple computing-memory stacks may be stacked 3D-DRAM chiplets.

[0052] The multi-chip package 101 transmits data from the input-output die via one or more peripheral component interconnects as shown in block 630. The input-output die may be further configured to communicate with at least one of external DRAM or external RDMA to interconnect with other computing packages.

[0053] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above may be utilized without departing from the subject matter defined by the claims, the previous description of the embodiments should be made by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the examples described herein and the provision of clauses expressed as "such as," "including," etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A computing package, characterized in that: include: Package substrate; as well as one or more computing clusters, the one or more computing clusters being located on the packaging substrate; wherein each of the one or more compute clusters includes a plurality of compute-memory stacks in communication with an input-output die; wherein each compute-memory stack comprises a plurality of memory dies stacked together with a low-power compute die; and The input-output die is configured to transmit data of the plurality of compute-memory stacks via one or more peripheral component interconnects.

2. The computing package of claim 1, wherein: Each low-power compute die of the plurality of compute-memory stacks is configured to operate at a power supply of approximately 40 W or less.

3. The computing package of claim 1, wherein: For a particular compute-memory stack of the plurality of compute-memory stacks, the low-power compute die has an area on the packaging substrate that is less than 30% larger than an area that is occupied by the plurality of memory dies.

4. The computing package of claim 1, wherein: Also includes: A plurality of computing clusters are located on the packaging substrate, and wherein the input-output die of each computing cluster is connected to one or more input-output dies of other computing clusters located on the packaging substrate.

5. The computing package of claim 4, wherein: The packaging substrate includes four compute clusters, and wherein each compute cluster contains four or more compute-memory stacks.

6. The computing package of claim 4, wherein: The packaging substrate includes two compute clusters, and wherein each compute cluster contains eight or more compute-memory stacks.

7. The computing package of claim 1, wherein: Each compute cluster contains at least one inactive standby compute-memory stack.

8. The computing package of claim 1, wherein: The multiple compute-memory stacks are stacked 3D-DRAM chiplets.

9. The computing package of claim 1, wherein: The computational package is configured to operate as a large model processing unit.

10. The computing package of claim 1, wherein: The input-output die is further configured to communicate with at least one of external DRAM or external remote direct memory access (RDMA) to interconnect with other computing packages.

11. A method of calculation, characterized in that include: receiving a processing command at one or more computing clusters located on a packaging substrate; performing a computing operation based on the processing command using a plurality of compute-memory stacks in communication with the input-output die, wherein each compute-memory stack includes a plurality of memory dies stacked with a low-power compute die; as well as Data from the input-output die is transmitted through one or more peripheral component interconnects.

12. The method according to claim 11, characterized in that Each low-power compute die of the plurality of compute-memory stacks is configured to operate at a power supply of approximately 40 W or less.

13. The method according to claim 11, characterized in that For a particular compute-memory stack of the plurality of compute-memory stacks, the low-power compute die has an area on the packaging substrate that is less than 30% larger than an area that is occupied by the plurality of memory dies.

14. The method according to claim 11, characterized in that Performing the computing operation further includes performing the computing operation using a plurality of computing clusters located on the packaging substrate, and wherein the input-output die of each computing cluster is connected to one or more input-output dies of other computing clusters located on the packaging substrate.

15. The method according to claim 14, characterized in that The packaging substrate includes four compute clusters, and wherein each compute cluster contains four or more active compute-memory stacks.

16. The method according to claim 14, characterized in that The packaging substrate includes two compute clusters, and wherein each compute cluster contains eight or more active compute-memory stacks.

17. The method according to claim 11, characterized in that Each compute cluster contains at least one inactive standby compute-memory stack.

18. The method according to claim 11, characterized in that The multiple compute-memory stacks are stacked 3D-DRAM chiplets.

19. The method according to claim 11, characterized in that The input-output die is further configured to communicate with at least one of external DRAM or external remote direct memory access (RDMA) to interconnect with other computing packages.

20. A large model processing unit, characterized in that: include: One or more computing packages connected via a peripheral component interconnect, each computing package comprising: a packaging substrate; and one or more computing clusters, the one or more computing clusters being located on the packaging substrate; wherein each of the one or more compute clusters includes a plurality of compute-memory stacks in communication with an input-output die; wherein each compute-memory stack comprises a plurality of memory dies stacked together with a low-power compute die; and Wherein the input-output die is configured to transmit data of the plurality of compute-memory stacks through the one or more peripheral component interconnects.

Images