Techniques for configuring parallel processors for different application domains

Abstract

Techniques for configuring parallel processors for different application domains are disclosed. In various embodiments, the parallel processor includes a parallel processor module implemented in a first die and a memory system module implemented in a second die. The memory system module is coupled to the parallel processor module via an on-package link. The parallel processor module includes a plurality of processor cores and a plurality of cache memories. The memory system module includes a memory controller for accessing DRAM. Advantageously, the performance of the parallel processor module can be effectively customized for memory bandwidth requirements representing one or more application domains via the memory system module.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 154,286, entitled “COMPOSABLE ON-PACKAGE ARCHITECTURE FOR GRAPHICS PROCESSING UNIT DOMAIN SPECIALIZATION,” filed on February 26, 2021. The subject matter of that related application is hereby incorporated herein by reference. Technical Field

[0003] Various embodiments relate generally to computer architecture and parallel processor implementations, and more particularly to techniques for configuring parallel processors for different application domains. Background Art

[0004] As a general matter, parallel processors achieve very high levels of processing performance by executing a large number of threads in parallel across many different processing cores. As a result, parallel processors are often implemented across many different application domains, where related workloads can be decomposed into data blocks that can be processed in parallel to increase processing throughput and overall processing efficiency. In order to meet the different computing requirements across multiple target application domains, many parallel processors are designed as a single monolithic die with an "aggregate" architecture. In an aggregate architecture, the processing cores typically include dedicated hardware for each target application domain in order to increase computational throughput when processing associated workloads. For example, the processing cores included in a typical aggregate architecture include, but are not limited to, low-precision matrix arithmetic units and high-precision arithmetic units, which can increase the computational throughput of deep learning workloads and high-performance computing workloads, respectively.

[0005] One disadvantage of using a converged architecture for parallel processors is that the memory bandwidth allocation to off-chip dynamic random access memory (DRAM) is typically a tradeoff between relatively low memory bandwidth requirements for certain workloads and relatively high memory bandwidth requirements for certain other workloads. As a result, the allocated memory bandwidth is often underutilized and therefore wasted for workloads with relatively low memory bandwidth requirements, such as many high-performance computing workloads. Conversely, the allocated memory bandwidth can become a performance bottleneck for workloads with relatively high memory bandwidth requirements, such as many deep learning workloads, thereby limiting the performance gains attributable to any specialized hardware implemented for deep learning or other similar applications.

[0006] To address the relatively high memory bandwidth requirements in a converged architecture, the amount of on-chip cache memory can be increased to mitigate the relatively long access latencies associated with off-die DRAM. However, because converged architecture parallel processors are typically at or near the maximum manufacturable die size (e.g., approximately 800 mm 2 ) on a die, so increasing the amount of cache memory on the die may require removing other circuits from the parallel processor or limiting other circuits within the parallel processor. Removing or reducing circuits often, if not always, reduces overall processing performance, which is undesirable.

[0007] As previously indicated, what is needed in the art is a more efficient way to configure parallel processors for varying memory bandwidth requirements. Summary of the invention

[0008] An embodiment of the present invention provides a parallel processor, which includes: a parallel processor module implemented in a first die, wherein the parallel processor module includes multiple processor cores and multiple cache memories; and a memory system module implemented in a second die and coupled to the parallel processor module via an on-package link, wherein the memory system module includes a first memory controller for accessing a first dynamic random access memory (DRAM).

[0009] At least one technical advantage of the disclosed technology over the prior art is that, using the disclosed technology, the performance of parallel processor modules implemented on a single die can be effectively customized for different memory bandwidth requirements representing different application domains. Specifically, for workloads with relatively high memory bandwidth requirements, a parallel processor can be configured with a processor module that can access additional cache memory via one or more on-package, ultra-high bandwidth links, which can reduce DRAM access delays and memory bandwidth bottlenecks typically experienced by aggregated architecture parallel processors. Further, because the additional cache memory is implemented on a separate die, there is no need to remove circuits from the processor module or limit circuits within the processor module to make room for the additional cache memory. These technical advantages provide one or more technical improvements over prior art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to understand the relevant features of the various embodiments described above in more detail, the above briefly summarized inventive concept may be described in more detail by reference to various embodiments (some of which have been illustrated in the accompanying drawings). However, it should be noted that the attached drawings only illustrate typical embodiments of the inventive concept and are therefore not intended to limit the scope in any way, and there are other equally effective embodiments.

[0011] Figure 1 is a block diagram illustrating a system configured to implement one or more aspects of various embodiments;

[0012] Figure 2 A composable on-package module library implemented to generate a composable parallel processor according to various embodiments is shown;

[0013] Figure 3A According to various embodiments Figure 2 A block diagram of a three-dimensional (3D) high performance computing (HPC) high bandwidth memory (HBM-enabled) graphics processing unit (GPU).

[0014] Figure 3B It shows the Figure 3A A top view of a 3D HPC HBM-enabled GPU;

[0015] Figure 3C It shows the Figure 3A Side view of a 3D HPC HBM-enabled GPU;

[0016] Figure 4A According to various embodiments Figure 2 Block diagram of a 3D deep learning (DL) HBM enabled GPU;

[0017] Figure 4B It shows the Figure 4A A top view of a 3D DL HBM-enabled GPU;

[0018] Figure 4C It shows the Figure 4A Side view of a 3D DL HBM enabled GPU;

[0019] Figure 5A According to various embodiments Figure 2 A block diagram of a 2.5-dimensional (2.5D) HPC HBM-enabled GPU;

[0020] Figure 5B It shows the Figure 5A A top view of a 2.5D HPC HBM-enabled GPU.

[0021] Figure 5C It shows the Figure 5A Side view of a 2.5D HPC HBM-enabled GPU;

[0022] Fig. 6A According to various embodiments Figure 2 Block diagram of a 2.5D DL HBM enabled GPU;

[0023] Figure 6B It shows the Fig. 6A A top view of a 2.5D DL HBM-enabled GPU.

[0024] Figure 6C It shows the Fig. 6A A side view of a 2.5D DL HBM enabled GPU; and

[0025] Figure 7 is a flow chart of method steps for servicing memory access requests associated with a processor core included in a combinable parallel processor, according to various embodiments. DETAILED DESCRIPTION

[0026] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that these inventive concepts may be practiced without one or more of these specific details.

[0027] Exemplary System Overview

[0028] Figure 1 1 is a block diagram illustrating a system 100 configured to implement one or more aspects of various embodiments. As shown, system 100 includes, but is not limited to, a CPU 102 and a system memory 104 coupled to a parallel processing subsystem 112 via a memory bridge 105 and a communication path 113. In some embodiments, at least a portion of system memory 104 is a host memory associated with CPU 102. Memory bridge 105 is further coupled to an input / output (I / O) bridge 107 via a communication path 106, and I / O bridge 107 is in turn coupled to a switch 116. For purposes of illustration only, reference numbers identifying objects and alphanumeric characters in parentheses identifying instances where necessary are used herein to represent multiple instances of similar objects.

[0029] In operation, I / O bridge 107 is configured to receive user input information from input device 108 (such as a keyboard or mouse) and forward the input information to CPU 102 for processing via communication path 106 and memory bridge 105. Switch 116 is configured to provide connections between I / O bridge 107 and other components of system 100 (such as network adapter 118 and add-in cards 120 and 121).

[0030] As shown, I / O bridge 107 is coupled to system disk 114, which may be configured to store content, applications, and data used by CPU 102 and parallel processing subsystem 112. As a general matter, system disk 114 provides non-volatile storage for applications and data, and may include a fixed or removable hard disk drive, flash memory device, CD-ROM, DVD-ROM, Blu-ray, HD DVD, or other magnetic, optical, or solid-state storage device. Finally, although not explicitly shown, other components (such as a universal serial bus or other port connection, optical disk drive, DVD drive, film recording device, etc.) may also be connected to I / O bridge 107.

[0031] In various embodiments, memory bridge 105 may be a northbridge chip, and I / O bridge 107 may be a southbridge chip. In addition, communication paths 106 and 113, as well as other communication paths within system 100, may be implemented using any technically suitable protocol, including but not limited to Peripheral Component Interconnect Express, Accelerated Graphics Port, HyperTransport, or any other bus or point-to-point communication protocol known in the art.

[0032] In some embodiments, the parallel processing subsystem 112 includes, but is not limited to, one or more parallel processors. In some embodiments, each parallel processor is a graphics processing unit (GPU), which includes, but is not limited to, one or more streaming multiprocessors (SMs). Each SM includes, but is not limited to, multiple execution units also referred to herein as "processor cores". For purposes of illustration, each SM is also referred to herein as a "multithreaded processor core". In some embodiments, the parallel processors may be the same or different, and each parallel processor may be associated with a dedicated parallel processing (PP) memory, or may not be associated with a dedicated PP memory. In some embodiments, the PP memory associated with a given parallel processor includes, but is not limited to, one or more types of dynamic random access memory (DRAM). In some embodiments, the PP memory associated with a given GPU is also referred to as "device memory" associated with the GPU. In the same or other embodiments, each kernel launched on a given GPU resides in the device memory of the GPU.

[0033] In some embodiments, the parallel processing subsystem 112 incorporates circuits optimized for general processing. Such circuits may be incorporated across one or more GPUs that may be configured to perform general processing operations. In the same or other embodiments, the parallel processing subsystem 112 further incorporates circuits optimized for graphics processing. Such circuits may be incorporated across one or more GPUs that may be configured to perform graphics processing operations. In the same or other embodiments, any number of GPUs may output data to any number of display devices 110. In some embodiments, zero or more GPUs in the GPU may be configured to perform general processing operations rather than graphics processing operations, zero or more GPUs in the GPU may be configured to perform graphics processing operations rather than general processing operations, and zero or more GPUs in the GPU may be configured to perform general processing operations and / or graphics processing operations. In some embodiments, a software application executed under the control of the CPU 102 may start a kernel on one or more GPUs. The term "kernel" as used herein refers to an instruction set (e.g., a program, a function, etc.) that can be executed on a parallel processor.

[0034] In some embodiments, the parallel processing subsystem 112 may be coupled to Figure 1 The parallel processing subsystem 112 may be integrated with one or more other elements of the CPU 102 to form a single system. For example, the parallel processing subsystem 112 may be integrated with the CPU 102 and other connected circuits on a single chip to form a system on a chip. In the same or other embodiments, any number of CPUs 102 and any number of parallel processing subsystems 112 may be distributed across any number of shared geographic locations and / or any number of different geographic locations, and / or implemented in any combination in one or more cloud computing environments (i.e., packaged shared resources, software, data, etc.).

[0035] The system memory 104 may include, but is not limited to, any number and / or type of system software (e.g., operating system, device drivers, library programs, utilities, etc.), any number and / or type of software applications, or any combination thereof. The system software and software applications included in the system memory 104 may be organized in any technically feasible manner.

[0036] As shown, in some embodiments, system memory 104 includes, but is not limited to, a programming platform software stack 160 and software applications 150. Programming platform software stack 160 is associated with a programming platform for utilizing hardware in parallel processing subsystem 112 to accelerate computing tasks. In some embodiments, the programming platform may be accessible to software developers through, but not limited to, libraries, compiler directives, and / or extensions to programming languages. In the same or other embodiments, the programming platform may be, but is not limited to, a Compute Unified Device Architecture (CUDA) ( Developed by NVIDIA in Santa Clara, California), Radeon Open Compute Platform (ROCm), OpenCL ( developed by the Khronos group), SYCL, or Intel One API.

[0037] In some embodiments, the programming platform software stack 160 provides an execution environment for the software application 150 and zero or more other software applications (not shown). In the same or other embodiments, the software application 150 may include, but is not limited to, any computer software that can be launched on the programming platform software stack 160. In some embodiments, the software application 150 may be, but is not limited to, an artificial intelligence application or workload, a machine learning application or workload, a deep learning application or workload, a high performance computing application or workload, a virtual desktop infrastructure, or a data center workload.

[0038] In some embodiments, software application 150 and programming platform software stack 160 execute under the control of CPU 102. In the same or other embodiments, software application 150 may access one or more GPUs included in parallel processing subsystem 112 via programming platform software stack 160.

[0039] Note that the techniques described herein are illustrative rather than limiting and may be modified without departing from the broader spirit and scope of the invention. Many modifications and variations of the functionality provided by software application 150, programming platform software stack 160, CPU 102, parallel processing subsystem 112, GPU, SM, compute engine, and resource manager will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. For example, in some embodiments, CPU 102 may be replaced with any type of processor capable of launching kernels on a parallel processor.

[0040] It will be understood that the system shown herein is illustrative, and that variations and modifications are possible. The connection topology (including the number and arrangement of bridges, the number of CPUs 102, and the number of parallel processing subsystems 112) may be modified as desired. For example, in some embodiments, system memory 104 may be directly connected to CPU 102 rather than through memory bridge 105, and other devices may communicate with system memory 104 via memory bridge 105 and CPU 102. In some other alternative topologies, parallel processing subsystem 112 may be connected to I / O bridge 107 or directly to CPU 102 rather than being connected to memory bridge 105. In other embodiments, I / O bridge 107 and memory bridge 105 may be integrated into a single chip rather than existing as one or more discrete devices. Finally, in some embodiments, there may be no memory bridge 107 or memory bridge 105. Figure 1 For example, the switch 116 may be omitted, and the network adapter 118 and the add-in cards 120 , 121 may be connected directly to the I / O bridge 107 .

[0041] As described herein before, in order to meet the different computational requirements across multiple target application domains, many conventional parallel processors are designed as a single monolithic die with a converged architecture, in which a processing core typically includes dedicated hardware for each target application domain in order to increase computational throughput when processing associated workloads.

[0042] One disadvantage of using a converged architecture for parallel processors is that memory bandwidth allocation to off-die DRAM is typically a tradeoff between the relatively low memory bandwidth requirements of typical high performance computing workloads and the relatively high memory bandwidth requirements of typical deep learning workloads. For many converged architecture parallel processors, addressing the relatively high memory bandwidth requirements may require removing other circuits from the parallel processor or limiting other circuits within the parallel processor. Removing or reducing circuits typically (if not always) reduces overall processing performance, which is undesirable.

[0043] Configure parallel processors for different application domains

[0044] To address these issues, in some embodiments, the parallel processing subsystem 112 includes, but is not limited to, one or more composable parallel processors. As referred to herein, a "composable parallel processor" is a parallel processor that implements a composable on-package parallel processor architecture. In some embodiments, each composable parallel processor includes, but is not limited to, a parallel processor module (PPM), zero or more memory system modules (MSMs), and any amount (including none) and / or type of PP memory aggregated within a single package. In some embodiments, each MSM deepens the cache hierarchy of the associated PPM, provides one or more interfaces to the PP memory, or both. In the same or other embodiments, each PPM is reused in conjunction with various combinations of MSMs and PP memories to generate multiple composable parallel processors optimized for different application domains and / or workloads.

[0045] In some embodiments, to generate a combinable parallel processor, the PPM is selected from any number of available PPMs based on the type of parallel processor, the high-level architecture associated with the type of parallel processor, the packaging architecture, or any combination thereof. Some examples of types of parallel processors include, but are not limited to, GPUs, intelligent processing units, neural processing units, tensor processing units, neural network processors, data processing units, visual processing units, or any other type of unit capable of executing a large number of threads in parallel across multiple processing cores. For illustrative purposes, a PPM associated with a GPU is also referred to herein as a "GPU module" (GPM). Combinable parallel processors including but not limited to GPMs are also referred to herein as "combinable GPUs." Some examples of packaging architectures are 2.5D and three-dimensional 3D.

[0046] In some embodiments, to specialize the selected PPM, zero or more MSMs are selected from any number of available MSMs that are compatible with the selected PPM. The MSMs may be selected to enhance the baseline memory system provided by the selected PPM based on desired memory system requirements (e.g., typical memory system requirements associated with an application domain), one or more types of on-chip PP memory, one or more types of memory interfaces, or any combination thereof. As referred to herein, a memory interface enables a composable parallel processor to access any amount and / or type of PP memory (e.g., DRAM). In the same or other embodiments, the memory interface includes, but is not limited to, any number of memory controllers, any number of physical interfaces (PHYs), and any number of I / O pads.

[0047] Now return to see Figure 1In some embodiments, the parallel processing subsystem 112 includes, but is not limited to, one or more composable parallel processors. Each of the composable parallel processors included in the parallel processing subsystem 112 may be a GPU, an intelligence processing unit, a neural processing unit, a tensor processing unit, a neural network processor, a data processing unit, a visual processing unit, or any other type of unit capable of executing a large number of threads in parallel across multiple processing cores.

[0048] In some embodiments, each of the one or more combinable parallel processors included in parallel processing subsystem 112 can be connected to any amount (including no) and / or type of PP memory dedicated to that combinable parallel processor in any technically feasible manner. In the same or other embodiments, any number of combinable parallel processors are integrated into a single package with any amount (including no) and / or type of PP memory. In some embodiments, parallel processing subsystem 112 is a single chip that is also a combinable parallel processor.

[0049] For illustrative purposes, the following combines Figure 2-6C Some embodiments are described in detail in which one or more composable parallel processors are generated based on GPMs and MSMs derived from a single exemplary GPU. Note that the techniques described herein are illustrative and not limiting, and may be varied without departing from the broader spirit and scope of the invention. In particular, any number and / or type of composable parallel processors may be assembled from any number of PPMs and any number of MSMs generated in any technically feasible manner.

[0050] For example, in some embodiments, any number of PPMs and any number of MSMs may be derived from any number and / or type of parallel processors having any number and / or type of architecture. Further, each combinable parallel processor may be integrated with any amount (including none) and / or type of PP memory via any number and / or type of interfaces in any technically feasible manner. In some embodiments, any type of combinable parallel processor may be aggregated within a single chip having any amount and / or type of PP memory to generate a memory-enabled combinable parallel processor. A memory-enabled combinable parallel processor is also referred to herein as a "memory-enabled combinable parallel processor".

[0051] For purposes of illustration, the HPC designation is used herein in conjunction with MSMs and composable parallel processors (e.g., composable GPUs) that are specialized for the relatively low memory bandwidth requirements associated with many HPC workloads and many HPC applications, and the DL designation is used herein in conjunction with MSMs and composable parallel processors that are specialized for the relatively high memory bandwidth requirements associated with many DL workloads and many DL applications.

[0052] However, as will be appreciated by those skilled in the art, some HPC workloads and some HPC applications have relatively high memory bandwidth requirements and, therefore, can be executed more efficiently by a DL-specified combinable parallel processor than by an HPC-specified combinable parallel processor. Conversely, some DL workloads and some DL applications have relatively low memory bandwidth requirements and, therefore, can be executed more efficiently by an HPC-specified combinable parallel processor than by a DL-specified combinable parallel processor.

[0053] Thus, as used herein, the designation HPC corresponds to an exemplary low memory bandwidth class that is associated with many, but not all, HPC workloads and many other workloads across a wide variety of application domains. The designation DL, as used herein, corresponds to an exemplary high memory bandwidth class that is associated with many, but not all, DL workloads and many other workloads across a wide variety of application domains. Note that the exemplary memory bandwidth classes described herein are illustrative and not limiting, and the disclosed techniques may be applied to any number of different memory bandwidth classes corresponding to different ranges and / or types of memory bandwidth requirements.

[0054] Figure 2 A composable on-package module library 210 implemented to generate a composable parallel processor according to various embodiments is shown. The composable on-package module library 210 includes, but is not limited to, any number of PPMs and any number of MSMs. Each PPM and each MSM is designed to be implemented on a single die. For purposes of explanation, an instance of a PPM implemented on a single die is also referred to herein as a "PPM" and a "PPM die." An instance of an MSM implemented on a single die is also referred to herein as an "MSM" and an "MSM die."

[0055] In some embodiments, each PPM is associated with a packaging architecture. Based on the associated packaging architecture, a PPM die can be integrated with zero or more compatible MSM dies using on-package aggregation to generate a composable parallel processor. In some embodiments, the resulting composable parallel processor includes, but is not limited to, no memory interface, one or more on-package memory interfaces, or one or more off-package memory interfaces. Based on the associated packaging architecture and the number and type of memory interfaces, a composable parallel processor can be assembled into a single package with zero or more PP dies. As used herein, a "PP die" refers to a die that includes, but is not limited to, any amount and / or type of PP memory.

[0056] The composable on-package module library 210 may be generated in any technically feasible manner. In some embodiments, the architecture of each of any number and / or type of monolithic parallel processors is separated into a baseline PPM and a baseline MSM in order to generate the composable on-package module library 210. Each monolithic parallel processor may be separated in any technically feasible manner that facilitates optional expansion of the memory system of the baseline PPM.

[0057] In some embodiments, for each of any number of packaging architectures, a baseline PPM is modified to generate a PPM that can be integrated with one or more MSMs using any type of on-package aggregation consistent with the packaging architecture. In the same or other embodiments, for each PPM, an associated baseline MSM is modified to generate at least one MSM that can be integrated with the PPM. In some embodiments, any amount and / or type of functionality can be added to a given PPM, any amount and / or type of functionality can be removed from a given PPM, any amount and / or type of functionality can be transferred from a given PPM to any number of associated MSMs based on any amount and / or type of criteria, or any combination thereof.

[0058] In some embodiments, any amount and / or type of functionality may be added to or removed from the PPM to fine-tune the PPM based on the least demanding target application domain (e.g., HPC). Figure 3A As described in more detail, in some embodiments, to enable the PPM to be deployed as a composable parallel processor that does not include an MSM, any number and / or type of disabling memory interfaces may be added to the PPM. In some embodiments, and as described below in conjunction with Figure 3B As described in more detail, because the PPM is to be integrated with at least one MSM before being deployed as a composable parallel processor, a memory interface is not included in the PPM.

[0059] For each PPM, any number of MSMs may be generated based on the PPM, other associated MSMs, associated baseline MSMs, or any combination thereof to support any number and / or type of memory system requirements. In some embodiments, for each PPM, at least one MSM is generated for each target application domain. In the same or other embodiments, each MSM specializes the memory system of the associated PPM in any technically feasible manner.

[0060] As shown, in some embodiments, the composable on-package module library 210 includes, but is not limited to, a 3D GPM 220, a 2.5D GPM 230, any number (including none) and / or type of other PPMs (indicated via ellipses), a 3D DL MSM 222, a 2.5D HPC MSM 232, a 2.5D DL MSM 234, and any number (including none) and / or type of other MSMs (indicated via ellipses). The 3D GPM 220, the 2.5D GPM 230, the 3D DL MSM 222, the 2.5D HPC MSM 232, the 2.5D DL MSM 234 are derived from a single monolithic GPU.

[0061] In some embodiments, to generate the 3D GPM 220, 2.5D GPM 230, 3D DL MSM 222, 2.5D HPCMSM 232, 2.5D DL MSM 234, the associated monolithic GPU is separated into a baseline GPM and an associated baseline MSM based on a memory hierarchy. More specifically, the portion of the memory hierarchy below the level 2 (L2) cache is delegated to the baseline MSM, and the remainder of the monolithic GPU is delegated to the baseline GPM. Advantageously, when the memory hierarchy is partitioned in this manner, existing L2 cache bandwidth filtering within the baseline GPM can significantly reduce inter-die bandwidth requirements outside the GPM.

[0062] In some embodiments, based on the 3D packaging architecture, the baseline GPM is modified to generate the 3D GPM 220. The 3D GPM 220 may be deployed as a composable GPU that does not include an MSM, and may optionally be integrated with a single instance of one of any number of associated MSMs. The baseline MSM is modified to generate the 3D DL MSM 222 and any number (including zero) of other MSMs that may be integrated with the 3D GPM 220. As described in more detail below, the 3D DL MSM 222 expands the memory system of the 3D GPM 220 for the DL.

[0063] In the same or other embodiments, based on the 2.5D packaging architecture, the baseline GPM is modified to generate a 2.5D GPM 230. Prior to deployment as a composable GPU, the 2.5D GPM 230 will be integrated with one or two instances of one of any number of associated MSMs or one instance of each of two associated MSMs. The baseline MSM is modified to generate a 2.5D HPC MSM 232, a 2.5D DL MSM 234, and any number (including zero) of other MSMs that can be integrated with the 2.5D GPM 230. As described in more detail below, in some embodiments, the 2.5D HPC MSM 232 enables the 2.5D GPM 230 to interface with a PP memory. The 2.5D DL MSM 234 further enhances the memory system of the 2.5D GPM 230 for the DL relative to the 2.5D HPC MSM 232.

[0064] As depicted via arrows, different instances of 3D GPM 220 may be specialized to generate any number of composable GPUs for each of any number and / or type of application domains. In some embodiments, and as described below in conjunction with Figure 3A and Figure 4A Described in more detail, the 3D GPM 220 can be deployed without an MSM or vertically stacked with a single compatible MSM via integration on a 3D package. In some embodiments, the 3D GPM 220 is pre-equipped with a pipelined memory system optimized for HPC and enables the 3D GPM 220 to interface with one or more HBM stacks via any number of HBM interfaces that can be collectively disabled or enabled. In some embodiments, each HBM stack includes, but is not limited to, multiple DRAM dies vertically integrated or stacked using through silicon vias (TSVs). Each HBM interface can be any type of memory interface that enables the 3D GPM 220 to access at least a portion of the DRAM included in at least one HBM stack in any technically feasible manner.

[0065] The HBM interface included in the 3D GPM 220 may be disabled or enabled in any technically feasible manner. In some embodiments, any number and / or type of memory switches included in the 3D GPM 220 may be configured to route memory access requests for data that does not reside in one or more cache memories included in the 3D GPM 220 to a memory controller included in the HBM interface or to an on-package MSM, thereby enabling or disabling the HBM interface, respectively.

[0066] To specialize the 3D GPM 220 as a GPU for HPC, the HBM interface in the 3D GPM 220 is enabled. The resulting specialized 3D GPM is a GPU, which is also referred to herein as a 3D HPC GPU 262. Figure 3B and Figure 3C As described in more detail, in some embodiments, the 3D HPC GPU 262 may be aggregated with one or more HBM stacks within a single package using 2.5D on-silicon interposer integration to generate a 3D HPC HBM-enabled GPU 272. As used herein, an "HBM-enabled GPU" refers to a GPU aggregated with one or more HBM stacks in a single package.

[0067] In the same or other embodiments, each of any number of MSMs compatible with 3D GPM 220 deepens the cache hierarchy of 3D GPM 220, enabling 3D GPM 220 to interface with any amount (including none) and / or type of PP memory, or both. Figure 4A Described in more detail, the 3D DL MSM 222 deepens the cache hierarchy of the 3D GPM 220 and enables the 3D GPM 220 to interface with multiple HBM stacks.

[0068] To specialize the 3D GPM 220 as a GPU for DL, the HBM interface in the 3D GPM 220 is disabled and the 3D GPM 220 is integrated with the 3D DL MSM 222 via 3D on-package integration. The resulting composable GPU is also referred to herein as a 3D DL GPU 264. Figure 4B and Figure 4C As described in more detail, in some embodiments, the 3D DL GPU 264 may be aggregated with one or more HBM stacks within a single package integrated using a 2.5D on-silicon interposer to generate a 3D DL HBM-enabled GPU 274 .

[0069] As depicted via arrows, different instances of 2.5D GPM 230 may be dedicated to generating any number of composable GPUs for each of any number and / or type of application domains. In some embodiments, and as described below in conjunction with Figure 5A and Fig. 6A As described in more detail, the 2.5D GPM 230 may be integrated via on-package integration of a 2.5D package, using a planar die stacking approach, with one or two instances of a compatible MSM, or an instance of each of two compatible MSMs.

[0070] In some embodiments, because 2.5D GPM 230 will be integrated with at least one MSM before being deployed as a composable GPU, 2.5D GPM 230 does not include any memory interfaces. In the same or other embodiments, each of any number of MSMs compatible with 2.5D GPM 230 enables 2.5D GPM 230 to interface with any amount and / or type of PP memory and optionally deepen the cache hierarchy of 2.5D GPM 230.

[0071] As the following combination Figure 5A As described in more detail, in some embodiments, the 2.5D HPC MSM 232 includes, but is not limited to, one or more HBM interfaces that enable the 2.5D GPM 230 to be connected to one or more HBM stack interfaces. To specialize the 2.5D GPM 230 as a GPU for HPC, the 2.5D GPM 230 is aggregated with one or two instances of the 2.5D DL MSM 232 via 2.5D on-package integration. The resulting composable GPU is also referred to herein as a 2.5D HPC GPU 266. As described below in conjunction with Figure 5B and Figure 5C As described in more detail, in some embodiments, the 2.5D HPC GPU 266 may be aggregated with one or more HBM stacks within a single package using 2.5D on-silicon interposer integration to generate a 2.5D HPC HBM-enabled GPU 276 .

[0072] As the following combination Fig. 6A As described in more detail, the 2.5D DL MSM 234 enables the 2.5D GPM 230 to interface with an additional HBM stack relative to the 2.5D HPC MSM 232 and deepen the cache hierarchy of the 2.5D GPM 230. To specialize the 2.5D GPM 230 as a GPU for DL, in some embodiments, the 2.5D GPM 230 is aggregated with one or two instances of the 2.5D DL MSM 234 via 2.5D on-package integration. The resulting composable GPU is also referred to herein as a 2.5D DL GPU 268. As described below in conjunction with Figure 5B and Figure 5C As described in more detail, in some embodiments, the 2.5D DL GPU 268 may be aggregated with one or more HBM stacks within a single package using 2.5D on-silicon interposer integration to generate a 2.5D DL HBM-enabled GPU 278 .

[0073] As depicted via dashed lines, in some embodiments, one or more instances of 3D HPC GPU 262, 3D HPC HBM-enabled GPU 272, 3D DL GPU 264, 3D DL HBM-enabled GPU 274, 2.5D HPC GPU 266, 2.5D HPC HBM-enabled GPU 276, 2.5D DL GPU 268, 2.5D DL HBM-enabled GPU 278, any number of other GPUs (depicted via ellipses), or any combination thereof may be included in the 3D HPC GPU 262. Figure 1 In the parallel processing subsystem 112.

[0074] It is noteworthy that a composable GPU including a 3D GPM 220 can be attached to at most one MSM. In contrast, a composable GPU including a 2.5D GPU 230 can be attached to one or two MSMs. Each of the MSMs can be implemented on any die that is less than or equal to the maximum die size. Therefore, a combined GPU including a 2.5D GPM 230 can include up to twice the amount of additional cache relative to a composable GPU including a 3D GPM 220. In addition, because the MSMs are connected to the 2.5D GPM 230 in a planar manner, the MSMs can increase the total available die edge of the composable GPU. The additional die edge can be used to implement additional memory bandwidth and / or additional PP memory (e.g., HBM stack).

[0075] Implementing 3D GPU modules across different application domains

[0076] Figure 3A According to various embodiments Figure 2 2 is a block diagram of a 3D HPC HBM enabled GPU 272. As shown, in some embodiments, the 3D HPC HBM enabled GPU 272 includes, but is not limited to, a 3D HPC GPU 262 and HBM stacks 390(1)-390(6). In some embodiments, any positive number of HBM stacks may be included in the 3D HPC HBM enabled GPU 272. Also as shown, the 3D HPC HBM enabled GPU 272 is included in a package 398(1). Notably, the 3D HPC GPU 262 is also a 3D GPM 220(1). The 3D GPM 220(1) is an example of a 3D GPM 220 deployed as a 3D HPC HBM enabled GPU 272 without an attached MSM.

[0077] As shown, in some embodiments, 3D GPM 220(1) includes, but is not limited to, SMs 320(1)-320(N), level 1 (L1) caches 322(1)-322(N), a network on chip (NoC) 330, an L2 cache 340, memory switches 350(1)-350(M), and disabling memory controllers 352(1)-352(M), where N and M may be any positive integers. As previously described in conjunction with Figure 1 As described, in some embodiments, each of SMs 320(1)-320(N) is a multi-threaded processor core including, but not limited to, a plurality of processor cores. In some embodiments, each of SMs 320(1)-320(N) includes, but is not limited to, any number and / or type of specialized cores for each of any number of target application domains that increase computational throughput when processing associated workloads. For example, in some embodiments, each of SMs 320(1)-320(N) includes, but is not limited to, any number of ray tracing cores and / or any number of tensor cores that increase computational throughput for different high performance computing visual computing workloads and a variety of deep learning workloads, respectively.

[0078] As shown, SMs 320(1)-320(N) are associated with L1 caches 322(1)-322(N), respectively. In some embodiments, L1 cache memories 322(1)-322(N) are included in SMs 320(1)-320(N). In some other embodiments, L1 cache memories 322(1)-322(N) are external to SMs 320(1)-320(N), respectively. As shown, in some embodiments, L1 caches 322(1)-322(N) are connected to L2 cache 340 via NoC 330. In some embodiments, any number of L1 caches may be connected to any number of L2 caches in any technically feasible manner.

[0079] In some embodiments, L2 cache 340 is a distributed L2 cache including, but not limited to, L2 cache slices 342(1)-342(M). In the same or other embodiments, L2 cache 340 acts as a coherence point for SMs 320(1)-320(N), meaning that SMs 320(1)-320(N) access the same copy of memory locations via L2 cache 340. In some embodiments, L1 caches 322(1)-322(N) and L2 cache 340 are collectively referred to as "cache memory" associated with 3D GPM 220(1) and reside in the die implementing 3D GPM 220.

[0080] As shown, in some embodiments, the L2 cache slices 342(1)-342(M) are connected to memory switches 350(1)-350(M), respectively. The memory switches 350(1)-350(M) are configured to route memory access requests associated with L2 cache misses to the disabling memory controllers 352(1)-352(M), respectively, instead of to the 3D UHB link interface. The 3D UHB link interface is not used in the 3D GPM 220(1). As described below in conjunction with Figure 4A As described in detail, the memory switches 350 ( 1 )- 350 (M) enable the 3D GPM 220 to be configured with or without an attached MSM, and the 3D UHB link interface enables the 3D GPM 220 to be connected to the MSM.

[0081] The disabling memory controllers 352(1)-352(M) may be any type of on-die memory controller relative to the 3D GPM 220. In the context of the disabling memory controllers 352(1)-352(M), as used herein, "disabled" indicates that the disabling memory controllers 352(1)-352(M) are not used when an instance of the 3D GPM 220 is attached to an MSM. In some embodiments, the number of L2 cache slices may be different from the number of memory switches, the number of memory switches may be different from the number of disabling memory controllers, the number of L2 cache slices may be different from the number of disabling memory controllers, or any combination thereof.

[0082] Although not explicitly shown, in some embodiments, the disabling memory controllers 352(1)-352(M) are included in the HBM memory interface. In the same or other embodiments, each of the HBM memory interfaces includes, but is not limited to, one or more of the disabling memory controllers 352(1)-352(M), any number of PHYs, and any of the HBM I / O pads. In some embodiments, the HBM I / O pads are connected to the HBM stacks 390(1)-390(6) via 2.5D interposer integration on silicon (e.g., via TSVs). In some embodiments, connecting the HBM I / O pads to the HBM stacks 390(1)-390(6) enables each of the disabling memory controllers 352(1)-352(M) to access at least a portion of at least one of the HBM stacks 390(1)-390(6). In some embodiments, the 3D GPM 220(1) implements, but is not limited to, any amount and / or type of connections to implement the 3D GPM 220(1) between the die and the interposer ( Figure 3A Power is transmitted between the two devices (not shown).

[0083] In some embodiments, any instance of 3D GPM 220 (including 3D GPM 220(1)) may include, but is not limited to, any number and / or type of caches, any number of levels of a cache hierarchy, and any amount and / or type of memory logic. The memory logic may determine whether data associated with any type of memory access request resides within a cache included in 3D GPM 220 in any technically feasible manner.

[0084] In the same or other embodiments, the 3D GPM 220 may include any number and / or types of switching mechanisms that may be configured to route memory access requests that are not satisfied by a cache within the 3D GPM 220 to a memory interface included in the 3D GPM 220 or to an MSM that is connected to the 3D GPM 220 via integration on a 3D package in any technically feasible manner.

[0085] The cache and / or memory logic included in 3D GPM 220 may implement any amount and / or type of cache-related operations and / or optimization techniques in any technically feasible manner. For example, in some embodiments, 3D GPM 220 may implement any amount and / or type of cache filtering techniques, cache compression techniques, any other cache optimization techniques, or any combination thereof.

[0086] Figure 3B It shows the Figure 3A 3D HPC HBM enabled GPU 272 is shown in top view 312. As shown, 3D HPC HBM enabled GPU 272 is included in package 398(1). 3D GPM 220(1) is implemented in a single die, and each of HBM stacks 390 is implemented via multiple vertically stacked dies. In some embodiments, 3D GPM 220(1) is implemented in a single die having a size at or near a maximum possible size. In some embodiments, disabling memory controller 352 is positioned along two edges of 3D GPM 220(1), and HBM stack 390 is positioned along the same two edges in relatively close proximity to disabling memory controller 352.

[0087] Figure 3C It shows the Figure 3A3D HPC HBM enabled GPU 272 is shown in side view 314. As shown, in some embodiments, 3D GPM 220(1) is stacked vertically on top of interposer 396(1), and interposer 396(1) is stacked vertically on top of package 398(1). In the same or other embodiments, interposer 396(1) overlaps 3D GPM 220(1) on each side, and package 398(1) overlaps interposer 396(1) on each side. Disabled memory controller 352 is located at or near the bottom of 3D GPM 220(1) and is connected to HBM stack 390 through interposer 396(1) via HBM I / O (not shown) and HBM I / O connection 392.

[0088] Figure 4A According to various embodiments Figure 2 3D DL HBM enabled GPU 274. As shown, in some embodiments, 3D DL HBM enabled GPU 274 includes, but is not limited to, 3D DL GPU 264 and HBM stacks 390(1)-390(6). In some embodiments, any positive number of HBM stacks may be included in 3D DL HBM enabled GPU 274. As shown, 3D DL HBM enabled GPU 274 is included in package 398(2).

[0089] As shown, in some embodiments, 3D DL GPU 264 includes, but is not limited to, 3D GPM 220(2) and 3D DL MSM 222. 3D GPM 220(2) is an example of 3D GPM 220 integrated with 3D DL MSM 222 prior to being deployed as 3D DL GPU 264. In some embodiments, the components included in 3D GPM 220(2) are the same as those previously integrated with 3D DL MSM 222. Figure 2 The components described are the same as those included in 3D GPM 220(1). More specifically, in some embodiments, 3D GPM 220(2) includes, but is not limited to, SMs 320(1)-320(N), L1 caches 322(1)-322(N), NoC 330, L2 cache 340, memory switches 350(1)-350(M), and disabling memory controllers 352(1)-352(M), where N and M may be any positive integers.

[0090] Compared to 3D GPM 220(1), memory switches 350(1)-350(M) included in 3D GPM 220(2) are configured to route memory access requests associated with L2 cache misses to 3D UHB link interfaces (not explicitly shown) instead of disabling memory controllers 352(1)-352(M), respectively. Thus, disabling memory controllers 352(1)-352(M) included in 3D GPM 220(2) are not used and are effectively disabled. Although not shown, in some embodiments, disabling memory controllers 352(1)-352(M) are included in unused HBM memory interfaces in 3D GPM 220(2).

[0091] As previously combined Figure 3A As described, in some embodiments, SMs 320(1)-320(N) are associated with L1 caches 322(1)-322(N), respectively. In some embodiments, L1 caches 322(1)-322(N) are contained within SMs 320(1)-320(N). In some other embodiments, L1 caches 322(1)-322(N) are external to SMs 320(1)-320(N), respectively. As shown, in some embodiments, L1 caches 322(1)-322(N) are connected to L2 cache 340 via NoC 330. In some embodiments, any number of L1 caches may be connected to any number of L2 caches in any technically feasible manner. In the same or other embodiments, the number of L2 cache slices may be different from the number of memory switches, the number of memory switches may be different from the number of memory controllers that may be disabled, the number of L2 cache slices may be different from the number of memory controllers that may be disabled, or any combination thereof.

[0092] The 3D UHB link interface enables the 3D GPM 220(2) to be vertically integrated with a single compatible MSM via 3D on-package integration. The 3D UHB link interface can be implemented in any technically feasible manner. In some embodiments, the 3D UHB link interface includes, but is not limited to, any number (including none) and / or types of components, any amount (including none) of area reserved on any number of silicon layers, any number of pads (e.g., copper bonding pads), or any combination thereof. In some embodiments, the 3D GPM 220(2) implements, but is not limited to, any number and / or types of connections to connect the die and interposer ( Figure 4A Power is transmitted between the two devices (not shown).

[0093] As depicted by the bidirectional arrows between the 3D GPM 220(2) and the 3D DL MSM 222, the 3D DL MSM 222 is connected to the 3D GPM 220(2) via 3D on-package integration. More precisely, the 3D DL MSM 222 is vertically attached to the 3D DL MSM 222 via a 3D ultra-high bandwidth (UHB) link using high-density, in-die bonding. In some embodiments, the 3D DL MSM 222 is positioned between the interposer and the 3D GPM 220(2). In the same or other embodiments, the 3D DL MSM 222 enables, but is not limited to, any amount and / or type of connections between the 3D GPM 220(2) and the interposer via any number of TSVs.

[0094] As shown, the 3D DL MSM 222 includes, but is not limited to, a level 3 (L3) cache 470 and memory controllers 480(1)-480(M). The 3D DL MSM 222 receives a memory access request associated with an L2 cache miss from the 3D GPM 220(2) via a 3D UHB link. The 3D DL MSM 222 includes, but is not limited to, any amount and / or type of memory logic that enables the 3D DL MSM 222 to service memory access requests in a hierarchical manner via the L3 cache 470 and the HBM stacks 390(1)-390(6). In operation, if data associated with a memory request received via the 3D UHB link resides with the L3 cache 470, the 3D DL MSM 222 uses the L3 cache 470 to satisfy the memory request. Otherwise, 3D DL MSM 222 uses HBM stack 390(1)-390(6) to configure memory controller 480(1)-480(M) to satisfy the memory request. 3D DL MSM 222 relays any data fetched from L3 cache 470 and HBM stack 390(1)-390(6) to 3D GPM 220(2) via 3D UHB link.

[0095] The L3 cache 470 deepens the cache hierarchy of the 3D GPM 220 (2) and is also referred to as the last level cache (LLC) of the 3D DL GPU 264. The L3 cache 470 can be implemented architecturally in any technically feasible manner. In some embodiments, the L3 cache 470 is implemented architecturally as another level of memory-side cache supporting the L2 cache 340. In the same or other embodiments, the L3 cache 470 is neither included nor excluded. In some embodiments, the L2 cache 340 acts as a consistency point for the SM 320 (1)-320 (N), and therefore the L3 cache 470 does not require consistency with the L2 cache 340. In some embodiments, from the perspective of the 3D DL GPU 264, the cache lines (not shown) present in the L2 cache 340 replace the cache lines (not shown) present in the L3 cache 470. In the same or other embodiments, when a cache line is evicted from L2 cache 340 in 3D DL GPU 264, 3D DL GPU 264 writes the cache line back to L3 cache 470. In some embodiments, 3D GPM 220(2) does not route any memory access request to L3 cache 470 until the memory access request has been serviced via L2 cache 340.

[0096] In some embodiments, L3 cache 470 includes, but is not limited to, L3 cache slices 472(1)-472(M). As shown, in some embodiments, L3 cache slices 472(1)-472(M) are connected to L2 cache slices 342(1)-342(M) via 3D UHB links and memory switches 350(1)-350(M), respectively. In some embodiments, L3 cache slices 472(1)-472(M) are connected to memory controllers 480(1)-480(M), respectively.

[0097] The memory controllers 480(1)-480(M) receive memory access requests associated with L3 cache misses and, in response, interact with the HBM stacks 390(1)-390(6) in any technically feasible manner to satisfy the memory access requests. Although not explicitly shown, in some embodiments, the memory controllers 480(1)-480(M) are included in the HBM memory interface. In the same or other embodiments, each of the HBM memory interfaces includes, but is not limited to, one or more of the memory controllers 480(1)-480(M), any number of PHYs, and any of the HBM I / O pads.

[0098] In some embodiments, the HBM I / O pads of the 3D DL MSM 222 are connected to the HBM stacks 390(1)-390(6) via 2.5D on-silicon interposer integration (e.g., via TSVs). In some embodiments, connecting the HBM I / O pads of the 3D DL MSM 222 to the HBM stacks 390(1)-390(6) enables each of the memory controllers 480(1)-480(M) to access at least a portion of at least one of the HBM stacks 390(1)-390(6).

[0099] In some embodiments, the number of L3 cache slices may be different from the number of memory switches, the number of L2 cache slices, or both the number of memory switches and the number of L2 cache slices. In the same or other embodiments, the number of L3 cache slices may be different from the number of memory controllers.

[0100] In some embodiments, any instance of 3D GPM 220 (including 3D GPM 220(2)) may include, but is not limited to, any number and / or type of caches, any number of levels of a cache hierarchy, and any amount and / or type of memory logic. The memory logic may determine whether data associated with any type of memory access request resides within a cache included in 3D GPM 220 in any technically feasible manner.

[0101] In the same or other embodiments, the 3D GPM 220 may include, but is not limited to, any number and / or type of switching mechanisms that may be configured to route memory access requests that are not satisfied by a cache within the 3D GPM 220 to a memory interface included in the 3D GPM 220 or to an MSM that is connected to the 3D GPM 220 via integration on the 3D package in any technically feasible manner.

[0102] In some embodiments, the 3D DL MSM 222 may include, but is not limited to, any number and / or type of caches, any number of levels of a cache hierarchy, and any amount and / or type of memory logic. The levels of cache included in the 3D DL MSM 222 deepen the cache memory hierarchy, and therefore deepen the memory hierarchy of the portion of the 3D DL GPU 264 implemented in the 3D GPM 220. The memory logic may determine whether data associated with any type of memory access request resides within a cache included in the 3D DL LSM 222 in any technically feasible manner.

[0103] The cache and / or memory logic included in 3D GPM 220 and 3D DL MSM 222 may implement any amount and / or type of cache-related operations and / or optimization techniques in any technically feasible manner. For example, in some embodiments, 3D GPM 220 may implement any amount and / or type of cache filtering techniques, cache compression techniques, any other cache optimization techniques, or any combination thereof.

[0104] Figure 4B It shows the Figure 4A 4. A top view 412 of a 3D DL HBM enabled GPU 274 is shown. As shown, the 3D DL HBM enabled GPU 274 is included in a package 398(2). The 3D GPM 220(2) is implemented in a single die, and each of the HBM stacks 390 is implemented via multiple vertically stacked dies. Although not visible in the top view 412, the 3D DL MSM 222 is implemented in a single die vertically located below the 3D GPM 220(2). In some embodiments, the 3D GPM 220(2) is implemented in a single die having a size at or near a maximum possible size. In the same or other embodiments, the 3D GPM 220(2) is implemented in a single die having a size at or near a maximum possible size. In some embodiments, the disabling memory controller 352 is disabled and positioned along two edges of the 3D GPM 220(2). The HBM stack 390 is positioned relatively close to the disabling memory controller 352 along the same two edges.

[0105] Figure 4C It shows the Figure 4A 414 of a 3D DL HBM enabled GPU 274. As shown, in some embodiments, 3D GPM 220(2) is vertically stacked on top of 3D DL MSM 222, 3D DL MSM 222 is vertically stacked on top of interposer 396(2), and interposer 396(2) is vertically stacked on top of package 398(2). In some embodiments, 3D GPM 220(2) is the same size as 3D DL MSM 222. In the same or other embodiments, interposer 396(2) overlaps 3D DL MSM 222 on each side, and package 398(2) overlaps interposer 396(2) on each side. The disabling memory controller 352 is positioned at or near the bottom of the 3D GPM 220 ( 2 ), and the memory controller 480 is positioned below the disabling memory controller 352 at or near the top of the 3D DL MSM 222 .

[0106] As shown, in some embodiments, memory controller 480 is connected to HBM stack 390 through interposer 396(2) via HBM I / O included in 3D DL MSM 222 and HBMI / O connection 392. In the same or other embodiments, 3D GPM 220(2) is connected to 3D DL MSM 222 via 3D UHB link 428. Notably, 3D UHB link 428 is significantly shorter and faster than each HBM I / O connection 392.

[0107] Implementing 2.5D GPU modules across different application domains

[0108] Figure 5A According to various embodiments Figure 2 2.5D HPC HBM enabled GPU 276. As shown, in some embodiments, 2.5D HPC HBM enabled GPU 276 includes, but is not limited to, 2.5D HPC GPU 266 and HBM stacks 390(1)-390(6), and is implemented in package 398(3). In some embodiments, any positive number of HBM stacks may be included in 2.5D HPC HBM enabled GPU 276.

[0109] As shown, in some embodiments, the 2.5D HPC GPU 266 includes, but is not limited to, a 2.5D GPM 230, a 2.5D HPC MSM 232(1), and a 2.5D HPC MSM 232(2). The 2.5D HPC MSM 232(1) and the 2.5D HPC MSM 232(2) are two instances of the 2.5D HPC MSM 232. In some embodiments, the 2.5D GPM 230 includes, but is not limited to, SMs 320(1)-320(N), L1 caches 322(1)-322(N), NoCs 330, and L2 caches 340. In some embodiments, the SMs 320(1)-320(N), L1 caches 322(1)-322(N), NoCs 330, and L2 caches 340 included in the 2.5D GPM 230 are similar to those previously described in conjunction with the present invention. Figure 2 The components included in the described 3D GPM 220(1) are the same.

[0110] In some embodiments, L1 caches 322(1)-322(N) are included in SMs 320(1)-320(N). In some other embodiments, L1 caches 322(1)-322(N) are external to SMs 320(1)-320(N), respectively. As shown, in some embodiments, L1 caches 322(1)-322(N) are connected to L2 cache 340 via NoC 330. In some implementations, any number of L1 caches may be connected to any number of L2 caches in any technically feasible manner.

[0111] In some embodiments, 2.5D GPM 230 may include, but is not limited to, any number and / or type of caches, any number of levels of a cache hierarchy, and any amount and / or type of memory logic. The caches and / or memory logic included in 2.5D GPM 230 may implement any number and / or type of cache-related operations and / or optimization techniques in any technically feasible manner. For example, in some embodiments, 2.5D GPM 230 may implement any number and / or type of cache filtering techniques, cache compression techniques, any other cache optimization techniques, or any combination thereof.

[0112] Advantageously, in some embodiments, 2.5D GPM 230 does not include any memory interface and can use the "freed" area to implement additional computing resources. Instead of implementing a memory interface, if 2.5D GPM 230 determines that the data associated with the memory access request does not reside within a cache included in 2.5D GPM 230, 2.5D GPM 230 sends a request to one of the two 2.5D USB link interfaces ( Figure 5A 2.5D GPM 230 may determine that data associated with the memory access request does not reside within a cache included in 2.5D GPM 230 and route the memory access request to one of the 2.5D USB link interfaces in any technically feasible manner.

[0113] The 2.5D USB link interface enables the 2.5D GPM 230 to be integrated with one or two instances of a single compatible MSM or one instance of each of two compatible MSMs in a planar manner via 2.5D on-package integration. The 2.5D USB link interface may be implemented in any technically feasible manner. In some embodiments, each 2.5D USB link interface includes, but is not limited to, any number (including none) and / or type of components, any number (including none) of areas reserved on any number of silicon layers, any number of pads (e.g., copper bonding pads), or any combination thereof. In some embodiments, two 2.5D USB link interfaces included in a 2.5D GPM 230 may be the same as each other or may be different from each other.

[0114] As depicted by the double-headed arrows, the 2.5D GPM 230 is planarly connected to both the 2.5D HPC MSM 232(1) and the 2.5D HPC MSM 232(2) via 2.5D on-package integration. More specifically, in some embodiments, one of the 2.5D USB link interfaces included in the 2.5D GPM 230 is attached to the 2.5D USB link interface included in the 2.5D HPC MSM 232(1) via one 2.5D UHB link. Another 2.5D USB link interface included in the 2.5D GPM 230 is attached to the 2.5D USB link interface included in the 2.5D HPC MSM 232(2) via another 2.5D UHB link. In some embodiments, the 2.5D HPC MSM 232(1) and the 2.5D HPC MSM 232(2) are located on opposite sides of the 2.5D GPM 230.

[0115] In some embodiments, each of the 2.5D USB link interfaces included in 2.5D GPM 230 may be the same as or different from the 2.5D USB link included in each instance of 2.5D HPC MSM 232. In the same or other embodiments, one of the 2.5D USB link interfaces included in 2.5D GPM 230 is connected to a 2.5D USB link interface included in 2.5D HPC MSM 232(1) via a single 2.5D UHB link, and the other 2.5D link interfaces included in 2.5D GPM 230 are not used.

[0116] The 2.5D HPC MSM 232(1) and the 2.5D HPC MSM 232(2) receive memory access requests that cannot be satisfied by the 2.5D GPM 230 via the 2.5D UHB link via the cache hierarchy implemented in the 2.5D GPM 230. As shown, collectively, the 2.5D HPC MSM 232(1) and the 2.5D HPC MSM 232(2) include, but are not limited to, memory controllers 480(1)-480(M). In operation, the memory controllers 480(1)-480(M) receive and satisfy memory access requests received via the 2.5D UHB link using the HBM stack 390(1)-390(6). The memory controllers 480(1)-480(M) may interact with the HBM stack 390(1)-390(6) in any technically feasible manner to satisfy the memory access requests.

[0117] Although not explicitly shown, in some embodiments, memory controllers 480(1)-480(M) are included in the HBM memory interface. In the same or other embodiments, each of the HBM memory interfaces includes, but is not limited to, one or more of the memory controllers 480(1)-480(M), any number of PHYs, and any of the HBM I / O pads. The 2.5D HPCMSM 232(1) and 2.5D HPC MSM 232(2) relay any data extracted from the HBM stacks 390(1)-390(6) to the 2.5D GPM 230 via the 2.5D UHB link.

[0118] In some embodiments, the HBM I / O pads of the 2.5D HPC MSM 232(1) and the 2.5D HPC MSM 232(2) are connected to the HBM stack 390(1)-390(6) via 2.5D silicon on interposer integration (e.g., via TSVs). In some embodiments, connecting the HBM I / O pads of the 2.5D HPC MSM 232(1) and the 2.5D HPC MSM 232(2) to the HBM stack 390(1)-390(6) enables each of the memory controllers 480(1)-480(M) to access at least a portion of at least one of the HBM stacks 390(1)-390(6).

[0119] Memory access requests, memory controllers 480(1)-480(M), and integration of HBM stacks 390(1)-390(6) may be distributed between 2.5D HPC MSM 232(1) and 2.5D HPC MSM 232(2) in any technically feasible manner. In some embodiments, 2.5D HPC MSM 232(1) includes, but is not limited to, memory controllers 480(1)-480(M / 2), and HBM I / O pads of 2.5D HPC MSM 232(1) are connected to HBM stacks 390(1)-390(3) via 2.5D silicon interposer integration. In the same or other embodiments, 2.5D HPC MSM 232(2) includes, but is not limited to, memory controllers 480(M / 2+1)-480(M), and HBM I / O pads of 2.5D HPC MSM 232(2) are integratedly connected to HBM stacks 390(4)-390(6) via a 2.5D on-silicon interposer. In some embodiments, connecting the HBM I / O pads of 2.5D HPC MSM 232(2) to HBM stacks 390(4)-390(6) enables each of memory controllers 480(M / 2+1)-480(M) to access at least a portion of at least one of HBM stacks 390(1)-390(3).

[0120] In some embodiments, the 2.5D GPM 230 forwards memory requests associated with L2 cache misses corresponding to the L2 cache slices 342(1)-342(M / 2) to the 2.5D HPC MSM 232(1) via the 2.5D UHB link 528(1). The 2.5D HPC MSM 232(1) configures the memory controllers 480(1)-480(M / 2) using one or more of the HBM stacks 390(1)-390(3) to satisfy the memory requests. The memory controllers 480(1)-480(M / 2) may interact with the HBM stacks 390(1)-390(3) in any technically feasible manner to satisfy the memory access requests. The HPC MSM 232(1) relays any data fetched from the HBM stacks 390(1)-390(3) to the 2.5D GPM 230 via the 2.5D UHB link 528(1).

[0121] In the same or other embodiments, the 2.5D GPM 230 forwards memory requests associated with L2 cache misses corresponding to L2 cache slices 342(M / 2+1)-342(M) to the 2.5D HPC MSM 232(2) via the 2.5D UHB link 528(2). The 2.5D HPC MSM 232(2) uses one or more of the HBM stacks 390(4)-390(6) to configure the memory controllers 480(M / 2+1)-480(M) to satisfy the memory requests. The memory controllers 480(M / 2+1)-480(M) may interact with the HBM stacks 390(4)-390(6) in any technically feasible manner to satisfy the memory access requests. The HPCMSM 232 ( 2 ) relays any data extracted from the HBM stacks 390 ( 4 )- 390 ( 6 ) to the 2.5D GPM 230 via the 2.5D UHB link 528 ( 2 ).

[0122] Figure 5B It shows the Figure 5A 398(3). 2.5D GPM 230 is implemented in one die that may be at or near the maximum possible size. 2.5D HPC MSM 232(1) is implemented in another die on one side of 2.5D GPM 230, and 2.5D HPC MSM 232(2) is implemented in yet another die on the other side of 2.5D GPM 230.

[0123] As shown, in some embodiments, the 2.5D GPM 230, the 2.5D HPC MSM 232(1), and the 2.5D HPCMSM 232(2) implement two, one, and one 2.5D USB link interface 520, respectively. In some embodiments, three of the memory controllers 480 are positioned at or near opposite edges of the 2.5D HPC MSM 232(1) relative to the 2.5D GPM 230. The three memory controllers 480 are positioned at or near opposite edges of the 2.5D HPC MSM 232(2) relative to the 2.5D GPM 230. The HBM stacks 390 are distributed near the outer boundaries of the package 398(3) and are relatively close to the memory controllers 480.

[0124] Figure 5C It shows the Figure 5A514 of a 2.5D HPC HBM enabled GPU 276. As shown, in some embodiments, a portion of the HBM stack 390, the 2.5D HPC MSM 232(1), the 2.5D GPM 230, the 2.5D HPC MSM 232(2), and the remainder of the HBM stack 390 are sequentially arranged in a planar manner on top of an interposer 396(3). The interposer 396(3) is stacked on top of a package 398(3). In some embodiments, the package 398(3) overlaps the interposer 396(3) on each side.

[0125] As shown, in some embodiments, the memory controller 480 is distributed at or near the outer edges of the 2.5D HPC MSM 232(1) and the 2.5D HPC MSM 232(2), and is connected to the HBM stack 390 through the interposer 396(3) via the HBM I / O connections 392. The 2.5D UHB link interface 520 included in the 2.5D HPC MSM 232(1) is connected to one of the 2.5D UHB link interfaces 520 included in the 2.5D GPM 230 via the 2.5D UHB link 528(1). The 2.5D UHB link interface 520 included in the 2.5D HPC MSM 232(2) is connected to a different one of the 2.5D UHB link interfaces 520 included in the 2.5D GPM 230 via the 2.5D UHB link 528(2). Notably, each of the 2.5D UHB links 528 ( 1 ) and 528 ( 2 ) is significantly shorter and faster than each of the HBM I / O connections 392 .

[0126] Fig. 6A According to various embodiments Figure 22.5D DL HBM enabled GPU 278. In some embodiments, the available die edge of 2.5D DL HBM enabled GPU 278 is larger relative to 2.5D HPC HBM enabled GPU 276, 3D DL HBM enabled GPU 274, and 3D HPC HBM enabled GPU 272, and thus, in addition to HBM stacks 390(1)-390(5), 2.5D DL HBM enabled GPU 278 can also accommodate HBM stacks 390(6)-390(10). As shown, in some embodiments, 2.5D DL HBM enabled GPU 278 includes, but is not limited to, 2.5D DL GPU 268 and HBM stacks 390(1)-390(10) integrated within package 398(4). In some embodiments, any positive number of HBM stacks can be included in 2.5D HPC HBM enabled GPU 278, subject to die area limitations and package area limitations.

[0127] As shown, in some embodiments, the 2.5D DL GPU 268 includes, but is not limited to, a 2.5D GPM 230, a 2.5D DL MSM 234(1), and a 2.5D DL MSM 234(2). The 2.5D DL MSM 234(1) and the 2.5D DL MSM 234(2) are two examples of the 2.5D DL MSM 234. The 2.5D GPM 230 is previously described herein. Figure 5A However, before that, Figure 5A The 2.5D GPM 230 included in the depicted 2.5D HPC GPU 266 is connected to the 2.5D HPC MSM 232(1) and the 2.5D HPC MSM 232(2), while the 2.5D GPM 230 included in the 2.5D DL GPU 268 is connected to the 2.5D DL MSM 234(1) and the 2.5D DL MSM 234(2).

[0128] As depicted by the double-headed arrows, the 2.5D GPM 230 is planarly connected to both the 2.5D DL MSM 234(1) and the 2.5D DL MSM 234(2) via 2.5D on-package integration. Figure 5CAs shown in detail, in some embodiments, one of the 2.5D USB link interfaces 520 included in the 2.5D GPM 230 is attached to the 2.5D USB link interface 520 included in the 2.5D DL MSM 234(1) via a 2.5D UHB link 528(1). The remaining one of the 2.5D USB link interfaces 520 included in the 2.5D GPM 230 is attached to the 2.5D USB link interface 520 included in the 2.5D DL MSM 234(2) via a 2.5D UHB link 528(2). In some embodiments, the 2.5D DL MSM 234(1) and the 2.5D DL MSM 234(2) are located on opposite sides of the 2.5D GPM 230.

[0129] In some embodiments, each of the 2.5D USB link interfaces 520 included in the 2.5D GPM 230 may be the same as or different from the 2.5D USB link interface 520 included in each instance of the 2.5D DL MSM 234. In the same or other embodiments, one of the 2.5D USB link interfaces 520 included in the 2.5D GPM 230 is connected to the 2.5D USB link interface 520 included in the 2.5D DL MSM 234(1) via the 2.5D UHB link 528(1), and the other of the 2.5D USB link interfaces 520 included in the 2.5D GPM 230 is not used.

[0130] As shown, in some embodiments, 2.5D DL MSM 234(1) and 2.5D DL MSM 234(2) receive memory access requests associated with L2 cache misses from 2.5D GPM 230 via 2.5D UHB link 528(1) and 2.5D UHB link 528(2), respectively. As shown, 2.5D DL MSM 234(1) and 2.5D DL MSM 234(2) collectively include, but are not limited to, L3 cache 470 and memory controllers 480(1)-480(M). In some embodiments, the 2.5D DL MSM 234 includes, but is not limited to, memory logic that enables the 2.5D DL MSM 234 to service any amount and / or type of memory access requests in a hierarchical manner via a portion of the L3 cache 470 included in the 2.5D DL MSM 234 and any portion (including all) of the HBM stacks 390 ( 1 )- 390 ( 10 ) connected to the 2.5D DL MSM 234 .

[0131] The L3 cache 470 deepens the cache hierarchy of the 2.5D GPM 230 and is also referred to as the LLC of the 2.5D DL GPU 268. The L3 cache 470 can be implemented architecturally and distributed between the 2.5D DL MSM 234(1) and the 2.5D DL MSM 234(2) in any technically feasible manner. In some embodiments, the L3 cache 470 is implemented architecturally as another level of memory-side cache supporting the L2 cache 340. In the same or other embodiments, the L3 cache 470 is neither inclusive nor exclusive. In some embodiments, the L2 cache 340 acts as a consistency point for the SMs 320(1)-320(N), and therefore the L3 cache 470 does not require consistency with the L2 cache 340. In some embodiments, from the perspective of the 2.5D DL GPU 268, a cache line (not shown) present in the L2 cache 340 replaces a cache line (not shown) present in the L3 cache 470. In the same or other embodiments, when a cache line is evicted from L2 cache 340 in 2.5D DL GPU 268, 2.5D DL GPU 268 writes the cache line back to L3 cache 470. In some embodiments, 2.5D GPM 230 does not route any memory access request to L3 cache 470 until the memory access request has been serviced via L2 cache 340.

[0132] In some embodiments, L3 cache 470 includes, but is not limited to, L3 cache slices 472(1)-472(M). As shown, in some embodiments, L3 cache slices 472(1)-472(M) are connected to L2 cache slices 342(1)-342(M) via 2.5D UHB links 528(1) and 2.5D UHB links 528(2), respectively, and memory switches 350(1)-350(M). In the same or other embodiments, L3 cache slices 472(1)-472(M) are connected to memory controllers 480(1)-480(M), respectively.

[0133] In some embodiments, the 2.5D DL MSM 234(1) includes, but is not limited to, L3 cache slices 472(1)-472(M / 2) and memory controllers 480(1)-480(M / 2). The 2.5D GPM 230 forwards a memory request associated with an L2 cache miss corresponding to an L2 cache slice 342(1)-342(M / 2) to the 2.5D DL MSM 234(1) via a 2.5D UHB link 528(1). If data associated with the memory request forwarded to the 2.5D DL MSM 234(1) via the 2.5D UHB link 528(1) resides within an L3 cache slice 472(1)-472(M / 2), the 2.5D DL MSM 234(1) satisfies the memory request using the L3 cache slice 472(1)-472(M / 2). Otherwise, the 2.5D DL MSM 234(1) uses one or more of the HBM stacks 390(1)-390(5) to configure the memory controller 480(1)-480(M / 2) to satisfy the memory request. The memory controller 480(1)-480(M / 2) may interact with the HBM stacks 390(1)-390(5) in any technically feasible manner to satisfy the memory access request. The 2.5D DL MSM 234(1) relays any data fetched from the L3 cache slices 472(1)-472(M / 2) and the HBM stacks 390(1)-390(5) to the 2.5D GPM 230 via the 2.5D UHB link 528(1).

[0134] In some embodiments, the 2.5D DL MSM 234(2) includes, but is not limited to, L3 cache slices 472(M / 2+1)-472(M) and memory controllers 480(M / 2+1)-480(M). The 2.5D GPM 230 forwards a memory request associated with an L2 cache miss corresponding to the L2 cache slices 342(M / 2+1)-342(M) to the 2.5D DL MSM 234(2) via the 2.5D UHB link 528(2). If data associated with the memory request forwarded to the 2.5D DL MSM 234(2) via the 2.5D UHB link 528(2) resides within the L3 cache slices 472(M / 2+1)-472(M), the 2.5D DL MSM 234(2) satisfies the memory request using the L3 cache slices 472(M / 2+1)-472(M). Otherwise, the 2.5D DL MSM 234(2) uses one or more of the HBM stacks 390(6)-390(10) to configure the memory controller 480(M / 2+1)-480(M) to satisfy the memory request. The memory controller 480(M / 2+1)-480(M) may interact with the HBM stacks 390(6)-390(10) in any technically feasible manner to satisfy the memory access request. The 2.5D DL MSM 234(2) relays any data fetched from the L3 cache slices 472(M / 2+1)-472(M) and the HBM stacks 390(6)-390(10) to the 2.5D GPM 230 via the 2.5D DUHB link 528(2).

[0135] Although not explicitly shown, in some embodiments, memory controllers 480(1)-480(M) are included in the HBM memory interface. In the same or other embodiments, each of the HBM memory interfaces includes, but is not limited to, one or more of memory controllers 480(1)-480(M), any number of PHYs, and any of the HBM I / O pads.

[0136] In some embodiments, the HBM I / O pads of the 2.5D DL MSM 234(1) are connected to the HBM stacks 390(1)-390(5) via 2.5D on-silicon interposer integration (e.g., via TSVs). In some embodiments, connecting the HBM I / O pads of the 2.5D DL MSM 234(1) to the HBM stacks 390(1)-390(5) enables each of the memory controllers 480(1)-480(M / 2) to access at least a portion of at least one of the HBM stacks 390(1)-390(5).

[0137] In some embodiments, the HBM I / O pads of the 2.5D DL MSM 234(2) are connected to the HBM stacks 390(6)-390(10) via 2.5D on-silicon interposer integration (e.g., via TSVs). In some embodiments, connecting the HBM I / O pads of the 2.5D DL MSM 234(2) to the HBM stacks 390(6)-390(10) enables each memory controller 480(M / 2+1)-480(M) to access at least a portion of at least one of the HBM stacks 390(6)-390(10).

[0138] In some embodiments, the number of L3 cache slices may be different from the number of memory switches, the number of L2 cache slices, or both the number of memory switches and the number of L2 cache slices. In the same or other embodiments, the number of L3 cache slices may be different from the number of memory controllers. In some embodiments, the number of HBM stacks 390 included in a 2.5DDL GPU 268 may be any positive integer.

[0139] In some embodiments, the 2.5D GPM 230 may include, but is not limited to, any number and / or type of caches, any number of levels of a cache hierarchy, and any amount and / or type of memory logic. The memory logic may determine in any technically feasible manner whether data associated with any type of memory access request resides within a cache included in the 2.5D GPM 230. When the 2.5D GPM 230 is attached to two MSMs, the memory logic may determine whether to forward a given memory access request via the 2.5D UHB link 528(1) or the 2.5D UHB link 528(2) in any technically feasible manner.

[0140] In some embodiments, the 2.5D DL MSM 234 may include, but is not limited to, any number and / or type of caches, any number of levels of a cache hierarchy, and any amount and / or type of memory logic. The levels of cache included in the 2.5D DL MSM 234 deepen the cache hierarchy, and therefore deepen the memory hierarchy of the 2.5D DL GPU 268 implemented in part in the 2.5D GPM. The memory logic may determine whether data associated with any type of memory access request resides within a cache included in the 3D DL MSM 234 in any technically feasible manner.

[0141] The cache and / or memory logic included in 2.5D GPM 230 and 2.5D DLMSM 234 may implement any cache organization and any number and / or type of cache related operations and / or optimization techniques in any technically feasible manner. For example, in some embodiments, 2.5D GPM 230 may implement any number and / or type of cache filtering techniques, cache compression techniques, any other cache optimization techniques, or any combination thereof.

[0142] Figure 6B It shows the Fig. 6A 612 of a top view of a 2.5D DL HBM enabled GPU 278. As shown, the 2.5D DL HBM enabled GPU 278 is included in a package 398(4). The 2.5D GPM 230 is implemented in one die that may be at or near the maximum possible size. The 2.5D DL MSM 234(1) is implemented in another die on one side of the 2.5D GPM 230, and the 2.5D DL MSM 234(2) is implemented in yet another die on the other side of the 2.5D GPM 230. The dies associated with the 2.5D DL MSM 234(1) and the 2.5D DL MSM 234(2) may each be at or near the maximum possible size.

[0143] As shown, in some embodiments, the 2.5D GPM 230, the 2.5D DLMSM 234(1), and the 2.5D DL MSM 234(2) implement two, one, and one 2.5D USB link interface 520, respectively. In some embodiments, a portion of the memory controller 480 is distributed adjacent to the available edge of the 2.5D DL MSM 234(1). The remaining memory controller 480 is distributed adjacent to the available edge of the 2.5D DL MSM 234(2). The HBM stacks 390(1)-390(10) are distributed close to the outer edge of the package 398(4), relatively adjacent to the memory controller 480.

[0144] Figure 6C It shows the Fig. 6A614 of a 2.5D DL HBM enabled GPU 278. As shown, in some embodiments, a portion of HBM stack 390, 2.5D DL MSM 234(1), 2.5D GPM 230, 2.5D DL MSM 234(2), and the remainder of HBM stack 390 are sequentially arranged in a planar manner on top of interposer 396(4). Interposer 396(4) is stacked on top of package 398(4). In some embodiments, package 398(4) overlaps interposer 396(4) on each side.

[0145] As shown, in some embodiments, the memory controller 480 is distributed at or near the outer edge of the 2.5D DL MSM 234(1) and the 2.5D DL MSM 234(2), and is connected to the HBM stack 390 through the interposer 396(4) via the HBM I / O connection 392. The 2.5D UHB link interface 520 included in the 2.5D DL MSM 234(1) is connected to one of the 2.5D UHB link interfaces 520 included in the 2.5D GPM 230 via a 2.5D UHB link 528(1). The 2.5D UHB link interface 520 included in the 2.5D DL MSM 234(2) is connected to a different one of the 2.5D UHB link interfaces 520 included in the 2.5D GPM 230 via a 2.5D UHB link 528(2). Notably, each of the 2.5D UHB link 528 ( 1 ) and the 2.5D UHB link 528 ( 2 ) is significantly shorter and faster than each of the HBM I / O connections 392 .

[0146] Figure 7 is a flow chart of method steps for servicing a memory access request associated with a processor core included in a combinable parallel processor according to various embodiments. Figure 1-6C Although the method steps are described in the context of a system for performing the method steps, those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present invention.

[0147] As shown, method 700 begins at step 702, where a processor core in a PPM issues a memory access request. At step 704, memory logic in the PPM causes a portion of a cache hierarchy included in the PPM to service the memory access request. At step 706, memory logic in the PPM determines whether the memory access request is satisfied. At step 706, if memory logic in the PPM determines that the memory access request is satisfied, method 700 terminates.

[0148] However, if at step 706, if the memory logic in the PPM determines that the memory access request cannot be satisfied, the method 700 proceeds to step 708. At step 708, the memory logic in the PPM routes the memory access request to the encapsulation onlink unless the memory logic in the PPM includes a memory switch 350 configured to route the memory access request to a disabling memory controller 352 included in the PPM. At step 710, if the memory logic in the PPM routes the memory access request to the disabling memory controller 352, the method proceeds to step 712. At step 712, the disabling memory controller 352 interacts with the PP memory coupled to the PPM to satisfy the memory access request. The method 700 then terminates.

[0149] However, if at step 710, the memory logic in the PPM does not route the memory access request to the disabling memory controller 352, the method proceeds directly to step 714. At step 714, the memory logic transmits the memory access request to the MSM via the encapsulated onlink. At step 716, if the MSM includes one or more additional cache levels, the MSM services the memory access request via the additional cache levels.

[0150] At step 718, if the memory access request is satisfied, then method 700 terminates. However, if the memory access request is not satisfied at step 718, then method 700 proceeds to step 720. At step 716, one of memory controllers 480(1)-480(M) interacts with a PP memory coupled to the MSM to satisfy the memory access request. Method 700 then terminates.

[0151] In summary, the disclosed techniques can be used to customize the memory systems of different instances of a reusable PPM to generate parallel processors dedicated to different application domains. To generate a parallel processor, an instance of a PPM is implemented on a single die and aggregated with zero or more other dies within a single package, each die enhancing the memory system of the PPM. In some embodiments, each of the other dies implements, but is not limited to, an instance of one of any number of MSMs or any number of PP memories (e.g., DRAM). In some embodiments, each MSM deepens the cache hierarchy of the PPM, modifies the number and / or type of memory interfaces, or any combination thereof.

[0152] In some embodiments, the 3D GPM is a PPM that includes, but is not limited to, a multithreaded processor core with a dedicated L1 cache, a distributed L2 cache, a disabling memory controller, an HBM memory interface, a memory switch, and one or more interfaces for one or more 3D UHB links. To specialize an instance of the 3D GPM for HPC, the memory switch is configured to route memory access requests associated with an L2 cache miss to the disabling memory controller. The HBM memory interface included in the resulting instance is attached to the HBM stack using 2.5D silicon interposer integration to generate a 3D HPC HBM-enabled GPU. In operation, when a disabling memory controller receives a memory access request via the memory switch, the disabling memory controller interacts with the HBM stack via the HBM memory interface to satisfy the memory access request.

[0153] In order to specialize an instance of a 3D GPM for DL, an instance of a 3D GPM is vertically attached to an instance of a 3DDL MSM via one or more 3D UHB links using high-density intra-die bonding. The 3D DL MSM provides a connection between the 3D GPM and the silicon interposer via through-silicon vias. The 3D DL MSM includes, but is not limited to, an interface for a 3D UHB link, an L3 cache, a memory controller, and an HBM memory interface. The memory switch included in the 3D GPM is configured to route memory access requests associated with L2 cache misses to the L3 cache via a 3D UHB link. The HBM memory interface included in the instance of the 3D DL MSM is attached to the HBM stack via a 2.5D silicon interposer integration to generate a GPU enabled with 3D DL HBM. In operation, the L3 cache serves memory access requests received via the 3D UHB link. The 3DDL MSM forwards memory access requests associated with L3 cache misses to the memory controller. When the memory controller receives a memory access request, the memory controller interacts with the HBM stack via the HBM memory interface to satisfy the memory access request.

[0154] In some embodiments, the 2.5D GPM is a parallel processor module that includes, but is not limited to, a multithreaded processor core with a dedicated L1 cache, a distributed L2 cache, an HBM memory interface, and one or more interfaces for one or more 2.5D UHB links. In order to specialize an instance of the 2.5D GPM for HPC, an instance of the 2.5D GPM is attached to one or two instances of a 2.5D HPC MSM via one or more 2.5D UHB links using high-density intra-die bonding. Each instance of the 2.5D HPC MSM includes, but is not limited to, a memory controller and an HBM memory interface. The HBM memory interface included in the instance of the 2.5D HPC MSM is attached to the HBM stack via a 2.5D silicon interposer integration to generate a 2.5D HPC HBM-enabled GPU. In operation, the 2.5D GPM routes memory accesses associated with L2 cache misses to a memory controller included in the instance of the 2.5D HPC MSM via a 2.5D UHB link. When the memory controller receives a memory access request, the memory controller interacts with the HBM stack via the HBM memory interface to satisfy the memory access request.

[0155] In order to specialize an instance of a 2.5D GPM for DL, an instance of a 2.5D GPM is attached to two instances of a 2.5D GPM via two or more 2.5D UHB links using high-density intra-die bonding. Each instance of a 2.5D DL MSM includes, but is not limited to, half of an L3 cache, a memory controller, and an HBM memory interface. The HBM memory interface included in the 2.5D DL MSM is attached to the HBM stack via a 2.5D silicon interposer integration to generate a GPU enabled with a 2.5D DL HBM. Relative to a 2.5D GPM, two 2.5D DL MSMs increase the total available die edge and can therefore be attached to an additional HBM stack. In operation, in an instance of a 2.5D DL MSM, a 2.5D GPM routes memory accesses associated with L2 cache misses to a memory controller via a 2.5D UHB link. When the memory controller receives a memory access request, the memory controller interacts with the HBM stack via the HBM memory interface to satisfy the memory access request.

[0156] At least one technical advantage of the disclosed technology over the prior art is that, using the disclosed technology, the performance of parallel processor modules implemented on a single die can be effectively customized for different memory bandwidth requirements representing different application domains. Specifically, for workloads with relatively high memory bandwidth requirements, a parallel processor can be configured with a processor module that can access additional cache memory via one or more on-package, ultra-high bandwidth links, which can reduce DRAM access delays and memory bandwidth bottlenecks typically experienced by aggregated architecture parallel processors. Further, because the additional cache memory is implemented on a separate die, there is no need to remove circuits from the processor module or restrict circuits within the processor module to make room for the additional cache memory. These technical advantages provide one or more technical improvements over prior art methods.

[0157] In some embodiments, a parallel processor includes: a parallel processor module implemented within a first die, wherein the parallel processor module includes multiple processor cores and multiple cache memories; and a memory system module implemented within a second die and coupled to the parallel processor module via an on-package link, wherein the memory system module includes a first memory controller for accessing a first dynamic random access memory DRAM.

[0158] 2. The parallel processor of clause 1, wherein the plurality of cache memories comprises at least one level 1 cache and a level 2 cache.

[0159] 3. The parallel processor of clause 1 or 2, wherein the memory system module further comprises a level 3 cache.

[0160] 4. The parallel processor of any of clauses 1-3, wherein the plurality of cache memories comprises a first portion of a cache hierarchy and the memory system module comprises a second portion of the cache hierarchy.

[0161] 5. The parallel processor of any of clauses 1 to 4, wherein the parallel processor module further comprises a second memory controller and a switch configured to disable the second memory controller.

[0162] 6. The parallel processor of any of clauses 1 to 5, wherein the first die and the second die are vertically stacked within a single package.

[0163] 7. The parallel processor of any of clauses 1 to 6, wherein the first die and the second die are stacked laterally within a single package.

[0164] 8. The parallel processor of any of clauses 1 to 7, wherein the on-package link comprises an on-package 3-dimensional ultra-high bandwidth link or an on-package 2.5-dimensional ultra-high bandwidth link.

[0165] 9. The parallel processor of any of clauses 1 to 8, wherein a third die comprises a first DRAM, the third die being coupled to the second die via 2.5-dimensional silicon-on-interposer integration.

[0166] 10. The parallel processor of any of clauses 1 to 9, wherein the first DRAM is implemented in a third die, the third die being included in a vertical stack of DRAM dies.

[0167] 11. A parallel processor according to any of clauses 1-10, wherein a first processor core is included in the plurality of processor cores, the first processor core comprising at least one of a tensor core or a ray tracing core.

[0168] 12. The parallel processor of any of clauses 1 to 11, wherein a level 2 cache is included in the plurality of cache memories, the level 2 cache comprising a coherence point for the plurality of processor cores.

[0169] 13. In some embodiments, a computer-implemented method for servicing memory access requests associated with multiple processor cores comprises: determining that data corresponding to a first memory access request does not reside within multiple cache memories, the multiple cache memories residing within a first die, wherein the first memory access request is associated with a first processor core implemented within the first die; and in response, transmitting the first memory access request to a memory system module via a package on-link, wherein the memory system module is implemented within a second die and includes a first memory controller.

[0170] 14. A computer-implemented method according to clause 13, wherein transmitting the first memory access request to the memory system module includes: transmitting the first memory access request to a switch included in the first die, the switch routing the memory access request to the memory system module via the link on the package instead of routing the memory access request to a memory controller on the die.

[0171] 15. The computer-implemented method according to clause 13 or 14 further includes: in response to receiving the first memory access request, determining that data associated with the first memory access request does not reside in a last level cache included in the memory system module, and accessing the data associated with the first memory access request via the first memory controller.

[0172] 16. The computer-implemented method of any of clauses 13-15, further comprising: in response to receiving the first memory access request, accessing data associated with the first memory access request via a last level cache included in the memory system module.

[0173] 17. The computer-implemented method of any of clauses 13-16, further comprising: evicting a first cache line from a level 2 cache included in the plurality of cache memories; and writing the first cache line to a level 3 cache included in the memory system module.

[0174] 18. The computer-implemented method of any of clauses 13-17, wherein a level 2 cache in the plurality of cache memories comprises a coherence point for the plurality of processor cores.

[0175] 19. The computer-implemented method of any of clauses 13-18, further comprising: receiving, via the encapsulated on-link, data accessed via the first memory access request.

[0176] 20. A computer-implemented method according to any of clauses 13-19, wherein determining that data corresponding to the first memory access request does not reside in the multiple cache memories includes: determining that the data corresponding to the first memory access request does not reside in a first-level cache, the first-level cache being included in the multiple cache memories and associated with the first processor core; and determining that the data corresponding to the first memory access request does not reside in a second-level cache, the second-level cache being included in the multiple cache memories.

[0177] Any and all combinations of any claim elements recited in any claim and / or any elements described in this application in any manner are within the scope of embodiments and protection.

[0178] The description of the various embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0179] Aspects of the present embodiment may be embodied as a system, method or computer program product. Therefore, aspects of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.) or an embodiment of a combination of software and hardware, all of which are collectively referred to as "modules", "systems" or "computers" herein. In addition, any hardware and / or software technology, process, function, component, engine, module or system described in the present disclosure may be implemented as a circuit or a group of circuits. In addition, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media, which contains computer-readable program code.

[0180] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples (non-exhaustive) of computer-readable storage media may include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, a flash memory, an optical fiber, a portable optical disc read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this document, a computer-readable storage medium may be any tangible medium that may include or store a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0181] Aspects of the present disclosure are described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. When the instruction is executed by the processor of a computer or other programmable data processing device, it is possible to implement the function / action specified in one or more boxes of the flowchart and / or block diagram. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, application-specific processors or field programmable gate arrays.

[0182] The flowchart and block diagram in the figure show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, segment or part of the code, and the code includes one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative embodiments, the functions indicated in the box may not occur in the order indicated in the figure. For example, two boxes displayed in succession can actually be executed roughly at the same time, or these boxes can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented by a system based on special-purpose hardware, which performs a specified function or action or a combination of special-purpose hardware and computer instructions.

[0183] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, the scope of which is determined by the appended claims.

Claims

1. A parallel processor, comprising: a parallel processor module implemented in a first die, wherein the parallel processor module includes a plurality of processor cores, a plurality of cache memories, and a first memory controller, the first memory controller accessing a first dynamic random access memory DRAM for a first application and being prohibited from accessing the first DRAM for a second application; as well as A memory system module is implemented within a second die and coupled to the parallel processor module via an on-package link, wherein the memory system module includes a second memory controller for accessing the first DRAM. 2 . The parallel processor according to claim 1 , wherein the plurality of cache memories include at least one level 1 cache and a level 2 cache.

3. The parallel processor of claim 1, wherein the memory system module further comprises a level 3 cache.

4. The parallel processor of claim 1, wherein the plurality of cache memories comprises a first portion of a cache hierarchy and the memory system module comprises a second portion of the cache hierarchy. 5 . The parallel processor of claim 1 , wherein the parallel processor module further comprises a switch configured to disable the first memory controller.

6. The parallel processor of claim 1, wherein the first die and the second die are vertically stacked within a single package.

7. The parallel processor of claim 1, wherein the first die and the second die are stacked laterally within a single package. 8 . The parallel processor of claim 1 , wherein the on-package link comprises an on-package 3-dimensional ultra-high bandwidth link or an on-package 2.5-dimensional ultra-high bandwidth link.

9. The parallel processor of claim 1, wherein a third die comprises a first DRAM, the third die being coupled to the second die via 2.5-dimensional silicon-on-interposer integration.

10. The parallel processor of claim 1, wherein the first DRAM is implemented in a third die, the third die included in a vertical stack of DRAM dies.

11. The parallel processor of claim 1, wherein a first processor core is included in the plurality of processor cores, the first processor core comprising at least one of a tensor core or a ray tracing core.

12. The parallel processor of claim 1, wherein a level 2 cache is included in the plurality of cache memories, the level 2 cache comprising a coherency point for the plurality of processor cores.

13. The parallel processor of claim 1, wherein the first application comprises a high performance computing (HPC) application associated with a first amount of memory bandwidth demand.

14. The parallel processor of claim 13, wherein the second application comprises a deep learning (DL) application associated with a second amount of memory bandwidth requirements greater than the first amount.

15. A computer-implemented method for servicing memory access requests associated with a plurality of processor cores, the method comprising: determining that data corresponding to a first memory access request does not reside within a plurality of cache memories residing within a first die, wherein the first memory access request is associated with a first processor core implemented within the first die, and wherein the first die includes a first memory controller that accesses a first dynamic random access memory DRAM for a first application and is prohibited from accessing the first DRAM for a second application; as well as In response, the first memory access request is transmitted to a memory system module via an on-package link, wherein the memory system module is implemented within a second die and includes a second memory controller that accesses the first DRAM.

16. The computer-implemented method of claim 15, wherein transmitting the first memory access request to the memory system module comprises: The first memory access request is transmitted to a switch included in the first die, and the switch routes the memory access request to the memory system module via the on-package link instead of routing the memory access request to the first memory controller.

17. The computer-implemented method of claim 15, further comprising: In response to receiving the first memory access request, determining that data associated with the first memory access request does not reside within a last level cache included in the memory system module, and accessing the data associated with the first memory access request via the second memory controller.

18. The computer-implemented method of claim 15, further comprising: In response to receiving the first memory access request, data associated with the first memory access request is accessed via a last level cache included in the memory system module.

19. The computer-implemented method of claim 15, further comprising: evicting a first cache line from a level 2 cache included in the plurality of cache memories; as well as The first cache line is written to a level 3 cache included in the memory system module.

20. The computer implemented method of claim 15, wherein a level 2 cache included in the plurality of cache memories comprises a coherency point for the plurality of processor cores.

21. The computer-implemented method of claim 15, further comprising: Data accessed via the first memory access request is received via the encapsulated onlink.

22. The computer-implemented method of claim 15, wherein determining that data corresponding to the first memory access request does not reside within the plurality of cache memories comprises: determining that data corresponding to the first memory access request does not reside in a first level cache included in the plurality of cache memories and associated with the first processor core; as well as It is determined that data corresponding to the first memory access request does not reside in a second level cache included in the plurality of cache memories.

Images