Synchronous memory management unit in multi-grain processor architecture

Abstract

The invention relates to a synchronous memory management unit in a multi-grain processor architecture, and particularly describes support of distributed graphics and computing engines in a multi-grain parallel processing system (e.g., a multi-grain graphics processing unit (GPU)) architecture, and synchronous memory management in such an architecture. Each die has a memory management unit (MMU). For messages originating from any die, the specified MMU serializes processing of at least one memory-related message type, and processing of at least some memory-related message types is performed locally on the original die.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 566,359, filed on March 17, 2024, the entire contents of which are incorporated herein by reference. This application is also related to U.S. Application No. 18 / 606,924, filed on March 15, 2024, entitled “Method and Apparatus for Supporting Distributed Graphics and Compute Engines and Synchronization in Multi-Die Parallel Processor Architectures,” the entire contents of which are also incorporated herein by reference. Technical Field

[0003] The present technology generally relates to multi-dielet processors. More specifically, the technology herein relates to distributing graphics and compute engines across multiple dies and synchronizing memory operations in such processors. Background Art

[0004] The demand for processors with extensive parallel processing capabilities, such as graphics processing units (GPUs), continues to grow. The processing demands placed on such processors are also rapidly increasing in terms of complexity, workload size, and throughput rate.

[0005] The need for GPUs to handle rapidly growing workloads of increasing complexity, size, and throughput has driven the need to pack more and more components onto the GPU semiconductor die—typically a thumb-sized square of flat semiconductor material (such as silicon) cut from a wafer, on which circuitry is fabricated. The more components packed onto a die, the more functionality the integrated circuit chip package that includes the die can provide. Consequently, chip designers have been working to pack more and more components, and a variety of components, onto each physical die.

[0006] There are physical limits on how many components can be packed onto a single die. For example, packing more transistors generates more heat, which can damage the chip unless cooling is properly managed. Having more components (usually smaller components) on a single die can also make interconnects difficult to implement and can cause signal issues on the interconnects, among other things. Furthermore, despite Moore's Law, some components may have a minimum physical size beyond which they cannot be easily miniaturized further.

[0007] Therefore, as workload demands continue to grow, efforts to pack more components onto a single die may encounter insurmountable limitations in the number of components, component types, or physical size of the processor. In addition to packing a greater number and type of components onto a single die to meet growing workload demands, other approaches to expanding the processing capacity and capabilities of processors such as GPUs can be explored. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A multi-die GPU is shown according to some embodiments of the present disclosure.

[0009] Figure 2 Another multi-die GPU is shown in accordance with some embodiments of the present disclosure.

[0010] Figure 3 Schematically illustrates an example of a translation lookaside buffer (TLB) hierarchy arrangement in each die of a multi-die GPU according to some embodiments of the present disclosure.

[0011] Figure 4 A multi-die processing system (eg, Figures 1 to 3 Flowchart of the binding process in the system shown in ).

[0012] Figure 5 A multi-die GPU (e.g., Figures 1 to 3 Flowchart of TLB invalidation in the system shown in FIG.

[0013] Figure 6A A multi-die GPU (e.g., Figures 1 to 3 Flowchart of a page table walk in the system shown in .

[0014] Figure 6B An example flow chart illustrating the process of address translation service (ATS) requests and responses in a multi-die GPU according to some embodiments of the present disclosure is shown.

[0015] Figure 6C A flow chart of a fault reporting process that may be used in a multi-die GPU according to some embodiments of the present disclosure is shown.

[0016] Figure 7 An example GPU on a die of a multi-die GPU is shown with its frame buffer hub (FBHUB) and high-speed hub (HSHUB) identified relative to some of their interconnects, in accordance with some embodiments of the present disclosure.

[0017] Figure 8An example parallel processing unit of a GPU is shown in accordance with some embodiments.

[0018] Figure 9A According to some embodiments Figure 8 An example general processing cluster (GPC) within a parallel processing unit.

[0019] Figure 9B Shown Figure 8 An example memory partitioning unit of a parallel processing unit.

[0020] Figure 10A FIG. 1 shows an MMA state machine circuit system according to some embodiments. Figure 9A An example of a streaming multiprocessor (SM).

[0021] Figure 10B Conceptually illustrates the process of performing a multi-processor (e.g., Figure 10A The four sub-partitions implemented in the SM) shown in .

[0022] Figure 11A is used Figure 8 An example conceptual diagram of a processing system implemented with a parallel processing unit (PPU).

[0023] Figure 11B is a block diagram of an exemplary system in which the various architecture and / or functionality of the various previous embodiments may be implemented. DETAILED DESCRIPTION

[0024] To address the problem of how to squeeze more and more components onto a single die, it has been proposed to interconnect multiple physical dies ("dies") together to form larger, more complex processing systems (e.g., GPUs). In this disclosure, each of the multiple dies in such a larger processing system is referred to as a "die." Such larger processing systems are referred to herein as "multi-die processing systems." Designing processing systems that extend beyond the confines of a single manufactured die, as is the case with multi-die processing systems, provides a new path to scalability and eliminates some previously existing physical limitations.

[0025] For ease of description, the present disclosure refers to a multi-die GPU having two independently manufactured dies, where each die contains one or more streaming multiprocessors (SMs), specialized and general-purpose hardware engines, and associated routing components for executing applications on behalf of processes launched by the CPU or GPU. In some embodiments, the dies may be identical (i.e., identical or substantially identical hardware). In this case, the multi-die GPU may have duplicate components between the two dies. Depending on the application, such duplicate components may be used or may be redundant. In some embodiments, the GPUs on the dies may not be identical, and some GPUs may have different sets of hardware units. Example embodiments are necessary to accommodate different scenarios in which multiple GPUs with different capabilities are combined.

[0026] However, with the formation of such larger GPUs, it is often necessary or effective to protect software from having to understand the physical layout of such larger GPUs. For example, such protection may be necessary to ensure that multi-die GPUs can interoperate in many usage scenarios without requiring extensive modifications to customize the software stack for each scenario. Such protection of software from having to understand the detailed organization of the multi-die GPU may, at least in some cases, allow different multi-die GPUs to be retrofitted into existing software (e.g., GPU driver software). Such protection may also "future-proof" multi-die designs by ensuring that software can run with such multi-die GPUs regardless of the specific design (e.g., number of dies, type and number of hardware engines on each die, etc.).

[0027] The multi-die processing system of an exemplary embodiment is configured to be viewed completely, or at least largely, as a single processor by software (e.g., driver software running on a CPU of a system including the multi-die processing system). One aspect of presenting the multi-die processing system as a single processor to software includes presenting a single logical memory management unit (MMU) to the software. Because each die requires memory translation and protection, and the die is to be exposed to software as a single processor, the multi-die processing system is configured to synchronize multiple MMUs (particularly the highest-level MMU on each die) with each other to form a single logical MMU.

[0028] If the hardware mechanisms described in this disclosure did not provide synchronization between the MMUs of multiple dies, software would have to provide the necessary coordination between the individual MMUs. This would also require developers to be specifically aware that they are dealing with multiple GPUs in a manner similar to that required for a peer-to-peer combination of multiple independent GPUs. In addition to negatively impacting performance, this software-based coordination would also incur high development costs.

[0029] Figure 11 shows a multi-die GPU 100 according to some embodiments of the present disclosure. The multi-die GPU 100 includes two dies, namely, die-1 102a and die-2 102b. Each die includes a GPU, where die-1 102a has GPU-1 104a and die-2 102b has GPU-2 104b. In some embodiments, GPU-1 and GPU-2 may each correspond to a processor such as Figure 8 A parallel processing unit (PPU) 800 is shown.

[0030] The die can be interconnected via one or more high-speed links 116 for exchanging commands, data, and / or control. In some embodiments, the high-speed links are NVLink interfaces. In embodiments where a multi-die GPU 100 includes two die, the chip-to-chip high-bandwidth interface on each die can be one or more high-speed links 116. Embodiments are not limited to the number of die included in a multi-die GPU, or the number of GPUs on each die. A multi-die GPU can also have any number of die without GPUs.

[0031] Each die includes multiple hardware engines. In the illustrated embodiment, die-1 includes hardware engine-1 106a and hardware engine-2 108a, and die-2 includes hardware engine-1 106b and hardware engine-2 108b. The hardware engines may include computing units, graphics units, encoding / decoding units, encryption / decryption units, etc. As described above, in some embodiments, die-1 and die-2 may each include similar hardware engines. Figure 8 The processor of the PPU in the embodiment of the present invention includes multiple general processing clusters (GPCs) (e.g., Figure 9A ), and each GPC includes Figure 9A and Figure 10A . SMs can execute both compute and graphics workloads and can be considered hardware engines. When "hardware engines" are referenced herein, this includes, but is not limited to, hardware engines used in graphics and compute applications, video encoding or decoding, units in GPC and SYS clusters, and units with DMA capabilities. Embodiments are not limited to any specific number of hardware engines and / or any type of hardware engines on each GPU.

[0032] Each die also includes at least one memory management unit (MMU). Similar to the CPU, the GPU uses an MMU to translate virtual addresses used by a program or process into physical addresses in memory. This is because programs often use a larger virtual address space than the physical memory available on the GPU. The MMU acts as a bridge, mapping these virtual addresses to their corresponding physical locations, enabling efficient memory access for various GPU operations. The MMU in the GPU facilitates memory protection by enforcing access permissions. They can control which parts of memory different processes running on the GPU can access. This helps prevent unauthorized access, data corruption, and potential security vulnerabilities. In addition, the MMU can help manage memory allocation and usage for different tasks running concurrently on the GPU.

[0033] In some embodiments, each die may have one or more additional MMUs, each configured to service memory-related requests at the component level on the die. For example, some embodiments may include a GPC MMU for each GPC in a GPU (see Figure 9A ) and / or input / output MMU (IO MMU). The highest level MMU on each die is referred to herein as a hub MMU or "HUBMMU". In some embodiments, the HUBMMU is a GPU MMU ("GMMU") with a separate virtual address space and page table for each process. In the illustrated embodiment, HUBMMU-1 110a is located on die-1 and HUBMMU-2 is located on die-2. Each HUBMMU enables the hardware engines on its die to access memory by providing virtual address translation, memory protection, etc. All HUBMMUs of a multi-die GPU (HUBMMU-1 and HUBMMU-2 in this example) collaborate to synchronize memory accesses for entities on all dies and present a single logical MMU to software.

[0034] The ability to present a monolithic view of multi-die GPU 100 is provided so that external entities (such as, for example, driver software 122 executing on CPU 120 communicating with multi-die GPU 100 via interface 118 (e.g., a PCI interface)) can view multi-die GPU 100 as a single monolithic GPU. In other words, the monolithic GPU view enabled by the circuitry in multi-die GPU 100 (including the circuitry of the HUBMMU) allows software (e.g., driver software 122) to be aware of the specific structure of multi-die GPU 100. U.S. Application No. 18 / 606,924, entitled “Method and Apparatus for Supporting Distributed Graphics and Compute Engines and Synchronization in Multi-Die Parallel Processor Architectures,” filed on March 15, 2024, and incorporated herein by reference in its entirety, describes a hardware implementation in a multi-die GPU for mapping between its die-local hardware engine identifiers and a globally unique engine identifier, thereby enabling software to view the multi-die GPU as a single monolithic GPU.

[0035] Before describing the mechanism for synchronizing the MMUs across all dies of a multi-die GPU, as provided in an embodiment of the present disclosure, the importance of the MMU can be illustrated by describing its role in memory access by hardware engines. Consider an example process running on a hardware engine (e.g., SM) on die-1 that needs to access an object in memory. The process first generates a virtual address pointing to the object's location in GPU memory (local / remote) or CPU memory. This virtual address is then used in an instruction to access the object. The process or the corresponding hardware engine sends a request containing the generated virtual address to the MMU. The MMU can look up the virtual address in its translation lookaside buffer (TLB), which is a cache that stores recently translated addresses, thereby increasing translation speed. If the virtual address is found in the TLB, the corresponding physical address in GPU memory (local / remote) or CPU memory is retrieved from the TLB entry. This physical address points to the actual location of the object being looked up in GPU memory (local / remote) or CPU memory. If the virtual address is not found in the TLB, a TLB miss occurs. The MMU then performs a more complex lookup process using the page table. A page table is a data structure that maps virtual addresses to physical addresses in a larger memory area. The MMU accesses the page table stored in memory to identify the physical address that corresponds to the virtual address. This process may involve traversing multiple levels of page tables, depending on the system's memory organization. When the page table traversal is complete, the MMU obtains the physical address of an object in GPU memory (local / remote) or CPU memory. The requesting hardware engine receives the physical address from the MMU and uses it to access an object in GPU memory (local / remote) or CPU memory.

[0036] In the example process described above, the MMU can also determine whether the requesting process and / or hardware engine is authorized to access the requested memory. Unauthorized memory access attempts may cause the MMU to generate a fault to be reported to the software and / or CPU. Therefore, in this way, the MMU is a key component of the GPU's functioning.

[0037] Figure 2 Another multi-die GPU 200 is shown, in accordance with some embodiments. Figure 2 Example communications involving HUBMMUs in an example 2 die handling system 200 are schematically illustrated.

[0038] Multi-die GPU 200 includes die-1 202a and die-2 202b. Die-1 includes HUBMMU 210a, high-speed hub (HSHUB) 214a, and frame buffer hub (FBHUB) 212a. Die-2 includes HUBMMU 210b, HSHUB 214b, and FBHUB 212b.

[0039] The binding table is a data structure used to store information about the association between virtual memory regions and physical memory frames assigned to different processes running on the GPU. Each entry in the binding table may include: the virtual address range used by the process, the starting address of the physical memory frame allocated for the corresponding virtual address range, and the access rights (read, write, execute) associated with the memory region used for protection.

[0040] The Page Table Binding Cache (PDB Cache) is a hardware cache located in the HUBMMU. It can store the currently used page table roots of the virtual-to-physical addresses of various GPU processes. A TLB miss requires retrieving the page table roots of the process and performing a page table walk using the retrieved roots.

[0041] When a program attempts to access data using a virtual address, the MMU first checks the page table binding cache. If a translation (a mapping between virtual and physical addresses) for that specific address is found in the cache, the MMU can efficiently translate the address and grant access to the corresponding physical memory location. This avoids the need to access the home page table (which is typically slower than the cache), thereby reducing memory access latency and improving overall performance.

[0042] The binding table and / or PDB cache in the HUBMMU 210a can be accessed by a hub translation lookaside buffer (HUBTLB) or a process that accesses the HUBTLB and causes a TLB miss. When the miss is caused by a respective lookup of another TLB, the other TLBs (e.g., L2 TLB and Link / CE TLB) can access the HUBTLB. Due to a binding table lookup or PDB cache lookup or page table walk, the TLB can update its TLB entries using entries provided by the HUBMMU. In some embodiments, as a result of a binding table and PDB cache lookup and a page table walk, the TLB can update its TLB entries using entries provided by the HUBMMU.

[0043] The HUBMMU on one die communicates with other HUBMMUs on other dies in a multi-die GPU regarding memory requests, memory responses, bind requests and responses, TLB invalidation requests and responses, and various acknowledgements. In some embodiments, these communications can be made to other dies via FBHUB 212a.

[0044] ATS requests, ATS responses, ATS shutdown / deactivation (ATSD), and related acknowledgements may be communicated between the HUBMMU and the high-speed hub (HSHUB) 214a.

[0045] The communication between the HUBMMU 210 b and the various TLBs, HSHUB 214 b and FBHUB 212 b of the second die 202 b may be the same or similar to that of the HSHUB 214 a of the first die 202 a .

[0046] In addition, HUBMMU-1 and HUBMMU-2 can be configured to exchange messages, such as bind requests and acknowledgments, TLB invalidation requests and acknowledgments, fault reports, ATS requests, virtual address bus / breakpoint (VAB) dump requests, and the like. In some embodiments, at least some messages sent from one HUBMMU to another HUBMMU are transmitted over a connection 224 that is distinct from the die-to-die connection (e.g., 116). In some embodiments, connection 224 is a dedicated HUBMMU-to-HUBMMU connection over the die-to-die connection (e.g., 116).

[0047] In the illustrated two-die processing system 200, Die-1 202a and Die-2 202b have different types of network interfaces, with Die-1 including an NVLINK interface 226 and a PCIE interface 228, and Die-2 including a chip-to-chip interface 230. In an example embodiment, the PCIE interface 228 can be used for memory access, while the NVLINK interface 226 and the chip-to-chip (C2C) interface 230 can be used to connect to the CPU and other devices (e.g., other GPUs, multi-die processing systems, etc.). In some embodiments, Die-1 and Die-2 can be identical and have the same network interface hardware, but one or more network interfaces on each die may be disabled or unused. For example, although Die-1 and Die-2 may both include NVLINK, PCIE, and C2C connectors, the C2C connector on Die-1 may be inactive, while the NVLINK and PCIE interfaces on Die-2 may be inactive.

[0048] exist Figure 1 and Figure 2 In the illustrated embodiment, each die includes a HUBMMU. However, in some embodiments, at least some of the die may include component-level MMUs, such as one or more GPC-level MMUs ("GPCMMUs") and high-speed hub-level MMUs ("HSHUBMMUs"). In some embodiments, all three types of MMU modules (GPCMMU, HUBMMU, and HSHUBMMU) are active in both die. The GPCMMU and HSHUBMMU in one die do not communicate with the GPCMMU and HSHUBMMU in another die. Instead, the GPCMMU / HSHUBMMU in one die communicates with the HUBMMU on the same die, and then the HUBMMU in one die communicates with the HUBMMU in the other die.

[0049] In some embodiments, HUBMMU to HUBMMU communication can be performed over a dedicated bus. The dedicated bus can be used to synchronize the MMUs in the two dies for binding operations, TLB invalidation, fault reporting, security vector updates, and ATS request processing. Although the MMU dedicated bus is used for synchronization commands as described above, ATS responses, ACK communications, other memory commands, and data transfers (for GMMU page table traversal memory access data, fault grouping, VAB data dump to memory) can be sent over the FBHUB of the corresponding die. This can provide faster HUBMMU synchronization.

[0050] Thus, in some embodiments, both die of a 2-die processing system have the HUBMMU active, and communication between them is minimal.Example die MMU operations may include several message types and request / response types.

[0051] All engines and other clients on a die are served by the MMU on that die. The HUBMMU in one die performs page table walks independently without any information from the HUBMMUs on other dies. The TLBs in one die do not communicate with the TLBs in other dies. For example, in some embodiments, each die may have a HUBTLB and one or more component-level TLBs (e.g., any of the uTLB, gpcL1TLB, gpcL2TLB, and LinkTLB), but no TLB on one die communicates with any TLB on other dies.

[0052] Engines and other clients send bind requests and TLB invalidation requests to the HUBMMU on the same die. The HUBMMUs communicate with each other to keep the MMUs synchronized, but both dies independently execute the bind operation in parallel to generate a combined acknowledgement to the requester.

[0053] In some embodiments, a bind operation is always initiated by the primary die. The secondary die forwards its local bind request to the primary die. A TLB invalidation operation may also always be initiated by the primary die. The secondary die forwards its local invalidation request to the primary die. Fault reporting is always initiated by the primary die. Faults from the secondary die are forwarded to the primary die. A VAB memory dump is always initiated by the primary die. VAB dump requests from the secondary die are forwarded to the primary die. ATS / ATSD request, response, and acknowledgment communications (acknowledgment communications with the HSHUB) are always completed by the secondary die. The primary die sends its ATS request to the designated secondary die's HUBMMU for forwarding to the HSHUB (and ultimately to the system MMU (SMMU)).

[0054] The PDB cache in one die is not a mirror of the PDB cache in other dies and works independently when processing bind commands, page table walks, and invalidations. For bind commands, the PDB ID associated with the same PDB may be different on different dies.

[0055] As described above, some MMU operations are performed by only one die, while the MMUs of other dies work in sync with the die performing those operations. Furthermore, in at least some embodiments, each die can be configured to operate as an independent GPU (e.g., for production and testing). Thus, embodiments of the present disclosure enable each die in a multi-die processing system to know its die identifier and operational role.

[0056] Each HUBMMU can determine whether it is configured as a primary die, an auxiliary die, or a standalone die in its operational role, for example, based on the configuration settings of one or more fuses and / or registers. Each HUBMMU can also determine its die identifier based on the settings of one or more fuses and / or registers. The die identifier is used to uniquely identify a die within a multi-die processing system.

[0057] In some embodiments, HUBMMU to HUBMMU communication may be achieved via a connection made through a die-to-die connection crossbar. The HUBMMU to HUBMMU connection may be configured to carry packets for a specific set of operations: MMU binding (e.g., original binding packets from the client), TLB invalidation, ATS request (ATR), ATS invalidation (ATSD), ATS response (ATRsp) - data packets from secondary to primary, fault (internal) packets, VAB dump request, VAB (internal) mask-data packets, and confirmation of the above operations. The HUBMMU to HUBMMU connection may be reserved specifically for the above-identified packets. In some embodiments, the connection may be shared with other components (e.g., FBHUB) to enable these components to also transmit selected types of packets.

[0058] In some embodiments, the HUBMMU includes a buffer for queuing packets to be transmitted and / or received. In some embodiments, the HUBMMU can implement blocking threads and non-blocking threads to transmit messages. For example, a blocking thread can transmit binds, TLB invalidations, blocking faults, VABs, ATSs, and related acknowledgments, while a non-blocking thread can be used to transmit non-blocking faults and related acknowledgments.

[0059] Figure 3 Schematically illustrates an example of a hierarchical TLB arrangement in each die of a 2-die processing system according to some embodiments. Figure 3In the 2-die processing system, Die-1 302a is configured as a primary die and Die-2 302b is configured as a secondary die.

[0060] The master die 302a includes a HUBMMU (HUBMMU-1) 310a, which communicates with an FBHUB (FBHUB-1) 312a and an HSHUB (HSHUB-1) 314a, also located in the die. HUBMMU-1 310a includes a fill unit 332a, which is configured to communicate with a die TLB hierarchy 334a. Fill unit 332a is also configured to communicate with the fill unit of HUBMMU (HUBMMU-2) 310b in die-2 302b via interface 324. Fill unit 332a may include a PDB cache and a binding table. Fill unit 332a may also include a fault state machine and / or ATS wrapper functionality.

[0061] The die TLB hierarchy 334a may include a hub-level TLB and a level 1 (L1) TLB for each GPC on the die to speed up access to the corresponding L1 cache. The die TLB hierarchy 334a may also include one or more level 2 (L2) TLBs for one or more GPCs on the die, and / or an L1 TLB for a corresponding link (e.g., a network interface). TLB invalidations from the HUBMMU are communicated to the corresponding local (e.g., component-level) TLBs in the die TLB hierarchy, and the corresponding local TLBs return corresponding acknowledgments.

[0062] Die-2 302b may have the same configuration as the TLB hierarchy, including die TLB hierarchy 334b. The configuration of HUBMMU 310b, fill unit 332b, FBHUB 312b, and HSHUB 314b may be the same or similar to the corresponding components in die-1 302a.

[0063] Figure 4 A multi-die processing system (eg, Figures 1 to 3 FIG4 is a flow chart 400 of a bonding process in a system shown in FIG4. In the configuration shown, Die-1 is configured as primary and Die-2 is configured as secondary. The multi-die processing system is configured so that the primary die performs the bonding operation.

[0064] The bind operation links an engine (e.g., graphics engine, copy / DMA engine (CE), security / integrity engine, etc.) to an instance block in the HUBMMU so that the HUBMMU can translate subsequent memory requests from that engine (or from that engine's client). The instance block holds the corresponding page table pointer.

[0065] In some embodiments, the HUBMMU does not distinguish which engine can be bound to which die, although in some cases only the VEID (graphics engine), base address register (BAR) engine, CE and PMA engines are expected to be accessible in both dies.

[0066] The HUBMMU for each die can include a binding table and a PDB cache. The binding table and PDB cache in each HUBMMU can be configured (e.g., resized) to support all engines on all die in a multi-die GPU. However, the binding table and PDB cache in one die are configured to operate independently of the binding tables and PDB caches in other die. Therefore, the same PDB ID may point to different page tables in different die. Binding table entries can be indexed by engine ID, so entries with the same index will have the same binding information in all die.

[0067] The primary die is configured to initiate the bonding process. The secondary die forwards the bonding packet to the primary die to initiate the bonding process.

[0068] When a client (e.g., a hardware engine) on Die-1 or Die-2 issues a bind request (e.g., 402a and 402b), the request is sent to the HUBMMU on the same die. On Die-1, the HUBMMU (HUBMMU-1) is configured to handle the bind request because Die-1 is configured as the primary. Conversely, when the HUBMMU (HUBMMU-2) on Die-2 receives the bind request 402b from the local client, Die-2 forwards the bind request (402c) to HUBMMU-1 because Die-2 is configured as the secondary.

[0069] The bind request packet may be generated by the engine or as a result of the engine writing to a register (eg, a privileged register).

[0070] Since HUBMMU-1 receives bind requests from all other die of the multi-die processing system, HUBMMU-1 serializes the bind requests 404. For example, the incoming bind requests may be stored in a queue.

[0071] HUBMMU-1 may then arbitrate (406) between the serialized bind requests to select a bind request to process.

[0072] At 408, the selected bind request is processed. Processing the bind request may include allocating a physical memory region for the requesting client and determining a virtual address for the allocated region. A new entry may be added to the binding table, or an existing corresponding entry may be updated in the binding table. The entry may include a mapping between the virtual address and the allocated memory, as well as associated permissions.

[0073] At 410, HUBMMU-1 transmits the binding information obtained at 408 (such as, for example, the contents of the binding table entry) to other dies. In some embodiments, the binding information may include the engine ID of the sender of the binding request, PDB cache information, and a mapped instance block pointer. Otherwise, the binding table and PDB cache may become out of sync.

[0074] At 412, HUBMMU-2 receives the binding information from HUBMMU-1 and updates its own binding table accordingly. For example, HUBMMU-2 adds an entry to its binding table.

[0075] Each die updates its local TLB at 414 (414a and 414b) after its binding table is updated, and the corresponding TLB confirms 416 (416a and 416b) the update. The TLB update can be initiated by propagating the binding information to all related TLBs on the same die through the HUBMMU.

[0076] After HUBMMU-1 receives confirmation from its local TLB update and other die (die-2 returns confirmation 418), HUBMMU-1 notifies the sender of the bind request that the bind request has been completed and returns the binding information to the requester. For example, if the selected bind request originated from die-1, an confirmation 420a including the virtual address of the allocated memory and optional other parameters is sent to the requesting local client. If the selected bind request originated from die-2, an confirmation 420b including the virtual address of the allocated memory and optional other parameters is sent to the HUBMMU on the requesting die, which forwards the confirmation and virtual address, etc. to the requesting local client. For example, HUBMMU-1 transmits confirmation 420b to HUBMMU-2, which then sends it (422c) to the requesting local client on die-2.

[0077] In many cases, a bind operation is the first requirement before a process can be assigned work. The bind operation provides a link between the memory system (in this case, the page table) and the context and engine that is executing the work. Therefore, after the engine initiates the bind and the confirmation is received, the work can be assigned to the engine through the work dispatcher.

[0078] This design ensures that, regarding a bound request, no matter which die the request originates from, only the master die accesses the memory to service the bound request.

[0079] The process involves handshaking that occurs locally on each die to ensure that the local component-level TLBs are synchronized, as well as handshaking that occurs between the dies to ensure that they are synchronized.

[0080] Figure 5A multi-die GPU (e.g., Figures 1 to 3 Flowchart 500 of TLB invalidation in FIG.

[0081] TLB invalidation is the process of removing obsolete entries from the Translation Lookaside Buffer (TLB) in a multi-die GPU. TLB invalidation can be triggered by changes in the mapping between virtual and physical addresses (e.g., memory writes, page table updates), context switches, address space changes, etc.

[0082] TLB invalidation may be triggered by the MMU, engine, or client, or by software.

[0083] In the configuration shown, Die-1 is configured as primary and Die-2 is configured as secondary. The multi-die processing system is configured to have the primary die perform a TLB invalidation operation.

[0084] TLB invalidation invalidates translations (e.g., TLB lines) that have been fetched from the GMMU page table walk or from the ATS (IOMMU / SMMU). However, TLB invalidation does not invalidate page table entries (PTEs) or binding information.

[0085] TLB invalidations can originate from a hardware engine or client on a die, or can be generated based on register writes (e.g., privileged register writes). A TLB invalidation request in one die invalidates the TLB line on all dies. However, each die invalidates independently and in parallel.

[0086] The TLB invalidation packet may have the following information sent over the interface to other HUB MMUs: virtual address, PDB entry, indication of whether the invalidation is for an ATS entry, originating hub ID, and requester ID.

[0087] In the illustrated embodiment, a TLB invalidation request is issued by an engine or other client on die-1. The engine or client sends a request 502 to the local HUBMMU, HUBMMU-1. The request may be initiated by the engine, a client on the engine, a front-end context switch (FECS), a GPU system processor (GSP), a high-speed hub (HSHUB), or generated by writing to a privileged register. For example, the engine or client may send a method-based invalidation request, the HSHUB may send an ATS invalidation request (ATSD), the FECS / GSP may send a uCode-based TLB invalidation request, or the HUBMMU TLB invalidation privilege register may be used to generate a PRI-based TLB invalidation.

[0088] At 504, HUBMMU-1 transmits an invalidation request to all other dies in the multi-die processing system.

[0089] At 506, HUBMMU-1 and HUBMMU-2 can process TLB invalidation in parallel (e.g., 506a and 506b). If the TLB invalidation information indicates that a system membar (memory barrier operation) is required, the source MMU (HUBMMU-1 in this example) performs an IO flush operation at the end of the invalidation operation. Each HUBMMU independently retrieves the PDB ID to be invalidated from the PDB cache based on the input PDB, so the PDB ID may be different for the same PDB.

[0090] At 508 , HUBMMU- 2 returns an acknowledgment to HUBMMU- 1 indicating that TLB invalidation on Die- 2 is complete.

[0091] In another example, if a TLB invalidation request originates from die-2, the request is sent from the requesting client to HUBMMU-2, and HUBMMU-2 forwards the request to HUBMMU-1. This is because HUBMMU-1, as the primary die, is configured to handle TLB invalidations, while HUBMMU-2, as the secondary die, is configured to forward TLB invalidations to the primary die for processing.

[0092] After receiving confirmation from its local TLB invalidation and other dies, HUBMMU-1 returns confirmation to the requesting client at 510 .

[0093] Figure 6A A multi-die GPU (e.g., Figures 1 to 3 Flowchart 600 of a page table walk in a GPU (GPU) shown in FIG. In the configuration shown, die-1 is configured as primary and die-2 as secondary. The multi-die processing system is configured such that each die independently performs a page table walk to find a TLB miss from the local TLB.

[0094] Each HUBMMU can have its own page directory entry (PDE) / page table entry (PTE) cache to accelerate GPU MMU (GMMU) page table walks. In this paper, GMMU page table walks are also referred to as HUBMMU page table walks. On a local TLB miss, the HUBMMU, or more specifically the fill unit within the HUBMMU, performs a GMMU page table walk based on the engine binding points found in its own binding table and PDB cache.

[0095] At 602 , a hardware engine issues a memory request to a local HUBMMU (eg, HUBMMU- 1 on Die- 1 , HUBMMU- 2 on Die- 2 ).

[0096] At 604, the local HUBMMU performs a TLB lookup and misses.

[0097] At 606, the local HUBMMU performs a GMMU page table walk. A miss occurs.

[0098] At 608, the local HUBMMU transmits the page table walk memory fetch via its local FBHUB. The page table walk memory fetch request is sent to the local FBHUB. The FBHUB sends the request through the memory crossbar to the memory. The target memory may be located in the local video memory attached to all dies, or it may be located in the system memory. The system memory can be connected to the multi-die processing system via a chip-to-chip (C2C) interface or a PCIe interface. The memory fetch response flows through the reverse path.

[0099] At 610 , FBHUB fetches memory and at 612 obtains the requested memory.

[0100] At 614, the FBHUB returns the memory response to the local HUBMMU.

[0101] Based on the obtained memory, the HUBMMU updates the tables at 616. For example, the HUBMMU may update the local TLB by returning page table entries (or information from them) to the local TLB comprising the HUBTLB.

[0102] At 618, the memory response is returned to the requesting engine.

[0103] It should be noted that in some cases, a GMMU page table walk may require an Address Translation Services (ATS) fetch. In this case, the ATS request is handled in a manner similar to any other ATS request in a multi-die processing system.

[0104] In many systems, data is provided from the host to the GPU through one of several memory management API calls provided by the CUDA framework, such as CudaMallocManaged and CudaMemCpy. Some systems are able to avoid using CUDA calls for memory management and access the same data on both the GPU and the CPU. This is achieved through the Address Translation Services (ATS) technology, which provides a unified virtual address space for data allocated using malloc and new. ATS enables the CPU and GPU to share a single per-process page table, enabling all CPU and GPU threads to access all system-allocated memory, which can reside on physical CPU or GPU memory. The CPU heap, CPU thread stacks, global variables, memory-mapped files, and inter-process memory are accessible to all CPU and GPU processes.

[0105] In some embodiments, a multi-die GPU is connected to a C2C interface (e.g., see Figure 2 ) is connected to the CPU, which interface in some implementations may include hardware-based memory consistency, which enables only the required data to be transferred instead of migrating entire pages to and from the GPU. It can also enable lightweight synchronization primitives between the GPU and CPU threads by enabling native atomic operations from the CPU and GPU. In some embodiments, the ATS can utilize the direct memory access (DMA) copy engine on the GPU to accelerate bulk transfers of pageable memory between the host and GPU. Such embodiments can enable applications to oversubscribe the GPU's memory and directly utilize CPU memory (system memory) with high bandwidth. Access to such large amounts of memory further facilitates high-performance computing applications, graphics, virtual reality, and artificial intelligence applications.

[0106] ATS can be used with virtual addresses or from system physical addresses. Two types of ATS can be used: one that can translate virtual addresses to physical addresses, and one that can translate GPU virtual addresses to GPU physical addresses and then translate GPU physical addresses to system physical addresses.

[0107] In some embodiments, it may be necessary to generate an ATS request as part of a page table walk. This request may be generated by the HUBMMU during the page table walk. In some embodiments, each die may independently perform a page table walk and may issue an ATS request at a certain level. The type of ATS request may depend on the level of the page table walk at which the request is issued.

[0108] exist Figure 2 In the example embodiment, both dies are shown as supporting ATS request (ATR) / ATS response (ATRsp) between the HUBMMU (e.g., the HUBMMU's fill unit) and the HSHUB. However, in the example embodiment, the Die-1 function may not be used. Instead, Die-1 forwards its ATR to Die-2.

[0109] At Die-2, the HUBMMU fill unit can accommodate up to a preconfigured maximum (e.g., 256) ATRs from each of Die-1 and Die-2 and serialize the ATRs. Up to a predetermined maximum (e.g., 256 x 2 = 512) ATRs can be sent over the HUBMMU-HSHUB interface on Die-2 (or, in some implementations, the interface between the HUBMMU fill unit and the HSHUB), from which it will be routed to a high-speed interface (e.g., C2C) configured to connect to the CPU. Die-2 receives corresponding acknowledgments and sends corresponding acknowledgments back to Die-1 for any requests originating from Die-1.

[0110] Note that ATR / ATRsp propagation is centralized (e.g., at Die-2), but generation is done independently and in parallel on each die. Also note that another option is for each fill unit to send its ATR to its respective HSHUB, and then allow HSHUB-1 (rather than HUBMMU-1's fill unit) to forward its ATR to Die-2 for transmission over the C2C interface. This design allows for efficient processing of ATRs because ATS communication, in some embodiments, primarily occurs over the C2C interface, and at least in some embodiments, the only C2C interface for a multi-die processing system is on Die-2.

[0111] The die-1 ATR is forwarded to die-2. The HUBMMU on die-2 (e.g., the fill unit of the HUBMMU) then sends the ATR (e.g., the ATR from die-1 or die-2) to the die-2 HSHUB to the C2C to the system MMU (SMMU). The SMMU processes the ATR and returns the corresponding ATRsp to the die-2 HSHUB.

[0112] The die-2HSHUB sends the ATRsp to the die-2HUBMMU (e.g., the fill unit of the HUBMMU). The target ID in the ATR and ATRsp can be used to identify the target. The ATRsp can also include one or more of the die ID, ATS cache ID, cache line ID, and entry ID. The source can use these parameters to subsequently update its local table.

[0113] Figure 6B An example flow chart 610 of an ATS request and response process in a multi-die GPU according to some embodiments of the present disclosure is shown. As described above, an ATS request may be generated by a HUBMMU on any die during a page table traversal. A multi-die GPU may be configured such that, while the generation of the ATS is distributed among all die, the related communications with other devices (e.g., CPU, SMMU, IOMMU) and receiving corresponding responses are centralized on the die configured in a secondary role. Figure 6B In the scenario shown, on a 2-die GPU (e.g. Figures 1 to 3 In any 2-die GPU), according to an embodiment, die-1 is configured as primary and die-2 is configured as secondary. Figure 2 As described, the C2C interface is active on die-2 and unavailable (eg, does not exist or is set to inactive) on die-1. The described scenario is for an ATR generated by the HUBMMU of die-1 (HUBMMU-1).

[0114] At 612, HUBMMU-1 generates an ATR. This may be the result of a page table walk as described above. Since the multi-die GPU is configured to use die-2 as the serialization point for ATR / ATRsp, HUBMMU-1 forwards the ATR to die-2.

[0115] At 614, the HUBMMU of die-2 (HUBMMU-2) receives the ATR from HUBMMU-1 and serializes it with any other ATS requests (e.g., originating from die-1 or die-2). It then arbitrates between the serialized ATSs and sends the ATR to its local HSHUB.

[0116] At 616 , the HSHUB on die- 2 (HSHUB- 2 ) sends the ATR to the C2C interface of any of the CPU, SMMU, etc.

[0117] At 618, the corresponding ATRsp is received at the C2C interface on Die-2 and forwarded to the local HSHUB-2.

[0118] At 620, HSHUB-2 forwards the ATRsp to the local HUBMMU-2.

[0119] At 622, HUBMMU-2 forwards the ATRsp to HUBMMU-1.

[0120] Based on the ATRsp, the HUBMMU-1 may update the local TLB using the information contained in the ATRsp.

[0121] In terms of message flow, ATSD invalidation is Figure 5 The ATS invalidation invalidates the TLB entries indicated as ATS entries. ATSD invalidation requests and responses (like TLB invalidations) are carried out through HSHUB-2. The ATSD invalidation request is received from the SMMU at HSHUB-2 and forwarded to HUBMMU-2. HUBMMU-2 handles it similarly to a locally generated TLB invalidation and forwards the invalidation request to HUBMMU-1. HUBMMU-1 initiates the corresponding TLB update locally and notifies the other dies to perform the TLB update in parallel. Subsequently, after the TLB update has been confirmed locally and in all other dies, HUBMMU-1 transmits the confirmation of the ATSD invalidation to HUBMMU-2, which sends it to the SMMU via HSHUB-2.

[0122] As described above, in some embodiments, an ATS request to the IOMMU / SMMU is sent by an auxiliary die in a multi-die processing system. Figures 1 to 3In the two-die processing system shown, ATS requests are sent through the secondary die HSHUB because, in certain embodiments, this die has an active C2C interface over which ATS communication is performed. In some embodiments, if PCIe ATS requests are to be sent, such requests can be sent through the primary die, which has an active PCIe interface. Each HUBMMU can include an ATS wrapper function for forwarding ATS messages.

[0123] Each die's MMU (e.g., TLB and fill unit in the HUBMMU) detects faults independently. However, the HUBMMU of the primary die is configured to report faults regardless of where the fault originates (e.g., on which die). If a fault occurs in a secondary die, the secondary die forwards the fault information (e.g., fault type / ID, fault parameters, source engine / client of the faulted command / request, source die ID, etc.) to the primary die, which then processes the information to generate a fault packet.

[0124] If a fault buffer exists, the fault packet may be written to memory, written to a register, or reported in other ways. As described above, the fault information may include the ID of the engine or other component that caused the fault to be generated. This helps the CPU and / or other fault handling entities initiate actions to respond to the fault.

[0125] Figure 6C FIGURE 1 illustrates an embodiment of a multi-die GPU (e.g., Figures 1 to 3 Flowchart 630 of the fault reporting process for any of the 2-die GPUs in FIG.

[0126] At 632 , an engine in die- 2 or a process on an engine transmits the memory request to its local HUBMMU (HUBMMU- 2 ).

[0127] At 634, HUBMMU-2 detects a fault (eg, a memory access violation). Each HUBMMU may implement a fault state machine to detect faults on the local die.

[0128] At 636, HUBMMU-2 reports the fault to the HUBMMU (HUBMMU-1) on Die-1.

[0129] At 638, HUBMMU-1 serializes the error report and notifies the CPU, SMMU, or other fault handling entity of the fault. The fault handling entity can then use the information received in the fault report to respond to the fault.

[0130] The vidmem access bit (VAB) is a technique used to track memory segments that have been accessed or modified since tracing was enabled or the last VAB dump and clear at a specific point in time. It focuses on the physical addresses used by the program, providing valuable information for paging, debugging, or VM swapping. A VAB dump captures the system state at a specific point in time, typically triggered by events such as privileged register writes or channel methods.

[0131] In some embodiments, the VAB is dumped by the HUBMMU upon receiving a VAB dump request (e.g., such as a TLB invalidation). Each die independently tracks the VAB in its VAB mask (e.g., a 4K-bit VAB). In response to a VAB dump request, the following steps may be performed in order.

[0132] For example, on the master die, the host sends a VAB dump packet to the local HUBMMU. The host sends the VAB dump request as part of a TLB invalidation packet. However, no invalidation is performed on this packet. A field in the invalidation packet can be used to identify that the invalidation request is for a VAB dump.

[0133] The master HUBMMU sends the VAB dump packet to the slave HUBMMU (or vice versa if received at the slave die).

[0134] Both HUBMMUs independently collect VABs from each tracking replica in the die.

[0135] The final merged VAB mask is stored in the HUBMMU. The auxiliary HUBMMU sends the final merged VAB to the main HUBMMU and divides it into multiple blocks.

[0136] The master HUBMMU may merge the VAB masks from the auxiliary HUBMMUs and write the merged VAB masks to memory.

[0137] Figure 7An example GPU 700 is shown on a die of a multi-die GPU according to some embodiments of the present disclosure, with its frame buffer hub (FBHUB) and high-speed hub (HSHUB) identified in relation to some of their interconnections. GPU 700 includes two processors 705-1 and 705-2, each processor including multiple processing units 710 connected to cache memory 715. Respective crossbar switches 712-1 and 712-2 interconnect the processing units 710 and memory 715 in processors 705-1 and 705-2. High-speed hub (HSHUB) 718 connects all processing units 710 and cache memory 715. Frame buffer hub (FBHUB) 720 provides access to system and / or host memory 722 for processing units 710, which are connected to FBHUB 720 via HSHUB 718.

[0138] Example GPU Architecture

[0139] Describes the Figure 1 6 . The following information is for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be optionally combined, without or in addition to the other features described.

[0140] Figure 8 A parallel processing unit (PPU) 800 is shown that may be included on a die of a multi-die GPU according to an embodiment. In one embodiment, the PPU 800 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 800 is a latency-hidden architecture designed for processing many threads in parallel. A thread (e.g., an execution thread) is an instantiation of a set of instructions configured to be executed by the PPU 800. In one embodiment, the PPU 800 is a graphics processing unit (GPU) that is configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (e.g., a liquid crystal display (LCD) device). In other embodiments, the PPU 800 may be used to perform general-purpose computations. In some other embodiments, the PPU 800 is configured to implement large neural networks in deep learning applications or other high-performance computing applications.

[0141] One or more PPUs 800 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPUs 800 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulation, drug discovery, disease diagnosis, weather forecasting, big data analysis, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimization, personalized user recommendations, and more.

[0142] like Figure 8 As shown, the PPU 800 includes an input / output (I / O) unit 805, a front-end unit 815, a scheduler unit 820, a work distribution unit 825, a hub 830, a crossbar switch (Xbar) 870, one or more general processing clusters (GPCs) 850, and one or more partition units 880. The PPU 800 can be connected to a host processor or other PPUs 800 via one or more high-speed NVLink 810 interconnects. The PPU 800 can be connected to a host processor or other peripheral devices via an interconnect 802. The PPU 800 can also be connected to a memory including a plurality of memory devices 804. In an embodiment, the memory 804 may include a plurality of dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem, with multiple DRAM dies stacked within each device.

[0143] The NVLink 810 interconnect enables the system to expand and include a combination of one or more PPUs 800 and one or more CPUs, supporting cache coherency between the PPU 800 and the CPU and CPU mastering. Data and / or commands can be transferred by the NVLink 810 to / from other units of the PPU 800 through the hub 830, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 11A and Figure 11B NVLink 810 is described in more detail.

[0144] The I / O unit 805 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via the interconnect 802. The I / O unit 805 can communicate with the host processor directly via the interconnect 802, or through one or more intermediate devices (e.g., a memory bridge). In one embodiment, the I / O unit 805 can communicate with one or more other processors (e.g., one or more PPUs 800) via the interconnect 802. In one embodiment, the I / O unit 805 implements a Peripheral Component Interconnect Express (PCIe) interface to communicate over a PCIe bus, and the interconnect 802 is a PCIe bus. In alternative embodiments, the I / O unit 805 can implement other types of well-known interfaces to communicate with external devices.

[0145] The I / O unit 805 decodes packets received via the interconnect 802. In one embodiment, the packets represent commands configured to cause the PPU 800 to perform various operations. The I / O unit 805 transmits the decoded commands to various other units of the PPU 800, as the commands may specify. For example, some commands may be transmitted to the front end unit 815. Other commands may be transmitted to the hub 830 or other units of the PPU 800, such as one or more copy engines, video encoders, video decoders, a power management unit, etc. (not explicitly shown). In other words, the I / O unit 805 is configured to route communications between the various logical units of the PPU 800.

[0146] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 800 for processing. The workload may include several instructions and the data to be processed by these instructions. A buffer is an area in memory that is accessible (e.g., read / write) to both the host processor and the PPU 800. For example, the I / O unit 805 may be configured to access a buffer in system memory connected to the interconnect 802 via a memory request transmitted via the interconnect 802. In one embodiment, the host processor writes a command stream into the buffer and then transmits a pointer to the beginning of the command stream to the PPU 800. The front end unit 815 receives pointers to one or more command streams. The front end unit 815 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 800.

[0147] Front-end unit 815 is coupled to scheduler unit 820, which configures each GPC 850 to process tasks defined by one or more streams. Scheduler unit 820 is configured to track state information related to the various tasks managed by scheduler unit 820. The state may indicate which GPC 850 a task is assigned to, whether a task is active or inactive, a priority level associated with the task, and the like. Scheduler unit 820 manages the execution of multiple tasks on one or more GPCs 850.

[0148] Scheduler unit 820 is coupled to work distribution unit 825, which is configured to dispatch tasks for execution on GPCs 850. Work distribution unit 825 can track a number of scheduled tasks received from scheduler unit 820. In one embodiment, work distribution unit 825 manages a pending task pool and an active task pool for each GPC 850. The pending task pool can include a number of slots (e.g., 32 slots) containing tasks assigned to be processed by a particular GPC 850. The active task pool can include a number of slots (e.g., 4 slots) for tasks that are actively being processed by GPC 850. When a GPC 850 completes execution of a task, the task is evicted from the active task pool of GPC 850 and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 850. If an active task is idle on a GPC 850 , such as while waiting for data dependencies to be resolved, the active task may be evicted from the GPC 850 and returned to the pending task pool while another task in the pending task pool is selected and scheduled for execution on the GPC 850 .

[0149] Work distribution unit 825 communicates with one or more GPCs 850 via XBar 870. XBar 870 is an interconnect network that couples many units of PPU 800 to other units of PPU 800. For example, XBar 870 can be configured to couple work distribution unit 825 to a specific GPC 850. Although not explicitly shown, one or more other units of PPU 800 can also be connected to XBar 870 via hub 830.

[0150] Tasks are managed by the scheduler unit 820 and dispatched to the GPCs 850 by the work distribution unit 825. The GPCs 850 are configured to process tasks and generate results. The results may be used by other tasks within the GPC 850, routed to different GPCs 850 via the XBar 870, or stored in memory 804. The results may be written to memory 804 via a partition unit 880, which implements a memory interface for reading from and writing data to memory 804. The results may be transferred to another PPU 804 or CPU via NVLink 810. In one embodiment, the PPU 800 includes a number U of partition units 880, which is equal to the number of separate and distinct memory devices 804 coupled to the PPU 800. Figure 9B The partition unit 880 is described in more detail.

[0151] In an embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications executing on the host processor to schedule operations to be executed on the PPU 800. In an embodiment, the PPU 800 executes multiple computing applications simultaneously, and the PPU 800 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 800. The driver kernel outputs the tasks to one or more streams being processed by the PPU 800. Each task can include one or more groups of related threads, referred to herein as warps. In an embodiment, a warp includes 32 related threads that can be executed in parallel. Collaborating threads can refer to multiple threads that include instructions to execute tasks and can exchange data through shared memory (SMEM). According to some embodiments, threads, cooperative threads, and hierarchical groupings of threads (e.g., cooperative thread arrays (CTAs) and cooperative group arrays (CGAs)) are described in more detail in U.S. application Ser. No. 17 / 691,621, the entire contents of which are hereby incorporated by reference. According to some embodiments, SMEM is described in U.S. application Ser. No. 17 / 691,690, which is hereby incorporated by reference.

[0152] Figure 9A It shows the embodiment according to Figure 8 PPU 800 GPC 850. Figure 9AAs shown, each GPC 850 includes several hardware units for processing tasks. In one embodiment, each GPC 850 includes a pipeline manager 910, a pre-raster operation unit (PROP) 915, a raster engine 925, a work distribution crossbar (WDX) 980, a memory management unit (MMU) 990 and one or more data processing clusters (DPCs) 920. It can be understood that Figure 9A The GPC 850 may include other hardware units to replace or supplement Figure 9A The unit shown in .

[0153] In one embodiment, the operation of GPC 850 is controlled by pipeline manager 910. Pipeline manager 910 manages the configuration of one or more DPCs 920 to process tasks assigned to GPC 850. In one embodiment, pipeline manager 910 can configure at least one of one or more DPCs 920 to implement at least a portion of a graphics rendering pipeline, a neural network, and / or a compute pipeline. For example, with respect to the graphics rendering pipeline, DPC 920 can be configured to execute vertex shader programs on programmable streaming multiprocessors (SMs) 940. Pipeline manager 910 can also be configured to route packets received from work distribution unit 825 to appropriate logic units within GPC 850. For example, some packets can be routed to fixed-function hardware units in PROP 915 and / or raster engine 925, while other packets can be routed to DPC 920 for processing by primitive engine 935 or SM 940.

[0154] PROP unit 915 is configured to route data generated by raster engine 925 and DPC 920 to the raster operation (ROP) unit, in conjunction with Figure 9B The PROP unit 915 may also be configured to perform optimization of color mixing, organize pixel data, perform address translation, and the like.

[0155] Each DPC 920 included in the GPC 850 includes an M-pipe controller (MPC) 930, a primitive engine 935, and one or more SMs 940. The MPC 930 controls the operation of the DPC 920 and routes packets received from the pipeline manager 910 to appropriate units in the DPC 920. For example, packets associated with vertices may be routed to the primitive engine 935, which is configured to fetch vertex attributes associated with the vertices from the memory 804. Conversely, packets associated with shader programs may be transmitted to the SM 940.

[0156] SM 940 includes a programmable streaming processor configured to process tasks represented by multiple threads. Each SM 940 is multi-threaded and configured to concurrently execute multiple threads (e.g., 32 threads) from a specific thread group. In one embodiment, SM 940 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group (e.g., a warp) is configured to process a different data set based on the same set of instructions. All threads in a thread group execute the same instructions. In another embodiment, SM 940 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group is configured to process a different data set based on the same set of instructions, but individual threads in a thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, thereby enabling concurrency between warps and serial execution within a warp when threads within the warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby enabling equal concurrency between all threads within and between warps. When execution state is maintained for each separate thread, threads executing the same instruction can converge and execute in parallel for maximum efficiency. Figure 10A SM 940 is described in more detail. Figure 10B Conceptually illustrates the process of performing a multi-processor (e.g., Figure 10A The four sub-partitions 1091-1094 are implemented in the SM) shown in FIG.

[0157] The MMU 990 provides an interface between the GPC 850 and the partition unit 880. The MMU 990 can provide virtual-to-physical address translation, memory protection, and arbitration of memory requests. In an embodiment, the MMU 990 provides one or more translation lookaside buffers (TLBs) for performing translation of virtual addresses into physical addresses in the memory 804.

[0158] Figure 9B It shows the embodiment according to Figure 8 The memory partition unit 880 of the PPU 800. Figure 9BAs shown, the memory partition unit 880 includes a raster operations (ROP) unit 950, a level 2 (L2) cache 960, and a memory interface 970. The memory interface 970 is coupled to the memory 804. The memory interface 970 can implement a 32-, 64-, 128-, or 1024-bit data bus, etc., for high-speed data transfer. In an embodiment, the PPU 800 includes U memory interfaces 970, one memory interface 970 for each pair of partition units 880, where each pair of partition units 880 is connected to a corresponding memory device 804. For example, the PPU 800 can be connected to up to Y memory devices 804, such as a high-bandwidth memory stack or graphics double data rate version 5, synchronous dynamic random access memory, or other types of persistent memory.

[0159] In an embodiment, memory interface 970 implements an HBM2 memory interface, and Y is equal to half of U. In an embodiment, the HBM2 memory stack is located on the same physical package as the PPU 800, which can significantly save power and area compared to traditional GDDR5 SDRAM systems. In an embodiment, each HBM2 stack includes four memory dies, and Y is equal to 4, where the HBM2 stack includes two 128-bit channels per die, for a total of 8 channels, with a data bus width of 824 bits.

[0160] In an embodiment, memory 804 supports single-error correction, double-error detection (SECDED) error correction code (ECC) to protect data. ECC provides higher reliability for computing applications that are sensitive to data corruption. Reliability is particularly important in large-scale cluster computing environments where PPU 800 processes very large data sets and / or long-running applications.

[0161] In an embodiment, the PPU 800 implements a multi-level memory hierarchy. In an embodiment, the memory partitioning unit 880 supports unified memory to provide a single unified virtual address space for the CPU and PPU 800 memory, thereby enabling data sharing between virtual memory systems. In an embodiment, the frequency of PPU 800 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 800 where the pages are accessed more frequently. In an embodiment, NVLink 810 supports address translation services, allowing the PPU 800 to directly access the CPU's page tables and providing the PPU 800 with full access to the CPU's memory.

[0162] In an embodiment, the copy engine transfers data between multiple PPUs 800 or between a PPU 800 and a CPU. The copy engine can generate a page fault for an address that is not mapped in the page table. The memory partition unit 880 can then service the page fault, mapping the address into the page table, after which the copy engine can perform the transfer. In conventional systems, memory is fixed (e.g., non-pageable) for multiple copy engine operations between multiple processors, significantly reducing the available memory. With hardware page faults, addresses can be passed to the copy engine without having to worry about whether the memory page is resident, and the copy process is transparent.

[0163] Memory partition unit 880 can retrieve data from memory 804 or other system memory and store it in L2 cache 960, which is located on-chip and shared between each GPC 850. As shown, each memory partition unit 880 includes a portion of L2 cache 960 associated with the corresponding memory device 804. Lower levels of cache can then be implemented in various units within GPC 850. For example, each SM 940 can implement a level 1 (L1) cache. The L1 cache is private memory dedicated to a specific SM 940. Data can be retrieved from L2 cache 960 and stored in each L1 cache for processing within the functional units of SM 940. L2 cache 960 is coupled to memory interface 970 and XBar 870.

[0164] The ROP unit 950 performs graphics raster operations related to pixel color, such as color compression, pixel blending, etc. The ROP unit 950 also implements depth testing in conjunction with the raster engine 925, receiving the depth of the sample position associated with the pixel fragment from the culling engine of the raster engine 925. The depth is tested against the corresponding depth in the depth buffer of the sample position associated with the fragment. If the fragment passes the depth test for the sample position, the ROP unit 950 updates the depth buffer and transmits the result of the depth test to the raster engine 925. It will be appreciated that the number of partition units 880 may be different than the number of GPCs 850, and therefore, each ROP unit 950 may be coupled to each GPC 850. The ROP unit 950 tracks packets received from different GPCs 850 and determines to which GPC 850 the result generated by the ROP unit 950 is routed via the Xbar 870. Although the ROP unit 950 is included in Figure 9B 850, but in other embodiments, ROP unit 950 may be external to memory partition unit 880. For example, ROP unit 950 may reside in GPC 850 or another unit.

[0165] Figure 10A It shows the embodiment according to Figure 9A Streaming multiprocessor 940. Figure 10A As shown, the SM 940 includes an instruction cache 1005, one or more scheduler units 1010, a register file 1020, one or more processing cores 1050, one or more special function units (SFUs) 1052, one or more load / store units (LSUs) 1054, an interconnection network 1080, and a SMEM / L1 cache 1070.

[0166] As described above, the work distribution unit 825 dispatches tasks for execution on the GPCs 850 of the PPU 800. Tasks are assigned to specific DPCs 920 within the GPC 850, and if the task is associated with a shader program, the task may be assigned to an SM 940. The scheduler unit 1010 receives tasks from the work distribution unit 825 and manages the scheduling of instructions for one or more thread blocks assigned to the SM 940. The scheduler unit 1010 schedules thread blocks for execution as warps of parallel threads, where each thread block consists of at least one warp. In one embodiment, each warp includes 32 threads. The scheduler unit 1010 can manage multiple different thread blocks, assigning different thread blocks to different warps, and then dispatching instructions from multiple different cooperation groups to various functional units (e.g., core 1050, SFU 1052, and LSU 1054) during each clock cycle.

[0167] Cooperative Group Arrays (CGA) provide a programming model for organizing groups of communicating threads, allowing developers to express the granularity of thread communication, enabling richer and more efficient expression of parallel decompositions. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. Traditional programming models provide a single, simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than a thread block and synchronize within the defined group to achieve higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0168] Cooperative group arrays enable programmers to explicitly define thread groups at sub-block (e.g., as small as a single thread) and multi-block granularity and perform collective operations on threads, such as synchronization within cooperative groups. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local context without making assumptions about convergence. Cooperative group array primitives support new cooperative parallelism patterns, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks. According to some embodiments, hierarchical groupings of threads, such as cooperative thread arrays (CTAs) and cooperative group arrays (CGAs), are described in more detail in U.S. application Ser. No. 17 / 691,621, the entire contents of which are hereby incorporated by reference.

[0169] The dispatch unit 1015 is configured to transmit instructions to one or more functional units. In an embodiment, the scheduler unit 1010 includes two dispatch units 1015, which enable dispatching two different instructions from the same warp in each clock cycle. In alternative embodiments, each scheduler unit 1010 may include a single dispatch unit 1015 or additional dispatch units 1015.

[0170] Each SM 940 includes a register file 1020 that provides a set of registers for the functional units of the SM 940. In one embodiment, the register file 1020 is partitioned between each functional unit so that each functional unit is allocated a dedicated portion of the register file 1020. In another embodiment, the register file 1020 is partitioned between the different warps executed by the SM 940. The register file 1020 provides temporary storage for operands connected to the data paths of the functional units.

[0171] Each SM 940 includes multiple processing cores 1050. In one embodiment, the SM 940 includes a large number (e.g., 128, etc.) of different processing cores 1050. Each core 1050 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (ALU) and an integer ALU. In one embodiment, the ALU implements the IEEE 754-2008 standard for floating-point arithmetic.

[0172] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 1050. Specifically, tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inference.

[0173] In some embodiments, the transposition hardware is included in the processing core 1050 or another functional unit (e.g., SFU 1052 or LSU 1054) and is configured to generate diagonally stored matrix data and / or generate the original matrix and / or the transposed matrix from the diagonally stored matrix data. The transposition hardware can be provided within the SMEM 1070 in the register file 1020 load path to the SM 940.

[0174] Each SM 940 also includes multiple SFUs 1052 that perform specialized functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 1052 may include a tree traversal unit (e.g., TTU 943) configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 1052 may include a texture unit (e.g., texture unit 942) configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D texel array) from memory 1004 and sample the texture map to generate sampled texture values ​​for use by shader programs executed by the SM 940. In one embodiment, the texture map is stored in the SMEM / L1 cache 970. The texture unit implements texture operations, such as filtering operations, using mip-maps (e.g., texture maps of different levels of detail). In one embodiment, each SM 940 includes two texture units.

[0175] Each SM 940 also includes multiple LSUs 1054 for implementing load and store operations between the SMEM / L1 cache 1070 and the register file 1020. Each SM 940 includes an interconnect network 1080 that connects each functional unit to the register file 1020 and connects the LSUs 1054 to the register file 1020 and the SMEM / L1 cache 1070. In one embodiment, the interconnect network 1080 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 1020 and to connect the LSUs 1054 to memory locations in the register file 1020 and the SMEM / L1 cache 1070.

[0176] SMEM / L1 cache 1070 is an on-chip memory array that allows data storage and communication between SM 940 and primitive engine 935, as well as between threads in SM 940. In an embodiment, SMEM / L1 cache 1070 includes 128KB of storage capacity and is located in the path from SM 940 to partition unit 1080. SMEM / L1 cache 1070 can be used to cache reads and writes. One or more of SMEM / L1 cache 1070, L2 cache 960, and memory 1004 are backing stores.

[0177] Combining data cache and SMEM functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used as a cache by programs that do not use SMEM. For example, if SMEM is configured to use half of its capacity, texture and load / store operations can use the remaining capacity. The integration within SMEM / L1 cache 1070 enables SMEM / L1 cache 1070 to serve as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0178] In the context of this disclosure, an SM or "streaming multiprocessor" refers to a processor designed in accordance with the architecture described in USP7,447,873 to Nordquist, including improvements and modifications thereto, and implemented, for example, in many generations of NVIDIA GPUs. For example, an SM may include multiple processing engines or cores configured to concurrently execute multiple threads arranged in multiple single instruction multiple data (SIMD) groups (e.g., warps), wherein each thread in the same SIMD group executes the same data processing program comprising a sequence of instructions on different input objects, and different threads of the same SIMD group are executed using different processing engines or cores in the processing engines or cores. The SM may also typically provide (a) a local register file having multiple channels, wherein each processing engine or core is configured to access a different subset of the channels; and instruction issue logic configured to select one of the SIMD groups and issue one of the instructions of the same data processing program to each of the multiple processing engines in parallel, wherein each processing engine executes the same instruction in parallel with each other processing engine using the accessible subset of the local register file channels. The SM typically also includes core interface logic configured to initiate execution of one or more SIMD groups. As shown in the figure, such SMs have been architected to provide fast local SMEM, enabling data sharing / reuse and synchronization among all threads of a CTA executing on the SM.

[0179] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 9AThe fixed-function graphics processing unit shown in FIG is bypassed, creating a simpler programming model. In the general parallel computing configuration, the work distribution unit 1025 assigns and distributes thread blocks directly to the DPC 920. The threads in the block execute the same program, use unique thread IDs in computations to ensure that each thread produces unique results, use the SM 940 to execute the program and perform computations, use the SMEM / L1 cache 1070 to communicate between threads, and use the LSU 1054 to read and write global memory through the SMEM / L1 cache 1070 and the memory partition unit 1080. When configured for general parallel computing, the SM 940 can also write commands that the scheduler unit 1020 can use to start new work on the DPC 920.

[0180] The PPU 800 or a multi-die GPU including multiple PPUs as described in the present disclosure may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., a wireless handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 800 is embodied on a single semiconductor substrate. In another embodiment, the PPU 800 is included in a system-on-chip (SoC) along with one or more other devices (e.g., additional PPUs 800, memory 1004, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0181] Example computing system

[0182] As developers demonstrate and exploit greater parallelism in applications like artificial intelligence computing, systems with multiple GPUs and CPUs are being adopted across a wide range of industries. High-performance GPU-accelerated systems with tens to thousands of compute nodes are deployed in data centers, research facilities, and supercomputers to solve increasingly large problems. As the number of processing devices in high-performance systems increases, communication and data transfer mechanisms need to scale to support the increased bandwidth.

[0183] Figure 11A According to one embodiment, the use includes Figure 8 Conceptual diagram of a processing system 1100 implemented with a multi-die GPU 1000 comprising two or more PPUs 800. The exemplary system 1100 may be configured to implement the methods disclosed herein. The processing system 1100 includes a CPU 1130, a switch 1155, and multiple multi-die GPUs, each with its own memory 1004. NVLink 1010 provides a high-speed communication link between each multi-die GPU. Although Figure 11AA specific number of NVLinks 1010 and interconnect 1002 connections are shown, but the number of connections for each multi-die GPU and CPU 1130 may vary. Switch 1155 interfaces between interconnect 1002 and CPU 1130. The multi-die GPU, memory 1004, and NVLinks 1010 may be located on a single semiconductor platform to form a parallel processing module 1125. In embodiments, switch 1155 supports two or more protocols to interface between the various connections and / or links.

[0184] In another embodiment (not shown), NVLink 1010 provides one or more high-speed communication links between each multi-die GPU and CPU 1130, and switch 1155 interfaces between interconnect 1002 and each multi-die GPU. The multi-die GPUs, memory 1004, and interconnect 1002 can be located on a single semiconductor platform to form parallel processing module 1125. In yet another embodiment (not shown), interconnect 1002 provides one or more communication links between each multi-die GPU and CPU 1130, and switch 1155 interfaces between the multi-die GPUs using NVLink 1010 to provide one or more high-speed communication links between the multi-die GPUs. In another embodiment (not shown), NVLink 1010 provides one or more high-speed communication links between the multi-die GPUs and CPU 1130 via switch 1155. In yet another embodiment (not shown), interconnect 1002 provides one or more communication links directly between the multi-die GPUs. One or more NVLink 1010 high-speed communication links may be implemented as a physical NVLink interconnect or an on-chip or on-die interconnect using the same protocol as NVLink 1010 .

[0185] In the context of this specification, a single semiconductor platform may refer to a single semiconductor-based integrated circuit manufactured on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with enhanced connectivity that simulates on-chip operation and is a substantial improvement over conventional bus implementations. Of course, various circuits or devices may also be provided individually or in various semiconductor platform combinations according to user needs. Alternatively, the parallel processing module 1125 may be implemented as a circuit board substrate, and each of the multi-die GPU 1000 and / or memory 1004 may be a packaged device. In one embodiment, the CPU 1130, the switch 1155, and the parallel processing module 1125 are located on a single semiconductor platform.

[0186] In an embodiment, NVLink 1010 allows direct load / store / atomic access from CPU 1130 to memory 1004 of each multi-die GPU 1000. In an embodiment, NVLink 1010 supports coherency operations, allowing data read from memory 1004 to be stored in the cache hierarchy of CPU 1130, thereby reducing cache access latency for CPU 1130. In an embodiment, NVLink 1010 includes support for Address Translation Services (ATS), allowing multi-die GPU 1000 to directly access page tables within CPU 1130. One or more NVLinks 1010 can also be configured to operate in a low-power mode.

[0187] Figure 11B An exemplary system 1165 is shown in which various architectures and / or functionality of various previous embodiments can be implemented. The exemplary system 1165 can be configured to implement the methods disclosed herein.

[0188] As shown, a system 1165 is provided that includes at least one central processing unit 1130 connected to a communication bus 1175. The communication bus 1175 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol. The system 1165 also includes a main memory 1140. Control logic (software) and data are stored in the main memory 1140, which can take the form of random access memory (RAM).

[0189] System 1165 also includes an input device 1160, a parallel processing system 1125, and a display device 1145, such as a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display, etc. User input can be received from input device 1160, such as a keyboard, mouse, touchpad, microphone, etc. Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 1165. Alternatively, the modules can be placed separately or on various combinations of semiconductor platforms according to user needs.

[0190] Furthermore, the system 1165 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a wired network, etc.) through the network interface 1135 for communication purposes.

[0191] System 1165 may also include an auxiliary storage device (not shown). The auxiliary storage device includes, for example, a hard disk drive and / or a removable storage drive, representing a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner.

[0192] Computer programs or computer control logic algorithms may be stored in the main memory 1140 and / or the secondary memory. Such computer programs, when executed, enable the system 1165 to perform various functions. The memory 1140, storage device, and / or any other storage are possible examples of computer-readable media.

[0193] The architecture and / or functionality of the various previous figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application-specific system, and / or any other desired system. For example, the system 1165 can take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., a wireless handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0194] The application program may be implemented by an application executed by a host processor (e.g., a CPU). In one embodiment, a device driver may implement an application programming interface (API) that defines various functions that the application program can utilize to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of a multi-die GPU 1000, which includes two or more PPUs 800. The API provides an abstraction for programmers, enabling them to utilize specialized graphics hardware (e.g., a PPU 800) to generate graphics data without requiring them to utilize the specific instruction set of the PPU 800. An application may include API calls that are routed to the device driver of the multi-die GPU. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver may perform operations by executing instructions on the CPU. In other cases, the device driver may perform operations at least in part by initiating operations on the multi-die GPU 1000 using an input / output interface between the CPU and the multi-die GPU. In one embodiment, the device driver is configured to implement a graphics processing pipeline using the hardware of the PPU 800.

[0195] Various programs can be executed within a multi-die GPU 1000 that includes two or more PPUs 800 to implement various processing stages of an application. For example, a device driver can launch a kernel on a PPU 800 to execute a processing stage on one SM 940 (or multiple SMs 940). The device driver (or the initial kernel executed by the PPU 800) can also launch other kernels on the PPU 800 to execute other processing stages. If the application processing includes a graphics processing pipeline, some stages of the graphics processing pipeline can be implemented on fixed unit hardware, such as a rasterizer or data assembler implemented within the PPU 800. It will be understood that the results of a kernel can be processed by one or more intermediate fixed-function hardware units and then processed by a subsequent kernel on the SM 940.

[0196] The technology disclosed herein can be incorporated into any processor that can be used to process neural networks, such as a central processing unit (CPU), a graphics processing unit (GPU), an intelligence processing unit (IPU), a neural processing unit (NPU), a tensor processing unit (TPU), a neural network processor (NNP), a data processing unit (DPU), a vision processing unit (VPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc. Such a processor can be incorporated into a personal computer (e.g., a laptop), a data center, an Internet of Things (IoT) device, a handheld device (e.g., a smartphone), a vehicle, a robot, or any other device that performs neural network inference, training, or any other processing. Such a processor can be used in a virtualized system so that an operating system executing in a virtual machine on the system can utilize the processor.

[0197] As an example, a processor incorporating the techniques disclosed herein can be used to process one or more neural networks in a machine to recognize, classify, manipulate, process, operate, modify, or navigate physical objects in the real world. For example, such a processor can be used in an autonomous vehicle (e.g., a car, motorcycle, helicopter, drone, airplane, ship, submarine, delivery robot, etc.) to enable the vehicle to move in the real world. Additionally, such a processor can be used in a robot in a factory to select parts and assemble the parts into assemblies.

[0198] As an example, a processor incorporating the techniques disclosed herein may be employed to process one or more neural networks to identify one or more features in an image or to alter, generate, or compress an image. For example, such a processor may be employed to enhance images rendered using rasterization, ray tracing (e.g., using NVIDIA RTX), and / or other rendering techniques. In another example, such a processor may be employed to reduce the amount of image data transmitted from a rendering device to a display device over a network (e.g., the Internet, a mobile telecommunications network, a WIFI network, and any other wired or wireless network system). Such transmission may be utilized to stream image data from a server or data center in the cloud to a user device (e.g., a personal computer, a video game console, a smartphone, other mobile device, etc.) to enhance services that stream images, such as NVIDIA GeForce Now (GFN), Google Stadia, and the like.

[0199] By way of example, a processor incorporating the techniques disclosed herein may be used to process one or more neural networks for any other type of application that can utilize a neural network. For example, such applications may involve translation from one spoken language to another, recognition and negation of sounds in audio, detection of anomalies or defects in the production of goods and services, monitoring of living and / or non-living things, medical diagnosis, decision making, and the like.

[0200] As an example, a processor incorporating the techniques disclosed herein may be used to implement neural networks, such as large language models (LLMs), to generate content (e.g., images, videos, text, articles, audio, etc.), respond to user queries, solve problems in mathematics and other fields, and the like.

[0201] All patents, patent applications, and publications cited herein are incorporated by reference for all purposes as if expressly set forth.

[0202] While the invention has been described in connection with what are presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but on the contrary, the invention is intended to cover various modifications and equivalent arrangements within the spirit and scope of the appended claims.

Claims

1. A multi-die processing system comprising: a first memory management unit MMU on the first die; as well as A second MMU on a second die, where The first MMU on the first die is configured to process memory-related requests of a first type from the first die and the second die and to process memory-related requests of a second type from the first die, and The second MMU on the second die is configured to forward the first type of memory-related requests to the first MMU on the first die and to process the second type of memory-related requests from the second die.

2. The multi-die processing system of claim 1 , wherein the second MMU is further configured to process a third type of memory-related requests received from the second die and the first die, and the first MMU is further configured to forward the third type of memory-related requests to the second MMU.

3. The multi-die processing system of claim 2 , wherein the first MMU is configured to serialize the first type of memory-related requests received from the first die and the second die, and the second MMU is configured to serialize the third type of memory-related requests received from the first die and the second die.

4. The multi-die processing system of claim 2 , wherein the first MMU is configured to generate a message for transmission on a first type of network interface in response to at least one memory-related message of the first type, wherein the second MMU is configured to generate a message for transmission on a second type of network interface in response to at least one memory-related message of the third type, wherein the second type of network interface is located on the second die. 5 . The multi-die processing system of claim 2 , wherein the first MMU and the second MMU are each configured to detect an operating role according to a respectively set configuration setting. 6 . The multi-die processing system of claim 5 , wherein a third MMU located on a third die of the multi-die processing system detects an independent operating role based on the configuration setting. 7 . The multi-die processing system of claim 5 , wherein the first MMU detects a primary operating role based on corresponding configuration settings, and the second MMU detects a secondary operating role based on corresponding configuration settings.

8. The multi-die processing system of claim 1, wherein the first type of memory-related request comprises a bind request.

9. The multi-die processing system of claim 8 , wherein, in response to the bind request, the first MMU is configured to: obtain binding information corresponding to the bind request, update at least one of a binding table and a page directory binding cache in a memory of the first die, and transmit the binding information to the second die.

10. The multi-die processing system of claim 9, wherein the bind request originates from the second die and is transmitted by the second die to the first die, and wherein the second MMU is further configured to: in response to receiving the bind information from the first die, update at least one of a binding table and a page directory binding cache in a memory of the first die, and respond to the source of the bind request.

11. The multi-die processing system of claim 10, wherein: In response to receiving the binding information, each of the first MMU and the second MMU is further configured to update at least one translation lookaside buffer (TLB) on the local die according to the binding information.

12. The multi-die processing system of claim 1, wherein the first type of memory-related request further comprises a fault report request, a TLB invalidation request, or a VAB dump request.

13. The multi-die processing system of claim 12, wherein the second type of memory-related request comprises an address translation request, and each of the first MMU and the second MMU is configured to perform a TLB and / or page table lookup in response to the address translation request.

14. The multi-die processing system of claim 13 , wherein the second type of memory-related request further comprises: ECC interrupts, GPU page table walks, memory barrier requests, memory-to-memory input-output requests, or packet generation for work creation requests.

15. The multi-die processing system of claim 1 , wherein the second MMU is configured to, in response to a memory barrier request from the second die, transmit a request preceding the memory barrier request from a hardware engine in a first stage, and transmit one or more input / output refresh (I / O) refresh requests in a second stage.

16. The multi-die processing system according to claim 1 , wherein the second MMU is configured to, in response to a memory barrier request from the second die, transmit a request preceding the memory barrier request from a hardware engine in a first stage, and transmit an input / output refresh (I / O refresh) request to the first MMU in a second stage, and the first MMU is configured to serialize execution of the I / O refresh request. 17 . The multi-die processing system of claim 1 , wherein the first MMU and the second MMU are respectively configured to program one or more registers on the first die and the second die via software.

18. The multi-die processing system of claim 1, wherein the first MMU and the second MMU are each configurable in one or three settings: primary, secondary, or independent.

19. The multi-die processing system of claim 1, further comprising a high-speed interface connecting the first die and the second die, and the second MMU is configured to forward the first type of memory-related request to the first MMU through the high-speed interface.

20. The multi-die processing system of claim 19, wherein the high-speed interface is a dedicated connection for die-to-die communication within the multi-die processing system.

21. The multi-die processing system of claim 1, wherein the hardware engine remapping circuitry is configured to determine a globally unique die identifier for each die based on a die identification fuse signal.

22. The multi-die processing system of claim 1, wherein the hardware engine remapping circuitry comprises a plurality of die-level remapping circuits, and each die-level remapping circuit is configured to operate independently of other ones of the die-level remapping circuits.

23. The multi-die processing system of claim 9, wherein each die-level remapping circuit is configured to apply an offset to a hardware engine identifier in a received message, wherein the offset is determined based on a die identification fuse signal.

24. The multi-die processing system of claim 1, wherein each die comprises a plurality of processing engines.

25. The multi-die processing system of claim 1, wherein the plurality of parallel processors on respectively different dies including the first die and the second die are located in a single package, wherein the package is configured to be connected to a central processing unit (CPU) in a computer.

26. The multi-die processing system of claim 1, wherein each of the dies is a graphics processing unit (GPU).

27. A method performed by a multi-die processing system, the multi-die processing system being configured to process multiple types of memory-related requests, wherein the multi-die processing system includes a first memory management unit (MMU) on a first die and a second MMU on a second die, the method comprising: processing, by the first MMU, memory-related requests of a first type from the first die and the second die and memory-related requests of a second type received from the first die; Forwarding, by the second MMU, the memory-related request of the first type to the first MMU; as well as The second type of memory-related request received from the second die is processed by the second MMU.

Images