Hardware-assisted page migration in multi-die grain processing system

Abstract

Embodiments of the present disclosure relate to hardware-assisted page migration in a multi-die grain processing system. A hardware mechanism at each die grain of a multi-die grain processing system knows engine page table bindings at all die grains, thereby providing accurate traffic notifications to software (e.g., a unified virtual memory driver) for on-demand page migration between system memory and GPU memory. The mechanism broadcasts binding information to an access counter on each die grain so that the access counter can associate an engine requesting memory access with a bound virtual memory page and generate corresponding notifications containing useful information. A flexible multi-die grain counter clearing function enables software to clear access counters.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This Chinese patent application claims priority to and the benefit of U.S. non-provisional patent application No. 18 / 607,525, filed on March 17, 2024, entitled “HARDWARE ASSISTED PAGE MIGRATION IN A MULTI-DIELET PROCESSING SYSTEM,” the entire disclosure of which is incorporated herein in its entirety for all purposes. This U.S. non-provisional patent application is related to commonly assigned co-pending patent application No. 63 / 566,359, filed on the same date, entitled “Synchronizing Memory Management Units in Multi-Dielet Processor Architectures,” (23-SC-01018US01 / 6610-169), which is incorporated herein by reference for all purposes as if expressly set forth herein. This co-pending application is referred to herein as “Synchronizing MMUs.” Technical Field

[0003] Embodiments relate to hardware-assisted memory page migration in a multi-dielet processing system, such as a processing system including a multi-chip module (MCM) graphics processing unit (GPU). Background Art

[0004] like Figures 1A-1C As shown, a multiprocessor system typically includes a processor, such as a CPU 14, having a system memory 12 that stores data and instructions in memory pages 18. The processing system may further include one or more auxiliary processors 100, such as a GPU (graphics processing unit). In the case where the auxiliary processor is a GPU 100, the application executed by the auxiliary processor may be a graphics application, including, for example, texture-based graphics pipeline processing, and more recently, ray tracing and computational applications including, but not limited to, tensor operations and deep neural networks.

[0005] exist Figure 1AIn one scenario shown, the CPU 14 may generate or otherwise provide instructions and data directed to the auxiliary processor 100 and arrange for the data and instructions to be stored in the dual-ported system memory 12. The CPU / SOC 14 may then instruct the auxiliary processor 100 to execute an application program from the dual-ported system memory 12. The auxiliary processor 100 will retrieve the instructions and data from the dual-ported system memory 12, perform the operations indicated by the instructions, and store the resulting data in the system memory 12. The CPU 14 may then retrieve the resulting data from the system memory 12.

[0006] like Figure 1A As shown, the problem with this approach is that the path used by the auxiliary processor 100 to access instructions and data in the system memory 12 may be a bottleneck. Each access may take many cycles to complete, which slows down the auxiliary processor 100. Some of the high latency can be hidden by techniques such as using local caches in the auxiliary processor 100, but as the auxiliary processor 100 has become faster and has higher throughput, Figure 1A The high latency shown in becomes unacceptable.

[0007] Figure 1B An improvement is shown in which the auxiliary processor 100 is now able to access instructions and data from local memory 16 (such as video memory). This is a bit like your elementary school-age child being able to attend a new school built near your home instead of traveling across town to attend school. Before the auxiliary processor 100 begins processing work, the CPU 14 copies the instructions and data for the work from system memory 12 to video memory 14 local to the auxiliary processor. The auxiliary processor 100 can now fetch instructions and retrieve data from the relatively low-latency local memory 16 instead of from the relatively high-latency system memory 12. However, this copying takes time, so the execution of instructions by the auxiliary processor 100 is typically delayed until at least some of the memory pages 18 containing the required instructions and data are copied from system memory 12 to the local memory 16 of the auxiliary processor 100.

[0008] About a decade ago, NVIDIA introduced Unified Memory ("UM") (see Figure 1C) solves this problem, allowing the use of a single pointer for both CPU functions and GPU kernels. This allows the coprocessor 100 and the CPU / SOC 14 to directly access each of system memory 12 and video memory 16 (i.e., all system memory is mapped into a common, unified address space). NVIDIA's CUDA 8 and Pascal architectures further significantly improve unified memory functionality by adding on-demand page migration, eliminating the need for developers to write applications that manage data movement between the CPU / SOC 14 and video memory 16. For example, the CUDA "cudaMallocManaged" API call allocates memory that the unified memory system will automatically manage. See, e.g., docs.nvidia.com / cuda / cuda-runtime-api / group__CUDA RT__MEMORY.html; US20150206277; US11182309; US11132326; Apple Yard, “PASCAL AND CUDA 8.0,” NVIDIA (July 2016).

[0009] Figure 2A and Figure 2B The following diagram shows how the unified memory architecture can automatically service page faults on an empty GPU and an oversubscribed GPU, respectively. In the case of oversubscription, memory pages are first evicted from GPU memory to system memory, and then the requested memory is transferred from the CPU to the GPU. None of this needs to be managed by the application - the hardware and driver software handle it automatically. For example, when the GPU 100 core working set exceeds the GPU's physical memory capacity, that is, when the GPU memory becomes "oversubscribed", pages are automatically swapped in and out of GPU memory on demand. The page migration engine allows GPU threads to fault on non-resident memory accesses, so the system can automatically migrate pages from anywhere in the system to the GPU's local memory on demand for efficient processing. Whenever a specific GPU touches data managed by unified memory, this data can be automatically migrated to the GPU's local memory, or the software driver can establish direct access through an available interconnect (e.g., PCIe or NVLINK).

[0010] Many applications benefit from this GPU memory oversubscription and page migration capability. For example, consider a ray tracing engine that shoots rays that can bounce in any direction depending on the surface of the material. If the entire scene doesn't fit in GPU memory (which is often the case given the size of typical ray tracing acceleration data structures), rays can easily hit unavailable surfaces and must be fetched from system memory. In this case, calculating which pages should be migrated to GPU memory at what time is nearly impossible because the directions the rays may take are unpredictable. The Pascal architecture's true GPU page faulting capability provides on-demand page migration from system memory to GPU local memory. The GPU's Unified Virtual Memory (UVM) driver can not only migrate pages from the SOC / CPU (system memory) to the GPU's video memory, but can also migrate pages from a peer GPU's video memory ("peermem") to the GPU's local video memory.

[0011] The UVM driver uses several heuristic-based algorithms to monitor and effectively manage page migrations. Within the page migration engine, there's an access counter circuit—a module or circuit within the GPU hardware—that tracks and counts GPU processor core accesses to memory remote from the GPU (e.g., system memory, peer memory) at a software-programmable granularity. Typically, this granularity corresponds to the default page size of the GPU's video memory. When the number of accesses to a remote page reaches a certain threshold (programmable by software), indicating that the GPU has touched that page more than a certain number of times, the access counter module notifies the UVM driver (via a notification buffer) of binding information (aperture, guest virtual address, instance block pointer, etc.) to uniquely identify that surface. This is achieved by hardware writing to the notification buffer in a preallocated memory range accessible to the driver and simultaneously sending an interrupt to the driver. The UVM driver then decides, based on performance-enhancing heuristics, whether to migrate the page to the GPU's local memory so that the GPU can continue to access the page, but with lower latency. If and when the UVM driver does migrate a page, it takes the usual steps to verify and ensure consistency.

[0012] For more information on page migration mechanisms in such systems, see, for example, US20150082001; US11741015; US11789649; US20140281358; US20140281297; US20140281299; US20140281323; US20140281324; and T. Zheng et al., “Towards high performance paged memory for GP Us,” in 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA), Barcelona, ​​Spain, 2016, pp. 345–357, doi:10.1109 / HPCA.2016.7446077. For more background on GPU memory oversubscription, see Li et al., “A Framework for Memory Oversubscription Management in Graphics Processing Units,” ASPLOS ’19, April 13–17, 2019; developer.nvidia.com / blog / improving-gpu-memory-oversubscription-performance / .

[0013] Until now, for each GPU, there has been a separate access counter circuit or module and an associated "notification buffer" as described above - notifications are generated by different I / Os that share the same space in the notification buffer. But recently, some in the semiconductor industry have proposed that each GPU "chip" be composed of multiple chip modules ("MCMs") - that is, package-level integration of multiple die modules to build larger logic processing systems that can achieve continued performance scaling beyond Moore's Law. For example, there have been proposals to divide the GPU into more basic GPU modules ("GPMs") that are easier to manufacture, each on its own die, and use high-bandwidth, energy-efficient signaling techniques to integrate multiple die on the package. See Arunkumar et al., “MCM-GPU: Multi-Chip-Module GPUs for Continued Performance Scalability,” International Symposium on Computer Architecture (ISCA) (ACM 2017), research.nvidia.com / publication / 2017-06_mcm-gpu-multi-chip-module-gpus-continued-performance-scalability, http: / / dx.doi.org / 10.1145 / 3079856.3080231; “TSMC’s New Wafer-on-Wafer Process to Empower NVIDIA and AMD GPU Designs,” in Proceedings of the IEEE International Symposium on Computer Architecture (ISCA) (ACM 2017), http: / / dx.doi.org / 10.1145 / 3079856.3080231; Designs,” engineering.com (May 3, 2018), engineering.com / story / tsmcs-new-wafer-on-wafer-process-to-empower-nvidia-and-amd-gpu-designs.

[0014] Such as existing technology Figure 3A and Figure 3BThis approach for an example graphics processing unit ("GPU"), as shown in

[15] , implements the MCM-GPU as a collection of GPMs that share resources and presents to software and programmers as a single monolithic (meaning a single die), integrated, single high-performance processing system. Multi-chip modules can be replicated (identical) or dedicated, as needed for custom functionality, process technology, area, and / or power requirements. Multi-chip modules are connected together by wires in a common integrated circuit package (e.g., by stacking in one implementation, see

[15] ). Figure 4A 、 Figure 4B ) to operate together as a single, integrated processing system. This approach enables resource sharing of underutilized fabric within a single processing system and eliminates the hardware duplication that would be required if each die contained its own completely independent processing system.

[0015] There are challenges in coordinating operations between different multi-chip modules to continue to provide hardware support for page migration in multi-die GPUs. Summary of the Invention

[0016] In one aspect, a method is provided according to the present disclosure. The method includes transmitting a memory binding to a first die and at least one additional die; subsequently receiving a memory access request; and generating a page migration notification on the first die or the at least one additional die based on associating the received memory access request with the previously transmitted memory binding.

[0017] In another aspect, the present disclosure provides a multi-chip module, comprising: at least one input / output port; a first memory management unit (MMU) configured to exchange memory binding information with a second MMU on another multi-chip module; and an access counter circuit operatively coupled to the first MMU, the access counter circuit tracking memory accesses performed through the at least one input / output port and selectively generating notifications for memory page migration; wherein the access counter circuit is configured to track memory accesses originating from an engine disposed on the other multi-chip module.

[0018] In yet another aspect, the present disclosure provides a method for commanding access counters for page migration, comprising: targeting all access counters on all multi-chip modules; and broadcasting a command to the access counters on all multi-chip modules.

[0019] In a further aspect, the present disclosure provides a host method for commanding an access counter for page migration. The method includes: targeting the access counter based on a tag received from a notification from the access counter; retrieving a die identifier based on the tag; and sending a command to the counter on the die indicated by the die identifier. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A 、 Figure 1B 、 Figure 1C An example prior art system architecture including a CPU and a secondary processor such as a GPU is shown.

[0021] Figure 2A 、 Figure 2B A prior art example of page migration between system memory and GPU memory is shown.

[0022] Figure 3A An example multi-chip module (MCM) GPU is shown.

[0023] Figure 3B An internal high level block diagram of an example of a multi-chip module GPU is shown.

[0024] Figure 4A 、 Figure 4B An example non-limiting MCM package is shown, wherein Figure 4A shows an example of 2.5D die-to-die routing, Figure 4B An example of 3.5D die-to-die routing is shown.

[0025] Figure 5 A high-level schematic diagram showing page migration within a computing system including an MCM multi-die GPU is shown.

[0026] Figure 5A A high-level schematic diagram showing page migration within a computing system including two or more MCM multi-die GPUs is shown.

[0027] Figure 6 A block diagram of a high-level MCM multi-die GPU architecture including memory access request and data routing between dies is shown.

[0028] Figure 6A A block diagram of a high-level MCM multi-die GPU architecture with four dies in the same package is shown.

[0029] Figure 7 An example, non-limiting memory access counter circuit is shown.

[0030] Figure 8An example, non-limiting binding flow diagram is shown.

[0031] Figure 9 An example host purge operation is shown. DETAILED DESCRIPTION

[0032] Example embodiments make it possible to provide a unified view of page migration when circuits such as access counter circuits (for tracking and counting GPU processor cores to memory remote from the GPU) and related notification mechanisms are instantiated or occur on or across multiple dies.

[0033] In particular, the access counter module is tightly coupled to the GPU's I / O. In an example dual-die GPU configuration, the primary die has PCIe+NVLink ports, and the secondary die has a C2C port. Each die has an access counter module that tracks GPU processor core accesses originating from that die's I / O. In an example embodiment, each access counter module is allocated its own notification buffer (storage space for writing notifications).

[0034] However, in different dual-die configurations (with or without MIG mode enabled), GPU processor engines / cores from any die (MIG) are allowed to access I / O in the same or different die. The access counter module cannot correctly generate notifications to the software unless each access counter module is updated with information related to the GPU processor core's I / O access (e.g., the instance block pointer (IBP) bound to the graphics page table corresponding to the absolute virtual engine ID carried by the GPU processor core access). Note: The acronym "MIG" should not be confused with "migration", but stands for something completely different, namely: "Multi-Instance GPU". See the description and more details at the end of this specification. In addition, the term "engine" in this document can include a processing core, a cluster of processing cores, or any other unit or functional block that accesses memory to perform work.

[0035] Thus, example embodiments herein provide a hardware implementation that allows for multi-die consistency of engine page table binding state, virtual engine identifier (VEID) information, and other instance block and page table data, to allow any die's access counter circuit to accumulate accesses and send accurate notifications to software to suggest and implement page migrations. Example, non-limiting embodiments also provide a hardware mechanism to orchestrate clearing (CLEAR) operations of one or all counters across multiple die.

[0036] Example Implementation

[0037] Figure 5 An example block diagram of a computing system 50 including a multi-die GPU 100 is shown, including a first die 100 ( 0 ) and a second die 100 ( 1 ). Figure 5A Another example embodiment of a computing system 50' is shown having multiple multi-die GPUs 100, 100', each of which has access counter circuitry to enable, for example, page migration between system memory and video memory and between the respective GPU video memories 16, 16'. In the example shown, each of the die dies 100(0), 100(1) has access counter circuitry for monitoring and recording memory accesses. These access counter circuits are each responsible for generating the above-described notifications. However, typically, the access counter circuitry on one die needs to be aware of what the other die are doing with respect to "memory binding" (i.e., mapping between the graphics or compute engine and the shared memory address space) in order to generate such notifications.

[0038] Figure 6 A more detailed view is shown. In the example shown, the die 100(0), 100(1) includes multiple clusters of GPCs 104, each of which includes multiple processing engines / cores 102 (e.g., streaming multiprocessors in one particular example). These GPCs 104 are what perform work, i.e., thread blocks that are part of a GPU software application. The work can be graphics or compute, and can perform any number of tasks, such as image generation, ray tracing, object collision testing, machine learning (e.g., deep neural networks or other neural networks), matrix or tensor calculations, or any number of other applications.

[0039] GPC 104 accesses memory external to GPU 100 via memory crossbar switch (MXBAR) 106. GPC 104 or its constituent processor cores may be assigned an identifier, such as a VEID (virtual engine identifier), for use in accessing memory. Figure 6As shown, MXBAR 106 communicates with corresponding high-speed hubs 120, 122 via a hub crossbar switch (HXBAR) 108. These high-speed hubs 120, 122 are in turn connected to external input / output ports. In this example, die 100(0) has active HSHUBNVL hubs 120(0), 120(J) connected to corresponding NVLINK I / O ports, and die 100(1) has active HSHUBNVL hubs 122(0), 122(K) connected to corresponding C2C (chip-to-chip) I / O ports. Through these I / O ports, GPC 104 can access memory, such as system memory, "peer" memory such as video memory connected to a peer GPU, and the like.

[0040] In one embodiment, the two dies 100(0), 100(1) are identical in structure, so they each have on-chip HSHUBC2C and HSHUBNVL hubs. Figure 6 Only a subset of these hubs are shown because, in one embodiment, one or another of these hubs may be inactive on a given die as shown. Figure 6 In the configuration shown, by routing access requests between the die via the C2C HBI interconnect, GPCs 104 and engines on either die can access memory via the NVL ports on die 100(0), and GPCs and engines on either die can access memory via the C2C ports. In other words, in this embodiment, engines on either die can access memory through the I / O ports of either die. See, for example, Synchronous MMU. Figure 6A Another embodiment is shown that includes four dies in the same package, which are not necessarily identical (in this example, the top two dies are identical, the bottom two dies are identical, but the top die and the bottom die are not identical).

[0041] Figure 6 and Figure 6A Also shown is an access counter circuit or module 118 as described above. In this example, each die 100(0), 100(1) has an access counter circuit or module. Thus, there is an access counter circuit or module 118(0) disposed on die 100(0), and an access counter circuit or module 118(1) disposed on die 100(1). In the example embodiment, each die has exactly one access counter circuit or module 118 that tracks external memory accesses via the I / O port of the die where the counter circuit is located.

[0042] exist Figure 6 In the dual die example shown, the two access counter circuits or modules 118(0), 118(1) may be instances of the same access counter circuit or module. Figure 7 As shown, access counter circuitry 118 includes any number of counter groups. Whenever a hub on a die accesses remote memory through its hub, the corresponding access counter circuitry 118 on that die increments one of its counters corresponding to the (programmable-sized) memory page / block containing the remotely accessed memory address. When the counter reaches a certain (programmable) threshold, indicating that the die has accessed that memory block / page a certain or determined number of times, access counter circuitry 118 generates a notification. In an exemplary embodiment, the software driver does not need (or desire) to be notified by hardware every time the GPU accesses any page in remote memory. Instead, it is notified when the GPU accesses a particular memory page more than a certain (programmable) number of times, so that the software driver can consider migrating that page to GPU-local memory. In this case, the software driver is making a heuristic determination that the GPU is likely to frequently access the same page in the future that it has accessed multiple times in the past. This is somewhat like a customer receiving a discount rewards card from a coffee shop after purchasing a certain number of coffees, which makes it more efficient for the customer to obtain the same coffee from the same shop in the future.

[0043] In an exemplary embodiment, the access counter circuit 118 on each die operates independently of the other access counter circuits on each other die. For example, in the illustrated embodiment, the access counter circuit 118(0) on die 100(0) is informed of every memory binding and access via the NVL ports of die 100(0), but is unaware of the details of memory binding and accesses made by another die via the C2C ports of the other die. Similarly, the access counter circuit 118(1) on die 100(1) is informed of every binding and access made via the C2C ports of the die, but is unaware of the details of accesses made by die 100(0) via the NVL ports on the other die. However, all accesses across the two die dies 100(0), 100(1) are recorded and counted by one or the other (i.e., a combination thereof) of the two access counter circuits 118(0), 118(1) operating independently but in series. In the example embodiment, accesses tracked by access counter circuit 118 ( 0 ) are therefore not tracked by access counter circuit 118 ( 1 ), and vice versa.

[0044] Binding information passed to the access counter circuit

[0045] Thus, in an example embodiment, each I / O link memory access initiated by GPU die 100 will cause relevant information to be written to access counter circuit 118 on that die. The internal on-die link on die 100(0) between HSHUBNVL 120 and access counter circuit 118(0) provides certain information to access counter circuit 118 for each access on the NVL port:

[0046] Virtual Address (VA)

[0047] The PA value used to index the counter within the access counter circuit 118

[0048] VEID (Virtual Engine Identifier)

[0049] GFID

[0050] ·etc.

[0051] Similarly, the internal on-die link between HSHUBC2C 122 and access counter circuit 118 ( 1 ) on die 100 ( 1 ) provides the same type of information to the access counter circuit for each access on the C2C port.

[0052] Memory Management Unit and Binding Operations

[0053] Reference again Figure 6 , also shown are several additional GPU components for supporting virtual memory access and binding. First among the additional GPU components is a memory management unit (MMU) 110. Each die has an MMU 110, which is used, among other things, to manage virtual memory in a manner that hides memory latency. Also shown is a front-end context switch (FECS) 116, which configures the MMU 110 to provide correct virtual-to-physical address mapping. Typically, execution of a given GPU context waits until the FECS 116 has configured all page tables associated with that GPU context. Once the FECS 116 generates a "done" signal indicating that the page table and related configuration steps are complete, the scheduler or work distributor (not shown) within the GPU can begin scheduling tasks associated with the page table. The FECS 116 associates each page table with an address space identifier (ASID) via a bind command sent to the MMU 110. Each thread block is, in turn, associated with an ASID, which is attached to each virtual memory request generated by the thread block during execution. See, for example, US20140123146.

[0054] An additional communication path called "FBHUB" (Frame Buffer Hub) 114 can also carry message packets between the die dies 100(0), 100(1) via an additional inter-die crossbar switch CXBAR 112. For example, access counter circuit 118 can write to a memory buffer at a software-specified memory address via the FBHUB. As described above, the C2C HBI (Chip-to-Chip High Bandwidth Interface) between the die dies 100(0), 100(1) passes data back and forth between the die.

[0055] In the exemplary embodiment, the MMU 110(0) on one die 100(0) communicates with the MMU via the C2C HBI interface (see Figure 6 The FBHUB 114 communicates with other MMUs 110(1) residing on other dies in a multi-die GPU (see vertical "bind" arrow on the right). Communications may include information about memory requests, memory responses, bind requests and responses, TLB invalidate requests and responses, and various acknowledgements, fault reports, ATS requests and responses, ATS descriptor (ATSD) invalidations, video memory access bits (VABs), dump requests, etc. In some embodiments, these communications may continue to other dies via the FBHUB 114. More details on the binding process can be found in Synchronizing MMUs.

[0056] In the exemplary embodiment herein, when the MMU 100 returns a bind ack (acknowledgement), it provides the absolute VEID from the original bind request as well as the relative VEID. This new acknowledgment method allows the FBHUB for each die to be symmetrical when handling graphics binding. This symmetry provides additional benefits as we scale the MCM design to include more than two dies.

[0057] In an exemplary embodiment, the binding table is a data structure that stores information about the association between virtual memory regions and physical memory frames allocated to different engines / processes running on the GPU. Each entry in the binding table may include: the virtual address range used by the process, the starting address of the physical memory frame allocated for the corresponding virtual address range, and the access rights (read, write, execute) associated with the memory region for protection.

[0058] The page table binding cache (PDB cache) is a hardware cache located in (each) MMU 110. It can store translations between recently used virtual addresses and physical addresses and is intended to improve performance by reducing the need to access the home page table stored in memory every time memory is accessed. When a program attempts to access data using a virtual address, the MMU 110 first checks the page table binding cache. If a translation (mapping between virtual addresses and physical addresses) for that particular address is found in the cache, the MMU 110 can efficiently translate the address and authorize access to the corresponding physical memory location. This avoids the need to access the home page table (which is typically slower to access than the cache), thereby reducing memory access latency and improving overall performance.

[0059] The binding table and / or PDB cache in MMU 110 may be accessed by a hub translation lookaside buffer (TLB) or a process that accesses the TLB and causes a TLB miss. As a result of a binding table or PDB cache lookup or page table walk, the TLB may update the TLB entry with an entry provided by MMU 110.

[0060] As described above, when a new binding is created, an instance block pointer (IBP) is associated with the absolute virtual engine ID carried by the GPU processor core that initiated the bind request. The IBP is stored in the graphics page table to represent the graphics binding and therefore corresponds to the GPC 104 (and, in some embodiments, the engine / processor core 102) bound to the memory being accessed during the bind. The MMU 110 has this information (i.e., for all MIG partitions), but the access counter circuit 118 also needs to know which engine and corresponding IBP is already bound to the memory page it is counting accesses to. This is because the access counter circuit 118 provides and passes this context information to the software driver in the form of notifications for the purpose of on-demand page migration. In an exemplary embodiment, the access counter circuit 118 writes (a) information from the binding process and (b) accesses by the processor core 102 to the notification buffer context.

[0061] Make each access counter circuit aware of each binding across multiple dies

[0062] To provide the above, the exemplary non-limiting embodiment provides a hardware mechanism that enables the independent access counter circuit 118 (at each die) to know the engine page table bindings at all die, thereby providing accurate notification to the software (UVM driver) for page migration in a multi-die configuration. This is achieved by each MMU 110 on each die broadcasting specific binding information to all other MMUs on all other die (or in Figure 6 In the dual-die example, MMU 110(0) broadcasts the binding information to MMU 110(1), and vice versa. Once an acknowledgment of the broadcast is received, the binding information that the MMU 110 on each die now possesses is used to update the access counter circuit 118 on the local die. In this way, each access counter circuit 118 on each die is continuously synchronized with each binding on any die as binding occurs. Then, when any engine on any die initiates a further memory access, the access counter circuit 118 on the die through which the access is made will identify the VEID associated with the engine's memory access request and can correlate it with other information (e.g., IBP) from the binding table that the access counter circuit stores in a lookup table (LUT).

[0063] In traditional GPUs, because software requires the IBP and GPU virtual address (GVA) to process notifications, the access counter module or circuit 118 maintains a lookup table (LUT) for VEID to IBP mapping. For MIG, the access counter circuit needs to report the MIG-relative VEID in notifications sent to software, although it still needs to maintain the LUT based on the absolute VEID. For more information about MIG, see the following information.

[0064] In an exemplary embodiment, at graphics bind time, access counter circuit 118 obtains the required information from MMU 110 to populate its LUT (see Figure 7 In a multi-die configuration, since pattern binding can occur from any die independently of the other dies, the exemplary non-limiting embodiment provides a mechanism to make the same VEID-IBP LUT information available to each access counter module or circuit 118 on all dies and keep them consistent.

[0065] Specifically, if Figure 8As shown, when any graph bind occurs from die X (e.g., die 100(0)), the example embodiment ensures that the access counter circuit (e.g., 118(1)) of die Y (e.g., die 100(1)) is updated with the bind information before the LUT in die X (Y represents all other die). In this mechanism, the MMU 110 broadcasts the bind information to all die dies via a dedicated die-to-dielet interconnect (C2C HBI). The MMU 110 also ensures that the LUT of the access counter circuit of die Y is updated before sending a bind-ack to die X. In this example, die Y receives a bind-ack for a bind that was not initiated by itself, and it does not need to forward the bind-ack to the client. A new identifier bit or other indicator in the bind-ack distinguishes this special bind-ack from a regular bind-ack (when Y is the bind initiator).

[0066] Example: Binding a Stream

[0067] In the previous design, FBHUB 114 used the following sequence to prepare the binding-related fields in the notification message of access counter circuit 118:

[0068] 1. FECS sends a bind request to FBHUB, which contains the relative veid, IBP, aperture and subid1 of the SMC (processor core).

[0069] 2. FBHUB forward FECS is bound to MMU via subid2.

[0070] 3. FBHUB caches the binding information (veid+ibp+aperture) and stores it locally in CAM1. The key is subid2.

[0071] 4. The MMU returns a bind-ack / nack (negative acknowledgment) with subid 2 and absolute veid.

[0072] 5.FBHUB checks the binding ack and finds the local CAM1 with subid2, and moves the binding information to CAM2. The key is the absolute veid.

[0073] 6. FBHUB returns a binding ack to FECS using subd1.

[0074] 7.ACTRS / ATCRSTC receives updates with absolute veid.

[0075] 8.ACTRS generates GVA notification with absolute veid to FBHUB.

[0076] 9. FBHUB uses the absolute veid to look up the local CAM2 and fills in the engine id, aperture and instance block pointer in the final notify message.

[0077] In the example embodiment of this document, FECS 116 can send the binding to the local die, but remote access can be forwarded to the remote die. In this case, the ACTRs module in the remote die generates a notification to the remote FBHUB 114. The remote FBHUB 114 does not have the corresponding binding information.

[0078] Example embodiments may use an existing interface (mmuhubtlb2hub) to pass bonding information from MMU 110 to FBHUB 114 in a remote die. Other embodiments may use a new or different path to transmit this information from MMU 100 to FBHUB 114. In an example embodiment, MMU 110 reformats an existing bonding ack packet to return bonding information (remote die) or bonding ack (local die) with the following fields:

[0079] subid / Not used for remote binding information Absolute engine id (same as existing VEID engine id in the bind ack) resp (bind ack / nack) Relative engine id (new id from local bind request) IBP instance block pointer (new pointer from local bind request) Aperture (new aperture from local bind request) Flag for remote bind information or local bind ack (new)

[0080] In an exemplary embodiment, FBHUB 114 monitors for new bind acknowledgement packets to update the internal binding table. The bind_ack packet is used to update the internal binding table to cache information based on the FECS bind request initiated by the die.

[0081] Figure 8An example multi-die binding flow is shown where die 0 is the primary die and die 1 is the secondary die. On each die, a client can send a bind request. For die 0, the bind request will pass through die 0 MMU 110(0). This will cause MMU 110(0) to broadcast the bind request ("D0 Bind Req.") to the remote die, in this case MMU 110(1) on die 1. Remote die 1 then forwards a Bind ACTR Bind Ack to the remote die's FBHUB 114(1). At the same time, the remote die's MMU 110(1) sends a D1 bind ack message to the initiating die's MMU 110(0), which passes the ack back to D0 FBHUB 114(0) as usual. The new part is the dual die interaction, which involves passing the D0 Bind Request to 100(1) and generating a Bind ACTR Bind Ack on 100(1) for the remote die to apply to its FBHUB 114(1). In the example embodiment, the flow is symmetric for a client Bind Request entering die 100(1), resulting in a Bind ACTR Bind Ack being generated on the other die D0's FBHUB 114(0).

[0082] For example, assume that a client is using the master die 100(0) for a binding request. The client's binding request enters the master die's MMU 110(0). The master die's MMU 110(0) forwards the request to the remote die's MMU 110(1) via the inter-die chip-to-chip interconnect. The remote die's MMU 110(1) forwards the binding information to the remote die's FBHUB 114(1), which transmits the binding information to the remote die's access counter circuit 118(1). Once the access counter circuit 118(1) has received the update (to avoid contention), the remote die's MMU 110(1) provides an acknowledgment to the master die's MMU 110(0). The master die's MMU 110(0) will then send a binding acknowledgment to the master die's FBHUB 114(0). The access counter circuit 118(0) of the master die will receive the same binding information via the binding confirmation sent by the master die's MMU 110(0) to the master die's FBHUB 114(0). In this way, the access counter circuits 118(0), 118(1) on each die receive the binding information. Although this example shows a system with a single master die and a single slave die, it can be extended to any number of slave dies to support scalable MCM designs.

[0083] In this example, the access stores the binding information in internal RAM wherever the MMU 110 sends it to the access counter circuit 118. The access counter circuit 118 checks the VEID and based on that knows that this is the binding information including the engine ID, IBP and aperture.

[0084] After the binding is complete, when the access counter circuit 118 later receives a remote memory update with the same VEID based on the engine with the VEID that generated the memory access request, the access counter can look up the corresponding binding information and use it to trigger a notification. If the corresponding counter has been incremented enough times to exceed the threshold, this indicates that there is enough traffic for that absolute engine ID VEID to notify the software driver that the virtual memory page associated with the corresponding IBP is a candidate for page migration. The access counter circuit 118 will send the relative engine ID, IBP, and aperture to the software driver. The software driver can then migrate the corresponding page from remote memory to local memory.

[0085] Clear counter

[0086] In addition to the above, the exemplary non-limiting embodiment provides a hardware mechanism for software to effectively manage the access counter circuit (e.g., clear one or all counters from any remote die). The exemplary embodiment allows software to clear the access counter circuit 118 (invalidate and reset the tracker) for a variety of reasons (during boot time or after processing a notification / successful page migration). In a multi-die configuration, the hardware provides multiple ways to clear the access counter circuit 118: In the exemplary embodiment, clearing can be done in three ways.

[0087] 1. Clear all labels in all ACTR modules as targets. (This is broadcasting)

[0088] 2. Target a purge to a specific tag located on the local die. (This is not sent die to die)

[0089] 3. Target a purge to a specific tag located on a remote die. (This is forwarded to the necessary die).

[0090] More specifically, clears can be triggered via software writable registers. Given that the notification buffers are now separate for each die, they have their own control registers, and die information is visible to software via notifications (based on the notification buffer that generated the notification). Therefore, software can choose to clear all trackers / counters in a die by writing to the corresponding trigger registers on the die. See Figure 9 .

[0091] Software can use host (HOST) methods (i.e., methods that consume and produce host data on the SOC / CPU) to perform clear operations. Host methods provide control over the scope of the clear operation and allow targeting all counters in the GPU (including counters located on all dies) or targeting a specific counter regardless of which access counter module 118 it resides in.

[0092] In an example embodiment, a clear can be issued via a host method to target all access counters in the system. For example, during a GPU reset or before an application teardown and context switch operation, it may be necessary to clear all access counters to prevent accidental notifications. When the FBHUB 114 receives a method-based clear targeting all counters, the clear is sent to the local access counter module 118 and the access counter modules of all other die. The die-to-die interconnect (CTC HBI) is used to broadcast this information.

[0093] In an example embodiment, a purge can also target only specific trackers / counters 118 based on a tag (received from a previous notification) and can be issued to a die that did not generate a notification. To address this issue in an example embodiment, the access counter module or circuit 118 (at the die X that generated the notification) encodes the dielet_id in the notify_tag field of the original notification. This allows the access counter module or circuit 118 (at the die that received the purge operation) to look up the dielet_id from the tag (received with the purge operation) and forward the purge to the correct die (X in this example) via the die-to-die interconnect.

[0094] Multi-instance GPU

[0095] Regarding the "MIG" mentioned above, this acronym stands for NVIDIA's Multi-Instance GPU ("MIG") feature, which allows a GPU to be securely partitioned into multiple separate GPU virtual instances, providing multiple tenants with their own dedicated GPU resources. MIG enables multiple GPU instances to run in parallel on a single, physical GPU. For example, see the "NVIDIA Multi-Instance GPU User Guide," docs.NVIDIA.com / datacenter / tesla / mig-User-Guide / index.html (accessed November 17, 2022); US20230288471. To the GPU application, the instance it is running on "appears" to be a complete GPU, even though it is actually virtualized. This feature is particularly beneficial for workloads that do not fully saturate the GPU's compute capacity and allows different tenants to run different workloads in parallel, maximizing the utilization of the GPU hardware. MIG ensures that one tenant cannot affect the work or schedules of other tenants, and also provides enhanced security isolation. Each instance's processor has independent access to the entire memory system—the on-chip crossbar switch ports, L2 cache banks, memory controllers, and DRAM address buses are all uniquely assigned to a single instance. This ensures that one tenant's workload can run with predictable throughput and experience predictable latency with the same L2 cache allocation and DRAM bandwidth, while another tenant is threshing its cache or saturating the DRAM interface. MIG can partition available GPU compute resources (including "GPC" clusters of processing cores such as streaming multiprocessors or SMs and other GPU engines such as replication engines or decoders) to provide well-defined quality of service (QoS) and provide fault isolation for different applications such as virtual machines (VMs), containers, or processes so that a faulty process does not take down others.

[0096] Example use case

[0097] The technology disclosed herein can be incorporated into any processor that can be used to process neural networks, such as a central processing unit (CPU), a graphics processing unit (GPU), an intelligence processing unit (IPU), a neural processing unit (NPU), a tensor processing unit (TPU), a neural network processor (NNP), a data processing unit (DPU), a vision processing unit (VPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc. Such a processor can be integrated into a personal computer (such as a laptop), a data center, an Internet of Things (IoT) device, a handheld device (e.g., a smartphone), a vehicle, a robot, or any other device that performs inference, training, or any other processing of a neural network. Such a processor can be used in a virtualized system so that an operating system executed in a virtual machine on the system can utilize such a processor.

[0098] As an example, a processor incorporating the techniques disclosed herein can be used to process one or more neural networks in a machine to identify, classify, manipulate, process, operate, modify, or navigate physical objects in the real world. For example, such a processor can be used in an autonomous vehicle (e.g., a car, motorcycle, helicopter, drone, airplane, ship, submarine, logistics robot, etc.) to enable the vehicle to move in the real world. In addition, such a processor can be used in a robot in a factory to select parts and assemble the parts into assemblies.

[0099] As an example, a processor incorporating the techniques disclosed herein can be used to process one or more neural networks to identify one or more features in an image, or to change, generate, or compress an image. For example, such a processor can be used to enhance images rendered using rasterization, ray tracing (e.g., using NVIDIA RTX), and / or other rendering techniques. In another example, such a processor can be employed to reduce the amount of image data transmitted from a rendering device to a display device over a network (e.g., the Internet, a mobile telecommunications network, a WIFI network, and any other wired or wireless network system). Such transmission can be used to stream image data from a server or a data center in the cloud to a user device (e.g., a personal computer, a video game console, a smartphone, other mobile devices, etc.) to enhance services that stream images, such as NVIDIA GeForceNow (GFN), Google Stadia, and the like.

[0100] As an example, a processor incorporating the techniques disclosed herein can be used to process one or more neural networks for any other type of application that can utilize a neural network. For example, such applications can involve translation from one spoken language to another, recognition and negation of sounds in audio, detection of anomalies or defects in the production of goods and services, monitoring of living and / or non-living things, medical diagnosis, decision making, etc.

[0101] As an example, a processor incorporating the techniques disclosed herein can be used to implement neural networks such as large language models (LLMs) to generate content (e.g., images, videos, text, articles, audio, etc.), respond to user queries, solve problems in mathematics and other fields, and so on.

[0102] All patents, patent applications, and publications cited herein are incorporated by reference herein for all purposes as if expressly set forth, and no statement in any such incorporated by reference item that might be construed as a disclaimer of subject matter shall apply as a disclaimer of subject matter herein unless such disclaimer is expressly stated and confirmed in the written record of this specification or the prosecution history of this patent.

[0103] While the invention has been described in connection with what is presently considered to be the most practical and preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments, but on the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method comprising: transferring memory bindings to the first die and at least one additional die; subsequently receiving a memory access request; as well as A page migration notification is generated on the first die or the at least one additional die based on associating the received memory access request with the previously transmitted memory binding.

2. The method of claim 1 , wherein transmitting comprises: The memory binding is communicated to a first circuit on the first die and a second circuit on the at least one additional die, and the generating is performed by only one of the first circuit and the second circuit.

3. The method according to claim 2, further comprising: A memory bind request is received, and a responsive memory bind acknowledgement is broadcast to each of the first circuitry on the first die and the second circuitry on the at least one additional die.

4. The method of claim 2, wherein generating comprises: The page migration notification is conditionally generated if the received memory access request causes one of the first circuitry and the second circuitry to indicate that memory access traffic for pages associated with the transferred memory binding is above a threshold.

5. A multi-chip module comprising: at least one input / output port; a first memory management unit (MMU) configured to exchange memory binding information with a second MMU on another multi-chip module; as well as an access counter circuit operatively coupled to the first MMU, the access counter circuit tracking memory accesses performed through the at least one input / output port and selectively generating a notification for a memory page migration; The access counter circuit is configured to track memory accesses originating from an engine disposed on another multi-chip module. The multi-chip module of claim 5 , wherein the engine comprises a graphics engine.

7. The multi-chip module of claim 5, wherein the engine comprises a compute engine.

8. The multi-chip module of claim 5, wherein the multi-chip module and the another multi-chip module are configured to be interconnected within a common package.

9. The multi-chip module of claim 5, wherein the access counter circuit is further configured to generate a notification when a memory access count by the engine to a particular memory page exceeds a predetermined threshold.

10. The multi-chip module of claim 5, wherein the memory binding information comprises at least an identifier of the engine and an associated instance block pointer.

11. The multi-chip module of claim 5, wherein the memory binding information further comprises an aperture and a remote binding indicator.

12. A method for controlling an access counter for page migration, comprising: Target all access counters on all multi-chip modules; as well as Broadcasts the command to the access counters on all multi-chip modules. The method of claim 12 , wherein the command comprises a clear command.

14. The method of claim 12, wherein the broadcast operation sends a command to all die using a link on the die to access all access counter instances.

15. A host method for commanding an access counter for page migration, comprising: targeting the access counter based on a tag received from a notification from the access counter; retrieving a die identifier based on the tag; as well as A command is sent to the access counter on the die indicated by the die identifier. The method of claim 15 , wherein the command comprises a clear command.

17. The method of claim 15, wherein the notification from the access counter indicates that the access counter has counted at least a determined number of memory accesses by an engine to a memory page.

Images