Unified virtual memory system in multi-GPU environment

Abstract

The invention provides a unified virtual memory system in a multi-GPU environment, each GPU is provided with an RDMA cache, the RDMA cache is used for storing remote data obtained through an RDMA mechanism, and all computing units on the GPU share the RDMA cache; a reverse access counter is arranged on each GPU, and the reverse access counter is used for recording the mode of a far-end GPU accessing a data page, so that data page migration is carried out when the mode of the far-end GPU is inconsistent with the currently accessed GPU and the mode of the far-end GPU reaches a preset threshold value. According to the unified virtual memory system in the multi-GPU environment, the cross-GPU access performance in the multi-GPU environment is effectively improved based on the global shared RDMA cache and the reverse access counter.

Description

Technical Field

[0001] This invention relates to the field of virtual memory technology, and in particular to a unified virtual memory system in a multi-GPU environment. Background Technology

[0002] Graphics Processing Units (GPUs) were initially used for rendering computer graphics. Due to their massively parallel computing capabilities, they have also been widely applied in scientific computing, machine learning, and signal processing in recent years. With the rise of Large Language Models (LLMs) in recent years, the industry's demand for computing power has increased significantly. A single large language model can have hundreds of billions of parameters. Both LLM training and inference heavily rely on the computing power of AI chips such as GPUs to capture input semantics and infer output results. During the inference process of a large language model, GPU memory needs to store the model's weights and activation values, while the training process requires storing gradient values ​​from the neural network's backward computation, optimizer parameters, etc., potentially requiring several times more memory than inference. As one of the most important AI chips, GPUs utilize large-capacity, high-bandwidth memory, multi-core computing focused on massively parallel computing efficiency, and powerful scheduling capabilities to enable efficient LLM training and inference.

[0003] As upper-layer applications continue to expand, GPU architecture is also undergoing changes. In the traditional single-GPU era, early GPUs were primarily used for graphics rendering and media encoding / decoding in user terminal products, rather than general-purpose computing tasks. Some engineers noticed the efficient parallel computing capabilities of GPUs and proposed GPU-based hardware architectures for general-purpose computing, such as scientific computing, machine learning, and deep learning. To meet this demand, chip manufacturers also launched their own general-purpose computing software stacks (such as NVIDIA's CUDA and AMD's HIP), and significantly enhanced general-purpose computing capabilities in new GPU hardware architecture designs, such as introducing application-specific integrated circuits (ASICs) for matrix multiplication. GPUs as general-purpose computing chips are sometimes also called GPGPUs (General Purpose GPUs) to emphasize their difference from earlier GPU products focused solely on graphics. All of these changes pertain to the architecture of a single GPU chip.

[0004] In recent years, with the development of large language models, the law of scale has been discovered, stating that increasing the number of parameters will bring about a qualitative change in the inference ability of large language models. The expansion of model parameters also poses new challenges to the storage capacity of modern GPUs. Flagship high-performance GPUs on the market often only provide around 100GB of video memory, while the weight data of some large language models already exceeds this value, making single-GPU inference challenging, and corresponding model training even more impractical. Furthermore, transistor expansion is becoming increasingly difficult, making it hard to increase the number of cores on a single GPU chip. Therefore, multi-GPU architectures have begun to be adopted, connecting multiple GPUs to collaboratively complete computational tasks. In addition, even for smaller deep neural network models, to achieve efficient model training and inference, multi-GPU architectures are being used, partitioning data along different dimensions, such as different batches of data, different layers of the neural network, and different dimensions of tensors, which can also bring performance improvements. Multi-GPU architectures require communication library frameworks and efficient communication hardware support between multiple GPUs, as well as multi-GPU management capabilities at the application, GPU driver, and runtime levels. Figure 1 The image shows a multi-GPU architecture based on PCIe connectivity. Figure 2 The diagram shows the interconnect architecture of a multi-GPU system with NVLINK connection provided, where PCIe connection is retained as an auxiliary connection.

[0005] Virtual memory is a memory management mechanism jointly implemented by the operating system and hardware. When using virtual memory to manage CPU main memory, virtual memory provides each process with a logically isolated address space. Its size is not limited by the actual physical memory installed in the computer system, and contiguous data in the virtual address space may not reside in contiguous physical memory. Each process's virtual address space is managed by the operating system. The address space of virtual memory is often managed at the page level, and the operating system uses a page table to record the mapping between the virtual address of a page and its actual physical address. When a process accesses data in physical memory using a virtual address, it first needs to perform virtual-to-physical address translation to obtain the data's actual physical address, and then use the physical address to send a request to memory. Virtual-to-physical address translation depends on the page table stored in memory and also on dedicated address translation hardware. In the earliest processor architectures that supported virtual-to-physical address translation, it was necessary to traverse page tables at all levels. The dedicated hardware unit on the CPU, the Memory Management Unit (MMU), provides the ability to perform page table walks, which can efficiently read page tables at all levels to obtain the required page table entries (PTEs). To address frequent virtual-to-physical address translation requests and reduce the overhead of virtual-to-physical address translation, a dedicated hardware Translation Lookaside Buffer (TLB) is introduced. The TLB's function is to cache page table entries obtained after page table roaming in its hardware storage. With the TLB, virtual-to-physical translation requests will first query the TLB. If the TLB hits, the corresponding page table entry is returned directly. If the TLB fails, a fallback to the MMU's page table roamer is required to traverse all levels of the page table, find the corresponding address mapping information, and cache the information in the TLB.

[0006] Virtual memory on a GPU works similarly. The GPU Memory Management Unit (GPUGMMU) provides address translation capabilities for the entire GPU. To accelerate address translation, each Compute Unit (CU) has a private L1 TLB, while all CUs share a single L2 TLB. During memory access, the address translation request is first sent to the L1 TLB; if the L1 TLB is missing, the request is sent to the L2 TLB; if the L2 TLB also fails to find the corresponding entry, the request is sent to the page table rovers, which read the corresponding page table entry from the GPU page table and return the corresponding physical address.

[0007] Before the introduction of Unified Virtual Memory (UVM), GPU memory models relied on manual host-to-device (H2D) and device-to-host (D2H) memory copying. This required copying storage from host-side RAM to GPU memory before kernel execution by calling runtime APIs, and then copying the data back to the CPU after the kernel finished executing on the GPU. On the other hand, as the dataset size and model parameter count of deep learning applications continue to increase, GPUs sometimes need to access working sets larger than their own GPU memory; this scenario is known as memory oversubscribing. Before the introduction of UVM, memory oversubscribing relied on the functionality of programmers or deep learning frameworks. By layering neural networks or batching input data, only a subset of input data or model weights resides on the GPU, and frequent H2D and D2H operations are used to use host-side RAM as backup storage for GPU memory. Due to factors such as available GPU memory, model structure, and weight size, finding a subset that a GPU can accommodate is complex and error-prone. Furthermore, data management by programmers and deep learning frameworks is often performed at the granularity of neural network operators. All of these factors highlight the limitations of this memory over-provisioning approach.

[0008] The unified virtual memory mechanism introduced in modern GPUs allows processors in CPU-GPU and multi-GPU systems to share the same virtual address space and access it using the same pointer, greatly benefiting programmers. The unified virtual memory mechanism is managed by the GPU host driver, eliminating the need for programmers to manually intervene in memory allocation within multi-GPU systems or perform H2D / D2H data transfers. More importantly, unified virtual memory provides a transparent data placement mechanism for programmers, managing data at the granularity of GPU pages or cache lines with the cooperation of the GPU driver. The actual resident location of a data page can be on any processor in the system.

[0009] Under a unified virtual memory mechanism, there are typically two address translation modes: centralized address translation and distributed address translation. In centralized address translation, each GPU no longer maintains its own private page table; all address translation requests, after a TLB miss, are directly translated by accessing the host-side IOMMU. In distributed address translation, each GPU still maintains its own private page table, while the host-side IOMMU maintains a centralized page table to record the actual location of each page. Each GPU's private page table primarily stores PTEs residing in local video memory, but by introducing additional page table fields (such as the Aperture field in the NVIDIA Pascal page table), it is also possible to map the physical address of a remote GPU to the local GPU's page table.

[0010] To enable remote data access across GPUs, there are generally two mechanisms:

[0011] I. On-demand Page Migration

[0012] This access method is suitable for access between the GPU and CPU, as well as cross-GPU data access. Specifically, such as... Figure 3 As shown, when GPU1 attempts to access a data page on GPU2, the following steps occur:

[0013] ① The memory access request issued by GPU1 CU first arrives at the TLB, but is missing at each level of TLB. The GMMU page table roamer also fails to find the corresponding PTE in the page table, so it sends the request to the centralized page table of the host IOMMU.

[0014] ② The IOMMU generates a page fault request and notifies the host driver. After receiving the interrupt request, the driver reads the page fault request buffer, performs some preprocessing, and then executes the page fault interrupt handler.

[0015] ③ Refresh GPU2 to ensure data consistency. This step includes TLB knockdown, write-back and invalidation of the corresponding L1 and L2 cache lines, clearing the instruction pipeline and memory access transactions of the current CU, etc.

[0016] ④ GPU2 notifies the driver that the refresh is complete.

[0017] ⑤ The GPU driver sends a command to GPU1, and the Page Migration Controller (PMC) of GPU1 uses DMA to transfer the corresponding page from GPU2 to GPU1.

[0018] ⑥GPU1 notifies the GPU driver that the migration is complete.

[0019] ⑦ The driver notifies the IOMMU. If GPU1 and GPU2 have independent local page tables, the corresponding page table entries of GPU1 and GPU2 also need to be modified, and the modifications are written to the local page table of the GPU via PCIe. (8) The driver notifies the TLB of GPU1 to replay the address translation request that was interrupted before.

[0020] II. Direct Cache Access (DCA)

[0021] This access method is only used for data access between GPUs. It employs a first-touch strategy, meaning that the data page is allocated to that GPU the first time it is accessed. This data page remains permanently on that GPU; subsequent accesses by other GPUs to this page are at the cache line level, and data is accessed via Remote Data Memory Access (RDMA) mechanisms such as PCIe and NVLink. This strategy ensures that a data page exists in the memory of only one GPU at a time. Specifically, as... Figure 4 As shown, when GPU1 accesses data located on GPU2 for the first time through the direct cache access mechanism, the following steps occur:

[0022] ① The memory access request issued by GPU1 CU first reaches the TLB, but all levels of TLB are missing, and the GMMU page table roamer also fails to find the corresponding PTE in the GPU1 page table.

[0023] ② The physical address of the data page on GPU 2 is obtained by accessing the centralized page table on the host side via the IOMMU. In some distributed address translation designs, to reduce consistency overhead, a mapping is usually not established in the GMMU page table. Other designs add the virtual-physical address mapping to the page table of GPU1, and use a specific field of the PTE (such as Aperture used in NVIDIA GPU PTE) to mark that the physical page is actually located in a remote GPU, in order to distinguish it from data pages local to GPU1. Since the remote physical address obtained through the centralized page table on the host side does not belong to the local physical address space, this address translation is not cached by the TLB.

[0024] ③ GPU1 replays the memory access request and retrieves the data from GPU2's L2 cache at the cache line granularity.

[0025] ④ GPU2 sends data to GPU1 via RDMA, and the data is cached in GPU1's L1 cache. However, to ensure data consistency, when the compiler recognizes that the memory access address is in unified virtual memory and may reside on other GPUs, it usually bypasses the cache to access memory. Therefore, the data in the L1 cache line is often not actually used.

[0026] The two GPU memory access modes described above have different applicable scenarios. In MCM-GPU (Multi-Chip Module GPU), direct cache access is commonly used. This is mainly because the on-chip network of the MCM-GPU provides extremely low latency and high bandwidth. However, in multi-GPU architectures, it is often necessary to combine both memory access modes simultaneously.

[0027] However, regardless of the cross-GPU memory access mode, the Non-Uniformity Memory Access (NUMA) problem exists. When using on-demand page migration, page migration incurs significant overhead, including page fault handler activation, TLB knockdown, GPU page table entry modification, and DMA data transfer. For example, a page fault on a 4KB data page can cause a latency of approximately 20-50 microseconds. With direct cache access, each access to remote GPU data needs to be performed at the cache behavior granularity, and each request introduces two inter-GPU interconnect latency operations. Quantitative analysis of common multi-GPU workloads reveals different memory access patterns across different workloads and different stages of the same workload. A data page might only be read by all GPUs during the initialization phase and then not be accessed again; other data pages might be accessed by different GPUs at different stages. Therefore, there is no one-size-fits-all static page placement strategy; a more efficient remote memory access framework is needed that combines the features of on-demand page migration and direct cache access to optimize NUMA memory access in multi-GPU systems with unified virtual memory.

[0028] Since the NVIDIA Volta architecture, a unified virtual memory access counter mechanism has been introduced. This counter is a hardware component on the GPU that records the number of access requests made by the GPU to remote GPUs. Page migration occurs and the corresponding count is reset to zero only when the count reaches a fixed static threshold (according to NVIDIA KMD configuration, the default threshold for a 64KB page is 256); otherwise, direct cache access is used. This mechanism can improve the ping-pong effect when multiple GPUs access the same data page to some extent, but there is still room for optimization in multi-GPU collaborative workloads. Specifically, during on-demand page migration, when the same data page is frequently accessed by n (n≥3) GPUs simultaneously within a certain time window, the access counter threshold may be reached successively within a short period, and the ping-pong problem persists. For example, when a data page is accessed by GPU1, GPU2, and GPU3 simultaneously in a similar pattern, the data actually resides on GPU1, and GPU2 and GPU3 reach the access counter threshold successively, resulting in a page migration from GPU1 to GPU2 to GPU3. When performing direct cache access, remote data is cached in the L1 cache. Since each CU's L1 cache is private, duplicate entries exist across different CUs, and the limited capacity of the L1 cache reduces the cache hit rate for direct cache access. Furthermore, because the TLB does not cache such address mappings, page roaming overhead is unavoidable.

[0029] Existing research has focused on one of these two memory access modes, improving unified virtual memory in multi-GPU systems. Griffin, proposed by T. Baruah et al., employs a series of architectural designs for multi-GPU memory access scenarios. Its Dynamic Page Classifier (DPC) component categorizes page access modes into five types: Mostly Dedicated, Shared, Streaming, Owner-Shifting, and Out-of-Interest, and only performs page migration for Owner-Shifting pages. However, DPC relies on the host-side driver frequently reading the access counters of all GPUs and analyzing the data to determine the access mode of each page, leading to significant PCIe bandwidth consumption. For example, using the hyperparameters used by Griffin, collecting 2200 bytes / GPU access counter readings over 1000 clock cycles would consume 2.2 GB / GPU / s of PCIe bandwidth in a multi-GPU system with a GPU clock frequency of 1 GHz. Analyzing the access counter readings also consumes significant CPU computing resources. On the other hand, Belayneh et al. proposed a dedicated RDMA cache for direct cache access, serving as a shared cache for all CUs. However, this design focuses solely on direct cache access, completely abandoning the page migration mechanism and failing to leverage its advantages in terms of locality. In modern multi-GPU systems where both remote memory access mechanisms exist, a unified design for the RDMA cache and page migration mechanism is necessary. Summary of the Invention

[0030] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a unified virtual memory system in a multi-GPU environment, which effectively improves cross-GPU access performance in a multi-GPU environment based on a globally shared RDMA cache and a reverse access counter.

[0031] This invention provides a unified virtual memory system in a multi-GPU environment. Each GPU is equipped with an RDMA cache, which stores remote data acquired through the RDMA mechanism. All computing units on the GPU share the RDMA cache. Each GPU is also equipped with a reverse access counter, which records the mode of the remote GPU accessing the data page. Data page migration is performed when the mode of the remote GPU is inconsistent with that of the currently accessing GPU and the mode of the remote GPU reaches a preset threshold.

[0032] In one embodiment of the present invention, the RDMA cache detects the percentage of cache lines that have only been accessed once when they are evicted, and determines that the RDMA cache is in streaming access mode when the percentage exceeds a percentage threshold.

[0033] In one embodiment of the present invention, when the RDMA cache is in the streaming access mode, the RDMA cache carries a streaming access flag when sending a data request to the remote GPU. The streaming access flag is used to prevent the remote GPU from migrating the data page to the GPU requesting the data.

[0034] In one embodiment of the present invention, when a write hit occurs, the write to the cache line of the RDMA cache is synchronously updated to the L2 cache of the GPU that owns the data page; when a write miss occurs, the write is directly written to the L2 cache of the GPU that owns the data page, without writing to the RDMA cache of the current GPU; when an explicit synchronization instruction is executed, the L1 cache of the current GPU and the RDMA cache of the remote GPU work together to perform a cache clearing operation.

[0035] In one embodiment of the present invention, the reverse access counter uses the Boyer-Moore voting algorithm to dynamically identify the mode of the remote GPU accessing the data page.

[0036] In one embodiment of the present invention, the reverse access counter includes a reverse access counter table and a reverse access counter controller; the reverse access counter controller is used to listen for access requests to the current GPU L2 cache and determine whether the access request is an RDMA access request from a remote GPU or a local L1 cache access request; the reverse access counter table is used to record the mode of the remote GPU accessing the data page.

[0037] In one embodiment of the present invention, for RDMA access requests from remote GPUs, the physical address of the access request and the source GPU are collected; for local L1 cache access requests, the physical address corresponding to the access request is collected.

[0038] In one embodiment of the present invention, the reverse access counter table includes a page physical address identifier, an access GPU identifier, and an access frequency counter.

[0039] In one embodiment of the present invention, for shared data pages, multiple GPUs adopt a read-only sharing mode or a producer-consumer mode, wherein the reverse access counters of the producer and the consumer will not frequently increase the access frequency count of the shared data page, thereby avoiding triggering invalid page migrations.

[0040] In one embodiment of the present invention, when the remote GPU enables RDMA caching and a cache hit occurs, no RDMA request is generated for the GPU that is the data page holder.

[0041] As described above, the unified virtual memory system, system, medium, and device in a multi-GPU environment described in this invention have the following beneficial effects:

[0042] (1) Based on global shared RDMA and reverse access counter, the cross-GPU access performance in a multi-GPU environment is effectively improved;

[0043] (2) It supports both on-demand page migration and direct cache access mechanisms, and can flexibly decide whether to perform page migration based on the current workload.

[0044] (3) Reduced the ping-pong phenomenon in on-demand page migration when local GPU data access and remote GPU access coexist;

[0045] (4) No longer occupy L1 data cache to avoid contention with local cache; reduce PCIe bandwidth usage that may be involved in page management mechanism. Attached Figure Description

[0046] Figure 1 The diagram shown is a schematic representation of a multi-GPU architecture based on a PCIe connection in one embodiment;

[0047] Figure 2 The diagram shown is a schematic representation of a multi-GPU architecture based on an NVLINK connection in one embodiment;

[0048] Figure 3 The diagram shown illustrates on-demand page migration using centralized address translation in one embodiment.

[0049] Figure 4 The diagram shown illustrates direct cache access using centralized address translation in one embodiment.

[0050] Figure 5 This diagram illustrates the location of the RDMA cache in the cache hierarchy in one embodiment of the present invention.

[0051] Figure 6 The diagram shown is a schematic representation of a write operation in an RDMA cache write call in one embodiment of the present invention;

[0052] Figure 7 The diagram shown is a schematic representation of a non-write allocation in an RDMA cache write miss according to an embodiment of the present invention. Detailed Implementation

[0053] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0054] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0055] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0056] In one embodiment, in the unified virtual memory system of the multi-GPU environment of the present invention, each GPU is equipped with an RDMA cache. The RDMA cache is used to store remote data acquired through the RDMA mechanism, and all computing units on the GPU share the RDMA cache. In the direct cache access scenario, the data of the remote GPU is no longer cached in the L1 cache private to each CU, but is cached in a dedicated RDMA cache shared by all CUs on a single GPU. This design can reduce L1 cache contention between local memory access and remote memory access, reduce duplicate remote data in each CU's L1 cache, and thus increase the hit rate of the RDMA cache. Simultaneously, considering the unified multi-GPU model, a caching strategy is designed for the RDMA cache to ensure data consistency across multiple GPUs. The RDMA cache also has the function of detecting streaming access patterns. When a streaming access pattern is detected, it prevents the remote GPU holding the data page from migrating the page to the GPU requesting the data.

[0057] Each GPU is equipped with a reverse access counter, which records the mode of remote GPUs accessing data pages. Data page migration is initiated when the mode of the remote GPU differs from that of the currently accessing GPU and reaches a preset threshold. The GPU's access counter no longer records requests from remote GPUs. Instead, by introducing the reverse access counter, each GPU records the mode of GPUIDs accessing a specific data page within a given time window, as well as the frequency of that GPUID. Access from the local GPU is also included in the statistics, mitigating the ping-pong phenomenon when multiple GPUs access the same data. The reverse access counter controller updates the frequency field and / or mode field based on the source GPUID of the memory access request, using the Boyer-Moore majority voting algorithm, and issues a page migration request to the driver only when the mode is not the local GPUID. This event-driven approach reduces the PCIe bandwidth consumption of the driver in collecting access counter values.

[0058] Specifically, in a typical multi-GPU system, when the local TLB is missing, if a page table roaming via GMMU or IOMMU reveals that the requested data resides on a remote GPU, the direct cache access mechanism retrieves the data from the L2 cache of the data page owner at the cache line granularity via RDMA and caches the data in the local L1 cache. However, the L1 cache has limited capacity and must simultaneously serve memory access requests from local threads and remote data access, leading to severe resource contention. Furthermore, different CUs may contain identical copies of remote data in their L1 caches, wasting cache space and increasing the overhead of maintaining cache consistency. More critically, to ensure data consistency, the compiler may compile memory access instructions with cache bypass options when it detects that the memory access address belongs to a uniformly allocated virtual memory space, rendering the cache invalid.

[0059] Therefore, this invention moves the caching of remote data to a lower level, serving all CUs to increase hit rate. Simultaneously, a corresponding caching strategy needs to be designed for the RDMA cache to ensure cache consistency. To this end, such as... Figure 5 As shown, this invention constructs a globally shared RDMA cache on top of all CUs, treating it as an independent level in the GPU memory hierarchy. This RDMA cache is dedicated to storing remote data acquired via the RDMA mechanism. All CUs access this cache via the on-chip network, and its physical location is close to the memory controller.

[0060] In one embodiment, the RDMA cache has a streaming access mode detection function. Streaming access refers to data fetched by the GPU from a remote GPU being used only once. To this end, an access count is added to the RDMA cache to detect the proportion of cache lines with one-touch-eviction (i.e., accessed only once when evicted) out of all cache line allocations. When this proportion exceeds a threshold, the RDMA cache is determined to be in streaming access mode. When the RDMA cache is in streaming access mode, it carries a streaming access flag when sending data requests to the remote GPU. This streaming access flag is used to prevent the remote GPU from migrating data pages to the GPU requesting the data.

[0061] The RDMA cache consistency model is adapted to the existing unified multi-GPU model. In the unified multi-GPU model, different Cooperative Thread Arrays (CTAs) of the same kernel are distributed to multiple GPUs for execution, and data access uses a unified address space, thus enabling existing GPU programs to run on multiple GPUs without additional adaptation. In this model, to maintain compatibility with the memory consistency model of single-GPU programs, each GPU's RDMA cache should be equivalent to the L1 cache in the single-GPU model (analogous to the case where each CTA is distributed to a different CU). The L1 cache adopts a write-through, write-no-allocate strategy, meaning that when a cache write hit occurs, the cache at that level and the next level are updated, and when a write miss occurs, the data is directly written to the next level cache. To maintain consistency with the single-GPU cache consistency model, a similar strategy is also adopted between the RDMA cache and the data page owner L2 cache. Specifically, the RDMA cache consistency model in this invention is as follows:

[0062] 1) such as Figure 6 As shown, when a write hit occurs, the write to the cache line of the RDMA cache is synchronously updated to the L2 cache of the GPU, which is the owner of the data page;

[0063] 2) such as Figure 7 As shown, when a write is missing, the data is directly written to the L2 cache of the GPU that owns the data page, without needing to be written to the RDMA cache of the current GPU.

[0064] 3) When an explicit synchronization instruction is executed, the L1 cache of the current GPU and the RDMA cache of the remote GPU work together to perform a cache clearing operation.

[0065] In a multi-GPU environment, when a GPU needs to access data located on a remote GPU, data is transferred at the page level using an on-demand page migration mechanism. This process involves a series of TLB knockdowns and page table entry operations, resulting in significant overhead. Furthermore, when multiple GPUs access the same physical page simultaneously, a noticeable ping-pong effect occurs. Assuming a data page initially resides on GPU1, GPU2 and GPU3 frequently access this page concurrently. When GPU2's access counter reaches its threshold first, the system migrates the page from GPU1 to GPU2. Subsequently, GPU3's access counter also reaches its threshold, requiring the page to be migrated from GPU2 to GPU3. This frequent migration operation not only consumes significant PCIe bandwidth but also incurs overhead from TLB knockdowns and page table updates. Therefore, traditional access counter designs are not well-suited for multi-GPU collaborative workloads. Therefore, this invention proposes a reverse access counter to replace the original access counter.

[0066] In one embodiment, the reverse access counter of the present invention has the following characteristics:

[0067] 1) The reverse access counter tracks pages at the granularity of GPU pages (typically 4KB).

[0068] 2) The reverse access counter tracks which remote GPUs receive access requests for each data page, and uses the Boyer-Moore majority voting algorithm to track the mode of the GPU IDs accessing a page.

[0069] 3) If the mode of the GPU ID accessed for a certain page is not the current GPU ID, then the ownership of the page is considered to have changed. If the frequency counter value continues to increase until it reaches the threshold, the GPU driver is notified to perform page migration.

[0070] 4) Configurable frequency counter threshold for triggering page migration.

[0071] 5) Reverse access counter entries only include pages that have been requested from remote locations. This excludes pages that are only accessed locally, saving space in the reverse access counter entries.

[0072] The reverse access counter uses the Boyer-Moore voting algorithm to dynamically identify the mode of the remote GPUs accessing the data page. This algorithm can efficiently find elements that appear more than half the time in streaming data and can be used to determine the dominant GPU in the access pattern. Specifically, if we let the sequence of GPU IDs that access a data page be A, and define the mode of A as the value of the element that appears more than half the length of A, then we can record the mode in A with constant space complexity using a mode field and a frequency field, and update the counter entries and retrieve the mode value with constant time complexity. Specifically, the Boyer-Moore voting algorithm is as follows:

[0073] Initialization: When accessing address addr, if there is no corresponding entry in the reverse access counter and the access request does not come from the local GPU, initialize the majority GPU ID field m to the current visitor GPU ID, and initialize the frequency field i = 1.

[0074] Access Update: For each access to address `addr`, denote the requested GPU ID as `id`. If the request originates from a local CU, `id` represents the local GPU ID; otherwise, `id` represents the ID of the remote requesting GPU. If `i` = 0, then `m` = `id`, and `i` = 1. Otherwise, if `m` = `id`, then `i` = `i+1`. If `i` is already saturated, it stops incrementing. Otherwise, `i` = `i-1`.

[0075] Migration decision: When the frequency counter exceeds a preset threshold and most GPU IDs are not local GPU IDs, a page migration request is sent to the driver.

[0076] Table 1 shows the difference in behavior between the traditional access counter and the reverse access counter of the present invention under different page attributes.

[0077] Table 1. Behavior of traditional access counters and reverse access counters of the present invention under different page attributes.

[0078]

[0079]

[0080] In one embodiment of the present invention, the reverse access counter includes a Reverse Access Counter Table (RACT) and a Reverse Access Counter Controller (RACC). The RACT monitors access requests to the current GPU L2 cache and determines whether the access request is an RDMA access request from a remote GPU or a local L1 cache access request. The RACT records the mode of the remote GPU accessing the data page. When page table roaming accesses the page table stored in local video memory, it may access the L2 cache. However, since RACT does not create entries for addresses that are only accessed locally, the page table roamer's memory access requests will not affect or interfere with the reverse access counter. Because the reverse access counter is not on the critical path of the GPU memory access structure, its value update process does not cause additional overhead to the memory access process.

[0081] RACC only monitors L2 cache accesses, therefore it does not count L1 cache hits. Similarly, if the remote GPU enables RDMA caching and a cache hit occurs, no RDMA request will be generated to the data page holder. For shared pages, this includes two common page sharing modes: read-only shared and producer-consumer.

[0082] Let GPU 1 be the data page holder. In read-only shared mode, under initial conditions, if GPU 1 and GPU 2 access the data page simultaneously, GPU 1's L1 and L2 data caches and GPU 2's RDMA cache will be filled at the cache line level. Since they are accessing the same data page, they cancel each other out, keeping the counter frequency at a low level and not triggering page migration. If the access frequency of GPU 2 increases and becomes the mode, it indicates that GPU 2 has a high RDMA cache miss rate, possibly due to severe RDMA cache conflicts (RDMA cache capacity miss, conflict miss) or GPU 2 needs to access more data in the page (RDMA cache forced miss). In either case, placing the data page on GPU 2 is the better choice.

[0083] In the producer-consumer model, assuming GPU 1 is the producer and GPU 2 is the consumer, GPU 1's writes need to write through the L1 cache to update the L2 cache. Due to the memory consistency model, when GPU 2 reads this data, GPU 2's RDMA cache is bypassed, and data is requested from GPU 1's L2 cache via RDMA. During this process, GPU 1's L2 cache is accessed, and the accesses from GPU 1 and GPU 2 cancel each other out; therefore, GPU 1's reverse access counter does not increment.

[0084] In one embodiment of the present invention, for RDMA access requests from a remote GPU, the physical address of the access request and the source GPU are collected; for local L1 cache access requests, the physical address corresponding to the access request is collected. Assume GPU 1 is the consumer and GPU 2 is the producer. GPU 2's writes require writing through the RDMA cache and updating GPU 1's L2 cache. GPU 1's reads also need to bypass the local L1 cache and read from the L2 cache. These two actions cancel each other out and do not increase the frequency count. Therefore, the reverse access counter, in conjunction with the RDMA cache, can adapt to two common scenarios of shared pages and dynamically determine the optimal placement location in read-only shared scenarios, placing data pages on the GPU with higher memory access requirements.

[0085] As shown in Table 2, the reverse access counter table includes page physical address identifier, access GPU identifier, and access frequency counter.

[0086]

[0087] The overhead introduced by a reverse access counter on a GPU is: n = N × (W id +W cnt +W tag Where N is the number of entries in RACT, W id W represents the width of the GPU ID field. cnt This represents the width of the frequency field. For example, in a typical multi-GPU system, the number of GPUs per CPU node is usually less than 8, i.e., W. id Only 3 bits are needed. Let N = 1024, W cnt =4. With a fixed page size of 4KB and using a 49-bit physical address, the required tag width is 49 - 12 - logN = 25. Therefore, the storage overhead introduced by the reverse access counter is 1024 × (3 + 4 + 25) = 32KB. If using large pages with 2MB granularity, then W... tag It can be reduced to 16, with a total storage overhead of 1024 × (3 + 4 + 16) = 23KB.

[0088] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A unified virtual memory system in a multi-GPU environment, characterized in that, Each GPU is equipped with an RDMA cache, which is used to store remote data obtained through the RDMA mechanism. All computing units on the GPU share the RDMA cache. Each GPU is equipped with a reverse access counter, which is used to record the mode of the remote GPU accessing the data page, so as to perform data page migration when the mode of the remote GPU is inconsistent with the current accessing GPU and the mode of the remote GPU reaches a preset threshold.

2. The unified virtual memory system in a multi-GPU environment according to claim 1, characterized in that, The RDMA cache detects the percentage of cache lines that were accessed only once when they were evicted. When the percentage exceeds a threshold, the RDMA cache is determined to be in streaming access mode.

3. The unified virtual memory system in a multi-GPU environment according to claim 2, characterized in that, When the RDMA cache is in the streaming access mode, the RDMA cache carries a streaming access flag when sending a data request to the remote GPU. The streaming access flag is used to prevent the remote GPU from migrating data pages to the GPU requesting the data.

4. The unified virtual memory system in a multi-GPU environment according to claim 1, characterized in that, When a write hit occurs, the write to the cache line of the RDMA cache is synchronously updated to the L2 cache of the GPU, which is the owner of the data page; When a write is missing, the data is written directly to the L2 cache of the GPU that owns the data page, without needing to be written to the RDMA cache of the current GPU. When an explicit synchronization instruction is executed, the L1 cache of the current GPU and the RDMA cache of the remote GPU work together to perform a cache clearing operation.

5. The unified virtual memory system in a multi-GPU environment according to claim 1, characterized in that, The reverse access counter uses the Boyer-Moore voting algorithm to dynamically identify the mode of the remote GPU accessing the data page.

6. The unified virtual memory system in a multi-GPU environment according to claim 1, characterized in that, The reverse access counter includes a reverse access counter table and a reverse access counter controller; the reverse access counter controller is used to listen for access requests to the current GPU L2 cache and determine whether the access request is an RDMA access request from a remote GPU or a local L1 cache access request. The reverse access counter table is used to record the mode of the remote GPU accessing the data page.

7. The unified virtual memory system in a multi-GPU environment according to claim 6, characterized in that, For RDMA access requests from remote GPUs, collect the physical address of the access request and the source GPU; for local L1 cache access requests, collect the physical address corresponding to the access request.

8. The unified virtual memory system in a multi-GPU environment according to claim 6, characterized in that, The reverse access counter table includes a page physical address identifier, an access GPU identifier, and an access frequency counter.

9. The unified virtual memory system in a multi-GPU environment according to claim 1, characterized in that, For shared data pages, multiple GPUs adopt a read-only shared mode or a producer-consumer mode, wherein the reverse access counters of the producer and the consumer will not increase the access frequency count due to the shared data page.

10. The unified virtual memory system in a multi-GPU environment according to claim 1, characterized in that, When the remote GPU enables RDMA caching and a cache hit occurs, no RDMA request will be generated for the GPU that is the data page holder.

Images