A heterogeneous GPU system and data transmission method

Abstract

This application belongs to the field of GPU technology, specifically to a heterogeneous GPU system and data transmission method, including an access interface module and a data transmission module. The access interface module encapsulates the native memory management interfaces of different GPUs and provides a consistent memory access interface. A GPU memory transmission interface is designed based on the unilateral semantics of libibverbs. The data transmission module uses GDR and RDMA transmission primitives to achieve direct data transmission to remote heterogeneous GPU memory. A dual data buffer is set up, with the sender and receiver each maintaining two data buffers residing in the GPU memory, and these buffers are allocated through a unified GPU memory access interface. The advantage of this application is that it designs a unified memory operation interface and communication interface, supporting memory interoperability between heterogeneous GPUs.

Description

Technical Field

[0001] The present application belongs to the field of GPU technology, and specifically relates to a heterogeneous GPU system and a data transmission method. Background Art

[0002] Modern GPUs vary significantly in memory capacity, computing power, and architectural design. For example, NVIDIA's A100 and H100 GPUs offer 40GB and 80GB of video memory, respectively, while AMD's MI250X GPU offers up to 128GB of video memory. In practical applications, especially in heterogeneous computing clusters, GPUs from different vendors need to collaborate to share data and complete model training tasks. However, existing communication libraries (such as NCCL, RCCL, and UCC) only support data transfer between homogeneous GPUs and cannot enable direct communication between GPU memory of different vendors.

[0003] Existing communication libraries are primarily designed for homogeneous GPU environments and lack support for direct memory transfers with heterogeneous GPUs. For example, NCCL (NVIDIA Collective Communications Library): Designed specifically for NVIDIA GPUs, it relies on the CUDA ecosystem and cannot support memory communication with GPUs from other vendors, such as AMD and Intel. RCCL (ROCm Collective Communications Library): Based on the AMD ROCm ecosystem, it supports collective communication with AMD GPUs, but cannot achieve direct memory transfers with NVIDIA or Intel GPUs. UCX (Unified Communication X): Although it supports cross-platform communication, it is still limited by vendor-specific drivers and interfaces for direct GPU memory transfers. MPI-aware GDR (MessagePassing Interface with GPU Direct RDMA): Combines RDMA technology to achieve GPU memory communication, but only supports GPUs from the same vendor and cannot achieve direct memory transfers across vendors. Summary of the Invention

[0004] Based on the above problems, this application designs a unified memory operation interface and communication interface to support memory interoperability between heterogeneous GPUs. The technical solution is:

[0005] A heterogeneous GPU system includes an access interface module and a data transmission module;

[0006] The access interface module: encapsulates the native memory management interface of different GPUs and provides a consistent memory access interface; designs the GPU memory transmission interface based on the unilateral semantics of libibverbs;

[0007] The data transmission module uses GDR and RDMA transmission primitives to realize direct data transmission to remote heterogeneous GPU memory; sets up dual data buffers, and the sender and receiver each maintain two data buffers residing in the GPU memory, which are allocated through a unified GPU memory access interface.

[0008] Preferably, the data transmission module sets two buffers. During the data transmission process, one buffer is used for GPU memory copying, and the other buffer is used to transmit data through the network. In the next cycle, the roles of the two buffers are exchanged.

[0009] Preferably, during program execution, the access interface module sets a memory management base class Memory for managing video memory of different GPU architectures;

[0010] HostMemory is used to manage host memory; CudaMemory is used to manage NVIDIA GPU video memory; RocmMemory is used to manage AMD GPU video memory; NeuwareMemory is used to manage Cambricon MLU video memory.

[0011] Preferably, in order to enable direct access of the host channel adapter HCA to the GPU memory, the peer agent API provided by Mellanox NICs and their OFED drivers is utilized to enable the host channel adapter HCA to access the peer memory buffer, thereby establishing a DMA data path from the Mellanox NIC to the GPU memory;

[0012] Host-to-device transfers are implemented through CUDA's cudaMemcpyHostToDevice, ROCm's hipMemcpyHostToDevice, or the corresponding MLU APIs; device-to-host transfers are implemented through CUDA's cudaMemcpyDeviceToHost, ROCm's hipMemcpyDeviceToHost, or the corresponding MLU APIs; device-to-device transfers are implemented through direct cross-GPU video memory transfers via the PCIe bus.

[0013] Preferably, in order to achieve message synchronization between the sender and the receiver, a status buffer is designed, which includes writability information and the size of the data to be sent. The buffer controls the data transmission process; the status buffer is maintained on both sides of the host memory, and the message is synchronized through a unilateral write operation; when the sender completes writing the data to the receiver, it updates the message in the sender's status buffer and sends it to the receiver. After the receiver successfully detects the message, it starts to read the sent data, thereby ensuring synchronization and efficient data processing.

[0014] A heterogeneous GPU data transmission method includes the following steps:

[0015] S1. Device initialization and configuration:

[0016] Install a module that supports GPU Direct RDMA on the target host. Users configure the target device by calling the constructor of the memory management base class Memory and passing in the device type and device number.

[0017] S2. Creation of video memory management instance:

[0018] After the device configuration is completed, Memory will dynamically create the corresponding video memory management class instance according to the device type specified by the user through the createMemoryClass method;

[0019] S3. Unified call of video memory operation:

[0020] Users can directly complete video memory operations through the unified interface provided by Memory;

[0021] S4. Use pipeline double buffer strategy for data transmission.

[0022] Preferably, during initialization in step S1, the GPU Direct RDMA module and the RDMA environment are checked to see if they are correctly configured, and necessary resources are initialized;

[0023] In step S3, after the video memory is allocated, it will be registered with the RDMA subsystem through the ibv_reg_mr interface so that RDMA can directly access these memory areas.

[0024] Preferably, the pipeline double buffer data transmission strategy in step S4 is as follows:

[0025] S41. The sender copies the data block to a data buffer and sends a message to the receiver to notify the total size of the data to be sent;

[0026] S42. The sender publishes two write events on the data buffer. The first event is used to write the actual data to the remote GPU memory, and the second event is used to synchronize a readability message to the receiver. These two events are published sequentially on the same QP to ensure in-order execution of RDMA. At the same time, the sender starts copying the data to the second data buffer.

[0027] S43. Once the receiver detects the readability notification, it immediately copies the data from its data buffer to the application and sends a writability notification to the sender. At the same time, the second data buffer begins receiving network data.

[0028] S44. Receive the next readability notification immediately after the current iteration is completed.

[0029] Preferably, regardless of the latency between the data copy and the RDMA write, the next RDMA write to the same memory region will occur after the copy is complete.

[0030] Preferably, when the GPU memory copy latency is higher than the RDMA write latency when the message size is below a certain threshold, the transmission latency is minimized by configuring the size of the data buffer to be close to the threshold.

[0031] Compared with the prior art, this application has the following beneficial effects:

[0032] This application implements the direct memory transmission capability of heterogeneous video memory, and supports direct memory transmission between graphics cards such as Cambrian, Hygon, and NVIDIA.

[0033] The high-performance transmission solution designed in this application achieves higher bandwidth and lower latency compared to unoptimized solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure is the experimental result.

[0035] Figure 2 Transmission flow chart. DETAILED DESCRIPTION

[0036] The technical solution of the present application is described in detail below through specific embodiments and drawings. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application, and the specific technical features can be combined with each other.

[0037] A heterogeneous GPU system includes an access interface module and a data transmission module. The access interface module encapsulates the native memory management interfaces of different GPUs and provides a consistent memory access interface. A GPU memory transmission interface is designed based on the unilateral semantics of libibverbs. The data transmission module uses GDR and RDMA transmission primitives to implement direct data transmission to remote heterogeneous GPU memory. A dual data buffer is set up, with the sender and receiver each maintaining two data buffers residing in the GPU memory. These buffers are allocated through a unified GPU memory access interface.

[0038] This interface encapsulates the native memory management interface of each GPU and provides a consistent memory access interface, such as memory allocation, copying, and release. This allows upper-level programs to access the memory of different GPUs in a unified manner, without having to worry about differences in the underlying hardware. During program execution, UHM automatically adapts to the specific GPU type based on the host device type, ensuring seamless compatibility in heterogeneous environments. This design allows upper-level applications to be migrated between different GPUs without modification, simplifying development and maintenance.

[0039] The unified operation interface for heterogeneous GPU memory is as follows:

[0040] Core data structure definition and implementation:

[0041] This paper defines a unified memory management interface for heterogeneous GPUs, shielding the memory management details of different GPU architectures and providing users with an efficient and easy-to-use memory management solution. The core data structure includes the memory management base class MemoryBase, which defines the basic memory management interface, including memory allocation, release, and data transfer functions. Specific subclasses such as CudaMemory and RocmMemory implement these interfaces based on different GPU architectures.

[0042] The unified interface class Memory provides a user-friendly interface for managing video memory across different GPU architectures. By setting the device type, users can automatically create corresponding video memory management class instances. For different GPU architectures, this invention implements the following video memory management classes:

[0043] HostMemory is used to manage host memory; CudaMemory is used to manage NVIDIA GPU video memory; RocmMemory is used to manage AMD GPU video memory; NeuwareMemory is used to manage Cambricon MLU video memory.

[0044] Regarding interface implementation and functionality expansion, the memory allocation and release interfaces use APIs specific to different GPU architectures. For NVIDIA GPUs, memory allocation and release are implemented using the CUDA API (e.g., cudaMalloc and cudaFree); for AMD GPUs, memory allocation and release are implemented using the ROCm API (e.g., hipMalloc and hipFree); and for Cambricon MLU, memory allocation and release are implemented using the MLU API (e.g., mluMalloc and mluFree).

[0045] In terms of data transmission interface, host-to-device transmission is implemented through CUDA's cudaMemcpyHostToDevice, ROCm's hipMemcpyHostToDevice, or the corresponding MLU API; device-to-host transmission is implemented through CUDA's cudaMemcpyDeviceToHost, ROCm's hipMemcpyDeviceToHost, or the corresponding MLU API; device-to-device transmission is achieved through the PCIe bus to directly transmit the video memory across the GPU, avoiding the additional overhead of data passing through the host memory.

[0046] In terms of initialization and resource management, each memory management class (such as CudaMemory) sets the current device and initializes the corresponding GPU context during initialization. Through the init method of the Memory class, users can automatically complete the creation and initialization of the memory management instance.

[0047] 1) Users need to install a module that supports GPU Direct RDMA on the target host, such as NVIDIA's GDR module or AMD's peermem module. These modules are typically integrated with the GPU driver and OFED driver to provide the necessary support for RDMA communication operations with video memory.

[0048] 2) The user creates a communication endpoint instance using the memory communication endpoint class. During instantiation, the communication endpoint class automatically detects the GPU Direct RDMA environment and prepares for subsequent memory operations and RDMA communication.

[0049] 3) The user allocates video memory using the memory operation interface provided by the communication endpoint class. Once allocated, the communication endpoint class calls the ibv_reg_mr interface to register the video memory with the RDMA subsystem. This process ensures direct RDMA access to the video memory, enabling efficient data transfer between video memories.

[0050] 4) Users can use the RDMA primitive interfaces (such as rdma_write and rdma_read) encapsulated by the communication endpoint class to directly read and write to the peer's video memory. These operations, based on RDMA's unilateral semantics, can significantly reduce data transmission latency and system overhead.

[0051] Heterogeneous video memory operation process:

[0052] 1. Device initialization and configuration:

[0053] The user completes the configuration of the target device by calling the Memory class constructor and passing in the device type (such as MemoryType::NVIDIA_GPU or MemoryType::AMD_GPU) and the device number. This process ensures that the graphics memory management module can accurately identify the target device type and allocate appropriate resources to it.

[0054] 2. Create a video memory management instance:

[0055] After device configuration is complete, the Memory class dynamically creates a corresponding memory management class instance based on the user-specified device type using the createMemoryClass method. For example, for NVIDIA GPUs, a CudaMemory instance is created; for AMD GPUs, a RocmMemory instance is created. This mechanism enables dynamic adaptation of memory management to different GPU architectures.

[0056] 3. Unified call of video memory operation:

[0057] Users can directly perform video memory operations through the unified interfaces provided by the Memory class (such as allocate_buffer, free_buffer, copy_host_to_device, etc.) without having to worry about the underlying implementation details. This design greatly simplifies user experience while ensuring the efficiency and consistency of video memory operations.

[0058] 4. Heterogeneous memory point-to-point direct transmission solution:

[0059] The GPU Direct RDMA (GDR) module is a technology used to enable direct memory transfers between GPUs. This module enables the Host Channel Adapter (HCA) to directly access GPU memory without going through the traditional CPU memory copy process. This not only reduces data transmission latency but also improves bandwidth utilization, making it particularly suitable for inter-GPU communication in high-performance computing and distributed systems. GDR modules are typically provided by GPU vendors, such as NVIDIA's GDR module and AMD's peermem module. These modules provide underlying support for RDMA communication through integration with the OFED driver.

[0060] Leveraging the GPU Direct RDMA module and the RDMA libverbs library, this paper designs a video memory communication endpoint class that encapsulates primitive operations for video memory registration, queue pair creation, and data transfer. This class primarily includes the following core functions:

[0061] Initialization and resource management: During initialization, this class checks whether the GPU Direct RDMA module and RDMA environment are configured correctly, and initializes necessary resources such as protection domains (PDs) and queue pairs (QPs).

[0062] Memory operation interface: Through this interface, users can easily allocate and release memory. After the memory is allocated, it will be registered with the RDMA subsystem through the ibv_reg_mr interface so that RDMA can directly access these memory areas.

[0063] RDMA data transfer: This class encapsulates RDMA one-sided operations (such as rdma_write and rdma_read), allowing users to directly read and write to the peer's video memory. These operations are based on RDMA's native semantics, ensuring efficient and reliable data transfer.

[0064] Through the memory communication endpoint class, users can seamlessly integrate memory operations with RDMA communication, simplifying the implementation process of direct memory transfer between GPUs.

[0065] Pipeline double buffer data transfer strategy:

[0066] In a heterogeneous GPU environment, there is a significant performance difference between direct memory reads and writes and memory copies. Memory reads and writes are usually implemented through RDMA, which has the characteristics of low latency and high bandwidth; while memory copies may involve more system overhead, especially when data needs to be transferred between GPUs. In addition, registering large blocks of memory for buffers may lead to a waste of memory resources, especially when memory resources are limited. Therefore, it is particularly important to design a transmission strategy that can efficiently utilize memory resources and balance memory copy latency and transmission latency. Figure 2The pipeline double buffer data transmission strategy designed by the present invention is demonstrated:

[0067] Phase 1: The sender copies the data block to a data buffer and simultaneously sends a message to the receiver informing it of the total size of the data to be sent. The sender and receiver can use this information to calculate the number of transmission iterations required.

[0068] Phase 2: The sender publishes two write events on the data buffer. The first event actually writes the data to the remote GPU memory, and the second event synchronizes a readability message to the receiver. These two events are published sequentially on the same QP to ensure in-order RDMA execution. Simultaneously, the sender begins copying the data to the second data buffer. In this way, RDMA data transfer and message synchronization overlap with GPU memory copying.

[0069] Phase 3: Once the receiver detects a readability notification, it immediately copies the data from its data buffer to application memory and sends a writability notification to the sender. Simultaneously, the second data buffer begins receiving network data. Regardless of the latency between the data copy and the RDMA write, the next RDMA write to the same memory region occurs after the copy is complete. This ensures that data is not overwritten, thus preserving data integrity.

[0070] Receiving the next readability notification (for the second buffer) immediately after the current iteration completes maximizes pipeline efficiency. GPU memory copy latency exceeds RDMA write latency when message sizes are below a certain threshold. Therefore, we can achieve this by configuring the data buffer size close to this threshold. This pipeline design allows us to hide RDMA data transfer latency within GPU memory copy latency, significantly reducing network transmission latency for large data blocks.

[0071] This solution designs two key components: a state buffer (for message synchronization) and a double data buffer (for pipeline data transmission), which reside in the host memory and GPU memory respectively.

[0072] The status buffer is used for message synchronization: To achieve message synchronization between the sender and the receiver, this application designs a status buffer that includes writability information and the size of the data to be sent. This buffer controls the data transmission process. Since the GPU memory cannot be directly accessed by the CPU program, it is impossible to embed the transmission control message directly into the data during communication like the polling design of FaRM. Instead, this solution maintains the status buffer on both sides of the host memory and synchronizes the message through a unilateral write operation. When the sender finishes writing the data to the receiver, it updates the message in the sender's status buffer and sends it to the receiver. After the receiver successfully detects the message, it starts to read the sent data, thus ensuring synchronization and efficient data processing.

[0073] Dual data buffers for pipelined data transfer: The sender and receiver each maintain two data buffers residing in GPU memory to store the actual data. These buffers are allocated through a unified GPU memory access interface. During data transfer, one buffer is used for GPU memory copying while the other is used to transfer data over the network. In the next cycle, the roles of the two buffers are reversed. This dual design overlaps RDMA data transfers with GPU memory copies, thus achieving pipelined transfers.

[0074] To enable direct HCA (Host Channel Adapter) access to GPU memory, we leveraged the peergent API provided by Mellanox NICs and their OFED (OpenFabrics Enterprise Distribution) driver. This API enables the HCA to access peer memory buffers, thereby establishing a DMA data path from the Mellanox NIC to the GPU memory. GPU drivers support this functionality by implementing a peer memory client (static struct peer_memory_client_ex). These modules are commonly referred to as GPU Direct RDMA (GDR). GPU vendors such as Nvidia and AMD provide kernel modules to support this functionality. For example, the Nvidia driver includes a GDR module, while the AMD driver embeds the peermem module. However, existing direct GPU memory transfer solutions, such as xCCL, UCX, and MPI-aware GDR, only support data transfers between homogeneous GPUs. To overcome this limitation, we extended the libibverbs library for GDR and RDMA and integrated them into a unified memory access interface. By leveraging libibverbs interfaces (such as ibv_reg_mr) to register heterogeneous GPU memory, direct data transfer to remote heterogeneous GPU memory is achieved using GDR and RDMA transfer primitives.

[0075] This application designed an experiment to verify the significant improvement in data transfer efficiency achieved by the unified heterogeneous GPU memory access method described in this invention. This experiment compared four different data transfer schemes, including: Ideal (using only RDMA writes without additional replication latency); G2H2G (host memory transfer scheme): using host memory as a transfer to achieve inter-GPU memory transfer; NoPipe (serial copy / transfer scheme): a non-pipelined version of the UHM method described in this invention; and UHM (the method described in this invention).

[0076] The experimental results are as follows Figure 1. Under small data volumes (less than 8192 bytes), the transmission latency of UHM is comparable to that of the NoPipe solution and significantly lower than that of the G2H2G solution. The average latency of the UHM and NoPipe solutions is reduced by approximately 50%-70% compared to the G2H2G solution. Under large data volumes (greater than 32768 bytes), UHM significantly reduces the latency of the G2H2G and NoPipe solutions. Experimental data show that UHM reduces the average latency of the G2H2G solution by 38.1%-87.2% and the average latency of the NoPipe solution by 23.5%-76.7%. The performance of UHM is close to the theoretical limit of the Ideal solution. By adopting a pipeline design based on dual data buffers, UHM significantly reduces the latency introduced by memory copying. Especially in large data transmission scenarios, the performance optimization effect of UHM is particularly significant, fully demonstrating its superiority in improving data transmission efficiency. These results show that the unified heterogeneous GPU memory access method described in the present invention can effectively reduce data transmission latency and significantly improve system performance, especially in complex heterogeneous environments.

[0077] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A heterogeneous GPU system, characterized in that: It includes an access interface module, a data transmission module, a status buffer and a double data buffer, wherein the status buffer and the double data buffer reside in the host memory and the GPU memory respectively; The access interface module encapsulates native memory management interfaces of different GPUs and provides a consistent memory access interface; Design GPU memory transfer interface based on libibverbs unilateral semantics; The data transmission module uses GDR and RDMA transmission primitives to realize direct data transmission to remote heterogeneous GPU memory; sets up dual data buffers, and the sender and receiver each maintain two data buffers residing in the GPU memory, and these buffers are allocated through a unified GPU memory access interface; The status buffer is used for message synchronization and controls the data transmission process: the status buffer is maintained on both sides of the host memory and synchronizes messages through unilateral write operations. When the sender completes writing data to the receiver, it updates the message in the sender's status buffer and sends it to the receiver. After the receiver successfully detects the message, it starts reading the sent data, thus ensuring synchronous and efficient data processing. Dual data buffers are used for pipelined data transfer: the sender and receiver each maintain two data buffers residing in GPU memory to store the actual data; these buffers are allocated through a unified GPU memory access interface; during data transfer, one buffer is used for GPU memory copying while the other buffer is used to transmit data over the network; in the next cycle, the roles of the two buffers are reversed; Pipeline double buffer data transfer strategy: Phase 1: The sender copies the data block to a data buffer and sends a message to the receiver to inform the total size of the data to be sent. The sender and receiver can calculate the required number of transmission iterations based on this information. Phase 2: The sender publishes two write events on the data buffer; the first event is used to write the actual data to the remote GPU memory, and the second event is used to synchronize the readability message to the receiver. These two events are published sequentially on the same QP to ensure the ordered execution of RDMA. At the same time, the sender starts copying the data to the second data buffer; Phase 3: Once the receiver detects a readability notification, it immediately copies the data from its data buffer to the application memory and sends a writability notification to the sender. At the same time, the second data buffer starts receiving network data. Regardless of the latency between the data copy and the RDMA write, the next RDMA write to the same memory region will occur after the copy is completed. The next readability notification is received immediately after the current iteration is completed. When the GPU memory copy latency is higher than the RDMA write latency when the message size is below a certain threshold, the above goal can be achieved by configuring the size of the data buffer to be close to the threshold. Through this pipeline design, the RDMA data transmission latency is hidden in the GPU memory copy latency, reducing the network transmission latency of large-block data transmission.

2. The heterogeneous GPU system according to claim 1, wherein: During program execution, the access interface module sets the memory management base class Memory to manage the video memory of different GPU architectures; HostMemory is used to manage host memory; CudaMemory is used to manage NVIDIA GPU video memory; RocmMemory is used to manage AMD GPU video memory; NeuwareMemory is used to manage CambriconMLU video memory.

3. The heterogeneous GPU system according to claim 1, wherein: To enable direct HCA access to GPU memory, the peer agent API provided by Mellanox NICs and their OFED drivers is used to enable the HCA to access the peer memory buffer, thereby establishing a DMA data path from the Mellanox NIC to the GPU memory. Host-to-device transfers are implemented through CUDA's cudaMemcpyHostToDevice, ROCm's hipMemcpyHostToDevice, or the corresponding MLU APIs; device-to-host transfers are implemented through CUDA's cudaMemcpyDeviceToHost, ROCm's hipMemcpyDeviceToHost, or the corresponding MLU APIs; device-to-device transfers are implemented through the PCIe bus to achieve direct cross-GPU video memory transfers.

4. A heterogeneous GPU data transmission method, configured in the heterogeneous GPU system according to any one of claims 1 to 3, characterized in that: The following steps are involved: S1. Device initialization and configuration: Install a module that supports GPU Direct RDMA on the target host. Users configure the target device by calling the constructor of the memory management base class Memory and passing in the device type and device number. S2. Create a video memory management instance: After the device configuration is completed, Memory will dynamically create the corresponding video memory management class instance according to the device type specified by the user through the createMemoryClass method; S3. Unified call of video memory operation: Users can directly complete video memory operations through the unified interface provided by Memory; S4. Use pipeline double buffer strategy for data transmission.

5. The heterogeneous GPU data transmission method according to claim 4, characterized in that: During initialization in step S1, the GPU Direct RDMA module and RDMA environment are checked to see if they are configured correctly, and the necessary resources are initialized. In step S3, after the video memory is allocated, it will be registered with the RDMA subsystem through the ibv_reg_mr interface so that RDMA can directly access these memory areas.

6. The heterogeneous GPU data transmission method according to claim 4, characterized in that: The next RDMA write to the same memory region will occur after the copy is complete.

Images