Heterogeneous GPU system and data transmission method

Abstract

The invention belongs to the technical field of GPUs, and particularly relates to a heterogeneous GPU system and a data transmission method.The heterogeneous GPU system comprises an access interface module and a data transmission module; the access interface module is used for packaging native memory management interfaces of different GPUs and providing consistent memory access interfaces; a GPU memory transmission interface is designed on the basis of unilateral semantics of libibverbs; the data transmission module is used for realizing direct data transmission of a remote heterogeneous GPU memory by using GDR and RDMA transmission primitives; double data buffer areas are arranged, a sender and a receiver respectively maintain two data buffer areas residing in a GPU memory, and the buffer areas are distributed through a unified GPU memory access interface. The method has the advantages that a unified video memory operation interface and a communication interface are designed, and video memory interoperation between heterogeneous GPUs is supported.

Description

Technical Field

[0001] The present application belongs to the field of GPU technology, and specifically relates to a heterogeneous GPU system and a data transmission method. Background Art

[0002] Modern GPUs vary significantly in memory capacity, computing power, and architectural design. For example, NVIDIA's A100 and H100 GPUs offer 40 GB and 80 GB of video memory capacity, respectively, while AMD's MI250X GPU offers up to 128 GB of video memory capacity. In practical applications, especially in heterogeneous computing clusters, GPUs from different vendors need to work together to complete data sharing and model training tasks. However, existing communication libraries (such as NCCL, RCCL, and UCC) only support data transmission between homogeneous GPUs and cannot achieve direct communication of GPU memory across vendors.

[0003] Existing communication libraries are mainly designed for homogeneous GPU environments and lack support for direct memory transfer of heterogeneous GPUs. For example, NCCL (NVIDIA Collective Communications Library): Designed specifically for NVIDIA GPUs, it relies on the CUDA ecosystem and cannot support memory communication of GPUs from other manufacturers such as AMD and Intel. RCCL (ROCm Collective Communications Library): Based on the AMD ROCm ecosystem, it supports collective communication of AMD GPUs, but cannot achieve direct memory transfer with NVIDIA or Intel GPUs. UCX (Unified Communication X): Although it supports cross-platform communication, it is still limited by vendor-specific drivers and interfaces in terms of direct GPU memory transfer. MPI-aware GDR (Message Passing Interface with GPU Direct RDMA): It combines RDMA technology to achieve GPU memory communication, but only supports GPUs from the same vendor and cannot achieve direct memory transfer across vendors. Summary of the invention

[0004] Based on the above problems, this application designs a unified video memory operation interface and communication interface to support video memory interoperability between heterogeneous GPUs. The technical solution is: A heterogeneous GPU system includes an access interface module and a data transmission module; The access interface module: encapsulates the native memory management interface of different GPUs and provides a consistent memory access interface; designs the GPU memory transmission interface based on the unilateral semantics of libibverbs; The data transmission module: uses GDR and RDMA transmission primitives to realize direct data transmission to remote heterogeneous GPU memory; sets up double data buffers, and the sender and the receiver each maintain two data buffers residing in the GPU memory, and these buffers are allocated through a unified GPU memory access interface.

[0005] Preferably, the data transmission module sets two buffers. During the data transmission process, one buffer is used for GPU memory copying, and the other buffer is used to transmit data through the network. In the next cycle, the roles of the two buffers are exchanged.

[0006] Preferably, during the execution of the program, the access interface module sets a memory management base class Memory for managing video memory of different GPU architectures; HostMemory is used to manage host memory; CudaMemory is used to manage NVIDIA GPU video memory; RocmMemory is used to manage AMD GPU video memory; NeuwareMemory is used to manage Cambricon MLU video memory.

[0007] Preferably, in order to realize direct access of the host channel adapter HCA to the GPU memory, the peer agent API provided by Mellanox NICs and their OFED drivers is used to realize the host channel adapter HCA access to the peer memory buffer, thereby establishing a DMA data path from the Mellanox NIC to the GPU memory; Host-to-device transmission is implemented through CUDA's cudaMemcpyHostToDevice, ROCm's hipMemcpyHostToDevice, or MLU's corresponding APIs; device-to-host transmission is implemented through CUDA's cudaMemcpyDeviceToHost, ROCm's hipMemcpyDeviceToHost, or MLU's corresponding APIs; device-to-device transmission is implemented through the PCIe bus to achieve direct transmission of video memory across GPUs.

[0008] Preferably, in order to achieve message synchronization between the sender and the receiver, a status buffer is designed, which includes writability information and the size of the data to be sent, and the buffer controls the data transmission process; the status buffer is maintained on both sides of the host memory, and the message is synchronized through a unilateral write operation; when the sender completes writing the data to the receiver, it updates the message in the sender's status buffer and sends it to the receiver. After the receiver successfully detects the message, it starts to read the sent data, thereby ensuring synchronous and efficient data processing.

[0009] A heterogeneous GPU data transmission method comprises the following steps: S1. Equipment initialization and configuration: Install a module that supports GPU Direct RDMA on the target host. The user completes the configuration of the target device by calling the constructor of the memory management base class Memory and passing in the device type and device number. S2. Creation of video memory management instance: After the device configuration is completed, Memory will dynamically create the corresponding video memory management class instance according to the device type specified by the user through the createMemoryClass method; S3. Unified call of video memory operation: Users can directly complete video memory operations through the unified interface provided by Memory; S4. Use pipeline double buffer strategy for data transmission.

[0010] Preferably, during initialization in step S1, it is checked whether the GPU Direct RDMA module and the RDMA environment are configured correctly, and necessary resources are initialized; In step S3, after the video memory is allocated, it will be registered with the RDMA subsystem through the ibv_reg_mr interface so that RDMA can directly access these memory areas.

[0011] Preferably, the pipeline double buffer data transmission strategy in step S4 is as follows: S41. The sender copies the data block to a data buffer and sends a message to the receiver to notify the total size of the data to be sent; S42. The sender publishes two write events on the data buffer. The first event is used to write the actual data to the remote GPU memory, and the second event is used to synchronize the readability message to the receiver. These two events are published sequentially on the same QP to ensure the orderly execution of RDMA. At the same time, the sender starts to copy the data to the second data buffer. S43. Once the receiver detects the readability notification, it immediately copies the data from its data buffer to the application and sends a writability notification to the sender. At the same time, the second data buffer starts receiving network data. S44. Receive the next readability notification immediately after the current iteration is completed.

[0012] Preferably, regardless of the latency between the data copy and the RDMA write, the next RDMA write to the same memory region will occur after the copy is complete.

[0013] Preferably, when the GPU memory copy latency is higher than the RDMA write latency when the message size is below a certain threshold, the transmission latency is minimized by configuring the size of the data buffer to be close to the threshold.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This application implements the direct memory transmission capability of heterogeneous video memory, and supports direct memory transmission between graphics cards such as Cambrian, Hygon, and NVIDIA.

[0015] The high-performance transmission solution designed in this application achieves higher bandwidth and lower latency compared to unoptimized solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a graph of the experimental results.

[0017] Figure 2 The figure is a transmission flow chart. DETAILED DESCRIPTION

[0018] The technical solution of the present application is described in detail below through specific embodiments and drawings. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application, and the specific technical features may be combined with each other.

[0019] A heterogeneous GPU system comprises an access interface module and a data transmission module; the access interface module encapsulates native memory management interfaces of different GPUs and provides a consistent memory access interface; a GPU memory transmission interface is designed based on the unilateral semantics of libibverbs; the data transmission module uses GDR and RDMA transmission primitives to realize direct data transmission to remote heterogeneous GPU memory; a double data buffer is set, a sender and a receiver respectively maintain two data buffers residing in the GPU memory, and these buffers are allocated through a unified GPU memory access interface.

[0020] This interface encapsulates the native memory management interface of each GPU and provides a consistent memory access interface, such as memory allocation, memory copying, and memory release. In this way, upper-level programs can access the memory of different GPUs in a unified manner without paying attention to the differences in the underlying hardware. During program execution, UHM automatically adapts to the specific GPU type based on the type of host device, ensuring seamless compatibility in heterogeneous environments. This design allows upper-level applications to migrate between different GPUs without modification, simplifying the complexity of development and maintenance.

[0021] The unified operation interface of heterogeneous GPU memory is as follows: Core data structure definition and implementation: The present invention defines a unified heterogeneous GPU memory management interface, shielding the memory management details of different GPU architectures, and providing users with an efficient and easy-to-use memory management solution. The core data structure includes a memory management base class MemoryBase, which defines the basic interface of memory management, including functions such as memory allocation, release, and data transmission. Specific subclasses such as CudaMemory and RocmMemory implement these interfaces according to different GPU architectures.

[0022] The unified interface class Memory provides a user-friendly interface for managing video memory of different GPU architectures. By setting the device type, the user can automatically create a corresponding video memory management class instance. For different GPU architectures, the present invention implements the following video memory management classes: HostMemory is used to manage host memory; CudaMemory is used to manage NVIDIA GPU video memory; RocmMemory is used to manage AMD GPU video memory; NeuwareMemory is used to manage Cambricon MLU video memory.

[0023] In terms of interface implementation and function expansion, the video memory allocation and release interface uses corresponding APIs according to different GPU architectures. For NVIDIA GPU, CUDA API (such as cudaMalloc and cudaFree) is used to allocate and release video memory; for AMD GPU, ROCm API (such as hipMalloc and hipFree) is used to allocate and release video memory; for Cambricon MLU, MLU API (such as mluMalloc and mluFree) is used to allocate and release video memory.

[0024] In terms of data transmission interface, host-to-device transmission is implemented through CUDA's cudaMemcpyHostToDevice, ROCm's hipMemcpyHostToDevice or MLU's corresponding API; device-to-host transmission is implemented through CUDA's cudaMemcpyDeviceToHost, ROCm's hipMemcpyDeviceToHost or MLU's corresponding API; device-to-device transmission is achieved through the PCIe bus to directly transmit the video memory across GPUs, avoiding the additional overhead of data passing through the host memory.

[0025] In terms of initialization and resource management, each memory management class (such as CudaMemory) will set the current device and initialize the corresponding GPU context during initialization. Through the init method of the Memory class, users can automatically complete the creation and initialization of the memory management instance.

[0026] 1) Users need to install a module that supports GPU Direct RDMA on the target host, such as NVIDIA's GDR module or AMD's peermem module. These modules are usually integrated with the GPU driver and OFED driver to provide the necessary support for RDMA communication operations on the video memory.

[0027] 2) The user creates a communication endpoint instance through the memory communication endpoint class. During the instantiation process, the communication endpoint class automatically detects the GPU Direct RDMA environment and prepares for subsequent memory operations and RDMA communications.

[0028] 3) The user allocates video memory through the video memory operation interface provided by the communication endpoint class. After the allocation is completed, the communication endpoint class will call the ibv_reg_mr interface to register the video memory with the RDMA subsystem. This process ensures that RDMA can directly access the video memory, thereby achieving efficient data transmission between video memories.

[0029] 4) Users can use the RDMA primitive interfaces (such as rdma_write and rdma_read) encapsulated by the communication endpoint class to directly read and write the peer's video memory. These operations are based on the unilateral semantics of RDMA and can significantly reduce data transmission latency and system overhead.

[0030] Heterogeneous video memory operation process: 1. Equipment initialization and configuration: The user completes the configuration of the target device by calling the constructor of the Memory class and passing in the device type (such as MemoryType::NVIDIA_GPU or MemoryType::AMD_GPU) and the device number. This process ensures that the video memory management module can accurately identify the target device type and allocate corresponding resources to it.

[0031] 2. Creation of video memory management instance: After the device configuration is completed, the Memory class will dynamically create the corresponding memory management class instance according to the device type specified by the user through the createMemoryClass method. For example, for NVIDIA GPU, a CudaMemory instance will be created; for AMD GPU, a RocmMemory instance will be created. This mechanism realizes the dynamic adaptation of memory management for different GPU architectures.

[0032] 3. Unified call of video memory operation: Users can directly complete video memory operations through the unified interface provided by the Memory class (such as allocate_buffer, free_buffer, copy_host_to_device, etc.) without having to worry about the underlying implementation details. This design greatly simplifies the user's usage complexity while ensuring the efficiency and consistency of video memory operations.

[0033] 4. Heterogeneous video memory point-to-point direct transmission solution: The GPU Direct RDMA (GDR) module is a technology used to implement direct memory transfer between GPUs. Through this module, the host channel adapter (HCA) can directly access the GPU memory without going through the traditional CPU memory copy process. This not only reduces the latency of data transmission, but also improves bandwidth utilization, and is especially suitable for inter-GPU communication in high-performance computing and distributed systems. GDR modules are usually provided by GPU manufacturers, such as NVIDIA's GDR module and AMD's peermem module, which provide underlying support for RDMA communication through integration with the OFED driver.

[0034] Using the GPU Direct RDMA module and the RDMA libverbs library, the present invention designs a video memory communication endpoint class that encapsulates the primitive operations of video memory registration, queue pair creation, and data transmission. This class mainly includes the following core functions: Initialization and resource management: During initialization, this class checks whether the GPU Direct RDMA module and RDMA environment are configured correctly, and initializes necessary resources such as protection domains (PDs) and queue pairs (QPs).

[0035] Video memory operation interface: Through this interface, users can easily allocate and release video memory. After the video memory is allocated, it will be registered to the RDMA subsystem through the ibv_reg_mr interface so that RDMA can directly access these memory areas.

[0036] RDMA data transmission: This class encapsulates RDMA unilateral operations (such as rdma_write and rdma_read), allowing users to directly read and write the peer's video memory. These operations are based on the native semantics of RDMA, ensuring the efficiency and reliability of data transmission.

[0037] Through the memory communication endpoint class, users can seamlessly integrate memory operations with RDMA communication, simplifying the implementation process of direct memory transfer between GPUs.

[0038] Pipeline double buffer data transfer strategy: In heterogeneous GPU environments, there is a significant performance difference between direct memory reads and writes and memory copies. Memory reads and writes are usually implemented through RDMA, which has the characteristics of low latency and high bandwidth; while memory copies may involve more system overhead, especially when data needs to be transferred between GPUs. In addition, registering large blocks of memory for buffers may lead to a waste of memory resources, especially when memory resources are limited. Therefore, it is particularly important to design a transmission strategy that can efficiently utilize memory resources and balance memory copy latency and transmission latency. Figure 2 The pipeline double buffer data transmission strategy designed by the present invention is demonstrated: Phase 1: The sender copies the data block to a data buffer and sends a message to the receiver to inform the total size of the data to be sent. The sender and receiver can calculate the number of transmission iterations required based on this information.

[0039] Phase 2: The sender publishes two write events on the data buffer. The first event is used to write the actual data to the remote GPU memory, and the second event is used to synchronize the readability message to the receiver. These two events are published sequentially on the same QP to ensure the in-order execution of RDMA. At the same time, the sender starts copying the data to the second data buffer. In this way, RDMA data transfer and message synchronization overlap with GPU memory copy.

[0040] Phase 3: Once the receiver detects a readability notification, it immediately copies the data from its data buffer to the application memory and sends a writability notification to the sender. At the same time, the second data buffer starts receiving network data. Regardless of the latency of the data copy and RDMA write, the next RDMA write to the same memory area will occur after the copy is complete. This ensures that the data is not overwritten, thereby ensuring data integrity.

[0041] Receiving the next readability notification (for the second buffer) immediately after the current iteration is completed can maximize pipeline efficiency. The GPU memory copy latency is higher than the RDMA write latency when the message size is below a certain threshold. Therefore, we can achieve the above goal by configuring the size of the data buffer to be close to this threshold. With this pipeline design, we hide the RDMA data transfer latency in the GPU memory copy latency, thereby significantly reducing the network transmission latency of large block data transfers.

[0042] This solution designs two key components: a state buffer (for message synchronization) and a double data buffer (for pipeline data transmission), which reside in the host memory and GPU memory respectively.

[0043] Status buffer for message synchronization: In order to achieve message synchronization between the sender and the receiver, this application designs a status buffer, which includes writability information and the size of the data to be sent. This buffer controls the data transmission process. Since the GPU memory cannot be directly accessed by the CPU program, it is impossible to embed the transmission control message directly into the data during communication like the polling design of FaRM. Instead, this scheme maintains the status buffer on both sides of the host memory and synchronizes the message through unilateral write operations. When the sender completes writing the data to the receiver, it updates the message in the sender's status buffer and sends it to the receiver. After successfully detecting the message, the receiver starts reading the sent data, thereby ensuring synchronous and efficient data processing.

[0044] Dual data buffers for pipelined data transfer: The sender and receiver each maintain two data buffers that reside in GPU memory to store the actual data. These buffers are allocated through a unified GPU memory access interface. During data transfer, one buffer is used for GPU memory copy while the other buffer is used to transfer data over the network. In the next cycle, the roles of the two buffers are swapped. This dual design overlaps RDMA data transfer with GPU memory copy, thus achieving pipelined transfer.

[0045] To enable direct HCA (Host Channel Adapter) access to GPU memory, we leveraged the peergent API provided by Mellanox NICs and their OFED (OpenFabrics Enterprise Distribution) drivers. Through this API, HCA can access peer memory buffers to establish a DMA data path from Mellanox NIC to GPU memory. GPU drivers support this functionality by implementing a peer memory client (static structpeer_memory_client_ex), and these modules are often referred to as GPU Direct RDMA (GDR). GPU vendors such as Nvidia and AMD provide corresponding kernel modules to support these features. For example, the Nvidia driver includes a GDR module, while the AMD driver embeds a peermem module. However, existing direct GPU memory transfer solutions such as xCCL, UCX, and MPI-aware GDR only support data transfer between homogeneous GPUs. To overcome this limitation, we extend the libibverbs library for GDR and RDMA and integrate them into a unified memory access interface. Heterogeneous GPU memory is registered by leveraging libibverbs interfaces (such as ibv_reg_mr), and direct data transfer to remote heterogeneous GPU memory is implemented using GDR and RDMA transfer primitives.

[0046] This application designs an experiment to verify the significant improvement in data transmission efficiency of the unified heterogeneous GPU memory access method described in the present invention. This experiment compares four different data transmission schemes, including: Ideal (ideal scheme): only using RDMA write, no additional replication delay; G2H2G (host memory transfer scheme): using host memory as a transfer to achieve inter-GPU memory transmission; NoPipe (serial replication / transmission scheme): non-pipeline version of the UHM method described in the present invention; UHM (the method described in the present invention).

[0047] The experimental results are as follows Figure 1. Under small data volume (less than 8192 bytes), the transmission delay of UHM is comparable to that of the NoPipe solution and significantly lower than that of the G2H2G solution. The average delay of the UHM and NoPipe solutions is reduced by about 50%-70% compared with the G2H2G solution. Under large data volume (greater than 32768 bytes), UHM significantly reduces the delay of the G2H2G and NoPipe solutions. Experimental data show that UHM reduces the average delay of the G2H2G solution by 38.1%-87.2%, and reduces the average delay of the NoPipe solution by 23.5%-76.7%. The performance of UHM is close to the theoretical limit of the Ideal solution. By adopting a pipeline design based on dual data buffers, UHM significantly reduces the delay introduced by memory copying. Especially in the scenario of large data transmission, the performance optimization effect of UHM is particularly significant, fully demonstrating its superiority in improving data transmission efficiency. These results show that the unified heterogeneous GPU memory access method described in the present invention can effectively reduce data transmission delay and significantly improve system performance, especially in complex heterogeneous environments.

[0048] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A heterogeneous GPU system, characterized in that: It includes an access interface module and a data transmission module; The access interface module encapsulates native memory management interfaces of different GPUs and provides a consistent memory access interface; Design GPU memory transfer interface based on libibverbs unilateral semantics; The data transmission module: uses GDR and RDMA transmission primitives to realize direct data transmission to remote heterogeneous GPU memory; sets up double data buffers, and the sender and the receiver each maintain two data buffers residing in the GPU memory, and these buffers are allocated through a unified GPU memory access interface.

2. The heterogeneous GPU system according to claim 1, characterized in that: The data transmission module sets up two buffers. During the data transmission process, one buffer is used for GPU memory copying, while the other buffer is used to transmit data through the network. In the next cycle, the roles of the two buffers are swapped.

3. The heterogeneous GPU system according to claim 1, characterized in that: During program execution, the access interface module sets the memory management base class Memory, which is used to manage the video memory of different GPU architectures; HostMemory is used to manage host memory; CudaMemory is used to manage NVIDIA GPU video memory; RocmMemory is used to manage AMD GPU video memory; NeuwareMemory is used to manage Cambricon MLU video memory.

4. The heterogeneous GPU system according to claim 1, characterized in that: In order to achieve direct access of the host channel adapter HCA to the GPU memory, the peer agent API provided by Mellanox NICs and their OFED drivers is used to enable the host channel adapter HCA to access the peer memory buffer, thereby establishing a DMA data path from the Mellanox NIC to the GPU memory; Host-to-device transmission is implemented through CUDA's cudaMemcpyHostToDevice, ROCm's hipMemcpyHostToDevice, or MLU's corresponding APIs; device-to-host transmission is implemented through CUDA's cudaMemcpyDeviceToHost, ROCm's hipMemcpyDeviceToHost, or MLU's corresponding APIs; device-to-device transmission is implemented through the PCIe bus to achieve direct transmission of video memory across GPUs.

5. The heterogeneous GPU system according to claim 1, characterized in that: In order to achieve message synchronization between the sender and the receiver, a status buffer is designed, which includes writability information and the size of the data to be sent. This buffer controls the data transmission process; the status buffer is maintained on both sides of the host memory, and the message is synchronized through a unilateral write operation; when the sender completes writing data to the receiver, it updates the message in the sender's status buffer and sends it to the receiver. After successfully detecting the message, the receiver starts to read the sent data, thus ensuring synchronous and efficient data processing.

6. A heterogeneous GPU data transmission method, characterized in that: The following steps are involved: S1. Equipment initialization and configuration: Install a module that supports GPU Direct RDMA on the target host. The user completes the configuration of the target device by calling the constructor of the memory management base class Memory and passing in the device type and device number. S2. Creation of video memory management instance: After the device configuration is completed, Memory will dynamically create the corresponding video memory management class instance according to the device type specified by the user through the createMemoryClass method; S3. Unified call of video memory operation: Users can directly complete video memory operations through the unified interface provided by Memory; S4. Use pipeline double buffer strategy for data transmission.

7. The heterogeneous GPU data transmission method according to claim 6, characterized in that: During initialization in step S1, it checks whether the GPU Direct RDMA module and the RDMA environment are configured correctly, and initializes the necessary resources; In step S3, after the video memory is allocated, it will be registered with the RDMA subsystem through the ibv_reg_mr interface so that RDMA can directly access these memory areas.

8. The heterogeneous GPU data transmission method according to claim 6, characterized in that: Step S4 pipeline double buffer data transmission strategy is as follows: S41. The sender copies the data block to a data buffer and sends a message to the receiver to notify the total size of the data to be sent; S42. The sender publishes two write events on the data buffer. The first event is used to write the actual data to the remote GPU memory, and the second event is used to synchronize the readability message to the receiver. These two events are published sequentially on the same QP to ensure the orderly execution of RDMA. At the same time, the sender starts to copy the data to the second data buffer. S43. Once the receiver detects the readability notification, it immediately copies the data from its data buffer to the application and sends a writability notification to the sender. At the same time, the second data buffer starts receiving network data. S44. Receive the next readability notification immediately after the current iteration is completed.

9. The heterogeneous GPU data transmission method according to claim 6, characterized in that: The next RDMA write to the same memory region will occur after the copy is complete.

10. The heterogeneous GPU data transmission method according to claim 6, characterized in that: When GPU memory copy latency is higher than RDMA write latency when the message size is below a certain threshold, the transmission latency is minimized by configuring the size of the data buffer to be close to this threshold.

Images