A hybrid RPC communication system and method for heterogeneous GPU communication protocols

Abstract

This invention relates to a hybrid RPC communication system and method for heterogeneous GPU communication protocols. The system includes an application layer, a physical hardware layer, a heterogeneous hardware abstraction layer, a dynamic transport router, and a hybrid RPC engine layer. The application layer initiates communication requests between heterogeneous GPUs. The physical hardware layer includes the hardware and network interface cards of the heterogeneous GPUs. The heterogeneous hardware abstraction layer encapsulates the low-level memory registration and memory addressing interfaces of the heterogeneous GPUs, generating a unified virtual memory descriptor that is universal across platforms. The dynamic transport router dynamically determines the optimal data plane transmission path based on the hardware topology between the heterogeneous GPUs. The hybrid RPC engine layer includes a control plane RPC proxy and a data plane pass-through engine. The control plane RPC proxy performs communication negotiation, and the data plane pass-through engine directly performs zero-copy data transfer between the heterogeneous GPUs based on the optimal data plane transmission path. Compared with existing technologies, this invention improves communication performance and cross-platform compatibility in heterogeneous GPU clusters.

Description

Technical Field

[0001] This invention relates to the field of GPU communication technology, and in particular to a hybrid RPC communication system and method for heterogeneous GPU communication protocols. Background Technology

[0002] With the rapid development of large-scale artificial intelligence (AI) model training and high-performance computing (HPC), the computing power of a single node or a single vendor's graphics processing unit (GPU) can no longer meet the massive computing demands. In order to build large-scale computing clusters, it is often necessary to introduce heterogeneous GPUs from different vendors.

[0003] Currently, communication between distributed GPU clusters mainly relies on the following two methods: 1) Vendor-bonded collective communication libraries: such as NVIDIA's NCCL (NVIDIA Collective Communications Library) and AMD's RCCL. These technologies enable high-speed data transfer between GPUs through low-level hardware interfaces (such as NVLink and PCIe) or networks that support Remote Direct Memory Access (RDMA).

[0004] 2) Traditional Remote Procedure Call (RPC) protocols: such as gRPC, Thrift, etc. These protocols handle control and data flow through the CPU, and have high versatility and cross-platform compatibility.

[0005] The aforementioned prior art has the following drawbacks: 1) Closed vendor ecosystems and communication silos: Existing GPU cluster communication libraries (such as NCCL) are highly tied to specific vendor hardware architectures, making it impossible to establish direct communication links between heterogeneous GPUs (such as NVIDIA GPUs and AMD GPUs). In heterogeneous clusters, they can only degenerate into relaying through main memory (CPU Memory).

[0006] 2) Traditional RPC protocols have extremely poor performance, resulting in very high copy overhead: Traditional RPC protocols are CPU-centric. When used for GPU communication, the data transmission path is: sending-end GPU memory → sending-end CPU main memory → network card → receiving-end CPU main memory → receiving-end GPU memory. This multiple data copying not only consumes a large amount of CPU and memory bus bandwidth, but also introduces extremely high end-to-end latency, which is completely unable to meet the high-bandwidth, low-latency communication requirements of large AI models.

[0007] 3) Lack of a unified heterogeneous memory management and addressing mechanism: Different manufacturers' GPUs have different memory address spaces and memory registration mechanisms. Traditional communication protocols cannot recognize the memory handles of heterogeneous GPUs, making it impossible to achieve zero-copy transmission across manufacturers.

[0008] Existing resource management solutions aimed at integrating heterogeneous computing power have failed to fundamentally address the aforementioned shortcomings in high-performance communication. For example, patent application CN115964128A discloses a method and system for unified management, scheduling, and allocation of heterogeneous GPU resources using a centralized control node, which is essentially a heterogeneous computing power resource pooling scheme. This method virtualizes GPUs from different vendors and dynamically allocates them to computing tasks through a global resource view and policy engine. While this method primarily solves the problem of unified management and task scheduling of heterogeneous resources, its communication architecture still relies on traditional, CPU-centric network protocol stacks and fails to achieve high-performance direct data communication between GPUs from different vendors. When running tightly coupled distributed computing tasks, communication between task processes scheduled to different physical GPUs still faces high latency and low bandwidth issues, making it difficult to meet the extreme communication performance requirements of AI large-scale model training. Therefore, how to achieve flexible management of heterogeneous resources while breaking through the high-performance communication barriers between GPUs from different vendors remains a core technical challenge that urgently needs to be addressed. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art by providing a hybrid RPC communication system and method for heterogeneous GPU communication protocols, thereby improving communication performance and cross-platform compatibility in heterogeneous GPU clusters.

[0010] The objective of this invention can be achieved through the following technical solutions: A hybrid RPC communication system for heterogeneous GPU communication protocols includes: an application layer, a physical hardware layer, a heterogeneous hardware abstraction layer, a dynamic transport router, and a hybrid RPC engine layer. The application layer includes upper-layer business programs used to initiate communication requests between heterogeneous GPUs; The physical hardware layer includes the hardware of heterogeneous GPUs and network interface cards; The heterogeneous hardware abstraction layer is used to encapsulate the underlying memory registration and video memory addressing interfaces of heterogeneous GPUs, and generate a unified virtual memory descriptor that is universal across platforms. The dynamic transmission router is used to dynamically determine the optimal data plane transmission path based on the hardware topology relationship between heterogeneous GPUs. The hybrid RPC engine layer includes a control plane RPC proxy and a data plane pass-through engine. The control plane RPC proxy is used for communication negotiation through standard RPC protocols. The data plane passthrough engine is used to bypass the CPU main memory and directly perform zero-copy data transfer between heterogeneous GPUs according to the optimal data plane transmission path.

[0011] Furthermore, the heterogeneous hardware abstraction layer includes multiple vendor plugins that automatically detect the local GPU type when the system starts up and load the corresponding underlying memory registration and video memory addressing functions, thereby mapping the underlying APIs of heterogeneous GPUs to a unified abstract interface.

[0012] Furthermore, the unified virtual memory descriptor includes a vendor identifier, a virtual base address, a memory length, an IPC handle, a remote access key, and a physical topology identifier.

[0013] Furthermore, the communication negotiation specifically includes handling connection establishment, authentication, heterogeneous topology discovery, and memory handle exchange based on the unified virtual memory descriptor.

[0014] According to another aspect of the present invention, a hybrid RPC communication method for heterogeneous GPU communication protocols is provided, which performs communication using the hybrid RPC communication system for heterogeneous GPU communication protocols described above, and includes the following steps: The heterogeneous hardware abstraction layer identifies the type and hardware topology information of the local heterogeneous GPU, registers and locks the memory of the GPU to be communicated, and generates a unified virtual memory descriptor that is universal across platforms. The application layer initiates a communication request between heterogeneous GPUs. The control plane RPC agent of the sending end communicates and negotiates with the control plane RPC agent of the receiving end through the standard RPC protocol, and exchanges the unified virtual memory descriptor in the RPC request. After the receiving end's control plane RPC proxy parses the unified virtual memory descriptor, it calls the local heterogeneous hardware abstraction layer to map it into a local memory access handle that can be recognized and accessed by the receiving end's GPU. By using a dynamic transmission router, the hardware connection affinity between the sending GPU and the receiving GPU is evaluated based on the hardware topology information between them, and the optimal data plane transmission path is dynamically decided based on the hardware connection affinity evaluation results. The decision result of the optimal data plane transmission path is converted into a routing instruction, and data transmission between heterogeneous GPU memory is performed through the data plane pass-through engine according to the routing instruction.

[0015] Furthermore, the specific steps for evaluating the hardware connectivity affinity between the transmitting and receiving GPUs based on the hardware topology information, and for dynamically deciding on the optimal data plane transmission path based on the hardware connectivity affinity evaluation results, include: Based on the hardware topology information, calculate the logical distance between the transmitting GPU and the receiving GPU; Based on the logical distance and combined with the real-time monitored channel bandwidth, latency and load status, the potential transmission paths are weighted and scored according to the preset performance weight table to calculate the candidate transmission paths. The candidate transmission paths are evaluated for protocol compatibility. If it is determined that the underlying drivers of the sending GPU and the receiving GPU support mutual access through the general direct memory access engine, a direct peer-to-peer interconnection transmission path is generated. If it is determined that the underlying drivers of the sending GPU and the receiving GPU do not support mutual access through the general direct memory access engine, a unified direct memory access relay path through an intermediate buffer is generated.

[0016] Furthermore, based on the hardware topology information, the specific steps for calculating the logical distance between the transmitting GPU and the receiving GPU include: The physical location relationship between the transmitting GPU and the receiving GPU is determined based on the hardware topology information. Based on the physical location relationships, assign logical distance values ​​representing communication overhead to different connection scenarios: If the sending GPU and the receiving GPU are mounted under the same PCIe root complex and support peer-to-peer interconnection, then the minimum logical distance value will be assigned. If the sending GPU and the receiving GPU are on the same physical machine but span different non-uniform memory access architecture nodes, then the allocation characterizes the intermediate logical distance value that has access overhead through the CPU memory controller. If the sending GPU and the receiving GPU are on different physical machines but are in the same subnet through a network that supports remote direct memory access, then a relatively high logical distance value representing network transmission overhead is assigned. If the sending GPU and the receiving GPU can only communicate via a traditional TCP / IP network, then the highest logical distance value is assigned.

[0017] Furthermore, when the decision result of the optimal data plane transmission path is converted into a routing instruction, and data transmission between heterogeneous GPU memory is performed through the data plane pass-through engine according to the routing instruction, if the decision result supports hardware pass-through, then a direct memory access operation is triggered according to the optimal data plane transmission path and the local memory access handle, bypassing the central processing unit's main memory, and zero-copy data transmission is completed directly between the sending GPU and the receiving GPU.

[0018] Furthermore, when the decision result of the optimal data plane transmission path is converted into a routing instruction, and data transmission between heterogeneous GPU memory is performed through the data plane pass-through engine according to the routing instruction, if the decision result does not support hardware pass-through, a downgrade process is triggered, and the data is transferred via a single copy through the paged memory of the central processing unit.

[0019] Furthermore, after confirming the completion of data transmission between heterogeneous GPU memory, the data plane passthrough engine sends a transmission completion signal to the receiving end's control plane RPC agent. Upon receiving the transmission completion signal, the receiving end's control plane RPC agent generates a standard RPC response message and returns it to the sending end's control plane RPC agent. The sending end's control plane RPC agent then returns the standard RPC response message to the application layer to notify that the communication request between the heterogeneous GPUs has been completed.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs a system architecture comprising an application layer, a physical hardware layer, a heterogeneous hardware abstraction layer, a dynamic transport router, and a hybrid RPC engine layer. The heterogeneous hardware abstraction layer maps the different memory registration and addressing APIs of heterogeneous GPUs to a unified abstract interface and generates a unified virtual memory descriptor containing all necessary metadata, providing standardized operation objects for the upper-layer dynamic transport router and hybrid RPC engine. The dynamic transmission router is responsible for intelligent path decision-making. The hybrid RPC engine layer includes a control plane RPC proxy and a data plane pass-through engine, which decouples control signaling from data transport. This solves the architectural contradiction of traditional solutions that are either tied to specific vendors or have low performance, and achieves communication performance that approaches the hardware limit while maintaining versatility.

[0021] 2. This invention separates the control plane and data plane in the hybrid RPC engine layer. The control plane RPC proxy reuses the standard RPC protocol to handle lightweight negotiation tasks such as connection, authentication, and memory handle exchange, ensuring cross-platform compatibility. The data plane pass-through engine is dedicated to executing zero-copy data transmission under the optimal path determined by the dynamic transport router. This solves the problems of traditional single RPC protocols that are CPU-centric and require the data plane to pass through main memory, resulting in multiple copies and performance bottlenecks. It achieves flexible and universal control flow and efficient pass-through of data flow.

[0022] 3. This invention designs an independent dynamic transmission router that can dynamically calculate logical distance and evaluate connection affinity based on real-time acquired hardware topology and connection relationships, and intelligently decide the optimal data plane transmission path for each communication request. This solves the problem that fixed and static transmission strategies cannot adapt to diverse hardware deployment scenarios and thus cannot maximize the utilization of hardware pass-through capabilities. This gives the system architecture adaptive optimization capabilities and high robustness in different environments. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of a hybrid RPC communication system for heterogeneous GPU communication protocols proposed in this invention; Figure 2 This is a flowchart illustrating a hybrid RPC communication method for heterogeneous GPU communication protocols proposed in this invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0025] The following English abbreviations are involved: Graphics Processing Unit (GPU) Remote Procedure Call (RPC) Heterogeneous Hardware Abstraction Layer (H-HAL) Network Interface Card (NIC) Direct Memory Access (DMA) Base Address Register (BAR) Example 1 This embodiment provides a hybrid RPC communication system for heterogeneous GPU communication protocols, such as... Figure 1 As shown, it includes: application layer, physical hardware layer, heterogeneous hardware abstraction layer, dynamic transport router and hybrid RPC engine layer.

[0026] The application layer includes upper-layer business programs used to initiate communication requests between heterogeneous GPUs.

[0027] The physical hardware layer includes the hardware of heterogeneous GPUs and network interface cards.

[0028] The Heterogeneous Hardware Abstraction Layer (HBIL) encapsulates the low-level memory registration and memory addressing interfaces of heterogeneous GPUs and generates a unified virtual memory descriptor that is universal across platforms. The HBIL includes plugins from multiple vendors, automatically detecting the local GPU type at system startup and loading the corresponding low-level memory registration and memory addressing functions, thereby mapping the low-level APIs of heterogeneous GPUs to a unified abstract interface. The unified virtual memory descriptor includes a vendor identifier, virtual base address, memory length, IPC handle, remote access key, and physical topology identifier.

[0029] Dynamic transport routers are used to dynamically determine the optimal data plane transmission path based on the hardware topology relationships between heterogeneous GPUs. They maintain a hardware topology map of the heterogeneous cluster and, before data transmission, determine the transmission path based on the location relationship between the sending and receiving GPUs. For example, if GPU A and GPU B are on the same physical machine and mounted on the same PCIe Switch, PCIe P2P transmission is selected; if they are across machines, GPUDirect RDMA transmission based on the network interface card is selected.

[0030] The hybrid RPC engine layer includes a control plane RPC proxy and a data plane pass-through engine.

[0031] The control plane RPC agent handles lightweight control signaling (e.g., communication requests, state synchronization, memory handle swapping) for communication negotiation via standard RPC protocols. Communication negotiation specifically includes handling connection establishment, authentication, heterogeneous topology discovery, and memory handle swapping based on a unified virtual memory descriptor. This part reuses existing TCP / IP or standard RPC mechanisms to ensure cross-platform compatibility.

[0032] The data plane passthrough engine is used to bypass CPU main memory and directly perform zero-copy data transfer between heterogeneous GPUs based on the optimal data plane transfer path. After receiving the optimal data plane transfer path dynamically determined by the dynamic transfer router and the memory handle exchanged by the control plane RPC proxy, it directly initiates a Direct Memory Access (DMA) operation to achieve zero-copy data transfer from GPU memory to GPU memory.

[0033] Example 2 This embodiment provides a hybrid RPC communication method for heterogeneous GPU communication protocols, such as... Figure 2 As shown, communication is performed using a hybrid RPC communication system with a heterogeneous GPU communication protocol as provided in Example 1, including the following steps: S1. Identify the type and hardware topology information of the local heterogeneous GPU through the heterogeneous hardware abstraction layer, register and lock the memory of the GPU to be communicated, and generate a unified virtual memory descriptor that is universal across platforms.

[0034] When a communication node starts up, the heterogeneous hardware abstraction layer scans the local PCIe topology, identifies the GPU model, and pre-registers a pinned memory or GPU video memory with the Network Interface Card (NIC) to generate a unified virtual memory descriptor that is universal across platforms.

[0035] After the system starts up, the heterogeneous hardware abstraction layer first performs hardware auditing through the kernel-mode interface or PCIe scanning mechanism to achieve self-discovery of heterogeneous devices.

[0036] Hardware identification scanning: Traverse the system PCIe bus to extract the vendor ID and device ID of each accelerator card. For example, 0x10de is identified as an NVIDIA device, and 0x1002 is identified as an AMD device.

[0037] Physical topology modeling: By reading the system sysfs file or calling a topology discovery library (such as hwloc), a physical connectivity graph of local resources is constructed. This records the PCIe layer where the GPU resides, its NUMA node, and its associated RDMA network interface card.

[0038] Based on the self-discovery results of heterogeneous devices, the system performs pluggable runtime binding: Dynamic library loading: The system dynamically loads the corresponding manufacturer's adapter based on the device type.

[0039] Unified Interface Mapping: The heterogeneous hardware abstraction layer defines a standard function pointer table that maps vendor-specific memory management APIs to abstract interfaces. For example, NVIDIA's cudaHost Register and AMD's hip host register are uniformly mapped to hhal_mem_register.

[0040] To ensure zero-copy data transfer, permission alignment between video memory and peripherals must be completed: Memory locking: H-HAL calls the underlying driver interface to mark the GPU memory area to be communicated as "non-swapable" to prevent the operating system from moving memory addresses during DMA transfer.

[0041] Peer-to-peer access authorization: If two heterogeneous GPUs are detected to be under the same PCIe root complex, H-HAL will automatically enable peer-to-peer memory access permissions.

[0042] Network card memory registration: If cross-machine communication is involved, H-HAL will work with the RDMA driver to create a memory protection domain for the video memory area and generate a key for remote access.

[0043] Finally, the system encapsulates the complex low-level information into a lightweight unified virtual memory descriptor.

[0044] The unified virtual memory descriptor contains a vendor identifier, virtual base address, memory length, IPC handle, remote access key, and physical topology identifier. This descriptor is registered with the local RPC service's resource management center, awaiting encapsulation by the RPCheeader in step S2. Through the unified virtual memory descriptor, heterogeneous nodes on the other end do not need to understand the physical characteristics of the sending GPU; they can reconstruct the access path in their local H-HAL simply by using the descriptor.

[0045] S2. The application layer initiates a communication request between heterogeneous GPUs. The sending end's control plane RPC agent communicates and negotiates with the receiving end's control plane RPC agent through the standard RPC protocol, and exchanges the unified virtual memory descriptor encapsulated in the RPC request.

[0046] When the application layer initiates a heterogeneous call, the sending end's control plane RPC proxy packages the local GPU's virtual memory descriptor into the RPC header and sends it to the receiving end. After parsing, the receiving end calls the local heterogeneous hardware abstraction layer to map it to a physical or virtual address that can be recognized by the local GPU.

[0047] When the sending end initiates a hybrid RPC call, the control plane proxy automatically intercepts the request. If it detects that the parameter to be transmitted is located in the GPU memory, it inserts a "heterogeneous feature field" into the header field of the standard RPC protocol (such as HTTP / 2 or a custom binary protocol). This field contains the unified virtual memory descriptor generated in step S1 and the physical topology capabilities of the current node (such as the highest supported PCIe version, RDMA protocol version, etc.).

[0048] S3. After the receiving end's control plane RPC proxy parses the unified virtual memory descriptor, it calls the local heterogeneous hardware abstraction layer to map it into a local memory access handle that can be recognized and accessed by the receiving end's GPU.

[0049] After receiving the RPC request, the receiving end's control plane RPC proxy performs the following verification: Vendor interoperability assessment: Check whether the sender and receiver vendor combinations (such as NVIDIA-to-AMD) are on the H-HAL support list.

[0050] Memory view translation: Using local H-HAL mapping functions, UMD descriptors in the header are translated into handles understandable by the local device driver. For example, a remote access address on the other end is translated into a target address for local RDMA writes.

[0051] The two communicating parties reached a consensus through multiple rounds of lightweight signaling exchange: Capability set intersection calculation: Both parties compare their supported transmission methods (P2P, RDMA, CXL) and select the common protocol with the highest bandwidth and lowest latency.

[0052] Security credential exchange: Exchange temporary session keys or memory protection identifiers to ensure that only designated hardware units can read and write the corresponding video memory area during data transmission, preventing cross-process or cross-node video memory attacks.

[0053] After successful negotiation, the server returns a tagged RPC ACK header to the client, informing the sender that the data plane channel is ready. At this point, the control plane releases control of the communication task and wakes up the data plane pass-through engine to perform the actual physical relocation.

[0054] S4. Using a dynamic transmission router, the hardware connection affinity between the sending GPU and the receiving GPU is evaluated based on the hardware topology information between them, and the optimal data plane transmission path is dynamically decided based on the hardware connection affinity evaluation results.

[0055] If two heterogeneous GPUs support direct interaction via Compute Express Link (CXL) or network RDMA, they are considered to support hardware passthrough.

[0056] The specific steps for evaluating the hardware connectivity affinity between the transmitting and receiving GPUs based on the hardware topology information, and for dynamically deciding on the optimal data plane transmission path based on the hardware connectivity affinity evaluation results, include: Based on the hardware topology information, the logical distance between the transmitting GPU and the receiving GPU is calculated. Specific steps include: Determine the physical location relationship between the transmitting GPU and the receiving GPU based on the hardware topology information; Based on physical location relationships, assign logical distance values ​​representing communication overhead to different connection scenarios: If the sending GPU and the receiving GPU are mounted under the same PCIe root complex and support peer-to-peer interconnection, then the minimum logical distance value will be assigned. If the sending GPU and the receiving GPU are on the same physical machine but span different non-uniform memory access architecture nodes, then the allocation characterizes the intermediate logical distance value that has access overhead through the CPU memory controller. If the sending GPU and the receiving GPU are on different physical machines but are in the same subnet through a network that supports remote direct memory access, then a relatively high logical distance value representing network transmission overhead is assigned. If the sending GPU and the receiving GPU can only communicate via a traditional TCP / IP network, then the highest logical distance value is assigned.

[0057] Based on the logical distance and combined with real-time monitoring of channel bandwidth, latency, and load status, potential transmission paths are weighted and scored according to a preset performance weight table to calculate candidate transmission paths.

[0058] The protocol compatibility of candidate transmission paths is judged. If it is determined that the underlying drivers of the sending GPU and the receiving GPU support mutual access through the general direct memory access engine, a direct peer-to-peer interconnection transmission path is generated. If it is determined that the underlying drivers of the sending GPU and the receiving GPU do not support mutual access through the general direct memory access engine, a unified direct memory access relay path through the intermediate buffer is generated.

[0059] S5. The decision result of the optimal data plane transmission path is converted into a routing instruction, and the data transmission between heterogeneous GPU memory is carried out through the data plane pass-through engine according to the routing instruction.

[0060] The decision on the optimal data plane transmission path is transformed into a series of atomic transmission instructions, including the source address descriptor, the destination address descriptor, the selected transmission engine ID, and the transmission trigger mode. These instructions are then dispatched to the data plane pass-through engine for execution.

[0061] When the decision result of the optimal data plane transmission path is converted into a routing instruction, and the data transmission between heterogeneous GPU memory is carried out through the data plane pass-through engine according to the routing instruction, if the decision result supports hardware pass-through, then the direct memory access operation is triggered according to the optimal data plane transmission path and the local memory access handle, bypassing the central processing unit's main memory, and completing zero-copy data transmission directly between the sending GPU and the receiving GPU.

[0062] The data plane pass-through engine performs asynchronous, high-performance transmission according to the following logic: Transfer descriptor and DMA list construction: The engine breaks down atomic instructions into tasks executable by the underlying hardware. If the transferred data is not contiguous in video memory, the engine automatically constructs a discrete-aggregate list to ensure that a single DMA operation can handle multiple memory segments. Logical addresses in the unified virtual memory descriptor are mapped to actual physical addresses and populated into the DMA descriptor table.

[0063] Heterogeneous pass-through execution: Based on the routing decision, the corresponding hardware migration engine is invoked, triggering the peer-to-peer DMA controller between heterogeneous GPUs. The sending GPU directly initiates a PCIe transaction, pushing data to the target GPU's BAR space without passing through the system main memory. This triggers the network card engine supporting GPU Direct. The network card directly reads data from the sending GPU's memory using PCIe Peer-to-Peer technology and encapsulates it into a RoCE packet. After unpacking, the receiving network card directly writes the payload to the target GPU's memory, achieving zero-copy cross-machine data transfer at the physical layer.

[0064] To maximize throughput, the data plane employs an asynchronous pipeline mechanism: after a transmission request is submitted, the control plane returns immediately, allowing the application layer to continue processing subsequent computation tasks. The engine inserts memory barrier instructions at the start and end of data transmission to ensure memory consistency between computation and communication tasks, preventing the reading of incomplete old data.

[0065] For ultra-low latency tasks, the completion status is detected by polling the memory flag; for long-cycle tasks, a hardware interrupt is used. Once the hardware layer confirms that the data has been written back to the target memory, the data plane engine sends a local signal to the control plane to trigger the response steps.

[0066] After confirming the completion of data transfer between heterogeneous GPU memory, the data plane passthrough engine sends a transfer completion signal to the receiving end's control plane RPC agent. Upon receiving the transfer completion signal, the receiving end's control plane RPC agent generates a standard RPC response message and returns it to the sending end's control plane RPC agent. The sending end's control plane RPC agent then returns the standard RPC response message to the application layer to notify that the communication request between the heterogeneous GPUs has been completed.

[0067] When the decision result of the optimal data plane transmission path is converted into routing instructions, and data is transferred between heterogeneous GPU memory via the data plane pass-through engine according to the routing instructions, if the decision result does not support hardware pass-through, a degradation process is triggered, and the data is transferred via a single copy through the CPU's paged memory. For example, when the PCIe switch between two heterogeneous GPUs disables P2P functionality due to security policies, or the network card does not support direct reading of specific GPU memory, the system adds a degradation processing step. H-HAL first allocates a block of host fixed memory on the host CPU, and adopts a single copy strategy. First, the data is pulled from GPU A to the host fixed memory through GPU A's DMA engine, and then the data is pushed from the host fixed memory to GPU B's memory through GPU B's DMA engine. Although CPU memory is introduced, compared with the multiple memory allocations and copies of traditional RPC, the efficiency and stability of heterogeneous communication are still significantly improved.

[0068] The rest is the same as in Example 1.

[0069] Example 3 When the system's application scenario involves communication between heterogeneous GPUs on the same machine, both AMD and NVIDIA GPUs are connected within the same server for single-machine heterogeneous inference collaboration. Since cross-machine networking is not involved, network security credential exchange and complex network RDMA negotiation steps can be reduced. The dynamic transport router directly detects that both GPUs are under the same PCIe tree, and H-HAL directly invokes the operating system-level inter-process communication (IPC) mechanism to share file descriptors, completing the GPU memory copy directly via PCIe P2P.

[0070] The rest is the same as in Example 1.

[0071] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A hybrid RPC communication system for heterogeneous GPU communication protocols, characterized in that, include: Application layer, physical hardware layer, heterogeneous hardware abstraction layer, dynamic transport router and hybrid RPC engine layer, The application layer includes upper-layer business programs used to initiate communication requests between heterogeneous GPUs; The physical hardware layer includes the hardware of heterogeneous GPUs and network interface cards; The heterogeneous hardware abstraction layer is used to encapsulate the underlying memory registration and video memory addressing interfaces of heterogeneous GPUs, and generate a unified virtual memory descriptor that is universal across platforms. The dynamic transmission router is used to dynamically determine the optimal data plane transmission path based on the hardware topology relationship between heterogeneous GPUs. The hybrid RPC engine layer includes a control plane RPC proxy and a data plane pass-through engine. The control plane RPC proxy is used for communication negotiation through standard RPC protocols. The data plane passthrough engine is used to bypass the CPU main memory and directly perform zero-copy data transfer between heterogeneous GPUs according to the optimal data plane transmission path.

2. The hybrid RPC communication system for heterogeneous GPU communication protocols according to claim 1, characterized in that, The heterogeneous hardware abstraction layer includes plugins from multiple vendors. When the system starts up, it automatically detects the local GPU type and loads the corresponding underlying memory registration and video memory addressing functions, thereby mapping the underlying APIs of heterogeneous GPUs to a unified abstract interface.

3. The hybrid RPC communication system for heterogeneous GPU communication protocols according to claim 1, characterized in that, The unified virtual memory descriptor includes a vendor identifier, virtual base address, memory length, IPC handle, remote access key, and physical topology identifier.

4. The hybrid RPC communication system for heterogeneous GPU communication protocols according to claim 1, characterized in that, The communication negotiation specifically includes handling connection establishment, authentication, heterogeneous topology discovery, and memory handle exchange based on the unified virtual memory descriptor.

5. A hybrid RPC communication method for heterogeneous GPU communication protocols, characterized in that, Communication is performed using a hybrid RPC communication system based on a heterogeneous GPU communication protocol as described in any one of claims 14, comprising the following steps: The heterogeneous hardware abstraction layer identifies the type and hardware topology information of the local heterogeneous GPU, registers and locks the memory of the GPU to be communicated, and generates a unified virtual memory descriptor that is universal across platforms. The application layer initiates a communication request between heterogeneous GPUs. The control plane RPC agent of the sending end communicates and negotiates with the control plane RPC agent of the receiving end through the standard RPC protocol, and exchanges the unified virtual memory descriptor in the RPC request. After the receiving end's control plane RPC proxy parses the unified virtual memory descriptor, it calls the local heterogeneous hardware abstraction layer to map it into a local memory access handle that can be recognized and accessed by the receiving end's GPU. By using a dynamic transmission router, the hardware connection affinity between the sending GPU and the receiving GPU is evaluated based on the hardware topology information between them, and the optimal data plane transmission path is dynamically decided based on the hardware connection affinity evaluation results. The decision result of the optimal data plane transmission path is converted into a routing instruction, and data transmission between heterogeneous GPU memory is performed through the data plane pass-through engine according to the routing instruction.

6. The hybrid RPC communication method for heterogeneous GPU communication protocols according to claim 5, characterized in that, The specific steps for evaluating the hardware connectivity affinity between the transmitting and receiving GPUs based on the hardware topology information, and for dynamically deciding on the optimal data plane transmission path based on the hardware connectivity affinity evaluation results, include: Based on the hardware topology information, calculate the logical distance between the transmitting GPU and the receiving GPU; Based on the logical distance and combined with the real-time monitored channel bandwidth, latency and load status, the potential transmission paths are weighted and scored according to the preset performance weight table to calculate the candidate transmission paths. The candidate transmission paths are evaluated for protocol compatibility. If it is determined that the underlying drivers of the sending GPU and the receiving GPU support mutual access through the general direct memory access engine, a direct peer-to-peer interconnection transmission path is generated. If it is determined that the underlying drivers of the sending GPU and the receiving GPU do not support mutual access through the general direct memory access engine, a unified direct memory access relay path through an intermediate buffer is generated.

7. The hybrid RPC communication method for heterogeneous GPU communication protocols according to claim 6, characterized in that, Based on the hardware topology information, the specific steps for calculating the logical distance between the transmitting GPU and the receiving GPU include: The physical location relationship between the transmitting GPU and the receiving GPU is determined based on the hardware topology information. Based on the physical location relationships, assign logical distance values ​​representing communication overhead to different connection scenarios: If the sending GPU and the receiving GPU are mounted under the same PCIe root complex and support peer-to-peer interconnection, then the minimum logical distance value will be assigned. If the sending GPU and the receiving GPU are on the same physical machine but span different non-uniform memory access architecture nodes, then the allocation characterizes the intermediate logical distance value that has access overhead through the CPU memory controller. If the sending GPU and the receiving GPU are on different physical machines but are in the same subnet through a network that supports remote direct memory access, then a relatively high logical distance value representing network transmission overhead is assigned. If the sending GPU and the receiving GPU can only communicate via a traditional TCP / IP network, then the highest logical distance value is assigned.

8. The hybrid RPC communication method for heterogeneous GPU communication protocols according to claim 5, characterized in that, When the decision result of the optimal data plane transmission path is converted into a routing instruction, and data transmission between heterogeneous GPU memory is performed through the data plane pass-through engine according to the routing instruction, if the decision result supports hardware pass-through, then a direct memory access operation is triggered according to the optimal data plane transmission path and the local memory access handle, bypassing the central processing unit's main memory, and zero-copy data transmission is completed directly between the sending GPU and the receiving GPU.

9. The hybrid RPC communication method for heterogeneous GPU communication protocols according to claim 5, characterized in that, When the decision result of the optimal data plane transmission path is converted into a routing instruction, and data transmission between heterogeneous GPU memory is performed through the data plane pass-through engine according to the routing instruction, if the decision result does not support hardware pass-through, a degradation process is triggered, and the data is transferred through the paged memory of the central processing unit in a single copy.

10. The hybrid RPC communication method for heterogeneous GPU communication protocols according to claim 5, characterized in that, After confirming the completion of data transmission between heterogeneous GPU memory, the data plane passthrough engine sends a transmission completion signal to the receiving end's control plane RPC agent. Upon receiving the transmission completion signal, the receiving end's control plane RPC agent generates a standard RPC response message and returns it to the sending end's control plane RPC agent. The sending end's control plane RPC agent then returns the standard RPC response message to the application layer to notify that the communication request between the heterogeneous GPUs has been completed.

Images