Heterogeneous GPU data transmission system and method
By combining the Apache Arrow format and topology-aware scheduling units, the interoperability problem between heterogeneous GPUs is solved, enabling efficient and transparent data transmission, reducing development complexity and performance overhead, and forming a high-performance heterogeneous GPU data transmission infrastructure.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF SCI & TECH OF CHINA
- Filing Date
- 2026-04-13
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, heterogeneous GPUs cannot directly communicate with each other, resulting in a closed hardware ecosystem, redundant general transmission paths and high performance overhead, and fragmented communication interfaces leading to high development complexity.
By constructing a unified tensor abstraction layer in the Apache Arrow format, the private tensor data structures of heterogeneous GPUs are losslessly mapped to standardized memory objects. A topology-aware scheduling unit is introduced to select the optimal transmission path based on the physical location of the node, achieving zero-copy within the same machine or cross-machine network transmission, avoiding redundant copying and serialization operations.
It enables efficient and transparent data transmission between heterogeneous GPUs, significantly improving transmission efficiency, reducing development and maintenance complexity, and providing a location-transparent communication interface.
Smart Images
Figure CN122019450A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of robot middleware and distributed computing technology, and in particular to a heterogeneous GPU data transmission system and method. Background Technology
[0002] In the fields of robotics development, autonomous driving, and high-performance computing, integrating graphics processing units (GPUs, collectively referred to as "heterogeneous GPUs") from different manufacturers (such as NVIDIA, AMD, and Huawei Ascend) or employing different computing architectures to meet the demands of high-frequency, large-scale tensor data exchange in distributed robotic systems or AI computing clusters has become a key means of improving computing performance. The core prerequisite for achieving efficient collaboration is realizing efficient and transparent data transmission between these heterogeneous GPUs: efficiency requires low latency and high throughput; transparency requires shielding upper-layer computing tasks from the differences in underlying hardware and the complexity of network topology.
[0003] However, existing technologies have the following typical drawbacks in achieving this goal: First, the ecosystem is closed, making it difficult for heterogeneous hardware to communicate directly. The native high-speed interconnect technologies provided by major GPU manufacturers (such as NVIDIA's NVLink) are proprietary protocols and dedicated physical interfaces. They are incompatible between different manufacturers and different generations of architectures, making it impossible for heterogeneous GPUs to establish direct data channels, thus forming a hardware and protocol barrier that is difficult to overcome.
[0004] Second, the common transmission path is redundant, resulting in significant performance loss. To circumvent ecosystem barriers, the industry generally adopts a common path via CPU memory: data must undergo multiple copies: "sending GPU memory → host memory → network → host memory → receiving GPU memory". Simultaneously, to adapt to network transmission, high-overhead serialization and deserialization operations must be performed on the CPU. This process incurs huge data copying overhead and CPU computational burden, significantly increasing transmission latency and reducing effective bandwidth, becoming a system performance bottleneck.
[0005] Third, fragmented communication interfaces lead to high development and deployment costs. Existing distributed frameworks lack a unified communication abstraction, requiring developers to explicitly select and invoke different underlying communication interfaces (such as CUDA IPC, Unix domain sockets, network sockets, etc.) based on the physical deployment relationship between the sender and receiver (same process, same machine cross process, cross machine network). This results in deep coupling between business logic and hardware topology, a lack of system location transparency, and significantly increases the development, debugging, maintenance, and cross-environment migration costs of distributed applications.
[0006] Therefore, existing technical solutions have long been limited by the triple challenges of hardware incompatibility, high performance overhead, and high development complexity when dealing with the data transmission requirements of heterogeneous GPUs. There is a lack of a technical solution that can systematically solve the above defects at the same time. Summary of the Invention
[0007] This application proposes a heterogeneous GPU data transmission scheme to address the problems in the prior art, such as the inability of heterogeneous GPUs to directly interoperate due to the closed hardware ecosystem, the large performance overhead caused by the redundancy of general transmission paths and serialization overhead, and the high development complexity caused by the fragmented communication interfaces and lack of location transparency.
[0008] A first aspect of this application provides a heterogeneous GPU data transmission system, including a sending node, a receiving node, and a topology-aware scheduling unit, comprising: The sending node has a first GPU and is configured to call a cross-platform tensor encapsulation tool library to encapsulate tensor data from the first GPU into memory objects conforming to the Apache Arrow format. The topology-aware scheduling unit is used to sense the physical location relationship between the sending node and the receiving node, and based on the location relationship, select a same-machine zero-copy transmission path or a cross-machine network transmission path for the encapsulated memory object. The sending node is also used to send the memory object conforming to the Apache Arrow format to the receiving node through the path selected by the topology-aware scheduling unit; The receiving node has a second GPU and is configured to call the cross-platform tensor wrapping tool library to load data from the memory object into the second GPU; The first GPU and the second GPU are heterogeneous.
[0009] In some embodiments of this application, the cross-platform tensor wrapping tool library wraps tensor data into memory objects conforming to the Apache Arrow format, including: The cross-platform tensor encapsulation tool library extracts the memory pointer, span, dimension, and data type of the tensor; Based on the extracted information, the cross-platform tensor wrapper library constructs memory descriptors in Apache Arrow format.
[0010] In some embodiments of this application, the topology-aware scheduling unit selects a transmission path based on the physical location relationship, including: If the process identifiers of the sending node and the receiving node are the same, the topology-aware scheduling unit determines that the transmission path is intra-process pointer passing. If the machine identifiers of the sending node and the receiving node are the same but their process identifiers are different, then the topology-aware scheduling unit determines that the transmission path is a zero-copy transmission path on the same machine. If the machine identifiers of the sending node and the receiving node are different, the topology-aware scheduling unit determines the transmission path as a cross-machine network transmission path.
[0011] In some embodiments of this application, when the selected transmission path is a zero-copy transmission path on the same machine, the sending node mounts the memory object conforming to the Apache Arrow format to the shared memory segment; The identifier of the shared memory segment is transmitted to the receiving node via inter-process communication.
[0012] In some embodiments of this application, when the selected transmission path is a cross-machine network transmission path, the sending node transmits the memory object conforming to the Apache Arrow format based on the Zenoh communication protocol, and there is no need to perform serialization operation on the memory object during the transmission process.
[0013] The second aspect of this application provides a heterogeneous GPU data transmission method, executed by a sending node in the system described in the first aspect of this application, including: The tensor data in the first GPU contained in the sending node is encapsulated into a memory object conforming to the Apache Arrow format; Obtain the location information of the receiving node; Based on the location information, a transmission path is obtained, wherein the transmission path is determined by the topology-aware scheduling unit in the system based on the location information, including a zero-copy transmission path within the same machine or a cross-machine network transmission path. The memory object conforming to the Apache Arrow format is sent to the receiving node via the obtained transmission path, so that the data therein can be loaded into the second GPU contained in the receiving node. The first GPU and the second GPU are heterogeneous.
[0014] In some embodiments of this application, obtaining the location information of the receiving node includes: The location information is obtained by querying a global registry center, which records the machine identifier, process identifier, and GPU device information of the nodes in the system.
[0015] In some embodiments of this application, encapsulating the tensor data in the first GPU contained in the sending node into a memory object conforming to the Apache Arrow format includes: When the tensor data is stored non-contiguously, calculate the logical offset of the tensor data; The tensor data and its logical offsets are described in the Apache Arrow RecordBatch pattern.
[0016] A third aspect of this application provides a heterogeneous GPU data transmission method, executed by a receiving node in the system described in the first aspect of this application, including: Receive a memory object conforming to the Apache Arrow format from the sending node, wherein the memory object is encapsulated from tensor data in a first GPU contained in the sending node; Using the cross-platform tensor wrapper library, the data in the memory object is loaded into the second GPU contained in the receiving node; The first GPU and the second GPU are heterogeneous.
[0017] In some embodiments of this application, loading data from the memory object into the second GPU using the cross-platform tensor wrapping tool library includes: The cross-platform tensor encapsulation tool library is used to parse the memory descriptor in the memory object conforming to the Apache Arrow format, and the memory is directly mapped to the address space of the second GPU for access via memory mapping.
[0018] In summary, the heterogeneous GPU data transmission system and method proposed in the embodiments of this application, by constructing a unified tensor abstraction layer based on the Apache Arrow format, losslessly maps the private tensor data structures of various heterogeneous GPUs into standardized memory objects, achieving a unified cross-platform description of data. This breaks down the data format barriers caused by the closed hardware ecosystem and fundamentally solves the problem of heterogeneous GPUs being unable to directly interoperate. By introducing a topology-aware scheduling unit, which adaptively selects either "zero-copy within the same machine" or "cross-machine network" transmission paths based on the physical location of nodes, this mechanism ensures that data is always transmitted via the optimal path: in the same-machine scenario, shared memory and other technologies are used to avoid redundant copies via host memory; in the cross-machine scenario, the "store-as-transfer" characteristic of the Apache Arrow format eliminates the need for CPU serialization / deserialization. Thus, the overhead of multiple data copies and format conversions in existing general transmission paths is completely eliminated, significantly improving transmission efficiency and solving the problem of high performance overhead. Through the synergy of the aforementioned "unified abstraction" and "intelligent scheduling," a location-transparent communication interface is provided for upper-layer applications. Application developers only need to perform unified send / receive calls, without needing to be aware of or adapt to the complex underlying hardware topology and network differences. This achieves complete decoupling of business logic and physical deployment, greatly reducing the complexity of system development, maintenance, and migration, thus solving the problem of high development complexity. Therefore, this application, through the organic combination of the above technical features, forms a complete high-performance heterogeneous GPU data transmission infrastructure. Attached Figure Description
[0019] The features and advantages of this application will become clearer with reference to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings: Figure 1 This is a schematic diagram of a common transmission path in existing technologies; Figure 2 A flowchart of the decision logic of the topology-aware scheduling unit according to some embodiments of this application; Figure 3 A schematic diagram of the architecture of a heterogeneous GPU data transmission system is shown in some embodiments of this application; Figure 4 A flowchart of a heterogeneous GPU data transmission method according to some embodiments of this application; Figure 5 A flowchart of a heterogeneous GPU data transmission method is shown in some embodiments of this application. Detailed Implementation
[0020] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this application are one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.
[0021] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.
[0022] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.
[0023] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.
[0024] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.
[0025] In computing clusters with extremely high real-time requirements, such as those used in distributed robotics and autonomous driving, processing high-frequency sensor data streams and complex model inference necessitates integrating computing power from graphics processing units (GPUs, collectively referred to as "heterogeneous GPUs") from different manufacturers (such as NVIDIA, AMD, and Huawei Ascend) or with different architectures. A core prerequisite for achieving efficient computing power collaboration is the ability to perform efficient and transparent data transmission between these heterogeneous GPUs. However, existing technological solutions have long faced systemic shortcomings in achieving this goal.
[0026] To overcome the fundamental barrier of heterogeneous GPUs being unable to directly interoperate due to a closed hardware ecosystem, the industry generally adopts a common software transmission path via host (CPU) memory, such as... Figure 1 (A schematic diagram of a general transmission path is shown). This scheme provides fragmented communication interfaces for different physical deployment topologies: intra-machine communication requires calling the inter-process communication (IPC) interface, while cross-machine communication requires calling interfaces based on network protocols such as TCP / IP. Because this scheme lacks a unified, location-transparent abstraction layer, application developers must explicitly select and call the corresponding underlying interfaces based on the physical deployment relationship between the communicating parties. This directly leads to high development complexity and forces business logic to be deeply coupled with the hardware topology.
[0027] More seriously, this general path itself suffers from severe performance defects. For example... Figure 1 As shown, data needs to undergo multiple cross-address space copies: "sending GPU memory → host memory → (IPC / network channel) → host memory → receiving GPU memory". Simultaneously, to adapt to network transmission, high-overhead serialization and deserialization operations are typically performed on the CPU. These redundant data copies and high-overhead format conversions result in significant performance overhead, making them a bottleneck for real-time system processing.
[0028] To systematically address the triple challenges of hardware incompatibility, high performance overhead, and high development complexity, this application proposes a method and system for optimizing heterogeneous GPU data transmission based on Apache Arrow. First, this application utilizes a cross-platform tensor encapsulation tool library to parse and extract features from raw tensor data from heterogeneous GPUs, and then uniformly converts (encapsulates) it into a standardized Apache Arrow columnar memory format, thereby overcoming the hardware compatibility barrier at the software level.
[0029] Furthermore, this application uses a topology-aware scheduling unit to dynamically sense the physical location (same machine / cross machine) of communication nodes and select the optimal transmission path for the encapsulated Arrow format data object. The core decision-making logic of this topology-aware scheduling unit is as follows: Figure 2(The topology-aware scheduling unit decision logic flowchart is shown below): It first determines whether the two communicating parties are located on the same physical host (i.e., "same machine?"). If the determination is yes (Y), it triggers zero-copy transmission on the same machine, completely eliminating memory copying through technologies such as shared memory; if the determination is no (N), it triggers cross-machine network transmission based on high-performance protocols such as Zenoh, and uses the Arrow format to achieve non-serialization. Through "unified data abstraction" and such... Figure 2 The close integration of "intelligent path scheduling" shown in this application enables high-performance data transmission while providing complete location transparency for upper-layer applications, thereby significantly reducing the difficulty of developing and maintaining distributed systems.
[0030] The heterogeneous GPU data transmission system of this application will be described in detail below with reference to the accompanying drawings.
[0031] Figure 3 This is a schematic diagram of a heterogeneous GPU data transmission system architecture according to some embodiments of this application. For example... Figure 3 As shown, the system consists of three core parts: a transmitting node, a topology-aware scheduling unit, and a receiving node. Wherein: 1. Sending node: like Figure 3 As shown on the left, the sending node is the initiator of data transmission. It has the first GPU and is configured to call the cross-platform tensor wrapping tool library.
[0032] First GPU: It is the source of the tensor data to be transmitted. The "heterogeneous GPU" scenario addressed in this application specifically refers to the situation where the first GPU and the second GPU at the receiving end may come from different manufacturers (such as NVIDIA, AMD, Huawei Ascend) or adopt different computing architectures.
[0033] Cross-platform tensor wrapper tool library: The cross-platform tensor wrapping tool library provides a standard tensor wrapping interface. In some embodiments of this application, the tool library, as a software component, can be independently integrated into the local runtime environments of the sending node and the receiving node, respectively, in the form of a dynamic link library, static dependency, or plugin. The sending node is configured to call the local tool library to wrap tensor data from the first GPU into memory objects conforming to the Apache Arrow format.
[0034] The core of this application lies in using a general-purpose tensor description toolkit to decouple and map hardware-private tensor data structures into a columnar layout conforming to the Apache Arrow specification. Apache Arrow is a columnar in-memory data format whose "store-as-transfer" and "zero-copy" design principles allow data to be used for computation or network transmission without conversion, which is the cornerstone of achieving high-performance transmission.
[0035] Taking the Ascend CANN domestic heterogeneous hardware scenario as an example, the tool library directly accesses tensor objects in the video memory by calling the underlying CANN (Compute Architecture for Neural Networks) interface. First, it extracts the underlying memory pointer, memory stride, tensor shape, and data type of the tensor, and then constructs an "Arrow Buffer" descriptor in memory.
[0036] The tool library logically maps heterogeneous memory blocks to contiguous memory spaces for Arrow by calculating hardware-specific alignment requirements. For non-contiguous tensors (such as image data that has undergone padding), the tool library automatically calculates their logical offsets and converts them into Arrow RecordBatch schema descriptions. In this way, tensors that were originally proprietary within the closed ecosystem of domestic GPUs are transformed into a widely applicable form through Arrow tensor descriptors.
[0037] Meanwhile, to avoid fragmentation caused by frequent memory allocation and deallocation, the tool library includes built-in reference counting management for heterogeneous memory. When a tensor is encapsulated in Arrow format, the underlying memory handle is locked until all communication nodes have finished reading, ensuring data integrity during asynchronous transmission.
[0038] 2. Topology-aware scheduling unit Figure 3 The topology-aware scheduling unit shown in the central section is the system's intelligent decision-making center, maintaining communication with both the sending and receiving nodes. Its decision-making logic is as follows: Figure 2 (The flowchart of the decision-making logic of the topology-aware scheduling unit is shown.) The specific working method is as follows: Once the data is ready, the topology-aware scheduling unit will intervene and dynamically select the optimal path based on the physical locations of the communicating parties. The entire process is completely transparent to the upper-layer application code.
[0039] In some embodiments of this application, the system further includes a global registry center for storing and managing node topology information. On one hand, during the startup phase, the system completes the registration of all node information through the global registry center, recording the unique characteristic information of each node, including but not limited to: Machine ID, Process ID (PID), and its associated GPU device serial number. On the other hand, when node A attempts to send a tensor to node B, the topology-aware scheduling unit executes the following judgment logic: Same process path: If the PIDs are the same, the pointer is passed directly through the function call, achieving zero overhead in an absolute sense.
[0040] Intra-process communication (IPC) path: If the Machine IDs are the same but the PIDs are different, the system determines that it is a single machine with multiple cards or a single machine with a heterogeneous environment, and automatically triggers zero-copy transfer based on shared memory.
[0041] Cross-machine path (Network): If the Machine ID is different, a network transmission path based on a high-performance communication protocol (such as Zenoh) is triggered. At the same time, the storage-as-transmission feature of Arrow format is utilized to eliminate serialization / deserialization overhead, thereby optimizing the data flow.
[0042] 3. Receiver node like Figure 3 As shown on the right, the receiving node is the destination for data transmission and includes a second GPU. The first GPU and the second GPU are heterogeneous GPUs.
[0043] The receiving node is configured to call the local cross-platform tensor wrapping tool library to load the data in the received Apache Arrow format memory object into the video memory address space of the second GPU.
[0044] The above three core components are passed through Figure 3 The architecture shown enables connectivity and collaboration. Data transmission paths within the system are dynamically determined by the topology-aware scheduling unit, primarily including intra-machine zero-copy transmission paths and cross-machine network transmission paths.
[0045] Same-machine zero-copy transfer path: In scenarios where the sender and receiver are on the same physical host, the system significantly reduces the user-space copy overhead common in traditional middleware through memory mapping technology.
[0046] First, when the system detects that communication is occurring between local processes, it allocates a shared memory segment in the system kernel aligned with the size of the Arrow data block. The system directly mounts the Arrow memory object to be sent to this shared segment without any format conversion. Second, the sender transmits only the key of this shared memory and Arrow schema metadata to the receiver via a lightweight IPC channel (such as a Unix Domain Socket or pipe). After obtaining the key, the receiver directly maps the shared memory segment to its own process address space through system calls such as mmap (Memory Mapping). Since the data itself conforms to the Arrow standard layout, the receiver does not need to perform deserialization or format parsing, achieving "in-place access" to the data, thereby compressing the latency of intra-machine communication to the nanosecond level in typical scenarios.
[0047] Cross-machine network transmission path: In cross-machine transmission scenarios, this application solves the performance bottleneck caused by CPU serialization in traditional frameworks (such as Protobuf / JSON in ROS), and realizes "storage as transmission".
[0048] Traditional communication frameworks require the CPU to parse and encode each field of an object before sending it. In this application, the encapsulated Arrow tensor is already in a format that can be directly sent over the network in memory. The communication engine uses zero-copy sending technology to directly send the Arrow Buffer in memory through the network card. Existing transport layer protocols are deeply aligned with the Arrow format, only adding descriptive information such as offset and length to the network packet header. This design allows the data read from video memory or main memory by the network card to be directly in a computable format, greatly reducing the CPU usage of the sending end. Furthermore, after receiving the network packet, the receiver can directly map the received message to an Arrow object and hand it over to the upper-layer node for processing. This approach skips the complex process of "receiving byte stream -> constructing object -> filling data".
[0049] The static architecture, component functions, and core interaction principles of the heterogeneous GPU data transmission system of this application have been described in detail above with reference to the accompanying drawings. Based on this system, this application also provides a corresponding data transmission method. The specific steps and dynamic flow of the method will be described in detail below from the execution perspectives of the sending and receiving nodes, with reference to the accompanying drawings.
[0050] Figure 4 This is a flowchart illustrating a heterogeneous GPU data transmission method according to some embodiments of this application. The method is... Figure 3 The sending node in the system executes this. For example... Figure 4 As shown, the method includes: S410, the tensor data in the first GPU contained in the sending node is encapsulated into a memory object conforming to the Apache Arrow format.
[0051] This step is performed by the sending node calling a cross-platform tensor wrapping tool library. This tool library, as a software component, is integrated into the sending node's local runtime environment, and the sending node is configured to call this tool library to complete the wrapping.
[0052] S420, obtain the location information of the receiving node.
[0053] In some embodiments of this application, this step involves the sending node obtaining the identifier of the target receiving node. The identifier is a unique identifier for the receiving node within the system (e.g., node name, ID, etc.). The sending node can determine this identifier through application-layer task configuration or other methods. In some embodiments of this application, since a global registry center is maintained in the system, which records the mapping relationship between each node identifier and its detailed characteristic information (including machine identifier, process identifier, etc.), knowing the receiving node identifier means that its location information can be determined at the system level.
[0054] S430, Based on the location information, obtain the transmission path, wherein the transmission path is determined by the topology-aware scheduling unit in the system based on the location information, including a same-machine zero-copy transmission path or a cross-machine network transmission path.
[0055] After obtaining the receiver node's identifier, the sending node initiates a path request to the topology-aware scheduling unit, carrying the receiver node's identifier in the request. Upon receiving the request, the topology-aware scheduling unit queries the global registry center to obtain the receiver node's detailed location information based on the identifier, and then executes actions based on this location information. Figure 3 The decision logic shown determines the final transmission path (zero-copy within the same machine or cross-machine network), and then returns this path decision result to the sending node. Thus, the sending node obtains the transmission path.
[0056] S440, the memory object conforming to the Apache Arrow format is sent to the receiving node through the obtained transmission path, so as to load the data therein into the second GPU contained in the receiving node, wherein the first GPU and the second GPU are heterogeneous.
[0057] The sending node executes the final data transmission based on the path determined by S430: If the path is a zero-copy transfer path within the same machine, then zero-copy data access between processes is achieved by mounting the Arrow memory object to the shared memory segment and passing only its identifier.
[0058] If the path is a cross-machine network transmission path, then based on the Zenoh protocol, the encapsulated Arrow memory object is directly transmitted over the network without serialization.
[0059] Figure 5 This is a flowchart illustrating a heterogeneous GPU data transmission method according to some embodiments of this application. The method is... Figure 3 The receiving node in the system executes this. For example... Figure 5 As shown, the method includes: S510, receiving a memory object conforming to the Apache Arrow format from a sending node, wherein the memory object is encapsulated from tensor data in a first GPU contained in the sending node.
[0060] The receiving node receives data from the sending node according to the transmission path determined by the topology-aware scheduling unit: If the path is a zero-copy transmission path on the same machine, the receiving node obtains the identifier (key) of the shared memory segment through inter-process communication (IPC) and directly accesses the Arrow format memory object in the shared memory segment through memory mapping (such as mmap).
[0061] If the path is a cross-machine network transmission path, the receiving node receives the network data stream based on the Zenoh communication protocol and directly parses and maps it into a memory object in Arrow format.
[0062] S520, using the cross-platform tensor encapsulation tool library, the data in the memory object is loaded into the second GPU included in the receiving node, wherein the first GPU and the second GPU are heterogeneous.
[0063] The receiving node is configured to call a local cross-platform tensor wrapping tool library. This tool library obtains the data layout, type, and address information by parsing the memory descriptor in the Arrow format memory object received by the S510. Subsequently, through efficient memory operations (e.g., directly mapping to the address space of the second GPU via memory mapping), the data is loaded into the local second GPU memory, completing a full heterogeneous GPU data transfer.
[0064] In summary, the heterogeneous GPU data transmission system and method proposed in the embodiments of this application, by constructing a unified tensor abstraction layer based on the Apache Arrow format, losslessly maps the private tensor data structures of various heterogeneous GPUs into standardized memory objects, achieving a unified cross-platform description of data. This breaks down the data format barriers caused by the closed hardware ecosystem and fundamentally solves the problem of heterogeneous GPUs being unable to directly interoperate. By introducing a topology-aware scheduling unit, which adaptively selects either "zero-copy within the same machine" or "cross-machine network" transmission paths based on the physical location of nodes, this mechanism ensures that data is always transmitted via the optimal path: in the same-machine scenario, shared memory and other technologies are used to avoid redundant copies via host memory; in the cross-machine scenario, the "store-as-transfer" characteristic of the Apache Arrow format eliminates the need for CPU serialization / deserialization. Thus, the overhead of multiple data copies and format conversions in existing general transmission paths is completely eliminated, significantly improving transmission efficiency and solving the problem of high performance overhead. Through the synergy of the aforementioned "unified abstraction" and "intelligent scheduling," a location-transparent communication interface is provided for upper-layer applications. Application developers only need to perform unified send / receive calls, without needing to be aware of or adapt to the complex underlying hardware topology and network differences. This achieves complete decoupling of business logic and physical deployment, greatly reducing the complexity of system development, maintenance, and migration, thus solving the problem of high development complexity. Therefore, this application, through the organic combination of the above technical features, forms a complete high-performance heterogeneous GPU data transmission infrastructure.
[0065] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.
[0066] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that other implementations can also be executed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.
[0067] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0068] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.
Claims
1. A heterogeneous GPU data transmission system, comprising a sending node, a receiving node, and a topology-aware scheduling unit, characterized in that, include: The sending node has a first GPU and is configured to call a cross-platform tensor encapsulation tool library to encapsulate tensor data from the first GPU into memory objects conforming to the Apache Arrow format. The topology-aware scheduling unit is used to sense the physical location relationship between the sending node and the receiving node, and based on the location relationship, select a zero-copy transmission path within the same machine or a cross-machine network transmission path for the encapsulated memory object. The sending node is also used to send the memory object conforming to the Apache Arrow format to the receiving node through the path selected by the topology-aware scheduling unit; The receiving node has a second GPU and is configured to call the cross-platform tensor wrapping tool library to load data from the memory object into the second GPU; The first GPU and the second GPU are heterogeneous.
2. The system according to claim 1, characterized in that, The cross-platform tensor wrapper library wraps tensor data into in-memory objects conforming to the Apache Arrow format, including: The cross-platform tensor encapsulation tool library extracts the memory pointer, span, dimension, and data type of the tensor; Based on the extracted information, the cross-platform tensor wrapper library constructs memory descriptors in Apache Arrow format.
3. The system according to claim 1, characterized in that, The topology-aware scheduling unit selects a transmission path based on the physical location relationship, including: If the process identifiers of the sending node and the receiving node are the same, the topology-aware scheduling unit determines that the transmission path is intra-process pointer passing. If the machine identifiers of the sending node and the receiving node are the same but their process identifiers are different, then the topology-aware scheduling unit determines that the transmission path is a zero-copy transmission path on the same machine. If the machine identifiers of the sending node and the receiving node are different, the topology-aware scheduling unit determines the transmission path as a cross-machine network transmission path.
4. The system according to claim 1, characterized in that: When the selected transmission path is a zero-copy transmission path on the same machine, the sending node will mount the memory object conforming to the ApacheArrow format to the shared memory segment; The identifier of the shared memory segment is transmitted to the receiving node via inter-process communication.
5. The system according to claim 1, characterized in that: When the selected transmission path is a cross-machine network transmission path, the sending node transmits the memory object conforming to the Apache Arrow format based on the Zenoh communication protocol, and there is no need to perform serialization operation on the memory object during the transmission process.
6. A heterogeneous GPU data transmission method, characterized in that, Performed by the sending node in the system of claim 1, including: The tensor data in the first GPU contained in the sending node is encapsulated into a memory object conforming to the Apache Arrow format; Obtain the location information of the receiving node; Based on the location information, a transmission path is obtained, wherein the transmission path is determined by the topology-aware scheduling unit in the system based on the location information, including a zero-copy transmission path within the same machine or a cross-machine network transmission path. The memory object conforming to the Apache Arrow format is sent to the receiving node via the obtained transmission path, so that the data therein can be loaded into the second GPU contained in the receiving node. The first GPU and the second GPU are heterogeneous.
7. The method according to claim 6, characterized in that, The step of obtaining the location information of the receiving node includes: The location information is obtained by querying a global registry center, which records the machine identifier, process identifier, and GPU device information of the nodes in the system.
8. The method according to claim 6, characterized in that, The step of encapsulating the tensor data in the first GPU contained in the sending node into a memory object conforming to the Apache Arrow format includes: When the tensor data is stored non-contiguously, calculate the logical offset of the tensor data; The tensor data and its logical offsets are described in the Apache Arrow RecordBatch pattern.
9. A heterogeneous GPU data transmission method, characterized in that, Performed by the receiving node in the system of claim 1, including: Receive a memory object conforming to the Apache Arrow format from the sending node, wherein the memory object is encapsulated from tensor data in a first GPU contained in the sending node; Using the cross-platform tensor wrapper library, the data in the memory object is loaded into the second GPU contained in the receiving node; The first GPU and the second GPU are heterogeneous.
10. The method according to claim 9, characterized in that, The step of loading data from the memory object into the second GPU using the cross-platform tensor wrapping tool library includes: The cross-platform tensor encapsulation tool library is used to parse the memory descriptor in the memory object conforming to the Apache Arrow format, and the memory is directly mapped to the address space of the second GPU for access via memory mapping.