RDMA-based end-to-end multi-core data direct connection method

Abstract

The invention relates to an end-to-end multi-core data direct connection method based on RDMA (Remote Direct Memory Access), which comprises the following steps: 1) an equipment enumeration step: identifying and acquiring type information and a physical DMA memory of each heterogeneous chip; 2) a chip memory binding and virtual mapping step: binding a DMA memory with a file descriptor, and constructing a unified virtual address space to realize cross-chip memory pooling management; (3) a data direct connection step: binding a file descriptor with an RDMA memory registration object, establishing an end-to-end data transmission channel, and finishing high-speed data transmission under the condition of no CPU (Central Processing Unit) participation; and 4) a resource release step: uniformly recovering equipment resources and updating state information after the task is completed. By taking the data processing unit as a core, the unified memory view is constructed, and cross-chip RDMA straight-through transmission is realized, so that CPU intervention and host memory overhead are remarkably reduced, and the overall performance of the system is improved.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, and in particular relates to an end-to-end multi-core data pass-through method based on RDMA, which is suitable for application scenarios with strict requirements for high throughput and low latency communication, such as intelligent computing, edge AI processing, and heterogeneous acceleration of data centers. Background Technology

[0002] With the rapid development of information technologies such as large-scale models, artificial intelligence (AI), and big data, the intelligence level of airborne systems is undergoing profound changes, and the new generation of aviation equipment is evolving towards a higher degree of autonomous decision-making and real-time perception. The widespread application of artificial intelligence algorithms in key tasks such as target recognition, flight control, and situational awareness has placed unprecedentedly high demands on the performance, efficiency, and real-time performance of computing platforms.

[0003] Currently, most aircraft in service employ a CPU-centric computing approach, with CPU+DSP and CPU+FPGA configurations used to expand computing power for specific tasks. However, this traditional architecture faces significant bottlenecks when dealing with large-scale intelligent tasks: on the one hand, frequent data copying and transmission between multiple chips significantly increases system latency, making it difficult to meet the response requirements of high-real-time tasks; on the other hand, when handling large-scale model inference and training tasks with rapidly increasing parameter sizes, existing platforms are limited by bandwidth and data scheduling efficiency, resulting in severe data transmission bottlenecks that severely restrict the deployment efficiency and application expansion capabilities of intelligent algorithms in airborne environments. Summary of the Invention

[0004] The purpose of this invention is to provide an end-to-end multi-chip data pass-through method based on RDMA to solve the problems of high latency, limited bandwidth, and low resource utilization in data transmission between heterogeneous chips in existing airborne and edge computing systems. This invention, by building a unified memory view around a Data Processing Unit (DPU) and realizing cross-chip RDMA pass-through, significantly reduces CPU intervention and host memory overhead, thereby improving overall system performance.

[0005] The technical solution of this invention is as follows: This invention is an end-to-end multi-core data pass-through method based on RDMA, and its special feature is that the method includes the following steps:

[0006] 1) Device enumeration step: Identify and obtain the type information of each heterogeneous chip and its physical DMA memory;

[0007] 2) Chip memory binding and virtual mapping steps: bind DMA memory to file descriptors and build a unified virtual address space to achieve cross-chip memory pooling management;

[0008] 3) Data pass-through step: By binding the file descriptor to the RDMA memory registration object, an end-to-end data transmission channel is established to complete high-speed data transmission without CPU involvement;

[0009] 4) Resource release step: After the task is completed, the equipment resources are uniformly recycled and the status information is updated.

[0010] Further, the specific steps of step 1) are as follows: During the system initialization phase, all heterogeneous computing devices that support data pass-through capability are enumerated to identify the hardware type, physical DMA memory base address and size, and memory access attributes of each chip, which serve as the basic resource input for subsequent data transmission path establishment and memory scheduling strategies.

[0011] Furthermore, the specific steps of step 2) are as follows:

[0012] 2.1) Allocate consistent DMA memory regions on each heterogeneous computing chip and encapsulate them into file descriptors that can be shared across devices through the dma_buf mechanism to support transparent transfer and reuse of memory objects;

[0013] 2.2) By combining the IOMMU and page table management mechanisms, DMA memory regions distributed across different chips are mapped to a globally unified virtual address space, constructing a logically consistent and physically distributed cross-device memory view, thus shielding the underlying physical address differences.

[0014] Furthermore, the specific steps of step 3) are as follows:

[0015] 3.1) Bind the file descriptor to the memory registration object in the RDMA communication mechanism, and call the RDMA driver interface to establish the physical address mapping relationship of the memory region in the RDMA context;

[0016] 3.2) Based on the RoCE V2 protocol stack, initiate remote memory write or read operations to complete high-speed data transmission across nodes or devices without relying on CPU participation, and realize end-to-end zero-copy data pass-through between source devices and target devices.

[0017] Furthermore, the specific steps of step 4) are as follows: After the data processing task is completed, the resources related to the heterogeneous devices used are uniformly reclaimed, including unbinding the file descriptor from the memory registration object, canceling the virtual address mapping, releasing the DMA memory, and updating the device status information so that the subsequent task scheduling module can perform dynamic resource allocation and load balancing.

[0018] This invention provides an end-to-end multi-chip data pass-through method based on RDMA, aiming to solve the problems of high data transmission latency, prominent bandwidth bottlenecks, and low utilization of heterogeneous resources in existing computing systems when processing large-scale artificial intelligence and big data tasks. This method uses a DPU as its core and achieves efficient data pass-through and memory sharing between heterogeneous computing chips such as GPUs, NPUs, and FPGAs through a hardware-software collaborative mechanism. This invention effectively reduces host memory overhead, improves the scheduling efficiency of heterogeneous computing resources and the overall system performance, and is suitable for the high-performance, low-latency computing requirements of airborne intelligent task scenarios. Attached Figure Description

[0019] Figure 1 This is a flowchart of the RDMA transmit queue management based on the hardware blind read mechanism of the present invention. Detailed Implementation

[0020] The end-to-end multi-core data pass-through method based on RDMA provided by this invention includes the following steps:

[0021] 1) Device enumeration step: Identify and obtain the type information of each heterogeneous chip and its physical DMA memory;

[0022] 2) Chip memory binding and virtual mapping steps: bind DMA memory to file descriptors and build a unified virtual address space to achieve cross-chip memory pooling management;

[0023] 3) Data pass-through step: By binding the file descriptor to the RDMA memory registration object, an end-to-end data transmission channel is established to complete high-speed data transmission without CPU involvement;

[0024] 4) Resource release step: After the task is completed, the equipment resources are uniformly recycled and the status information is updated.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0026] See Figure 1 The end-to-end multi-core data pass-through method based on RDMA provided by this invention includes the following steps in a specific embodiment:

[0027] 1) Device enumeration steps: During the system initialization phase, enumerate all heterogeneous computing devices that support data pass-through capabilities, identify the hardware type, physical DMA memory base address and size, and memory access attributes of each chip, and use them as the basic resource input for subsequent data transmission path establishment and memory scheduling strategies.

[0028] 2) Unified Virtual Mapping Steps: Enable the DPU to directly access the DMA memory space on the target heterogeneous chip through file descriptors, specifically including the following sub-steps:

[0029] 2.1) Allocate consistent DMA memory regions on each heterogeneous computing chip and encapsulate them into file descriptors that can be shared across devices through the dma_buf mechanism to support transparent transfer and reuse of memory objects;

[0030] 2.2) By combining the IOMMU and page table management mechanisms, DMA memory regions distributed across different chips are mapped to a globally unified virtual address space, constructing a logically consistent and physically distributed cross-device memory view, thus shielding the underlying physical address differences.

[0031] 3) Data pass-through step: Implement an end-to-end data pass-through process without CPU intervention, specifically including the following sub-steps:

[0032] 3.1) Bind the file descriptor to the memory registration object (Memory Region, MR) in the RDMA communication mechanism, and call the RDMA driver interface to establish the physical address mapping relationship of the memory region in the RDMA context;

[0033] 3.2) Based on the RoCE V2 protocol stack, initiate remote memory write (RDMA Write) or read (RDMA Read) operations to complete high-speed data transmission across nodes or devices without relying on CPU participation, and realize end-to-end zero-copy data pass-through between source devices and target devices.

[0034] 4) Resource release steps: After the data processing task is completed, the resources related to the heterogeneous devices used are uniformly reclaimed, including unbinding the file descriptor from the memory registration object, canceling the virtual address mapping, releasing the DMA memory, and updating the device status information so that the subsequent task scheduling module can perform dynamic resource allocation and load balancing.

[0035] In a specific application embodiment of the present invention, the application system includes heterogeneous computing units such as DPU, GPU, NPU, and FPGA, all of which are connected to the CPU via PCIe links. The DPU runs a Linux operating system and a custom memory management driver, and is responsible for performing device enumeration, memory virtualization, and RDMA channel management.

[0036] After the application system starts, the DPU executes step 1), identifying each device and obtaining its DMA memory information via lspci and ioctl calls. Then, it executes step 2), mapping each device's memory to file descriptors using mmap and calling ibv_reg_mr() from the rdma-core library to complete memory registration and build a unified virtual address space. Next, it executes step 3), where when a remote compute node needs to transfer data to the local compute node, the DPU establishes an RDMA write / read channel between the remote compute node's output buffer and the local compute node's input buffer, triggering ibv_post_send() to send a WR(WorkRequest). The data is directly written to the NPU memory via the RDMA engine, without CPU intervention. Finally, it executes step 4), where, after the task is completed, the DPU calls ibv_dereg_mr() to release the MR object and update the resource pool state.

[0037] The above are merely specific embodiments disclosed in this invention, but the scope of protection disclosed in this invention is not limited thereto. The scope of protection disclosed in this invention should be determined by the scope of the claims.

[0038] The technical contents of this invention and those not specifically described in the above embodiments are the same as those in the prior art.

[0039] The present invention is not limited to the above embodiments; all embodiments described herein can be implemented and have the aforementioned good effects.

Claims

1. An end-to-end multi-core data pass-through method based on RDMA, characterized in that: The method includes the following steps: 1) Device enumeration step: Identify and obtain the type information of each heterogeneous chip and its physical DMA memory; 2) Chip memory binding and virtual mapping steps: bind DMA memory to file descriptors and build a unified virtual address space to achieve cross-chip memory pooling management; 3) Data pass-through step: By binding the file descriptor to the RDMA memory registration object, an end-to-end data transmission channel is established to complete high-speed data transmission without CPU involvement; 4) Resource release step: After the task is completed, the equipment resources are uniformly recycled and the status information is updated.

2. The end-to-end multi-core data pass-through method based on RDMA according to claim 1, characterized in that: The specific steps of step 1) are as follows: During the system initialization phase, all heterogeneous computing devices that support data pass-through capability are enumerated to identify the hardware type, physical DMA memory base address and size, and memory access attributes of each chip, which serve as the basic resource input for subsequent data transmission path establishment and memory scheduling strategies.

3. The end-to-end multi-core data pass-through method based on RDMA according to claim 2, characterized in that: The specific steps of step 2) are as follows: 2.1) Allocate consistent DMA memory regions on each heterogeneous computing chip and encapsulate them into file descriptors that can be shared across devices through the dma_buf mechanism to support transparent transfer and reuse of memory objects; 2.2) By combining the IOMMU and page table management mechanisms, DMA memory regions distributed across different chips are mapped to a globally unified virtual address space, constructing a logically consistent and physically distributed cross-device memory view, thus shielding the underlying physical address differences.

4. The end-to-end multi-core data pass-through method based on RDMA according to claim 3, characterized in that: The specific steps of step 3) are as follows: 3.1) Bind the file descriptor to the memory registration object in the RDMA communication mechanism, and call the RDMA driver interface to establish the physical address mapping relationship of the memory region in the RDMA context; 3.2) Based on the RoCE V2 protocol stack, initiate remote memory write or read operations to complete high-speed data transmission across nodes or devices without relying on CPU participation, and realize end-to-end zero-copy data pass-through between source devices and target devices.

5. The end-to-end multi-core data pass-through method based on RDMA according to claim 4, characterized in that: The specific steps of step 4) are as follows: After the data processing task is completed, the resources related to the heterogeneous devices used are uniformly reclaimed, including unbinding the file descriptor from the memory registration object, canceling the virtual address mapping, releasing the DMA memory, and updating the device status information so that the subsequent task scheduling module can perform dynamic resource allocation and load balancing.

Images