A full-stack deep observable auditing method for RDMA high-performance network traffic

Abstract

The application provides a full-stack deep observable auditing method for RDMA high-performance network flow, belongs to the technical field of RDMA network monitoring, and adds a layer of LD_PRELOAD hijacking library on the basis of the original call logic framework of RDMA application, uses the LD_PRELOAD hijacking library to hijack the function call of the user-mode driver to obtain the RDMA resource identifier and operation content, generates auditing data containing resource-level granularity according to the RDMA resource identifier and operation content, and the original call logic framework of the RDMA application includes a user-mode application, a user-mode driver, a kernel-mode software stack and hardware, and the LD_PRELOAD hijacking library is arranged between the user-mode application and the user-mode driver. The application can audit the RDMA network flow at the function level and the resource level (QP / CQ / WR), which is far higher than the level of traditional network equipment that can only see IP / UDP flow.

Description

Technical Field

[0001] This invention relates to RDMA network traffic auditing, and more particularly to a full-stack, deep observable auditing method for high-performance RDMA network traffic, belonging to the field of RDMA network monitoring technology. Background Technology

[0002] With the increasing prevalence of RDMA in data centers, high-performance computing, and AI training, the need for monitoring, auditing, and troubleshooting its traffic is becoming increasingly urgent. However, the traditional bypass characteristic of RDMA makes it a "blind spot" in the network, rendering existing network monitoring tools (such as kernel-based packet capture tools) ineffective. Therefore, a method for effectively monitoring and auditing RDMA traffic is urgently needed.

[0003] Currently, the industry's traditional methods for traffic auditing are through switch port mirroring + Roce packet parsing, or through EBPF kernel tracing. However, these methods all have different drawbacks, which will be detailed below.

[0004] Traditional Method 1: Switch Port Mirroring (SPAN) + RoCE Packet Parsing

[0005] Method: Mirror RoCE traffic to an analysis server equipped with parsing tools using the port mirroring function of the data center switch.

[0006] Disadvantages:

[0007] 1) It is impossible to associate application layer semantics. Only network layer RoCE packets (traditional IP, port, etc.) can be seen. It is impossible to know which QP, which WR, or which Cmid corresponds to this packet. The IBverbs function call sequence of the application layer cannot be seen.

[0008] 2) Requires special switch support and has complex configuration;

[0009] 3) Real-time auditing performance will be slightly worse because it requires copying traffic, and there is a copying operation for the traffic to be audited in the middle.

[0010] 4) Encrypted traffic cannot be parsed. If RoCEv2 IPsec encryption is used, it is completely unanalyzable.

[0011] Traditional Method 2: Kernel tracepoints (such as BPF / eBPF)

[0012] Method: Hooking kernel system functions via eBPF

[0013] Disadvantages:

[0014] 1) Implementation relies on the kernel's RDMA subsystem; its versatility may be affected by different Linux distributions. A relatively new kernel version (typically >= 4.4) is required, and BPF support needs to be enabled. Production environments may require additional configuration (such as setting kernel parameters).

[0015] 2) For RDMA devices that are entirely driven in user space (such as some user-space driven network cards), there are not enough trace points at the kernel level;

[0016] 3) Each function call requires entering the kernel, executing the eBPF program, and then returning to user space. This can introduce significant performance overhead, especially for frequently called functions (such as ibv_post_send).

[0017] 4) eBPF programs run in kernel space and require privileged permissions (root or CAP_BPF, etc.). eBPF programs must pass the kernel verifier's check to ensure they do not compromise system stability;

[0018] 5) Writing eBPF programs requires understanding kernel programming constraints (such as finite loops, stack size, etc.) and usually requires cooperating with user-space loaders.

[0019] The following section introduces the technical background and traffic auditing background of RDMA.

[0020] (1) RDMA technology:

[0021] like Figure 1 As shown, RDMA (Remote Direct Memory Access) allows a local node to directly access the memory of a remote node. Due to its high throughput and low latency, RDMA has become a key network technology for data centers and high-performance computing clusters. The term "direct" means that remote memory can be read and written as if it were local memory, bypassing the complex TCP / IP network protocol stack of traditional Ethernet. This process is invisible to the remote node, and most of the work in this read / write process is done by hardware rather than software.

[0022] The core technology model of RDMA is based on abstract objects such as Queue Pair (QP), Work Request (WR), and Completion Queue (CQ):

[0023] Queue Pair (QP): Each communication connection is represented by a queue pair consisting of a send queue (SQ) and a receive queue (RQ). The application sends and receives data by submitting a WR to the QP.

[0024] Work request (WR): Describes a specific operation (such as sending or receiving) that the application wants to perform.

[0025] Completion Queue (CQ): Used to notify the application whether its submitted WR has been completed.

[0026] Currently, RDMA is mainly implemented through three physical networks: InfiniBand, RoCE, and iWARP. At the software level, industry standards provide a unified programming interface (Verbs interface) through user-space libraries such as libibverbs and librdmacm, enabling applications to easily invoke RDMA functions.

[0027] Thanks to its superior performance, RDMA has been widely used in core scenarios such as high-performance computing, hyperconverged infrastructure, distributed storage systems (such as Ceph, NVMe-oF) and artificial intelligence training clusters, becoming an indispensable network foundation for modern data centers.

[0028] libibverbs is a user-space library for the InfiniBand verb interface, providing functionality for creating and managing low-level resources such as queue pairs (QPs), protection domains (PDs), and completion queues (CQs). `rdma_create_qp` is a function in libibverbs used to create a queue pair (QP).

[0029] librdmacm (RDMA CM interface) is a library for connection management, providing advanced functions such as connection establishment, address resolution, and event handling. It is typically used in conjunction with libibverbs, but the two belong to different interface layers.

[0030] In RDMA programming, the typical workflow is as follows:

[0031] Use libibverbs to create resources (such as PD, CQ, QP, etc.).

[0032] Use librdmacm to establish connections (such as resolving addresses, establishing connections, listening, etc.).

[0033] Then libibverbs is used for data transfer.

[0034] (2) The significance of RDMA traffic auditing:

[0035] Despite its superior performance, RDMA's "kernel bypass" design philosophy, while delivering high performance, also presents significant "observability" challenges. Traditional network monitoring tools (such as tcpdump and Wireshark) operate on the operating system kernel protocol stack, while RDMA traffic completely bypasses the kernel protocol stack, rendering these tools ineffective for RDMA traffic and turning RDMA networks into a "black box" or "blind spot."

[0036] This lack of observability poses a serious challenge to the operation, troubleshooting, and security of RDMA networks:

[0037] Troubleshooting is difficult: When RDMA networks experience performance jitter, packet loss, or communication interruptions, maintenance personnel lack effective tools to pinpoint the root cause of the problem. They cannot easily capture and analyze packets to determine whether the problem originates in the network link, the switch, or the application itself, as they can with TCP / IP traffic. Summary of the Invention

[0038] To address the aforementioned issues, this invention provides a full-stack, deep observable auditing method for high-performance RDMA network traffic, enabling finer-grained auditing of RDMA network traffic.

[0039] To achieve the above objectives, the technical solution of the present invention is as follows: a full-stack depth observable auditing method for high-performance RDMA network traffic, which adds an LD_PRELOAD hijacking library to the original calling logic framework of RDMA applications, hijacks user-mode driven function calls using the LD_PRELOAD hijacking library to obtain RDMA resource identifiers and operation content, and generates audit data containing resource-level granularity based on the RDMA resource identifiers and operation content;

[0040] The original calling logic framework of RDMA applications includes user-mode applications, user-mode drivers, kernel-mode software stacks and hardware, with the LD_PRELOAD hijacking library deployed between the user-mode applications and user-mode drivers.

[0041] Furthermore, the LD_PRELOAD hijacking library contains a generic user-mode driver interface.

[0042] Furthermore, the user-space driver includes the librdmacm and libibverbs core libraries.

[0043] Furthermore, the method includes the following steps:

[0044] S1, preload the LD_PRELOAD hijacking library into the target application. The LD_PRELOAD hijacking library contains wrapper functions with the same name as the target functions in the user-mode driver standard library.

[0045] S2, when the user-mode application APP calls the target function, the LD_PRELOAD hijacking library intercepts the call to the target function, thereby intercepting the user-mode driver interface function call;

[0046] S3, parse the parameters of the intercepted function call, and obtain the RDMA resource identifier and operation content related to this call;

[0047] S4. Generate audit data with resource-level granularity based on the RDMA resource identifier and operation content;

[0048] S5, output the audit data for visualization or analysis of the RDMA traffic of the target application.

[0049] Furthermore, in step S3, the RDMA resource identification and operation content includes creating resources, sending data, and receiving data.

[0050] Furthermore, the audit data in step S4 includes function input parameter analysis, generation of complete call trace audit logs, and generation of context-related logs.

[0051] Furthermore, function input parameter analysis includes memory length and address analysis; complete call trace audit logs include tracing the complete call chain from user-mode API calls to hardware execution; and generating context-related logs associates network traffic with specific application behaviors and user identities.

[0052] Furthermore, step S5 includes visualizing or analyzing RDMA resource information, network information, user name information, and function call trajectory information.

[0053] Furthermore, the method can also audit data related to encrypted RDMA calls.

[0054] The beneficial effects of the full-stack depth observable auditing method for high-performance RDMA network traffic of the present invention are as follows:

[0055] The method of this invention provides audit granularity at both the function level and the resource level (QP / CQ / WR), far exceeding the level of traditional network devices that can only see IP / UDP traffic. Traditional network auditing is based on IP / port. The core of this invention lies in proposing the concept of traffic auditing for native RDMA objects (such as QP, CQ, WR, etc.). This means that the auditing system can understand the semantics of RDMA, for example, it can display "the throughput of a certain QP queue pair", "the latency distribution of a certain completion queue CQ", and "the success / failure status of a certain work request WR".

[0056] The method of this invention requires no modification to the application, driver, or switch configuration. It can be implemented through user-space library hijacking, resulting in extremely low deployment costs and easy integration into existing environments. All RDMA operations are intercepted and copied by hijacking (e.g., using LD_PRELOAD or modifying dynamic link libraries) the API functions of core libraries such as librdmacm and libibverbs (e.g., ibv_post_send, ibv_post_recv, ibv_poll_cq, etc.).

[0057] The method of this invention allows maintenance personnel to precisely pinpoint which application and which QP is experiencing problems when RDMA networks experience performance jitter, packet loss, or errors, significantly reducing the Mean Time To Repair (MTTR). After hijacking the function, not only can the function call be obtained, but more importantly, the function parameters (such as QP handles and WR contents) can be correlated with the underlying data packets and integrated with network layer information (such as IP addresses and RoCEv2 UDP ports) to form a unified and readable audit view. This solves the problem of correlating RDMA application layer semantics with network layer packets.

[0058] Compared to kernel packet capture or hardware probes, the method of this invention can perform user-mode hijacking and auditing logic very efficiently, and can selectively audit specific traffic as needed, with minimal impact on business performance. Attached Figure Description

[0059] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0060] Figure 1 This is a schematic diagram of the implementation of RDMA technology in the prior art;

[0061] Figure 2 This is a diagram illustrating the existing technology and the RDMA call logic framework of this invention.

[0062] Figure 3 This is a diagram of the RDMA call logic framework of the present invention;

[0063] Figure 4 This is a schematic diagram of the hijacking database auditing logic of the present invention;

[0064] Figure 5 This is a schematic diagram of the encrypted data auditing logic of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0066] Explanation of technical terms

[0067] RDMA: Remote Direct Memory Access is an advanced network communication technology that allows one computer (the initiating computer) to directly read and write memory to another computer (the target computer) over a network without the involvement of the target computer's central processing unit (CPU).

[0068] The ibverbs interface is a set of low-level hardware-oriented interfaces that provide the most direct and fundamental control over RDMA devices, adhering to the philosophy of "providing basic building blocks." Its responsibilities include creating and managing core RDMA objects such as protection domains (PDs), completion queues (CQs), queue pairs (QPs), and memory regions (MRs).

[0069] QP: Queue Pair (QP) is the core communication endpoint of RDMA technology. Each QP consists of a Send Queue and a Receive Queue. Applications initiate RDMA operations by submitting work requests to these queues.

[0070] CQ: Completion Queue (CQ) is a data structure used to notify an application that its submitted RDMA operation (such as send or receive) has been completed. A CQ can be shared by multiple QPs.

[0071] MR: Memory Region is a user-space memory buffer that is registered to the RDMA network card. Memory registration is a crucial step; it locks and maps virtual memory addresses to physical addresses, and sets access permissions for this memory, enabling the RDMA network card to directly perform DMA (Direct Memory Access) operations on it.

[0072] CM Connection Establishment: CM (Communication Manager) is a high-level, connection-oriented abstraction interface built on top of the IB Verbs interface, designed to simplify the process of establishing reliable connections. Responsibilities: Manages both connections and underlying resources. It not only includes the functionality to create QPs (Qualifiers), but also encapsulates the entire complex process of discovering peers, exchanging parameters, handshaking, and transitioning a QP to a usable state (RTS). The CM interface encapsulates this complexity.

[0073] This invention provides a full-stack, deep observable auditing method for high-performance RDMA network traffic. By hijacking the Verbs interface of the RDMA core library, it aims to achieve transparent interception and auditing of RDMA operations. This method can precisely correlate network traffic with core resources of applications, such as QP, CQ, and WR, thereby providing unprecedentedly refined visualization capabilities for RDMA network operation, troubleshooting, and security. The core of this invention's technical solution lies in: hijacking API function calls of standard libraries such as libibverbs and librdmacm from the target application by preloading a custom dynamic link library; copying and parsing function parameters (such as QP handles, WR contents, and CQE entries) within the hijacked functions and associating them with network layer information; finally, aggregating, storing, and displaying the processed audit data to provide an audit view that reflects the details of RDMA resource-level operations. The technical solution of this invention is described in detail below.

[0074] Example 1

[0075] Combination Figure 2 and Figure 3 As shown, the present invention provides a full-stack depth observable auditing method for high-performance RDMA network traffic, which adds an LD_PRELOAD hijacking library to the original call logic framework of existing RDMA applications. Figure 2 The left figure shows the original calling logic framework of existing RDMA applications. Figure 2 The right figure shows the calling logic framework of the RDMA application of the present invention. In addition to user-mode application, user-mode driver, kernel-mode software stack and hardware, the logic framework of the present invention also includes a hijacking library set between user-mode application and user-mode driver, namely LD_PRELOAD hijacking library.

[0076] User-space applications include various apps that call RDMA functions. User-space drivers provide a unified programming interface (Verbs interface) through libibverbs and rdmacm to facilitate app calls to RDMA functions. In RDMA programming, libibverbs is used to create resources (such as PD, CQ, QP, etc.), and librdmacm is used to establish connections (such as address resolution, connection establishment, listening, etc.). Data transfer is performed; the kernel-mode software stack, as the privileged layer of the operating system, is responsible for resource arbitration, security control, and the core framework. Kernel modules in the diagram (such as ib_core.ko) implement the core framework of the RDMA subsystem, while ib_uverbs.ko specifically handles system calls from user-mode libibverbs, verifying and converting user-mode requests before forwarding them to the hardware driver. It is the fundamental guarantee for the secure execution of user-mode functions. Hardware is the physical foundation of the system, namely the network adapter (RNIC) supporting RDMA functionality, responsible for ultimately executing data transfer instructions and realizing direct data transfer between the network card and remote memory. The LD_PRELOAD hijacking library contains the common libibverbs and rdmacm interfaces (that is, re-encapsulating the same-named ibverbs and rdmacm functions in the hijacking library). Thus, when upper-layer user-mode applications (APPs) call the original user-mode driver's libibverbs and rdmacm interfaces, they will preferentially go to the newly encapsulated corresponding functions in the LD_PRELOAD hijacking library for hijacking, hijacking the API functions of core libraries such as librdmacm and libibverbs (such as ibv_post_send, etc.). (ibv_post_recv, ibv_poll_cq, etc.); After the above adjustments, the functions in LD_PRELOAD obtain the call parameters and message content of the user-space application APP. Finally, resource auditing is performed in the functions encapsulated by the LD_PRELOAD hijacking library. After hijacking the function, the function parameters (such as QP handle, WR content) are associated with the underlying actual transmitted data packets, and network layer information (such as IP address, RoCEv2 UDP port, etc.) is integrated to form a unified and readable audit view.

[0077] Auditing is conducted based on the resource hijacking function being called and its input parameters, and the results are recorded in the resource log. For example... Figure 3As shown: For example, the newly encapsulated QP creation function `rdma_create_qp` in the LD_PRELOAD hijacking library calls the `rdma_create_qp` function in the LD_PRELOAD hijacking library when the upper-layer user-space application (APP) calls `rdma_create_qp`. Then, by auditing the input parameters of this function, some attribute information of the created QP can be obtained, such as whether a UD type QP or an RC type QP is created. The return value of the function call can be used to determine whether the creation was successful. If creation fails, the reason for the failure is recorded, thus obtaining a complete QP resource log. In addition, some commonly used network traffic information, such as IP address and port, can also be obtained from the LD_PRELOAD hijacking library. This allows the resource log to be correlated with the network log, thereby generating a unified audit view that simultaneously contains RDMA resource layer information (CQ, QP, MR, etc.) and network layer flow information.

[0078] Example 2

[0079] This embodiment is based on embodiment 1, combined with Figures 4-5 The method of the present invention will be described in further detail below.

[0080] The present invention provides a full-stack depth-of-observation auditing method for high-performance RDMA network traffic, comprising the following steps:

[0081] S1, preload a custom dynamic link library, namely the LD_PRELOAD hijacking library, into the target application. The LD_PRELOAD hijacking library is deployed between the user-mode application APP and the user-mode driver standard library (i.e., the RDMA standard library, specifically including the librdmacm and libibverbs core libraries). The LD_PRELOAD hijacking library contains wrapper functions with the same name as the target functions in the user-mode driver standard library.

[0082] S2, when the user-mode application APP calls the target function, the LD_PRELOAD hijacking library intercepts the call to the target function, thereby intercepting the call to the user-mode driver standard library Verbs interface function;

[0083] S3, parse the parameters of the intercepted function call to obtain the RDMA resource identifier and operation content related to this call, such as whether it is creating a resource, sending data, or receiving data, etc. Figure 4 Semantic analysis of function calls in [the context of the text];

[0084] S4. Based on the RDMA resource identifier and operation content, generate audit data with resource-level granularity, specifically including function input parameter analysis (such as memory length, address, etc.), generating a complete call trace audit log (tracking the complete call chain from user-mode API call to hardware execution), and generating context-related logs (associating network traffic with specific application behavior and user identity).

[0085] S5, output the audit data for visualizing or analyzing the RDMA traffic of the target application, corresponding to... Figure 4 The system generates audit view logs (including RDMA resource information, network information, user name information, function call trace information, etc.).

[0086] The context-associated logs generated in step S4 include:

[0087] 1) Capture the network layer data packet corresponding to the function call;

[0088] 2) Associate the audit data with the network layer data packets to generate a unified audit view that includes both RDMA resource layer information (CQ, QP, MR, etc.) and network layer flow information.

[0089] Furthermore, the method of this invention can audit resource information and network information (IP, port, etc.) created by RDMA's qp, cq, mr, etc. through a hijacking library, and can also audit encrypted data information. The auditing of encrypted data information is performed through the following method:

[0090] This invention achieves log analysis and auditing by hijacking the underlying calls to the libibverbs and rdmacm interfaces. Since RDMA encryption and decryption are typically hardware-based, the hijacking library added in this invention is only called before data encryption (for outgoing messages) and after decryption (for received data). Therefore, this invention can obtain the decrypted data through these interfaces and then perform analysis and auditing. Other existing solutions obtain data that is either encrypted or unencrypted, making this encrypted data unauditable. Furthermore, this invention has no impact on the performance of auditing encrypted traffic because the observation point is plaintext, not involving the cost of decryption. Decryption is performed by hardware, meaning the performance overhead introduced by auditing encrypted traffic is negligible.

[0091] Combination Figure 5 As shown, the specific encrypted data auditing process includes the following:

[0092] Regarding the encryption of transmitted data, the user-space application (APP) first sends data, which is then hijacked and audited by the LD_PRELOAD hijacking library using libibverbs and rdmacm APIs. The data is then encrypted by hardware before being transmitted to the network.

[0093] For data decryption, the process begins by acquiring network data, which is then decrypted by hardware. Next, the LD_PRELOAD hijacking library executes API hijacking and auditing using libibverbs and rdmacm, and finally returns the data to the user-space application (APP).

[0094] The auditing method described above in this invention can be used to assess its effectiveness by examining the audited logs. This allows us to view the RDMA traffic logs, which are resource-level associated logs.

[0095] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A method for full-stack deep observable auditing of RDMA high-performance network traffic, the method comprising: On the basis of the original calling logic framework of RDMA application, an LD_PRELOAD hijacking library is added. The LD_PRELOAD hijacking library is used to hijack the function calls of user-mode driven functions to obtain RDMA resource identifiers and operation content. Based on the RDMA resource identifiers and operation content, audit data containing resource-level granularity is generated. ​ The original calling logic framework of RDMA applications includes user-mode applications, user-mode drivers, kernel-mode software stacks and hardware, with the LD_PRELOAD hijacking library deployed between the user-mode applications and user-mode drivers.

2. The method according to claim 1, characterized in that, The LD_PRELOAD hijacking library contains a generic user-mode driver interface.

3. The method according to claim 2, characterized in that, The user-space driver includes the librdmacm and libibverbs core libraries.

4. The method according to claim 3, characterized in that, The method includes the following steps: S1, preload the LD_PRELOAD hijacking library into the target application. The LD_PRELOAD hijacking library contains wrapper functions with the same name as the target functions in the user-mode driver standard library. S2, when the user-mode application APP calls the target function, the LD_PRELOAD hijacking library intercepts the call to the target function, thereby intercepting the user-mode driver interface function call; S3, parse the parameters of the intercepted function call, and obtain the RDMA resource identifier and operation content related to this call; S4. Generate audit data with resource-level granularity based on the RDMA resource identifier and operation content; S5, output the audit data for visualization or analysis of the RDMA traffic of the target application.

5. The method according to claim 4, characterized in that, Step S3 includes RDMA resource identification and operation content such as resource creation, data transmission, and data reception.

6. The method according to claim 4, characterized in that, The audit data in step S4 includes function input parameter analysis, generation of complete call trace audit logs, and generation of context-related logs.

7. The method according to claim 6, characterized in that, Function input parameter analysis includes memory length and address analysis; complete call trace audit logs include tracing the complete call chain from user-mode API calls to hardware execution; and generating context-related logs associates network traffic with specific application behaviors and user identities.

8. The method according to claim 4, characterized in that, The visualization or analysis in step S5 includes RDMA resource information, network information, user name information, and function call trajectory information.

9. The method according to any one of claims 1-8, characterized in that, The method can also audit data related to encrypted RDMA calls.

Images