A lightweight method for cross-process sharing of RDMA reliable connections

Abstract

The application discloses a lightweight RDMA reliable connection cross-process sharing method, relates to the field of data center networks and remote direct memory access communication technologies, and designs a shared IP queue to realize multiplexing of multiple processes QP, and compresses the scale of a global hardware QP to the O(n) level of a cluster node. First, a shared memory transceiving queue is created based on a target node GUID, and a statistical array and a DB mapping table are constructed; further, a user state realizes multi-threaded writing of WQE without a global lock through atomic operation, truncated exponential backoff, modulus slot allocation and bitmap marking; finally, a kernel scheduler adopts WRR weighted round robin, combines ctz bit operation to batch deliver WQE, and efficiently recycles CQE with the aid of a hot queue, and reduces the network card overhead by batch delivering messages in a single doorbell. The method executes a large amount of logic in the user state, reduces context switching and lock conflict, guarantees bandwidth isolation of multiple applications, is suitable for high-concurrency scenes such as distributed training and key value storage, effectively alleviates the RNIC cache invalidation problem, and significantly optimizes the transmission delay of small packets.

Description

Technical Field

[0001] This invention relates to the field of data center networks and remote direct memory access communication technology, specifically a lightweight RDMA reliable connection cross-process sharing method. Background Technology

[0002] Remote Direct Memory Access (RDMA) is a core communication technology for high-performance services such as distributed training and key-value storage in current data centers. Commercial RNICs lack built-in large-capacity DRAM, relying solely on limited on-chip SRAM to store connection metadata such as QPC and WQE. A large number of concurrent QPs frequently trigger cache invalidation, causing a significant drop in network throughput. To reduce the total number of QPs and alleviate NIC cache performance degradation, existing optimization schemes based on reliable connection (RC) fall into two main branches: the first is intra-process QP multiplexing schemes such as XRC and DCT, which leverage the native hardware characteristics of the NIC to achieve connection sharing within a single process with low software overhead; the second is kernel-based cross-process QP multiplexing schemes such as LITE, SRM, and KRCORE, which use the kernel's intermediate management layer to overcome process isolation, enabling multiple processes to share the same physical QP, resulting in more significant QP compression. Both schemes alleviate the QP bloat problem from intra-process and inter-process perspectives, respectively, but both inherently suffer from the inability to balance multiplexing scale with software overhead, making them unsuitable for large-scale multi-process concurrent communication scenarios.

[0003] The shortcomings of in-process QP reuse schemes and their negative impact on business are as follows: In-process QP reuse schemes can only achieve connection resource sharing within a single process and cannot break through the process isolation boundary of the operating system. This is the core reason why their QP reduction capability is limited. XRC only decouples the resources of the sending and receiving ends, and can only reduce the QP scale from O(np^2) to O(np). In multi-process cluster scenarios, the number of QPs is still massive. DCT relies on dynamic temporary connections to reduce persistent QPs, but the connection must be repeatedly established and destroyed for each message transmission, which will generate additional waiting latency. This shortcoming has a particularly significant impact on small packet services. In distributed storage and real-time computing scenarios where short messages below 1KB account for a high proportion, frequent connection switching will significantly increase the overall communication latency. Even with extremely low software layer overhead, it cannot solve the core pain points of QP cache invalidation and small packet performance degradation in large-scale clusters.

[0004] The Defects and Performance Impact of Kernel-Based Cross-Process QP Multiplexing Schemes: While kernel-based cross-process QP multiplexing schemes can compress the QP scale to the node level, they are constrained by the RDMA permission isolation mechanism. Resource management, request forwarding, and completion event distribution are all completed in kernel mode, resulting in a dual performance bottleneck. On the one hand, applications need to enter the kernel through system calls to send messages, constantly generating user-mode and kernel-mode context switches; on the other hand, schemes such as SRM use kernel CPU to poll and pull WQE, copy WQE, and continuously poll the completion queue CQE, monopolizing CPU computing power for a long time. Actual test data shows that even when the number of QPs is controlled at a very low level, the throughput in a single CPU environment decreases by 52.2% compared to the theoretical hardware peak, making CPU computing power a new bottleneck for communication performance. Both mainstream existing schemes suffer from high latency for small packets and excessive CPU overhead, respectively.

[0005] Therefore, there is an urgent need for a reliable connection sharing scheme for cross-process RDMA with low overhead and high reuse efficiency. Summary of the Invention

[0006] 1. The technical problem to be solved by the present invention The purpose of this invention is to propose a lightweight method for cross-process sharing of reliable RDMA connections, which can achieve maximum inter-process QP reuse with extremely low software overhead, and is used to solve the problem of RNIC performance degradation caused by RNIC cache miss and software CPU bottleneck in previous work.

[0007] 2. Technical Solution To achieve the above objectives, the present invention provides the following technical solution: a lightweight RDMA reliable connection cross-process sharing method, comprising the following steps: S1, IP queue initialization: The application process carries the target node GUID to create a temporary WQE cache and sends it to the kernel scheduler. The scheduler retrieves shared memory based on the GUID and creates IP transceiver queues of a specified depth as needed. It returns the associated handle to complete the WQE binding, and simultaneously builds a DB mapping table and creates a global array to count the total amount of WQE issued and CQE reclaimed in the transceiver queues. This completes the initialization configuration of cross-node IP queue resources and index statistics structure. S2, Application process writes to IP queue WQE: The application process reads the difference between the send and receive statistics array through atomic operations to determine the queue balance. If the queue is not full, the send count is atomically incremented. In case of concurrent conflicts, the count is rolled back and retried exponentially. After verification, the modulo is taken to locate the slot and write to WQE. The corresponding slot bitmap is updated synchronously and the source identifier is configured for WQE, realizing multi-threaded safe writing to WQE without global lock. S3, the kernel scheduler sends WQE to the RDMA network card: The scheduler builds a sending count array and a hotspot polling queue. Based on the application RCQP weight, it uses WRR weighted polling to traverse the IP queue. It uses bitmap bit operations to extract consecutive valid WQE in batches. A single doorbell sends batches to the network card and updates the sending count. After processing, the queue is added to the hotspot queue for polling and recycling CQE. If recycling is successful, the bitmap is cleared, the receive count is incremented, and the built-in CQE is filtered. The whole mechanism reduces lock contention, doorbell overhead, and context switching, achieving low-overhead cross-process QP reuse and bandwidth isolation.

[0008] Preferably, S1 specifically includes: S101. Each application process creates a temporary WQE buffer of a hardware QP structure without a network card and sends it to the kernel-mode scheduler along with the target node GUID. S102. The kernel scheduler searches the shared memory based on the target node's GUID and determines whether the corresponding IP sending queue and IP receiving queue exist. If they do not exist, it creates an IP sending queue and an IP receiving queue with a set queue depth in the shared memory, generates an associated handle, returns it to the application process, and binds the application process's temporary buffer's WQE to the corresponding IP queue. S103. The scheduler constructs a DB mapping table to store the IP queue number qn and the corresponding doorbell register number; S104. Create arrays A[n] and B[n] in shared memory, where n is the total number of cluster nodes; array A[xx] records the total number of WQEs sent by IP number xx in the sending queue, and array B[xx] records the total number of CQEs recovered by IP number xx in the receiving queue.

[0009] Preferably, S2 specifically includes: S201. The application process reads A[xx] and B[xx] through atomic read operations, calculates the difference A[xx]-B[xx] to obtain the number of WQEs currently pending in the IP queue; S202. If the difference is equal to the queue depth, the queue is determined to be full, and the issuance of WQE is paused; if the difference is less than the depth, an atomic increment operation is performed to increment A[xx] by 1. S203, re-verify A[xx]-B[xx]; if the difference is greater than depth, it means that there is a multi-threaded concurrent slot contention. Perform atomic decrement of 1 on A[xx], and retry S201~S203 in a loop after sleeping based on the truncation exponential backoff algorithm. S204. After the verification is passed, calculate the IP queue slot corresponding to the current WQE based on the modulo of depth using A[xx], and write the WQE into the slot. S205. Each IP queue is equipped with an independent bitmap. After writing to WQE, the corresponding slot bitmap is marked as 1. The driver configures a send_singled flag for each WQE and uniquely identifies the application process from which the WQE originates by using wr_id.

[0010] As a preferred embodiment, S3 specifically includes: S301. The scheduler creates an array C[xx] in shared memory to record the total number of WQEs that the doorbell has sent to the xx IP queue; at the same time, it maintains a hotspot queue Q to poll the receiver's CQE; each application process maintains a weight table in shared memory, and the weight value is equal to the application's native RCQP quantity. S302. The scheduler reads the weight table and uses the weighted round-robin (WRR) algorithm to traverse all IP queues. When traversing a single IP queue, it compares A[xx] with C[xx]. If they are equal, it skips the IP queue. S303. The scheduler reads the bitmap associated with the IP queue, and quickly retrieves the slot range with consecutive valid WQEs starting from the head of the queue by inverting the bitmap using ctz bitwise operations; it then sends all WQEs in the slot range to the RNIC network card in one go using a single doorbell command, and updates the IP queue head pointer; and it also updates the value of C[xx] synchronously. S304. After completing the doorbell distribution, add the current IP queue number qn to the tail of the hotspot queue Q; read the head qn of the hotspot queue Q and poll the corresponding IP receiving queue to obtain CQE; if CQE is not fully obtained, dequeue the head qn and reinsert it to the tail of the queue. S305. If a CQE is successfully acquired, the bitmap flag of the corresponding slot is set to 0, and an auto-increment operation is performed on B[xx] synchronously. If the wr_id of the CQE is a driver-built-in identifier, the CQE is not distributed to the application process. The method avoids global lock contention in the shared IP queue of multiple processes through a fine-grained slot addressing mechanism, reduces doorbell overhead through batch distribution of continuous slots, and ensures bandwidth allocation isolation for multiple applications through weighted round-robin scheduling. It does not require the kernel to continuously monopolize the CPU for polling or high-frequency user-mode / kernel-mode context switching, thus achieving low-overhead cross-process QP reuse and reducing the probability of RNICQPC cache miss.

[0011] Preferably, the execution logic of the truncation exponential backoff algorithm is as follows: 2 n+1 ·t0+random(0,t0) In the formula, t0 is the complete time taken to write to the IP queue in a single WQE operation, and n is the number of retries; the sleep duration is set to a fixed upper limit t. max To avoid indefinite periods of dormancy.

[0012] Preferably, the WQE slot allocation method abandons the native RCQP tail contention mechanism. Each WQE is directly mapped to a unique fixed slot based on the global incrementing counter A[xx] modulo the queue depth depth. There is no global lock contention at the tail of the queue when multiple processes write concurrently.

[0013] Preferably, the hotspot queue Q only stores the IP queue numbers of CQEs to be reclaimed. The scheduler prioritizes polling nodes in the hotspot queue to reduce CPU resource consumption caused by invalid polling of idle IP queues. The weights of the weighted polling WRR algorithm correspond one-to-one with the original number of independent RCQPs for each application. The software layer scheduling matches the application's native bandwidth allocation requirements, offsetting the bandwidth squeeze problem of RNIC hardware RR polling of a single IP queue.

[0014] Preferably, the scheduler uses CTZ bitwise operations to locate consecutive valid WQE slots at the head of the queue in one go, and issues multiple WQE slots in a single doorbell instruction, reducing network card overhead caused by frequent doorbell register interactions.

[0015] As a preferred approach, the WQE cache buffer created during the application process initialization phase exists only in the host shared memory, does not request QP hardware context from the RNIC network card, and only occupies a single network card QP resource through a unified IP queue, thus compressing the global QP number to the O(n) level, where n is the number of remote nodes in the cluster.

[0016] As a preferred option, it is suitable for distributed training, distributed file systems, and high-concurrency RDMA communication scenarios for key-value storage. Compared with existing inter-process QP reuse schemes that rely on kernel-exclusive CPU polling and frequent system calls, it reduces communication throughput loss by more than 50% under a single CPU, reduces transmission latency for small messages (within 1KB), and alleviates throughput attenuation caused by frequent misses of the RNIC on-chip SRAM QPC cache.

[0017] Compared with existing technologies, the lightweight RDMA reliable connection cross-process sharing method provided by this invention has the following advantages: Compared to in-process QP multiplexing schemes, this approach balances extreme QP reduction capabilities with small message transmission performance. Traditional in-process QP multiplexing technologies such as XRC, DCT, and Flock can only be used within a single process sharing a connection, resulting in limited QP compression. DCT also requires frequent dynamic connection establishment and destruction, which introduces significant additional latency for short messages smaller than 1KB, severely impacting the performance of small data packet-intensive services. The overall concept of the in-process QP multiplexing scheme is as follows: Figure 1 As shown. Assuming there are n nodes, and each node has p processes, then the fully connected communication mode requires O(n^2) processes. p) QP connections. This invention achieves global QP reuse based on a cross-process shared IP queue, which can compress the scale of the entire network QP to the O(n) level of the number of server nodes, fundamentally reducing the number of QPC cache entries that need to be maintained on the RNIC, and significantly reducing the performance loss caused by cache misses; at the same time, it does not require repeated connection disassembly and assembly like DCT, and there is no additional waiting overhead for short messages. In high-frequency small packet scenarios such as distributed training and in-memory databases, the throughput and latency performance are far superior to in-process reuse schemes.

[0018] Compared to kernel-centralized cross-process QP reuse solutions, this significantly reduces CPU kernel switching and copying overhead. The overall approach of cross-process reuse solutions such as LITE and SRM is as follows: Figure 2 As shown. Assuming there are n nodes and p processes, since each node only needs one QP connection, even with full connection communication, only O(n) connections are required. Existing cross-process QP reuse schemes are limited by the RDMA permission isolation mechanism, with all resource scheduling and request forwarding logic being executed in kernel mode. In the default RDMA driver, QP generally adopts a whole queue exclusive lock mechanism, which can cause intense lock contention when multiple processes concurrently send WQE. To avoid lock contention when multiple processes send data to the same node-level queue Node QP, the most advanced working SRM allows the application to send WQE to uQP, and then the kernel CPU scheduler polls different uQPs and pulls WQE to the node-level queue, uniformly distributing completion events. This will continuously occupy CPU cores, causing a serious CPU bottleneck. In actual single-CPU scenarios, the throughput drops by nearly 50% compared to the theoretical peak of the hardware. This invention designs a distributed slot writing mechanism based on timestamp modulo, where each process independently writes to its own dedicated queue slot, eliminating global queue lock contention, such as Figure 3 In the original scheme, after WQE was written to the queue, the RDMA network card was immediately notified via the doorbell mechanism to pull WQE from the node-level queue. If the slots with existing WQE in the queue were not contiguous (i.e., some slots had not yet received WQE), the network card could not correctly parse the corresponding slot's WQE, thus resulting in an error. To avoid this problem, this scheme decouples the doorbell mechanism from the WQE writing to the queue. The kernel CPU scheduler quickly checks whether the WQE in the node-level queue is contiguous using a bitmap. If it is contiguous, multiple packets are delivered to the network card in a single doorbell command to achieve a WQE batch delivery mechanism, further improving the network card's throughput efficiency. Since the doorbell does not require WQE transport, and the WQE batch delivery mechanism further reduces the overhead of the kernel CPU scheduler, this invention avoids high-frequency system calls and global data copying, greatly releasing CPU computing power, solving the computing power bottleneck problem of traditional kernel solutions, and simultaneously achieving low lock conflicts, fair bandwidth scheduling, and high hardware utilization efficiency in large-scale multi-process concurrent scenarios.

[0019] Furthermore, existing cross-process sharing and RNIC hardware default polling scheduling of all QPs can easily disrupt the original bandwidth allocation of multiple services when multiple services share a single IP queue. This invention, when batch distributing WQE through the kernel CPU scheduler, adopts a weighted polling scheduling strategy, allocating network card scheduling opportunities based on the original number of QPs for each application, strictly preserving the original bandwidth isolation and resource allocation semantics of each service; Attached Figure Description Figure 1 This is a schematic diagram of a lightweight RDMA reliable connection cross-process sharing method according to the present invention. Figure 2 This is the in-process QP reuse scheme of the present invention; Figure 3 This is a previous inter-process reuse scheme of the present invention; Figure 4 This is the optimized inter-process reuse scheme of the present invention; Figure 5 This is a bar chart comparing RC and SRM in Embodiment 2 of the present invention with the present solution. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0022] Example 1, please refer to Figures 1 to 5 As shown: To address the issues mentioned in the technical solutions, this solution proposes a more efficient inter-process reuse scheme that can achieve maximum inter-process QP reuse with extremely low software overhead. This scheme aims to resolve the RNIC performance degradation issues caused by RNIC cache misses and software CPU bottlenecks in previous work. The biggest problem with inter-process reuse schemes is that severe lock contention inevitably occurs when different processes share a single QP. SRM attempts to avoid lock contention by actively pulling messages from different processes through the scheduler, but this incurs excessive CPU overhead, thus reducing performance. To distinguish it from application QPs, we refer to the shared QP as an IP queue.

[0023] What sets us apart from existing technologies is our proposal to solve this problem. QPSwitch consists of three core components.

[0024] First, to address the lock contention issue arising from different processes sharing a QP, we propose a fine-grained resource reservation scheme. In the native RC, each QP queue is a circular FIFO queue with many slots. Each WQE enters the slot at the tail of the queue and is then sent to the RNIC in sequence. Therefore, the lock is applied to the entire queue. In our scheme, each WQE is sorted according to its own sending timestamp, modulo the queue length according to the sending order, and then sent to the corresponding slot within the queue, instead of competing for the slot at the tail of the queue.

[0025] Secondly, although the above scheme avoids lock contention, it may lead to out-of-order delivery of WQEs or errors (it's possible that WQE #1 goes to slot #1, and WQE #2 goes to slot #2, but WQE #2 arrives in the IP queue slot before WQE #1). To solve this problem, we set up a scheduler to check the slots in each IP queue and only notify the RNIC of the positions of consecutive slots with WQEs, starting from the head of the queue, so that the RNIC can acquire the WQEs. Simultaneously, the scheduler acquires the CQEs of the IP queues and distributes them to the corresponding CQ queues for the applications.

[0026] Finally, RNICs are generally considered to serve different QPs in a round-robin fashion, meaning that if there are n QPs, each QP has a 1 / n probability of being served by the RNIC. Therefore, reducing the number of QPs through IP queues may lead to an incorrect bandwidth allocation ratio for the application. Thus, the scheduler needs to schedule wqe messages multiplexed into IP queues based on the application's original QP allocation ratio.

[0027] This application provides a lightweight method for cross-process sharing of reliable RDMA connections, as detailed below: I. IP Queue Initialization (1) The application creates a buffer for temporary storage of wqe using ibv_create_qp (one buffer corresponds to one rcqp. This buffer has no qp structure and does not need to be maintained by the network card), and then passes parameters to enter kernel mode.

[0028] (2) Based on the GUID issued by the application, the kernel-mode scheduler checks whether the IP send and receive queues have been created. If not, a new IP send queue and IP receive queue are created in shared memory, with the depth of each queue set to depth, which can be set to a larger value. Then, the handle (context, PD, etc.) is returned to the application, and the wqe sent by the application's rcqp will be bound to the IP queue.

[0029] (3) The scheduler maintains a db mapping table to record the sequence number of the db register corresponding to the qn of each IP queue, so that the doorbell can be given to the network card later.

[0030] (4) The scheduler creates two arrays A[n] and B[n] in shared memory, each with a capacity of n nodes. A[xx] is responsible for counting the number of wqe sends in the shared IP sending queue of node xx, and B[xx] is responsible for counting the number of cqe acquires in the shared IP receiving queue of node xx.

[0031] II. Application --> IP Queues (1) When each application calls ibv_post_send to send a wqe, it first obtains the values ​​of A[xx] and B[xx] through atomic read and calculates the difference between them, which represents the number of wqes waiting for CQE in the xx IP queue.

[0032] (2) Then apply the calculation of the value of A[xx]-B[xx]. If it is depth, it means that it is full and stop sending wqe to the IP queue.

[0033] (3) If A[xx] - B[xx] != depth, then add 1 to A[xx] atomically via fetch_add. (Even if the application has many wr->next, only one can be fetch_add at a time.) (4) Before sending wqe to the IP queue, perform the judgment of A[xx]-B[xx] again. If A[xx]-B[xx]<=depth, it means that wqe can be sent to the IP queue. If A[xx]-B[xx]>depth, it means that wqe can no longer be sent (thread contention has occurred, and other threads have already sent wqe). The application atomically decrements A[xx] by 1. Then the application function enters sleep mode and is periodically woken up by referring to the truncation exponential backoff algorithm to repeat the operations in (2) and (3).

[0034] (5) The truncation exponential backoff algorithm is as follows: every 2 n-1 The CPU is woken up once every t0 + random(0, t0), where t0 is the entire process of the application sendingwqe to the IP queue after completing A+1. To avoid excessively long backoff times, 2 n-1 t0 + random(0, t0) has an upper bound t max ; (6) If in step (3), the first occurrence of A[xx]-B[xx]<depth and the second occurrence of A[xx]-B[xx]<=depth, it indicates that a wqe can be sent to the IP queue. At this time, the slot position in the IP queue where the wqe should be placed shall be found according to A[xx]%depth, and then the wqe is placed therein.

[0035] (7) Currently, it is tentatively determined that the driver sets the send_singled flag for each wqe to generate a cqe, and identifies this part of wqes through the wr_id field.

[0036] (8) In addition, the application also needs to maintain a bitmap for the shared IP queue numbered A[xx]. Each element of the bitmap is represented by 0 / 1, indicating whether there is a wqe in a certain slot of the shared IP queue numbered A[xx]. After placing the WQE into the corresponding slot each time, the corresponding position in the bitmap is marked as 1.

[0037] III. IP Queue - Network Interface Card (1) The kernel scheduler maintains an array C[xx] in the shared memory, which represents the number of wqes of a certain IP queue that have completed doorbell operations by the scheduler. The scheduler also maintains a hot queue Q for fast polling of cqes.

[0038] (2) It is tentatively determined that the scheduler performs fair scheduling on all IP queues based on a weighted round-robin algorithm (to ensure resource isolation of the native application semantics). The weight is the actual number of RCQPs on each IP queue. The weight table is maintained by the application in the shared memory, and the scheduler reads the weight table to determine the scheduling ratio for each IP queue.

[0039] Software WRR, hardware can adopt RR; software RR, hardware may need to adopt WRR; (3) When polling the IP queues, the scheduler will first read the value of A[xx] and the value of C[xx]. If A[xx] is equal to C[xx], it indicates that there is no wqe in this IP queue, and the queue is skipped. Otherwise, the scheduler reads the bitmap corresponding to this IP queue (4) When the scheduler is ready to send the IP queue numbered xx, it uses the ctz bit operation instruction (CountTrailingZeros, which is used to calculate the number of consecutive zeros starting from the least significant bit in the binary representation of an integer) for negation to quickly obtain consecutive slots with WQEs starting from the head pointer of the shared QP queue in the current shared QP from the bitmap. Then the pointer of the last one in these slots is sent to the network interface card via doorbell (one db operation for multiple wqes, and the number of wqes can be set with an upper limit). Then the scheduler updates the head pointer.

[0040] (5) After doorbell completes the wqe of IP queue number xx, the scheduler will add the qn of the current IP sending queue to the tail of the hotspot queue Q, and obtain the CQE from the corresponding IP receiving queue based on the qpn of the head of the current hotspot queue Q. If it is not polled or not polled completely, the head of the hotspot queue is dequeued and then added to the tail.

[0041] (6) If the corresponding IP queue's CQE is obtained, the scheduler sets the bitmap of the corresponding WQE for the IP sending queue to 0 and updates B[xx]. In addition, if the wr_id field in this part of the CQE indicates that the CQE was added by the driver itself, then the CQE should not be distributed to the application CQ.

[0042] Example 2: Based on Embodiment 1, but with some differences, the present invention provides a lightweight RDMA reliable connection cross-process sharing method with specific examples and accompanying drawings. The specific details are as follows: Key-value storage scenarios often require maintaining a large number of QP connections. We conducted relevant tests using Twitter Key-Value storage as an example. In Twitter Key-Value, approximately 80% of the data is less than 1KB. We used two Dell R750 servers, each equipped with an Nvidia CX6 RDMA network card with a bandwidth of 200Gbps, and the network cards were directly connected for data transmission.

[0043] We tested the native RDMA-driven RC and the cross-process QP multiplexing SRM (using one CPU core, with the kernel node-level queue count set to 32). Our solution (using one CPU core, with the kernel node-level queue count set to 32) achieved throughput under Twitter KeyValue mode traffic at a QP count of 256 (no cache misses) and a QP count of 16384 (cache misses will occur, referred to as 16K). See the figure: like Figure 5 As shown, when RC decreases from 256 QPs to 16K QPs, the throughput drops by about 22.0%, which is due to cache misses.

[0044] The SRM scheme sets the number of node-level queues to 32, so even if the number of application QPs increases, its throughput will not decrease due to cache misses. However, because SRM requires frequent WQE copying by the CPU, the CPU becomes the performance bottleneck of RDMA, resulting in a 43.5% decrease in throughput compared to RC at 256 QPs. Although increasing the number of application QPs will increase the WQE per node-level queue, thus bringing some performance improvement, its throughput at 16K QPs is still 13% lower than RC.

[0045] Our solution involves the application sending WQE to each slot of the node-level queue, requiring only a small amount of CPU to handle doorbell operations and necessary polling for completion information. Therefore, at 256 QPs, the throughput is essentially the same as RC. When the number of QPs increases to 16K, the throughput remains at 173Gbps, a 25.4% improvement over RC.

[0046] Please refer to the above work process. Figures 1 to 5 .

[0047] It should be noted that the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0048] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight RDMA reliable connection cross-process sharing method, characterized in that, Includes the following steps: S1, IP queue initialization: The application process carries the target node GUID to create a temporary WQE cache and sends it to the kernel scheduler. The scheduler retrieves shared memory based on the GUID and creates IP transceiver queues of a specified depth as needed. It returns the associated handle to complete the WQE binding, and simultaneously builds a DB mapping table and creates a global array to count the total amount of WQE issued and CQE reclaimed in the transceiver queues. This completes the initialization configuration of cross-node IP queue resources and index statistics structure. S2, Application process writes to IP queue WQE: The application process reads the difference between the send and receive statistics array through atomic operations to determine the queue balance. If the queue is not full, the send count is atomically incremented. In case of concurrent conflicts, the count is rolled back and retried exponentially. After verification, the modulo is taken to locate the slot and write to WQE. The corresponding slot bitmap is updated synchronously and the source identifier is configured for WQE, realizing multi-threaded safe writing to WQE without global lock. S3, the kernel scheduler sends WQE to the RDMA network card: The scheduler builds a sending count array and a hotspot polling queue. Based on the application RCQP weight, it uses WRR weighted polling to traverse the IP queue. It uses bitmap bit operations to extract consecutive valid WQE in batches. A single doorbell sends batches to the network card and updates the sending count. After processing, the queue is added to the hotspot queue for polling and recycling CQE. If recycling is successful, the bitmap is cleared, the receive count is incremented, and the built-in CQE is filtered. The whole mechanism reduces lock contention, doorbell overhead, and context switching, achieving low-overhead cross-process QP reuse and bandwidth isolation.

2. The lightweight RDMA reliable connection cross-process sharing method according to claim 1, characterized in that, S1 specifically includes: S101. Each application process creates a temporary WQE buffer of a hardware QP structure without a network card and sends it to the kernel-mode scheduler along with the target node GUID. S102. The kernel scheduler searches the shared memory based on the target node's GUID and determines whether the corresponding IP sending queue and IP receiving queue exist. If they do not exist, it creates an IP sending queue and an IP receiving queue with a set queue depth in the shared memory, generates an associated handle, returns it to the application process, and binds the application process's temporary buffer's WQE to the corresponding IP queue. S103. The scheduler constructs a DB mapping table to store the IP queue number qn and the corresponding doorbell register number; S104. Create arrays A[n] and B[n] in shared memory, where n is the total number of cluster nodes; array A[xx] records the total number of WQEs sent by IP number xx in the sending queue, and array B[xx] records the total number of CQEs recovered by IP number xx in the receiving queue.

3. The lightweight RDMA reliable connection cross-process sharing method according to claim 1, characterized in that, S2 specifically includes: S201. The application process reads A[xx] and B[xx] through atomic read operations, calculates the difference A[xx]-B[xx] to obtain the number of WQEs currently pending in the IP queue; S202. If the difference is equal to the queue depth, the queue is determined to be full, and the issuance of WQE is paused; if the difference is less than the depth, an atomic increment operation is performed to increment A[xx] by 1. S203, re-verify A[xx]-B[xx]; if the difference is greater than depth, it means that there is a multi-threaded concurrent slot contention. Perform atomic decrement of 1 on A[xx], and retry S201~S203 in a loop after sleeping based on the truncation exponential backoff algorithm. S204. After the verification is passed, calculate the IP queue slot corresponding to the current WQE based on the modulo of depth using A[xx], and write the WQE into the slot. S205. Each IP queue is equipped with an independent bitmap. After writing to WQE, the corresponding slot bitmap is marked as 1. The driver configures a send_singled flag for each WQE and uniquely identifies the application process from which the WQE originates by using wr_id.

4. The lightweight RDMA reliable connection cross-process sharing method according to claim 1, characterized in that, S3 specifically includes: S301. The scheduler creates an array C[xx] in shared memory to record the total number of WQEs that the doorbell has sent to the xx IP queue; at the same time, it maintains a hotspot queue Q to poll the receiver's CQE; each application process maintains a weight table in shared memory, and the weight value is equal to the application's native RCQP quantity. S302. The scheduler reads the weight table and uses the weighted round-robin (WRR) algorithm to traverse all IP queues. When traversing a single IP queue, it compares A[xx] with C[xx]. If they are equal, it skips the IP queue. S303. The scheduler reads the bitmap associated with the IP queue, and quickly retrieves the slot range with consecutive valid WQEs starting from the head of the queue by inverting the bitmap using ctz bitwise operations; it then sends all WQEs in the slot range to the RNIC network card in one go using a single doorbell command, and updates the IP queue head pointer; and it also updates the value of C[xx] synchronously. S304. After completing the doorbell distribution, add the current IP queue number qn to the tail of the hotspot queue Q; read the head qn of the hotspot queue Q and poll the corresponding IP receiving queue to obtain CQE; if CQE is not fully obtained, dequeue the head qn and reinsert it to the tail of the queue. S305. If a CQE is successfully acquired, the bitmap flag of the corresponding slot is set to 0, and an auto-increment operation is performed on B[xx] synchronously. If the wr_id of the CQE is a driver-built-in identifier, the CQE is not distributed to the application process. The method avoids global lock contention in the shared IP queue of multiple processes through a fine-grained slot addressing mechanism, reduces doorbell overhead through batch distribution of continuous slots, and ensures bandwidth allocation isolation for multiple applications through weighted round-robin scheduling. It does not require the kernel to continuously monopolize the CPU for polling or high-frequency user-mode / kernel-mode context switching, thus achieving low-overhead cross-process QP reuse and reducing the probability of RNICQPC cache miss.

5. A lightweight RDMA reliable connection cross-process sharing method according to claim 3, characterized in that, The execution logic of the truncated exponential backoff algorithm is as follows: 2 n+1 ·t0+random(0,t0) In the formula, t0 is the complete time taken to write to the IP queue in a single WQE operation, and n is the number of retries; the sleep duration is set to a fixed upper limit t. max To avoid indefinite periods of dormancy.

6. A lightweight RDMA reliable connection cross-process sharing method according to claim 1, characterized in that, The WQE slot allocation method abandons the native RCQP tail contention mechanism. Each WQE is directly mapped to a unique fixed slot based on the global incrementing counter A[xx] modulo the queue depth depth. There is no global lock contention at the tail of the queue when multiple processes write concurrently.

7. A lightweight RDMA reliable connection cross-process sharing method according to claim 4, characterized in that, The hotspot queue Q only stores the IP queue numbers of CQEs to be reclaimed. The scheduler prioritizes polling nodes in the hotspot queue to reduce CPU resource consumption caused by invalid polling of idle IP queues. The weights of the weighted polling WRR algorithm correspond one-to-one with the original number of independent RCQPs for each application. The software layer scheduling matches the application's native bandwidth allocation requirements to offset the bandwidth squeeze problem of RNIC hardware RR polling of a single IP queue.

8. A lightweight RDMA reliable connection cross-process sharing method according to claim 1, characterized in that, The scheduler uses CTZ bitwise operations to locate consecutive valid WQE slots at the head of the queue in one go, and issues multiple WQE slots in a single doorbell instruction, reducing network card overhead caused by frequent doorbell register interactions.

9. A lightweight RDMA reliable connection cross-process sharing method according to claim 1, characterized in that, The WQE cache buffer created during the application process initialization phase exists only in the host shared memory. It does not request QP hardware context from the RNIC network card, but only occupies a single network card QP resource through a unified IP queue, thus compressing the global QP number to the O(n) level, where n is the number of remote nodes in the cluster.

Images