Kernel mode rdma qp sharing algorithm and hybrid connection pool optimization method and system

Abstract

The application provides a kernel mode RDMA QP sharing algorithm and a hybrid connection pool optimization method and system, comprising: using RDMA in a kernel mode, avoiding overflow of a NIC network card due to sharing to multiple user processes by using an RDMA QP sharing algorithm, and introducing a hybrid connection pool design, so that connection latency is reduced from a millisecond level to a microsecond level without affecting original performance.

Description

Technical Field

[0001] This invention relates to the field of kernel-mode RDMA technology, specifically to a kernel-mode RDMA QP sharing algorithm and a hybrid connection pool optimization method and system. Background Technology

[0002] RDMA technology, with its high throughput and low latency, is widely used in modern data center applications. RDMA provides support for direct host access to remote server memory. Compared to the traditional TCP / IP protocol stack, RDMA can complete data transmission between applications and network cards without going through the operating system kernel. Multiple machines use QP (queue pair) and MR (memory region) for data exchange. Based on zero-copy and kernel bypass characteristics, RDMA achieves high throughput and low latency.

[0003] RDMA also faces scalability and slow handshake connection establishment issues. Firstly, RDMA network cards need to maintain communication resources such as QP / MR. As the number of communication resources to be managed increases, the management load on the network card rises, reducing its communication performance. Secondly, the handshake connection establishment latency of RDMA is in the millisecond range, which is 15,000 times slower than the microsecond-level communication latency. When applications need to quickly initiate connection establishment and perform data transmission, the handshake connection establishment becomes a bottleneck.

[0004] Patent document CN107147722A (application number: CN201710355317.0) discloses an IB-RTI method based on the RDMA communication mechanism, which includes: creating an RDMA protocol connection port; registering the server-side RDMA operation cache; accepting the client's connection request and replying with a connection success instruction when the client requests a connection based on the RDMA protocol; exchanging QP information and modifying the client's QP information status level; receiving data information sent by the established connection client and sending data information to the established connection client.

[0005] Currently, the approaches to addressing scalability and slow handshake connection speeds include: one is cross-thread level QP sharing in user space. This user-space QP sharing scheme can reduce the resource maintenance cost of a single-machine network card, but it lacks a reasonable QP resource protection strategy. Without a proper QP resource protection strategy, QP may become unusable, affecting other threads using the resource. Another approach is to use kernel-level RDMA for resource sharing. This method can achieve cross-application level network card resource sharing at the expense of some performance. However, this method also lacks a resource protection strategy and does not solve the slow handshake connection speed problem. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a kernel-mode RDMA QP sharing algorithm and a hybrid connection pool optimization method and system.

[0007] The kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization method provided by the present invention include:

[0008] Step S1: When performing the queue enqueue operation, for the list of tasks that need to be inserted into the work queue, determine whether the conditions for direct enqueue are met. If they are met, insert them directly and then execute step S3; otherwise, execute step S2.

[0009] Step S2: Perform a dequeue operation on the queue, dequeue all tasks in the first batch of the queue, and then return to step S1 to continue execution;

[0010] Step S3: Set the type of the last task in the enqueued work queue list to signaled;

[0011] Step S4: When the user-mode application starts, use Dynamic Connection QP to establish a handshake connection;

[0012] Step S5: After the dynamic connection QP connection is established, use the dynamic connection QP for communication;

[0013] Step S6: Start the reliable connection QP handshake in the background. When the reliable connection QP handshake is completed, migrate the hybrid connection pool from dynamic connection QP to reliable connection QP.

[0014] Step S7: After the migration is complete, use reliable connection QP for communication.

[0015] Preferably, if the length of the task list is less than or equal to the available queue length of the work queue, then the direct enqueue condition is met.

[0016] Preferably, after the host kernel module is loaded into the kernel, it registers metadata with the global metadata server. The dynamic connection QP obtains the handshake information from the peer through the global metadata server and directly communicates with the peer server based on the metadata.

[0017] Preferably, when the user program requests a connection for the first time, the background starts a reliable connection QP connection. After the background task is completed, the foreground reliable connection QP is notified of the successful connection. At this time, the ongoing dynamic connection QP request is completed first, and all dynamic connection QP tasks are cleared before the migration from reliable connection QP to dynamic connection QP is performed.

[0018] Preferably, the connection pools for both dynamic connection QP and reliable connection QP are established in the kernel. Multiple hosts are configured with the same kernel version, network card driver, and kernel module. Each host is connected to the others via an unlimited bandwidth protocol. The global metadata server is connected to each host via an unlimited bandwidth protocol.

[0019] The kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization system provided by the present invention includes:

[0020] Module M1: When performing an enqueue operation on the queue, for the list of tasks that need to be inserted into the work queue, determine whether the conditions for direct enqueueing are met. If they are met, insert them directly and then execute module M3; otherwise, execute module M2.

[0021] Module M2: Performs the dequeue operation on the queue, dequeueing all tasks in the first batch from the queue, and then returns to module M1 to continue execution;

[0022] Module M3: Sets the type of the last task in the enqueued work queue to signaled;

[0023] Module M4: Uses dynamic connection QP to establish a handshake connection when the user-space application starts;

[0024] Module M5: After the Dynamic Connection QP connection is established, use Dynamic Connection QP for communication;

[0025] Module M6: Initiates a reliable connection QP handshake in the background. When the reliable connection QP handshake is complete, it performs a migration of the hybrid connection pool from dynamic connection QP to reliable connection QP.

[0026] Module M7: After migration is complete, use reliable connection QP for communication.

[0027] Preferably, if the length of the task list is less than or equal to the available queue length of the work queue, then the direct enqueue condition is met.

[0028] Preferably, after the host kernel module is loaded into the kernel, it registers metadata with the global metadata server. The dynamic connection QP obtains the handshake information from the peer through the global metadata server and directly communicates with the peer server based on the metadata.

[0029] Preferably, when the user program requests a connection for the first time, the background starts a reliable connection QP connection. After the background task is completed, the foreground reliable connection QP is notified of the successful connection. At this time, the ongoing dynamic connection QP request is completed first, and all dynamic connection QP tasks are cleared before the migration from reliable connection QP to dynamic connection QP is performed.

[0030] Preferably, the connection pools for both dynamic connection QP and reliable connection QP are established in the kernel. Multiple hosts are configured with the same kernel version, network card driver, and kernel module. Each host is connected to the others via an unlimited bandwidth protocol. The global metadata server is connected to each host via an unlimited bandwidth protocol.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) The kernel-mode QP sharing algorithm is adopted, which reduces the number of QPs on a single network card and reduces the network card resource consumption;

[0033] (2) The resource protection strategy of kernel-mode QP sharing is adopted to avoid the work queue overflow problem that may occur when multiple applications share the same QP;

[0034] (3) By adopting a hybrid connection pool scheme, the application connection latency is reduced from the microsecond level to the millisecond level. Attached Figure Description

[0035] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0036] Figure 1 Here is the flowchart for the QP sharing algorithm;

[0037] Figure 2 Flowchart for shared connection pool;

[0038] Figure 3 This is a schematic diagram of the hardware device structure of the present invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0040] Example 1:

[0041] This invention uses RDMA in kernel mode and avoids the overflow of the NIC due to sharing with multiple user processes through the RDMA QP sharing algorithm. It also introduces a hybrid connection pool design to reduce the original millisecond-level connection latency to the microsecond level without affecting the original performance.

[0042] like Figure 1 and Figure 2 This is a software flowchart of the QP sharing algorithm and hybrid connection pool in this invention.

[0043] The steps of the QP sharing algorithm include:

[0044] When the system needs to insert an RDMA request list (wr list), it first checks if there is available space in the current work queue to place the request list. The system determines whether insertion is possible based on the following: if the length of the wr list is less than or equal to the length of the available queue, then it can be enqueued. If there is insufficient space to insert the request, the system first processes completed RDMA requests, dequeues them, and updates the number of completed and incomplete requests based on the request information. The system repeats this process until there is enough space to enqueue the request. It is assumed that the length of the wr list is always less than the total length of the work queue.

[0045] After the enqueue conditions are met, the system sets the unsignaled counter to 0 and checks the request header information to determine if there is memory access permission. If there is no permission, an "access denied" error is returned directly; if there is permission, the last task in the request list is set to signaled, ensuring that the request list forms a task batch. Then, the system sends the request list to the RDMA network card, and the RDMA request list is successfully inserted.

[0046] The steps for hybrid connection pooling include:

[0047] Hybrid connection pooling refers to the combined use of dynamic connection QP pooling and reliable connection QP pooling. When the system needs to establish a handshake with a remote RDMA node, it first checks if a reusable reliable connection QP cached connection already exists in the kernel. If it does, the system returns directly; otherwise, the system checks if there is a remote metadata cache locally. If not, the system pulls the remote node's metadata information from the metadata server in the cluster and updates the metadata cache. After completing this step, the dynamic connection QP pool can directly communicate with the remote node.

[0048] Meanwhile, the background will asynchronously establish a reliable connection QP connection pool. Once the reliable connection QP asynchronous connection is completed, the system will clear the work queue of the original dynamic connection QP connection pool and migrate the connection from the dynamic connection QP connection pool to the reliable connection QP connection pool, thus completing the connection process of the hybrid connection pool.

[0049] like Figure 3As shown, this invention employs connection pools for both dynamic connection QP and reliable connection QP, both established within the kernel. Multiple hosts are configured with the same kernel version, network card driver, and kernel module. Each host connects to the others via the InfiniBand protocol, and the global metadata server connects to each host via the InfiniBand protocol as well. Each host maintains a hybrid connection pool locally to switch between different QP types.

[0050] In summary, the kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization method proposed in this example can address the shortcomings of existing solutions: 1. By adopting a kernel-mode QP sharing algorithm, the number of QPs on a single network interface card (NIC) is reduced, thus lowering NIC resource consumption; 2. By employing a kernel-mode QP sharing resource protection strategy, the work queue overflow problem that may occur when multiple applications share the same QP is avoided; 3. By adopting a hybrid connection pool scheme, the application connection latency is reduced from the microsecond level to the millisecond level.

[0051] Example 2:

[0052] This invention also provides a kernel-mode RDMA QP sharing algorithm and a hybrid connection pool optimization system. The kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization system can be implemented by executing the process steps of the kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization method. That is, those skilled in the art can understand the kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization method as a preferred embodiment of the kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization system.

[0053] The kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization system provided by the present invention includes: Module M1: When performing a queue enqueue operation, for the task list that needs to be inserted into the work queue, it determines whether the direct enqueue condition is met. If it is met, it is directly inserted, and then Module M3 is executed; otherwise, Module M2 is executed. Module M2: Performs a queue dequeue operation, dequeueing all tasks in the first batch from the queue, and then returns to Module M1 to continue execution. Module M3: Sets the type of the last task in the enqueued work queue list to signaled. Module M4: When the user-mode application starts, it uses dynamic connection QP to perform a handshake connection. Module M5: After the dynamic connection QP connection is completed, it uses dynamic connection QP for communication. Module M6: Starts a reliable connection QP handshake in the background. When the reliable connection QP handshake is completed, it performs a migration of the hybrid connection pool from dynamic connection QP to reliable connection QP. Module M7: After the migration is completed, it uses reliable connection QP for communication.

[0054] If the length of the task list is less than or equal to the available queue length of the work queue, then the condition for direct enqueueing is met.

[0055] After the host kernel module is loaded into the kernel, it registers metadata with the global metadata server. The dynamic connection QP obtains the handshake information from the peer through the global metadata server and directly communicates with the peer server based on the metadata.

[0056] When the user program makes its first connection request, the background starts a reliable connection QP connection. After the background task is completed, it notifies the foreground reliable connection QP that the connection was successful. At this time, the ongoing dynamic connection QP request is completed first, and all dynamic connection QP tasks are cleared before the migration from reliable connection QP to dynamic connection QP is performed.

[0057] The connection pools for both Dynamic Connection QP and Reliable Connection QP are established in the kernel. Multiple hosts are configured with the same kernel version, network card driver, and kernel module. Each host is connected to the others via an unlimited bandwidth protocol. The global metadata server is also connected to each host via an unlimited bandwidth protocol.

[0058] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0059] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization method, characterized in that, include: Step S1: When performing the queue enqueue operation, for the list of tasks that need to be inserted into the work queue, determine whether the conditions for direct enqueue are met. If they are met, insert them directly and then execute step S3; otherwise, execute step S2. Step S2: Perform a dequeue operation on the queue, dequeue all tasks in the first batch of the queue, and then return to step S1 to continue execution; Step S3: Set the type of the last task in the enqueued work queue list to signaled; Step S4: When the user-mode application starts, use Dynamic Connection QP to establish a handshake connection; Step S5: After the dynamic connection QP connection is established, use the dynamic connection QP for communication; Step S6: Start the reliable connection QP handshake in the background. When the reliable connection QP handshake is completed, migrate the hybrid connection pool from dynamic connection QP to reliable connection QP. Step S7: After the migration is complete, use reliable connection QP for communication; If the length of the task list is less than or equal to the available queue length of the work queue, then the condition for direct enqueueing is met. When the user program makes its first connection request, the background starts the reliable connection QP connection. After the background task is completed, it notifies the foreground reliable connection QP that the connection is successful. At this time, the ongoing dynamic connection QP request is completed first. After clearing all dynamic connection QP tasks, the migration from reliable connection QP to dynamic connection QP is performed. The connection pools for both Dynamic Connection QP and Reliable Connection QP are established in the kernel. Multiple hosts are configured with the same kernel version, network card driver, and kernel module. Each host is connected to the other via an unlimited bandwidth protocol. The global metadata server is connected to each host via an unlimited bandwidth protocol. After the host kernel module is loaded into the kernel, it registers metadata with the global metadata server. The dynamic connection QP obtains the handshake information from the peer through the global metadata server and directly communicates with the peer server based on the metadata.

2. A kernel-mode RDMA QP sharing algorithm and hybrid connection pool optimization system, characterized in that, include: Module M1: When performing an enqueue operation on the queue, for the list of tasks that need to be inserted into the work queue, determine whether the conditions for direct enqueueing are met. If they are met, insert them directly and then execute module M3; otherwise, execute module M2. Module M2: Performs the dequeue operation on the queue, dequeueing all tasks in the first batch from the queue, and then returns to module M1 to continue execution; Module M3: Sets the type of the last task in the enqueued work queue to signaled; Module M4: Uses dynamic connection QP to establish a handshake connection when the user-space application starts; Module M5: After the Dynamic Connection QP connection is established, use Dynamic Connection QP for communication; Module M6: Initiates a reliable connection QP handshake in the background. When the reliable connection QP handshake is complete, it performs a migration of the hybrid connection pool from dynamic connection QP to reliable connection QP. Module M7: After migration is complete, use reliable connection QP for communication; If the length of the task list is less than or equal to the available queue length of the work queue, then the condition for direct enqueueing is met. When the user program makes its first connection request, the background starts the reliable connection QP connection. After the background task is completed, it notifies the foreground reliable connection QP that the connection is successful. At this time, the ongoing dynamic connection QP request is completed first. After clearing all dynamic connection QP tasks, the migration from reliable connection QP to dynamic connection QP is performed. The connection pools for both Dynamic Connection QP and Reliable Connection QP are established in the kernel. Multiple hosts are configured with the same kernel version, network card driver, and kernel module. Each host is connected to the other via an unlimited bandwidth protocol. The global metadata server is connected to each host via an unlimited bandwidth protocol. After the host kernel module is loaded into the kernel, it registers metadata with the global metadata server. The dynamic connection QP obtains the handshake information from the peer through the global metadata server and directly communicates with the peer server based on the metadata.

Images