A method and system for processing data of an RDMA network based on driving middleware

Abstract

The application provides an RDMA network data processing method and system based on driving middleware, and the method is applied to a network card driver, and the method comprises the following steps: obtaining a request of an RC type connection created by an application, constructing a request of a UD type connection based on the request of the RC type connection; obtaining a WQE of the RC type issued by the application, and cutting a large message in the WQE into a plurality of small messages with an MTU granularity; delivering the WQE to a connection of the UD type which is in an RTS state, adding a number to each small message in sequence, adding a number and an identification field to each ordinary message or small message in sequence, and sending to a destination; obtaining feedback information of the destination, determining whether data retransmission is needed based on the feedback information, and performing data retransmission on the corresponding ordinary message or small message. The scheme is applied to a network card driver program, a request of a UD type connection is constructed based on a request of an RC type connection, a large QP does not need to be constructed for the UD type connection, and the stability of network scale expansion is ensured.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to an RDMA network data processing method and system based on driver middleware. Background Technology

[0002] RDMA (Remote Direct Memory Access) technology was developed to address the latency issues in server-side data processing during network transmission. RDMA quickly moves data from one system's memory to a remote system's memory without intervention from the operating systems of either communicating system. Unlike TCP / IP, it uses a memory registration mechanism that allows the network interface card (NIC) to directly read user-space memory data and offloads network protocols to the NIC hardware. This eliminates the overhead of copying and moving data packets between user space and kernel space, as well as context switching, significantly reducing CPU overhead. This enables high-throughput, low-latency network communication, making it particularly suitable for large-scale parallel computer clusters.

[0003] RDMA initially built lossless data center networks using Infiniband (IB) dedicated network hardware, providing high-performance networks for HPC (High Performance Computing) clusters. However, current data center network equipment is mostly Ethernet equipment, and it is a lossy network, resulting in packet loss. The emergence of second-generation converged Ethernet RDMA allows Ethernet clusters to apply RDMA technology. This is achieved by encapsulating RDMA transport layer packets under the UDP (User Datagram Protocol) transport layer, and relying on PFC (Priority-based Flow Control) to provide lossless support. RDMA supports three basic transport types: RC (Reliable Connection), UC (Unreliable Connection), and UD (Unreliable Datagram). Connection-oriented transport types require both communicating parties to establish a QP (Queue Pair) for communication.

[0004] Therefore, when the network is large, the number of QPs that need to be built is quite large, which limits the expansion of the network. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an RDMA network data processing method based on driver middleware to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides an RDMA network data processing method based on driver middleware. This method is applied to a network card driver and includes the following steps:

[0007] Get the request for the RC type connection created by the application, and construct the request for the UD type connection based on the request for the RC type connection;

[0008] Get the RC type WQE issued by the application, combine multiple WQE into a data packet, divide the WQE in the data packet into large messages and ordinary messages based on a preset first length threshold, and divide the RC type large messages into multiple MTU granular small messages.

[0009] The weight of a large message is calculated based on its data length to determine whether it belongs to a short-flow queue or a long-flow queue. Large messages belonging to the short-flow queue are combined with ordinary messages to form a short-flow queue; large messages belonging to the long-flow queue are formed into a long-flow queue.

[0010] The short-flow queue and long-flow queue are combined into a sending queue. The ordinary messages or small messages in the sending queue are passed to the UD type connection that is already in the RTS state. Number and identification fields are added to each ordinary message or small message in sequence and sent to the destination.

[0011] Obtain feedback information from the destination, and determine whether data retransmission is required based on the feedback information. If data retransmission is required, retransmit the corresponding ordinary message or small message.

[0012] Using the above scheme, this scheme is applied to the network card driver and the network card driver is adjusted. The UD type connection request is constructed based on the RC type connection request. The UD type connection is different from the RC type connection. For the RC type connection, both communicating parties need to establish a pair of QP (Queue Pair) for communication. For the UD type connection, only one one-to-many QP needs to be built at each end. It does not need to build a large number of QPs, which ensures the stability of network scaling.

[0013] In some embodiments of the present invention, after the step of constructing a UD type connection request based on an RC type connection request, the method further includes the step of converting the RC connection type connection information into the connection information required by the UD connection type, and converting the state machine of the UD connection from the initial state to the RTS state.

[0014] In some embodiments of the present invention, in the step of dividing a large RC-type message into multiple small messages of MTU granularity, the large message is cached and the position of each small message in the large message is recorded.

[0015] In some embodiments of the present invention, the weight of a large message is calculated based on its data length to determine whether it belongs to a short-flow queue or a long-flow queue. Large messages belonging to the short-flow queue are combined with ordinary messages to construct a short-flow queue. The step of constructing large messages belonging to the long-flow queue into a long-flow queue further includes:

[0016] For large messages, a first judgment value is calculated based on the largest weight in the data packet and the weight of the large message. The first judgment value is compared with a first threshold. A second judgment value is calculated based on the length of the smallest ordinary message in the data packet and the length of the large message. The second judgment value is compared with a second threshold.

[0017] If for a large message, the first judgment value is greater than the first threshold and the second judgment value is less than or equal to the second threshold, then the large message is determined to belong to the short-flow queue, and the SJF algorithm is used to calculate the large message and ordinary messages to complete the reordering.

[0018] If for a large message, the first judgment value is greater than the first threshold and the second judgment value is greater than the second threshold, then the large message is determined to belong to the short-flow queue and is placed after the reordered large messages and ordinary messages.

[0019] If for a large message, the first judgment value is less than or equal to the first threshold, then the large message is determined to belong to the long queue, and a weight-based round-robin algorithm is used to reorder the large messages in the long queue.

[0020] In some embodiments of the present invention, in the step of determining whether a large message belongs to a short-flow queue or a long-flow queue by calculating the weight of the large message based on its data length, the weight of the large message is calculated based on the following formula:

[0021] W = L + L MTU -1 / L MTU ;

[0022] Where W represents the weight, L represents the length of the large message, and L MTU This indicates the length of a unit MTU.

[0023] In some embodiments of the present invention, in the step of calculating a first determination value based on the largest weight in the data packet and the weight of the large message, the ratio of the largest weight in the data packet to the weight of the large message is calculated as the first determination value; in the step of calculating a second determination value based on the length of the smallest ordinary message in the data packet and the length of the large message, the ratio of the length of the large message to the length of the smallest ordinary message in the data packet is calculated as the second determination value.

[0024] In some embodiments of the present invention, in the step of combining the short-flow queue and the long-flow queue into a sending queue, the short-flow queue is placed before the long-flow queue to form a sending queue.

[0025] In some embodiments of the present invention, the feedback information includes confirmation of receipt of a small message. In the step of obtaining the feedback information from the destination and determining whether data retransmission is required based on the feedback information, if confirmation of receipt of a normal message or a small message is received within a preset time period, it is determined that the corresponding normal message or small message has been successfully received.

[0026] In some embodiments of the present invention, in the step of obtaining feedback information from the destination and determining whether data retransmission is required based on the feedback information, if no confirmation of receipt of a normal message or a small message is received within a preset time period, the corresponding normal message or small message will be retransmitted.

[0027] In some embodiments of the present invention, in the step of retransmitting the corresponding ordinary message or small message, if a small message is retransmitted, the number of the corresponding small message is obtained, the position of the small message in the large message is determined based on the number of the small message, the corresponding small message is extracted from the large message based on the position of the small message in the large message, and the small message is sent to the destination.

[0028] In some embodiments of the present invention, in the step of adding numbering and identification fields to each ordinary message or small message in sequence and sending it to the destination, a preset identification field is inserted before the payload.

[0029] In some embodiments of the present invention, the large message is divided into multiple processing segments. In the step of obtaining feedback information from the destination, the processing segments are traversed from front to back. If all the small messages in the processing segment have been received, the corresponding processing segment is deleted from the cached large message, and the length of the deleted processing segment is recorded.

[0030] In some embodiments of the present invention, in the step of obtaining feedback information from the destination, after all the small messages in the last processing segment of the cached large message have been received, the last processing segment is deleted, and the length of each processing segment is deleted from the record.

[0031] A second aspect of the present invention also provides an RDMA network data processing system based on driver middleware. The system includes a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0032] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned RDMA network data processing method based on driver middleware.

[0033] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention will become apparent from the description and the accompanying drawings.

[0034] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0035] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0036] Figure 1 This is a schematic diagram of one embodiment of the RDMA network data processing method based on driver middleware of the present invention;

[0037] Figure 2 This is a schematic diagram of another embodiment of the RDMA network data processing method based on driver middleware of the present invention.

[0038] Figure 3 A schematic diagram added to the identification field of this invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0040] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0041] With the maturity of RoCEv2 (RDMA over Converged Ethernet v2) technology, RDMA can be deployed on existing network infrastructure in data centers, making it the mainstream solution for high-speed network communication in data centers. As data center networks continue to expand, a single storage or computing cluster may contain tens of thousands of servers and associated network cards. For distributed storage clusters, the scale of distributed interconnected RDMA networks is enormous due to the limited storage space on a single machine. For distributed machine learning clusters, due to the limitations of GPU memory and computing power, distributed training and inference tasks based on data parallelism, pipelined parallelism, and tensor parallelism have become the mainstream. However, as the scale of storage clusters and AI training / inference clusters continues to expand, a series of performance issues will arise, limiting the scalability of training or storage clusters. The main reasons for these scalability limitations are as follows:

[0042] 1. Due to the requirements of lossless RDMA networks, RoCEv2 networks typically enable Priority-based Flow Control (PFC) to prevent packet loss. However, PFC uses hop-by-hop backpressure for flow control, which can lead to side effects such as PFC head-of-line blocking, PFC deadlock, and PFC storms. As RDMA service clusters such as HPC (High Performance Computing) and GPUs grow larger, the side effects of PFC become increasingly significant, resulting in decreased service performance and hindering large-scale RDMA deployment across Points of Demand (PODs).

[0043] 2. The scalability limitations of large-scale distributed services are mainly reflected in the limitations of connection expansion. Specifically, as the number of connections (or QPs) on the end-side network card increases, RDMA performance experiences a sharp drop. The root cause of this performance drop is that a large number of QP connections occupy a significant amount of on-board storage, leading to buffer overflow. Therefore, the network card needs to reread the lost cached data via PCIe. This behavior causes a sharp decrease in transmission throughput and an increase in FCT (Functional Cross-Counting). Large-scale distributed services typically use RC QPs for communication. Each pair of processes requires one pair of QPs. In a full-mesh, fully connected communication scenario with N servers and M processes per server, a single network card must handle O(M*M*N) QPs. As the scale of the distributed cluster continues to expand, the increased number of machines inevitably exacerbates the connection scalability problem.

[0044] 3. Corresponding to the RC QP connection type is the UD QP connection type. UD connections have the following characteristics: UD is a datagram-based communication method, allowing one connection to communicate with multiple other nodes, offering the advantage of connection expansion and reducing the number of QP connections on the network card from O(M*M*N) to O(M). However, UD does not have the reliability guarantee of RC; lost data cannot be retransmitted. Furthermore, the message size for a single transmission in a UD connection is limited to no more than the MTU, while the message size for a single transmission in an RC connection can reach up to 2GB.

[0045] The current state of RDMA networks in data centers is as follows: commercial RDMA network cards implement retransmission using a fallback-to-N packet loss recovery mechanism, while RoCEV2 networks ensure zero packet loss through PFC. Services use RC connections to guarantee transmission reliability. The scalability of the current network configuration is limited. Therefore, the industry has proposed many solutions, summarized in Table 1 below:

[0046] Table 1

[0047]

[0048] Hardware Solution: Existing technologies analyzed all data structures involved in the RDMA conceptual model and removed as many as possible through RDMA protocol header modifications and architectural innovations (including a cacheless QP scheduler and memory-free selective duplication). We implemented a full-featured SRNIC prototype using an FPGA, capable of supporting high-performance transmission over 10K connections. Existing technologies address RDMA scalability issues by transmitting connection state to the other end. Leveraging asymmetric communication patterns in data center applications, existing technologies allow the low-concurrency communication end to maintain state for the high-concurrency other end, thus putting the bottleneck RNIC in a stateless mode. Throughput was improved under high concurrency by implementing this on an FPGA board with 10Gbps network ports.

[0049] Software Solution: Existing technologies use shared connections to alleviate the problem of limited connection scalability. It utilizes QP multiplexing between threads within a node, then distributes the QPs to the target thread via the receiving thread. Lock-free queues are used to avoid contention, significantly reducing the number of QPs in the storage service cluster. XRC is a new transport type specified by IB to enhance scalability. In multi-core, multi-process FULL MESH communication, it reduces the number of connections from O(N*P*P) to O(N*P), a significant improvement for applications with a large number of processes.

[0050] Hardware and software combined solution: Existing technology uses network interface cards to implement the RD protocol stack to overcome the connection expansion problem, and moves out-of-order reordering to software middleware for coordination. This solution has been deployed in data center RDMA networks.

[0051] Existing technologies have drawbacks. While hardware solutions address both network and connectivity scalability from the network interface card (NIC) architecture perspective, this FPGA-based approach is difficult to deploy incrementally. New hardware solutions that abandon existing commercial NIC architectures all suffer from the aforementioned deployment difficulties. From a software perspective, QP inter-process sharing can alleviate this problem, but it introduces head-to-head blocking and lock contention issues. Furthermore, this approach doesn't decouple the number of machines, failing to fundamentally solve the scalability problem. Moreover, all the aforementioned software solutions require adaptation modifications to upper-layer applications, making transparent deployment impossible. Hardware-software hybrid solutions, from a protocol stack perspective, use datagram protocols and design self-developed NICs, deploying them in private networks through a hardware-software hybrid approach. While this solves the scalability problem, it cannot achieve universality across heterogeneous data centers.

[0052] like Figure 1 As shown, this invention proposes an RDMA network data processing method based on driver middleware. This method is applied to network card drivers and includes the following steps:

[0053] In practical implementation, the technical entry point of this invention is the end-side, applied to the network card driver, making it a purely end-side solution without any network-side design. The framework of this invention is mainly divided into two major modules: a queue management switching module and a selective retransmission module.

[0054] In the queue management switching module, step S100 involves obtaining the request for an RC type connection created by the application, and constructing a UD type connection request based on the RC type connection request.

[0055] In some embodiments of the present invention, RC type connection is a Reliable Connection, and UD type connection is an Unreliable Datagram.

[0056] In practice, datagram communication (UD) only requires one QP to complete one-to-many communication. All transmissions support SEND / RECV. RC and UC support writing, but only RC supports reading. In RC, RNIC is responsible for retransmission after loss. Conversely, in UC and UD, RNIC ignores packet loss.

[0057] Step S200: Obtain the RC type WQE issued by the application, combine multiple WQE into a data packet, divide the WQE in the data packet into large messages and ordinary messages based on a preset first length threshold, and divide the RC type large messages into multiple small messages with MTU (Maximum Transmission Unit) granularity.

[0058] In practice, if the data length of WQE is greater than the first length threshold, it is determined to be a large message; if it is not greater, it is determined to be a normal message.

[0059] In practice, the first length threshold can be the length value of the MTU.

[0060] In the specific implementation process, this solution uses a constructor to build data packets;

[0061] Specifically, the constructor's function is to connect WQEs on different threads of the same process through a singly linked list and send them to the splitter and scheduler for processing at once. The length of the constructed linked list is determined by the number of CQEs generated by the network card.

[0062] The specific construction logic is as follows:

[0063] 1. When the previous round of construction stops and enters the splitter and scheduler for processing, a new round of construction begins;

[0064] 2. During construction, all WQEs from different threads within the process are stored in a linked list in the order they arrived. Over time, the WQE linked list grows continuously.

[0065] 3. Assume that the previous round of WQE linked list generated M small messages with MTUs after being split. We call these M small messages a batch. The network card processes and sends the small messages in order, and each sent small message generates a CQE. When N CQEs are generated in this batch, the constructor is notified to stop construction, and the WQE linked list of this round is sent to the splitter and scheduler for processing.

[0066] In the specific implementation process, a segmenter is used to divide large messages into multiple smaller messages with an MTU (Maximum Transmission Unit) granularity.

[0067] Step S300: Calculate the weight of the large message based on its data length to determine whether it belongs to the short stream queue or the long stream queue. Combine the large messages that belong to the short stream queue with ordinary messages to construct a short stream queue; construct the large messages that belong to the long stream queue into a long stream queue.

[0068] Step S400: Combine the short-flow queue and the long-flow queue into a sending queue, pass the ordinary messages or small messages in the sending queue to the UD type connection that is already in the RTS state, add number and identification fields to each ordinary message or small message in sequence, and send it to the destination.

[0069] In the specific implementation process, a scheduler is used to complete step S400.

[0070] In the retransmission module, step S500 involves obtaining feedback information from the destination and determining whether data retransmission is required based on the feedback information. If data retransmission is required, the corresponding ordinary message or small message is retransmitted.

[0071] Using the above scheme, this scheme is applied to the network card driver and the network card driver is adjusted. The UD type connection request is constructed based on the RC type connection request. The UD type connection is different from the RC type connection. For the RC type connection, both communicating parties need to establish a pair of QP (Queue Pair) for communication. For the UD type connection, only one one-to-many QP needs to be built at each end. It does not need to build a large number of QPs, which ensures the stability of network scaling.

[0072] In some embodiments of the present invention, after the step of constructing a UD type connection request based on an RC type connection request, the method further includes the step of converting the RC connection type connection information into the connection information required by the UD connection type, and converting the state machine of the UD connection from the initial state to the RTS state.

[0073] In some embodiments of the present invention, in the step of dividing a large RC-type message into multiple small messages of MTU granularity, the large message is cached and the position of each small message in the large message is recorded.

[0074] In some embodiments of the present invention, the weight of a large message is calculated based on its data length to determine whether it belongs to a short-flow queue or a long-flow queue. Large messages belonging to the short-flow queue are combined with ordinary messages to construct a short-flow queue. The step of constructing large messages belonging to the long-flow queue into a long-flow queue further includes:

[0075] like Figure 2 As shown, in step S310, for a large message, a first determination value is calculated based on the largest weight in the data packet and the weight of the large message, and the first determination value is compared with a first threshold. A second determination value is calculated based on the length of the smallest ordinary message in the data packet and the length of the large message, and the second determination value is compared with a second threshold.

[0076] In the specific implementation process, the weight of all large messages or ordinary messages in all data packets is calculated by calculating the weight of large messages. The largest weight in the data packet is the maximum value among all weights, and the length of the smallest ordinary message in the data packet is the data length of the ordinary message with the smallest data length in the data packet.

[0077] Step S320: If for a large message, the first judgment value is greater than the first threshold and the second judgment value is less than or equal to the second threshold, then the large message is determined to belong to the short-flow queue, and the SJF algorithm is used to calculate the large message and ordinary messages to complete the reordering.

[0078] In practice, the SJF algorithm is the shortest completion time algorithm.

[0079] In the specific implementation process, when using the SJF algorithm to calculate the large message and ordinary messages and complete the reordering, the goal is to ensure that the completion time of all WQEs in the short-flow queue is minimized.

[0080] The start time is denoted as T0, which is the time when the WQE enters the constructor. The index i represents the WQE sequence after each reordering. `rate` represents the network interface card's transmission rate, the hardware completion time of the previous WQE, the completion time of the 0th WQE, the time the Tdoorbell packet enters the scheduler, and the time the i-th WQE enters the constructor.

[0081] ΔT i =size i / rate+ΔT i-1 ;

[0082]

[0083] Where, ΔT i Let ΔT represent the software completion time of the i-th WQE. i-1 Represents the software completion time of the (i-1)th WQE, size i This represents the data length of the i-th WQE, and rate represents the transmission rate of the network interface card. This represents the hardware completion time of the i-th WQE. T represents the time when the i-th WQE enters the constructor. doorbell This indicates the time when the data packet enters the scheduler.

[0084] The input to the SJF algorithm is calculated based on the above formula. The SJF algorithm is then used to rearrange the WQE, find the optimal solution that satisfies the target, and obtain the sorted short-flow queue.

[0085] Step S330: If for a large message, the first judgment value is greater than the first threshold and the second judgment value is greater than the second threshold, then the large message is determined to belong to the short-flow queue and placed after the reordered large messages and ordinary messages.

[0086] Step S340: If for a large message, the first determination value is less than or equal to the first threshold, then the large message is determined to belong to the long queue, and the large messages in the long queue are reordered using a weight-based polling algorithm.

[0087] In practice, the weight-based polling algorithm is the Weight Round Robin algorithm.

[0088] Using the above scheme, this scheme first makes a comprehensive judgment based on weight and data length. According to the comprehensive judgment result, large messages or ordinary messages in the data packet are placed in the long flow queue or short flow queue to ensure the transmission efficiency of subsequent data transmission.

[0089] In some embodiments of the present invention, in the step of determining whether a large message belongs to a short-flow queue or a long-flow queue by calculating the weight of the large message based on its data length, the weight of the large message is calculated based on the following formula:

[0090] W = L + L MTU -1 / L MTU ;

[0091] Where W represents the weight, L represents the length of the large message, and L MTU This indicates the length of a unit MTU.

[0092] In some embodiments of the present invention, in the step of calculating a first determination value based on the largest weight in the data packet and the weight of the large message, the ratio of the largest weight in the data packet to the weight of the large message is calculated as the first determination value; in the step of calculating a second determination value based on the length of the smallest ordinary message in the data packet and the length of the large message, the ratio of the length of the large message to the length of the smallest ordinary message in the data packet is calculated as the second determination value.

[0093] In some embodiments of the present invention, in the step of combining the short-flow queue and the long-flow queue into a sending queue, the short-flow queue is placed before the long-flow queue to form a sending queue.

[0094] Using the above scheme, the short-flow queue is placed before the long-flow queue in the sending queue. The WQE in the short-flow queue often has a faster sending speed, so it is processed first in the sending queue to ensure processing efficiency.

[0095] In some embodiments of the present invention, the feedback information includes confirmation of receipt of a small message. In the step of obtaining the feedback information from the destination and determining whether data retransmission is required based on the feedback information, if confirmation of receipt of a normal message or a small message is received within a preset time period, it is determined that the corresponding normal message or small message has been successfully received.

[0096] In some embodiments of the present invention, in the step of obtaining feedback information from the destination and determining whether data retransmission is required based on the feedback information, if no confirmation of receipt of a normal message or a small message is received within a preset time period, the corresponding normal message or small message will be retransmitted.

[0097] In some embodiments of the present invention, in the step of retransmitting the corresponding ordinary message or small message, if a small message is retransmitted, the number of the corresponding small message is obtained, the position of the small message in the large message is determined based on the number of the small message, the corresponding small message is extracted from the large message based on the position of the small message in the large message, and the small message is sent to the destination.

[0098] By adopting the above scheme, this scheme can save large messages as a whole, ensuring their integrity and preventing data distortion caused by segmentation. In the process of data retransmission, the original large messages are re-segmented, which facilitates the preservation of large messages and ensures the accuracy of retransmitted messages.

[0099] like Figure 3 As shown, in some embodiments of the present invention, in the step of adding numbering and identification fields to each ordinary message or small message in sequence and sending it to the destination, a preset identification field is inserted before the payload.

[0100] In some embodiments of the present invention, the large message is divided into multiple processing segments. In the step of obtaining feedback information from the destination, the processing segments are traversed from front to back. If all the small messages in the processing segment have been received, the corresponding processing segment is deleted from the cached large message, and the length of the deleted processing segment is recorded.

[0101] Using the above scheme, large messages are segmented. After the previous segment is received, the preceding segment data can be deleted first, and the length of the deleted data is recorded. On the one hand, the buffer area can be used efficiently, and on the other hand, the accuracy of subsequent data processing can be ensured by recording the length of the deleted data, avoiding errors caused by data deletion.

[0102] In some embodiments of the present invention, in the step of obtaining feedback information from the destination, after all the small messages in the last processing segment of the cached large message have been received, the last processing segment is deleted, and the length of each processing segment is deleted from the record.

[0103] In the specific implementation process, the processing steps of the queue management switching module are as follows:

[0104] 1. Upper-layer applications call the Verbs interfaces provided by the underlying user-space Libbverbs library through communication libraries such as Libfabric, UCX, and NCCL. Verbs consists of two parts: control path interfaces and data path interfaces. The control path includes interfaces for protection domain creation, memory registration, connection creation, and state machine modification; the data path includes interfaces for WQE message delivery and completion queue polling.

[0105] 2. The application first allocates a protection domain by calling ibv_alloc_pd and registers memory with ibv_reg_mr for resource creation during the communication initialization phase. This process is directly embedded from user space into kernel space via system calls and allocates and creates resources for communication with the hardware network card.

[0106] 3. When an application communicates with a remote node, it needs to establish a reliable connection. When the application requests an RC type connection from the user-space driver through the Verbs interface ibv_create_qp, the driver middleware intercepts the RC type request through the QP connection type overriding module and changes the connection request to UD type. It registers a UD type QP context (QPC) in the underlying hardware RDMA network card. At this time, the QP connection is in the initial INIT state and returns a pointer to the UD connection to the upper-layer application. The upper-layer application is unaware that the connection type has been modified and will still perform subsequent processing based on the RC type connection.

[0107] 4. When the application modifies the state machine of the connection by passing the RC connection type parameters through the Verbs control path interface ibv_modify_qp, the driver middleware also obtains the passed parameters and converts them into the parameters required for the UD connection type (UD type parameters are a subset of RC type parameters), and gradually changes the connection state from the initialization state INIT to the RTR and RTS states. At this time, the connection has the ability to send and receive.

[0108] 5. After steps 1, 2, 3, and 4 above, the creation and configuration of communication resources and communication queue connections are complete. When the application sends a WQE to the connection pointer returned in step 3 through the data path interface ibv_post_send, the driver middleware first records and caches the address of the WQE message in memory and the continuous length of the data in memory, placing it in a dedicated buffer queue for data backup by the packet loss retransmission module.

[0109] 6. In order to meet the message size limit of UD connection type, the above WQE needs to be split after backup, dividing the memory size specified by WQE into memory blocks with MTU granularity.

[0110] 7. Encapsulate the memory blocks divided in step 6 into individual data frames, and fill in the custom header fields according to the order of segmentation. Figure 2 Fill in the four fields shown. PKT_TYPE represents the packet type, MSG_ID represents the message sequence number before processing by the message segmentation module in step 6, SGE_LEN represents the length of the segmented sub-data, and SGE_OFFSET represents the byte offset of the sub-data fragment within the entire original data. The message carrying the user-defined message fields is encapsulated into a complete payload and sent to the UD connection using a WQE specified by a uniform MTU size. The UD connection uses a doorbell mechanism to notify the network card to send data packets to the network. At this time, the bitmap is initialized to all 0s, and the timer is initialized. Steps 5, 6, and 7 are repeated as the application continuously sends WQE messages.

[0111] In the user-mode select retransmission module's processing steps,

[0112] 1. After receiving a data packet, the receiving end parses the packet's continuity using a custom field carried in the payload and sends the received packet information to the sender by constructing a user-space ACK response message. When out-of-order packet loss is detected, a NAK is immediately constructed and sent to the sender. The NAK only carries the sequence number of the lost data packet for the sender to selectively retransmit.

[0113] 2. The sending end receives the ACK / NAK signal returned from the receiving end through the control queue CTRL QP.

[0114] 3. Upon receiving the ACK signal, parse the sequence number of the data packet carried in the ACK control signal packet, match the sequence number with the corresponding bit in the bitmap, and set the bit to 1 to indicate that the corresponding data packet has been received correctly.

[0115] 4. When a NAK signal is received, the sequence number of the data packet corresponding to the NAK is parsed, which means that the data packet corresponding to the sequence number has been lost. The corresponding data is immediately extracted from the message buffer queue in step 5 of module one, encapsulated into a WQE message payload, and the lost data packet is retransmitted through a separate headless blocking retransmission queue RETRY QP.

[0116] 5. If the ACK reception time in step 3 exceeds a certain threshold and the bitmap is still not entirely set to 1, it indicates that a data packet timeout event has occurred. At this time, the sequence number of the timed-out data packet is determined based on the empty positions in the bitmap. This sequence number is used to determine the location of the lost data packet in memory. The data address and length are retrieved from the buffer queue in step 5 of module one, and the packet is retransmitted through the retransmission queue. Simultaneously, the timer is reset during retransmission to prevent the next timeout event.

[0117] 6. Each sub-message sent in batches by the sending end will generate an ACK or NAK at the receiving end. After the sending end receives these control messages, it will repeat steps 3 and 4 according to the control signal type.

[0118] In its implementation, this invention addresses the scalability issues present in current large-scale RDMA networks and proposes a technical approach to reduce the deployment difficulty of data centers.

[0119] The scalability and deployment issues to be addressed can be summarized as follows:

[0120] 1. Network scalability in large-scale networks, namely, eliminating the negative effects of PFC such as head-of-line congestion, PFC storm, and PFC deadlock in the current network, and providing high-performance transmission guarantees to improve network scalability.

[0121] 2. Scalability of connections in large-scale networks, that is, supporting more connections per network card, avoiding performance degradation caused by frequent cache loss under large-scale service connection numbers, and supporting more service connections under large-scale networking.

[0122] 3. This invention needs to be easily and incrementally deployed in existing data centers. No modifications are required to the network interface card (NIC) hardware, and no interface changes are needed for upper-layer applications; deployment is achieved simply by adding a NIC user-space driver middleware.

[0123] By adopting the above scheme, this approach eliminates the drawbacks of PFC deadlock and storms by disabling PFC, overcoming network scalability constraints in large-scale RDMA networking. A user-space selective retransmission scheme is designed to improve recovery efficiency in the event of packet loss, increasing throughput and flow completion time in loss-tolerant networks. By converting RC connections to UD connections and adapting the underlying driver, the storage resources occupied by the underlying connection context are reduced, allowing more connections to be accommodated on a single network card, lowering the frequency of connection context cache loss in large-scale networks, and improving connection scalability. The scheme is designed from a user-space driver perspective, requiring no modification to the underlying hardware. It is also transparent to upper-layer services, requiring no modification to the upper-layer service interfaces.

[0124] In this plan:

[0125] 1. The present invention proposes a technical solution for a connectivity scalable system implementation for data center RDMA networks. Specifically, it includes a connection type conversion function based on the RDMA network card user-space driver (including implementation methods for state machine transitions and message slicing processing).

[0126] 2. The present invention proposes a network scalability system implementation solution for data center RDMA networks. Specifically, it includes a selective retransmission function implemented based on the RDMA network card user-space driver (including the implementation of a custom header field carried in the load and the execution method of the end-side retransmission queue).

[0127] This invention also provides an RDMA network data processing system based on driver middleware. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0128] This invention also provides an RDMA network data processing system based on driver middleware. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0129] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned RDMA network data processing method based on driver middleware. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0130] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0131] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0132] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0133] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for RDMA network data processing based on driver middleware, characterized in that, This method is applied to network card drivers and includes the following steps: Get the request for the RC type connection created by the application, and construct the request for the UD type connection based on the RC type connection request; Get the RC type WQE issued by the application, combine multiple WQE into a data packet, divide the WQE in the data packet into large messages and ordinary messages based on a preset first length threshold, and divide the RC type large messages into multiple MTU granular small messages. The weight of a large message is calculated based on its data length to determine whether it belongs to a short-flow queue or a long-flow queue. Large messages belonging to the short-flow queue are combined with ordinary messages to form a short-flow queue; large messages belonging to the long-flow queue are formed into a long-flow queue. For a large message, a first determination value is calculated based on the largest weight in the data packet and the weight of the large message. This first determination value is compared with a first threshold. A second determination value is calculated based on the length of the smallest ordinary message in the data packet and the length of the large message. This second determination value is compared with a second threshold. If, for a large message, the first determination value is greater than the first threshold and the second determination value is less than or equal to the second threshold, then the large message is determined to belong to the short-flow queue, and the SJF algorithm is used to reorder the large message and ordinary messages. If, for a large message, the first determination value is greater than the first threshold and the second determination value is greater than the second threshold, then the large message is determined to belong to the short-flow queue and placed after the reordered large messages and ordinary messages. If, for a large message, the first determination value is less than or equal to the first threshold, then the large message is determined to belong to the long-flow queue, and a weighted round-robin algorithm is used to reorder the large messages in the long-flow queue. The short-flow queue and long-flow queue are combined into a sending queue. The ordinary messages or small messages in the sending queue are passed to the UD type connection that is already in the RTS state. Number and identification fields are added to each ordinary message or small message in sequence and sent to the destination. Obtain feedback information from the destination, and determine whether data retransmission is required based on the feedback information. If data retransmission is required, retransmit the corresponding ordinary message or small message.

2. The RDMA network data processing method based on driver middleware according to claim 1, characterized in that, Following the step of constructing a UD type connection request based on an RC type connection request, the process further includes steps of converting the RC connection type establishment information into the establishment information required by the UD connection type, and transitioning the UD connection state machine from the initial state to the RTS state.

3. The RDMA network data processing method based on driver middleware according to claim 1, characterized in that, In the step of dividing a large RC-type message into multiple smaller messages with MTU granularity, the large message is cached and the position of each smaller message within the large message is recorded.

4. The RDMA network data processing method based on driver middleware according to claim 1, characterized in that, In the step of determining whether a large message belongs to the short or long queue by calculating its weight based on its data length, the weight of the large message is calculated based on the following formula: in, This represents the weight, and L represents the length of the large message. This indicates the length of a unit MTU.

5. The RDMA network data processing method based on driver middleware according to claim 1, characterized in that, In the step of calculating the first judgment value based on the largest weight in the data packet and the weight of the large message, the ratio of the largest weight in the data packet to the weight of the large message is calculated as the first judgment value; in the step of calculating the second judgment value based on the length of the smallest ordinary message in the data packet and the length of the large message, the ratio of the length of the large message to the length of the smallest ordinary message in the data packet is calculated as the second judgment value.

6. The RDMA network data processing method based on driver middleware according to claim 1, characterized in that, In the step of combining the short-flow queue and the long-flow queue into a sending queue, the short-flow queue is placed before the long-flow queue to form a sending queue.

7. The RDMA network data processing method based on driver middleware according to any one of claims 1-6, characterized in that, The feedback information includes confirmation of receipt of small messages. In the step of obtaining feedback information from the destination and determining whether data retransmission is needed based on the feedback information, if confirmation of receipt of ordinary messages or small messages is received within a preset time period, it is determined that the corresponding ordinary message or small message has been successfully received. If no confirmation of receipt of a regular message or a small message is received within the preset time period, the corresponding regular message or small message will be retransmitted.

8. The RDMA network data processing method based on driver middleware according to claim 7, characterized in that, The large message is divided into multiple processing segments. In the step of obtaining feedback information from the destination, the processing segments are traversed from front to back. If all the small messages in the processing segment have been received, the corresponding processing segment is deleted from the cached large message, and the length of the deleted processing segment is recorded. When all the small messages in the last processing segment of the cached large message have been received, the last processing segment is deleted, and the length of each processing segment is deleted.

9. A RDMA network data processing system based on driver middleware, characterized in that, The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1-8.

Images