A user-mode-based RDMA network QoS coordination method

Abstract

The application provides a user state-based RDMA network QoS coordination method, which comprises the following steps: a sending end configures different QoS initial parameters for different service flows, and sends data packets with the same sending window size; a receiving end carries a network congestion control signal in an ACK packet returned after receiving the packet; the sending end acquires the network congestion control signal; the sending end adjusts the sending window by using a user state congestion control algorithm; the sending end periodically reports QoS parameters and service quality of the service flow to a correction node; and the correction node adjusts the QoS parameters by using a parameter adjustment algorithm according to the QoS parameters and service quality of the service flow reported by the sending end, and delivers the QoS parameters to the sending end. The application performs fine-grained control on a single service flow, so that more priority levels are realized, and the correction module can recover the pre-set QoS as soon as possible, so that good fairness, convergence and stability are achieved.

Description

Technical Field

[0001] This invention belongs to the field of network communication technology, and particularly relates to a user-space-based RDMA network QoS coordination method. Background Technology

[0002] With the rapid development of next-generation information technologies such as cloud computing, big data, and AI, high-performance applications such as distributed machine learning, distributed storage, and cloud computing are placing higher demands on data center network transmission performance. Remote Direct Memory Access (RDMA) technology provides high-speed network performance with low latency and high bandwidth through host-side kernel bypass zero-copy and hardware offloading of the software protocol stack, and is widely used in data centers and large-scale distributed systems.

[0003] In data center networks, Quality of Service (QoS) refers to traffic prioritization and resource reservation control mechanisms, with detailed metrics including latency, bandwidth, jitter, and packet loss rate. Because data centers deal with arbitrarily changing traffic scenarios, different network flows need to coordinate service levels to improve network efficiency, depending on the service protocol.

[0004] Current commercial Ethernet RDMA (RDMA over Converged Ethernetv2, RoCEv2) uses a Priority Flow Control (PFC) mechanism based on the IP header's DSCP field to isolate different service traffic and prioritize high-priority services. However, the PFC standard specifies eight priority categories, but due to issues such as PFC response time, propagation delay, and small switch buffers, current network environments can only utilize two of these eight priorities for RDMA. Even if all are used, it is difficult to meet the needs of data centers with mixed cloud services, leading to QoS ambiguity and frequent failures. Therefore, further research and optimization of RDMA network QoS mechanisms are needed to meet the ever-changing data center network requirements.

[0005] Existing QoS technologies are typically designed for specific systems, making it difficult to find a unified solution suitable for RDMA networks. This limits the application of QoS technologies. Furthermore, switch-based QoS schemes involve rate control and priority scheduling within switch queues. However, schemes like HCSFQ and AIFO discard packets based on optimization objectives, contradicting the lossless underlying network characteristics upon which RDMA networks rely. Therefore, optimization and improvements specifically for RDMA networks are necessary.

[0006] In conclusion, a complete and efficient data center RDMA network environment still requires QoS support. To address the challenges of cloud data center resource management, it is necessary to explore more granular QoS schemes to differentiate RDMA network service traffic, given the diversity of data center and application architectures. Summary of the Invention

[0007] This invention addresses the shortcomings of existing technologies by providing a user-space-based RDMA network QoS coordination method, comprising:

[0008] The sending end configures different QoS initial parameters for different service flows and sends data packets with the same sending window size;

[0009] After receiving the message, the receiving end carries the network congestion control signal in the returned ACK message;

[0010] The sending end receives ACK packets and obtains network congestion control signals;

[0011] The transmitting end adjusts the transmitting window based on the congestion control signal and the initial QoS parameters using a user-space congestion control algorithm.

[0012] The sending end periodically reports the QoS parameters and service quality of the service flow to the correction node;

[0013] The correction node adjusts the QoS parameters based on the QoS parameters and service quality of the service flow reported by the sender using a parameter adjustment algorithm, and then sends the adjustment parameters back to the sender.

[0014] Furthermore, the sending end configures different QoS initial parameters for different service flows and sends data packets with the same sending window size, including:

[0015] A user-space data transmission deployment scheme combining unreliable connections and unreliable datagrams using RDMA is adopted. Based on user-space reliability mechanisms and user-space congestion control, initial QoS parameters are configured for different service traffic priorities. target_delay, αi, and hαi are used as QoS impact parameters. Among them, target_delay is the expected delay of service traffic. The smaller the value of target_delay, the smaller the tolerance for queuing delay of data packets in the network. αi and hαi are the speed adjustment granularity of the user-space congestion control algorithm in the service flow growth stage. The larger the values ​​of αi and hαi, the faster the speed increase.

[0016] Furthermore, after receiving the message, the receiving end carries a network congestion control signal in the returned ACK message, including:

[0017] Round-trip time is used as the congestion control signal. The receiver carries the timestamp information of the relevant reception / transmission events in the ACK message and feeds it back to the sender as a congestion control signal.

[0018] Furthermore, the sending end receives the ACK message and obtains the network congestion control signal, including:

[0019] The sending end receives the congestion control signal carried in the ACK packet, and, in conjunction with the relevant time measured by the local network card, uses a hardware and software co-operated RTT precision measurement technology to calculate the network round-trip time value.

[0020] Furthermore, the transmitting end adjusts the transmitting window using a user-space congestion control algorithm based on the congestion control signal and in conjunction with the initial QoS parameters, including:

[0021] The sending end compares the calculated actual network round-trip time value measure_delay with the pre-configured target_delay, and uses an additive, multiplicative, and subtractive user-space congestion control algorithm to adjust the sending window of the service traffic in order to increase or decrease the speed.

[0022] Furthermore, the sending end periodically reports the QoS parameters and service quality of the service flow to the correction node, including:

[0023] A separate correction node is used to collect service traffic information from the sending end and correct erroneous events. The sending end triggers a collection and reporting event at each time interval, reporting the QoS parameters, bandwidth, latency and sending window size of the service flow to the correction node.

[0024] Furthermore, the correction node adjusts the QoS parameters using a parameter adjustment algorithm based on the QoS parameters and service quality of the service flow reported by the sending end, and then sends the adjusted parameters back to the sending end, including:

[0025] The correction node adjusts the QoS parameters based on the QoS parameters and service quality of the service flow reported by the sender, using a parameter adjustment algorithm, and sends the parameter modification suggestions from the sender back to the sender to reduce QoS ambiguity and errors.

[0026] This invention provides a user-space-based RDMA network QoS coordination method that requires only one queue on the endpoint and one on the switch, without relying on changes to the switch. It only requires corresponding configuration on the host side, making it highly portable. This invention enables fine-grained control over individual service traffic to achieve more priority levels. Furthermore, the correction module of this invention can quickly restore the pre-set QoS, achieving excellent fairness, convergence, and stability. Attached Figure Description

[0027] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0028] Figure 1 A flowchart of a user-space-based RDMA network QoS coordination method provided in this embodiment of the invention;

[0029] Figure 2 A structural diagram of a user-space-based RDMA network QoS coordination system provided in an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of an RDMA user-space deployment scheme based on UC and RD provided in an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of the RTT measurement timestamp provided in an embodiment of the present invention;

[0032] Figure 5 A flowchart of a user-space-based congestion control algorithm provided for an embodiment of the present invention;

[0033] Figure 6 A flowchart of the QoS parameter correction algorithm provided in an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] like Figure 1 As shown, this embodiment of the invention provides a user-space-based RDMA network QoS coordination method, including:

[0036] Step 101: The sending end configures different QoS initial parameters for different service flows and sends data packets with the same sending window size.

[0037] Due to the non-programmability of RNIC (RDMA-capable network adapter), this embodiment of the invention adopts a user-space data transmission deployment scheme that combines RDMA's Unreliable-Connection (UC) and Unreliable Datagram (UD). This scheme supports user-space reliability mechanisms and congestion control mechanisms, solving the pain points of non-programmability of network cards and the inability to modify hardware after unloading. It enables the management and control of service traffic at the software, i.e., user-space level.

[0038] like Figure 3 As shown, this embodiment of the invention uses the UC communication mode of RDMA to send data packets. Compared with the RC (Reliable Connection) communication mode, it reduces hardware reliability overhead. However, the software layer (user layer) uses a separate UD queue pair (QP) to send the ACK message and places the UD QP with high priority. This avoids the header blocking problem caused by data packets and speeds up congestion feedback. Previously, due to the non-programmable nature of the RDMA network card RNIC, RDMA network QoS could only be guaranteed by priority queue division, using intermediate switches and network card priorities. This method had a coarse granularity and made it difficult to manage each service traffic individually. Now, the user-space-based data transmission scheme can achieve individual management of each service flow at the application layer.

[0039] Based on different service priority requirements, latency-sensitive flows, online searches, and financial transactions can be classified as high priority, while background flows such as data copying / backup will be classified as low priority. Different priority service flows are configured with different QoS parameters, including target_delay, αi, and hαi. Target_delay represents the expected delay of the service flow and is the primary QoS parameter. Target_delay is the sum of base_delay and queue_delay. Base_delay represents the propagation delay of the point-to-point link between the sender and receiver, which is a fixed value. Queue_delay represents the service flow's tolerance for queuing delay in the network. A smaller target_delay indicates a lower tolerance for queuing delay, thus accelerating the deceleration of the service flow and lowering its priority. Conversely, a higher target_delay allows for greater tolerance for queuing delay, enabling the service flow to be sent at a higher rate and increasing its priority. αi and hαi are the granularity of the user-space congestion control algorithm during the service flow acceleration phase; a larger value indicates a faster acceleration. When priority control based on `target_delay` causes QoS ambiguity, these two parameters can be modified. A larger parameter allows for faster growth of service flows, while a smaller parameter results in slower growth, enabling the differentiation of multiple service flows within a short time. The initial value of the sending window is set to the size of a Bandwidth Delay Product (BDP), and the sending end initially sends data packets at line rate (maximum sending rate).

[0040] Step 102: After receiving the message, the receiving end carries the network congestion control signal in the returned ACK message.

[0041] In this step, round-trip time is used as the congestion control signal. The receiver carries the timestamp information of the relevant receive / send events in the ACK packet, which is fed back to the sender as a congestion control signal.

[0042] Step 103: The sending end receives the ACK message and obtains the network congestion control signal.

[0043] In this step, the sending end receives the congestion control signal carried in the ACK message, and, in conjunction with the relevant time measured by the local network card, uses a hardware and software co-operated RTT precision measurement technology to calculate the network round-trip time value.

[0044] In steps 102-103, Round-Trip Time (RTT) is used as the congestion control signal. RTT, as a simple and effective congestion signal, has been deployed and verified on a large scale. RDMA utilizes the hardware timestamp function of the RNIC to obtain the timestamp of the data packet or acknowledgment packet completion event to calculate the RTT. RDMAWQE (Work Queue Element) is used to describe the relevant parameters and data of RDMA operation. When transmitting / receiving data, RDMA first sends a WQE to the network card. After the network card reads the WQE, the hardware submits a CQE (Completion Queue Element) to the CQ (Completion Queue) to notify the upper layer of the WQE completion status. RDMA completion events include: TX Completion (transmission completed) and RX Completion (reception completed). When data is successfully sent, a TX Completion transmission success event is generated. When data is successfully received by the other end, an RX Completion reception success event is generated. Figure 4 As shown, after a data packet is sent from the sending network card, a TX Completion event is generated. The hardware timestamps the TX Completion, which can then be obtained from the CQE in the CQ, denoted as t1. When it is received by the remote network card, an RX Completion event is generated, and the hardware timestamps the RX Completion, denoted as t2. After processing the data packet, the remote host sends an ACK packet, with the sending time denoted as t4. When the network card sends the data packet, a TX Completion event is generated, with a timestamp of t5. When the sending network card receives the data packet, an RX Completion event is generated, with a timestamp of t6. Since the receiving host can only obtain the hardware timestamp of the ACK TX Completion event after the ACK has been sent, the sending end can no longer perceive t4. Therefore, in this embodiment of the invention, the result of (t6-t1)-(t4-t2) is used as the actual measured RTT value, ignoring the queuing delay at the host end. However, because a high-priority UD QP is used as the transmission connection for the ACK response packet, the congestion signal (delay information) can be fed back more quickly, so the impact is small.

[0045] Step 104: The sending end adjusts the sending window using a user-space congestion control algorithm based on the congestion control signal and the initial QoS parameters.

[0046] The sending end compares the calculated actual network round-trip time value measure_delay with the pre-configured target_delay, and uses an additive, multiplicative, and subtractive user-space congestion control algorithm to adjust the sending window of the service traffic in order to increase or decrease the speed.

[0047] For example, this invention employs a user-space congestion control algorithm, modified from Google's Swift algorithm. The core idea is based on AIMD (Additive Increase Multiplicative Decrease), which aims to make the measured RTT (measure_delay) close to the target RTT (target_delay). Figure 5 As shown in the diagram. In the algorithm, cwnd represents the sending window size, β is the multiplicative decrement factor, max_mdf is the maximum multiplicative decrement coefficient, num_acked is the number of successfully received packets, and t_last_decrease is the time of the last multiplicative window reduction. When an ACK packet is received, a congestion window algorithm is triggered, which quickly adjusts congestion based on measure_delay. When measure_delay is less than target_delay, additive windowing is performed; when measure_delay is greater than target_delay, multiplicative windowing is performed, with the reduction depending on the distance to target_delay. The multiplicative windowing algorithm is limited to occurring only once per RTT, thus preventing multiple responses to the same congestion event. The time of the last multiplicative windowing is recorded, and it is checked whether the distance is now greater than one RTT. αi is the additive increase factor, and hαi is the super-additive increase factor. In this invention, hαi is used instead of αi to flexibly adjust the windowing speed.

[0048] Step 105: The sending end periodically reports the QoS parameters and service quality of the service flow to the correction node.

[0049] In this step, a separate correction node is used to collect service traffic information from the sending end and correct erroneous events. At each time interval, the sending end triggers a collection and reporting event, reporting the QoS parameters, bandwidth, latency, and sending window size of the service flow to the correction node.

[0050] Step 106: The correction node adjusts the QoS parameters according to the QoS parameters and service quality of the service flow reported by the sender using a parameter adjustment algorithm, and then sends the adjusted parameters to the sender.

[0051] In steps 105-106, QoS errors may occur in practice, where the existing QoS differs from the expected QoS. This is due to the randomness of the windowing sequence and the influence of the `num_acked` parameter in the windowing algorithm. This embodiment of the invention adopts a centralized control approach, utilizing a separate correction server node to collect data and modify "error" events. Every time interval T, the sending node (end) triggers a collection and reporting event, reporting the QoS parameters (target_delay, αi, hαi), bandwidth (bw), latency (measure_delay), and sending window size (cwnd) of the service flow to the correction node. The modification node adjusts the QoS parameters based on the service flow information reported by the sending end, using a parameter adjustment algorithm, and sends the parameter modification suggestions from the error node (sender) back to the error node (sender) to reduce QoS ambiguity and errors. If the same modification is performed more than N times, the error node is abandoned.

[0052] like Figure 6 As shown, the correction node sorts the collected information list `info_list` in reverse order by `target_delay`, then iterates through the information set. When the `bw` value of an entry with a larger `target_delay` is smaller than the `bw` value of the next entry with a smaller `target_delay`, a "mistake" event occurs. The information of the entry with the larger `target_delay` is recorded as the "mistake" entry. When a "mistake" occurs, the correction node adjusts the parameters. When the measured `measure_delay` in the "mistake" entry is less than or equal to `target_delay`, it indicates that the windowing phase is underway. When the maximum window limit has not been reached, `αi` and `hαi` can be adjusted to increase these two values, allowing the erroneous connection to have a faster windowing opportunity. Here, both values ​​are incremented by one simultaneously, completing one correction (non-aggressive adjustment mode). When the maximum window limit has been reached, `target_delay` can be adjusted to increment by 1, and it is checked whether the latest value exceeds the `target_delay` value of the entry in `info_list` preceding the "mistake" entry. If it does, no modification can be made. When the measured `measure_delay` in the "Error" entry is greater than `target_delay`, it indicates that the window reduction phase is underway. When the minimum window limit is reached, `target_delay` can be adjusted by decreasing it by 1. It is then checked whether the latest value is lower than the `target_delay` value in the next entry in `info_list` after the "Error" entry. If it is lower, no modification can be made. After the above algorithm processing, the correction node will provide suggestions for parameter modifications to the error node.<target_delay,αi,hαi> And send this information to the faulty node.

[0053] This invention provides a user-space-based RDMA network QoS coordination method that requires only one queue on the endpoint and the switch, does not depend on changes to the switch, and only requires corresponding configuration on the host side, thus exhibiting high portability. This invention enables fine-grained control over individual service traffic to achieve more priority levels. Simultaneously, the correction module of this invention can quickly restore the pre-set QoS, achieving excellent fairness, convergence, and stability.

[0054] This invention also provides a user-space-based RDMA network QoS coordination system, such as... Figure 2 As shown, the coordination system includes a control plane 201 and a data plane 202. The control plane 201 includes an RTT measurement module 204 and a user-space congestion control and parameter configuration module 205. The data plane 202 includes a receiver 203 and a transmitter 206. The receiver receives data packets sent by the transmitter, and the transmitter receives ACK response packets sent by the receiver. The RTT measurement module, after receiving the congestion signal from the receiver's ACK, calculates the actual network RTT using a hardware-software co-operated RTT precision measurement technique and feeds it back to the congestion control module. The user-space congestion control and parameter configuration module calculates the transmission window based on QoS parameters and RTT delay information using a congestion control algorithm.

[0055] The present invention has been described in detail above with reference to specific embodiments and exemplary examples; however, these descriptions should not be construed as limiting the present invention. Those skilled in the art will understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and embodiments of the present invention without departing from the spirit and scope of the invention, and all such modifications and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A user-space-based RDMA network QoS coordination method, characterized in that, include: The sending end configures different QoS initial parameters for different service flows and sends data packets with the same sending window size; After receiving the message, the receiving end carries the network congestion control signal in the returned ACK message; The sending end receives ACK packets and obtains network congestion control signals; The transmitting end adjusts the transmitting window based on the congestion control signal and the initial QoS parameters using a user-space congestion control algorithm. The sending end periodically reports the QoS parameters and service quality of the service flow to the correction node; The correction node adjusts the QoS parameters based on the QoS parameters and service quality of the service flow reported by the sender using a parameter adjustment algorithm, and then sends the adjustment parameters back to the sender.

2. The user-space-based RDMA network QoS coordination method according to claim 1, characterized in that, The sending end configures different QoS initial parameters for different service flows and sends data packets with the same sending window size, including: A user-space data transmission deployment scheme combining unreliable connections and unreliable datagrams using RDMA is adopted. Based on user-space reliability mechanisms and user-space congestion control, initial QoS parameters are configured for different service traffic priorities. target_delay, αi, and hαi are used as QoS impact parameters. Among them, target_delay is the expected delay of service traffic. The smaller the value of target_delay, the smaller the tolerance for queuing delay of data packets in the network. αi and hαi are the speed adjustment granularity of the user-space congestion control algorithm in the service flow growth stage. The larger the values ​​of αi and hαi, the faster the speed increase.

3. The user-space-based RDMA network QoS coordination method according to claim 1, characterized in that, After receiving the message, the receiving end carries a network congestion control signal in the returned ACK message, including: Round-trip time is used as the congestion control signal. The receiver carries the timestamp information of the relevant reception / transmission events in the ACK message and feeds it back to the sender as a congestion control signal.

4. The user-space-based RDMA network QoS coordination method according to claim 1, characterized in that, The sending end receives the ACK message and obtains the network congestion control signal, including: The sending end receives the congestion control signal carried in the ACK packet, and, in conjunction with the relevant time measured by the local network card, uses a hardware and software co-operated RTT precision measurement technology to calculate the network round-trip time value.

5. The user-space-based RDMA network QoS coordination method according to claim 1, characterized in that, The transmitting end adjusts the transmitting window based on the congestion control signal and in conjunction with the initial QoS parameters, using a user-space congestion control algorithm, including: The sending end compares the calculated actual network round-trip time value measure_delay with the pre-configured target_delay, and uses an additive, multiplicative, and subtractive user-space congestion control algorithm to adjust the sending window of the service traffic in order to increase or decrease the speed.

6. The user-space-based RDMA network QoS coordination method according to claim 1, characterized in that, The sending end periodically reports the QoS parameters and service quality of the service flow to the correction node, including: A separate correction node is used to collect service traffic information from the sending end and correct erroneous events. The sending end triggers a collection and reporting event at each time interval, reporting the QoS parameters, bandwidth, latency and sending window size of the service flow to the correction node.

7. The user-space-based RDMA network QoS coordination method according to claim 1, characterized in that, The correction node adjusts the QoS parameters based on the QoS parameters and service quality of the service flow reported by the sender, using a parameter adjustment algorithm, and then sends the adjusted parameters back to the sender, including: The correction node adjusts the QoS parameters based on the QoS parameters and service quality of the service flow reported by the sender, using a parameter adjustment algorithm, and sends the parameter modification suggestions from the sender back to the sender to reduce QoS ambiguity and errors.

Images