Wide-area RDMA network congestion control fairness compensation method

By deploying the active notification points and temporary response points of programmable switches in the RDMA network, the unfairness of congestion control in the wide-area RDMA network is solved, ensuring the fairness of near-source and near-destination, and improving network performance.

CN120499101APending Publication Date: 2025-08-15BEIHANG UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202510769289.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-10
Publication Date
2025-08-15

AI Technical Summary

Technical Problem

The existing RDMA network congestion control algorithm has unfair problems in wide-area scenarios, resulting in delayed feedback signals of cross-data center traffic, resulting in excessive congestion responsibility for internal traffic, affecting network performance.

Method used

The programmable switch designs active notification points and temporary response points. By marking congestion on the switch and actively sending congestion notification packets, feedback signals in advance to solve the problem of near-source inequity, and temporarily slowing the cross-DC traffic before the sender slows down, ensuring a fair response to near-destination.

Benefits of technology

The fairness of near-source and near-destination in wide-area RDMA network is achieved, which improves overall network performance and reduces the difference in stream completion time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120499101A_ABST
    Figure CN120499101A_ABST
Patent Text Reader

Abstract

The invention provides a congestion control fairness compensation method for a wide-area RDMA (Remote Direct Memory Access) network. The system is composed of an active notification point PNP and a temporary response point TRP which are deployed on an external switch ESW connecting a DC and a WAN. When near-source congestion occurs, the switch marks congestion for the data packet, and the active notification point clears the congestion mark and actively sends a congestion notification packet CNP through the corresponding sending end; when a receiving end receives the data packet containing the congestion mark, responding to the congestion notification packet; the temporary reaction point checks the congestion notification packet, determines the severity of the congestion, and temporarily slows the corresponding cross-DC traffic before the sender begins to decelerate. Therefore, the congestion feedback signal is fed back in advance by using the programmable switch, the congestion feedback fairness of the new-source is ensured, the congestion feedback signal is responded in advance by using the programmable switch, the congestion response fairness of the new-source is ensured, and finally, the fairness in the wide-area RDMA network is maintained, and the overall network performance is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of network communication technology, and more particularly, to a fairness compensation method for wide area RDMA network congestion control. Background Art

[0002] RDMA (Remote Direct Memory Access) technology integrates the network protocol stack into a network interface card (RNIC) specifically designed for RDMA. The one-way operations (WRITE, READ, and ATOMIC) provided by this NIC allow remote access to the target host's memory, bypassing the target host's CPU and operating system without requiring any additional data copying. This enables high throughput and low latency for data transmission while reducing CPU load. Due to its performance advantages, RDMA has been widely adopted in real-world production environments such as distributed deep learning and distributed storage. Adopting RDMA in cross-datacenter services is expected to meet stringent low-latency and high-throughput performance requirements.

[0003] Data centers have grown significantly to meet the demand for high-performance services. However, a single data center always has a ceiling. Therefore, companies are actively developing cross-data center services for three reasons. First, the rapid development of various distributed applications is placing higher demands on the performance of a single data center, leading to the need for cross-data center services. For example, with the rapid growth of the application user base, the demand for storage resources has surged, making it impossible for cloud service providers to rely on a single data center to store user data. Furthermore, the scale and training data for large language models are also rapidly increasing, requiring GPU resources far beyond the capacity of a single data center. Second, geographical differences lead to varying energy, labor, and land costs. Cloud service providers tend to build data centers in cost-effective cities and interconnect them for greater efficiency. Third, user expectations for reliable services are driving cloud service providers to build data centers in diverse locations to enhance overall system fault tolerance and prevent service interruptions caused by natural disasters. Adopting RDMA for cross-data center services holds the potential to meet stringent low-latency and high-throughput performance requirements.

[0004] RDMA's congestion control algorithm initially used the InfiniBand protocol, which is independent of Ethernet and has hindered large-scale deployment of RDMA. Later, RoCEv2 (RDMA over Converged Ethernet version 2) enabled RDMA deployment on Ethernet, promoting its widespread adoption in data centers. To achieve reliable and high-performance data transmission, RoCE requires Priority Flow Control (PFC) to implement a lossless network. However, PFC introduces congestion propagation issues such as head-of-line blocking, unfairness, PFC storms, and even PFC deadlocks, which can significantly negatively impact RDMA network performance. To mitigate the negative impact of PFC, RoCEv2 introduced the DCQCN congestion control mechanism. This mechanism monitors switch queue depth information and notifies the sender to proactively reduce the speed, maintaining a low queue depth on the switch and reducing the triggering of PFC.

[0005] However, RDMA congestion control algorithms can lead to unfairness. The currently widely used RDMA congestion control algorithms require switches to carry relevant information in the data packets to be forwarded, and rely on the receiving end to generate congestion feedback signals after receiving the relevant data packets, informing the sending end to actively reduce the speed. This feature will inevitably lead to the problem of inconsistent feedback signal arrival times in wide-area scenarios. For example, in Figure 1 In the WAN, the latency introduced by the wide area network (WAN) is generally in the thousands of microseconds (corresponding to a physical distance of several hundred kilometers), while the latency of each hop in the data center is generally in the few microseconds. Therefore, the sender of cross-DC (data center) traffic (blue traffic in the figure) will receive the congestion feedback signal (red traffic in the figure) much later than the sender of the DC's internal traffic (green traffic in the figure). At the same time, because cross-DC traffic maintains its original speed until receiving the feedback signal, while the traffic within the DC is continuously suppressed, the internal traffic within the DC ultimately bears all responsibility for network congestion, resulting in unfair congestion control. Compared to the performance of a single DC, in the test experiment, in a wide-area RDMA network with a distance of about 100 kilometers, the flow completion time of DC internal traffic increased by 118%, while the flow completion time of cross-DC traffic decreased by 74%. As the network load increases, congestion becomes more frequent, exacerbating this unfairness.

[0006] Currently, the related work on this problem, Annulus, only solves Figure 1The near-source unfairness problem in [1] is caused by the QCN mechanism of L3 routing on the ToR (Top of Rack) switch, which has no way to solve the unfairness caused by congestion occurring on the side close to the receiving end. Another advanced work, BiCC, partially alleviates the near-destination unfairness problem by limiting the number of flying bytes between the receiving switch and the receiving end. However, BiCC uses a virtual output queue (VQ) to temporarily store paused flows with the same destination, resulting in significant buffer overhead and lacking the required accuracy. In addition, the current solution for solving the unfairness problem in wide-area TCP networks mainly alleviates the unfairness problem by slowing down the congestion window reduction of the flow within the DC, which is not applicable to wide-area RDMA networks because RDMA networks do not have a congestion control window like TCP networks.

[0007] Therefore, it is of great significance to solve the above unfairness problem and design a compensation mechanism for the fairness of the RDMA network congestion control algorithm to improve the performance of the RDMA network in wide-area scenarios.

[0008] With the continuous development of software-defined networking (SDN) technology, network programmability has gradually expanded from the control plane of SDN to the data plane. This process has led to the emergence of programmable hardware switches and the protocol-independent programming language P4. Programmable hardware switches have similar power consumption and price to traditional switches, but offer programmability without sacrificing performance, enabling the execution of user-defined packet processing logic at line speed. P4 is a domain-specific programming language adapted for programmable switch architectures. Currently, many commercial programmable switches, such as Intel (Barefoot)'s Tofino switch and Cavium's XPliant switch, already support P4. The data plane pipeline of a programmable switch includes a certain amount of ALU computing resources and TCAM / SRAM storage resources. Programmers can write and run P4 programs on the programmable switch platform to implement line-speed network applications. Summary of the Invention

[0009] The purpose of the embodiments of the present disclosure is to provide a fairness compensation method for congestion control in a wide-area RDMA network. To address the shortcomings of existing RDMA network congestion control algorithms in wide-area scenarios, the present invention utilizes programmable switches to provide early feedback of congestion feedback signals, ensuring fairness in near-source congestion feedback. Furthermore, programmable switches are utilized to proactively respond to congestion feedback signals, ensuring fairness in near-destination congestion responses. Ultimately, this method maintains fairness in wide-area RDMA networks and improves overall network performance.

[0010] In a general aspect, a fairness compensation method for congestion control in a wide area RDMA network is provided, which consists of two parts: a proactive notification point PNP and a temporary reaction point TRP deployed on an external switch ESW connecting a DC and a WAN;

[0011] When congestion near the source occurs, the switch marks the data packet as congested; the switch actively notifies the point to clear the congestion mark and actively sends a congestion notification packet (CNP) through the corresponding sender;

[0012] When the receiver receives a data packet containing a congestion marker, it responds with a congestion notification packet; the temporary reaction point checks the congestion notification packet, determines the severity of the congestion, and temporarily slows down the corresponding cross-DC traffic before the sender starts to slow down.

[0013] The method for the active notification point to clear the congestion mark is as follows: after receiving a data packet with a congestion mark, the wide area RDMA network congestion control fairness compensation mechanism first queries the congestion feedback information recorded in the register. When the time difference between the current time and the last time the congestion feedback signal was sent is greater than a preset time interval, the active notification point deployed on the external switch connecting the DC and the WAN copies the original packet and transmits it to the egress pipe of the switch. In the egress pipe, the original data packet is modified to become a CNP message, and the egress port of the packet is set to the port when it entered the switch, and then the packet is sent back to the sending host.

[0014] The fairness compensation mechanism for wide area RDMA network congestion control clears the congestion mark on the data packet after receiving the data packet with the congestion mark.

[0015] The specific method for the active notification point to actively send the congestion notification packet CNP through the corresponding sending end is:

[0016] Use the sender's daemon to asynchronously send information to the external switch control plane connecting the DC and WAN;

[0017] In the RoCEv2 protocol, before the sender and receiver perform RDMA communication, the ibv_modify_qp function is used to set the queue pair status to initial, ready to receive, and ready to send in sequence. The ibv_modify_qp function is monitored using eBPF and captured on the host side with only the server-side queue pair number cQPN. Subsequently, the corresponding entry is inserted into the QP field table.

[0018] When the host calls the ibv_destroy_qp function, the ESW control plane is also notified to delete the corresponding QP field entry and reset the register storing its CNP timestamp.

[0019] In the process of sending the congestion notification packet CNP, in order to prevent SYN flood attacks, a TCP handshake monitoring mechanism based on SYN-Cookie is designed in the data plane. When the ESW receives a SYN packet, it replies with a SYN-ACK packet with a Cookie. If the ESW receives an ACK packet with the correct sequence number, it replies with a TCP-RST packet to re-establish the transmission and records the flow in the TCP-cookie register array. Subsequently, the TCP connection re-establishes the handshake process. The data plane forwards the flow information to the control plane only when the ACK packet is detected. Only when the control plane identifies the traffic as cross-data center traffic, the QP field table entry is inserted after receiving the daemon process packet from the host side.

[0020] The specific method of the temporary reaction point checking the congestion notification packet and temporarily slowing down the corresponding cross-DC traffic before the sender starts to slow down is as follows:

[0021] The temporary reaction point first pre-injects virtual packets to the designated ports through the control plane, and the virtual packets are transmitted in a loop. When it is detected that the packet rate needs to be reduced, the virtual packets are redirected to these pre-filled ports, circulate a predefined number of times, and then forwarded to the correct port.

[0022] In order to quickly respond to congestion notification packets (CNPs) and accurately identify target cross-data center traffic, a combination of data plane registers and control plane flow tables is designed. The data plane immediately writes to the register array CNP_Cache when receiving a CNP; at the same time, the CNP is sent to the switch CPU to perform flow table insertion; when the flow table insertion is performed to insert the table entry, the control plane resets the register to make the expired register entry available for reuse; at runtime, each data packet is hashed using its dstIp and sQPN to check for a hit in the CNP_Cache register array. If a miss occurs, the CNP_Store table is applied. If there is still a miss, it means that the flow is not the target of deceleration.

[0023] The specific method for determining the severity of congestion is as follows: when the ESW receives a CNP, the CNP reception time is recorded as the CNP time of the corresponding flow;

[0024] The first CNP increases loop_num and changes the flow state to rate-limited, indicating that packets for this flow need to be redirected to resolve congestion. Subsequently, each α CNP triggers an increase in loop_num, where α is a custom parameter. To prevent cross-DC flows from being blocked due to continuous CNPs, α itself gradually increases to ensure that the rate of increase in loop_num slows down over time. If no CNP is received within a continuous β time period (β is a custom parameter), the congestion is considered to have eased, the flow state is changed to recovery, indicating that the flow has entered the rate recovery phase, and cnp_num is reset to 0.

[0025] A multiplicative decreasing strategy is applied to α to ensure that if the flow experiences subsequent congestion, it will be aggressively rate-limited. When a packet is forwarded to the correct port, its loop_num header field is removed by the TRP.

[0026] The technical effects to be achieved by the embodiments of the present invention are:

[0027] Programmable switches are used to provide early feedback of congestion feedback signals to ensure fairness in near-source congestion feedback. Programmable switches are also used to respond to congestion feedback signals in advance to ensure fairness in near-destination congestion response. Ultimately, fairness in wide-area RDMA networks is maintained and overall network performance is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0029] Figure 1 is a schematic diagram of an example of unfair congestion feedback in a wide-area RDMA network in the prior art;

[0030] Figure 2 1 is a schematic diagram illustrating an architecture diagram of a fairness compensation method for wide area RDMA network congestion control according to an embodiment of the present disclosure;

[0031] Figure 3 Schematic diagram showing the architecture of a TCP handshake monitoring mechanism according to an embodiment of the present disclosure;

[0032] Figure 4 is a schematic diagram showing a logical architecture diagram of an active notification point according to an embodiment of the present disclosure;

[0033] Figure 5 is a schematic diagram showing a logic architecture diagram of a temporary reaction point according to an embodiment of the present disclosure;

[0034] Figure 6 is a schematic diagram showing a flow chart of a speed reduction algorithm according to an embodiment of the present disclosure;

[0035] Figure 7 It is a schematic diagram showing the main components and layout architecture of a network congestion control fairness compensation mechanism according to a specific application example of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and conciseness, descriptions of features known in the art may be omitted.

[0037] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will become clear after understanding the disclosure of this application.

[0038] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0039] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are used solely to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Thus, what is referred to as a first member, first component, first region, first layer, or first portion in the examples described herein may also be referred to as a second member, second component, second region, second layer, or second portion without departing from the teachings of the examples.

[0040] In the specification, when an element (such as a layer, region, or substrate) is described as being “on,” “connected to,” or “coupled to” another element, the element may be directly “on,” “connected to,” or “coupled to” the other element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on,” “directly connected to,” or “directly coupled to” another element, there may be no other elements present therebetween.

[0041] The terms used herein are intended only to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular is intended to include the plural. The terms "comprise," "include," and "have" indicate the presence of the recited features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0042] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure pertains after understanding the present disclosure. Unless expressly defined otherwise herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0043] Furthermore, in describing the examples, when it is deemed that a detailed description of well-known related structures or functions would cause ambiguous interpretation of the present disclosure, such detailed description will be omitted.

[0044] Figure 2 3 is a schematic diagram illustrating a fairness compensation method for wide-area RDMA network congestion control according to an embodiment of the present disclosure.

[0045] In order to achieve the above-mentioned purpose of the invention, the technical framework adopted by the present invention is as follows Figure 2 shown.

[0046] The fairness compensation mechanism for wide area RDMA network congestion control mainly consists of two parts: the proactive notification point deployed on the external switch (the switch connecting the DC and WAN, ESW), and the temporary reaction point deployed on the peer external switch. Figure 2 As shown in the figure, when near-source congestion occurs, the switch marks the data packet as congested. The proactive notification point removes the congestion mark and proactively sends a Congestion Notification Packet to the corresponding sender, mitigating the significant feedback delay caused by long-distance transmission and resolving near-source unfairness. When the receiver receives a packet with a congestion mark, it responds with a Congestion Notification Packet (CNP). Temporary reaction points can detect these CNPs and temporarily slow down the corresponding cross-DC traffic before the sender begins to slow down, resolving near-destination unfairness.

[0047] Active notification points:

[0048] In traditional RDMA congestion control algorithms, cross-DC flows must wait for the receiver to receive data packets with congestion markers and send back congestion control signals before responding to network congestion. However, the fairness compensation mechanism for wide-area RDMA network congestion control significantly advances congestion feedback signals, avoiding the excessively long delays imposed by wide-area networks.

[0049] However, RDMA packets only carry the server-side queue pair number (sQPN), which is a required field for generating a CNP. If we want the ESW to send a CNP, we must ensure that the ESW has the client queue pair number (cQPN) when the first packet arrives. This prevents the ESW from wanting to send a CNP but lacking a cQPN. Furthermore, given that large-scale data centers can generate tens of thousands of flows per second, the design of the active notification point must be efficient to minimize ESW resource overhead.

[0050] We utilize the Packet Replication Engine (PRE) provided by the programmable switch. Upon receiving a congestion-marked packet, the Wide Area RDMA Network Congestion Control Fairness Compensation mechanism first queries the congestion feedback information recorded in a register to align with the CNP feedback behavior of the DCQCN algorithm. Only when the time difference between the current time and the last time a congestion feedback signal was sent exceeds a pre-set interval will the active notification point deployed on the External Switch replicate the original packet and pass it to the switch's egress pipeline. In the egress pipeline, the original packet is modified to become a CNP message, and the packet's egress port is set to the port it entered the switch, before being resent back to the sending host. Furthermore, to prevent the External Switch at the other end of the WAN from issuing another congestion feedback signal for a congestion-marked packet, which could lead to excessive suppression, the Wide Area RDMA Network Congestion Control Fairness Compensation mechanism clears the congestion marker on the packet after receiving it.

[0051] We use the sender's daemon to asynchronously send information to the ESW control plane. In the RoCE v2 protocol, before the sender and receiver perform RDMA communication, the queue pair (QP) state is set to initial (INIT), ready to receive (RTR), and ready to send (RTS) in sequence using the ibv_modify_qp function. We use eBPF to monitor this function and capture the cQPN on the host side. Subsequently, the corresponding entry is inserted into the QP field table. Since eBPF can obtain and transmit the cQPN during the INIT phase of RDMA transmission, the control plane can implement the table insertion before the first RDMA packet arrives at the ESW. In addition, when the host calls the ibv_destroy_qp function, the ESW control plane is also notified to delete the corresponding QP field entry and reset the register storing its CNP timestamp.

[0052] Due to the existence of the virtual migration mechanism, the eBPF program on the host side cannot simply determine whether the established RDMA connection is intra-DC / inter-DC based on the target IP. Storing all daemon packets in ESW arbitrarily will waste scarce data plane resources. Due to the topological location of ESW, it will not receive any traffic within the DC. Therefore, the data plane of ESW can monitor the TCP connection establishment process to collect information about inter-DC traffic. In order to prevent SYN flood attacks from consuming ESW resources, we designed a TCP handshake monitoring mechanism based on SYN-Cookie on the data plane. When ESW receives a SYN packet, it replies with a SYN-ACK packet with a Cookie instead of maintaining the state of the TCP flow on the data plane. If ESW receives an ACK packet with the correct sequence number, it replies with a TCP-RST packet to re-establish the transmission and records the flow in the TCP-cookie register array. Subsequently, the TCP connection can re-establish the handshake process, such as Figure 3 As shown in the figure, since SYN-cookie validation only involves retransmission within the DC, it introduces only tens of microseconds of additional latency, negligible compared to the millisecond-level TCP handshake latency for cross-DC traffic. The data plane forwards flow information to the control plane only when it detects an ACK packet. Only when the control plane identifies the traffic as cross-DC traffic does it insert a QP field table entry after receiving a daemon packet from the host.

[0053] like Figure 4As shown in the figure, the active notification point can not only obtain the congestion feedback signal generated by the line speed with a low switch overhead, but also avoid overly frequent congestion feedback by using the timestamp stored in the register and the operation of clearing the congestion mark, thereby effectively and timely notifying the cross-DC flow of the need for speed reduction, greatly alleviating the near-source unfairness.

[0054] Temporary reaction points:

[0055] When near-destination congestion occurs, the fairness compensation mechanism for wide-area RDMA network congestion control differs from traditional solutions in that it deploys a congestion response mechanism on the external switch, enabling a quick response to congestion feedback signals from the peer end.

[0056] The temporary reaction point first pre-injects a number of dummy packets onto designated ports via the control plane. These packets are then looped. When a packet is detected requiring a rate reduction, it is redirected to these pre-populated ports, where it loops a predefined number of times before being forwarded to the correct port. As a result, these looped packets not only achieve the desired traffic reduction but are also physically isolated from the normal forwarding ports, preventing any impact on the receiving end.

[0057] In order to quickly respond to CNPs and accurately identify target cross-data center traffic, we designed a combination of data plane registers and control plane flow tables. Figure 5 As shown, the data plane writes to a register array (called CNP_Cache) immediately upon receiving a CNP. At the same time, the CNP is sent to the switch CPU, where flow table insertion is performed. When a table entry is inserted, the control plane resets the register, making expired register entries available for reuse. Therefore, the CNP_Cache register array only needs to store CNPs within the delay of inserting CNP_Store table entries, reducing its memory requirements. At runtime, each packet is hashed using its dstIp and sQPN to check for a hit in the CNP_Cache register array. If a miss occurs, the CNP_Store table is applied. If there is still a miss, it means that the flow is not a target for deceleration.

[0058] In addition, we designed a speed reduction algorithm. ESW can determine the severity of congestion based on the frequency of receiving CNP for the target flow. Figure 6As shown, when an ESW receives a CNP, it records the CNP reception time as the corresponding flow's CNP time and increments its cnp_num counter, which is used to calculate the loop_num parameter. Specifically, to achieve rapid rate reduction, the first CNP increases loop_num and changes the flow state to rate limiting, indicating that packets for this flow need to be redirected to resolve congestion. Subsequently, each α CNP triggers an increase in loop_num. To prevent cross-DC flows from being blocked due to continuous CNP reception, α itself is gradually increased, ensuring that the rate of loop_num increase gradually slows over time. Through this mechanism, different subflows receive different loop_num values, widening their time gaps and achieving rate reduction at the subflow granularity. If no CNPs are received for a continuous period of β, we consider the congestion to have eased. At this point, we change the flow state to recovery, indicating that the flow has entered the rate recovery phase, and reset cnp_num to 0. Simultaneously, we apply a multiplicative decrement strategy to α to ensure that if the flow experiences subsequent congestion, it will be aggressively rate-limited. When a packet is forwarded to the correct port, TRP removes its loop_num header field.

[0059] Specific application examples:

[0060] The fairness compensation mechanism for wide-area RDMA network congestion control runs only on the switches connecting the data center and the wide area network. The switches must be programmable. Data plane modules, such as congestion signal feedback and congestion signal response, must be written in P4. Control plane programs, responsible for updating data plane entries and registers, must also be written in C.

[0061] Since ESW changes the dstQPN field of the CNP packet, its ICRC checksum needs to be recalculated. However, for RoCEv2 packets, the ICRC field is redundant because the Ethernet frame already has a checksum. Therefore, we set it to disable ICRC. The main components and layout of the fairness compensation mechanism for wide-area RDMA network congestion control are as follows: Figure 7 During deployment, the host first compiles and runs the eBPF program written in C. The programmable switch system then compiles the P4 program using the toolchain provided by the system. The relevant scripts are then run to load the P4 program into the switch chip. The Python program is then used to configure the data plane state. The system then enters normal operation, maintaining congestion control fairness in the wide-area RDMA network.

[0062] While some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A fairness compensation method for wide area RDMA network congestion control, characterized in that: It consists of two parts: the proactive notification point (PNP) and the temporary response point (TRP), which are deployed on the external switch (ESW) connecting the DC and the WAN. When near-source congestion occurs, the switch marks the packet as congested; The active notification point clears the congestion mark and actively sends a congestion notification packet CNP through the corresponding sender; When the receiving end receives a data packet containing a congestion mark, it responds with a congestion notification packet; Temporary reaction points examine the congestion notification packets, determine the severity of the congestion, and temporarily slow down the corresponding cross-DC traffic before the sender starts to slow down.

2. A method for fairness compensation for wide area RDMA network congestion control according to claim 1, characterized in that: The method for the active notification point to clear the congestion mark is as follows: after receiving a data packet with a congestion mark, the wide area RDMA network congestion control fairness compensation mechanism first queries the congestion feedback information recorded in the register. When the time difference between the current time and the last time the congestion feedback signal was sent is greater than a preset time interval, the active notification point deployed on the external switch connecting the DC and the WAN copies the original packet and transmits it to the egress pipe of the switch. In the egress pipe, the original data packet is modified to become a CNP message, and the egress port of the packet is set to the port when it entered the switch, and then the packet is sent back to the sending host. The fairness compensation mechanism for wide area RDMA network congestion control clears the congestion mark on the data packet after receiving the data packet with the congestion mark.

3. A fairness compensation method for wide area RDMA network congestion control according to claim 2, characterized in that: The specific method for the active notification point to actively send the congestion notification packet CNP through the corresponding sending end is: Use the sender's daemon to asynchronously send information to the external switch control plane connecting the DC and WAN; Use eBPF to monitor the ibv_modify_qp function and capture the server-side queue pair number cQPN on the host side; then, insert the corresponding entry into the QP field table; When the host calls the ibv_destroy_qp function, the ESW control plane is also notified to delete the corresponding QP field entry and reset the register storing its CNP timestamp.

4. A method for fairness compensation for wide area RDMA network congestion control according to claim 3, characterized in that: In the process of sending the congestion notification packet CNP, in order to prevent SYN flood attacks, a TCP handshake monitoring mechanism based on SYN-Cookie is designed in the data plane. When the ESW receives a SYN packet, it replies with a SYN-ACK packet with a cookie. If the ESW receives an ACK packet with the correct sequence number, it replies with a TCP-RST packet to re-establish transmission and records the flow in the TCP-cookie register array. Subsequently, the TCP connection re-establishes the handshake process, and the data plane forwards the flow information to the control plane only when it detects the ACK packet; only when the control plane identifies the traffic as inter-data center traffic, it inserts the QP field table entry after receiving the daemon process packet from the host side.

5. The method for fairness compensation of wide area RDMA network congestion control according to claim 4, characterized in that: The specific method of the temporary reaction point checking the congestion notification packet and temporarily slowing down the corresponding cross-DC traffic before the sender starts to slow down is as follows: The temporary reaction point first pre-injects virtual packets to the designated ports through the control plane, and the virtual packets are transmitted in a loop. When it is detected that the packet rate needs to be reduced, the virtual packets are redirected to these pre-filled ports, circulate a predefined number of times, and then forwarded to the correct port. In order to quickly respond to congestion notification packets (CNPs) and accurately identify target cross-data center traffic, a combination of data plane registers and control plane flow tables is designed. The data plane immediately writes to the register array CNP_Cache when receiving a CNP; at the same time, the CNP is sent to the switch CPU to perform flow table insertion; when the flow table insertion is performed to insert the table entry, the control plane resets the register to make the expired register entry available for reuse; at runtime, each data packet is hashed using its dstIp and sQPN to check for a hit in the CNP_Cache register array. If a miss occurs, the CNP_Store table is applied. If there is still a miss, it means that the flow is not the target of deceleration.

6. A fairness compensation method for wide area RDMA network congestion control according to claim 5, characterized in that: The specific method for determining the severity of congestion is as follows: when the ESW receives a CNP, the CNP reception time is recorded as the CNP time of the corresponding flow; The first CNP increases loop_num and changes the flow state to rate-limited state, indicating that the packets of this flow need to be redirected to resolve congestion; Subsequently, every α CNPs triggers an increase in loop_num, where α is a custom parameter. To prevent cross-DC flows from being blocked due to continuous CNPs, α itself increases gradually to ensure that the rate of increase of loop_num slows down over time. If no CNPs are received within a continuous β time period (β is a custom parameter), the congestion is considered to have eased, the flow state is changed to recovery, indicating that the flow has entered the rate recovery phase, and cnp_num is reset to 0. A multiplicative decreasing strategy is applied to α to ensure that if the flow experiences subsequent congestion, it will be aggressively rate-limited. When a packet is forwarded to the correct port, its loop_num header field is removed by the TRP.

Citation Information

Cited By

  • Congestion control method based on reverse explicit notification and flow prediction

    CN122204786A

  • A congestion control method based on reverse explicit notification and traffic prediction

    CN122204786B