A flow control method and system based on gateway active probing

Abstract

This invention discloses a traffic control method and system based on gateway active probing, relating to the field of computer network communication technology. The invention constructs and sends probing messages to a wide area network (WAN) based on cross-domain traffic information. It determines the WAN transmission latency and bottleneck bandwidth based on received probing response messages. The token generation rate of the token bucket is adjusted according to the transmission latency and bottleneck bandwidth, and the token consumption rate is calculated based on the tokens consumed by the cross-domain traffic. When the token bucket's stock triggers a preset condition, a pause time is determined based on the token generation rate, token consumption rate, and data center internal latency. The source node is then controlled to delay sending traffic based on the pause time. This invention effectively avoids WAN congestion and packet loss, as well as PFC storms within the data center, significantly improving the overall efficiency of cross-domain collaborative services.

Description

Technical Field

[0001] This invention relates to the field of computer network communication technology, and in particular to a flow control method and system based on gateway active detection. Background Technology

[0002] With the rise of large models (such as ChatGPT and Sora) and the popularization of ultra-high-definition audio and video and real-time interactive applications, a single data center can no longer meet its exponentially growing computing power demand. Cross-regional multi-data center collaboration has become the mainstream architecture to support key businesses such as intelligent computing, media processing and distributed inference. Against this background, the communication traffic carried by the data center interconnection network exhibits complex characteristics such as diversification, high concurrency, strong bursts and low latency sensitivity. Typical businesses include but are not limited to: (1) Intelligent computing traffic transmission: cross-domain collaborative training of large models, whose communication traffic has strong periodicity, extremely high bursts and extreme sensitivity to tail latency; (2) Ultra-high-definition audio and video traffic transmission: such as 8K / VR live streaming, cloud gaming, etc., which require high throughput, low jitter and deterministic latency; (3) KVCache cross-domain distribution communication: used to cache key-value pair states in distributed inference, which has strict requirements for transmission efficiency and consistency.

[0003] These new types of traffic differ fundamentally from the steady-state traffic of traditional data centers. To meet the demands of high-performance communication, data centers commonly employ RDMA (Remote Direct Memory Access) technology that supports the RoCEv2 protocol. RDMA relies on a lossless network environment and achieves high-throughput, low-latency transmission within the local area network through PFC (Priority-based Flow Control) and the DCQCN congestion control algorithm utilizing the ECN (Explicit Congestion Notification) mechanism.

[0004] However, in wide-area interconnection scenarios across data centers, RDMA needs to rely on wide-area lossy networks for communication, and its existing design has serious limitations: On the one hand, forwarding devices in wide area networks (WANs) typically do not support features such as ECN marking and PFC flow control. This means they do not send ECN signals to the source to slow down traffic during network congestion, nor do they send PFC frames to suppress traffic transmission before packet loss. When the rate of traffic injected into the data center exceeds the WAN's forwarding capacity, the forwarding device's queue will continuously accumulate, eventually leading to packet loss once the buffer threshold is exceeded. Once packet loss occurs, RDMA will use an inefficient go-back-n mechanism to retransmit all data, causing a significant drop in effective throughput and a substantial increase in flow completion time (FCT), thus slowing down the entire service process.

[0005] On the other hand, high-throughput, high-burst traffic transmitted across wide area networks often creates many-to-one incast high-burst traffic at the data center gateway. Since the RDMA transmitter needs millisecond-level round-trip latency to detect the gateway's congestion signal, burst traffic continuously accumulates in the gateway's egress queue, easily triggering PFC pause frames for backpressure. These PFC pause frames propagate into the data center, affecting "innocent" communication flows within the data center and causing them to suffer performance losses due to head-of-line congestion.

[0006] The above analysis shows that existing flow control mechanisms are significantly inadequate in adapting to the multiple challenges of high burst traffic, high transmission latency, and strong dynamic congestion in cross-data center scenarios. Existing solutions such as Swing and BiCC either rely on PFC relays to optimize lossless long-distance transmission or only optimize local congestion feedback within the data center, neither of which solves the fundamental problems in wide-area lossy environments. Furthermore, newer congestion control algorithms such as HPCC and PowerTCP require coordinated hardware modifications across the entire network, resulting in high deployment costs.

[0007] To support the efficient and stable operation of critical services such as large-scale distributed intelligent computing, ultra-high-definition media services, and efficient inference cache distribution, a novel cross-domain traffic control scheme is urgently needed that can proactively detect wide-area congestion and accurately regulate burst traffic without modifying the existing wide area network infrastructure. Therefore, this invention proposes a cross-domain communication traffic control method and system based on gateway proactive detection. This system relies on a data center gateway and mainly includes the following functional modules: (1) a network detection module, used to detect wide area network capacity and latency in real time to enhance environmental awareness; (2) a token bucket management module, used to smooth and shape burst traffic through a token bucket data structure; and (3) a burst flow control module, used to accurately regulate the sending rate through fine-grained feedback. The scheme described in this invention effectively avoids congestion accumulation and packet loss while fully exploring and utilizing available bandwidth, ensuring the controllability and stability of long-distance, high-load communication tasks. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a traffic control method and system based on gateway active detection. This invention can proactively sense the available bandwidth and latency status of a wide area network, smooth burst traffic, and send fine-grained rate control commands to the source, thereby effectively avoiding wide area congestion and packet loss, as well as PFC storms within the data center, significantly improving the overall efficiency of cross-domain collaborative services.

[0009] To achieve the above objectives, the present invention proposes the following technical solutions.

[0010] A traffic control method based on gateway active probing includes the following steps: Based on cross-domain traffic information, probe messages are constructed and sent to the wide area network. The transmission latency and bottleneck bandwidth of the wide area network are determined based on the received probe response messages. The token generation rate of the token bucket is adjusted according to the transmission latency and bottleneck bandwidth, and the token consumption rate is obtained by statistically analyzing the tokens consumed by the token bucket according to the cross-domain traffic. When the token bucket's inventory triggers a preset condition, the pause time is determined based on the token generation rate, token consumption rate, and data center internal latency, and the source node is controlled to delay sending traffic based on the pause time.

[0011] Furthermore, based on cross-domain traffic information, probe messages are constructed and sent to the wide area network, including: Sample cross-domain traffic headers and construct probe request messages carrying sending timestamps and sequence numbers to send to the wide area network; Receive a probe response message returned by the receiving gateway along the original path. The probe response message carries the sending timestamp, sequence number, and arrival timestamp of the probe request message to the receiving gateway.

[0012] Furthermore, the transmission latency and bottleneck bandwidth of the wide area network are determined based on the received probe response messages, including: The transmission delay is determined based on the difference between the receiving time of the probe response message and the sending timestamp; The bottleneck bandwidth is determined based on the difference in arrival timestamps of adjacent probe response messages with the same sequence number and the size of the probe message.

[0013] Further, adjusting the token generation rate of the token bucket based on the transmission latency and bottleneck bandwidth includes: When the transmission delay increases relative to the previous cycle, the current token generation rate is calculated and reduced based on the change in transmission delay between the two cycles, the transmission delay of the current cycle, and the token generation rate of the previous cycle.

[0014] Further, adjusting the token generation rate of the token bucket based on the transmission latency and bottleneck bandwidth includes: When the transmission delay is reduced or equal to that of the previous period, the current token generation rate is calculated and increased based on the minimum round-trip delay, the transmission delay of the current period, the bottleneck bandwidth, and the token generation rate of the previous period.

[0015] Further, adjusting the token generation rate of the token bucket according to the bottleneck bandwidth includes: When the bottleneck bandwidth is updated, the token generation rate of the previous period is scaled proportionally according to the bottleneck bandwidth ratio of the current period to the previous period in order to update the current token generation rate.

[0016] Furthermore, the token consumption rate was statistically analyzed, including: The total amount of tokens consumed within a preset period is calculated, and the token consumption rate is determined based on the ratio of the total consumption to the period duration.

[0017] Furthermore, the pause time is determined based on the token generation rate, token consumption rate, and data center internal latency, including: Based on the ratio of token consumption rate to token generation rate, the sum of the intra-data center round-trip latency and the previous cycle's pause time is weighted and then subtracted from the intra-data center round-trip latency to update the pause time for the current cycle.

[0018] Furthermore, controlling the source node to delay sending traffic based on the pause time includes: A control message carrying the pause time is constructed and sent to the source node, so that the source node, after receiving the control message, sends subsequent traffic with a delay according to the pause time.

[0019] A traffic control system based on gateway active probing includes: The network detection module is used to construct and send detection messages to the wide area network based on cross-domain traffic information, and to determine the transmission latency and bottleneck bandwidth of the wide area network based on the received detection response messages. The token bucket management module is used to adjust the token generation rate of the token bucket according to the transmission latency and bottleneck bandwidth, and to calculate the token consumption rate based on the tokens consumed by the cross-domain traffic. The burst rate control module is used to determine the pause time based on the token generation rate, token consumption rate and data center internal latency when the token bucket's stock triggers preset conditions, and to control the source node to delay sending traffic based on the pause time.

[0020] The present invention has the following advantages.

[0021] (1) Low deployment cost: This invention is incrementally deployed on the gateway control plane in the form of a plug-in, without requiring any changes to the existing data plane traffic, network topology and WAN hardware devices, making it easy to deploy in practice.

[0022] (2) Enhance wide-area perception capability: Based on the characteristic that all cross-domain burst traffic passes through the data center gateway, this invention uses the token bucket data structure to obtain the burst level of cross-domain transmission traffic and actively probes to obtain the congestion status on the wide area network, obtains a global network status view, and notifies the source end of the status information by sending a pause control message, thereby enhancing RDMA's perception capability of the wide area network and avoiding the asymmetry between the burst rate of traffic and the wide-area forwarding capability.

[0023] (3) Precisely suppress traffic bursts: Based on the gateway actively detecting the status of the wide area network, and combined with the token bucket and fine-grained feedback mechanism, the token bucket is used to smooth the traffic, and through fine-grained pause time feedback, the precise and dynamic control of the source end sending rate that causes bursts is achieved, avoiding the impact on the normal transmission of traffic inside the data center, and effectively solving the problem of "overreaction" in traditional PFC.

[0024] (4) Ensure stable network transmission: This invention effectively controls bursts at the gateway, avoids congestion and packet loss of cross-domain burst traffic in the wide area network, and significantly improves the flow completion time; at the same time, it greatly reduces the occurrence of PFC storms, which improves the completion efficiency of traffic within the data center by several times, and is suitable for high-performance data center interconnection network environments with new distributed collaborative business requirements. Attached Figure Description

[0025] Figure 1 This is a main flowchart of a traffic control method based on gateway active probing in an embodiment of the present invention; Figure 2 Flowchart for wide area network state detection; Figure 3 This is a module interaction diagram of a traffic control system based on gateway active detection in an embodiment of the present invention; Figure 4 This is a diagram of the experimental network topology. Figure 5 A comparison chart of queue lengths for data center gateways and WAN routers under the default mechanism and the solution of this invention; Figure 6 A comparison chart of FCT for cross-data center traffic and intra-data center traffic of DCQCN and the present invention under various wide-area bandwidth bottlenecks; Figure 7 This is a comparison chart of FCT (Functional Transmission Coherence) for cross-data center traffic and intra-data center traffic of DCQCN and the present invention under various wide-area transmission delays. Detailed Implementation

[0026] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, detailed descriptions are provided below through embodiments.

[0027] This invention provides a traffic control method based on gateway active probing, the main process of which is as follows: Figure 1 As shown, the specific steps include:

[0028] Step S1: Construct and send probe messages to the wide area network based on cross-domain traffic information, and determine the transmission delay and bottleneck bandwidth of the wide area network based on the received probe response messages.

[0029] By periodically sampling communication traffic to the control plane and buffering header routing information to construct probe packets, delay probe packets and bandwidth probe packets are periodically sent to the wide area network (WAN). The receiving gateway receives and replies with probe response packets. Through periodic active probing and based on the information carried in the probe response packets, the transmission delay and bottleneck bandwidth values ​​of the WAN are calculated and output. The process is as follows: Figure 2 As shown.

[0030] Step S11: Construct and send the probe request message.

[0031] For cross-domain transmission traffic from inside the data center to the data center gateway, the packet headers of this transmission traffic are sampled periodically and cached locally. By setting a timer, the headers are periodically used as routing headers to construct latency probe request messages and bandwidth probe request messages. A sending timestamp is written into the messages, and a unique sequence number is written into the messages each period to mark the probe. The messages are then sent to the network egress of the data center gateway connected to the wide area network.

[0032] In an optional embodiment of the present invention, the initial transmission delay is the minimum basic round-trip delay (...). The initial bottleneck bandwidth is the outgoing link bandwidth of the data center gateway ( ).

[0033] Step S12: Generating and transmitting the probe response message.

[0034] After the probe request message arrives at the data center gateway on the receiving side, a corresponding probe response message is generated with high priority. The information carried in the request message is copied into the response message, and the arrival timestamp of the request message is inserted. The response message is then returned immediately along the original path.

[0035] Step S13: Network status parameters updated.

[0036] Upon receiving the response message, the requesting gateway updates the transmission delay record as follows: ) and bottleneck bandwidth ( ).

[0037] Step S131, Delay Calculation: The requesting party records the receiving timestamp and calculates the difference between it and the sending timestamp carried in the message, as an estimated transmission delay value. .

[0038] Step S132, Bandwidth Calculation: The requester constructs two bandwidth probe request messages of the same size and content every cycle and sends them continuously to the wide area network. After receiving two bandwidth probe response messages with the same sequence number, the difference between the reception timestamps carried in the messages is calculated. The ratio of the size of the request message to this difference is used as the bottleneck bandwidth estimate. .

[0039] Step S2: Adjust the token generation rate of the token bucket according to the transmission latency and bottleneck bandwidth, and calculate the token consumption rate based on the tokens consumed by cross-domain traffic.

[0040] A token bucket data structure is implemented to record the number of tokens in bytes. An equal number of tokens as the data packet size are consumed when routing communication traffic, and the token count is periodically increased based on the token generation rate. Simultaneously, the token generation rate is dynamically updated using transmission latency and bottleneck bandwidth obtained from WAN status probing as input. Furthermore, the token consumption rate is periodically calculated based on changes in the token bucket inventory, serving as an estimate of the bandwidth of burst traffic across data centers over a period of time. The token bucket has a capacity limit and a token generation rate. The initial token inventory is equal to the maximum token capacity, and the token generation rate is initialized to the data center gateway outgoing port link bandwidth. When the token supply is not full, new tokens are continuously generated according to the token generation rate.

[0041] Step S21: The token generation rate is dynamically updated.

[0042] According to the period Updated transmission latency and bottleneck bandwidth Token generation rate The calculation method is as follows: when That is, when the transmission delay increases relative to the previous period: when That is, when the transmission delay is reduced relative to the previous period: When the bottleneck bandwidth During the update: Wherein, the subscript is Indicates the latest value of the variable, with the subscript . This represents the previous value of the variable. This represents the minimum basic round-trip time under congestion-free conditions.

[0043] Step S22, token consumption and rate statistics.

[0044] For cross-domain traffic originating within the data center and destined for the data center gateway, the token inventory is reduced by an equal number of tokens based on the size of the data packet. This applies in each cycle. The system tracks the amount of tokens consumed within the current period and calculates the token consumption rate during that period. Upon arrival of each data packet, if the token balance falls below the pause threshold, a pause control message is sent to the source node.

[0045] In an optional embodiment of the present invention, when the token consumption rate is calculated... Then, the operation of calculating the pause time is triggered. When a data packet arrives and causes the token inventory to fall below the pause threshold, the operation of constructing and returning a pause control message to the source node is triggered.

[0046] Step S3: When the token bucket's inventory triggers the preset conditions, determine the pause time based on the token generation rate, token consumption rate, and internal latency of the data center, and control the source node to delay sending traffic based on the pause time.

[0047] Burst rate control is triggered when the token bucket inventory falls below the token bucket pause threshold. The monitored token consumption rate and token generation rate are used as inputs, where the token consumption rate corresponds to the actual burst rate, and the token generation rate corresponds to the rate at which burst traffic is forwarded to the wide area network under ideal conditions. Based on the ratio of these two rates, the time during which the source node generating the burst traffic should pause transmission is periodically calculated. After the token bucket triggers the pause threshold, a pause control message is generated for each burst traffic data packet and sent to the source node, with the pause time written into this message. This controls the actual burst rate in subsequent periods to converge towards the token generation rate.

[0048] In an optional embodiment of the present invention, a new control message (GNR NAK) is designed based on the Not Ready Receiver Negative Acknowledgment (RNR NAK) message in the Remote Direct Memory Access (RDMA) standard protocol. This GNR NAK message carries a pause duration, and the sender will delay for a specified time after receiving the message before continuing to send traffic.

[0049] Step S31, pause time calculation.

[0050] Because the internal topology of the data center satisfies that the path distance from each end node to the data center gateway is the same, and the data center is a congestion-free network, the bidirectional round-trip latency from the end nodes within the data center to the data center gateway is ( Given the known token generation rate () ) and token consumption rate ( ), each new cycle Pause time ( The update method is as follows: in, Indicates the pause time of the latest cycle. This indicates the pause time of the previous cycle. The initial value is 0.

[0051] Step S32: Pause control message transmission.

[0052] Based on the packet routing information, construct the GNR NAK pause control message returned by the original path, and set the currently determined pause time. It is written into the message and sent to the source node to regulate its transmission traffic at a specified delay time, thereby controlling its average burst rate.

[0053] This invention also provides a traffic control system based on gateway active probing, deployed on a gateway device connecting a data center and a wide area network, for executing the above-mentioned method. It includes the following three core modules, the composition and logical relationship of which are as follows: Figure 3 As shown: The network detection module is used to construct and send detection messages to the wide area network based on cross-domain traffic information, and to determine the transmission latency and bottleneck bandwidth of the wide area network based on the received detection response messages. The token bucket management module is used to adjust the token generation rate of the token bucket according to the transmission latency and bottleneck bandwidth, and to calculate the token consumption rate based on the tokens consumed by the cross-domain traffic. The burst rate control module is used to determine the pause time based on the token generation rate, token consumption rate and data center internal latency when the token bucket's stock triggers preset conditions, and to control the source node to delay sending traffic based on the pause time.

[0054] To verify the performance of the technical solution of this invention, a test platform was built using the simulation platform simAI. The platform was compared with the DCQCN end-to-end congestion control algorithm without network assistance in existing mainstream technologies. Simulation tests were conducted on a large-scale collaborative training scenario of a 128-node dual-datacenter interconnection model under different bottleneck bandwidths and wide-area transmission delays. The experimental topology is as follows: Figure 4 As shown, the evaluation focuses on the queue length, flow completion time (FCT) of transmitted traffic, and deceleration ratio (the ratio of actual FCT to the theoretical minimum FCT) of data center gateways and WAN routers. Experimental results are shown below. Figure 5 , Figure 6 , Figure 7As shown, this invention does not modify the original end-to-end DCQCN congestion control algorithm. Instead, it incrementally deploys the technical solution of this invention at the gateway location, and is labeled in the experimental results diagram as "DCQCN+GPFC". Experimental results show that compared with the original default congestion control algorithm DCQCN, the technical solution of this invention can reduce the queue length of bottleneck nodes (gateways and WAN routers) by more than 50%; under different WAN bottleneck bandwidths (50G~150G) and different WAN transmission latencies (1ms~9ms), the tail flow completion time of cross-domain traffic is reduced from tens of seconds to hundreds of milliseconds, and the average flow completion time is reduced from seconds to tens of milliseconds, with transmission efficiency improved by tens of times. The tail flow completion time of internal traffic is reduced by at least 90%, and the average flow completion time is reduced by at least 50%.

[0055] The embodiments described above are specific examples of the present invention, and their purpose is to explain in more detail the technical solutions proposed by the present invention and their beneficial effects. The description of the present invention is not intended to limit its scope of protection. In fact, any modifications, equivalent substitutions, or technical improvements made based on the concept of the present invention should be considered to fall within the scope of protection of the present invention.

Claims

1. A flow control method based on gateway active probing, characterized in that, Includes the following steps: Based on cross-domain traffic information, probe messages are constructed and sent to the wide area network. The transmission latency and bottleneck bandwidth of the wide area network are determined based on the received probe response messages. The token generation rate of the token bucket is adjusted according to the transmission latency and bottleneck bandwidth, and the token consumption rate is obtained by statistically analyzing the tokens consumed by the token bucket according to the cross-domain traffic. When the token bucket's inventory triggers a preset condition, a pause time is determined based on the token generation rate, token consumption rate, and data center internal latency, and the source node is controlled to delay sending traffic based on the pause time.

2. The method as described in claim 1, characterized in that, Based on cross-domain traffic information, probe messages are constructed and sent to the wide area network, including: Sample cross-domain traffic headers and construct probe request messages carrying sending timestamps and sequence numbers to send to the wide area network; Receive a probe response message returned by the receiving-side gateway along the original path. The probe response message carries the sending timestamp, sequence number, and arrival timestamp of the probe request message to the receiving-side gateway.

3. The method as described in claim 2, characterized in that, The transmission latency and bottleneck bandwidth of the wide area network are determined based on the received probe response messages, including: The transmission delay is determined based on the difference between the receiving time of the probe response message and the sending timestamp; The bottleneck bandwidth is determined based on the difference in arrival timestamps of adjacent probe response messages with the same sequence number and the size of the probe message.

4. The method as described in claim 1 or 3, characterized in that, Adjusting the token generation rate of the token bucket based on the transmission latency and bottleneck bandwidth includes: When the transmission delay increases relative to the previous cycle, the current token generation rate is calculated and reduced based on the change in transmission delay between the two cycles, the transmission delay of the current cycle, and the token generation rate of the previous cycle.

5. The method as described in claim 1 or 3, characterized in that, Adjusting the token generation rate of the token bucket based on the transmission latency and bottleneck bandwidth includes: When the transmission delay is reduced or equal to that of the previous period, the current token generation rate is calculated and increased based on the minimum round-trip delay, the transmission delay of the current period, the bottleneck bandwidth, and the token generation rate of the previous period.

6. The method as described in claim 1, characterized in that, Adjusting the token generation rate of the token bucket based on the bottleneck bandwidth includes: When the bottleneck bandwidth is updated, the token generation rate of the previous period is scaled proportionally according to the bottleneck bandwidth ratio of the current period to the previous period in order to update the current token generation rate.

7. The method as described in claim 1, characterized in that, The token consumption rate was statistically analyzed, including: The total amount of tokens consumed within a preset period is calculated, and the token consumption rate is determined based on the ratio of the total consumption to the period duration.

8. The method as described in claim 1, characterized in that, The pause time is determined based on the token generation rate, token consumption rate, and internal data center latency, including: Based on the ratio of token consumption rate to token generation rate, the sum of the intra-data center round-trip latency and the previous cycle's pause time is weighted and then subtracted from the intra-data center round-trip latency to update the pause time for the current cycle.

9. The method as described in claim 8, characterized in that, Based on the pause time, the source node delays sending traffic, including: A control message carrying the pause time is constructed and sent to the source node, so that the source node, after receiving the control message, sends subsequent traffic with a delay according to the pause time.

10. A flow control method based on gateway active probing, characterized in that, include: The network detection module is used to construct and send detection messages to the wide area network based on cross-domain traffic information, and to determine the transmission latency and bottleneck bandwidth of the wide area network based on the received detection response messages. The token bucket management module is used to adjust the token generation rate of the token bucket according to the transmission latency and bottleneck bandwidth, and to calculate the token consumption rate based on the tokens consumed by the cross-domain traffic. The burst rate control module is used to determine the pause time based on the token generation rate, token consumption rate and internal data center latency when the token bucket's stock triggers a preset condition, and to control the source node to delay sending traffic based on the pause time.

Images