A non-intrusive host-in and network fabric congestion control method based on active probing

Abstract

A non-intrusive host and network structure congestion control method based on active detection and machine learning, comprising: (1) a data acquisition and model training module adjusts the number of CPU cores and network traffic load in a real test environment to simulate different types and different degrees of congestion scenarios, and uses iperf3 to send long and short detection flows to collect network traffic index data, and trains a congestion identification model based on a decision tree; (2) a congestion identification module periodically sends long and short detection flows during system operation, collects network traffic characteristics in real time, and uses the congestion identification model to identify the types and degrees of host and network structure congestion in the current network; (3) a congestion relief module dynamically adjusts the sending rate according to the congestion identification result, and adaptively adjusts the detection flow sending period, thereby realizing differentiated control of different types of congestion. The present application does not need to modify the existing network equipment and protocol stack, can accurately identify different congestion types in complex network environment, and further improves the transmission performance and resource utilization efficiency of the data center network.

Description

Technical Field

[0001] This invention relates to the field of data center network and congestion control technology, and in particular to a non-intrusive host-based and network structure congestion control method based on active detection. Background Technology

[0002] With the rapid development of applications such as cloud computing, large-scale model training, distributed storage, and high-performance computing, the traffic volume and communication complexity in modern data center networks continue to increase. Especially in AI large-scale model training, parameter synchronization, and multi-tenant mixed deployment scenarios, a large amount of sudden traffic and high-concurrency communication can easily cause serious network congestion problems, resulting in decreased link utilization, increased flow completion time, and fluctuations in service performance.

[0003] Existing data center congestion control methods mostly rely on explicit switch feedback, network equipment upgrades, or modifications to the endpoint protocol stack. Examples include ECN-based explicit congestion notification mechanisms, switch queue monitoring mechanisms, and congestion awareness schemes assisted by dedicated hardware. While these methods can alleviate network congestion to some extent, they typically suffer from the following problems:

[0004] (1) Highly invasive. Existing methods often require modifications to switch hardware, network protocol stacks, or operating system kernels, making deployment complex and difficult to adapt to heterogeneous data center environments and existing network infrastructure.

[0005] (2) Difficulty in distinguishing different sources of congestion. Performance degradation in data centers may not only stem from link congestion in the network structure, but also from CPU resource contention within the host, insufficient protocol stack processing capacity, or resource contention in the virtualization environment. Traditional congestion control methods typically focus only on network-side indicators and cannot effectively distinguish between congestion within the host and network structure congestion, resulting in inaccurate rate adjustment strategies.

[0006] (3) Insufficient real-time performance and adaptability. Most existing solutions rely on fixed-period monitoring or static threshold judgment, which makes it difficult to detect changes in congestion level in a timely manner under complex dynamic load environments, and is prone to problems such as delayed congestion response or misjudgment.

[0007] (4) Lack of intelligent congestion identification capabilities. Traditional congestion control algorithms are usually based on empirical rules or fixed thresholds for judgment, which makes it difficult to make full use of multi-dimensional traffic characteristics to accurately classify and assess the degree of congestion in complex scenarios.

[0008] Therefore, there is an urgent need for a non-intrusive intelligent congestion control method that can identify both host-level congestion and network structure congestion without modifying existing network infrastructure, and has the ability to dynamically adjust in real time, in order to improve network performance and resource utilization efficiency in complex data center environments. Summary of the Invention

[0009] To overcome the problems of existing data center network congestion control methods being highly invasive, having difficulty distinguishing between in-host congestion and network structure congestion, and lacking dynamic adaptability, this invention proposes a non-invasive in-host and network structure congestion control method based on active detection.

[0010] This invention collects network traffic characteristics by actively sending long and short probe streams and analyzes the current network state using a decision tree-based congestion identification model, achieving online identification of both host-level and network structure-level congestion. Based on this, the sending end performs differentiated rate adjustments according to different congestion types, thereby improving network transmission performance and resource utilization efficiency.

[0011] Overall, the congestion control framework of this invention consists of three parts: a data acquisition and model training module, a congestion identification module, and a congestion mitigation module. The data acquisition and model training module constructs a realistic test environment to simulate different types and degrees of congestion scenarios and uses iperf3 to send probe traffic to collect network traffic indicator data, further training a decision tree-based congestion identification model. The congestion identification module periodically sends long and short probe streams during system operation and identifies the current congestion type and degree based on real-time collected network traffic characteristics. The congestion mitigation module dynamically adjusts the sending rate based on the congestion identification results and adaptively adjusts the probe stream sending cycle, reducing system overhead while ensuring real-time congestion detection.

[0012] This invention requires no modification to existing network devices and protocol stacks, has good non-intrusiveness and deployment compatibility, and can effectively improve network transmission efficiency and congestion control capabilities in complex data center environments.

[0013] The first aspect of this invention relates to a non-intrusive host-based and network structure congestion control method based on active detection, comprising the following steps:

[0014] (1) First, data collection and model training are required. The data collection and model training module simulates different types and degrees of congestion scenarios by constructing a real test environment, and uses iperf3 to send probe traffic to collect network traffic indicator data, and further trains the congestion identification model based on decision tree. The main steps are as follows:

[0015] Step (1.1): First, set up a real test bench environment and simulate different types of network congestion scenarios and congestion levels by adjusting the number of CPU cores and network traffic load on the host.

[0016] Step (1.2): Use the iperf3 tool at the sending end to send probe traffic, collect network traffic indicator data under various congestion scenarios, and form a training dataset.

[0017] Step (1.3): Based on the training dataset, train a decision tree-based congestion identification model to determine the type and degree of congestion in the current network.

[0018] (2) In the actual operation of the network system, the congestion identification module periodically sends long and short probe streams, and identifies the current congestion type and degree based on the real-time collected network traffic characteristics. The main steps are as follows:

[0019] Step (2.1): The sending end simultaneously sends a long probe stream and a short probe stream according to a preset probe period. The long probe stream is used to fully fill the network buffer to expose the congestion characteristics of the network structure, while the short probe stream is used to perceive changes in network latency and packet loss in real time with low overhead, enabling more accurate perception of congestion characteristics within the host. The long and short streams work together to comprehensively capture different types of congestion.

[0020] Step (2.2): During the probe stream transmission process, the sending end collects the following key metrics in real time: throughput, round-trip time (RTT), packet loss rate, jitter, congestion window (cwnd), and round-trip time variation (RTTVAR). These metrics constitute the original feature vector of the current network state;

[0021] Step (2.3): Input the collected traffic metrics into the pre-trained decision tree congestion identification model. The model outputs the current network congestion type (host-side congestion, switching structure congestion, or mixed congestion) and congestion severity (mild, moderate, severe). The determination result is used to select differentiated rate adjustment strategies in the subsequent congestion mitigation phase.

[0022] (3) After effectively identifying network congestion, the congestion mitigation module dynamically adjusts the sending rate based on the congestion identification results and adaptively adjusts the probe stream sending cycle, reducing system overhead while ensuring real-time congestion detection. The main steps are as follows:

[0023] Step (3.1): The sending end executes a differentiated rate adjustment strategy based on the congestion type and degree output during the congestion identification phase. If the current congestion is within the host, the rate adjustment factor is set to... i = low / medium / high. If the current network structure is congested, the speed regulation factor is set to... , i = low / medium / high. If the current congestion situation is mixed, then the minimum value of the above two is taken as the speed regulation factor;

[0024] Step (3.2): After completing the rate adjustment, to ensure the effectiveness of the rate adjustment, a new congestion detection period is set to guarantee the real-time performance of congestion detection. The sending end dynamically adjusts the sending period of the next probe traffic based on the current congestion level: if the current congestion level is light or there is no congestion, the detection period is appropriately increased to reduce detection overhead; if the current congestion level is heavy or the congestion type changes frequently, the detection period is shortened to ensure the real-time performance and sensitivity of congestion detection. The updated period will take effect immediately after the current congestion mitigation operation is completed and will be used in the next round of congestion identification phase.

[0025] A second aspect of the present invention relates to a non-intrusive host-based and network structure congestion control device based on active detection, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement a non-intrusive host-based and network structure congestion control method based on active detection of the present invention.

[0026] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements a non-intrusive host-based and network structure congestion control method based on active probing.

[0027] The working principle of this invention is:

[0028] (1) In the real test environment, the present invention first simulates different types and degrees of congestion scenarios by adjusting the number of CPU cores and network traffic load of the host, and uses iperf3 to send probe traffic to collect network traffic index data such as throughput, latency, and packet loss rate, and trains a congestion identification model based on decision tree to realize the classification and identification of congestion in the host and network structure.

[0029] (2) During the system operation phase, the sending end periodically sends long and short probe streams simultaneously to collect network traffic characteristics in real time and input them into the congestion identification model for analysis. Among them, the long probe stream is used to reflect the continuous link load status, and the short probe stream is used to detect instantaneous network fluctuations and sudden congestion, thereby improving the accuracy of congestion identification in complex network environments.

[0030] (3) During the congestion mitigation phase, the sending end executes a differentiated rate adjustment strategy based on the congestion type and degree identified by the model, and dynamically adjusts the sending cycle of the probe traffic to reduce the system probe overhead while ensuring the real-time performance of congestion detection.

[0031] The advantages of this invention are:

[0032] (1) This invention does not require modification of switches, network devices or operating system protocol stacks, has good non-intrusiveness and deployment compatibility, and can be adapted to existing data center network environments.

[0033] (2) This invention achieves accurate differentiation between congestion within the host and network structure congestion by combining long and short probe streams with a machine learning model, thereby improving the congestion identification capability in complex network environments.

[0034] (3) The present invention can dynamically adjust the transmission rate and detection period according to different types of congestion, thereby reducing additional network overhead while ensuring the real-time performance of congestion detection, thereby improving network transmission efficiency and resource utilization. Attached Figure Description

[0035] Figure 1 This is a framework diagram of the method of the present invention.

[0036] Figure 2 This is a flowchart of the process of the present invention. Detailed Implementation

[0037] The specific embodiments of the present invention are further described below with reference to the accompanying drawings.

[0038] Example 1

[0039] This embodiment relates to a non-intrusive host-based and network structure congestion control method based on active detection, including the following steps:

[0040] Step 1: Data Acquisition and Congestion Identification Model Construction. In a real testbed environment, different types and degrees of on-host congestion and network structure congestion scenarios are simulated by adjusting the number of host CPU cores, network background traffic load, and link bandwidth utilization. The sending end uses iperf3 to periodically send long and short probe streams to collect network traffic metrics such as throughput, latency, packet loss rate, and stream completion time. Based on the collected data, a decision tree congestion identification model is trained, enabling the model to classify and identify different congestion types and degrees.

[0041] Step 2: Implement online network congestion identification. For example... Figure 2 As shown, during system operation, the sending end periodically sends both long and short probe streams simultaneously. The long probe stream is used to monitor the continuous link load status, while the short probe stream is used to detect instantaneous network latency changes and sudden congestion. Subsequently, the network traffic characteristics corresponding to the probe streams are collected and input into the congestion identification model to perform real-time analysis of the current network status, obtaining the current congestion type and degree.

[0042] Step 3: Implement congestion mitigation and dynamic adjustment. When the system detects congestion, the sending end executes a differentiated rate control strategy based on the identified congestion type. When intra-host congestion is detected, the data injection rate of the sending end is preferentially reduced to alleviate host resource contention; when network structure congestion is detected, the sending rate is dynamically adjusted to reduce link load. Simultaneously, the system dynamically adjusts the probe traffic sending cycle according to the current network state, reducing the probe frequency under light load conditions to reduce system overhead, and increasing the probe frequency under high congestion conditions to ensure real-time congestion detection.

[0043] This invention collects network traffic characteristics by sending long and short probe streams, and combines them with a decision tree-based congestion identification model to achieve online identification of host-side congestion and network structure congestion. It also dynamically adjusts the sending rate and probe cycle according to different congestion types, thereby improving the data center network transmission performance and resource utilization efficiency.

[0044] This invention does not require modification of existing network equipment and protocol stacks, and can accurately identify different congestion types in complex network environments, thereby further improving the transmission performance and resource utilization efficiency of data center networks.

[0045] The system employs a leaf-spine topology (12 sender leaves, 2 receiver leaves, 4 spines), with all links having a bandwidth of 100 Gbps, a switch queue size of 1 MB, and a link packet loss rate of 0.01%. The host-side MTU is 1500 / MSS 1460, initial CWND is 100 MSS (146 KB), ssthresh is 500 MB, CUBIC β=0.7 / C=0.4, RTO range is 10ms, base RTT is 100μs, NIC buffer is 4 MB, CPU processing speed is 12 GB / s, and a total of 200 traffic flows are processed. Performance for the four traffic types is shown in Table 1.

[0046] Table 1

[0047]

[0048] Example 2

[0049] This embodiment relates to a non-intrusive host-based and network structure congestion control device based on active detection, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a non-intrusive host-based and network structure congestion control method based on active detection according to Embodiment 1.

[0050] Example 3

[0051] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements a non-intrusive host-based and network structure congestion control method based on active probing, as described in Embodiment 1.

[0052] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A non-intrusive host-based and network structure congestion control method based on active detection, characterized in that: Includes the following steps: (1) In the data collection and model training phase, a real test bench environment is first built, and different types of congestion are simulated by adjusting the number of CPU cores and network traffic load of the host. The sending end uses iperf3 to send probe traffic and collect network traffic index data under various congestion scenarios. During the model training phase, a congestion identification model based on decision trees is trained using the collected training data. (2) During the congestion identification phase, the sending end periodically sends long and short probe streams simultaneously to collect network traffic indicators and uses the congestion identification model to determine the current congestion type and degree. (3) During the congestion mitigation phase, the sending end performs differentiated rate adjustment according to the current network congestion type and dynamically updates the sending cycle of probe traffic to ensure the real-time performance of congestion detection.

2. The non-intrusive host-based and network structure congestion control method based on active detection as described in claim 1, characterized in that: Step (1) specifically includes: Step (1.1): First, set up a real test bench environment and simulate different types of network congestion scenarios and congestion levels by adjusting the number of CPU cores and network traffic load on the host. Step (1.2): Use the iperf3 tool at the sending end to send probe traffic, collect network traffic indicator data under various congestion scenarios, and form a training dataset; Step (1.3): Based on the training dataset, train a decision tree-based congestion identification model to determine the type and degree of congestion in the current network.

3. The non-intrusive host-based and network structure congestion control method based on active detection as described in claim 1, characterized in that: Step (2) specifically includes: Step (2.1): The sending end sends a long probe stream and a short probe stream simultaneously according to the preset probe period; the long probe stream is used to fully fill the network buffer to expose the congestion characteristics of the network structure, and the short probe stream is used to perceive network latency and packet loss changes in real time with low overhead, and to perceive the congestion characteristics within the host more accurately; the long and short streams work together to comprehensively capture different types of congestion. Step (2.2): During the probe stream transmission process, the sending end collects the following key indicators in real time: throughput, round-trip time (RTT), packet loss rate, jitter, congestion window (cwnd), and round-trip time variation (RTTVAR); these indicators constitute the original feature vector of the current network state; Step (2.3): Input the collected traffic metrics into the pre-trained decision tree congestion identification model. The model outputs the current network congestion type and congestion level. The judgment result is used to select differentiated rate adjustment strategies in the subsequent congestion mitigation stage.

4. The non-intrusive host-based and network structure congestion control method based on active detection as described in claim 3, characterized in that: The decision tree congestion identification model in step (2.3) outputs the current network congestion type, including host-side congestion, switching structure congestion, or mixed congestion; the congestion level includes light, medium, and severe.

5. The non-intrusive host-based and network structure congestion control method based on active detection as described in claim 1, characterized in that: Step (3) specifically includes: Step (3.1): The sending end executes a differentiated rate adjustment strategy based on the congestion type and degree output in the congestion identification phase; if the current congestion is within the host, the rate adjustment factor is set to... i = low / medium / high; if the current network structure is congested, the speed regulation factor is set to... i = low / medium / high; if the current situation is mixed congestion, then the minimum value of the two above is taken as the speed regulation factor; Step (3.2): After the rate adjustment is completed, in order to ensure the effectiveness of the rate adjustment, a new congestion detection period is set to ensure the real-time performance of congestion detection. The sending end dynamically adjusts the sending period of the next probe traffic according to the current congestion level: if the current congestion level is light or there is no congestion, the detection period is increased to reduce the detection overhead; if the current congestion level is heavy or the congestion type changes frequently, the detection period is shortened to ensure the real-time performance and sensitivity of congestion detection. The updated period will take effect immediately after the current congestion mitigation operation is completed and will be used in the next round of congestion identification stage.

6. A non-intrusive host-based and network structure congestion control device based on active detection, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the non-intrusive host-based and network structure congestion control method based on active probing as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements a non-intrusive host-based and network structure congestion control method based on active probing, as described in any one of claims 1-5.

Images