Rate-optimized congestion management

Abstract

The present disclosure relates to rate-optimized congestion management. A method and system for reducing congestion in a RoCEv2 network are provided. The method is configured to operate traffic from a sender node to a receiver node at scale in a data center. The described method has three phases: a fast start phase, a transition phase, and a regulation phase. In the fast start phase, the sender sends data to the receiver at a fast initial rate. This can continue until the receiver observes a congestion event. When this occurs, the sender reduces the data transmission rate when the method enters a transition phase. The method enters a regulation phase from the reduced rate, where the rate is increased using a combination of a feedback control loop and an additive increase multiplicative decrease (AIMD) algorithm.

Description

[0001] This application is a divisional application of an application with an application date of May 15, 2020, application number 202080051803.3, and invention name “Rate-Optimized Congestion Management”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Application No. 16 / 415,609, filed on May 17, 2019, entitled “RATE-OPTIMIZED CONGESTION MANAGEMENT,” the contents of which are incorporated herein by reference for all purposes. Background Art

[0004] Data center applications require high throughput and ultra-low latency from the network including the host network stack. Due to the high CPU overhead of standard TCP / IP, Remote Direct Memory Access (RDMA) over Ethernet (RoCEv2) is deployed in modern data centers as an alternative technology to achieve low latency and high performance. When heavily loaded, data center networks may experience congestion and cause packet loss. Therefore, the performance of RDMA degrades drastically, and RoCEv2 relies on Priority-based Flow Control (PFC) to achieve loss-free networks. PFC may cause congestion spread, where slowing down traffic in one area of ​​the network can cause slowing down traffic in other areas of the network. Therefore, congestion management is necessary to avoid congestion spread. Existing congestion management methods in RoCEv2 networks, such as Data Center Quantized Congestion Notification (DCQCN), may not work well when running at large scale, generating thousands of pause frames, poor network utilization and average throughput, and causing congested flows to complete very slowly. Summary of the invention

[0005] There is a need to efficiently manage congestion in data center networks at scale. Such an approach would achieve high link utilization, ultra-low latency, and network fairness. The disclosed method achieves these goals by dynamically changing data rates in response to congestion events and by combining a feedback control algorithm with an additive growth and multiplicative reduction (AIMD) algorithm.

[0006] In one aspect, a computer-implemented method for managing congestion in a network protocol is disclosed. The method includes detecting a congestion event in the network. The method next includes slowing down a data transmission rate from an initial rate to a minimum rate. The method next includes increasing the data transmission rate to half of the initial rate. The method next includes periodically monitoring a one-way delay of the network. Finally, the network includes adjusting the data transmission rate using a feedback control algorithm and an additive increase multiplicative decrease (AIMD) algorithm. In some embodiments, setting the initial data transmission rate also includes setting an initial window equal to a bandwidth delay product, and increasing the window size if congestion is not detected after setting the initial window. In some embodiments, the initial data transmission rate is equal to a line rate of the network. In some embodiments, the threshold delay is based on a round trip time for data packet transmission. In some embodiments, the minimum data transmission rate is at least less than one percent of the initial data transmission rate. In some embodiments, the feedback control loop includes a proportional integral controller. In some embodiments, transmitting data from a sender to a receiver at the minimum data transmission rate includes sending a congestion notification packet from the receiver to the sender, and updating the rate when the sender receives the congestion notification packet. In some embodiments, continuously monitoring the one-way delay includes keeping track of the total number of bytes of packets received. In some embodiments, the managed transmission rate is calculated after one cycle at a minimum transmission rate and one cycle at half the initial rate. In some embodiments, the cycle is based on the round trip time of the transmitted packets. In some embodiments, the initial transmission rate is based on the line rate between the sender and the receiver. In some embodiments, the initial transmission rate is the minimum rate between the link rate of the sender and the link rate of the receiver. In some embodiments, the AIMD algorithm includes performing addition on the managed rate if the managed rate is below a threshold and performing multiplication on the managed rate if the managed rate is above a threshold. In some embodiments, the addition is adding a positive number. In some embodiments, the multiplication is multiplication by a number between 0 and 1. In some embodiments, the computer-implemented method also includes obtaining at least an initial rate event by adjusting the data transmission rate within a threshold number of cycles and monitoring the network for congestion events.

[0007] In another aspect, a system for managing network congestion in a network is disclosed. The system includes a plurality of computing nodes. In addition to one or more processors, the computing nodes also include a memory for storing one or more previously obtained delay values ​​and an instruction set. The processor is configured to execute the instruction set to detect a congestion event in a physical layer link, slow down a data transmission rate from an initial rate to a minimum rate, increase the data transmission rate to half of the initial rate, periodically monitor the one-way delay of the network, and adjust the data transmission rate using a feedback control algorithm and an additive growth multiplicative reduction (AIMD) algorithm. In some embodiments, the network has at least one Ethernet link. In some embodiments, the plurality of computing nodes include a sender node and a receiver node. In some embodiments, setting an initial data transmission rate also includes setting an initial window equal to a bandwidth delay product, and increasing the window size if congestion is not detected after setting the initial window. In some embodiments, the initial data transmission rate is equal to the line rate of the Ethernet link. In some embodiments, the threshold delay is based on a round trip time for data packet transmission. In some embodiments, the minimum data transmission rate is at least less than one percent of the initial data transmission rate. In some embodiments, the feedback control loop includes a proportional integral controller. In some embodiments, transmitting data from a sender to a receiver at a minimum data transmission rate includes: sending a congestion notification packet from the receiver to the sender, and updating the rate when the congestion notification packet is received by the sender. In some embodiments, continuously monitoring the one-way delay includes keeping track of the total number of bytes of the received packets. In some embodiments, the managed transmission rate is calculated after one cycle at the minimum transmission rate and one cycle at half the initial rate. In some embodiments, the period is based on the round-trip time of the transmitted packets. In some embodiments, the initial transmission rate is based on the line rate between the sender and the receiver. In some embodiments, the initial transmission rate is the minimum line rate between the sender and the receiver. In some embodiments, the AIMD algorithm includes performing addition on the managed rate if the managed rate is below a threshold and performing multiplication on the managed rate if the managed rate is above a threshold. In some embodiments, the addition is adding a positive number. In some embodiments, the multiplication is multiplication by a number between 0 and 1.

[0008] In yet another aspect, we disclose a computer-implemented method for managing congestion in a network protocol. The method includes a fast startup phase, including setting a data rate to an initial data rate and performing data transmission between a sender and a receiver until the receiver detects a congestion event. The method also includes a transition phase, including reducing the data rate to a minimum data rate within a period, and then increasing the data rate to half of the initial rate within the period. The method also includes an adjustment phase, including modifying the data rate using a combination of a feedback control loop and a sum-reducing multiplicative growth algorithm. In some embodiments, the computer-implemented method also includes an initial setup phase before the fast startup phase, including sharing a clock rate and a line rate between the sender and the receiver. In some embodiments, the computer-implemented method also includes, before the fast startup phase, setting a window size to an initial window size for sending data packets. In some embodiments, the computer-implemented method also includes increasing the window size when no congestion event is detected within the network. In some embodiments, the window size increases exponentially. In some embodiments, the transition phase begins after the congestion event is detected. In some embodiments, the feedback control algorithm is a proportional-integral algorithm. In some embodiments, the feedback control loop can be described as the difference between the current rate, the proportional term, and the integral term. In some embodiments, the proportional term describes the difference between the current delay and the threshold delay, while the integral term describes the difference between the current delay and the previous delay. In some embodiments, the multiplicative growth is performed in response to a probabilistic event. In some embodiments, the multiplicative growth is performed at a lower frequency than the feedback control loop.

[0009] In yet another aspect, a system for managing network congestion in a network is disclosed. The system includes a plurality of computing nodes. The computing nodes include a memory for storing one or more previously obtained delay values ​​and an instruction set. The computing nodes also include one or more processors configured to execute the instruction set to implement a fast startup phase, including setting the data rate to an initial data rate and performing data transmission between a sender and a receiver until the receiver detects a congestion event, a transition phase, including reducing the data rate to a minimum data rate within a period, and then increasing the data rate to half of the initial rate within the period, and an adjustment phase, including modifying the data rate using a feedback control loop in combination with a sum-reduced multiplicative growth algorithm. In some embodiments, the system also includes an initial setup phase prior to the fast startup phase, including sharing a clock rate and a line rate between the sender and the receiver. In some embodiments, the system also includes, prior to the fast startup phase, setting the window size to an initial window size for sending data packets. In some embodiments, the system also includes increasing the window size when no congestion event is detected within the network. In some embodiments, the window size increases exponentially. In some embodiments, the transition phase begins after a congestion event is detected. In some embodiments, the feedback control algorithm is a proportional-integral algorithm. In some embodiments, the feedback control loop can be described as the difference between the current rate, the proportional term, and the integral term. In some embodiments, the proportional term describes the difference between the current delay and the threshold delay, while the integral term describes the difference between the current delay and the previous delay. In some embodiments, the multiplicative growth is performed in response to a probabilistic event. In some embodiments, the multiplicative growth is performed at a lower frequency than the feedback control loop. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The novel features of the subject matter are set forth with particularity in the appended claims. Reference is made to the following detailed description of illustrative embodiments in which the principles of the subject matter are utilized, and to the accompanying drawings (also referred to as "diagrams" and "figures"), in which:

[0011] Figure 1 is an exemplary block diagram illustrating communications over a link;

[0012] Figure 2 schematically illustrates a plurality of variables measured in a congestion management system according to embodiments described herein;

[0013] Figure 3 A flow chart showing a method for congestion management in a physical layer link;

[0014] Figure 4 The optimum operating point for communication is shown;

[0015] Figure 5shows a latency comparison of the disclosed method and DCQCN, using data from experiments conducted with both methods;

[0016] Fig. 6A and Figure 6B The diagram shows Figure 5 A diagram of the data flow in the network of the experiment; and

[0017] Fig. 7A and Figure 7B A comparison of queue occupancy between the disclosed method and DCQCN is illustrated. DETAILED DESCRIPTION

[0018] The disclosed method effectively manages congestion in a RoCEv2 network at the link layer (e.g., in an Ethernet network). The method can be implemented on communications between a sender node and a receiver node, which nodes can be connected by one or more network links. The sender and receiver can manage data streams from multiple users and may have to maintain efficiency and fairness in transmitting data between multiple users sending data over a link.

[0019] The disclosed method has three main phases: a fast start phase, a transition phase, and an adjustment phase. In the fast start phase, the sender and receiver can send data to each other at an initial rate until the receiver records a congestion event. A congestion event can be detected as a delay spike measured from the receiver's perspective. After a congestion event, the transition phase begins. In the transition phase, the sender first reduces the data rate to a minimum rate within a period, and then increases the data rate to half the initial rate to manage congestion. After the transition phase, the adjustment phase uses a combination of feedback control and an additive increase and multiplicative decrease (AIMD) algorithm to increase the rate while maintaining low latency, efficiency, and fairness.

[0020] The feedback control loop guides the main rate calculation in the regulation phase, where the rate increase or decrease is precisely adjusted using, for example, a proportional integral algorithm. The AIMD algorithm adds small perturbations that cause the flow to rebalance over a long period of time. This is done without causing excessive disturbance to the overall arrival rate of the congested data packet queue.

[0021] After the adjustment phase, the sender and receiver can eventually resume sending data at the initial rate. The method can restart when another congestion event is detected.

[0022] In one aspect, the method and system of the present disclosure are provided for determining the network congestion state based on the measurement of delay variation. The delay measurement can be a forward path delay or a round trip delay, such as a round trip time (RTT) delay. The change or change in delay can be used as a congestion signal, so that the sender can be notified of the degree of congestion in a timely manner.

[0023] Figure 1 1 is an exemplary block diagram illustrating communication through link 103 according to an embodiment of the subject matter described herein. A sender 101 and a receiver 105 communicate through link 103. In the example described herein, the system delegates specific sending and receiving tasks to the sender and the receiver. However, packets can be sent from the sender to the receiver, and also from the receiver to the sender. Communications can use congestion management mechanisms to control congestion. In some cases, the network congestion status along the link can be determined. The sender and the receiver can also be referred to as the source and the destination, respectively. In other cases, the network congestion for the round trip (from the sender to the receiver and back) can be determined.

[0024] The sender 101 and the receiver 105 can be various types of computing devices, such as personal computers, tablet computers, smart phones, set-top boxes, desktop computers, laptop computers, gaming systems, servers, data centers, and various other devices or systems. A plurality of other devices or components can be connected through a network. One or more senders can be connected to one or more receivers. The sender and the receiver can be connected to the network through wired and / or wireless connections. The sender or the receiver can be referred to as the source node and the destination node in the network. The node can be any device equipped with communication capabilities. The communication can be wired or wireless communication.

[0025] The sender and receiver each have a memory 120 and a CPU 130. The receiver can use RDMA to access items stored in the sender's memory while bypassing the sender's operating system.

[0026] Link 103 may be from a wireless network, a wired network, or a combination of both. For example, the network may include one or more of the Internet, an intranet, a cellular network, a home network, a personal area network, etc., through an ISP, a cellular or broadband cable provider, etc. Link 103 may be associated with an Internet protocol interface, such as one or more network components, data servers, connecting nodes, switches, etc. On the other hand, sender 101 and receiver 105 may be considered part of link 103.

[0027] Both sender 101 and receiver 105 include a network interface 140, which may be a network interface card (NIC). The network interface allows the sender and receiver nodes to interface with link 103. The network interfaces 140 may have intelligent capabilities, allowing them to implement algorithms to control the data transmission rate to nodes in the network.

[0028] Data packets 111 sent from a source node (e.g., a sender) to a destination node (e.g., a receiver) may or may not experience delays along the link. In some embodiments, changes in delay or measurements of changes can provide an immediate signal of the degree of congestion. The delay can be calculated by subtracting the send time of the data packet 111 from the arrival time of the data packet 111. In some embodiments, the send time of the data packet is provided by the sender's clock 107, and the arrival / receive time of the data packet is provided by the receiver's clock 109.

[0029] The sender clock 107 and the receiver clock 109 do not need to be synchronized. In some cases, the sender clock 107 and / or the receiver clock 109 can be unsynchronized free-running clocks. In some cases, the sender clock 107 and the receiver clock 109 operate at the same rate or frequency. Alternatively, the sender clock 107 and the receiver clock 109 can run at different rates or frequencies. In some cases, a common rate or frequency can be set when the connection is established. For example, a common rate or frequency is a predetermined frequency set by a connection protocol before transmitting data packets. In some cases, the rates or frequencies associated with the sender and the receiver, respectively, can be exchanged when the connection is established. In some cases, the receiver may be notified by the rate or frequency of the sender. In another case, the sender and the receiver may run at different frequencies that are not confirmed by each other. For example, both the sender and the receiver send a handshake request (e.g., a control packet with a handshake type) when the connection is established, wherein the handshake packet may include a common frequency that the sender and the receiver agree to use for communication or a frequency associated with the sender and the receiver, respectively. It is important to note that the clock frequency or rate can be transmitted at any time other than the connection establishment time. For example, a handshake packet including the clock frequency can be sent once at an interval when a notification message is generated or a timeout timer expires. The clock can include any type of local hardware or device clock, such as a NIC hardware time or a host device (e.g., CPU) clock. The sender clock and the receiver clock may be of the same type or different types. In some cases, a calibration or mapping scheme can be used to map the NIC timestamp to the host timestamp. In some cases, clock frequency drift may occur between the sender clock and the receiver clock, resulting in a tendency for delay to vary. Detection and correction of frequency drift will be discussed later.

[0030] A timestamp indicating the current time at which the data packet 111 is sent may be included in the data packet 111. The timestamp may be generated by a sender clock. The timestamp may be embedded in any field of the data packet. The timestamp may be multi-bit. The timestamp may include any number of bits, such as 32 bits or 64 bits. The timestamp may provide a time resolution of picoseconds, milliseconds, or microseconds.

[0031] In some embodiments, each data packet 111 has a timestamp. Alternatively, some data packets have a timestamp. For example, every X packets may include a timestamp indicating the sending time, where X may be any integer.

[0032] Upon receiving a data packet including a timestamp, the receiver may calculate a delay. The delay may be determined by subtracting the timestamp value in the data packet from the current value of the receiver's local clock, or vice versa. Since the current value of the receiver's local clock may not be synchronized with the sender's clock, the difference may not represent an absolute delay. Therefore, the time difference may be referred to as a pseudo-delay. In the present disclosure, the terms "pseudo-delay" and "delay" are equivalent, and "delay" should not be interpreted as representing an absolute delay.

[0033] The difference or change in delay can be used to predict the degree of congestion. The difference in the change in delay value can indicate a change in queuing delay. For example, a positive change in delay due to an increase in delay value may indicate an increase in queue, while a negative change in delay may indicate a decrease in queue. In the case of a positive change being detected, it is advantageous to react to the delay without waiting for a standing queue to be formed, thereby achieving a low-delay method. In the case of a negative change being detected, it is advantageous to adjust the sending rate to fully utilize the bandwidth. In some cases, a notification message may be generated when, for example, a change in delay is determined to exceed a certain degree by comparing the change in delay with a delay difference threshold. A notification message may be generated when an increase change or a decrease change exceeds a predetermined delay difference threshold. The congestion state may include detection of a delay change exceeding a threshold. The congestion state may include a measured delay change. The congestion state may also include any other variables related to the delay discussed later herein. In some embodiments, when determining the congestion state, the recipient may generate a notification message. The notification message may include multiple variables related to the delay.

[0034] Figure 2 Schematically illustrates a number of variables measured in a congestion management system according to an embodiment of the subject matter described herein. The sender 201 and the receiver 203 may be associated with local clocks 205, 207, respectively, such as Figure 1 The sender and the receiver can be configured to send and receive messages such as data packets, control packets, notification packets, etc. Figure 2 As shown, one or more data packets 211, 217, 219 may be transmitted from a sender 201 to a receiver 203. In some cases, the one or more data packets may be time stamped by the sender's local clock 205. It should be noted that a sender and a receiver are depicted in the communication system, however, any other components may be included in the system.

[0035] A data packet may be sent from a sender or a receiver at a first time and received by the other at a second time. The first time (e.g., T1) and the second time (e.g., T2) may be determined by local clocks 205, 207, respectively. Various time points such as T4 and T5, or T6 and T7 may occur simultaneously and / or in various orders. Note that each communication may include one or more packets; however, for ease of reading, communications are described as a single packet sent from one component to another. It should also be noted that for readability, communications are depicted as non-overlapping and at different times. In addition, unless the context indicates otherwise, any communication may overlap, may be in a different order, and may be forwarded between various network components (e.g., intermediate nodes, switches, etc.). It should be noted that any different number of messages may be sent and received between a sender and a receiver; similarly, a sender and a receiver may send messages to and / or receive messages from various other components not shown in the figure.

[0036] As described above, the extent of the delay may be determined based on a change in the delay. In some embodiments, a notification message 215, 216 may be generated when it is determined that the change in the delay exceeds a delay difference threshold. In some cases, a new interval 227 may begin when the notification message is generated. In some cases, a new interval 227 may begin when a congestion state is detected. In some cases, a new interval may begin at the reception time or arrival time (e.g., T2) of the last data packet 211 from the previous interval. The interval may have any time length, such as microseconds, milliseconds, seconds, etc.

[0037] As shown, the delay of the first data packet in the interval can be calculated 221. After the first data packet is received by the receiver 203, the receiver can subtract the timestamp value T3 in the data packet from the current value T4 of the receiver's local clock. Delay values ​​can be calculated for one or more messages or data packets 217 sent from the sender to the receiver. The delay difference or variation between two or more data packets can be calculated.

[0038] In some cases, the change in delay can be calculated as the difference between the current delay value and the previous delay value. The previous delay value can be the delay value of a single data packet, or the average value of a plurality of data packets previously obtained. In some cases, the previous delay value can be the delay value 221 of the first packet 213 in the interval 227. In some cases, the previous delay value can be the minimum delay value. In some cases, the previous delay value can be the average value of a plurality of previously obtained delay values, such as a moving average of a first plurality of data packets or a plurality of data packets. In some cases, the previous delay value can be generated based on data from the previous interval. The previous delay value can be fixed or variable during the interval 227. For example, when the minimum delay value tracked by the receiver 203 is used as the previous delay value, the previous delay value can change when the minimum value is updated. For example, the data packet 217 may have the minimum delay value 223 in the interval, and then for subsequent data packets, the change in delay can be calculated as the difference between the corresponding delay value and the minimum delay value. Similarly, when the delay value 221 of the first data packet is taken as the previous delay value, for all subsequent data packets in the interval, the change in delay may be calculated as the difference between the respective delay values ​​and the delay value of the first packet.

[0039] In some cases, the change in delay is determined as the difference between the current delay value and the previous delay value. The current delay value may be the delay value of a single packet. The current delay value may be a moving average of multiple packets. This may provide the advantage of avoiding instantaneous spikes in the delay measurement. Various other methods or algorithms may be used to filter spikes. In some cases, the change in delay may be a value derived from the current delay value and the previous delay value. For example, the change in delay may be dimensionless, i.e., the difference between the current delay value and the previous delay value divided by the previous delay value (or the current delay value).

[0040] In some cases, the last packet 219 in the interval is a packet with a delay variation that exceeds a delay difference threshold. The current delay of the last packet may or may not be a moving average. The variation may be the absolute value of the difference between the current delay value and the previous delay value. The difference may be positive or negative. When the delay difference threshold is met, a notification message 216 may be generated.

[0041] In some embodiments, the change in delay may be determined by the receiver. Alternatively, the change in delay may be determined by the sender. In some embodiments, the state of network congestion may be determined by the receiver. Alternatively, the state of network congestion may be determined by the sender.

[0042] In some embodiments, the notification message may be generated by the receiver. The notification message may be sent from the receiver to the sender. When it is determined that the change in delay exceeds the delay difference threshold (e.g., message 215 sent at T2), the notification message may be sent immediately. The notification message may be sent at a later time point than the end of the interval (e.g., message 216 sent at T9). The notification message may include multiple variables indicating an increase or decrease in congestion status or queuing delay. In some embodiments, at least some of the information contained in the notification message is used by the sender for further congestion management.

[0043] Figure 3 A flow chart of a method for congestion management in a data link (eg, an Ethernet link) is shown. The flow chart includes an initial setup phase, a fast start phase, a transition phase, and an adjustment phase. The method may be Figure 1 The method is implemented in an environment where data transmission is performed between a sender 101 and a receiver 105 .

[0044] The sender and receiver may share information about each other's clock rate and line rate during the initial setup phase 310. This information allows the sender and receiver to determine acceptable delays and thus be able to detect congestion events by registering abnormally large delay values ​​(e.g., values ​​above a certain threshold for a data flow between a particular sender and receiver).

[0045] In the fast start phase 320, the sender sends data to the receiver at an initial rate. The fast start phase 320 may be preceded by a window growth phase, in which the sender may set an initial window size for sending data packets and increase the size of the window as long as no congestion is detected within the network. The initial window size may be set to, for example, the bandwidth-delay product of the line rate multiplied by the minimum propagation delay time within the network. As the window size increases, the receiver may periodically (e.g., every 50KB of data received) notify the sender that congestion has not been detected by sending packets to the sender. The maximum size to which the window can grow may be determined by its link rate multiplied by the maximum value of the propagation delay within the network. The receiver may detect congestion events by observing spikes in the one-way delay of the data packets it receives from the sender. The spikes in the one-way delay may correspond to a delay greater than the round trip time of the packet. For example, a delay of 15-25 microseconds may cause the receiver to register a congestion event. In some embodiments, the receiver may measure a delay relative to the minimum delay in order to register a congestion event. In these embodiments, the receiver may configure a threshold delay above which it may register a detected delay as a congestion event.

[0046] The sender can increase the window size exponentially. The sender can also increase the size linearly, logarithmically, or stepwise. The window size can be increased until a threshold size is reached. The packet can signal the sender to increase the window size using bits.

[0047] After the window grows in the fast start phase 320, the sender sets an initial rate. The initial rate can be the line rate, such as 25GBit / s or 100GBit / s. The initial rate may be affected by factors such as the type of physical network connecting the sender and the receiver, the physical distance between the sender and the receiver, and the operational configuration of the sender and the receiver. As long as the window is not overloaded, the sender can send traffic at full speed. The receiver can communicate with the sender using CNP packets. For example, the receiver can send a CNP-W packet to notify the receiver to keep sending at line speed, or send a CNP-R packet when a congestion event is detected.

[0048] When the receiver detects a congestion event, the system enters a transition phase 330. The receiver may send a packet to the sender with a bit indicating that the sender should reduce its data transmission rate to a minimum rate. The minimum rate may be significantly less than the initial rate. For example, the minimum rate may be two to four orders of magnitude less than the initial rate. The system may stay in the transition state for one or more cycles. The period may be based on the round trip time (RTT) of the packet. For example, the period may be 25 microseconds. This is done to reduce congestion in the system. After maintaining the minimum rate for a certain number of cycles, the system may increase the data transmission rate to half of the initial transmission rate. The receiver may also measure the throughput between the sender and the receiver. If the throughput is less than half of the initial rate, the receiver may send a packet to the sender, notifying the sender to send at half of the measured throughput. The sender may reduce the sending rate when receiving the packet.

[0049] The system stays in the transition phase for two update cycles and then enters the adjustment phase 340. The adjustment phase can use a feedback control algorithm and additive increase and multiplicative decrease (AIMD) to modify the data transmission rate.

[0050] The feedback control algorithm may be a proportional integral (PI) algorithm. The feedback control algorithm may also be implemented using a proportional algorithm or a proportional integral derivative (PID) algorithm. The PI algorithm may ensure efficiency and low latency. The calculation of the data transmission rate may be periodically updated based on how far the current delay is from a threshold value (e.g., a minimum delay value) and whether the queue delay is currently trending up or down. The "proportional" aspect of the algorithm makes it more likely that the algorithm will reduce the data transmission rate when the delay is high, while the "integral" aspect of the algorithm makes it more likely that the data transmission rate will be reduced when the queue delay is increasing.

[0051] The feedback control loop can be described using the following expression.

[0052] R(n+1)=R(n)-α(Δ(n)-θ)-β(Δ(n)-Δ(n-1))

[0053] In this expression, the update rate R(n+1) is calculated by subtracting the proportional term α(Δ(n)-θ) and the integral term β(Δ(n)-Δ(n-1)) from the current rate R(n). The proportional term describes the difference between the current delay Δ(n) and the threshold delay θ multiplied by a constant. The integral term describes the difference between the current delay Δ(n) and the previous delay Δ(n-1) multiplied by different constants to incorporate increasing or decreasing delay trends into the expression.

[0054] The AIMD algorithm operates in conjunction with a feedback control algorithm to ensure fairness in the system. The AIMD algorithm can perform a summative increase in the data transmission rate if the delay decreases, and a multiplicative decrease if the delay increases. This process adds fairness to the links between data streams and can be performed over longer time scales than those required to control delays. For example, the system can perform a summative increase in the rates of different data streams until the receiver detects a congestion event. At this point, the system can perform a multiplicative decrease in the data rate to prevent congestion. This process can be repeated until the streams have equal data rates.

[0055] In some embodiments, while the system is in the throttling phase, summative increases may be performed continually. While this is occurring, the receiver may send packets, such as CNP-R packets, to the sender to increase the rate. Multiplicative decreases may be performed periodically. In some embodiments, the system may perform a multiplicative decrease after a specific number of throttling cycles. In other embodiments, multiplicative increases may be performed in response to a probabilistic event such as a "coin toss." The system may assign a multiplicative decrease probability to the "coin" and "toss the coin" as each packet arrives to determine whether to perform a multiplicative decrease in the rate during that cycle. For example, if the multiplicative decrease probability is 30%, then it can be expected that 3 out of 10 packet transmissions, a multiplicative decrease will be performed.

[0056] The system can jointly execute the feedback control loop and the AIMD loop. The AIMD process can be executed at a lower frequency than the feedback control loop because fairness can be achieved over a long time scale. Periodically executing AIMD, less frequently than executing the feedback control loop, allows rebalancing of data flows with less disruption to packets arriving at the queue.

[0057] The adjustment phase 340 may continue until the data transmission has returned to the initial data rate. If the system recovers to the initial rate within a number of cycles, the system may begin the fast start process 320 again.

[0058] Figure 4 An efficiency graph 400 of communication is shown. The graph shows how the effective throughput (the number of useful information bits delivered per unit time) and the congestion buffer size 420 increase as the packet load at the link layer increases. The graph shows that as the data load increases from zero, the effective throughput 410 also increases linearly. When the packet load 450 increases beyond the capacity 440, the curve transitions from linear to flat, forming an inflection point in the graph. This is the optimal operating point 430 of the system, with maximum effective throughput and negligible queues. The following figure shows the buildup of queues as the packet load increases beyond the inflection point, because not all packets can be transmitted by the system after this point. The figure shows that as the queues build up, the effective throughput 410 remains high even under heavy load. The system can operate optimally using small standing queues. As the load increases, the size of the queues may grow exponentially because enqueued packets may not be transmitted before new packets are added to the queue. At certain loads, packets may be dropped because the system may not be able to handle maintaining such a large queue. The figure shows that in the disclosed method, the system operates near the inflection point.

[0059] Figure 5 A latency comparison 500 of the disclosed method and DCQCN is shown using data from experiments conducted with both methods. Figure 5 In the experiments, 20,000 data flows arrived at the receiver as a Poisson process for DCQCN and the disclosed method. The flows came from 16 different ports with different round-trip times (RTTs), all with a destination of the receiver. The arrival rate of packets to the receiver was controlled to perform experiments with 20%, 40%, 60%, and 80% load on the link between the sender and the receiver. For these 16 ports in both DCQCN and the method, the minimum pause duration, average pause duration, and maximum pause duration of Ethernet frames were measured. As can be seen from the figure, under high load, the performance of DCQCN is very poor compared to the disclosed method. In fact, the pause duration of DCQCN deteriorates exponentially under high load. In contrast, under high load, the disclosed method does not generate pause frames even under high load.

[0060] Fig. 6A and Figure 6B Shown from Figure 5 Illustration of the data flow in the experimental network. Fig. 6A Graph 600 shows the throughput from the disclosed method, and Figure 6B Graph 650 of FIG. 650 shows the throughput from the DCQCN. Graph 650 shows a significant victim flow. Victim flows or reduced throughput flows indicate that congestion is spreading across the data center network. Therefore, Fig. 6A and Figure 6B It is shown that there is significant congestion spread when DCQCN is used, but the congestion is negligible when the disclosed method is used.

[0061] Fig. 7A and Figure 7B A comparison of queue occupancy between the disclosed method 700 and DCQCN 750 is shown. The graph shows that the system has less data in the queue than the DCQCN. Fig. 7A and Figure 7B The chart reflects the results of the experiment. In the experiment, there are 640 nodes in a 3-layer data center network. Each node has two 25Gbps links connected to a pair of top-of-rack (ToR) switches. The round-trip propagation delay between nodes is 12 microseconds. PFC is enabled. The 640 nodes continuously transmit data packets to a group of 32 nodes under a pair of ToR switches. Each of the 32 nodes also transmits data packets to the other 639 nodes. The experiment deleted one link pointing to one of the 32 nodes in the middle of the simulation and observed how the deletion of the link affected the other servers. Fig. 7A and Figure 7B The queue occupancy of the congested link from the experiment under the disclosed method and DCQCN is compared, and it is shown that the disclosed method reduces the queue occupancy to only one-tenth of that under DCQCN.

Claims

1. A computer-implemented method for managing congestion in a network protocol, the method comprising: a) Detect congestion events in the network; b) slowing down the data transmission rate from the initial data transmission rate to the minimum data transmission rate; c) increasing the data transmission rate to half of the initial data transmission rate; d) periodically monitoring the one-way delay of the network; and e) Using a feedback control loop in combination with a sum-increase-multiplicative-decrease AIMD algorithm to adjust the data transmission rate, wherein the feedback control loop includes a proportional-integral controller, and wherein the AIMD algorithm is executed at a lower frequency than the feedback control loop to adjust the data transmission rate.

2. The computer-implemented method of claim 1 , wherein monitoring a one-way delay comprises performing a comparison of the one-way delay to a threshold delay and determining a trend of the one-way delay, and wherein the threshold delay is based on a round trip time for data packet transmission.

3. The computer-implemented method of claim 1, wherein the minimum data transfer rate is at least one percent less than the initial data transfer rate.

4. The computer-implemented method of claim 1 , wherein slowing the data transmission rate from an initial data transmission rate to a minimum data transmission rate comprises: A congestion notification packet is sent from the receiver to the sender, and the rate is updated when the sender receives the congestion notification packet.

5. The computer-implemented method of claim 1, wherein monitoring the one-way delay comprises keeping track of a total number of bytes of packets received.

6. The computer-implemented method of claim 1 , further comprising: f) obtaining at least the initial data transmission rate by adjusting the data transmission rate over a threshold number of cycles; as well as g) Monitoring the network for congestion events.

7. A system for managing network congestion in a network, the system comprising a plurality of computing nodes, the computing nodes comprising: a) a memory for storing one or more previously obtained delay values ​​and instruction sets; and b) one or more processors configured to execute the set of instructions to perform operations including: i) detecting a congestion event in the network; ii) slowing down the data transmission rate from the initial data transmission rate to the minimum data transmission rate; iii) increasing the data transmission rate to half of the initial data transmission rate; iv) periodically monitoring the one-way delay of the network; and v) using a feedback control loop in combination with a sum-increase-multiplicative-decrease AIMD algorithm to adjust the data transmission rate, wherein the feedback control loop includes a proportional-integral controller, and wherein the AIMD algorithm is executed at a lower frequency than the feedback control loop to adjust the data transmission rate.

8. The system of claim 7, wherein the initial data transmission rate is determined at least in part by setting an initial window equal to a bandwidth delay product, and increasing the window size if no congestion is detected after setting the initial window.

9. The system of claim 7, monitoring the one-way delay comprises performing a comparison of the one-way delay to a threshold delay and determining a trend of the one-way delay, and wherein the threshold delay is based on a round trip time of a data packet transmission.

10. The system of claim 7, wherein the minimum data transmission rate is at least one percent less than the initial data transmission rate.

11. The system of claim 7, wherein the plurality of computing nodes comprises a sender node and a receiver node.

12. The system of claim 11, wherein slowing down the data transmission rate from the initial data transmission rate to the minimum data transmission rate comprises: A congestion notification packet is sent from the receiving party to the sending party, and a rate is updated when the sending party receives the congestion notification packet.

13. The system of claim 7, wherein monitoring the one-way delay comprises keeping track of a total number of bytes of packets received.

14. A computer-implemented method for managing network congestion in a network, the computer-implemented method comprising: a) a fast start-up phase, including setting the data rate to an initial data rate and performing data transmission between the sender and the receiver until the receiver detects a congestion event; b) a transition phase comprising reducing the data rate to a minimum data rate during a first period and then increasing the data rate to half of the initial data rate during a second period; and c) A regulation phase, comprising executing a feedback control loop in combination with a sum-increase-multiplicative-decrease AIMD algorithm to modify the data rate, wherein the feedback control loop is a proportional-integral algorithm, and wherein the AIMD algorithm is executed at a lower frequency than the feedback control loop to adjust the data transmission rate.

15. The computer-implemented method of claim 14, further comprising an initial setup phase prior to the fast startup phase, including sharing a clock rate and a line rate between the sender and the receiver.

16. The computer-implemented method of claim 14, wherein the transition phase begins after the congestion event is detected.

17. The computer implemented method of claim 14, wherein the feedback control loop can be described as the difference between a current rate, a proportional term, and an integral term.

18. The computer-implemented method of claim 17, wherein the proportional term describes a difference between a current delay and a threshold delay and the integral term describes a difference between a current delay and a previous delay.

19. The computer-implemented method of claim 14, wherein the multiplicative reduction is performed in response to a probabilistic event.

20. The computer implemented method of claim 14, wherein the multiplicative reduction is performed at a lower frequency than the feedback control loop.

21. A system for managing network congestion in a network, comprising a plurality of computing nodes, the computing nodes comprising: a) a memory for storing one or more previously obtained delay values ​​and instruction sets; and b) one or more processors, the processors being configured to execute the instruction set to implement at least the following: i) a fast start-up phase, including setting the data rate to an initial data rate and performing data transmission between the sender and the receiver until the receiver detects a congestion event; ii) a transition phase, comprising reducing the data rate to a minimum data rate during a first period and then increasing the data rate to half of the initial data rate during a second period; as well as iii) a regulation phase, comprising executing a feedback control loop in combination with an AIMD algorithm of summation increase and multiplicative decrease to modify the data rate, wherein the feedback control loop comprises a proportional-integral algorithm, wherein the AIMD algorithm is executed at a lower frequency than the feedback control loop to adjust the data transmission rate.

22. The system of claim 21 further comprising an initial setup phase prior to the fast startup phase, including sharing a clock rate and a line rate between the sender and the receiver.

23. The system of claim 21, wherein the transition phase begins after the congestion event is detected.

24. The system of claim 21, wherein the feedback control loop can be described as the difference between a current rate, a proportional term, and an integral term.

25. The system of claim 24, wherein the proportional term describes a difference between a current delay and a threshold delay and the integral term describes a difference between a current delay and a previous delay.

26. The system of claim 21, wherein the multiplicative reduction is performed in response to a probabilistic event.

27. The system of claim 21, wherein the multiplicative reduction is performed at a lower frequency than the feedback control loop.

Images