REVERSE LOSS DETECTION FOR BANDWIDTH ESTIMATION OF COMMUNICATION NETWORKS USING TOKEN BUCKETS

Abstract

A receiver computing device (120; 220; 610) comprising: a memory (614); and one or more processors (612) configured to execute machine-readable instructions stored in the memory (614) to perform a method comprising: Receiving (616) a set of test data packets transmitted over a network path through a communications network (130; 230) from a sender computing device (110; 210); if packet losses are detected in the set of test data packets, selecting (618) a subset of the set of test data packets; for each packet of the subset of the set of test data packets, calculating (620) a significant loss value for the packet based on the packet losses detected from the set of test packets; selecting (622) a first packet in the subset of the set of test data packets, the first selected packet corresponding to the last received packet in the subset for which the significant loss value is greater than zero; selecting (622) a second packet in the subset of the set of test data packets, the second packet corresponding to a next packet received after the first selected packet; determining (624) a bandwidth estimate for the network path based on a probe rate of the second packet that corresponds to a current rate at which the second packet was sent by the sender computing device; and Executing (626) an action in connection with the bandwidth estimation for the network path in the communication network (130; 230).

Description

[0001] Bandwidth measurement is an essential component of any network traffic engineering solution, including solutions that use a software-defined wide area network (SD-WAN). The framework needs to know how much bandwidth is available on each network path before deciding where to place / route network traffic and load balancing.

[0002] In a closed system, it's possible to take direct measurements on each of the network devices along the traffic routes. However, in many cases, direct measurements aren't possible. For example, the network devices may be located in a different management area or hidden by tunneling or encapsulation. This is the case with SD-WAN, where the SD-WAN gateway attempts to route traffic along the best path across the internet.

[0003] Better solutions are needed for SD-WAN and other emerging network architectures.

[0004] Document US 2016 / 0344608 A1 describes a method for testing the bandwidth for communications between a first network station and a second network station. The test is performed using the "Probe Rate Model" (PRM) by iteratively sending a series of probe packets at a constant packet rate per iteration. The probe rate and the number of probe packets in the series are changed from iteration to iteration.

[0005] Document US 2021 / 0243133 A1 describes a method for measuring available bandwidth. A series of probe packets is received, and a network transmission delay is measured for at least two packets.

[0006] The present invention is defined by independent claims 1, 9 and 17. Embodiments are subject to the respective dependent claims. Brief description of the drawings

[0007] The present disclosure will be described in detail in accordance with one or more different examples with reference to the following figures. The figures are for illustrative purposes only and show only typical examples. Fig. Figure 1 shows a computer environment for implementing various bandwidth estimation methods. Fig. 2 shows a computing environment for implementing packet loss regression to estimate bandwidth according to some examples of the disclosure. Fig. Figure 3 illustrates the types of methods for reducing rates in accordance with some examples according to the application. Fig. 4 shows a summary of processes for determining a bandwidth estimate in accordance with some examples according to the application. Fig. Figure 5 shows packet loss associated with reverse loss detection in conjunction with some examples according to the application. Fig. 6 is an example of a computer component that may be used to implement various features of the examples described in the present disclosure. Fig. Figure 7 shows a block diagram of an example computer system in which various of the examples described herein may be implemented.

[0008] The figures are not exhaustive and do not limit the present disclosure to the precise form disclosed. Detailed description

[0009] Bandwidth measurement is challenging in an SD-WAN architecture, at least because cross-traffic can significantly impact the bandwidth measurement of a wired or wireless communication link (e.g., between edge devices, different users or applications sharing the same connections, etc.). Cross-traffic is often dynamic and can fluctuate continuously. Therefore, accurate bandwidth measurements in an SD-WAN are almost impossible to determine.

[0010] If direct measurements are not possible, bandwidth estimation can be performed from two endpoints that can be controlled or otherwise used for measurements. Bandwidth estimation can be performed, for example, by probing the network path with specially crafted probe packets sent from one end of the path to the other. The receiving end measures the packet reception time and changes in the packet delay / timing pattern to estimate path properties such as path capacity, available bandwidth, or bulk transfer capacity.

[0011] In some examples, a probe rate model (PRM) can be used to estimate the available bandwidth of a network path (e.g., using research projects such as PathChirp, PathCos++, SLDRT, etc.). The PRM can create a temporary congestion condition (e.g., a number of data packets transmitted on the network above a threshold to reduce the bandwidth on the network, etc.). Once the temporary congestion condition is established on the network path, a controller can measure the increase in probe packet latency to infer when the path is in a congestion condition. By determining when the path is in a congestion condition and when it is not, these methods can then calculate an estimate of the available bandwidth.

[0012] PRM can have several limitations. For example, the standard PRM can infer a congestion condition based on packet timing and assume that there is no packet loss on the network path. However, buffers on the network path are not infinite, so packet loss can occur if the congestion condition is large enough. Furthermore, some network devices respond to the congestion condition by producing only packet loss and never increasing the latency, which completely overrides PRM techniques and prevents them from measuring available bandwidth. Furthermore, some network links are unreliable and can randomly drop packets, making bandwidth estimation difficult.

[0013] Examples of the present disclosure may estimate available bandwidth in light of network congestion. For example, the system may measure available bandwidth in a black-box network. Conventional systems may perform bandwidth estimates using various techniques described herein (e.g., actively probing data packets, passive measurements to measure delays, etc.). The technique described here may find the point at which packet loss ceases and use the probe rate as a bandwidth estimate. Additional measures may be performed, such as automatically rerouting packets and / or load balancing network traffic after the probe rate is determined, as described throughout the disclosure.

[0014] In some examples, the system may specify a constant or varying probe rate. For example, the probe packets in a chirp train (e.g., a set of probe packets) may be sent at a predefined pattern of probe rates. The pattern of probe rates may correspond to a decreasing-rate pattern, such that the first few probe packets are sent at a higher probe rate, and the probe rate is gradually decreased as packets are sent. Any of these patterns may match the probe rate. A second probe rate may be a different pattern than the probe rate, including another varied pattern of probe rates.

[0015] In some examples, the transmitting computing device may transmit a single chirp train corresponding to a series of probe packets. The probe rates within the chirp train may be predetermined. In some examples, the probe rate of a packet within a chirp train is unique, and the chirp train as a whole may correspond to many different probe rates.

[0016] In some examples, a significant packet loss value is calculated. The significant packet loss value may correspond to a final point in a chirp train at which two of the packets are lost consecutively, although other measures of significant packet loss may also be calculated, as described in the disclosure. The next point after the significant packet loss value is the point at which no two packets in the set of probe data packets are lost consecutively. The significant packet loss value may be calculated based on the packets lost from a series of probe packets in the chirp train. Once the significant loss value is "below or equal to" a threshold, the significant loss value may approach zero. The process may select a first and a second packet from the chirp train.The first selected packet may be the last received packet for which the significant loss value is greater than zero. The second packet may be the first packet received after the first selected packet. Using the second packet, the process may determine the probe rate of the second packet as an estimate of the available bandwidth of the network path and use this value to perform an action (e.g., measuring available bandwidth in a black-box network, rerouting traffic, load balancing network traffic, etc.).

[0017] Fig. Figure 1 illustrates a computing environment for implementing various bandwidth estimation processes. In this figure, the sender computing device 110 transmits one or more data packets over the network 130 to the receiver computing device 120. The sender computing device 110 and the receiver computing device 120 (e.g., clients, servers, etc.) may be endpoints in a communications network that interconnects the endpoints. Different sender computing devices may send data packets to different receiver computing devices, although for simplicity, a single sender computing device and a single receiver computing device are shown.

[0018] An available bandwidth may be measured for sending network communications between the sender computing device 110 and the receiver computing device 120 to determine the available bandwidth for that network path. The network bandwidth may correspond to how much of the bandwidth is unused and how much could be added to the network path. The network path may be a wired or wireless communication link to transmit data across the network 130 in a given time. The available bandwidth may be reduced by traffic already using the network path.

[0019] One example of how available bandwidth can be utilized is network traffic engineering. The communication path across network 130 between sender computing device 110 and receiver computing device 120 may include multiple possible paths from one endpoint to another. Multiple paths between endpoints typically improve resiliency and can increase network bandwidth.

[0020] When planning network traffic, the process may determine the characteristics of network traffic and network elements, as well as their connectivity, to both assist in network planning and to route traffic along different paths within that network. In some examples, the primary path between the sender computing device 110 and the receiver computing device 120 may add a secondary path to be used in the event of a primary path failure.

[0021] Network traffic engineering can consist of three parts. The first part is measurement, which involves measuring some attributes of the traffic and / or the network. The measurement part of network traffic engineering can involve determining the available bandwidth(s) of various network paths, as described here. The second part is optimization, which involves calculating an optimal traffic distribution. The third part is control, which involves reconfiguring the network to achieve the desired traffic distribution.

[0022] Another illustrative method for estimating network bandwidth involves software-defined networking (SDN). Software-defined networking (SDN) can manage networks by defining application programming interfaces (APIs). These APIs can enable the system to decouple the data path (e.g., packet forwarding) and the control plane (e.g., protocol intelligence) of network elements. In other words, a network controller, an entity external to the network element, can have fine-grained control and visibility over that network element. This can be used by the network controller to dynamically change the policies of network elements or to centralize the network's control plane and decision-making.

[0023] The SDN approach can be combined with network traffic engineering. SDN APIs typically define both measurement and control, enabling the network controller to measure the network and dictate traffic distribution through network traffic engineering.

[0024] One of the limitations of SDN is that it assumes a tight coupling between the network controller and the network elements. This may work for small to medium-sized communication networks, but is generally not transferable to larger networks. The efficiency of the SDN process is reduced if the network between the network controller and the network elements has limited performance (e.g., low bandwidth or high latency). Furthermore, the SDN approach generally does not allow for crossing administrative domain boundaries, as the various entities can only rely on controlled and limited interactions with each other.

[0025] Another example where available bandwidth can be utilized is software-defined wide area networking (SD-WAN). The SD-WAN technique can be implemented in computing environments distributed across multiple physical locations, for example, when the sender computing device 110 and the receiver computing device 120 are located at different physical locations. Consequently, the computing environment may include a number of local area networks (LANs) supporting the device's local physical locations, as well as a number of wide area network (WAN) connections interconnecting these local networks.

[0026] For example, a large bank or retailer may have multiple locations and branches. Each location or branch has a number of LANs. In a traditional configuration, all branches and locations are connected to a few central locations via dedicated WAN links (e.g., using routing techniques such as Multiprotocol Label Switching (MPLS), etc.). These few central locations may be connected to an external network (e.g., the Internet, etc.) via one or more WAN links. The dedicated WAN links may be provided by a telecommunications company, which typically guarantees high availability and quality of service, but also incurs high costs.

[0027] The SD-WAN process proposes using SDN principles to manage WAN connectivity. This can provide centralized visibility and control over the enterprise's WAN connectivity and reduce WAN connectivity costs. SD-WAN can reduce costs by replacing dedicated WAN connections with tunnels over the internet. In this case, each branch and site has WAN links connected to the internet, typically using inexpensive consumer-grade WAN technology (e.g., DSL modem, cable modem, wireless 3G, etc.). The network can implement a dedicated SD-WAN gateway at each branch and site to create private tunnels (e.g., a virtual private network (VPN), etc.) that enable secure connectivity to other branches and sites over the WAN links and the internet.

[0028] Another example of leveraging available bandwidth is tunnel handoff. If an SD-WAN gateway detects that a WAN link is down, the gateway can redirect traffic away from that WAN link to a tunnel that doesn't utilize that particular WAN link, also known as tunnel handoff. The SD-WAN gateway can create parallel tunnels across network 130 that utilize each WAN link and then use network traffic engineering to route traffic to the most appropriate tunnel to maximize available network capacity. In some examples, the SD-WAN gateway can monitor the performance of each tunnel in terms of latency and throughput and then load balance or assign each traffic type to the tunnel best suited for that traffic.

[0029] One component of such traffic engineering is a way to measure the performance of each tunnel. Each tunnel defines a network path through the network, with tunneled packets being processed by a set of network elements. The network path used by a tunnel (e.g., outside the tunnel) and the network path inside a tunnel are logically different because they have different addresses. However, in some cases, two network paths can pass through the same network elements, have nearly identical performance, and their performance characteristics can be highly correlated. Therefore, the performance of a tunnel can be measured by measuring the network path outside the tunnel or the network path inside the tunnel.

[0030] In some examples, it is difficult to measure the estimated bandwidth for the tunnel handoff process. Direct measurement or an SDN approach cannot be used for these network paths because the vast majority of network elements of an internet path are located in different administrative areas (e.g., at the various ISPs along the path), and it would be very difficult to obtain the complete list of all these elements (which is dynamic in most cases) and administrative access to them. In some examples, the path measurement may be performed using end-to-end network path estimation techniques and / or by sending probe packets from one end to the other, i.e., from the sender computing device 110 to the receiver computing device 120.

[0031] Various methods can be used to determine bandwidth and network path. For example, if direct measurement of network elements is not possible (e.g., if they are located in different administrative areas), the next best approach may be end-to-end bandwidth estimation.

[0032] The end-to-end estimation of the network path may involve actively checking the data packets transmitted in the network 130. For example, the sender computing device 110 at one end of the network path sends special check packets to the receiver computing device 120 at the other end of the network path. These packets can only be used to estimate the bandwidth and must not contain any actual data beyond the payload required for the network path estimation itself.

[0033] The estimation process may also involve passive measurements, either by measuring the delays experienced by the data transmitted along the network path or by modulating that data to exhibit specific properties. Another variation is a single-ended measurement, in which the method initiates probe packets from the sender computing device 110, which are sent back to the sender computing device 110.

[0034] Different methods can estimate different network path properties. Bandwidth estimation is a subset of network path estimation. Path capacity is the maximum amount of traffic bandwidth that can be sent when the network path is unused, i.e., without competing traffic. Available bandwidth (ABW) is the remaining path capacity, i.e., the capacity not currently being used by other traffic. Bulk transfer capacity (BTC) is the bandwidth a Transmission Control Protocol (TCP) connection would receive if placed on this network path. Latency is the delay in one direction from the sender to the receiver, and round-trip time (RTT) is the delay in both directions.

[0035] During active probing, the sender computing device 110 sends a series of specially created probe packet patterns to the receiver computing device 120. The packet pattern may be defined by the estimation method and may be designed to trigger specific behaviors of the network elements on the network path. In many cases, for example, the packet pattern is a probe train, where the packets and the interval between the packets are used to probe various bandwidths across the packet pattern. The receiver computing device 120 may measure the reception time of the packets and calculate the one-way delay of each packet (i.e., the time it takes for a packet to travel from the sender device to the receiver device). The receiver computing device 120 may examine the changes in the packet pattern.The estimation procedure uses a simplified network model to convert these measurements into estimates of various network path properties.

[0036] Two main classes of bandwidth estimation can be: the probe spacing model (PGM) and the probe rate model (PRM). The PGM assumes that the spacing between two closely spaced packets increases proportionally to the load of the busiest queue, which is due to the waiting time in that queue. The PRM assumes that the traffic pattern remains largely unchanged when packets are sent at a rate lower than the bottleneck bandwidth, while packets sent at a rate higher than the bottleneck bandwidth are subject to additional queue delay due to network path congestion.

[0037] In practice, PGM and PRM attempt to infer network path congestion by estimating variations in the latency experienced by packets at different network elements along the network path. Queue delay can affect the time it takes for a packet to traverse the network path. PGM and PRM can compare the one-way delay of different probe packets to estimate latency variations. For example, with PGM, two packets can be sent at a known send interval. The measured receive interval is assumed to be the sum of the send interval and the difference in latency between the packets.

[0038] Another illustrative method for estimating network bandwidth involves packet one-way delay (OWD). With PRM, the method can measure the delays of received packets to determine the network path bandwidth. The measurement for each packet is performed using the OWD method. OWD corresponds to the time difference between the time the packet was sent by the sender computing device 110 (e.g., send time) and the time it was received by the receiver computing device 120 over the network 130 (e.g., receive time). Some methods compare the OWD of multiple packets.

[0039] In some examples, the OWD of a packet may be equal to the propagation delay of the network path, the transmission time of the slowest link in the path, and the cumulative queuing delay across all network elements in the path. For each packet, the formula for determining the OWD may include: OWD(i)=pd+st(size)+sum(qd(e,i)) where: pd -> total propagation delay st(size) -> slowest transmission time for this packet size qd(e,i) -> Queue delay at element e for packet i

[0040] In some examples, PRM can assume a queueing model where qd(e,i) is a function of the congestion state at element e when packet i arrives.

[0041] OWD has several limitations. For example, the sender computing device 110 and the receiver computing device 120 may use different clocks to measure the departure and arrival of packets, and these clocks may not be perfectly synchronized. Therefore, it is difficult to accurately measure the OWD of packets. Fortunately, in most cases, the absolute OWD of a packet is not important, and only the difference between the OWDs of different packets is significant. Over the short time span of a test train, clock drift may be negligible, and the difference between OWDs may be somewhat more accurate. When using relative OWD, the relative OWD is the measured OWD of this packet minus the OWD of the first packet. This relative OWD can estimate the additional queuing delay of a packet (e.g., compared to the first packet) and can be negative.

[0042] Another illustrative method for estimating network bandwidth includes a PathChirp process, which may use an implementation of PRM. For example, PathChirp may send a periodic probe train from the sender computing device 110 to the receiver computing device 120. A probe train is a sequence of probe packets sent with a specific time interval between the dispatch of the probe packets. The time intervals between the packets define an instantaneous rate, and the network can respond to this rate by either leaving the interval largely unchanged or sending the packets at wider intervals. With PathChirp, the probe rate over the probe train can be increased by decreasing the time interval between adjacent probe packets.

[0043] In some examples, the PathChirp process may measure the relative one-way delay of each probe packet in the train and attempt to find the inflection point where the delay changes from relatively constant to a constant increase. This change may indicate a congested path condition. The probe rate associated with the inflection point can be used as the primary basis for estimating available bandwidth.

[0044] Another illustrative method for estimating network bandwidth is the excursion detection algorithm (EDA). EDA can be integrated with PathChirp to detect the inflection point (e.g., the point at which the network path becomes completely congested). The same EDA can also detect periods of temporary congestion in the portion of the test train prior to the inflection point. The idea behind EDA is that an increase in excursion detection (ODD) in the test train can indicate increasing congestion on the network path. EDA uses these OWD increases to detect both temporary and complete congestion.

[0045] In some examples, the EDA can analyze the relative OWD of packets (e.g., by comparing the OWD of a first packet with the OWD of the previous packet, etc.). If the OWD of a first packet is greater than the OWD of the previous packet, the EDA can identify the first packet as the potential start of an excursion. As the probe train continues, the EDA can attempt to find a second packet whose OWD is sufficiently lower than the OWD of the first packet at the start of the excursion. If such a packet is found, it is marked as the end of the potential excursion. The EDA can then filter out the potential deviation. If the potential deviation is deemed too small, it can be ignored. If it is large enough, the probe rate associated with the packets in the excursion is used in calculating the available bandwidth. If no end of the deviation is found (e.g.,If the OWD never drops below a threshold, the beginning of the last deviation is used as the inflection point. The probe rate associated with the packet at the inflection point forms the primary basis for calculating the available bandwidth.

[0046] Another illustrative method for estimating network bandwidth is the PathCos++ method. PathCos++ can estimate the available bandwidth of a network path using the PRM, i.e., self-congestion. For example, PathCos++ sends a periodic probe train. The time interval between two packets can define an instantaneous rate, and the network responds to this rate. The probe rate can be decreased over the probe train by increasing the time between probe packets to create a congested state and then gradually de-congest it. The PathCos++ process can measure the relative one-way delay of each probe packet in the train and attempt to find a pair of packets that have a similar one-way delay on either side of the congestion peak. The similar one-way delay can correspond to packets that have similar congestion.The PathCos++ process can attempt to find the packet pair with the greatest separation and then calculate the average reception rate of the probe packets between the two packets in the packet pair. The average reception rate can be used as an estimate of the available bandwidth.

[0047] Another illustrative method for estimating network bandwidth is the bump detection algorithm (BDA). BDA can be integrated with PathCos++ to select a pair of packets with similar congestion on either side of the congestion peak. The selected packet pair can be used to estimate the available bandwidth. The quality of the available bandwidth estimate can only be as good as the BDA's selection of these packets.

[0048] In BDA, the test train has a decreasing rate to first create a congested state of the network path (e.g., rate higher than the bottleneck) and then decongest that network path (e.g., rate lower than the bottleneck). This means that the OWD of packets throughout the test train first increases (a congested state) and then decreases (a decongested state). The large jump in OWD may represent the time of maximum congestion, and packets with similar OWD should have experienced similar congestion (e.g., a similar amount of queueing).

[0049] Another illustrative process for estimating network bandwidth is congestion control via the Transmission Control Protocol (TCP). For example, bandwidth estimation is primarily used for traffic engineering and deciding how to distribute traffic across network paths. Since the vast majority of traffic uses the TCP protocol, the estimation of available bandwidth is essentially used in traffic engineering as a prediction of TCP performance.

[0050] TCP performance on a network path is largely determined by TCP congestion control. TCP congestion control attempts to determine the optimal transmission rate for the traffic by determining when the path is congested and when it is not. When the path is not congested, TCP congestion control increases the transmission rate of the sender computing device 110. When the path is congested, TCP congestion control decreases the transmission rate of the sender computing device 110.

[0051] Another illustrative method for estimating network bandwidth includes packet loss due to errors and congestion. Many networks are lossy, meaning packets may be lost or intentionally dropped. Some networks are lossless, meaning they guarantee that no packets are lost. Lossless networks may only be suitable for relatively small networks because they lead to undesirable effects such as head-of-line blocking at large scales. For this reason, most larger networks and network paths are considered lossy.

[0052] One cause of packet loss is bit errors in network devices and transmission errors over noisy links. However, these errors occur rarely. Most connection technologies deployed on noisy channels, such as Wi-Fi and 3G, use link-level acknowledgments and retransmissions to conceal link losses, so actual packet loss due to noise and errors is very rare unless conditions are bad. This is because TCP assumes packet loss is due to congestion, so optimal performance is achieved by eliminating loss at the link level.

[0053] On a lossy network path, packet loss is therefore almost always caused by congestion, and sustained congestion can lead to packet loss. If the incoming traffic rate is consistently higher than what a link can carry, the imbalance can be resolved by discarding the excess traffic, resulting in packet loss. As a result, packet loss can be a reliable indicator of network congestion.

[0054] Another illustrative process for estimating network bandwidth involves base queue tail-drop losses. The connection between sender computing device 110 and receiver computing device 120 over network 130 may be a simple fixed-capacity pre-connect queue. Sender computing device 110 may transmit the packet from the queue over the connection as quickly as the connection allows and drop the packet when the queue is full.

[0055] Traffic tends to be bursty, allowing the queue to absorb temporary overflows in the input rate and smooth processing on the link. As long as the queue isn't full, each received packet is either transmitted if the link is idle or added to the queue for transmission when the link allows. If the queue is full, any received packet may be dropped. Losses can occur at the end of the queue, which is why they are also called tail-drop losses.

[0056] With a perfect queue and constant traffic, losses can be fine-grained and evenly distributed across the chirp train. In practice, losses can be clustered and not evenly distributed across the chirp train. The first cause can be bursty cross-traffic, which changes the queue load between bursts, allowing the queue to accept more probe packets and fewer during the burst. The second cause can be granular queue scheduling, where multiple packets are dequeued together. After scheduling, several slots in the queue are available, but after these are filled, no slots open until the next schedule. As a result, packet losses with tail-drop often occur in clusters.In the received data traffic at the receiver computing device 120, sequences with low or no loss alternate with sequences with very high or complete loss. This makes the congestion signal quite coarse.

[0057] A tail-drop queue can behave in two main ways. If cross-traffic does not saturate the queue, the queue is mostly empty. In this case, the congestion caused by the chirp train can cause the queue to fill and the delay to increase until the queue is full. If cross-traffic saturates the queue, the queue is mostly full. In this case, the congestion caused by the chirp train can cause the delay to not increase (i.e., the queue can no longer fill up) and packets to be dropped immediately. The delay can fluctuate depending on the intensity of the cross-traffic filling the queue.

[0058] Another illustrative method for estimating network bandwidth involves losses through active queue management (AQM). AQM can address the shortcomings of tail-drop, including high latency and a rough congestion signal. AQM can add additional processing to the queue to make losses more proportional to the degree of congestion.

[0059] AQM can implement Random Early Detection (RED), which defines a drop probability based on queue occupancy. When the queue is almost empty, the probability is close to 0, and dropping packets is very unlikely. When the queue is almost full, this probability is higher, and a packet is more likely to be dropped. When a packet is received, RED can determine whether to drop the packet or queue it based on the current drop probability and a random number. Packet loss becomes increasingly likely depending on queue occupancy.

[0060] AQM can use probabilistic algorithms other than RED (e.g., Blue, ARED, PIE, CoDel, etc.) to keep the queue slightly full by preemptively dropping packets. For example, the CoDel process can attempt to keep the queue delay below a threshold for all packets. Another example is the PIE process with burst protection, which prevents any packets from being dropped for the first 150 ms after the queue begins to fill.

[0061] In some examples, the short train of packets used by the estimation methods may be too short to trigger a strong AQM response, and the losses caused by AQM may be very small for each chirp train. It may be much more likely that the chirp train will overfill the queue and cause tail-drop losses than that significant AQM losses will occur. Therefore, AQM queues can be equated with tail-drop queues for the purposes of bandwidth estimation.

[0062] Another illustrative method for estimating network bandwidth involves rate limiters, policers, and token buckets. Rate limiters can be used to limit network traffic to a specific rate. This can be done for policy reasons, for example, as a result of a contractual agreement.

[0063] Rate limiters can be implemented with or without a queue. When using a queue, the rate limiter is similar to a fixed-capacity connection and can be managed through tail-dropping or with the help of an AQM. When no queue is used, the rate limiter implementation is simplified and can reduce resource consumption. They are often called policers or meters to distinguish them from other rate limiters. These rate limiters can be implemented using a token bucket to account for traffic frequency.

[0064] If a token bucket is implemented, the token bucket can contain virtual tokens connected to its maximum capacity and is replenished in tokens at the desired rate. If a packet arrives and the bucket is not empty, the packet is forwarded and a token is removed. If the bucket is empty, the packet is discarded. If the connection has been idle for a while, the bucket may become full, and excess tokens beyond capacity are discarded. The token capacity (or burst size) can allow for an unrestricted rate if the connection has been underutilized.

[0065] Token bucket-based rate limiters do not have a queue, so packets cannot experience additional delays due to congestion. The congestion signal may be accompanied by packet loss. Packet loss can be quite fine-grained if the bucket size is appropriate for the link capacity and traffic frequency (i.e., not too small and not too large). In practice, configuring the burst size is difficult, so most token buckets do not produce uniform packet loss patterns.

[0066] Another illustrative method for estimating network bandwidth involves the PRM model and packet loss. The PRM model assumes that no probe packets are lost along the network path and that every transmitted packet is received and can be used by the method. None of the existing PRM methods attempt to tolerate packet loss, with the exception of the NEXT-v2 method. The NEXT-v2 method assumes that packet loss is due to random link errors. If a packet is lost, the NEXT-v2 method attempts to reconstruct it by interpolating from neighboring packets. The NEXT-v2 method can tolerate limited packet loss and is not useful for losses due to congestion. Other methods may assume that no estimation can be made if packet loss occurs in a chirp train.In these cases, the entire chirp train can be discarded and no estimation is performed. This can significantly reduce the probability of obtaining a bandwidth estimate.

[0067] PRM techniques may not accurately estimate available bandwidth if the bottleneck on a network path is based on a token bucket. For example, PRM techniques measure the OWD increase due to congestion. With a token bucket, there is no queue, and the bottleneck may not respond to congestion by increasing delay. In this sense, the congestion state created by the PRM technique does not increase OWD. In the presence of a token bucket, PRM techniques typically fail or provide an incorrect estimate. Therefore, PRM techniques currently cannot estimate available bandwidth when the bottleneck is a token bucket.

[0068] With various other methods for estimating available bandwidth in light of network congestion, the examples of the present disclosure enhance these and other methods. The technique described here can find the point at which packet loss ceases and use the probe rate as a bandwidth estimate. Additional actions can be performed, such as automatically rerouting packets and / or load balancing network traffic, after the probe rate is determined, as described in the disclosure.

[0069] Fig. Figure 2 illustrates a computing environment for implementing packet loss regression to estimate bandwidth, according to some examples of the disclosure. In the figure, sender computing device 210 transmits one or more data packets over network 230 to receiver computing device 220. Sender computing device 210 and receiver computing device 220 (e.g., clients, servers, etc.) may be endpoints in a communications network that interconnects the endpoints. Different sender computing devices may send data packets to different receiver computing devices, although for simplicity, a single sender computing device and a single receiver computing device are shown.

[0070] The sender computing device 210 may generate a probe train and send the probe packets over the network 230 to the receiver computing device 220. The receiver computing device 220 may receive the probe packets and estimate the available bandwidth. The process may perform bandwidth estimation in the presence of a bottleneck, responding to a congested network path condition by dropping packets and not increasing the delay by implementing a token bucket. The process may find the point at which packet loss ceases and use the probe rate at that point as a bandwidth estimate.

[0071] This process of bandwidth estimation using packet loss regression can have advantages over other processes discussed here. For example, the packet loss regression process can estimate available bandwidth when the bottleneck only drops packets and does not increase delay. This method can be correlated with the bump detection algorithm (BDA) of some other estimation methods and requires minimal modifications to those methods. Implementing this process in networks can be simple for engineers or technicians and can involve little overhead (e.g., minimizing CPU utilization, running quickly so as not to slow down network path estimation, etc.).

[0072] In some examples, the process can implement a reverse token bucket loss detection algorithm. Bottlenecks based on token bucket loss can be problematic for bandwidth estimation because they do not increase delay, and the BDA cannot be used to calculate a bandwidth estimate. Instead, a token bucket is used. The token bucket cannot queue or store packets. Therefore, if the token bucket is overloaded with traffic, it drops the excess packets. The probability of loss depends directly on the difference between the incoming traffic rate and the outgoing traffic rate of the token bucket. R_drop=R_input−R_output P_drop=R_drop / R_input=1−(R_ouput / R_input)

[0073] The incoming traffic is a mixture of the sample traffic used for the estimation and the cross traffic. For simplicity, we assume constant cross traffic. The available bandwidth (ABW) is therefore the difference between the capacity and the cross traffic: R_input=SR_probes+SR_cross ABW=R_output−SR_cross

[0074] In most cases, the losses can be distributed evenly between probe and cross traffic, so that the probe reception rate (RR_probes) depends directly on the loss probability. P_drop=1−(R_output / (SR_probes+SR_cross)) RR_probes=(1−P_drop)*SR_probes RR_probes=(R_output / (SR_probes+SR_cross))*SR_probes

[0075] If the probe rate is greater than the ABW, the probe reception rate is equal to the ABW when cross-traffic is zero. In the presence of cross-traffic, the reception rate is higher than the ABW. This is comparable to what happens in queues. This rate is called the asymptotic data rate (ADR).

[0076] The above equations may hold when the token bucket is congested (e.g., when the check rate is greater than the ABW). When it is not congested (a de-congested state), no packet is dropped, and the egress rate of any traffic is equal to the ingress rate.

[0077] PRM techniques that use an EDA determine the ABW by detecting the onset of congestion, i.e., the point at which the network path transitions from its previously uncongested (e.g., empty, etc.), uncongested, or unloaded state to its congested state. For example, the sender computing device 210 sends probe packets at an increasing rate, and the EDA detects the increase in delay as a signal of congestion. Various types of rate reduction techniques are described in Fig. 3 illustrates.

[0078] PRM techniques that use EDA can be adapted for token buckets by determining when packet loss begins. If the check rate is below the ABW, there should be no loss.

[0079] Once the check rate exceeds the ABW, packet loss should occur. Therefore, a loss detection algorithm (LDA) can be implemented to find the first occurrence of significant loss in the chirp train and use the corresponding check rate as a bandwidth estimate. However, such an LDA does not work in practice. Most token buckets have space at the onset of congestion (earlier packets affect later packets). Token buckets have a burst size. If the token bucket has been uncongested for a while, a number of tokens accumulate in the bucket. These accumulated tokens allow traffic to be sent at a much higher rate than the token bucket's set rate until they are used up. If the check rate is below the ABW value, the tokens are replenished faster than they are consumed, so the bucket remains full.If the check rate exceeds the ABW, tokens are consumed faster than they are replenished, so the number of tokens shrinks, but because tokens are available, no packets are dropped yet. After a number of packets of the chirp train have been sent over the ABW, the bucket is empty, and the token bucket begins to drop packets, but by this time, the check rate of the chirp train has already increased beyond the ABW, leading to an overestimation. This effect can be somewhat mitigated by burst detection techniques, but it introduces significant measurement inaccuracies.

[0080] The method can apply the LDA method in reverse, based on a decreasing chirp train, to search for the transition to a relieved state. The decreasing chirp train can reduce the check rates so that the check rate in the token bucket eventually falls below the ABW value and the losses cease. Reverse LDA (RLDA) can consider the last occurrence of significant packet loss and use the corresponding check rate as a bandwidth estimate. In this way, the method can perform bandwidth estimation for token buckets with decreasing chirp trains, eliminating the effect of the burst.

[0081] In the RLDA scheme, the bottleneck may not be a queue and may have no memory during the transition from a congested state. A token bucket can be implemented where no memory is available during a rebound state, and RLDA can be used to obtain a bandwidth estimate from the rebound point. With a token bucket at the rebound point, losses can cease, and the receive rate can become equal to the send rate (e.g., the packet check rate, which corresponds to the instantaneous rate at which the packet was sent by the sender).

[0082] The RLDA process can find the last point with significant packet loss in the received chirp train. Finding the last point is difficult for several reasons. First, packet loss can occur due to transmission errors. Such packet loss is relatively rare, which is why the RLDA searches for significant packet loss. This means there must be a way to distinguish significant from non-significant losses, which can increase the estimation error. Second, in most cases, the cross-traffic is bursty. This can cause the instantaneous ABW to vary and the losses to become more clustered. Packet loss may stop before or after the average ABW due to the bursty nature of the cross-traffic. When trying to find a specific point, the process may not perform averaging over a section of the chirp train.Third, the speed of the chirp train gradually decreases toward the ABW to test different rates, so that the probability of loss can also gradually decrease to zero. Just before reaching the ABW, the probability of loss is very low. This means that the chirp train near the ABW does not contain much loss information, and the RLDA can detect with a very small number of packet losses.

[0083] The combination of these three limitations can make a simple determination of the point at which significant packet loss ceases difficult and quite noisy. This is a similar shortcoming to EDA-based techniques: determining the inflection point in the chirp train is not robust in noisy conditions.

[0084] In some examples, the process can find the last point of significant packet loss in the chirp train based on clusters of packet losses. If the packet losses are separated by a few received packets, they can be considered part of the same loss cluster. The number of tolerable received packets in a cluster is based on the number of lost packets, which allows for the bursty nature of cross-traffic to be taken into account. Isolated packet losses can be ignored, and the last loss of the last cluster is used as the last point of significant packet loss. Because the process can use a token bucket (and the token bucket has no memory), each packet loss can be independent of the others, and the trend can be consistent across all computations, requiring fewer computational resources for analysis.

[0085] In some examples, the process may implement extrapolation of the trend of packet loss information to determine where the probability reaches zero. The first problem is to identify the portion of the chirp train to use to estimate this trend (e.g., the token bucket passes through various states with different trends, so a portion of the chirp train corresponding to the congested state is selected). Once the probe rate is below the ABW, the token bucket is no longer in a congested state and has a different trend (constant zero losses), so the corresponding portion of the chirp train can be eliminated. This may involve eliminating all packets after the last packet with significant losses. The process may also eliminate the portion of the chirp train that occurs before the detected end of the burst (e.g.,Burst detection involves determining the point at which the token bucket's burst phase ends and the token bucket enters the congested state. An alternative is to look at the time of the onset of significant losses, which usually marks the beginning of the congested state, and eliminate packets before that. A third option is to determine the point at which the check rate in the chirp train begins to decrease and eliminate packets before that. This third option yields a simpler trend for the data and is therefore preferred for easier extrapolation.

[0086] Extrapolation can be performed on the set of packets between the end of the burst or the beginning of the rate reduction and the packet with the last significant loss. Extrapolation can help determine the test rate at which packet loss becomes zero.

[0087] In some examples, extrapolation may involve linear regression. For example, a line may be drawn through the data and the check rate at which the loss probability reaches zero may be determined. The check rate can be used as a bandwidth estimate. The trend may be nonlinear and contain a small error.

[0088] In some examples, extrapolation can implement nonlinear regression (e.g., curve fitting the packet loss formula to the packet loss data, etc.). Similarly, the fitted curve can be used to calculate the probe rate at which the loss probability reaches zero. The probe rate can be used as a bandwidth estimate. This method reduces the error but can increase complexity. P_drop=1−(R_output / (SR_probes+SR_cross))

[0089] In some examples, nonlinear regression can be implemented by curve fitting to directly calculate the unknown parameters R_output and SR_cross and use them to determine the DEV.

[0090] Fig. 4 shows a summary of processes for determining a bandwidth estimate in accordance with some examples according to the application.

[0091] In block 410, the sender computing device 210 may send packets over the network 230 to the receiver computing device 220. The receiver computing device 220 may receive packets transmitted over a path through the network.

[0092] In block 420, the receiver computing device 220 may find the packet at which significant losses end. For example, the number of packet losses may be less than a threshold. The time at which the losses end and / or the number of packet losses may be determined.

[0093] In some examples, a number of consecutive losses may be used as a value for significant losses.

[0094] In some examples, the losses may be considered significant for various reasons. For example, they may be significant if they are part of a group of packet losses and / or are separated from other losses by a few received packets.

[0095] In block 430, the receiver computing device 220 may determine a bandwidth estimate. The bandwidth estimate may be based, for example, on a probe rate associated with this packet and / or the actually received rate of a subset of the chirp train. The lower of these two values ​​may be used as the bandwidth estimate.

[0096] In some examples, the packet check rate may be used as the overall check rate and / or bandwidth estimate.

[0097] In some examples, when using linear regression on the packet loss data before the packet determined in block 420 to calculate the test rate at which losses reach zero, this calculated test rate is used as the bandwidth estimate.

[0098] In some examples, when using a non-linear curve fit of packet loss data prior to the packet determined in block 420 to calculate the test rate at which losses reach zero, this calculated test rate is used as the bandwidth estimate.

[0099] In some examples, when using a non-linear curve fit of packet loss data prior to the packet detected in block 420 to determine the token bucket egress rate and the amount of cross-traffic and using them to calculate a probe rate, that probe rate is used as a bandwidth estimate.

[0100] In block 440, one or more actions may be performed in response to determining the bandwidth estimate.

[0101] In some examples, the action may include generating a report message containing the available bandwidth estimate. The report message may be sent to a traffic engineering module of the receiver computing device 220, a human user, or an external computer system configured to report or receive notifications.

[0102] In some examples, the action may involve automatically redirecting packets to reduce packet loss on a particular network path. For example, a new probe packet may be transmitted to a controller node to redirect packets in a black-box network.

[0103] In some examples, the action may include initiating load balancing of the network traffic after determining the probe rate. For example, the sender computing device 210 may send a request to a controller node to initiate load balancing in a black-box network.

[0104] In some examples, the action may include enabling or disabling some network path routes, connections, or access to network devices. In some examples, the action may include creating or stopping some network tunnels.

[0105] In some examples, the action may include reconfiguring some connections or network devices. This may include relocating devices or other resources.

[0106] In some examples, the action may include starting or stopping some network applications. This may include starting or stopping a backup process on a specific network device or rebooting the device altogether.

[0107] There are several advantages explained in this disclosure. For example, when using the reverse loss detection algorithm (RLDA), the process can perform bandwidth estimation in the presence of token buckets or network bottlenecks that simply drop packets. Previously, bandwidth estimation was either not performed or exhibited a large estimation error.

[0108] Additionally, this method may provide a better estimate of available bandwidth, which can improve the efficiency of traffic engineering for most networks, including in the context of SD-WAN.

[0109] Furthermore, the method can be combined with other techniques to estimate available bandwidth. In some examples, the same chirp train can be used by both RLDA and a BDA method. This means that the same chirp train can be used when the bottleneck is a queue or a token bucket. This can be particularly useful when cross-traffic varies and the bottleneck of a network path alternates between a queue and a token bucket. In some examples, the system can be upgraded without refactoring the code or changing packet formats.

[0110] Fig. Figure 5 shows packet loss associated with reverse loss detection in conjunction with some examples according to the application.

[0111] For example, using RLDA, the process can find the point of last significant loss, similar to EDA or PathChirp, only in reverse and looking for losses instead of received packets.

[0112] In another example, the process may consider the cluster of the last significant loss point using the last significant loss point. The cluster may include losses separated by a few received packets. In some examples, the cluster may be calculated with num_rcv < 2 * num_losses.

[0113] In another example, the method using interpolation may use linear regression as the loss rate. If nonlinear regression is used, the curve may correspond to the loss probability on the data. The calculation may be: P=1−(R_out / (SR_probe+SR_cross))

[0114] It should be noted that the terms "optimize," "optimal," and the like, as used herein, may be used to mean making or achieving performance as effective or perfect as possible. However, as one skilled in the art reading this document will recognize, perfection cannot always be achieved. Accordingly, these terms may also mean making or achieving performance as good or effective as possible or practical under the circumstances, or making or achieving performance better than that achievable with other settings or parameters.

[0115] Fig. Figure 6 shows an example of a computing component that can be used to implement reverse loss detection for estimating the bandwidth of a communication network with token buckets in accordance with various examples. As shown in Fig. 6, the computing component 610 may be, for example, a server computer, a controller, or other similar computing component that can process data. In the example implementation of Fig. 6, the computer component 610 includes a hardware processor 612 and a machine-readable storage medium 614.

[0116] The hardware processor 612 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices capable of fetching and executing instructions stored in the machine-readable storage medium 614. The hardware processor 612 may fetch, decode, and execute instructions, such as instructions 616-626, to control processes or operations for burst preloading to estimate available bandwidth. Alternatively, or in addition to fetching and executing instructions, the hardware processor 612 may include one or more electronic circuits comprising electronic components for performing the functionality of one or more instructions, such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other electronic circuitry.

[0117] A machine-readable storage medium, such as machine-readable storage medium 614, may be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. For example, machine-readable storage medium 614 may be random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), a storage device, an optical disk, or the like. In some examples, machine-readable storage medium 614 may be a non-transitory storage medium, where the term "non-transitory" does not include the transitory transmission signals. As described in detail below, machine-readable storage medium 614 may be encoded with executable instructions, such as instructions 616-626.

[0118] Hardware processor 612 may execute instruction 616 to receive a set of test data packets. For example, hardware processor 612 may receive a set of test data packets from a sender computing device that are transmitted over a network path through a communications network.

[0119] Hardware processor 612 may execute instruction 618 to select a subset of test data packets. For example, if packet loss is detected in the set of test data packets, hardware processor 612 may select a subset of the set of test data packets.

[0120] Hardware processor 612 may execute instruction 620 to calculate a significant loss value. For example, for each packet in the subset of the set of test data packets, hardware processor 612 may calculate a significant loss value for the packet based on the packet losses. In some examples, the packet losses are determined from the set of test packets.

[0121] Hardware processor 612 may execute instruction 622 to select the first or second packet. For example, hardware processor 612 may select a first packet in the subset of the set of test data packets. The first selected packet may correspond to the last received packet in the subset for which the significant loss value is greater than zero.

[0122] In some examples, hardware processor 612 may select a second packet from the subset of test data packets. The second packet may correspond to a next packet received after the first selected packet.

[0123] Hardware processor 612 may execute instruction 624 to determine a bandwidth estimate. For example, hardware processor 612 may determine a bandwidth estimate for the network path based on a probe rate of the second packet that corresponds to a current rate at which the second packet was transmitted by the sender computing device.

[0124] Hardware processor 612 may execute instruction 626 to perform an action associated with the bandwidth estimation. For example, hardware processor 612 may perform an action associated with the bandwidth estimation for the network path in the communications network.

[0125] In some examples, the hardware processor 612 may calculate the significant loss value for a packet as the number of lost packets between the packet and a previously received packet.

[0126] In some examples, the hardware processor 612 may calculate the significant loss value for a packet as the number of lost packets between the packet and a previously received packet less a fixed offset.

[0127] In some examples, hardware processor 612 may use a clustering algorithm to create one or more clusters of packet losses. The clusters of packet losses may be statistically related to each other. For a packet of the subset of the set of probe data packets, if more than one of the packet losses between the packet and a previously received packet are part of a cluster with a size greater than a threshold, the significant loss value of that packet is set to one; otherwise, the significant loss value of the packet is set to zero.

[0128] In some examples, the hardware processor 612 may calculate the significant loss value for a packet as a value given by a linear regression of a number of packet losses between consecutively received packets in the subset of the set of probe data packets.

[0129] In some examples, the hardware processor 612 may calculate the significant loss value for a packet as a value given by a nonlinear curve fit of a number of packet losses between consecutively received packets in the subset of the set of probe data packets.

[0130] In some examples, the subset of the set of probe packets is the part of a chirp train in which losses are present.

[0131] In some examples, a subset of a chirp train has a decreasing rate.

[0132] Fig. Figure 7 shows a block diagram of an example computer system 700 in which various of the examples described herein may be implemented. For example, the Fig. 2 illustrated transmitter computer device 210 or receiver computer device 220 as an example computer system 700 in Fig.7. Computer system 700 includes a bus 702 or other communication mechanism for conveying information, and one or more hardware processors 704 coupled to bus 702 for processing information. For example, hardware processor(s) 704 may be one or more general-purpose microprocessors.

[0133] Computer system 700 also includes main memory 706, such as random access memory (RAM), a cache, and / or other dynamic storage devices connected to bus 702, for storing information and instructions to be executed by processor 704. Main memory 706 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 704. When such instructions are stored in storage media accessible to processor 704, computer system 700 becomes a special-purpose machine adapted to perform the operations specified in the instructions.

[0134] Computer system 700 also includes a read-only memory (ROM) 708 or other static storage device connected to bus 702 for storing static information and instructions for processor 704. A storage device 710, such as a magnetic disk, an optical disk, or a USB flash drive, etc., is provided and connected to bus 702 for storing information and instructions.

[0135] Computer system 700 may be connected to a display 712, such as a liquid crystal display (LCD) (or a touch screen), via bus 702 for displaying information to a computer user. An input device 714, including alphanumeric and other keys, is coupled to bus 702 for communicating information and command selections to processor 704. Another type of user input device is cursor control 716, such as a mouse, trackball, or cursor direction keys for communicating direction information and command selections to processor 704 and controlling cursor movement on display 712. In some examples, the same direction information and command selections as with cursor control may be implemented via receiving touches on a touchscreen without a cursor.

[0136] Computer system 700 may include a user interface module for implementing a graphical user interface, which may be stored on a mass storage device as executable software code executed by the computing device(s). This and other modules may include, for example, components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0137] In general, the word "component," "engine," "system," "database," "data store," and the like, as used herein, may refer to logic embodied in hardware or firmware, or to a collection of software instructions that may have entry and exit points and may be written in a programming language such as Java, C, or C++. A software component may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as BASIC, Perl, or Python. It is understood that software components may be callable by other components or by themselves, and / or may be called in response to detected events or interrupts. Software components configured to execute on computing devices may be embodied on a computer-readable medium, such as a hard disk.a compact disc, digital video disc, flash drive, magnetic disk, or other tangible medium, or as a digital download (and may be originally stored in a compressed or installable format that must be installed, decompressed, or decrypted before execution). Such software code may be stored partially or entirely in a memory of the executing computing device so that it can be executed by the computing device. Software instructions may be embedded in firmware, such as an EPROM. In addition, the hardware components may consist of interconnected logic units such as gates and flip-flops and / or programmable units such as programmable gate arrays or processors.

[0138] Computer system 700 may implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, causes or programs computer system 700 to be a special-purpose machine. According to one example, the techniques described herein are performed by computer system 700 in response to processor(s) 704 executing one or more sequences of one or more instructions contained in main memory 706. Such instructions may be read into main memory 706 from another storage medium, such as computing device 710. Execution of the instruction sequences contained in main memory 706 causes processor(s) 704 to perform the process steps described herein.In alternative examples, hard-wired circuits may be used instead of or in combination with software instructions.

[0139] The term "non-transitory" and similar terms as used herein refer to any media that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 710. Volatile media includes dynamic memory, such as main memory 706. Common forms of non-transitory media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or other magnetic data storage media, CD-ROMs, other optical data storage media, physical media with hole patterns, RAM, PROM and EPROM, FLASH EPROM, NVRAM, other memory chips or cartridges, and networked versions thereof.

[0140] Non-transitory media are distinct from transmission media but can be used in conjunction with them. Transmission media are involved in the transfer of information between non-transitory media. Examples of transmission media include coaxial cables, copper cables, and fiber optic cables, including the wires that make up bus 702. Transmission media can also take the form of sound or light waves, such as those generated in data communications via radio and infrared.

[0141] Computer system 700 also includes a network interface 718 connected to bus 702. Network interface 718 establishes a two-way data communication connection to one or more network connections connected to one or more local area networks. For example, network interface 718 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for establishing a data communication connection to a corresponding type of telephone line. As another example, network interface 718 may be a Local Area Network (LAN) card for establishing a data communication connection to a compatible LAN (or a WAN component for communicating with a WAN). Wireless connections may also be implemented.In each of these implementations, the network interface 718 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams containing various types of information.

[0142] A network connection typically enables data communication across one or more networks to other data devices. For example, a network connection may connect across a local area network to a host computer or to data devices of an Internet service provider (ISP). The ISP, in turn, provides data communication services across the worldwide packet data communications network, now commonly referred to as the "Internet." Both the local area network and the Internet use electrical, electromagnetic, or optical signals that carry digital data streams. The signals across the various networks and the signals on the network connection and across the network interface 718 that carry the digital data to and from the computer system 700 are examples of transmission media.

[0143] Computer system 700 can send messages and receive data, including program code, over the network(s), the network connection, and the network interface 718. In the Internet example, a server could transmit requested code for an application program over the Internet, the ISP, the local network, and the network interface 718.

[0144] The received code may be executed by processor 704 upon receipt and / or stored in storage device 710 or other non-volatile memory for later execution.

[0145] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in, and fully or partially automated by, code components executed by one or more computer systems or computer processors comprising computer hardware. The one or more computer systems or computer processors may also operate to support the performance of the corresponding operations in a cloud computing environment or as software as a service (SaaS). The processes and algorithms may be partially or fully implemented in application-specific circuitry. The various features and methods described above may be used independently or combined in various ways.Various combinations and subcombinations are intended to be within the scope of this disclosure, and certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular order, and the associated blocks or states may be executed in other suitable orders, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed examples. The execution of certain operations or processes may be distributed among computer systems or computer processors located not only in a single machine, but distributed across a number of machines.

[0146] As used herein, a circuit may be implemented in any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms may be implemented to form a circuit. In implementation, the various circuits described herein may be implemented as discrete circuits, or the described functions and features may be distributed, in part or in whole, among one or more circuits.Although various features or functional elements are individually described or claimed as separate circuits, these features and functions may be shared by one or more common circuits, and such description is not intended to assume or imply that separate circuits are required to implement these features or functions. If a circuit is implemented in whole or in part with software, that software may be implemented to operate with a computer or processing system capable of performing the functionality described with respect to it, such as computer system 700.

[0147] As used herein, the term "or" can be understood both inclusively and exclusively. Furthermore, descriptions of resources, operations, or structures in the singular should not be construed as excluding the plural. Conditional expressions such as "may," "could," "might," or "may" are generally intended to convey that certain examples include certain features, elements, and / or steps, while other examples do not, unless explicitly stated otherwise or understood differently in the relevant context.

[0148] Unless expressly stated otherwise, the terms and expressions used in this document, as well as their variations, are not to be interpreted as limiting but as open-ended. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar import are not to be construed as limiting the subject matter described to a particular period of time or to a subject matter available at a particular time, but should be understood to include conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future.The presence of broader words and phrases such as “one or more,” “at least,” “but not limited to,” or similar phrases in some cases should not be construed as meaning that the narrower case is intended or required in the absence of such broader phrases.

Claims

[1] A receiver computing device (120; 220; 610) comprising: a memory (614); and one or more processors (612) configured to execute machine-readable instructions stored in the memory (614) to perform a method comprising: Receiving (616) a set of test data packets transmitted over a network path through a communications network (130; 230) from a sender computing device (110; 210); if packet losses are detected in the set of test data packets, selecting (618) a subset of the set of test data packets; for each packet of the subset of the set of test data packets, calculating (620) a significant loss value for the packet based on the packet losses detected from the set of test packets; selecting (622) a first packet in the subset of the set of test data packets, the first selected packet corresponding to the last received packet in the subset for which the significant loss value is greater than zero; selecting (622) a second packet in the subset of the set of test data packets, the second packet corresponding to a next packet received after the first selected packet; determining (624) a bandwidth estimate for the network path based on a probe rate of the second packet that corresponds to a current rate at which the second packet was sent by the sender computing device; and Executing (626) an action in connection with the bandwidth estimation for the network path in the communication network (130; 230). [2] The receiver computing device (120; 220; 610) of claim 1, wherein the machine-readable instructions further perform the method comprising: Calculate the significant loss value for a packet as a number of packets lost between the packet and a previously received packet. [3] The receiver computing device (120; 220; 610) of claim 1, wherein the machine-readable instructions further execute the method comprising: Calculate the significant loss value for a packet as the number of packets lost between the packet and a previously received packet, minus a fixed offset. [4] The receiver computing device (120; 220; 610) of claim 1, wherein the machine-readable instructions further perform the method comprising: Using a clustering algorithm to generate one or more clusters of packet losses, wherein the clusters of packet losses are statistically related to each other; for a packet of the subset of the set of test data packets: if more than one of the packet losses between the packet and a previously received packet is part of a cluster with a size greater than a threshold, setting the significant loss value of that packet to one, otherwise, the packet's significant loss value is set to zero. [5] The receiver computing device (120; 220; 610) of claim 1, wherein the machine-readable instructions further execute the method comprising: Calculating the significant loss value for a packet as a value given by a linear regression of a number of packet losses between consecutively received packets in the subset of the set of probe data packets. [6] The receiver computing device (120; 220; 610) of claim 1, wherein the machine-readable instructions further execute the method comprising: Calculating the significant loss value for a packet as a value given by a nonlinear curve fitting of a number of packet losses between consecutively received packets in the subset of the set of probe data packets. [7] The receiver computing device (120; 220; 610) of any one of claims 1 to 6, wherein the subset of the set of test data packets is the part of a chirp train in which losses are present. [8] The receiver computing device of any one of claims 1 to 7, wherein a subset of a chirp train has a decreasing rate. [9] A computer-implemented method for determining a bandwidth estimate for a network path in a communications network (130; 230), the method comprising: Receiving (410) a set of test data packets transmitted over the network path through the communication network (130; 230) from a sender computing device (110; 210); if packet losses are detected in the set of test data packets, selecting a subset of the set of test data packets; for each packet of the subset of the set of test data packets, calculating a significant loss value for the packet based on the packet losses detected from the set of test packets; selecting (420) a first packet in the subset of the set of test data packets, the first selected packet corresponding to the last received packet in the subset for which the significant loss value is greater than zero; selecting (420) a second packet in the subset of the set of test data packets, the second packet corresponding to a next packet received after the first selected packet; determining (430) a bandwidth estimate for the network path based on a probe rate of the second packet that corresponds to a current rate at which the second packet was sent by the sender computing device; and Executing (440) an action in connection with the bandwidth estimation for the network path in the communication network (130; 230). [10] The method of claim 9, further comprising: Calculate the significant loss value for a packet as a number of packets lost between the packet and a previously received packet. [11] The method of claim 9, further comprising: Calculate the significant loss value for a packet as the number of packets lost between the packet and a previously received packet minus a fixed offset. [12] The method of claim 9, further comprising: Using a clustering algorithm to generate one or more clusters of packet losses, wherein the clusters of packet losses are statistically related to each other; for a packet of the subset of the set of test data packets: if more than one of the packet losses between the packet and a previously received packet is part of a cluster with a size greater than a threshold, setting the significant loss value of that packet to one, otherwise, the significant loss value of the packet is set to zero. [13] The method of claim 9, further comprising: Calculating the significant loss value for a packet as a value given by a linear regression of a number of packet losses between consecutively received packets in the subset of the set of probe data packets. [14] The method of claim 9, further comprising: Calculating the significant loss value for a packet as a value given by a nonlinear curve fitting of a number of packet losses between consecutively received packets in the subset of the set of probe data packets. [15] The method of any one of claims 9 to 14, wherein the subset of the set of test data packets is the part of a chirp train in which losses are present. [16] The method of any one of claims 9 to 15, wherein a subset of a chirp sequence has a decreasing rate. [17] A non-transitory computer-readable storage medium (614) storing a plurality of instructions executable by one or more processors (612), the plurality of instructions, when executed by the one or more processors (612), causing the one or more processors (612) to: Receiving (616) a set of test data packets transmitted over the network path through the communication network (130; 230) from a sender computing device (110; 210); if packet losses are detected in the set of test data packets, select a subset of the set of test data packets (618); for each packet of the subset of the set of test data packets, calculate a significant loss value for the packet based on the packet losses detected (620) from the set of test packets; selecting (622) a first packet in the subset of the set of test data packets, the first selected packet corresponding to the last received packet in the subset for which the significant loss value is greater than zero; selecting (622) a second packet in the subset of the set of test data packets, the second packet corresponding to a next packet received after the first selected packet; determining (624) a bandwidth estimate for the network path based on a probe rate of the second packet that corresponds to a current rate at which the second packet was sent by the sender computing device; and perform an action in connection with the bandwidth estimation for the network path in the communication network (130; 230) (626). [18] The computer-readable storage medium (614) of claim 17, further comprising: Calculate the significant loss value for a packet as a number of packets lost between the packet and a previously received packet. [19] The computer-readable storage medium (614) of claim 17, further comprising: Calculate the significant loss value for a packet as the number of lost packets between the packet and a previously received packet minus a fixed offset. [20] The computer-readable storage medium (614) of claim 17, further comprising: Using a clustering algorithm to generate one or more clusters of packet losses, wherein the clusters of packet losses are statistically related to each other; for a packet of the subset of the set of test data packets: if more than one of the packet losses between the packet and a previously received packet are part of a cluster with a size greater than a threshold, set the significant loss value of that packet to one, otherwise, the packet's significant loss value is set to zero.

Images