Device, method and medium for determining bandwidth estimation

Abstract

A system and method for measuring the available bandwidth available in a black box network by determining a probe rate for packet transmission between a sender and a receiver is provided. An optimal probe rate and bandwidth estimate can be determined. Additional actions can be performed, such as automatically rerouting packets and / or load balancing network traffic after the probe rate is determined.

Description

Background Art

[0001] Bandwidth measurement is an essential component of any network traffic engineering solution, including those using Software Defined Wide Area Networks (SD-WAN). The framework needs to know how much bandwidth is available on each network path before it can decide where to place / route and load balance network traffic.

[0002] In a closed system, it is possible to collect direct measurements on each network device on the traffic path. However, in many cases, it is not possible to collect direct measurements. For example, network devices may be in different administrative domains or may be hidden through tunnels or encapsulation. This is the case with SD-WAN, where the SD-WAN gateway attempts to direct traffic to the best path over the Internet.

[0003] SD-WAN and other increasingly important network architectures require better solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] According to one or more different examples, the present disclosure is described in detail with reference to the following drawings. These drawings are for illustration purposes only and depict only typical examples.

[0005] Figure 1 A computing environment for implementing various bandwidth estimation processes is shown.

[0006] Figure 2 A computing environment for implementing packet loss regression to estimate bandwidth according to some examples of the present disclosure is shown.

[0007] Figure 3 Types of rate reduction methods according to some examples of the present application are shown.

[0008] Figure 4 An overview of a process for determining a bandwidth estimate according to some examples of the present application is shown.

[0009] Figure 5 Packet losses associated with reverse loss detection are shown in conjunction with some examples of the present application.

[0010] Figure 6 are example computing components that can be used to implement various features of the examples described in this disclosure.

[0011] Figure 7 Depicted is a block diagram of an example computer system in which various examples described herein may be implemented.

[0012] The drawings are not intended to be exhaustive and do not limit the disclosure to the precise forms disclosed. DETAILED DESCRIPTION

[0013] Bandwidth measurement is difficult in an SD-WAN architecture, not least because cross traffic can greatly impact bandwidth measurements of wired or wireless communication links (e.g., between edge devices, various users or applications using the same link, etc.). Cross traffic is often dynamic and can fluctuate constantly, making it nearly impossible to determine accurate bandwidth measurements in an SD-WAN.

[0014] When direct measurement is not possible, bandwidth estimation can be achieved from two endpoints that can be controlled or otherwise used for measurement. For example, bandwidth estimation can be performed by probing a network path using specially crafted probe packets sent from one end of the path to the other end of the path. The receiving end measures the reception time of the packets and the changes in packet delay / time patterns to estimate path characteristics such as path capacity, available bandwidth, or bulk transfer capacity.

[0015] In some examples, a Probe Rate Model (PRM) can be used to estimate the available bandwidth of a network path (e.g., using research projects including PathChirp, PathCos++, SLDRT, etc.). The PRM can create a temporary congestion state (e.g., the number of data packets transmitted on the network exceeds a threshold to slow down the bandwidth on the network, etc.). Once a temporary congestion state is created on a network path, the controller can measure the increase in queuing delay of the probe packets to infer when the path is in a congested state. By determining when a path is in a congested state and when it is not in a congested state, these methods can calculate an estimate of the available bandwidth.

[0016] PRM can have several limitations. For example, standard PRM can use packet timing to infer congestion conditions and can assume that there is no packet loss on the network path. However, buffers on network paths are not infinite, so if the congestion condition is severe enough, packet loss may occur. Additionally, some network devices react to congestion conditions by exclusively generating packet losses and never increasing queuing delays, which completely defeats PRM methods and prevents them from measuring available bandwidth. Furthermore, some network links are unreliable and can randomly drop packets, hindering bandwidth estimation.

[0017] Examples of the present disclosure can estimate available bandwidth taking into account network congestion. For example, the system can measure the available bandwidth available in a black box network. Conventional systems can use various methods discussed herein (e.g., active probing of data packets, passive measurements of measured delays, etc.) to perform bandwidth estimation. The techniques described herein can find the point at which packet loss stops and use the probing rate as a bandwidth estimate. Additional actions can be performed, such as automatically rerouting packets and / or load balancing network traffic after determining the probing rate, as described throughout the disclosure.

[0018] In some examples, the system can determine a constant or variable probing rate. For example, the probing packets can be sent in a chirp train (e.g., a set of probing data packets) with a predefined pattern of probing rates. The pattern of the probing rate can correspond to a decreasing rate pattern, such that the first few probing packets are sent at a higher probing rate, and as the packets are sent, the probing rate gradually decreases. Each of these patterns can correspond to a probing rate. The second probing rate can be a pattern different from the probing rate, including a different changing probing rate pattern.

[0019] In some examples, the sending computing device may send a single chirp sequence corresponding to a set of sounding data packets. The sounding rate in the chirp sequence may be predetermined. In some examples, the sounding rate of the packet is unique within the chirp sequence, and the entire chirp sequence may correspond to many different sounding rates.

[0020] In some examples, a significant packet loss value is calculated. Although other measurements of significant packet loss can be calculated, the significant packet loss value can correspond to the last point in the chirp sequence where two packets can be lost consecutively, as described throughout the disclosure. The next point after the significant packet loss value is a position where no two packets are lost consecutively in the set of probe data packets. The significant packet loss value can be calculated based on packets lost from the set of probe data packets in the chirp sequence. Once the significant loss value is "lower than or equal to" a threshold, the significant loss value can be close to 0. The process can select a first packet and a second packet from the chirp sequence. The first packet selected can be the last received packet with a significant loss value greater than 0. The second packet can be the first packet received after the selected first packet. Using the second packet, the process can determine the probe rate of the second packet as an available bandwidth estimate for the network path and use the value to perform an action (e.g., measuring available bandwidth in a black box network, rerouting services, load balancing network services, etc.).

[0021] Figure 1 A computing environment for implementing various bandwidth estimation processes is shown. In this illustration, a sender computing device 110 transmits one or more data packets to a receiver computing device 120 via a network 130. The sender computing device 110 and the receiver computing device 120 (e.g., a client, a server, etc.) can be endpoints in a communication network that connects endpoints. Although a single sender computing device and a single receiver computing device are provided for simplicity of illustration, various sender computing devices can send data packets to various receiver computing devices.

[0022] The available bandwidth for sending network communications between the sender computing device 110 and the recipient computing device 120 may be measured to determine the available bandwidth of the network path. The network bandwidth may correspond to how much bandwidth is not being used and how much bandwidth can be added to the network path. A network path may be a wired or wireless communication link used to transmit data over the network 130 in a given amount of time. The available bandwidth may be reduced by traffic that is already using the network path.

[0023] Network traffic engineering is one example of how available bandwidth can be used. The communication path between the sender computing device 110 and the recipient computing device 120 via the network 130 may include multiple possible paths from one endpoint to another. Having multiple paths between endpoints generally improves resilience to failures and may increase network bandwidth.

[0024] In network traffic engineering, the process can determine the properties of network traffic and network elements and their connectivity, both of which help design the network and direct traffic to different paths in the network. In some examples, the primary path between the sender computing device 110 and the recipient computing device 120 can add a secondary path to be used in the event of a failure of the primary path.

[0025] Network traffic engineering can consist of three parts. The first part is measurement, where some properties of the traffic and / or network are measured. As described herein, the measurement part of network traffic engineering can be determined in conjunction with the available bandwidth (s) of various network paths. The second part is optimization, where the optimal allocation of traffic is calculated. The third part is control, where the network is reconfigured to achieve the desired traffic allocation.

[0026] Another illustrative network bandwidth estimation process includes software defined networking (SDN). Software defined networking (SDN) can manage the network by defining application programming interfaces (APIs). The API can allow the system to decouple the data path (e.g., packet forwarding) and control plane (e.g., protocol intelligence) of network elements. In other words, a network controller, an entity external to the network element, can have fine-grained control and visibility over the network element. This can be used by the network controller to dynamically change the policy of the network element or centralize the control plane and decision making of the network.

[0027] SDN methods can be combined with network traffic engineering. For example, SDN APIs usually define both measurement and control, which can enable a network controller to measure the network and instruct traffic allocation through network traffic engineering.

[0028] One of the limitations of SDN is that it assumes tight coupling between the network controller and the network elements. This works in small and medium-sized communication networks, but it does not usually scale to larger networks. If the network between the network controller and the network elements has limited performance (e.g., low bandwidth or high latency), the efficiency of the SDN process is reduced. In addition, SDN approaches usually do not allow crossing administrative domain boundaries, because different entities can only trust controlled and restricted interactions with each other.

[0029] Another example where available bandwidth may be used is a software defined wide area network (SD-WAN). The SD-WAN process may be implemented in a computing environment distributed across multiple physical locations, including where the sender computing device 110 and the recipient computing device 120 are located in different physical locations. Thus, the computing environment may include a set of local area networks (LANs) supporting local physical locations of the entities and a set of wide area network (WAN) links connecting these local networks to each other.

[0030] For example, a large bank or retailer may have multiple physical locations and branches. Each location or branch has a set of LANs. In a traditional configuration, all branches and locations are connected to a few central locations using dedicated WAN links (e.g., using routing technologies such as Multiprotocol Label Switching (MPLS)), and a few central locations may be connected to external networks (e.g., the Internet, etc.) using one or more WAN links. Dedicated WAN links may be provided by telecommunications companies, which typically comply with high availability and quality of service guarantees, but also have a higher monetary cost.

[0031] The SD-WAN process proposes to use SDN principles to manage WAN connectivity. This can provide centralized visibility and control over an entity's WAN connections and reduce the cost of WAN connectivity. SD-WAN can reduce costs by replacing dedicated WAN links with tunnels over the Internet. In this scenario, each branch office and location has a WAN link connected to the Internet, typically using inexpensive consumer WAN technology (e.g., digital subscriber line (DSL) modem, cable modem, wireless 3G, etc.). The network can implement special SD-WAN gateways in each branch office and location to create dedicated tunnels (e.g., virtual private network (VPN), etc.) to securely connect to other branches and locations over the WAN link and the Internet.

[0032] Another example of how available bandwidth can be used is tunnel switching. When the SD-WAN gateway detects that a WAN link is down, the gateway can direct traffic from that WAN link to a tunnel that does not use that particular WAN link, also known as tunnel switching. The SD-WAN gateway can create parallel tunnels on the network 130 using each WAN link, and then use network traffic engineering to direct traffic to the most appropriate tunnel, with the goal of making the best use of available network capacity. In some examples, the SD-WAN gateway can monitor the performance of each tunnel in terms of latency and throughput, and then load balance the traffic or map each traffic type to the most appropriate tunnel for that traffic.

[0033] An integral part of this traffic engineering is a way to measure the performance of each tunnel. Each tunnel defines a network path through the network, with tunnel packets being processed by multiple network elements. The network path used by the tunnel (e.g., outside the tunnel) and the network path inside the tunnel are logically different because they have different addresses. However, in some cases, two network paths can pass through the same network elements with nearly identical performance, and their performance characteristics can be strongly correlated. Therefore, the performance of a tunnel can be measured by measuring the network path outside the tunnel or inside the tunnel.

[0034] In some examples, it is difficult to measure bandwidth estimates for the tunnel switching process. For example, direct measurement or SDN methods cannot be used for these network paths because the vast majority of network elements of an Internet path are located in different administrative domains (e.g., various ISPs on the path), and it is difficult to obtain a complete list of all of these elements (dynamic in most cases) and management access to them. In some examples, path measurement can be done via end-to-end network path estimation methods and / or by sending probe packets from one end to the other, from the sender computing device 110 to the receiver computing device 120.

[0035] Various methods for determining bandwidth estimates and network path estimates may be implemented.For example, when it is not possible to directly measure network elements (eg, when they are in different administrative domains, etc.), the next best procedure may be end-to-end bandwidth estimation.

[0036] End-to-end network path estimation may include actively probing data packets transmitted in the network 130. For example, a sender computing device 110 at one end of the network path sends special probing packets to a receiver computing device 120 at the other end of the network path. These packets may only be used to estimate bandwidth and may not carry actual data beyond the data payload required for the network path estimation itself.

[0037] The estimation process may also include passive measurements, either by measuring the delay experienced by existing data transmitted on the network path, or by modulating the data to have special characteristics. Another variation is single-ended measurement, where the method is initiated by the sender computing device 110 with a probe packet that is reflected back to the sender computing device 110.

[0038] Different methods can estimate different properties of a network path. Bandwidth estimation is a subset of network path estimation. Path capacity is the maximum amount of traffic bandwidth that can be sent when the network path is idle, i.e. when there is no competing traffic. Available bandwidth (ABW) is the remaining / residual path capacity, i.e. the capacity that is not currently being used by other traffic. Bulk transfer capacity (BTC) is the bandwidth that a Transmission Control Protocol (TCP) connection would receive if placed on that network path. Latency is the one-way delay from sender to receiver, and round trip time (RTT) is the two-way delay.

[0039] With active probing, the sender computing device 110 sends a series of specially crafted probing packet patterns to the receiver computing device 120. The packet patterns can be defined by the estimation method and can be designed to trigger specific behaviors from network elements on the network path. For example, in many cases, the packet pattern is a sequence of probes, and the intervals between packets and packets are intended to probe various bandwidths across the packet pattern. The receiver computing device 120 can measure the reception time of the packets and calculate the one-way delay of each packet (i.e., the time it takes for the packet to reach the receiver device from the sender device). The receiver computing device 120 can check for changes in the packet pattern. The estimation method uses a simplified network model to convert these measurements into estimates of various network path characteristics.

[0040] For bandwidth estimation, two main categories can include the Probing Gap Model (PGM) and the Probing Rate Model (PRM). For the PGM, it is assumed that the interval between two closely transmitted packets will increase in proportion to the load on the most loaded queue due to the queuing delay on the queues. For the PRM, it is assumed that when packets are sent at a rate below the bottleneck bandwidth, the traffic pattern will remain essentially unchanged, while when packets are sent at a rate greater than the bottleneck bandwidth, the packets will suffer additional queuing delays due to the congested state of the network path.

[0041] In practice, PGM and PRM attempt to infer network path congestion by attempting to estimate the variation in queuing delays experienced by packets at different network elements in the network path. Queuing delays can affect the time required for a packet to traverse a network path. PGM and PRM can compare the one-way delays of various probe packets to estimate the variation in queuing delays. For example, using PGM, two packets can be sent with a known send interval. It is assumed that the measured receive interval is the sum of the differences in the send intervals and the queuing delays between the packets.

[0042] Another illustrative network bandwidth estimation process includes packet one-way delay (OWD). In PRM, the method can measure the delay of received packets to determine the network path bandwidth. The OWD process is used for the measurement of each packet. OWD corresponds to the time difference between when the sender computing device 110 sends the packet (e.g., the transmission time) and when the receiving computing device 120 receives the packet via the network 130 (e.g., the reception time). Some methods compare the OWD of multiple packets.

[0043] In some examples, the OWD of a packet may correspond to the propagation delay of the network path, the transmission time of the slowest link in the path, and the cumulative queuing delay of all network elements in the path. For each packet, a formula to determine the OWD may include:

[0044] OWD(i)=pd+st(size)+sum(qd(e,i))

[0045] in:

[0046] pd->total propagation delay

[0047] st(size)->slowest transmission time for this packet size

[0048] qd(e,i)->queueing delay for packet i at element e

[0049] In some examples, the PRM may assume a queuing model where qd(e,i) is a function of the congestion state at element e when packet i arrives.

[0050] OWD has several limitations. For example, the sender computing device 110 and the receiver computing device 120 may use different clocks to measure packet departures and arrivals, and these clocks may not be absolutely synchronized. Therefore, it is difficult to accurately measure the OWD of a packet. Fortunately, in most cases, the absolute OWD of a packet is not important, but rather the difference between the OWDs of different packets. Over the short time of a probe sequence, clock offsets may be negligible and the difference between OWDs may be slightly more accurate. When relative OWD is used, the relative OWD is the measured OWD of the packet minus the OWD of the first packet. The relative OWD can estimate the additional queuing delay of a packet (e.g., relative to the first packet) and can be negative.

[0051] Another illustrative network bandwidth estimation process includes a PathChirp process that may use an implementation of PRM. For example, PathChirp may send a periodic probe sequence from the sender computing device 110 to the receiver computing device 120. A probe sequence is a sequence of probe packets sent with a specific time interval between the departure of the probe packets. The time interval between packets defines the instantaneous rate, and the network may react to the rate by keeping the time interval substantially constant or spacing the packets. In PathChirp, the probe rate on the probe sequence may be increased by reducing the time interval between adjacent probe packets.

[0052] In some examples, the PathChirp process can measure the relative one-way delay of each probe packet in the sequence and attempt to find the inflection point where the delay changes from being relatively constant to consistently increasing. This change can indicate a congestion state in the path. The probe rate associated with the inflection point can serve as the primary basis for the available bandwidth estimate.

[0053] Another illustrative network bandwidth estimation process includes an Excursion Detection Algorithm (EDA). EDA can be integrated with PathChirp to help detect inflection points (e.g., where a network path becomes completely congested). The same EDA can also find periods of temporary congestion in the portion of the probe sequence before the inflection point. The idea of ​​EDA is that an increase in OWD in the probe sequence can indicate an increase in congestion in the network path. EDA uses these increases in OWD to detect both temporary congestion and complete congestion.

[0054] In some examples, the EDA may analyze the relative OWD of the packets (e.g., by comparing the OWD of the first packet to the OWD of the previous packet, etc.). If the OWD of the first packet is greater than the OWD of the previous packet, the EDA may identify the first packet as the possible start of an offset. When moving forward in the probing sequence, the EDA may attempt to find a second packet whose OWD is sufficiently lower than the OWD of the first packet at the start of the offset. If such a packet is found, it is marked as the possible end of the offset. The EDA may then filter out possible offsets. If a possible offset is deemed too small, it may be ignored. If it is large enough, the probing rate associated with the packets in the offset is used when calculating the available bandwidth. If the end of the offset is not found (e.g., the OWD never decreases beyond a threshold), the starting point of the final offset is used as the inflection point. The probing rate associated with the packets at the inflection point is the primary basis for calculating the available bandwidth.

[0055] Another illustrative network bandwidth estimation process includes the PathCos++ process. PathCos++ can use PRM to estimate the available bandwidth of a network path, i.e., self-congestion. For example, PathCos++ sends a periodic sequence of probes. The time interval between two packets can define the instantaneous rate, and the network will react to this rate. The probe rate can be reduced over the probe sequence by increasing the time between probe packets to create a congestion state and then gradually relieving the congestion. The PathCos++ process can measure the relative one-way delay of each probe packet in the sequence and try to find a pair of packets with similar one-way delays on both sides of the congestion peak. Similar one-way delays can correspond to packets with similar congestion. The PathCos++ process can try to find the widest interval packet pair and then calculate the average reception rate of the probe packets between the two packets of the packet pair. The average reception rate can be used as an available bandwidth estimate.

[0056] Another illustrative network bandwidth estimation process includes a bulge detection algorithm (BDA). BDA can be integrated with PathCos++ to select a pair of packets with similar congestion on both sides of a congestion peak. The selected packet pair can be used to estimate the available bandwidth. The quality of the available bandwidth estimate may only be as good as the BDA's selection of those packets.

[0057] In BDA, the probe sequence has decreasing rates to first create a congested state for the network path (e.g., a rate above the bottleneck) and then decongest the network path (e.g., a rate below the bottleneck). This means that on the probe sequence, the OWD of a packet first rises (congested state) and then falls (decongested state). Large bumps in OWD can indicate times of maximum congestion, and packets with similar OWD should experience similar congestion (e.g., similar amounts of queues).

[0058] Another illustrative network bandwidth estimation process includes transmission control protocol (TCP) congestion control. For example, the main use of bandwidth estimation is traffic engineering and deciding how to place traffic on network paths. Since the vast majority of traffic uses the TCP protocol, traffic engineering basically uses available bandwidth estimation as a prediction of TCP performance.

[0059] TCP performance on a network path is primarily governed by TCP congestion control. TCP congestion control attempts to determine the optimal rate for sending traffic by inferring when a path is congested and when it is not. If the path is not congested, TCP congestion control increases the sending rate from the sender computing device 110. If the path is congested, TCP congestion control decreases the sending rate from the sender computing device 110.

[0060] Another illustrative network bandwidth estimation process includes packet loss due to errors and congestion. For example, many networks are lossy, meaning that packets may be lost or intentionally dropped. Some networks are lossless, meaning that they guarantee that no packets will be lost. Lossless networks may only be suitable for relatively small networks because it can produce undesirable effects at scale, such as head-of-line blocking. Therefore, most larger networks and network paths are considered lossy.

[0061] One cause of packet loss is bit errors in network equipment and transmission errors on noisy links. However, these errors are uncommon. Most link technologies deploy perpetually noisy channels, such as WiFi and 3G, and use link-level acknowledgments and retransmissions to mask link losses, so actual packet loss due to noise and errors is rare unless conditions are very bad. The reason is that TCP assumes that packet loss is due to congestion, so the way to get the best performance is to eliminate link-level packet loss.

[0062] Therefore, on a lossy network path, packet loss is almost always caused by congestion, and sustained congestion can produce packet loss. If the input rate of traffic is consistently higher than the rate the link can transmit, this imbalance can be resolved by dropping the excess traffic and causing packet loss. Therefore, packet loss can be a reliable indicator of network congestion.

[0063] Another illustrative network bandwidth estimation process includes basic queue-tail drop loss. The link between the sender computing device 110 and the receiver computing device 120 via the network 130 can be a simple queue of fixed capacity before the link. The sender computing device 110 can send packets from the queue as fast as possible on the link and drop packets when the queue is full.

[0064] Traffic tends to be bursty, so the queue can accommodate temporary excesses in input rate and smooth out processing on the link. As long as the queue is not full, any received packet is delivered if the link is idle, or added to the queue for delivery when the link allows. When the queue is full, any received packet may be dropped. Losses may occur at the end of the queue, which is why it is called tail drop loss.

[0065] With perfect queuing and continuous traffic, losses can be fine-grained and spread smoothly over the chirp sequence. In practice, losses can be clustered and not uniform across the chirp sequence. A first reason can be bursty cross-traffic that changes the load on the queue during the burst. During a burst, the queue has the opportunity to accept more probe packets and fewer probe packets. A second reason can be granular queue scheduling, where multiple packets are removed from the queue together after multiple slots are available in the scheduling queue, but after these slots are filled, slots are not opened until the next scheduling. Therefore, packet losses with tail drops are usually clustered. In the traffic received at the receiving computing device 120, there are sequences with little or no loss, which alternate with sequences with high or complete loss. This makes the congestion signal quite coarse.

[0066] There are two main behaviors of a tail drop queue. If the cross traffic does not saturate the queue, the queue is mostly empty. In this case, congestion caused by the chirp sequence can cause the queue to fill up and increase delay until the queue is full, when delay stops increasing and packet loss begins. If the cross traffic saturates the queue, the queue is almost full. In this case, congestion caused by the chirp sequence can result in no delay increase (e.g., the queue cannot get any fuller) and immediate packet loss. Delay can fluctuate based on the load of the cross traffic filling the queue.

[0067] Another illustrative network bandwidth estimation process includes loss from active queue management (AQM). AQM can compensate for the shortcomings of tail drop, including high latency and its coarse congestion signal. AQM can add additional processing in the queue to make the loss more proportional to the congestion level.

[0068] AQM can implement Random Early Detection (RED), which defines a drop probability based on queue occupancy. When the queue is almost empty, the probability is close to 0, and it is unlikely that a packet will be dropped. When the queue is almost full, this probability is higher, and it is more likely that the packet will be dropped. When a packet is received, RED can use the current drop probability and a random number to determine whether the packet should be dropped or put into the queue. Packet loss becomes more probabilistic based on queue occupancy.

[0069] AQM can implement other probabilistic algorithms besides RED (e.g., Blue, ARED, PIE, CoDel, etc.) to maintain small queue occupancy by preventively dropping packets. For example, the encoding process can attempt to maintain the queue delay of all packets below a threshold. In another example, the PIE process features burst protection, where no packet drops occur within the first 150 ms after the queue starts to fill up.

[0070] In some examples, the short packet sequences used by the estimation technique may be too short to trigger a strong AQM response, and the loss caused by AQM on each chirp sequence may be very low. A chirp sequence may be more likely to overflow the queue and cause tail drop loss than to see significant AQM loss. Therefore, for the purpose of bandwidth estimation, an AQM queue can be considered the same as a tail drop queue.

[0071] Another illustrative network bandwidth estimation process includes a rate limiter, a policer, and a token bucket. A rate limiter can be used to help make network traffic conform to a specific rate. This may be for policy reasons, for example, as a result of a contractual agreement.

[0072] Rate limiters can be implemented with or without queues. When using queues, the rate limiter is similar to a fixed capacity link and can be managed by tail dropping or using AQM. When queues are not used, the implementation of the rate limiter is simplified and resource usage can be reduced. They are often called policers or meters to distinguish them from other rate limiters. These rate limiters can be implemented using token buckets to accommodate traffic bursts.

[0073] When a token bucket is implemented, the bucket can hold virtual tokens associated with its maximum capacity and refill tokens at a desired rate. When a packet arrives, if the bucket is not empty, the packet is passed and the token is removed. If the bucket is empty, the packet is discarded. If the link is underutilized for a period of time, the bucket may become full and tokens that exceed capacity are discarded. When the link is underutilized, the token bucket capacity (or burst size) can allow an unrestricted rate.

[0074] Token bucket based rate limiters have no queues, so packets may not experience additional delays due to congestion. Congestion signals can correspond to packet drops. If the bucket size is appropriate for the link capacity and traffic burst (i.e., not too small and not too large), packet loss can be fairly fine-grained. In practice, configuring the burst size is tricky, so most token buckets do not produce a smooth packet loss pattern.

[0075] Another illustrative network bandwidth estimation process includes a PRM model and packet loss. The PRM model assumes that there is no detection packet loss on the network path and that any packets that have been sent have been received and can be used by the method. Except for the Next-v2 process, existing PRM methods do not attempt to tolerate packet loss. The Next-v2 process assumes that packet loss is due to random link errors. When a packet is lost, the Next-v2 process attempts to reconstruct it by interpolating from adjacent packets. The Next-v2 process can tolerate limited packet loss, and it may not help with losses due to congestion. Other methods may assume that if there is packet loss in the chirp sequence, no estimation can be made. In these cases, the entire chirp sequence can be discarded and no estimation is made. This can significantly reduce the probability of obtaining a bandwidth estimate.

[0076] When the bottleneck on a network path is based on a token bucket, the PRM method may not accurately estimate the available bandwidth. For example, the PRM method measures the increase in OWD due to congestion. When using token buckets, there are no queues and the bottleneck may not react to congestion by increasing latency. In this sense, the congestion state created by the PRM method does not increase OWD. In the presence of token buckets, the PRM method will often fail or give incorrect estimates. Therefore, the PRM method may not currently estimate the available bandwidth when the bottleneck is a token bucket.

[0077] Examples of the present disclosure improve upon these and other methods using various other methods for estimating available bandwidth taking into account network congestion. The techniques described herein can find the point at which packet loss stops and use the probe rate as a bandwidth estimate. Additional actions can be performed, such as automatically rerouting packets and / or load balancing network traffic after determining the probe rate, as described throughout the disclosure.

[0078] Figure 2 A computing environment for implementing packet loss regression to estimate bandwidth according to some examples of the present disclosure is shown. In this illustration, a sender computing device 210 sends one or more data packets to a receiver computing device 220 via a network 230. The sender computing device 210 and the receiver computing device 220 (e.g., a client, a server, etc.) can be endpoints in a communication network connecting endpoints. Although a single sender computing device and a single receiver computing device are provided for simplicity of illustration, various sender computing devices can send data packets to various receiver computing devices.

[0079] The sender computing device 210 may generate a probe sequence and send the probe packets to the receiver computing device 220 via the network 230. The receiver computing device 220 may receive the probe packets and estimate the available bandwidth. The process may perform bandwidth estimation in the presence of a bottleneck that reacts to the congestion state of the network path by dropping packets without adding delay by implementing a token bucket. The process may find the point where packet loss stops and use the probe rate at that point as the bandwidth estimate.

[0080] This process of estimating bandwidth using packet loss regression can be more beneficial than other processes discussed herein. For example, the packet loss regression process can estimate available bandwidth when a bottleneck only drops packets without adding latency. The process can be related to the bulge detection algorithm (BDA) of some other estimation methods and requires minimal changes to those methods. Implementing the process in a network may be straightforward for an engineer or technician to implement and can be performed with low overhead (e.g., minimize CPU usage, execute quickly so as not to slow down network path estimation, etc.).

[0081] In some examples, the process can implement a reverse loss detection algorithm for a token bucket. Bottlenecks based on token bucket loss can be problematic for bandwidth estimation because they do not increase latency and BDA cannot be used to calculate bandwidth estimates. Instead, a token bucket is used. A token bucket cannot queue or store packets, so when the token bucket is in a traffic congestion state, it drops excess packets. The loss probability depends directly on the difference between the input traffic rate and the output rate of the token bucket.

[0082] R_drop=R_input-R_output

[0083] P_drop=R_drop / R_input=1-(R_output / R_input)

[0084] The input traffic is a mixture of probe traffic and cross traffic for estimation. For simplicity, we assume constant cross traffic. The available bandwidth (ABW) is the difference between capacity and cross traffic, so:

[0085] R_input=SR_probes+SR_cross

[0086] ABW=R_output-SR_cross

[0087] In most cases, the losses can be evenly distributed between the probes and the cross traffic, so the reception rate of probes (RR_probes) depends directly on the probability of losses.

[0088] P_drop=1-(R_output / (SR_probes+SR_cross))

[0089] RR_probes=(1-P_drop)*SR_probes

[0090] RR_probes=(R_output / (SR_probes+SR_cross))*SR_probes

[0091] If the probing rate is greater than the ABW, then when there is zero cross traffic, the probing receive rate is equal to the ABW. In the presence of cross traffic, the receive rate is higher than the ABW. This is similar to what happens in a queue, and this rate is called the asymptotic data rate (ADR).

[0092] When the token bucket is congested (for example, the detection rate is greater than ABW), the above formula may be true. If it is not congested (non-congested state), no packets will be dropped, and the output rate of each service is equal to its input rate.

[0093] The PRM technique using EDA determines ABW by finding the onset of congestion, i.e., the point at which a network path transitions from its previous non-congested (e.g., empty, etc.), non-congested, or decongested state to its congested state. For example, the sender computing device 210 sends probe packets at an increasing rate, and the EDA determines the increase in latency as a sign of congestion. Figure 3 Provides illustrations of various types of rate reduction methods.

[0094] By determining when packet loss starts, PRM techniques using EDA can be applied to the token bucket. If the detection rate is lower than ABW, there should not be any loss.

[0095] As soon as the probe rate exceeds the ABW, there should be packet loss, so a loss detection algorithm (LDA) can be implemented to find the first significant loss in the chirp sequence and use the corresponding probe rate as a bandwidth estimate. However, such an LDA does not work well in practice. Most token buckets have a memory of when congestion started (earlier packets affect later packets). The token bucket has a burst size, and when the token bucket has been free of congestion for a while, multiple tokens accumulate in the token bucket. These accumulated tokens enable traffic to be sent at a much higher rate than the rate promised by the token bucket, until they are exhausted. When the probe rate is below the ABW, tokens are replenished faster than they are consumed, so the bucket remains full. When the probe rate exceeds the ABW, tokens are consumed faster than they are replenished, so the number of tokens decreases, but no packets have been dropped yet because tokens are available. After several packets of the chirp sequence are sent over the ABW, the bucket is empty and the token bucket starts dropping packets, but by this time the probe rate of the chirp sequence has increased to exceed the ABW, resulting in an overestimation. This effect can be mitigated to some extent by burst detection techniques, but it adds significant inaccuracies to the measurements.

[0096] The process can reverse the LDA process based on the decreasing chirp sequence to find the transition to the decongested state. The decreasing chirp sequence can reduce the probing rate so that at some point the probing rate at the token bucket is below the ABW and the loss stops. The reverse LDA (RLDA) can look at the last significant packet loss that occurred and use the corresponding probing rate as the bandwidth estimate. This can enable the process to perform bandwidth estimation for token buckets with decreasing chirp sequences and eliminate the effects of bursts.

[0097] During RLDA, the bottleneck may not be a queue and there may be no memory during the transition out of the congested state. A token bucket may be incorporated where there is no memory during the decongested state and RLDA may be used to obtain a bandwidth estimate from the decongested point. With a token bucket at the decongested point, the loss may stop and the receive rate may become equal to the send rate (e.g., the packet probing rate corresponding to the instantaneous rate at which the sender is sending packets).

[0098] The RLDA process can find the last point of significant packet loss in the received chirp sequence. Finding the last point is difficult for several reasons. First, there may be packet loss due to transmission errors. This kind of packet loss is quite rare, which is why RLDA looks for significant packet loss. This means that there needs to be a way to distinguish which are significant losses and which are non-significant losses, which may increase the estimation error. Second, in most cases, cross traffic is bursty. This may cause the instantaneous ABW to change and make the loss more clustered. Due to the bursty cross traffic, the packet loss can stop before or after the average ABW. By trying to find a specific point, the process may not perform any averaging over a segment of the chirp sequence. Third, the rate of the chirp sequence gradually decreases towards the ABW to test various rates, so the probability of loss may also gradually decrease to zero. Just before reaching the ABW, the probability of loss is very low. This means that the chirp sequence does not contain much loss information close to the ABW, and RLDA can be determined using a very small amount of packet loss.

[0099] The combination of these three constraints makes it simple to determine the point where significant packet loss stops difficult and rather noisy. This is similar to the drawback of EDA-based techniques, namely that the determination of the inflection point in the chirp sequence is not robust in the presence of noise.

[0100] In some examples, the process can use clusters of packet losses to find the last point of significant packet loss in the chirp sequence. If packet losses are separated by several received packets, they can be considered part of the same loss cluster. The tolerable number of received packets in a cluster is based on the number of lost packets, which can take into account the burstiness of cross-traffic. Isolated packet losses can be ignored and the last loss of the last cluster is used as the last point of significant packet loss. Since the process can use token buckets (and token buckets have no memory), each packet loss can be independent of other packets, and the trends can be consistent in the calculations and may require less computing resources to analyze.

[0101] In some examples, the process can implement an extrapolation of the trend of packet loss information to find where it reaches zero probability. The first problem is to identify which part of the chirp sequence is used to estimate the trend (e.g., the token bucket experiences different regimes and has different trends, so a part of the chirp sequence is selected to correspond to a congested state). Once the probe rate is below the ABW, the token bucket is no longer in a congested state and has a different trend (constant zero loss), so the corresponding part of the chirp sequence can be eliminated. This can include eliminating all packets except the last packet with significant loss. The process can also eliminate parts of the chirp sequence before the end of the detected burst (e.g., when the token bucket is in a congested state). If burst detection is used, it detects the point where the burst phase of the token bucket ends and the token bucket begins to be in a congested state. An alternative is to look at the starting point of significant loss, which usually identifies the beginning of a congested state, and eliminate packets before that. A third option is to identify the point in the chirp sequence where the probe rate begins to decrease and eliminate packets before that. This third option provides a simpler trend for the data and is therefore preferred for simpler extrapolation.

[0102] Extrapolation can be performed on the set of packets between the end of the burst or the start of the rate reduction and the last significantly lost packet. This extrapolation can help find the detection rate at which the packet loss becomes zero.

[0103] In some examples, the extrapolation can implement linear regression. For example, a line can be fit through the data and the detection rate at which the loss probability is zero can be determined. The detection rate can be used as a bandwidth estimate. This trend can be nonlinear and introduce small errors.

[0104] In some examples, the extrapolation can implement nonlinear regression (e.g., by curve fitting a packet drop formula to packet loss data, etc.). Similarly, the fitted curve can be used to calculate the probe rate at which the loss probability is zero. The probe rate can be used as a bandwidth estimate. This approach reduces error but increases complexity.

[0105] P_drop=1-(R_output / (SR_probes+SR_cross))

[0106] In some examples, nonlinear regression may be implemented using curve fitting to directly calculate the unknown parameters R_output and SR_cross, and thereby calculate ABW.

[0107] Figure 4 An overview of a process for determining a bandwidth estimate according to some examples of the present application is shown.

[0108] At block 410, a sending computing device may send packets to a receiving computing device 220 via a network 230. The receiving computing device 220 may receive packets transmitted on a path through the network.

[0109] At block 420, the receiving computing device may find packets where significant loss ends. For example, the number of packet losses may be less than a threshold. The time when the loss ends and / or the number of packet losses may be determined.

[0110] In some examples, multiple consecutive losses may be used as significant loss values.

[0111] In some examples, these losses may be considered significant for various reasons. For example, they may be significant if they are part of a cluster of packet losses and / or are separated from other losses by less frequently received packets.

[0112] At block 430, the receiving computing device may determine a bandwidth estimate. For example, the bandwidth estimate may be based on a probe rate associated with the packet and / or an actual received rate of a subset of chirp sequences. The lower of these two values ​​may be used as the bandwidth estimate.

[0113] In some examples, the packet's probe rate may be used as an overall probe rate and / or bandwidth estimate.

[0114] In some examples, when linear regression is used on the packet loss data preceding the packet determined in block 420 to calculate the probe rate at which the loss reaches zero, the calculated probe rate is used as the bandwidth estimate.

[0115] In some examples, when the nonlinear curve fit of the packet loss data preceding the packet determined in block 420 is used to calculate the probe rate when the loss reaches zero, the calculated probe rate is used as the bandwidth estimate.

[0116] In some examples, the probe rate is used as a bandwidth estimate when a nonlinear curve fit of packet loss data preceding the packet determined in block 420 is used to determine the output rate of the token bucket and the amount of cross traffic and used to calculate the probe rate.

[0117] At block 440, one or more actions may be performed in response to determining the bandwidth estimate.

[0118] In some examples, the action may include generating a report notification including the available bandwidth estimate.The report notification may be sent to a traffic engineering module of the recipient computing device 220, a human user, or an external computing system configured to report or receive notifications.

[0119] In some examples, the action may include automatically rerouting packets to reduce the amount of packet loss on a particular network path. For example, a new probe packet may be sent to a controller node to help reroute packets in a black box network.

[0120] In some examples, the action may include initiating load balancing of network traffic after determining the probing rate.For example, the sender computing device 210 may send a request to the controller node to help initiate load balancing in the black box network.

[0121] In some examples, the action may include opening or closing some network path routes, links, or access to network devices. In some examples, the action may include creating or stopping some network tunnels.

[0122] In some examples, the action may include reconfiguring some links or network devices. This may include relocating devices or other resources.

[0123] In some examples, the action may include starting or stopping some network applications. This may include starting or stopping a backup process on a particular network device, or completely restarting the device.

[0124] Various benefits are discussed throughout this disclosure. For example, when using the Reverse Loss Detection Algorithm (RLDA), the process can perform bandwidth estimation in the presence of a token bucket or a network bottleneck that simply drops packets. Previously, bandwidth estimation was either not performed or would result in huge estimation errors.

[0125] Additionally, using this process, better estimates of available bandwidth can be made. This can improve the efficiency of traffic engineering for most networks, including in the context of SD-WAN.

[0126] Additionally, the process can be combined with other techniques to perform available bandwidth estimation. In some examples, RLDA and methods using BDA can use the same chirp sequence. This means that the same chirp sequence can be used when the bottleneck is a queue or a token bucket. This can be particularly useful when cross-traffic changes and the bottleneck of the network path alternates between queues and token buckets. In some examples, the system can be modified without having to restructure the code or change the packet format.

[0127] Figure 5 In conjunction with some examples of the present application, packet loss associated with reverse loss detection is shown.

[0128] For example, using RLDA, the process may find the last significant loss point which may be similar to EDA or PathChirp, but in reverse and looking for packets that were lost rather than received.

[0129] In another example, using the last significant loss point, the process can look at the last loss point cluster. The cluster can include several received packets separated by loss. In some examples, num_rcv<2*num_losses can be used to calculate the cluster.

[0130] In another example, using interpolation, the process can use linear regression as the loss rate. When using nonlinear regression, a curve can correspond to the loss probability on the data. The calculation can correspond to:

[0131] P=1-(R_out / (SR_probe+SR_cross))

[0132] It should be noted that the terms "optimize," "optimal," and the like as used herein may be used to mean obtaining or achieving performance as efficiently or perfectly as possible. However, as one of ordinary skill in the art who reads this document will recognize, perfection cannot always be achieved. Therefore, these terms may also include obtaining or achieving performance that is as good or efficient or practical as possible under given circumstances, or obtaining or achieving better performance than can be achieved using other settings or parameters.

[0133] Figure 6 An example computing component is shown that can be used to implement inverse loss detection for communication network bandwidth estimation using token buckets according to various examples. Figure 6 , computing component 610 may be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 6 In an example implementation of , computing component 610 includes a hardware processor 612 and a machine-readable storage medium for 614 .

[0134] The hardware processor 612 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for retrieving and executing instructions stored in the machine-readable storage medium 614. The hardware processor 612 may retrieve, decode, and execute instructions, such as instructions 616-626, to control the process or operation of burst preloading for available bandwidth estimation. As an alternative or in addition to retrieving and executing instructions, the hardware processor 612 may include one or more electronic circuits including electronic components for performing the functions of one or more instructions, such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other electronic circuits.

[0135] A machine-readable storage medium, such as a machine-readable storage medium, may be any electrical, magnetic, optical, or other physical storage device that contains or stores executable instructions. Thus, the machine-readable storage medium 614 may be, for example, a random access memory (RAM), a non-volatile RAM (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a storage device, an optical disk, etc. In some examples, the machine-readable storage medium 614 may be a non-transient storage medium, where the term "non-transient" does not include transient propagating signals. As described in detail below, the machine-readable storage medium 614 may be encoded with executable instructions, such as instructions 616-626.

[0136] The hardware processor 612 may execute instructions 616 to receive a set of probe data packets.For example, the hardware processor 612 may receive a set of probe data packets transmitted over a network path through a communication network from a sender computing device.

[0137] The hardware processor 612 may execute instructions 618 to select a subset of the probe data packets. For example, when packet loss is detected in the set of probe data packets, the hardware processor 612 may select a subset of the set of probe data packets.

[0138] The hardware processor 612 may execute the instructions 620 to calculate a significant loss value. For example, the hardware processor 612 may calculate a significant loss value for the packet based on the packet loss for each packet of the subset of the set of probe data packets. In some examples, packet loss is detected from the set of probe data packets.

[0139] The hardware processor 612 may execute instructions 622 to select the first packet or the second packet. For example, the hardware processor 612 may select the first packet in the subset of the set of detection data packets. The selected first packet may correspond to the last received packet in the subset, and the significant loss value for the last received packet is greater than zero.

[0140] In some examples, the hardware processor 612 can select a second packet in the subset of the set of probe data packets. The second packet can correspond to the next packet received after the selected first packet.

[0141] Hardware processor 612 may execute instructions 624 to determine a bandwidth estimate.For example, hardware processor 612 may determine a bandwidth estimate for the network path based on a probe rate of second packets corresponding to an instantaneous rate at which the sending computing device sends the second packets.

[0142] The hardware processor 612 may execute the instructions 626 to perform actions associated with bandwidth estimation.For example, the hardware processor 612 may perform actions associated with bandwidth estimation of a network path in a communication network.

[0143] In some examples, hardware processor 612 may calculate a significant loss value for a packet as the number of packets lost between the packet and a previously received packet.

[0144] In some examples, hardware processor 612 may calculate a significant loss value for a packet as the number of packets lost between the packet and a previously received packet minus a fixed offset.

[0145] In some examples, the hardware processor 612 may use a clustering algorithm to create one or more clusters of packet losses. The clusters of packet losses may be statistically related. For packets in a subset of the set of probe packets, when more than one packet in the packet loss between the packet and a previously received packet is part of a cluster whose size is greater than a threshold, the significant loss value of the packet is set to one, otherwise, the significant loss value of the packet is set to zero.

[0146] In some examples, hardware processor 612 may calculate a significant loss value for a packet as a value given by a linear regression of the number of packet losses between consecutively received packets in a subset of the set of probe data packets.

[0147] In some examples, hardware processor 612 may calculate a significant loss value for packets as a value given by a non-linear curve fit of the number of packet losses between consecutively received packets in a subset of the set of probe data packets.

[0148] In some examples, the subset of the set of sounding data packets is a portion of the chirp sequence where there is a loss.

[0149] In some examples, a subset of chirp sequences have decreasing rates.

[0150] Figure 7 A block diagram of an example computer system 700 is depicted in which various examples described herein may be implemented. For example, Figure 2 The sender computing device 210 or the receiver computing device 220 shown in FIG. 1 may be implemented as Figure 7 700. Computer system 700 includes a bus 702 or other communication mechanism for communicating information, and one or more hardware processors 704 coupled to bus 702 for processing information. Hardware processor(s) 704 may be, for example, one or more general purpose microprocessors.

[0151] The computer system 700 also includes a main memory 706, such as a random access memory (RAM), cache, and / or other dynamic storage device, coupled to the bus 702 for storing information and instructions to be executed by the processor 704. The main memory 706 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 704. When stored in a storage medium accessible to the processor 704, these instructions present the computer system 700 as a special-purpose machine that is customized to perform the operations specified in the instructions.

[0152] The computer system 700 also includes a read only memory (ROM) 708 or other static storage device coupled to the bus 702 for storing static information and instructions for the processor 704. A storage device 710, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to the bus 702 for storing information and instructions.

[0153] The computer system 700 may be coupled via the bus 702 to a display 712, such as a liquid crystal display (LCD) (or touch screen), for displaying information to a computer user. An input device 714, including alphanumeric and other keys, is coupled to the bus 702 for communicating information and command selections to the processor 704. Another type of user input device is a cursor control 716, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 704 and for controlling cursor movement on the display 712. In some examples, the same direction information and command selections as cursor control may be achieved by receiving touches on a touch screen without a cursor.

[0154] The computing system 700 may include a user interface module that implements a graphical user interface that may be stored in a mass storage device as executable software code executed by the computing device(s). This module and other modules may include components, such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0155] In general, the words "component", "engine", "system", "database", "data storage", etc. used herein may refer to logic contained in hardware or firmware, or to a collection of software instructions that may have entry and exit points written in programming languages ​​such as Java, C or C++. Software components can be compiled and linked into executable programs installed in dynamic link libraries, or can be written in interpreted programming languages ​​such as BASIC, Perl or Python. It should be understood that software components can be callable from other components or themselves, and / or can be called in response to detected events or interrupts. Software components configured for execution on a computing device can be provided on a computer-readable medium such as a compact disc, a digital video disc, a flash drive, a disk, or any other tangible medium, or as a digital download (and can be initially stored in a compressed or installable format that needs to be installed, decompressed or decrypted before execution). Such software code may be stored in part or in whole on a storage device of an executing computing device for execution by the computing device. Software instructions may be embedded in firmware such as an EPROM. It should also be understood that hardware components may include connected logic units such as gates and triggers, and / or may include programmable units such as programmable gate arrays or processors.

[0156] Computer system 700 can implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in conjunction with the computer system, makes computer system 700 a special purpose machine or programs computer system 700 to be a special purpose machine. According to one example, the techniques herein are performed by computer system 700 in response to processor(s) 704 executing one or more sequences of one or more instructions contained in main memory 706. Such instructions may be read into main memory 706 from another storage medium such as storage device 710. Execution of the sequences of instructions contained in main memory 706 causes processor(s) 704 to perform the process steps described herein. In alternative examples, hardwired circuitry may be used in place of or in combination with software instructions.

[0157] The term "non-transient medium" and similar terms used herein refer to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-volatile media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 710. Volatile media include dynamic memory, such as main memory 706. Common forms of non-transient media include, for example, floppy disks, disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cassettes, and network versions thereof.

[0158] Non-transient media are distinct from transmission media, but can be used in conjunction with transmission media. Transmission media participate in the transmission of information between non-transient media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the cables that make up bus 702. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0159] The computer system 700 also includes a network interface 718 coupled to the bus 702. The network interface 718 provides a bidirectional data communication coupling with one or more network links connected to one or more local networks. For example, the network interface 718 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the network interface 718 can be a local area network (LAN) card to provide a data communication connection to a compatible local area network (or a WAN component that communicates with a WAN). A wireless link can also be implemented. In any such implementation, the network interface 718 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0160] A network link typically provides data communication to other data devices through one or more networks. For example, a network link may provide a connection to a host computer or to data devices operated by an Internet service provider (ISP) through a local network. The Internet service provider, in turn, provides data communication services through the global packet data communication network now commonly referred to as the "Internet". Both the local network and the Internet use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link and through the network interface 718 carry digital data to and from the computer system 700, which are example forms of transmission media.

[0161] Computer system 700 can send messages and receive data, including program code, through the network(s), network links, and network interface 718. In the example of the Internet, a server may send the requested code for an application program through the Internet, an ISP, a local network, and network interface 718.

[0162] The received code may be executed by processor 704 as it is received, and / or stored in storage device 710 or other non-volatile storage for later execution.

[0163] Each process, method and algorithm described in the foregoing section may be included in a code component executed by one or more computer systems or computer processors including computer hardware, and fully or partially automated by the code component. One or more computer systems or computer processors may also operate to support the performance of related operations, which are in a "cloud computing" environment or as "software as a service" (SaaS). These processes and algorithms may be implemented in part or in whole in a dedicated circuit. The various features and processes described above may be used independently of each other, or may be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of the present disclosure, and certain methods or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states associated therewith may be executed in other appropriate sequences, or may be executed in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed examples. The execution of certain operations or processes may be distributed between computer systems or computer processors, not only residing in a single machine, but also deployed on multiple machines.

[0164] As used herein, any form of hardware, software or combination thereof may be utilized to implement a circuit. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FGAs, logic components, software routines or other mechanisms may be implemented to form a circuit. In implementations, the various circuits described herein may be implemented as discrete circuits, or the functions and features described may be partially or fully shared between one or more circuits. Even if the various features or elements of a function may be described or claimed separately as separate circuits, these features and functions may be shared between one or more common circuits, and such descriptions do not require or imply the need for separate circuits to implement these features or functions. In the case of using software to implement the circuit in whole or in part, such software may be implemented to operate with a computing or processing system such as a computer system 700 that is capable of performing the functions described therefor.

[0165] As used herein, the term "or" may be interpreted in an inclusive or exclusive sense. Furthermore, descriptions of resources, operations, or structures in the singular should not be construed to exclude the plural. Conditional language, such as, among others, "may," "could," "can," or "might," unless otherwise stated or understood in the context of use, is generally intended to convey that certain examples include certain features, elements, and / or steps, while other examples do not.

[0166] Unless expressly stated otherwise, the terms and phrases used in this document and variations thereof should be construed as open ended and not limiting. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar meaning should not be construed to limit the items described to a given time period or items available at a given time, but should be construed to include conventional, traditional, normal, or standard technology that may be available or known at any time now or in the future. In certain cases, the presence of broadening words and phrases such as "one or more," "at least," "but not limited to," or other similar phrases should not be construed as intending or requiring a narrowing of the scope where such broadening phrases may not be present.

Claims

1. A receiving computing device, comprising: Memory; as well as One or more processors configured to execute machine-readable instructions stored in the memory for performing a method comprising: receiving, from a sending computing device, a set of probe data packets transmitted over a network path through a communication network; selecting a subset of the set of probe data packets when a packet loss is detected in the set of probe data packets, wherein the subset of the set of probe data packets is a portion of a chirp sequence in which there is a loss; calculating, for each packet in the subset of the set of probe data packets, a significant loss value for the packet based on the packet loss detected from the set of probe data packets; selecting a first packet in the subset of the set of probe data packets, wherein the selected first packet corresponds to the last received packet in the subset, wherein the significant loss value for the last received packet is greater than zero; selecting a second packet in the subset of the set of probe data packets, wherein the second packet corresponds to a next packet received after the selected first packet; determining a bandwidth estimate for the network path based on a probe rate of the second packets corresponding to an instantaneous rate at which the second packets are sent by the sending computing device; and Actions associated with the bandwidth estimation for the network path in the communication network are performed.

2. The recipient computing device of claim 1, wherein the machine-readable instructions further execute the method, the method comprising: The significant loss value for a packet is calculated as the number of packets lost between the packet and a previously received packet.

3. The recipient computing device of claim 1 , wherein the machine-readable instructions further perform the method, the method comprising: The significant loss value for a packet is calculated as the number of packets lost between the packet and a previously received packet minus a fixed offset.

4. The recipient computing device of claim 1 , wherein the machine-readable instructions further perform the method, the method comprising: using a clustering algorithm to create one or more clusters of packet losses, wherein the clusters of packet losses are statistically correlated; For packets in the subset of the set of probe data packets: setting the significant loss value for the packet to one when more than one of the packet losses between the packet and a previously received packet is part of a cluster whose size is greater than a threshold, Otherwise, the significant loss value for the packet is set to zero.

5. The recipient computing device of claim 1 , wherein the machine-readable instructions further perform the method, the method comprising: The significant loss value for a packet is calculated as a value given by a linear regression of the number of packet losses between consecutively received packets in the subset of the set of probe data packets.

6. The recipient computing device of claim 1 , wherein the machine-readable instructions further perform the method, the method comprising: The significant loss value for a packet is calculated as a value given by a non-linear curve fit of the number of packet losses between consecutively received packets in the subset of the set of probe data packets.

7. The receiving computing device of claim 1, wherein the subset of chirp sequences has a decreasing rate.

8. A computer-implemented method for determining a bandwidth estimate for a network path in a communication network, the method comprising: receiving, from a sending computing device, a set of probe data packets transmitted over said communication network on said network path; selecting a subset of the set of probe data packets when a packet loss is detected in the set of probe data packets, wherein the subset of the set of probe data packets is a portion of a chirp sequence in which there is a loss; calculating, for each packet in the subset of the set of probe data packets, a significant loss value for the packet based on the packet loss detected from the set of probe data packets; selecting a first packet in the subset of the set of probe data packets, wherein the selected first packet corresponds to the last received packet in the subset, wherein the significant loss value for the last received packet is greater than zero; selecting a second packet in the subset of the set of probe data packets, wherein the second packet corresponds to a next packet received after the selected first packet; determining a bandwidth estimate for the network path based on a probe rate of the second packets corresponding to an instantaneous rate at which the second packets are sent by the sending computing device; as well as Actions associated with the bandwidth estimation for the network path in the communication network are performed.

9. The method according to claim 8, further comprising: The significant loss value for a packet is calculated as the number of packets lost between the packet and a previously received packet.

10. The method according to claim 8, further comprising: The significant loss value for a packet is calculated as the number of packets lost between the packet and a previously received packet minus a fixed offset.

11. The method according to claim 8, further comprising: using a clustering algorithm to create one or more clusters of packet losses, wherein the clusters of packet losses are statistically correlated; For packets in the subset of the set of probe data packets: setting the significant loss value for the packet to one when more than one of the packet losses between the packet and a previously received packet is part of a cluster whose size is greater than a threshold, Otherwise, the significant loss value of the packet is set to zero.

12. The method according to claim 8, further comprising: The significant loss value for a packet is calculated as a value given by a linear regression of the number of packet losses between consecutively received packets in the subset of the set of probe data packets.

13. The method according to claim 8, further comprising: The significant loss value for a packet is calculated as a value given by a non-linear curve fit of the number of packet losses between consecutively received packets in the subset of the set of probe data packets.

14. The method of claim 8, wherein the subset of chirp sequences has a decreasing rate.

15. A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors, the plurality of instructions, when executed by the one or more processors, causing the one or more processors to: receiving, from a sending computing device, a set of probe data packets transmitted over a network path through a communication network; selecting a subset of the set of probe data packets when a packet loss is detected in the set of probe data packets, wherein the subset of the set of probe data packets is a portion of a chirp sequence in which there is a loss; calculating, for each packet in the subset of the set of probe data packets, a significant loss value for the packet based on the packet loss detected from the set of probe data packets; selecting a first packet in the subset of the set of probe data packets, wherein the selected first packet corresponds to the last received packet in the subset, wherein the significant loss value for the last received packet is greater than zero; selecting a second packet in the subset of the set of probe data packets, wherein the second packet corresponds to a next packet received after selecting the first packet; determining a bandwidth estimate for the network path based on a probe rate of the second packets corresponding to an instantaneous rate at which the second packets are sent by the sending computing device; as well as Actions associated with the bandwidth estimation for the network path in the communication network are performed.

16. The non-transitory computer-readable storage medium of claim 15, further comprising: The significant loss value for a packet is calculated as the number of packets lost between the packet and a previously received packet.

17. The non-transitory computer-readable storage medium of claim 15, further comprising: The significant loss value for a packet is calculated as the number of packets lost between the packet and a previously received packet minus a fixed offset.

18. The non-transitory computer-readable storage medium of claim 15, further comprising: using a clustering algorithm to create one or more clusters of packet losses, wherein the clusters of packet losses are statistically correlated; For packets in the subset of the set of probe data packets: setting the significant loss value for the packet to one when more than one of the packet losses between the packet and a previously received packet is part of a cluster whose size is greater than a threshold, Otherwise, the significant loss value for the packet is set to zero.

Images