ESTIMATION OF AVAILABLE NETWORK BANDWIDTH USING A ONE-WAY DELAY NOISE FILTER WITH INCREASE DETECTION

Abstract

Procedure, comprehensive: Receiving (406) one-way delay measurements for probe packets (P.1; P.2; P.3) in a probe sequence sent over a network path (104) in a system (100) comprising a hardware processor (402); Grouping (408), by the system (100), the probe packets (P.1; P.2; P.3) into several pairs (506, 508; 510, 512) of probe packets (P.1; P.2; P.3) based on the one-way delay measurements that form a rise, wherein two probe packets (P.1; P.2; P.3) are grouped into a pair (408) if the two probe packets (P.1; P.2; P.3) have similar one-way delay measurements on each side of the rise; Calculate (410), by the system (100), a respective noise threshold value for each pair of the multiple pairs based on one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after a later transmitted probe packet (P.1; P.2; P.3) of the respective pair, wherein for the respective pair a statistic of the one-way delay measurements of all probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the respective pair is calculated; Select (412), by the system (100), one selected pair of the multiple pairs according to a comparison of the one-way delay measurements for the sounding packets (P.1; P.2; P.3) of the multiple pairs with the noise thresholds; and Estimating (414), by the system (100), an available bandwidth on the network path (104) based on the transmission times of the probe packets (P.1; P.2; P.3) in the selected pair.

Description

Description of the state of the art

[0001] Bandwidth measurement is a critical component of any network traffic planning solution, including SDWAN solutions. To properly route and balance traffic across a network, it's helpful to know how much bandwidth is available.

[0002] In a closed system, it is possible to collect direct measurements on each of the network devices along the traffic paths. However, in many cases, direct measurements are not possible, for example, when the network devices are located in different administrative domains, or when they are hidden by tunneling or route encapsulation. This is the case with a software-defined wide area network (SD-WAN), where the SD-WAN gateway attempts to route traffic over the internet using the best path.

[0003] US 2007 / 0217448 A1 describes estimating available bandwidth using multiple congestion streams. The testing transmitter receives latency delay information from the receiver based on the receiver's OWD measurements of at least two pairs of consecutive packets.

[0004] US 2015 / 0 333 993 A1 describes a method for estimating available bandwidth using probe packets and evaluating changes in propagation times at different transmission rates. Short description of the drawings

[0005] The present disclosure will be described in detail according to one or more different embodiments with reference to the following figures. The figures are for illustrative purposes only and depict merely typical or exemplary embodiments. Fig. 1 is a block diagram of a system for estimating available bandwidth using a probe sequence according to one embodiment. Fig. 2 shows timing characteristics of a probe sequence according to one embodiment. Fig. Figure 3 is a graph of relative one-way delay (OWD) for a probe sequence of 18 probes sent over a network path at decreasing rates, ie, with increasing drop distance (IDT). Fig. 4 is a block diagram of an exemplary computing component or device for estimating available bandwidth on a network path according to one embodiment. Fig. Figure 5 shows the noise thresholds for the probing sequence of Fig. 3. Fig. 6 shows a block diagram of an exemplary computer system in which various embodiments described herein may be implemented.

[0006] The figures are not exhaustive and do not limit the present disclosure to the precise form disclosed. Detailed description

[0007] Wide area networks (WANs) can exist across multiple domains, whether geographic, administrative, or organizational. A WAN administrator may have non-administrative access to all domains spanned by the WAN. An example of such a topology is a software-defined WAN (SD-WAN), in which a branch network (e.g., a corporate campus) is connected to a core network (e.g., a corporate headquarters) via one or more secure tunnels through the internet. While a network administrator for such an SD-WAN may have administrative control over network infrastructure devices (e.g., controllers, switches, routers, gateways, etc.) in the branch network and the core network, the administrator has no administrative control over the intermediate network infrastructure devices across the internet.

[0008] This lack of complete administrative control over the WAN's intermediate internet connections makes it impossible to directly measure the performance of network paths. When there is more than one path for transmitting data between two devices in a network, performance data for each of these paths can be used to select a preferred path for forwarding such data. This can reduce latency, increase throughput across all network paths, and reduce the likelihood of packets being dropped by network infrastructure devices. Referring again to the SD-WAN example, it may be impossible to directly measure the performance of one or more secure tunnel connection devices (e.g., user devices, servers, etc.) in the branch network with devices in the core network.However, determining which secure tunnel to use can improve network efficiency by reducing the likelihood of one of the tunnels becoming saturated (i.e., sending more data than the tunnel can handle without degrading performance).

[0009] If direct measurements are not possible, bandwidth estimation can be performed by two devices that can be controlled and used for measurements. This is usually done by probing the network path with specially crafted probe packets sent from a device at one end of the path (sender) to a device at the other end of the path (receiver). The receiving terminal measures the reception time of the packets and the changes in the one-way delay / timing pattern of the packet to estimate the path properties, such as path capacity, available bandwidth, and / or bulk transmission capacity. Path capacity is the maximum amount of data traffic volume per unit time that can be sent when the network path is idle, i.e., with no competing traffic. Available bandwidth (ABW) is the remaining / remaining path capacity, i.e.,The capacity not currently being used by other traffic. Bulk transfer capacity (BTC) is the bandwidth a TCP connection would receive if placed on this network path. Latency is the one-way delay (OWD) from the sender to the receiver, and round-trip time (RTT) is the bidirectional delay between the sender and the receiver.

[0010] The primary approach for end-to-end network path estimation is active probing. A device at one end of the internet path sends special probing packets to a device at the other end of the internet path. These packets are used only to estimate bandwidth and carry no actual data beyond the data payload required for the network path estimation itself.

[0011] During active probing, the sender sends a series of specially designed probe packet patterns. The packet pattern is defined by the estimation technique and designed to trigger specific behaviors from the network elements on the network path. For example, in many cases, the packet pattern is a probe sequence. The packets and the interval between packet transmissions are selected to probe different bandwidths across the packet pattern. The receiver measures the reception time of the packets, calculates a one-way delay of each packet (e.g., the time it takes for a packet to travel from the sender device to the receiver device), and examines changes in the packet pattern. The estimation technique uses a simplified network model to convert these measurements into estimates of various network path properties.

[0012] There are two main classes of bandwidth estimation techniques: those that use a probe gap model and those that use the probe rate model. For the probe gap model, it is assumed that when two packets are sent closely together, the gap between them increases proportionally to the load in the busiest queue along the path, due to the queuing delay in that queue. For the probe rate model, it is assumed that when packets are sent at a rate lower than the bottleneck bandwidth, the traffic pattern remains largely unchanged, whereas when packets are sent at a rate greater than the bandwidth available on the path bottleneck, that packet will incur additional queuing delay due to congestion.

[0013] One technique for estimating available bandwidth is called PathCos++. The PathCos++ technique is described in the paper "A Novel Hybrid Probing Technique for End-to-End Available Bandwidth Estimation" by Lin et al., 35th Annual IEEE Conference on Local Computer Networks, 2010. According to PathCos++, a sequence of probes is sent at decreasing speeds over the path to be measured. A receiver measures the probe reception rate between two points of equal congestion. Fig. Figure 3 is an example of a graph of relative OWDs for a probe sequence of 18 probes sent over a network path at decreasing rates, ie, with increasing IDTs. As can be seen from Fig. As can be seen in Figure 3, the graph shows a bump. Points of equal overload are points on opposite sides of the bump with similar OWD measurements.

[0014] PathCos++ and some other bandwidth estimation techniques use a bump detection algorithm to select these two points of equal congestion. However, the bump detection algorithm is very sensitive to noise in packet reception timing. In many cases, this noise prevents the algorithm from finding the correct pair of points, which can lead to large errors in bandwidth estimates.

[0015] Fig. 1 is a block diagram of an exemplary system 100 for estimating available bandwidth using a probe sequence. The system 100 in this example includes a transmitter 102 connected to a receiver 108 by a network path 104 in a network 106. The exemplary system 100 also includes a network controller 110. Each of the transmitter 102 and the receiver 108 may be implemented as a network device, such as a switch, a router, a gateway, a server, a client, or the like.

[0016] In operation, transmitter 102 generates a probe sequence comprising multiple packets at 112 and transmits the packets over the network path at 114. Receiver 108 receives the packets at 116 and estimates the available bandwidth based on the packets at 118. Receiver 108 provides the available bandwidth estimate to network controller 110. Network controller 110 may use the available bandwidth estimate to perform network traffic planning and the like at 120.

[0017] Fig. Figure 2 shows timing characteristics of a probe sequence according to one embodiment. Three packets are shown: P.1, P.2, and P.3. These three packets are both in the upper part of Fig. 1 when they leave the transmitter and in the lower part of the diagram when they arrive at the receiver. Fig. Figure 2 illustrates the departure distance (IDT), arrival distance (IAT), and one-way delay (OWD). The IDT of a packet is the time between the transmission of the packet and the transmission of the previous packet. The IDT can be calculated from the desired probing rate as IDT = packet size / probing rate. The transmitter is rarely absolutely accurate when transmitting packets, so the actual IDT of a packet may differ from the desired / theoretical IDT. The disclosed technology in various embodiments operates with both the desired IDT and the actual IDT, but generally provides better results with the actual IDT. The arrival distance (IAT) of a packet is the time between the reception of the packet and the reception of the previous packet.

[0018] The OWD of a packet is primarily determined by the propagation delay of the network path, the transmission time of the slowest link in the path, and the accumulated queuing delay in all network elements in the path. For each packet i, the OWD can be calculated as follows: OWD(i)=pd+st(size)+sum(qd(e,i)) with: pd -> total propagation delay; st(size) -> slowest transmission time for this packet size; and qd(e,i) -> queuing delay at element e for packet i.

[0019] These network path estimation techniques can assume a fairly simple queueing model where qd(e,i) is a function of the congestion at element e when packet i arrives.

[0020] To measure packet departure and arrival times, the transmitter and receiver use different clocks that may not be perfectly synchronized. Therefore, it can be difficult to accurately measure the OWD of packets. Fortunately, in most cases, it is not the absolute OWD of a packet that matters, but rather the difference between the OWDs of different packets. During the short time of a probe sequence, clock drift is negligible, and therefore the difference between the clocks can be within an acceptable range of accuracy. The disclosed technology can use this relative OWD for packet measurement. For each packet, the relative OWD of a packet can be the difference between the OWD of that packet and a fixed constant per probe sequence. For example, this can be the difference between the OWD of that packet and the OWD of the first packet in the probe sequence.This relative OWD estimates the additional queuing delay of a packet relative to another packet and can be a positive or negative value.

[0021] If there is no packet loss and no packet reordering, packets are sent and received in the same order. In this case, the IAT, IDT, and OWD of each packet are directly related. If OWDp is the one-way delay of the previous packet and OWDc is the one-way delay of the current packet, the IAT can be given by: IAT=OWDc−OWDp+IDT

[0022] Fig. Figure 3 is an example of a relative OWD graph for a probe sequence of 18 probes sent over a network path at decreasing rates, ie, with increasing IDTs. In some embodiments, each probe comprises only one probe packet. In other embodiments, as discussed below, each probe comprises multiple probe packets.

[0023] In Fig. 3 shows that the graph forms a "bump," and bump detection algorithms (BDA) can be used to locate the bump. The top of the bump represents the maximum relative OWD and indicates the bottleneck bandwidth. To calculate the bottleneck bandwidth, a packet pair is required. The goal of the BDA is to select a representative pair of packets with similar OWD on either side of a large bump in the OWD. Since the packet pair is used to estimate the available bandwidth, the quality of the available bandwidth estimate is only as good as the selection of these packets.

[0024] The probe sequence is sent at a decreasing rate. Therefore, the probe sequence first creates congestion on the network path (sending at a rate higher than the bottleneck) and then decongests that network path (sending at a rate lower than the bottleneck). This means that throughout the probe sequence, the OWD of packets initially increases due to congestion and then decreases due to decongestion. The peak of the increase in OWD represents the time of maximum congestion. Packets with similar OWD should have experienced similar congestion (similar amount in queues).

[0025] The BDA preferentially selects packets that are part of the same congestion period. If they are not part of the same congestion period, there may be some idle time between these two packets, and in this case, not much can be said other than the fact that the packet probing rate is lower than the available bandwidth. If two packets are part of the same congestion period and have similar congestion, the total probing rate between these two packets contributes fairly neutrally to the congestion, meaning that the total probing rate is close to the available bandwidth.

[0026] To avoid problems with the granularity of the OWD and achieve better averaging over time, the BDA can be implemented to locate a pair of packets as far apart as possible. Accordingly, it can search for packets with the maximum time between them in the probing sequence. However, the network path must remain congested between these two packets to meet the model requirements. This means that the two packets must be part of the OWD bump. In practice, the BDA attempts to find two packets around the base of the OWD bump.

[0027] An increase in OWD in the probe sequence theoretically indicates increased congestion, but in reality, OWD measurements are very noisy. This noise is due to the imperfection and non-ideal behavior of the various network elements and other constraints in packet scheduling.

[0028] Such noise in the OWD directly impacts the BDA algorithm. Noisy OWD values ​​can cause the BDA to select two packets with similar OWD values ​​that did not actually experience the same level of congestion, or packets that did not fall within the same congestion period. This leads to incorrect estimates of the available bandwidth.

[0029] PathCos++'s BDA can be challenged by typical OWDs measured on a shared network path. For example, on an idle Ethernet link, which represents the best-case scenario, the BDA exhibits a fairly high error rate. Furthermore, PathCos++ often underestimates the available bandwidth, and in some cases, the selected packet pair does not belong to the same congestion period.

[0030] A major source of measurement noise is packet stacking. Packet stacking occurs when packets in a queue are processed as a group rather than individually. This packet stacking can drastically impact the OWD of the packets. Earlier packets in the group have to wait until the group is processed and see their OWD artificially inflated, while the last packet in the group waits less time and therefore has a lower OWD.

[0031] The authors of PathChirp, a very popular bandwidth estimation technique, identified a common source of packet stacking: interrupt coalescence in the receiver network interface card (NIC). Their solution uses multiple packets per probe. A probe sequence consists of multiple probes, with each probe testing a specific rate on the network path. When stack mitigation is enabled, the sender sends multiple packets for each probe instead of a single packet per probe. All packets in a probe are sent at the same rate and therefore have the same IDT. The receiver filters the packets and selects only a single packet for each probe—the packet estimated to have suffered the least from packet stacking. The BDA is executed only on the selected packets, not on all received packets.

[0032] Fig. 4 is a block diagram of an exemplary computing component or device 400 for estimating the available bandwidth on a network path according to one embodiment. The computing component 400 may be, for example, a server computer, a controller, or other similar computing component capable of processing data. In the exemplary implementation of Fig. 4, the computing component 400 includes a hardware processor 402 and a machine-readable storage medium 404. In some embodiments, the computing component 400 may be an embodiment of the transmitter 102, the receiver 108, or the network controller 110 of Fig. 1, or a combination thereof.

[0033] The hardware processor 402 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices capable of retrieving and executing instructions stored in a machine-readable storage medium 404. The hardware processor 402 may fetch, decode, and execute instructions, such as instructions 406 through 414, to control processes or operations for estimating the available bandwidth on the network path. Alternatively, or in addition to retrieving and executing instructions, the hardware processor 402 may include one or more electronic circuits that include electronic components for performing the functionality of one or more instructions, such as a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other electronic circuitry.

[0034] A machine-readable storage medium, such as machine-readable storage medium 404, may be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. For example, machine-readable storage medium 404 may be random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), a storage device, an optical disk, and the like. In some embodiments, machine-readable storage medium 404 may be a non-transitory storage medium, where the term "non-transitory" does not include transitively propagating signals. As described in detail below, machine-readable storage medium 404 may be encoded with executable instructions, such as instructions 406 through 414.

[0035] Hardware processor 402 may execute instruction 406 to receive one-way delay measurements for each probe packet in a probe sequence sent over the telecommunications path. The OWD measurements may be calculated using any method. For example, the OWD of each packet in a probe sequence may be measured relative to the OWD of the first packet in the probe sequence. This approach may reduce the effect of synchronization errors between the transmitting and receiving clocks and may also reduce the bit lengths of the OWD measurements.

[0036] The disclosed technology may incorporate a novel technique for mitigating OWD noise in the boost detection algorithm for estimating available bandwidth. The improved BDA favors a packet pair whose OWD after boosting is greater than the OWD noise.

[0037] OWD noise is typically not uniform across the probe sequence. OWD noise tends to be larger when the network path is not congested and smaller when the network path is congested. When the network path is congested, scheduling is more predictable and there is less jitter. Furthermore, this noise decreases even more with a larger OWD relative to the OWD.

[0038] Previous techniques also tend to select packets too close to the base of the increase, that is, to select the widest pair with similar OWD, and run the risk that these packets may not be in the same congestion period, as in Fig. 5. In other words, there is an idle gap in the bottleneck between these two packets.

[0039] The disclosed technology favors pairs of packets higher than the baseline of the boost to avoid OWD noise. We use a statistical measure of the uncongested OWD after the boost as the OWD threshold, and we favor packet pairs with an OWD greater than this threshold.

[0040] Hardware processor 402 may execute instruction 408 to group the probe packets into multiple pairs based on the one-way delay measurements. The disclosed technology selects a number of candidate pairs of packets after packet stacking filtering and other validity filtering, i.e., packets suitable for performing bandwidth estimation. Each packet pair has a first packet and a second packet, selected mostly based on their OWD, with the first packet being transmitted before the second packet. For example, two of the probe packets may be grouped into a pair if the two probe packets have similar one-way delay measurements, and a probe packet was transmitted between the two probe packets with a maximum one-way delay measurement for the probe sequence.

[0041] Hardware processor 402 may execute instruction 410 to calculate a respective noise threshold for each pair based on the one-way delay measurements of all probe packets transmitted after the later probe packet of the pair. For each packet pair, the system calculates statistics using the OWD of all packets in the probe sequence after the final packet of the pair (the second packet, in the offload). These packets follow the congestion and are therefore representative of the OWD noise. In some embodiments, the system calculates both the average (mean) and the mean absolute deviation (MAD) of the OWD of these subsequent packets and combines both to define an OWD threshold. Our current OWD threshold is the average plus 1.5 times the deviation, which is approximately the 85th percentile of the OWD noise. Other OWD thresholds may be used.In some embodiments, calculating a respective noise threshold for a pair comprises calculating a mean average separation of the one-way delay measurements of all probe packets transmitted after the later-transmitted probe packet of the pair. In some embodiments, calculating a respective noise threshold for a pair comprises calculating a standard deviation of the one-way delay measurements of all probe packets transmitted after the later-transmitted probe packet of the pair. In some embodiments, calculating a respective noise threshold for a pair comprises calculating a cumulative distribution function of the one-way delay measurements of all probe packets transmitted after the later-transmitted probe packet of the pair. Fig. 5 shows the noise threshold values ​​502 for the probing sequence of Fig. 3. The OWD mean values ​​504 are also shown for reference.

[0042] At this point, the original BDA may filter out candidate packet pairs that do not satisfy the boost filter (the boost filter considers the highest OWD of the boost, the boost height, and width), and if there are remaining pairs, selects the widest one. The hardware processor 402 may execute instruction 412 to select one of the pairs according to the noise thresholds 502 and the one-way delay measurements for the probe packets of the pairs. In some embodiments, the pair with the lowest one-way delay measurement that exceeds the respective noise threshold 502 is selected. If there are packet pairs that satisfy the boost filter and for which the OWD of the final packet is higher than the OWD threshold, the widest packet pair is selected. Otherwise, if there are packet pairs that satisfy the boost filter but none have the OWD of the final packet higher than the OWD threshold, the packet pair with the highest OWD is selected.

[0043] In the example of Fig. 5, the pair 506, 508 is selected. For comparison, previous techniques would choose the pair 510, 512. Note that pair probe 512 is not part of the boost, so probes 510 and 512 are not part of the same congestion period and would therefore produce an inaccurate bandwidth estimate.

[0044] Hardware processor 402 may execute instruction 414 to estimate the available bandwidth on the telecommunications path based on the transmission times of the probe packets in the selected pair 506, 508. Many such bandwidth estimation techniques exist, such as the original PathCos++ method. One example is measuring the reception rate of probe packets between the two packets of the selected pair.

[0045] The bandwidth estimate for the telecommunication path may be used to perform network traffic planning for the network that includes the telecommunication path. The network traffic planning may include routing packets through the network, performing load balancing on the network, and the like. The bandwidth estimate for the telecommunication path may be used to select a transmission rate for Transmission Control Protocol (TCP) packets to be transmitted over the telecommunication path. The bandwidth estimate for the telecommunication path may be used to select a video coding rate for video to be transmitted over the telecommunication path. The bandwidth estimate for the telecommunication path may be reported to a user of the telecommunication path.

[0046] The main advantage of the disclosed technology is its performance. It improves packet pair selection and leads to much better available bandwidth estimates. Another advantage is that it is a simple modification of the bump detection algorithm (BDA), which is used by many of the best available bandwidth measurement techniques.

[0047] Fig. 6 shows a block diagram of an exemplary computer system 600 in which various embodiments described herein may be implemented. Computer system 600 includes a bus 602 or other communication mechanism for conveying information, and one or more hardware processors 604 coupled to bus 602 for processing information. Hardware processor(s) 604 may, for example, be one or more general-purpose microprocessors.

[0048] Computer system 600 also includes main memory 606, such as random access memory (RAM), a cache, and / or other dynamic storage devices coupled to bus 602 for storing information and instructions to be executed by processor 604. Memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 604. When stored in storage media accessible to processor 604, such instructions make computer system 600 a special-purpose machine adapted to perform the operations specified in the instructions.

[0049] Computer system 600 further includes a read-only memory (ROM) 608 or other static storage device coupled to bus 602 for storing static information and instructions for processor 604. A storage device 610, such as a magnetic disk, an optical disk, a USB flash drive, etc., is provided and coupled to bus 602 for storing information and instructions.

[0050] Computer system 600 may be coupled via bus 602 to a display 612, such as a liquid crystal display (LCD) (or a touchscreen), for displaying information to a computer user. An input device 614, including alphanumeric and other keys, is coupled to bus 602 for communicating information and command selections to processor 604. Another type of user input device is cursor control 616, such as a mouse, trackball, or cursor keys, for communicating directional information and command selections to processor 604 and controlling cursor movement on display 612. In some embodiments, the same directional information and command selections may be implemented as cursor control by receiving touches on a touchscreen without a cursor.

[0051] Computer system 600 may include a user interface module for implementing a GUI, which may be stored on a mass storage device as executable software code executed by the computing devices. This and other modules may include, for example, components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0052] In general, the words "component," "engine," "system," "database," "data store," and the like, as used herein, may refer to logic embodied in hardware or firmware, or in a collection of software instructions that may have entry and exit points and are written in a programming language such as Java, C, or C++. A software component may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as BASIC, Perl, or Python. It is understood that software components may be called by other components or by themselves, and / or may be called in response to detected events or interrupts.Software components configured to execute on computing devices may be provided on a computer-readable medium, such as a CD, digital video disc, flash drive, magnetic disk, or other tangible medium, or as a digital download (and may have originally been stored in a compressed or installable format that must be installed, decompressed, or decrypted prior to execution). Such software code may be stored partially or entirely on a storage device of the executing computing device for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. It is further understood that hardware components may consist of connected logic units, such as gates and flip-flops, and / or may consist of programmable units, such as programmable gate arrays or processors.

[0053] Computer system 600 may implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, causes or programs computer system 600 to be a special-purpose machine. According to one embodiment, the techniques contained herein are performed by computer system 600 in response to processor(s) 604 executing one or more sequences of one or more instructions contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage device 610. Execution of the sequences of instructions contained in main memory 606 causes processor(s) 604 to perform the process steps described herein.In alternative embodiments, hard-wired circuits may be used instead of or in combination with software instructions.

[0054] The term "non-transitory media" and similar terms as used herein refer to any media that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 610. Volatile media includes dynamic memory, such as main memory 606.Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, a hard disk, a solid-state drive, a magnetic tape or other magnetic data storage medium, a CD-ROM, another optical data storage medium, any physical medium with hole patterns, a RAM, a PROM and an EPROM, a FLASH EPROM, NVRAM, any other memory chip or any other cartridge, and networked versions thereof.

[0055] Non-transitory media differs from transmission media but can be used in conjunction with transmission media. Transmission media are involved in the transfer of information between non-transitory media. For example, transmission media include coaxial cable, copper wire, and fiber optics, including the wires comprising bus 602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communication.

[0056] Computer system 600 also includes a communications interface 618 coupled to bus 602. Network interface 618 provides a two-way data communications connection to one or more network connections connected to one or more local area networks. For example, communications interface 618 may be an ISDN (Integrated Services Digital Network) card, a cable modem, a satellite modem, or a modem for providing a data communications connection to a corresponding type of telephone line. As another example, network interface 618 may be a local area network (LAN) card for providing a data communications connection to a compatible LAN (or a WAN component for communicating with a WAN). Wireless connections may also be implemented.In such an implementation, the network interface 618 sends and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0057] A network connection typically provides data communication over one or more networks to other data devices. For example, a network connection may provide a connection over a local area network to a host computer or to data devices operated by an Internet service provider (ISP). The ISP, in turn, provides data communication services over the worldwide packet data communication network, now commonly referred to as the "Internet." The local area network and the Internet both use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network connection and across the communication interface 618 that carry the digital data to and from the computer system 600 are exemplary forms of transmission media.

[0058] Computer system 600 can send messages and receive data, including program code, over the network, the network connection, and the communications interface 618. In the Internet example, a server can transmit requested code for an application program over the Internet, the ISP, the local area network, and the communications interface 618.

[0059] The received code may be executed by the processor 604 when it is received and / or stored in the storage device 610 or other non-volatile memory for later execution.

[0060] All of the processes, methods, and algorithms described in the preceding sections may be embodied in code components executed by, and fully or partially automated by, one or more computer systems or computer processors comprising computer hardware. The one or more computer systems or computer processors may also be operated to support the performance of the relevant operations in a cloud computing environment or as software as a service (SaaS). The processes and algorithms may be partially or fully implemented in application-specific circuitry. The various features and methods described above may be used independently of one another or combined in various ways.Various combinations and subcombinations are intended to be within the scope of this disclosure, and certain methods or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular order, and the associated blocks or states may be performed in other suitable orders, or may be performed in parallel or in another manner. Blocks or states may be added to or removed from the disclosed example embodiments. The performance of certain operations or processes may be distributed among computer systems or computer processors located not only on a single computer, but deployed on multiple computers.

[0061] As used herein, a circuit may be implemented using any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms may be implemented to form a circuit. In implementation, the various circuits described herein may be implemented as discrete circuits, or the described functions and features may be distributed, in part or in whole, among one or more circuits.Although various features or elements of functionality may be individually described or claimed as separate circuits, these features and functions may be shared by one or more common circuits, and such description does not require or imply that separate circuits are required to implement such features or functionality. If a circuit is implemented in whole or in part using software, such software may be implemented to operate with a computer or processing system capable of performing the functions described in this context, such as computer system 600.

[0062] As used herein, the term "or" can be interpreted in either an inclusive or an exclusive sense. Furthermore, descriptions of resources, operations, or structures in the singular should not be read to exclude the plural. Conditional language, such as, but not limited to, "may," "could," "would," or "might," unless expressly stated otherwise or understood in the context used, is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not.

[0063] Terms and expressions used in this document, and variations thereof, should be interpreted as open-ended and not limiting unless expressly stated otherwise. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar import should not be construed to limit the described item to a particular time period or to an item available at a given time, but should instead be read to include conventional, traditional, normal, or standard technologies that may be available or known now or at some later time in the future.The presence of broadening words and expressions, such as “one or more,” “at least,” “but not limited to,” or similar expressions, in some cases does not mean that the narrower case is intended or required in cases where such broadening phrases may be absent.

[0064] Software-defined networking (SDN) is an approach to managing networks. SDN defines APIs that allow the data path (packet forwarding) and the control plane (protocol intelligence) of network elements to be decoupled. In other words, a network controller, an entity external to the network element, can have fine-grained control and visibility over that network element. This can be used by the network controller to dynamically change the policy of network elements or to centralize the network's control plane and decision-making.

[0065] The SDN approach is also well-suited for network traffic planning. SDN APIs typically define both measurement and control, allowing the network controller to measure the network and dictate traffic distribution.

[0066] One of the limitations of SDN is that it requires tight coupling between the network controller and the network elements. This can work well at small to medium scales, but typically does not scale to larger networks. Its efficiency is also reduced when the network between the network controller and the network elements has limited performance (e.g., low bandwidth or high latency). Furthermore, the SDN approach typically does not allow for crossing administrative domain boundaries, as different entities can only trust each other during controlled and limited interactions.

[0067] Software-defined WAN (SD-WAN) technology proposes using SDN principles to manage WAN connectivity. This can provide centralized visibility and control over the enterprise's WAN connectivity. Another goal of SD-WAN is to reduce the cost of WAN connectivity.

[0068] SD-WAN reduces costs by replacing these dedicated WAN connections with tunnels over the internet. In this case, each branch and site has multiple WAN connections connected to the internet. Typically, low-cost consumer WAN technologies such as DSL, cable modems, or wireless 3G are used. A dedicated SD-WAN gateway at each branch and site creates private tunnels (VPN tunnels) to securely connect to other branches and sites over the WAN connections and the internet.

[0069] Replacing dedicated WAN connections can reduce the cost of WAN connectivity. However, consumer WAN technology and the Internet do not offer the same guarantees of high availability and quality of service as older dedicated WAN connections.

[0070] The typical way SD-WAN solutions address this problem is for the SD-WAN gateway at each branch and site to have multiple WAN connections to the internet. The SD-WAN gateway creates parallel tunnels across the internet using each WAN connection, implements tunnel handoff, and uses network traffic planning to route traffic to the most appropriate network tunnel, with the goal of optimally utilizing available network capacity. For example, if the SD-WAN gateway detects that a WAN connection is down, it will route traffic from that WAN connection to a tunnel that doesn't use that particular WAN connection.

[0071] The SD-WAN gateway can perform more advanced traffic planning. For example, it can monitor the performance of each tunnel in terms of latency, available bandwidth, and throughput, and then load-balance the traffic or assign each traffic type to the tunnel most suitable for that traffic.

[0072] Network traffic planning typically examines the characteristics of network traffic and network elements and their connectivity to support both network design and direct traffic along different paths within that network. Appropriate traffic planning contributes to increasing network performance and reducing costs.

[0073] One of the simplest forms of network traffic planning is to add a backup link to the network to be used in the event of a primary link failure. Multi-rooted structures, such as fat-tree, are topologies that provide many such parallel paths. They have been studied extensively, and many approaches have been proposed for loading balancing packets onto such topologies. The Internet is a more complex network, divided among many entities and administrative domains. Therefore, traffic planning for the Internet is much more complex. On the Internet, each entity typically performs fairly advanced traffic engineering within its own domain, and cross-domain interactions are usually managed using BGP (Border Gateway Protocol) routing, which allows only fairly coarse-grained traffic planning.

[0074] Most traffic engineering techniques consist of three parts. The first part is measurement, which measures some attributes of the traffic and / or network. The second part is optimization, which calculates an optimal traffic distribution. The third part is control, which reconfigures the network to implement the desired traffic distribution.

[0075] A key component of such traffic planning is a way to measure the performance of each network tunnel. Each tunnel defines a network path across the Internet. The tunneled packets are processed by a number of network elements along the network path. The network path used by a tunnel (outside the tunnel) and the network path inside a tunnel are logically different (they have different endpoint addresses). However, these two network paths traverse the same network elements and, in most cases, exhibit nearly identical performance and highly correlated performance characteristics. Therefore, the performance of a network tunnel can be measured by measuring the network path outside or inside the tunnel. For the purposes of the disclosed technology, they are considered the same network path.

Claims

[1] Method comprising: Receiving (406) one-way delay measurements for probe packets (P.1; P.2; P.3) in a probe sequence sent over a network path (104) in a system (100) comprising a hardware processor (402); Grouping (408), by the system (100), the probe packets (P.1; P.2; P.3) into a plurality of pairs (506, 508; 510, 512) of probe packets (P.1; P.2; P.3) based on the one-way delay measurements that form a ridge, wherein two probe packets (P.1; P.2; P.3) are grouped (408) into a pair if the two probe packets (P.1; P.2; P.3) have similar one-way delay measurements on either side of the ridge; Calculating (410), by the system (100), a respective noise threshold for each pair of the plurality of pairs based on one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after a later transmitted probe packet (P.1; P.2; P.3) of the respective pair, wherein for the respective pair, a statistic of the one-way delay measurements of all probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the respective pair is calculated; Selecting (412), by the system (100), a selected pair of the plurality of pairs according to a comparison of the one-way delay measurements for the probe packets (P.1; P.2; P.3) of the plurality of pairs with the noise thresholds; and Estimating (414), by the system (100), an available bandwidth on the network path (104) based on the transmission times of the probe packets (P.1; P.2; P.3) in the selected pair. [2] The method of claim 1, wherein grouping (408) the probe packets (P.1; P.2; P.3) into a plurality of pairs comprises: Grouping (408) two of the probe packets (P.1; P.2; P.3) into a pair based on positions of one-way delay measurements of the two of the probe packets (P.1; P.2; P.3) on the bump, wherein a peak of the bump represents a maximum one-way delay measurement of a given probe packet (P.1; P.2; P.3) for the probe sequence, the given probe packet (P.1; P.2; P.3) having been transmitted between the two probe packets (P.1; P.2; P.3), and wherein selecting (412) the selected pair is performed using a bump detection process. [3] The method of claim 1 or 2, wherein calculating (410) a first noise threshold for a first pair of the plurality of pairs comprises: Calculating an average of the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the first pair; and Calculating a standard deviation of the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the first pair. [4] The method of any one of claims 1 to 3, wherein selecting (412) the selected pair of the plurality of pairs of probe packets (P.1; P.2; P.3) comprises: Selecting a pair of the plurality of pairs of probe packets (P.1; P.2; P.3) having a lowest one-way delay measurement that exceeds a respective one of the noise thresholds. [5] Method according to one of claims 1 to 4, further comprising: Performing network traffic planning for a network comprising the network path (104) based on the estimated (414) available bandwidth. [6] The method of claim 5, wherein performing network traffic planning for the network comprises routing packets through the network and / or performing load balancing for the network. [7] System (100), comprising: a hardware processor (402); and a non-transitory machine-readable storage medium (404) comprising instructions (414) executable on the hardware processor (402) to: receive one-way delay measurements for probe packets (P.1; P.2; P.3) in a probe sequence sent over a network path (104); grouping (408) the probe packets (P.1; P.2; P.3) into a plurality of pairs of probe packets (P.1; P.2; P.3) based on the one-way delay measurements forming a ridge, wherein two probe packets (P.1; P.2; P.3) are grouped into a pair (408) if the two probe packets (P.1; P.2; P.3) have similar one-way delay measurements on either side of the ridge; for each respective pair of the plurality of pairs, to calculate (410) a respective noise threshold based on the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after a later transmitted probe packet (P.1; P.2; P.3) of the respective pair, wherein for the respective pair a statistic of the one-way delay measurements of all probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the respective pair is calculated; using an increase detection process, selecting a selected pair of the plurality of pairs according to a comparison of the one-way delay measurements for the probe packets (P.1; P.2; P.3) of the plurality of pairs with the noise thresholds; and to estimate (414) an available bandwidth on the network path (104) based on the transmission times of the probe packets (P.1; P.2; P.3) in the selected pair. [8] The system (100) of claim 7, wherein grouping (408) the probe packets (P.1; P.2; P.3) into the plurality of pairs comprises: Grouping (408) two of the probe packets (P.1; P.2; P.3) into a pair based on positions of one-way delay measurements of the two of the probe packets (P.1; P.2; P.3) on the bump, wherein a peak of the bump represents a maximum one-way delay measurement of a given probe packet (P.1; P.2; P.3) for the probe sequence, wherein the given probe packet (P.1; P.2; P.3) was transmitted between the two probe packets (P.1; P.2; P.3), and wherein the bump detection process detects the one-way delay measurements of the two of the probe packets (P.1; P.2; P.3) on different sides of the bump relative to the peak of the bump. [9] The system (100) of claim 7 or 8, wherein calculating (410) a first noise threshold for a first pair of said plurality of pairs of probe packets (P.1; P.2; P.3) includes one or more of: Calculating an average of the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the first pair; Calculating a mean average distance to the average of the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the first pair; Calculating a standard deviation of the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the first pair; and Calculating a cumulative distribution function of the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the first pair. [10] The system (100) of any one of claims 7 to 9, wherein selecting (412) the selected pair of the plurality of pairs of probe packets (P.1; P.2; P.3) comprises: Selecting a pair of the plurality of pairs of probe packets (P.1; P.2; P.3) having a lowest one-way delay measurement exceeding a respective noise threshold of the noise thresholds. [11] The system (100) of any one of claims 7 to 10, wherein the method for detecting a peak of the peak comprises the selected pair of probe packets (P.1; P.2; P.3) associated with one-way delay measurements located on different sides of the peak relative to the peak of the peak. [12] The system (100) of any one of claims 7 to 11, wherein the instructions are executable on the hardware processor (402) to perform network traffic planning for a network based on the estimated (414) available bandwidth, the network traffic planning comprising: Routing packets through the network; or Perform load balancing on the network. [13] The system (100) of any one of claims 7 to 12, wherein the instructions are executable on the hardware processor (402) to: select a transmission rate for transmission control protocol (TCP) packets to be transmitted over the network path (104) based on the estimated (414) available bandwidth; or select a video coding rate for video to be transmitted over the network path (104) based on the estimated (414) available bandwidth. [14] A non-transitory machine-readable storage medium (404) comprising instructions that, when executed, cause a system (100) to: receive one-way delay measurements for probe packets (P.1; P.2; P.3) in a probe sequence sent over the network path (104); to group (408) the probe packets (P.1; P.2; P.3) into a plurality of pairs of probe packets (P.1; P.2; P.3) based on the one-way delay measurements forming an increase, wherein two probe packets (P.1; P.2; P.3) are grouped (408) into a pair if the two probe packets (P.1; P.2; P.3) have similar one-way delay measurements on either side of the ridge; calculating (410) for each respective pair of the plurality of pairs a respective noise threshold based on the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after a later transmitted probe packet (P.1; P.2; P.3) of the respective pair, wherein for the respective pair a statistic of the one-way delay measurements of all probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the respective pair is calculated; using an increase detection process, select (412) a selected pair of the plurality of pairs according to a comparison of the one-way delay measurements for the probe packets (P.1; P.2; P.3) of the plurality of pairs with the noise thresholds; and to estimate (414) an available bandwidth on the network path (104) based on the transmission times of the probe packets (P.1; P.2; P.3) in the selected pair. [15] The non-transitory machine-readable storage medium (404) of claim 14, wherein grouping (408) the probe packets (P.1; P.2; P.3) into multiple pairs comprises: Grouping (408) two of the probe packets (P.1; P.2; P.3) into a pair based on positions of one-way delay measurements of the two of the probe packets (P.1; P.2; P.3) on the bump, wherein a peak of the bump represents a maximum one-way delay measurement of a given probe packet (P.1; P.2; P.3) for the probe sequence, wherein the given probe packet (P.1; P.2; P.3) was transmitted between the two probe packets (P.1; P.2; P.3), and the bump detection process detects the one-way delay measurements of the two of the probe packets (P.1; P.2; P.3) on different sides of the bump relative to the peak of the bump. [16] The non-transitory machine-readable storage medium (404) of claim 14 or 15, wherein calculating (410) a first noise threshold for a first pair of said plurality of pairs of probe packets (P.1; P.2; P.3) comprises: Calculating an average of the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the first pair; or Calculating a mean average distance to the average of the one-way delay measurements of the probe packets (P.1; P.2; P.3) transmitted after the later transmitted probe packet (P.1; P.2; P.3) of the first pair. [17] The non-transitory machine-readable storage medium (404) of any one of claims 14 to 16, wherein selecting (412) the selected pair of the plurality of pairs of probe packets (P.1; P.2; P.3) comprises: Selecting a pair of the plurality of pairs of probe packets (P.1; P.2; P.3) having a lowest one-way delay measurement exceeding a respective noise threshold of the noise thresholds. [18] The non-transitory machine-readable storage medium (404) of any one of claims 14 to 17, wherein the bump detection process detects a peak of the bump and comprises the selected pair of probe packets (P.1; P.2; P.3) associated with one-way delay measurements located on different sides of the bump relative to the peak of the bump. [19] The non-transitory machine-readable storage medium (404) of claim 18, wherein another pair of said plurality of pairs of probe packets (P.1; P.2; P.3) is associated with one-way delay measurements that do not exceed a respective one of said noise thresholds. [20] The non-transitory machine-readable storage medium (404) of any one of claims 14 to 19, wherein the instructions, when executed, cause the system (100) to: select a transmission rate for transmission control protocol (TCP) packets to be transmitted over the network path (104) based on the estimated (414) available bandwidth; or select a video coding rate for video to be transmitted over the network path (104) based on the estimated (414) available bandwidth.

Images