Apparatus and method for stochastic multi_path packet switching

Abstract

A stochastic multi-path switching mechanism for use with multi-path internet connections configured to switch paths based on first order and second order changes in minimum round- trip times of network paths. The multi-path switching scheme is decoupled from underlying congestion control algorithms (CCA) and maintains fairness by executing path checking based on the smallest minimum round-trip time among the multiple paths of the multi-path network connection. Data transmission over a first and second path is managed by a first and second CCA respectively where the first and second CCA need not be the same. Path switching is performed by a multi-path switching scheme that switches paths based at least in part on first order and second order changes in the round-trip time of the multiple paths. Path checking is performed at an interval based on the intrinsic minimum possible period of network feedback.

Description

[0001] APPARATUS AND METHOD FOR STOCHASTIC MULTI PATH PACKET SWITCHING

[0002] TECHNICAL FIELD

[0003] The aspects of the disclosed embodiments relate generally to computer networking, and more particularly to multi-path network congestion control.

[0004] BACKGROUND

[0005] A proliferation of cellular and wireless services has provided modern mobile devices, such as smart phones and tablets, with the ability to connect to multiple network paths at once. For example, a mobile phone that is connected to both LTE as well as Wifi leads to an expectation that these multiple network paths can be combined to provide better performance. Traditional protocols require modifications to the protocol core to take advantage of multiple network paths. Efforts are currently underway to standardize protocol extensions to enable near simultaneous use of multiple paths for a single connection, however additional work is needed to achieve desired results.

[0006] Scheduling and balancing packets across different network paths is a difficult problem. Scheduling resources of a buffered network system, such as the internet, is a distributed and dynamic optimization problem. Multiple senders and receivers must modulate their use of shared network resources to obtain the most efficient use of the network. Network users need to balance obtaining the best performance for their own packets against the social need to remain fair to other users and avoid a systemic congestion induced collapse of the network.

[0007] Our modem world has become highly networked, and it is common for devices to access the internet over multiple distinct access networks. For example, a mobile phone may create multiple network links from the two common families of wireless networks: the ITU-R / 3GPP family, e.g. LTE, 5G, etc.; and the IEEE 802.11 ‘WiFi’ family. Consequently, interest has grown in enabling transport protocols to make (near) simultaneous use of available network connections in order to improve network reliability and / or throughput to remote servers on the Internet. Managing these multiple links adds an additional layer of multi-path scheduling and congestion control to a transport protocol. The multi-path problem is complicated by the need to balance the desire for improved performance against the need to remain fair to other common protocols that are also using the network

[0008] Thus, there is a need for improved apparatus and methods capable of efficiently managing multi-path network connections while remaining fair to other users and avoiding a systemic congestion induced collapse of the network. Accordingly, it would be desirable to provide methods and apparatus that addresses at least some of the problems described above.

[0009] SUMMARY

[0010] The aspects of the disclosed embodiments are directed to apparatus and methods for operating multi-path network connections. The aspects of the disclosed embodiments describe a stochastic multi-path switching algorithm that determines path switching decisions in part on the first order change in round-trip time and the second order change in round-trip time of the path, while maintaining fairness by performing path checking based on the intrinsic minimum possible period of network feedback. The use of stochastic multi-path switching decouples the multi-path switching scheme from the underlying congestion control algorithms, operates the network connection near Kleinrock optimum, and remains fair to other users of the network.

[0011] According to a first aspect, the above and further implementations and advantages are obtained by an apparatus including a processor communicatively coupled with a memory and a plurality of network interfaces. The memory includes program instructions that when executed by the processor cause the apparatus to send and receive data packets over a network connection, where the network connection includes a plurality of paths. The processor is configured to: manage data transmission over a first path in the plurality of paths based on a first congestion control algorithm; manage data transmission over a second path in the plurality of paths based on a second congestion control algorithm; switch data transmission from the first path to the second path based on a multi-path switching algorithm, wherein the multi-path switching algorithm switches paths based at least in part on a combination of a first order change in a round-trip time and a second order change in the round-trip time.

[0012] In a possible implementation form, the multi-path switching algorithm switches paths based at least in part on a packet loss of a path in the plurality of paths. Switching based on packet loss avoids increased packet delays that can occur on a lossy network path. In a possible implementation form, the multi-path switching algorithm is run at an interval based on a minimum-minimum round-trip time of all paths in the plurality of paths. Executing path switching at the intrinsic minimum possible period of network feedback improves fairness to other users on the network.

[0013] In a possible implementation form, the processor is configured to switch paths when the first order change in the round-trip time is greater than zero and the second order change in the round-trip time is greater than zero. Both the first order change in the round-trip time and the second order change in the round-trip time transition to greater than zero at a network operating point that is near the Kleinrock optimum.

[0014] In a possible implementation form, the first congestion control algorithm is different than the second congestion control algorithm. Because the stochastic multi-path switching algorithm is decoupled from the underlying CCA, each path can use a different CCA thereby allowing the CCA to be tailored for the specific type of network path.

[0015] In a possible implementation form, the first congestion control algorithm and the second congestion control algorithm include one or more of a Tahoe TCP congestion control algorithm, a Reno TCP congestion control algorithm, a BBR control algorithm, and a CUBIC congestion control algorithm. Decoupling the stochastic multi-path switching scheme from the underlying CCA allows use of any of the reliable and well understood CCA for per-path congestion control.

[0016] According to a second aspect, the above and further implementations and advantages are obtained by a method for switching paths in a multi-path network connection, wherein the multi-path network connection includes at least a current path and a second path, and wherein data transmission over the current path is managed based on a first congestion control algorithm, and data transmission over the second path is managed based on a second congestion control algorithm. The method includes: when the second path is not losing and the current path is losing, set the current path to the second path. When the current path is not losing and the second path is not losing: compute a first derivative of a round-trip time of the second path and a second derivative of the round-trip time of the second path; and set the current path to the second path based at least in part on the first derivative of the round-trip time of the second path and the second derivative of the round-trip time of the second path. The use of stochastic multipath switching decouples the multi-path switching scheme from the underlying congestion control algorithms, operates the network connection near Kleinrock optimum, and remains fair to other users of the network.

[0017] In a possible implementation form, the method is repeated based on a minimum-minimum round-trip time of the current path and the second path. Executing the path checking method at the minimum-minimum round-trip time promotes fairness to other users on the network.

[0018] In a possible implementation form, the current path is set to the second path when the first derivative of the round-trip time of the second path is greater than zero and the second derivative of the round-trip time of the second path is greater than zero. Both the first order change in the round-trip time and the second order change in the round-trip time transition to greater than zero at a network operating point near the Kleinrock optimum.

[0019] In a possible implementation form, the first congestion control algorithm and the second congestion control algorithm include one or more of a Tahoe TCP congestion control algorithm, a Reno TCP congestion control algorithm, a BBR congestion control algorithm, and a CUBIC congestion control algorithm. Decoupling the multi-path switching from the underlying CCA allows any of the reliable and well understood CCA to be used for per-path congestion control.

[0020] In a possible implementation form, the first congestion control algorithm is different than the second congestion control algorithm. Decoupling the multi-path switching from the underlying CCA allows the CCA of each path to be tailored to specific requirements of the type of network of each individual path.

[0021] In a possible implementation form, when a new path is added to the multi-path network connection, the method is configured to enter a probing phase. When in the probing phase the method is configured to: when the new path is not the best path, set the new path as the best path. A probing end time is then computed based on a minimum round-trip time of the new path, a pre-determined probing parameter, and a probing start time. The method then determines whether probing is finished based on a minimum round-trip time of the new path, a current time, and the probing end time; and when probing is finished, exit the probing phase. Inclusion of a probing phase allows initial values for the CWND, and metrics to be established for a newly added network path. According to a third aspect, the above and further implementations and advantages are obtained by a computer program product including a non-transitory computer readable media having stored thereon program instructions that when executed by a processor cause the processor to perform the method according to the second aspect. The use of stochastic multi-path switching decouples the multi-path switching scheme from the underlying congestion control algorithms, operates the network connection near Kleinrock optimum, and remains fair to other users of the network.

[0022] These and other aspects, implementation forms, and advantages of the exemplary embodiments will become apparent from the embodiments described herein considered in conjunction with the accompanying drawings. It is to be understood, however, that the description and drawings are designed solely for purposes of illustration and not as a definition of the limits of the disclosed invention, for which reference should be made to the appended claims. Additional aspects and advantages of the invention will be set forth in the description that follows, and in part will be obvious from the description, or may be learned by practice of the invention. Moreover, the aspects and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the following detailed portion of the present disclosure, the invention will be explained in more detail with reference to the example embodiments shown in the drawings, in which like references indicate like elements and:

[0025] Figure 1 illustrates a block diagram of an exemplary computing apparatus configured to employ stochastic multi-path switching methodology incorporating aspects of the disclosed embodiments;

[0026] Figure 2 illustrates a graph showing how network data rate, delay, and packet loss are affected by network load.

[0027] Figure 3 illustrates a graph showing changes in sign of the first and second derivative of roundtrip time incorporating aspects of the disclosed embodiments;

[0028] Figure 4 illustrates a flow chart showing the phases of a stochastic multi-path switching algorithm incorporating aspects of the disclosed embodiments;

[0029] Figure 5 illustrates a pictorial diagram depicting an exemplary path switching sequence incorporating aspects of the disclosed embodiments; Figure 6 illustrates a flow chart of an exemplary stochastic multi-path switching algorithm incorporating aspects of the disclosed embodiments; and

[0030] Figure 7 illustrates a flow chart of an exemplary path probing method incorporating aspects of the disclosed embodiments.

[0031] DETAILED DESCRIPTION OF THE DISCLOSED EMBODIMENTS

[0032] Figure 1 illustrates a block diagram of an exemplary computing apparatus 100 configured to employ stochastic multi-path switching (SMPS) methodology, also referred to herein as a stochastic multipath switching algorithm, incorporating aspects of the disclosed embodiments. The exemplary apparatus 100 of the disclosed embodiments generally comprises a computing apparatus 100 configured to provide improved use of multi-path network connections. Among other benefits, the use of the herein disclosed SMPS algorithm allows a multi-path network connection to operate closer to the Kleinrock optimum without sacrificing fairness to other users on the network and without any dependencies between the multi-path switching method and the underlying per path congestion control algorithms. These improvements and advantages are obtained in part by employing a novel SMPS algorithm that reacts to changes in round-trip time (RTT) of the multiple paths and is independent of the underlying congestion control algorithms employed by each path.

[0033] As used herein the terms ‘method’, ‘process’, and ‘algorithm’, are used interchangeably to describe a set of steps executed by a processing device, such as the exemplary apparatus 100, to manage transmission of data packets over a computer network.

[0034] Referring to Figure 1, in one embodiment the exemplary apparatus 100 comprises a processor 102 communicatively coupled to a memory 104 and a plurality of network interfaces 106a, 106b, ... 106x. The memory 104 comprises program instructions that when executed by the processor 102 cause the apparatus 100 to send and receive data packets over a network connection 108. The network connection 108 comprises a plurality of paths 110a, 110b, ... 1 lOx and the processor 102 is configured to manage data transmission over a first path 110a in the plurality of paths 110a, 110b, . . . 1 lOx based on a first congestion control algorithm; manage data transmission over a second path 110b in the plurality of paths 110a, 110b, ... 1 lOx based on a second congestion control algorithm; and switch data transmission from the first path 110a to the second path 110b based on a multi-path switching algorithm. The multi-path switching algorithm switches paths based at least in part on a combination of a first order change in a round-trip time (dRTT) and a second order change in the round-trip time (dA2RTT).

[0035] The processor 102 is capable of reading and writing data to / from the memory 104, and can send and receiving data over a computer network via the plurality of network interfaces 106a, 106b, . . . 106x. As will be discussed further below, the plurality of network interfaces 106a, 106b, . . . 106x is configured to create a multi-path network connection 108 over a computer network, such as the internet or other desired wide area network (WAN) or local area network (LAN), where the network connection 108 includes a plurality of paths 110a, 110b, ... HOx. For example, in certain embodiments, each path in the plurality of paths 110a, 110b, ... HOx, corresponds to a network interface in the plurality of network interfaces 106a, 106b, . . . 106x.

[0036] The apparatus 100 may be any desired type of computing apparatus or communications apparatus including but not limited to a mobile communications device such as a smartphone, wearable, or tablet. In certain embodiments the apparatus 100 may be a personal computing apparatus such as a laptop or other type of personal computing device, a server apparatus such as those used in cloud computing data centers, or any desired computing apparatus that includes a processor, a memory, and a means for communicating over a computer network. When desired the apparatus 100 may be a virtualized computing apparatus executing within a hypervisor.

[0037] As is shown in the example of Figure 1, the processor 102 is communicatively coupled to the memory 104 and is configured to read and perform operations on data stored in the memory 104. Optionally, the apparatus 100 may include a system storage (not shown) such as a disk drive or solid-state disk configured to provide high-capacity long term storage capabilities.

[0038] The processor 102 can generally be any suitable processing device or group of processing devices, including but not limited to a high-performance multi-core computer processing device such as those used in large cloud computing data centers, a multi-core or single core microprocessor such as those used in workstations and laptop computers, a processing device embedded in a system such as a system on a chip (SoC), or any suitable or specialized processing device such as those used in mobile communications devices, telecommunications equipment, and smart devices configured for the internet of things (loT). The memory 104 may include any desired type or combination of computer accessible memory, such as random-access memory (RAM), read-only memory (ROM), or other suitable types of volatile and non-volatile memory.

[0039] The world in which we live has become highly networked, and it is now common for computing devices, such as the apparatus 100, to be simultaneously connected to multiple distinct access networks to the internet. For example, it is common for mobile devices to be equipped with multiple wireless networks such as from the two different families of wireless networking. For example, a mobile phone may include a wireless network interface from the International Telecommunications Union Radiocommunications sector 3rdGeneration Partnership Program (ITU-R / 3GPP) family of wireless standards, such as LTE, 5G, etc., and also include a second wireless network interface from the Institute of Electrical and Electronic Engineers (IEEE) 802.11 “WiFi” family of wireless standards.

[0040] Because multiple network connections are readily available, interest has grown in enabling transport protocols to make (near) simultaneous use of available network connections for the purpose of improving network reliability and / or throughput to remote servers on the Internet. Adding multiple paths to a network connection adds an additional layer of multi-path scheduling and congestion control on top of the underlying transport protocol. This additional layer of multi-path control needs to improve network performance while remaining fair to other common protocols also using the network.

[0041] Multiple network connections are supported by the apparatus 100 through the plurality of network interfaces 106a, 106b, ... 106x. Each network interface 106a, 106b, ... 106x is communicatively coupled with the processor 102 and may be configured to couple with any suitable type of computer network configured to allow an apparatus 100 or other computerized device to share information and resources. The network interfaces 106a, 106b, ... 106x may include any desired type of physical media such as wired, optical, or wireless media. Wired networks include any desired type of computer network configured to use electric signals to carry information over network links, such as links constructed of conductive wires. Optical networks include any type of network configured to use optical signals to carry information over network links, such as links constructed of fibre-optic materials. Wireless networks include any suitable type of computer network, such as WIFI, Bluetooth, long term evolution (LTE), 5G, or other wireless broadband communication mechanisms, that uses radio signals to transmit information through air or other material permeable to radio signals.

[0042] System software included in the computing apparatus 100 may, when desired, provide a network communication stack such as an internet protocol suite commonly referred to as TCP / IP (transmission control protocol / intemet protocol) stack. The network communication stack may be configured to provide a reliable, connection-oriented transport service.

[0043] The processor 102 of the exemplary apparatus 100 is configured to manage data transmission over a first path 110a in the plurality of paths 110a, 110b, ... 11 Ox based on a first congestion control algorithm (CCA) and to manage data transmission over a second path 110b in the plurality of paths 110a, 110b, ... 1 lOx based on a second CCA. The first CCA and the second CCA may be any desired CCA suitable for controlling congestion or managing network traffic over a path in a network connection. For example, in certain embodiments the first CCA and the second CCA may be one or more of a Tahoe TCP congestion control algorithm, a Reno TCP congestion control algorithm, a bottleneck, bandwidth, and round-trip time (BBR) congestion control algorithm, a CUBIC congestion control algorithm, or other desired CCA. Because the SMPS algorithm is decoupled from the underlying per-path congestion control, when desired, the first CCA and the second CCA may include different CCA.

[0044] As will be discussed in more detail below, the processor 102 is configured to switch data transmission from the first path 110a to the second path 110b based on a SMPS algorithm, wherein the multi-path switching scheme switches paths based at least in part on a combination of a first order change in a round-trip time, denoted herein as dRTT, and a second order change in the round-trip time, denoted herein as d2RTT.

[0045] Figure 2 illustrates a graph 200 showing how network data rate 206, delay 208, and packet loss 210 are affected by network load. In the graph 200 network load is depicted along a horizontal axis 202 increasing to the right, and magnitude of the network data rate 206, delay 208, and packet loss 210 is depicted along a vertical axis 204 increasing upwards. The data profiles depicted in the graph 200 ignore traffic bursting.

[0046] At low network loads, as depicted in the area 212, there is no congestion and packets experience the minimum possible delay in transmission across the network. As network load increases, a point is reached, as indicated by the dashed line 218, where congestion starts to occur and data packets begin to experience additional queueing delays. As can be seen in the area 214, delay continues to increase as the network load increases until a second point is reached, as indicated by the dashed line 220, where the minimum buffering capacity of the network is exceeded. At this point 220, at least one router begins dropping packets and network capacity is reached. As network load increases past the network capacity 220, delay associated with delivered packets stabilises, and the rate of packet loss increases as illustrated by the area 216. Optimal network operation occurs at a point 218 where the network is exhibiting minimum delay with maximal throughput. This optimal operating point is often referred to as the ‘KI einrock sweet-spot,’ or ‘KI einrock optimum’.

[0047] Arange of decentralised ‘congestion control’ algorithms (CCA) have been developed to manage network congestion with Internet transport protocols. In general, these CCA modulate packet transmission, and react to perceived network state, to maintain the network at its optimal operating point. These algorithms control the rate at which packets are introduced into the system based on a combination of optimisation goals and feedback from the network.

[0048] In the classic ‘black-box’ architecture of TCP / IP on the Internet, the feedback is limited to ‘acknowledgement’ packets, with the sender taking the measurable metrics of the delay in an acknowledgement or the absence of an expected acknowledgement. The delay in an acknowledgement is the amount of time between sending a data packet and receiving an acknowledgement for that packet and is referred to herein as the round-trip time, abbreviated herein as ‘RTT’.

[0049] The absence of an expected acknowledgement indicates a packet loss and is often caused by congestion in the network. A network path that is experiencing packet loss may be referred to herein as ‘losing’, i.e. a network path that is losing will not receive acknowledgements for data packets that have been lost. The sender modulates the rate of data packet transmission based on the rate of acknowledgements, and its perception of previously transmitted packets having been lost. Data packet transmission is modulated with a connection window (CWND), which is the number of data packets “in-flight,” i.e., the number of data packets sent without receiving an acknowledgement from the receiver. A variety of different heuristics may be used by the sender to increase or decrease its CWND as acknowledgements are received or loss is perceived. A receiver of data may also modulate the rate at which acknowledgements are sent. Acknowledgements may be unintentionally delayed, such as when the receiving system is busy and its packet processing is delayed, or intentionally delayed when desired.

[0050] Several conventional TCP / IP protocols, such as those deriving from the 4.3 BSD Tahoe and Reno TCP congestion control implementations, modulate sender rate based on packet loss. In steady state, these congestion control protocols, probe for more bandwidth by additively increasing their CWND when there is no loss, and reducing, typically by halving, the CWND when loss is detected, thereby resulting in an oscillating process. A more recent refinement, known as the widely used TCP CUBIC algorithm, is based on a cubic probe-backoff function, with the inflection point rooted at the value of the CWND.

[0051] To overcome inherent drawbacks of loss-based congestion control feedback, newer approaches such as delay congestion avoidance (DCA) algorithms have been worked on. More recently, algorithms that combine DCA with loss-based peak bandwidth probing have seen some success when managing data transmission on a single network path.

[0052] When a network connection includes multiple paths, an additional layer of multi-path scheduling is required. Multi-path scheduling and congestion control algorithms are a logical evolution of loss-based and delay-based algorithms. Conventional multi-path switching algorithms are generally loss based in their back-off functions and so by their design cause a network to maximize RTT delay under load, and as a result, are unable to operate the network near the Kleinrock sweet spot.

[0053] The multi-path switching algorithms or schemes disclosed herein employ a SMPS scheme based on a hybrid loss and DCA based methodology, which is adapted to switch paths based on the higher-order rates of change of RTT delay while also responding to loss events. This multipath switching scheme seeks to minimise queue build-up in the network and hence minimizes RTT. The SMPS scheme also avoids the complexities of coupling changes to the CWND of each sub-flow or path, and instead uses a stochastic switching mechanism built on standard CCAs. This SMPS scheme responds to underlying delay-change and loss events in the per-path CCA by switching to other available paths. More congested paths will see more events, and will be switched away from more frequently. Lower min-RTT paths will have proportionately more opportunity to switch traffic to themselves, thus the stochastic nature of the SMPS algorithms. The SMPS embodiments described herein can work with any standard, single-path CCA which maintains a CWND and measures RTT.

[0054] Each path in a multi-path connection has its own minimum RTT (min-RTT) value. In certain embodiments, the min-RTT value is provided by the underlying per-path CCA implementations. Each path in a multi-path connection will have its own min-RTT. The smallest or minimum min-RTT value among all paths in a multi-path connection is referred to herein as the minimumminimum RTT, and is abbreviated herein as min-min-RTT.

[0055] The SMPS algorithm effectively considers whether to alternate its output path every min-min- RTT, sending a train of packets of up to the CWND of the best path in that period before considering whether to alternate to another path. To preserve fairness, the SMPS scheme behaves as a single-path sender over any given RTT. Throughput is achieved by being uncoupled between paths - the CWND on each path can grow to its full potential, rather than being restricted to a weighted share of an overall connection-level CWND. To achieve this, a potentially higher degree of “Happiness” is tolerated - reasoning there is no benefit to very finegrained switching of packets between paths which can lead to suboptimal results.

[0056] The exact degree of Happiness determines the balance between throughput and fairness. As the period of alternation is increased, the SMPS scheme must tend to the behaviour of the underlying single-path CCA sender and hence maximises fairness, so far as that underlying CCA is fair. As the period of alternation is decreased, the SMPS scheme tends to the behaviour of the uncoupled scheme, which minimises fairness and maximises throughput. The period of alternation in this scheme replaces the coupling that other multi-path congestion control algorithms (MPCCAs) use to balance goals, relying instead on the intrinsic minimum possible period of network feedback, the min-min-RTT, to govern the period of alteration. It has been shown empirically that the SMPS algorithms disclosed herein achieve better throughput than the conventional Linux multi-path TCP (MPTCP). Thus, in certain embodiments, a multi-path switching algorithm, such as the SMPS algorithm, may be run every min-min RTT thereby ensuring an appropriate degree of fairness.

[0057] Decoupling the multi-path switching algorithm from the underlying single path CCA has the advantage of allowing reliable well understood CCAs to be used to manage packet transmission within each path. Additionally, decoupling allows different paths to employ different CCA while path switching is still managed by a single multi-path switching algorithm, such as the SMPS algorithm described herein.

[0058] Figure 3 illustrates a graph 300 showing changes in sign of the first and second derivative of RTT incorporating aspects of the disclosed embodiments. In the graph 300, time is depicted along a horizontal axis 202 increasing to the right, and RTT is depicted along a vertical axis 204 increasing upwards. The first derivative of RTT, abbreviated herein as dRTT, may be referred to as the first-order change in RTT, and the second derivative of RTT, abbreviated herein as d2RTT, may be referred to as the second-order change in RTT. These higher-order changes in RTT provide useful information about how the RTT is changing and allow some degree of prediction. In shaded regions 308, RTT is generally stable. Shaded regions 310 correspond to regions where both higher-order rates of change of RTT are positive, dRTT > 0 and d2RTT > 0. Shaded region 312 corresponds to a region where dRTT > 0 and d2RTT < 0. Shaded region 314 corresponds to a region where dRTT < 0 and d2RTT < 0. Shaded region 316 corresponds to a region where dRTT < 0 and d2RTT > 0.

[0059] SMPS improves on conventional DCAs by incorporating higher-order rate of change of RTT into its decision process. A significant problem with DCAs is the noise in RTT, which persists even after smoothing, such as via exponential weighted averaging, to filter out higher frequency noise. The SMPS scheme further minimises the impact of remaining noise by reacting to the combination of first order change in RTT, dRTT, and the second-order change in RTT, d2RTT. The reasoning being that the Kleinrock optimum is characterised by the inflection point where RTT changes from generally stable to generally increasing, modulo noise.

[0060] As shown in Figure 3, considering d2RTT in combination with dRTT allows a switching algorithm to focus on the regions of the RTT regime where the delay signal corresponds to the Kleinrock optimum which occurs when dRTT > 0 and d2RTT > 0 as illustrated by the regions 310. The SMPS algorithm looks for regions 310 where dRTT > 0 and d2RTT > 0, and ignores, or filters out, other regions, 308, 312, 314, 316, where RTT is less useful as a switching signal. RTT is less useful as a switching signal in these other regions, 308, 312, 314, 316, either because it is converging on a new stable value after a change in network conditions or because the network is operating in a region where RTT is moving toward a loss condition. Use of the higher-order changes in RTT allows the SMPS scheme to focus on the desired regions, 310, of RTT, while avoiding influence of RTT noise in other less useful regions 308, 312, 314, 316.

[0061] Figure 4 illustrates a flow chart 400 showing the phases of a SMPS algorithm incorporating aspects of the disclosed embodiments. The SMPS algorithm of the disclosed embodiments proceeds through two phases as outlined in Figure 4. When a new path is added 402 to a network connection, the network connection enters a probing phase 404 where switching between available paths occurs over a longer interval than when the network connection is in the steady state 406. While in the probing phase 404, path checking occurs based on a predetermined probing parameter, Conn.n probing r Us, which represents a pre-determined multiplier for the current min-min-RTT of the connection being probed. Inclusion of a probing phase 404 allows the underlying per-path CCAs to probe the path and build up the CWND. After the probing phase 402 is completed, the SMPS scheme switches to its steady-state switching phase 406, where a path check procedure is run for each path every min-RTT.

[0062] As will be discussed further below, the path check procedure considers whether the next packet or packets should be sent on the path being checked or on the currently selected ‘best’ path by comparing recent first and second order changes in RTT and loss to that of the currently selected ‘best’ path. This process is stochastic in that paths with lower min-RTT will run the path check procedure more frequently and have more opportunity to select themselves as the best path, while paths having a longer min-RTT will have intrinsically fewer opportunities to select themselves. Each time the path check procedure is run on a path, that path will select itself as the best path when it is not experiencing loss and it has favourable dRTT and d2RTT values.

[0063] Table 1 provides pseudo-code for an exemplary path probing procedure appropriate for use during the probing phase 404 of a SMPS algorithm. At line 2 the new path is checked against the current best path. The new path needs to receive traffic for probing to take place, so when the new path is not the best path it is set to be the currently selected best path at line 3. At lines

[0064] 5, 6 and 7 an end time for the probing phase is computed based on the min RTT of the new path and a pre-determined probing parameter n probin rtts. At line 8 the current time plus the new path’s min RTT is checked against the probing end time to determine if probing is finished. When probing is finished, a probing finished indication is set at line 9. At line 11 an alarm is set to re-run path checking on the new path at an interval based on min RTT of the new path.

[0065] Table 2 provides pseudo-code for an exemplary path checking procedure appropriate for use during the steady state path switching phase 406 of an exemplary SMPS algorithm. The PathCheck procedure receives information about the connection, Conn, and a second path, Path. The connection, Conn, is configured to transmit data packets over the connection’s best path, Conn.BestPath, also referred to herein as the current path. Before proceeding with a switching determination, validity of the second path is checked at line 22 and when the second path is not valid, path checking exits. Next, at line 24, a check is made to determine whether the connection is in the probing phase. When the connection is probing, control is passed at line 26 to a probing procedure such as the probing phase procedure described above and with reference to Table 1. When the connection is not in the probing phase, the connection’s best path is then checked for validity at line 29. When the best path is not valid, the second path is set as the new best path of the connection at line 30.

[0066] Once the second path has been determined to be valid, and the connection is not in a probing phase, and the connection’s best path Conn.BestPath is determined to be valid, it is time to check whether the transmission characteristics of the best path and second path favour a path switch. The second path is checked for loss at line 31, and when the second path is losing packets, no path switch is indicated. When the second path is not losing, first and second order changes in RTT of the best path are checked. A first derivative dRTT of the best path’s RTT is obtained at line 32, and the second derivative d2RTT of the best path’s RTT is obtained at line 33. When both first derivative dRTT of the best path’s RTT and the second derivative d2RTT of the best path’s RTT are positive, or greater than zero, a path switch is indicated and the connection’s best path is set to the second path Path at line 37. Finally, at lines 40 and 41 of the PathCheck procedure, the next time to check the second path is set based on the min-RTT of the second path. Figure 5 illustrates a pictorial diagram 500 depicting an exemplary path switching sequence incorporating aspects of the disclosed embodiments. A first path is depicted along an upper horizontal line 502 and a second path is depicted along a lower horizontal line 504 with time increasing from left to right. A solid line indicates data is being transmitted through a path and a dashed line indicates data is not being transmitted through a path. Path switching is indicated by the patterned dashed lines 516 and 520.

[0067] When the illustrated interval begins, data is being transmitted through a first path as indicated by solid line 514. This condition remains through a full network feedback period of min-min- RTT 530. After the first min-min-RTT has elapsed, as indicated by the vertical line 506, the path check procedure is run for the second path 504 and determines that path switching is favourable so the path is switched 516 making the second path 504 the active path as indicated by the solid line 518. After the next min-min-RTT 524, path checking is run for the first path 502 at the time indicated by the vertical line 508 and path switching is determined to be unfavourable, so the second path 504 remains as the active path. After a third min-min-RTT 526 path checking is again run for the first path 502 at the time indicated by the vertical line 508 and path switching is again determined to be unfavourable, so the second path 504 remains as the active path. After another min-min-RTT 528, path checking is again run for the first path 502 at a time indicated by vertical line 512, and this time path switching is determined to be favourable so data transmission is switched back to the first path 502 as indicated by the path switch 520 and solid line 522.

[0068] Figure 6 illustrates a flow chart of an exemplary stochastic multi-path switching method 600 incorporating aspects of the disclosed embodiments. The exemplary SMPS method 600 of the disclosed embodiments generally employs the novel stochastic multi-path switching methodology, as described in detail above, to provide a multi-path switching method that is decoupled from the underlying per-path CCA and tends to operate a multi-path network connection near the Kleinrock optimum without sacrificing fairness to other users on the network and without any dependencies between the multi-path switching method 600 and the underlying per path CCA.

[0069] The exemplary method 600 is appropriate for use in any suitable multi-path network connection having any number of two or more paths, such as a network connection having at least a current path and a second path. By using the intrinsic minimum possible period of network feedback, the min-min-RTT, to govern the period of alteration, the method 600 is executed 602 each min- min-RTT of the multi-path network connection.

[0070] In a multi-path network connection, one of the paths is set as the current or best path for the network connection, and a path check method 600 is run on a second path to allow the second path to set itself as the best path when conditions are favourable for a path switch. The path check method 600 relies on the RTT and losing metrics derived from acknowledgement packets as described above.

[0071] An invalid path cannot be used, therefore, the path check method 600 begins by verifying 604 validity of the second path. When the second path is not valid, the method 600 returns 606 without further processing.

[0072] Path switching is not performed when the connection is in a probing phase, such as the probing phase 404 described above. The connection is checked 608 for the probing phase, and when the connection is in the probing phase control is passed 610 to a path probing procedure, such as the path probing procedure described above and with reference to Table 1. Once probing is finished, the path probing procedure will mark probing as finished thereby allowing steady state path checking to be performed during subsequent runs of the method 600.

[0073] Next, the current best path is checked for validity 612. When the current best path is not valid the second path is set 614 as the current best path.

[0074] The second path is then checked 616 to see if it is losing or dropping data packets. When the second path is losing data packets 626, path switching is not beneficial, further checking need not be performed and the path is not switched.

[0075] A first derivative dRTT of a round-trip time of the second path and a second derivative d2RTT of the RTT of the second path is computed 618. These first and second derivatives, also referred to as first and second order rates of change, of the RTT may be computed in any appropriate fashion. In certain embodiments the derivative values dRTT, d2RTT may be provided by an underlying CCA of the path. Optionally, these values dRTT, d2RTT may be computed as part of the path check method 600, or through any other suitable computational approach as desired. A check 620a is made to determine whether the current best path is losing data packets. When the current best path is losing 620a data packets, a path switch is indicated and the second path is set as the best path 622.

[0076] As discussed above, the first and second order rate of change dRTT, d2RTT of the RTT are both positive, or greater than zero, when the network is operating in a region near the Kelinrock optimum as shown by the regions 310 in Figure 3. A check is made to determine whether both the first and second order rate of change dRTT, d2RTT of the RTT are positive 620b, and when both the first and second order rate of change dRTT, d2RTT of RTT are positive 620b a path switch is indicated and the second path is set as the current best path 622.

[0077] Before exiting, an alarm is set 624 to trigger the next execution of the path check method 600. In the illustrated embodiment, the path check method 600 will be executed again after an interval corresponding to the min-RTT of the second path.

[0078] Figure 7 illustrates a flow chart of an exemplary path probing method 700 incorporating aspects of the disclosed embodiments. The exemplary path probing method 700 of the disclosed embodiments probes a newly added path to determine a starting value for the CWND size and determine initial values for path metrics including the min-RTT and whether the path is losing.

[0079] The method 700 is executed 702 at intervals based on the min RTT of the newly added path, referred to in the following description of the probing method 700 as the new path, and begins by checking 704 whether the new path is set as the current best path of the network connection. When the new path is not set as the current best path, the current best path is replaced with the new path 706 thereby allowing metrics to be gathered from the new path.

[0080] An end time for probing is then computed 708 based on the min-RTT of each path and a predetermined probing parameter by multiplying each path’ s min-RTT by the probing parameter to obtain a maximum time for the probing max time for each path. Adding the computed max time to the probing start time, i.e. the time at which probing was begun, provides an end time, end time, for the probing phase.

[0081] The method 700 determines whether probing is finished 710 by comparing the current time to the probing end time. When the current time is greater than or equal to the end time, probing is finished. Aprobing finished indication is then set 712, and an alarm is set to run a path check method, such as the exemplary path check method 600 described above, at an interval that is a min RTT of the new path in the future. When probing is not finished, the exemplary probing method 700 sets a time for the next execution of the path probing method after an interval of a min RTT of the new path. Thus, the minimum of the end-times over the available paths will govern the end-time of connection probing.

[0082] Thus, while there have been shown, described, and pointed out, fundamental novel features of the invention as applied to the exemplary embodiments thereof, it will be understood that various omissions, substitutions and changes in the form and details of apparatuses and methods illustrated, and in their operation, may be made by those skilled in the art without departing from the spirit and scope of the presently disclosed invention. Further, it is expressly intended that all combinations of those elements, which perform substantially the same function in substantially the same way to achieve the same results, are within the scope of the invention. Moreover, it should be recognized that structures and / or elements shown and / or described in connection with any disclosed form or embodiment of the invention may be incorporated in any other disclosed or described or suggested form or embodiment as a general matter of design choice. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.

Claims

CLAIMS1. An apparatus (100) comprising a processor (102) communicatively coupled with a memory(104) and a plurality of network interfaces (106a, 106b, ... 106x), wherein the memory (104) comprises program instructions that when executed by the processor (102) cause the apparatus (100) to send and receive data packets over a network connection (108), wherein the network connection (108) comprises a plurality of paths (110a, 110b, ... HOx), and wherein the processor (102) is configured to: manage data transmission over a first path (110a) in the plurality of paths (110a, 110b, ... 1 lOx) based on a first congestion control algorithm; manage data transmission over a second path (110b) in the plurality of paths (110a, 110b, ... 1 lOx) based on a second congestion control algorithm; switch data transmission from the first path (110a) to the second path (110b) based on a multi-path switching algorithm, wherein the multi-path switching algorithm switches paths based at least in part on a combination of a first order change in a round-trip time (dRTT) and a second order change in the round-trip time (d2RTT).

2. The apparatus (100) of claim 1 wherein the multi-path switching algorithm switches paths based at least in part on a packet loss of a path in the plurality of paths (110a, 110b, ...11 Ox).

3. The apparatus (100) of any one of the preceding claims wherein the multi-path switching algorithm is run at an interval based on a minimum-minimum round-trip time of all paths in the plurality of paths (110a, 110b, ... 1 lOx).

4. The apparatus (100) of any one of the preceding claims wherein the multi-path switching algorithm is independent of the first congestion control algorithm and the second congestion control algorithm.

5. The apparatus (100) of any one of the preceding claims wherein the processor is configured to switch paths when the first order change in the round-trip time (dRTT) is greater than zero and the second order change in the round-trip time (d2RTT) is greater than zero.

6. The apparatus (100) of any one of the preceding claims wherein the first congestion control algorithm is different than the second congestion control algorithm.

7. The apparatus (100) of any one of the preceding claims wherein the first congestion control algorithm and the second congestion control algorithm comprise one or more of a Tahoe TCP congestion control algorithm, a Reno TCP congestion control algorithm, a BBR congestion control algorithm, and a CUBIC congestion control algorithm.

8. A method (600) for switching paths in a multi-path network connection, wherein the multi-path network connection comprises at least a current path and a second path, and wherein data transmission over the current path is managed based on a first congestion control algorithm, and data transmission over the second path is managed based on a second congestion control algorithm, the method (600) comprising: when the second path is not losing (616) and the current path is losing (620a), set the current path to the second path; and when the current path is not losing and the second path is not losing: compute (618) a first derivative of a round-trip time of the current path and a second derivative of the round-trip time of the current path; and set (624) the current path to the second path based at least in part on the first derivative of the round-trip time of the current path and the second derivative of the roundtrip time of the current path.9 The method (600) according to claim 8 wherein the method (600) is repeated based on a minimum-minimum round-trip time of the current path and the second path.

10. The method (600) according to claim 8 or 9 wherein the current path is set (624) to the second path when the first derivative of the round-trip time of the current path is greater than zero and the second derivative of the round-trip time of the current path is greater than zero.

11. The method (600) according to any one of claims 8 through 10 wherein the first congestion control algorithm and the second congestion control algorithm comprise one or more of aTahoe TCP congestion control algorithm, a Reno TCP congestion control algorithm, a BBR congestion control algorithm, and a CUBIC congestion control algorithm.

12. The method (600) according to any one of claims 8 through 11, wherein the first congestion control algorithm is different than the second congestion control algorithm.

13. The method (600) according to any one of claims 8 through 12, wherein when a new path is added to the multi-path network connection, enter a probing phase (700), wherein the probing phase (700) comprises: when the new path is not the best path (704), set (706) the new path as the best path; compute (708) a probing end time based on the new path minimum round-trip time, a pre-determined probing parameter, and a probing start time; determine (710) whether probing is finished based on a minimum round-trip time of the new path, a current time, and the probing end time; and when probing is finished, exit (712) the probing phase.

14. A computer program product comprising a non-transitory computer readable media having stored thereon program instructions that when executed by a processor cause the processor to perform the method (600) according to any one of claims 8 through 12

Images