System and method for push-based data communication - Patents.com

Abstract

A data packet communication system is described that may be implemented as a physical push-type data packet communication device, such as a router, a gateway, or a controller circuit coupled to a router or gateway adapted to control data packet communication, the data packet communication system adapted to assess network capacity of each of a plurality of networks at the time of a monitored communication event and allocate data packets accordingly.

Description

[Technical field]

[0001] cross reference This application is a nonprovisional application of U.S. Patent Application Nos. 63 / 139286 and 63 / 183952, both entitled SYSTEMS AND METHODS FOR PUSH-BASED DATA COMMUNICATIONS, filed on January 19, 2021 and May 4, 2021, and claims all benefits, including priority thereto, of the same, which are hereby incorporated by reference in their entireties.

[0002] This application incorporates by reference the following applications in their entireties: PCT Application No. PCT / CA2017 / 051584, filed December 21, 2017, entitled "PACKET TRANSMISSION SYSTEM AND METHOD."

[0003] PCT Application No. PCT / CA2018 / 051012, filed August 22, 2018, entitled "SYSTEM AND METHOD FOR ASSESSING COMMUNICATION RESOURCES."

[0004] PCT Application No. PCT / CA2020 / 051090, filed August 7, 2020, entitled "SYSTEMS AND METHODS FOR MANAGING DATA PACKET COMMUNICATIONS."

[0005] TECHNICAL FIELD Embodiments of the present disclosure relate to the field of electronic data packet communication, and more particularly, embodiments relate to devices, systems, and methods for push-type data packet communication where multiple network connections are available.

[0006] Introduction Existing data packet communication solutions for the transmission of data packets over multiple available network connections rely on static routing logic to determine how to propagate data packets through the connections. For example, the system may employ a timer for each available connection, using each timer to determine when to pull a packet from an input queue for transmission on the connection associated with the timer. These solutions assume that these connections have similar operating characteristics and availability, trading performance for simplicity of implementation, and schedule the next available data packet for transmission over the connection with the most recently expired timer. Although the timer approach is simple, it may suffer from computational efficiency in certain circumstances.

[0007] Communications systems (e.g., LACP, ML-PPP) that treat multiple network connections as multiple instances of a connection with similar operating characteristics can lead to less effective use of the multiple connections when the networks have different operating characteristics because a particular network may be underutilized or overutilized. This inaccuracy in use can result in greater latency in data packet delivery, less reliable data packet delivery, more expensive data packet delivery, reduced throughput, etc.

[0008] Connection management solutions also encounter difficulty in dealing with applications in which the available connections or the operating characteristics of those connections change over time. For example, timer-based systems may not have an accurate view of the network performance of all other connections because the timers for each connection fire independently and may be insensitive to changes in network performance.

[0009] Systems and methods for data communications that more efficiently utilize multiple connections to transmit data packets with lower latency, increased throughput, and enable more reliable or less expensive delivery of data packets are desirable. Summary of the Invention [Means for solving the problem]

[0010] The systems and methods described herein include a push-type data packet communication system, which may be implemented as a physical push-type data packet communication device, such as a router, a gateway, or a controller circuit coupled to a router or gateway adapted to control data packet communication. In another embodiment, the system may be implemented as software for use with electronic hardware, such as a communication controller board having a processor, or software residing on a memory as an instruction set executed by the electronic hardware. Various embodiments are described herein for operating a push-type data packet communication system to improve data communication (e.g., latency, reliability, throughput, error correction) across multiple connections based on monitored data communication characteristics (e.g., network traffic) associated with a connection of the multiple connections. Corresponding computer-readable media (e.g., non-transitory computer-readable media having stored thereon a machine-interpretable instruction set for execution on the processor) operating in conjunction with data storage, computer processors, and computer memories are also provided.

[0011] The embodiments described herein provide technical improvements over alternative approaches such as the use of static routing tables, pull scheduling (e.g., the use of connection-specific timers that are not necessarily synchronized), etc. For example, the Linux kernel TCP stack uses a "pull" model to reduce recovery time for retransmitted packets. See: https: / / groups.google.com / g / bbr-dev / c / d14XAcuTBxo / m / Pwui1o3MBAAJ.

[0012] The push-based data packet communication system described herein is adapted to provide a router that can accommodate connection (link) aggregation for coarse-grained packet-based data distribution, enabled by real-time measurements of connection characteristics associated with mixed connections to remote peers.

[0013] As described in various embodiments herein, a multi-path connection aggregation system (alternatively referred to as a mixed connection system) may be modeled as a three-stage data pipeline having a first input queuing and scheduling stage, a second connection characteristic monitoring stage (e.g., providing congestion control and packet buffering and encapsulating pacing instructions), and a third transmission stage for conveying packets to a network interface (e.g., in accordance with the pacing instructions provided in the second stage).

[0014] The push-based data packet communication system includes a scheduler configured to queue data packets for transmission in response to monitored communication events (also called trigger events). Exemplary trigger events (other than timers) include ACK / NACK of a previously transmitted WAN packet, arrival of new data in an input queue, lack of feedback for a given amount of time, etc. The scheduler allocates the data packets to various networks based on the monitored characteristics at the time of the trigger event.

[0015] The monitored characteristics of an individual connection are a subset of the operational characteristics of the connection and may include, but are not limited to, estimated throughput, measured and / or expected latency, measured packet loss, and other characteristics that may be derived from measurements or expectations. In a clear example, the operational characteristics may include estimated throughput, measured and / or expected latency, measured packet loss, and other characteristics that may be derived from measurements or expectations, and the monitored characteristics may be limited to estimated throughput.

[0016] In contrast to the pull scheduler, the push scheduler evaluates the monitored characteristics of each of the multiple networks at the time of the monitored communication event and assigns the data packets to the connection that has available transmission capacity. In an exemplary embodiment, the push scheduler takes into account the data packet (application) requirements, the network order preferences configured by the administrator, the monitored characteristics of the connection (other than the transmission capacity) in addition to the available transmission capacity. The push scheduler continues to assign data packets to an available network when there is available transmission capacity associated with the network (the available transmission capacity is continuously updated to reflect the assigned data packets) and there are data packets available for transmission. In this way, the push scheduler may utilize the available transmission capacity of the multiple networks to a greater extent compared to the pull scheduler.

[0017] One aspect of the push scheduler is how the push scheduler is configured to computationally determine or establish a sorting order for controlling the routing of data packets to various network connections. Factors that influence the determination of the sorting order include keeping the sorting order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (splitting increases the likelihood of self-induced out-of-order and jitter events) and taking into account both measured characteristics of the connections and unmeasured external directives.

[0018] The measured characteristics may include packet loss during three separate sliding windows (500ms, 3s, 30s) (further summarized as the worst value during the three windows), current network round trip time (RTT), bandwidth and round trip propagation time (BBRv1) parameters, RtProp (network propagation delay), and BtlBw (throughput estimate). External directives may include priority (PR) rules among others, but may also include flow-specific rules / requirements.

[0019] In some embodiments, the measured properties of BBRv1 differ from the design of these properties in the original BBRv1 IETF draft. For example, to allow for embodiments that may run on systems with low runtime consistency, the size of the sliding window used to calculate BtlBw may vary, either statically or dynamically. Also, for similar runtime consistency reasons, the input parameters used to calculate RtProp may be slightly modified.

[0020] A specific improvement based on improving the networking routing order to control the use of network interfaces is described in some embodiments that utilizes a composite characteristic of round trip time (RTT) and packet loss to predict throughput based on the Mathis Equation to establish an upper bound on TCP throughput given RTT, packet loss, and MSS. The Mathis factor can be utilized to modify the sorting order for routing networked packets.

[0021] In some embodiments, over a period of time, push scheduling can also be configured to implicitly induce a sorting order along one or more implicit sorting dimensions. The implicit sorting dimensions force connections that experience periodic bad events (e.g., latency spikes due to congestion) to bubble to the bottom of the list. Connections with consistent behavior naturally bubble to the top and remain near the top.

[0022] Pull scheduling does not allow for increased utilization of network capacity because pulls occur based on connection-specific timers that are triggered independently of each other. The pull scheduler system includes a shared pull scheduler and timers associated with each network of a plurality of available networks, each timer having a wake-up value. When a timer wakes up to pull a packet from the pull scheduler, the decision as to which packet is scheduled is made based solely on the monitored characteristics of the current connection, since the potential wake-up times of other connections are unknown. The resulting packet scheduling decision is locally optimal for each individual connection at the time of the pull event, but may result in a globally suboptimal decision where a packet scheduled at an earlier pull event on one connection would have been scheduled more optimally on a separate connection where a pull request was imminent. The pull scheduler is configured to pull an amount of packets from the input queue based on the estimated bit rate of the corresponding network and the duration between each timer wake-up (e.g., 1 / 50 Hz, or 20 ms).

[0023] A technical challenge associated with a push scheduler is to ensure that data packets belonging to a particular data flow (e.g., packets containing audio files, video, etc.) are transmitted over a single network or a set of relatively stable networks sharing similar monitored characteristics that can meet the requirements of the flow. For example, transmitting audio packets belonging to a single data flow across two networks with different latencies requires a sequencer (i.e., hardware or software responsible for reassembling packets after transmission) to buffer and reorder the audio packets, possibly adding latency or failing to deliver data packets in the original order, resulting in the application perceiving a high-latency, inconsistent and poor networking encounter. A push scheduler that groups connections with similar monitored characteristics provides more direct control to assign data packets to the connection that best meets the flow requirements.

[0024] The proposed push scheduler is configured in some embodiments to order a plurality of networks based on a comparison of a monitored characteristic (e.g., round trip time, bottleneck bandwidth, etc.) to a reference range of network operating characteristics (e.g., a transmission reference range), where the range is defined to be insensitive to minor variations in the measured operating characteristic. The use of ranges allows the push scheduler to remain responsive to changes in the monitored characteristics of the networks while building in a degree of insensitivity to smaller variations in the monitored characteristics.

[0025] The proposed push scheduler may be configured to allocate data packets to multiple networks based on an ordering of the data packets, and to facilitate allocating data packets belonging to a particular flow to a single or relatively stable set of networks based on monitored characteristics of the networks. In an exemplary embodiment, the push scheduler stores the ordering and continues to transmit packets according to the ordering until the ordering is updated as a result of new or updated monitored characteristics.

[0026] The monitored characteristics may include a capacity factor, which is an aggregate value defined at least in part by dividing the bottleneck bandwidth by the round-trip propagation time. The capacity factor may be indicative of connections achieving higher bandwidth per millisecond of round-trip propagation time, which is indicative of an efficient network. Furthermore, the capacity factor may be determined by extrapolating the bottleneck bandwidth to the nearest order of magnitude (e.g., 1, 10, 100, 1000, 10,000, etc.) or an integer multiple thereof (e.g., 2, 3...20, 30, etc.), such that the capacity factor responds only to relatively large changes in the measured bottleneck bandwidth value. Similarly, the capacity factor may be determined by using a range of round-trip propagation times associated with the measured round-trip propagation time to reduce the sensitivity of the capacity factor to variations in the measured round-trip propagation time.

[0027] In an example embodiment, the proposed push scheduler may further order the multiple networks based on user input criteria. For example, the ordering may be based, at least in part, on a user having assigned a priority to each of the available networks, allowing the user to provide preferences for the order in which networks should be used based on external factors such as cost, information about future reliability or availability that cannot be measured or predicted, etc.

[0028] In response to the technical challenge of utilizing a larger amount of existing network transmission capacity, the push scheduler may be configured to identify and / or utilize transmission requirements associated with data packets queued for transmission by using a flow classification engine (e.g., the system may receive criteria that may be used to identify a particular data flow type and utilize predefined transmission requirements associated with the identified data flow type). By identifying or utilizing the transmission requirements of the data packets and comparing these transmission requirements with monitored characteristics of a plurality of networks, the push scheduler may be able to assign the data packets to networks that match the required transmission requirements in order to maximize the amount of data packets transmitted over networks with suitable monitored characteristics. It may still be possible to transmit packets over networks whose monitored characteristics do not match the transmission requirements, but this may result in applications interpreting "delayed" or out-of-order delivery of packets as network degradation, thereby generating packets that reduce the quality of the attempted service. Exemplary transmission requirements may include required latency throughout, etc.

[0029] In an exemplary embodiment, the push scheduler may use a flow classification engine packet into flows where each flow has a set of assigned transmission requirements. In another embodiment, each flow inherently has transmission requirements that form the characteristics of the flow. This flow mapping is based on one or more elements in the header of the data packet or associated with the data packet. These flow transmission requirements may then be used to assign the packet to one of multiple connections with comparable monitored characteristics. In an example where a data packet is classified as a VPN data flow, it has an inherent requirement that the data be delivered in order, and these requirements may include a flow transmission requirement that is sent over a connection with a certain minimum reliability.

[0030] For example, a TCP / IP header may be identified as a TCP / IP 5-tuple (e.g., source and destination IP addresses, source and destination port numbers, and protocol) and as a result determined to be a Virtual Private Network (VPN) data flow, which is a data flow that is more sensitive to ordering. As a result, the scheduler may receive a transmission requirement from the flow classification engine that requires that all of the data packets of the VPN data flow be delivered in order, regardless of the presence of other indicators, such as DSCP tags, on the individual packets that may contradict this requirement. For example, some of the packets may have DSCP tags that require low latency, while other packets may require high throughput. The scheduler may map (assign) packets to a connection with monitored characteristics that meet the particular transmission requirements of the DSCP tags, as long as they do not violate the ordering requirements of the encapsulating VPN data flow.

[0031] According to another example, the push scheduler may use a flow classification engine to map packets to flows whose transmission requirements prioritize low latency over reliability or throughput (e.g., a source may indicate a type of data, such as a live video stream, that may be sensitive to increased latency).

[0032] According to further exemplary embodiments, the push scheduler may assign priority transmission requirements to individual data packets independent of the transmission requirements derived from the flow classification. For example, the push scheduler may be configured to assign the highest level of priority to packets that are queued for retransmission (i.e., packets that have previously been transmitted unsuccessfully).

[0033] A technical challenge associated with a push scheduler implementing data flow or data packet priority transmission requirements is that non-urgent data packets may be continually moved out of the transmission queue, resulting in "starvation" of non-urgent packets in the system.

[0034] The push scheduler may be configured to pace the rate of packet transmission in response to non-urgent packet starvation by allocating a nominal wall clock time for transmission to each data packet. In an exemplary embodiment, packets are allocated wall clock time based on an estimated bottleneck bandwidth and the size of the data packet to prevent congestion.

[0035] The output queues of the network to which data packets are assigned at a pipeline stage may be configured to transmit the data packets according to the data packets' assigned nominal wall clock times without moving non-urgent packets in favor of priority packets. For example, at time zero, a non-urgent data packet may be assigned a nominal wall clock time for transmission at time 3. The urgent data packet may be scheduled for transmission at time 3 based on the amount of available transmission capacity in the network, which is reduced to accommodate the non-urgent data packet.

[0036] Continuing with the example, the push scheduler may be configured to queue non-urgent data packets for later transmission that has a later nominal wall clock time. For example, at time 0, if there is available network capacity at time 2, the push scheduler may transmit a non-urgent data packet with a time stamp of time 3 for transmission at time 2. The non-urgent data packet may then be rescheduled for transmission at time 3 in response to receiving a packet with a more urgent nominal wall clock time.

[0037] A technical challenge associated with a push scheduler is that pushing data packets based on estimated available network transmission capacity can result in the system losing robustness in the event of network failure or an unacceptable amount of network congestion.

[0038] The push-based scheduler may be configured to monitor network operating characteristics associated with each available network to determine whether the monitored characteristics satisfy a congestion threshold. In an exemplary embodiment, the congestion threshold is established from a baseline profile associated with each network. For example, the minimum latency measured for packets transmitted over a particular network may form a baseline value for the network, and subsequent transmissions along that network may be compared to the baseline value. In another exemplary embodiment, the threshold may be based, at least in part, on a baseline of packet round trip times. In other embodiments, the baseline may be calculated using more complex statistical techniques than taking the minimum latency.

[0039] To increase the robustness of the push scheduler, the scheduler may be configured to reduce the amount of transmission capacity defined as available for transmission in response to detecting a network failure or an unacceptable amount of network congestion, or to update the available transmission capacity other than following each monitored communication event. In an exemplary embodiment, the transmission capacity of the network may be updated only in response to receiving an acknowledgement that a previously transmitted data packet has been received (e.g., in response to receiving an ACK). According to some embodiments, for example, the defined transmission capacity is reduced to a theoretical transmission capacity that does not include a dead queue of packets in a buffer associated with the network. In a further exemplary embodiment, the defined transmission capacity is reduced to be equal to the amount of in-flight data packets throughout the particular network.

[0040] In some exemplary embodiments, one of the networks operates according to a bandwidth-on-demand (BOD) principle, where a network administrator and / or automated network monitoring agents allocate available bandwidth for transmission to transmitters based on previous transmissions or previous requests for bandwidth. This allocation may increase or decrease the allocated bandwidth depending on the administrator's or agent's prediction of future bandwidth usage. During this bandwidth allocation process, packet loss and / or latency may temporarily increase, requiring the scheduler to make more intelligent decisions for allocating packets to connections. For example, probing packets used to request more bandwidth may be marked with a DSCP tag indicating that the probing packets are of lower priority and are therefore candidates to be dropped first if necessary.

[0041] A technical challenge associated with a push scheduler pushing data packets to a bandwidth-on-demand network is that the bandwidth-on-demand network updates the bandwidth available for transmission infrequently, making it difficult to utilize the available capacity. For example, a bandwidth-on-demand network may have available capacity, but as a result of allocation delays, this capacity may go unused. Alternatively, if the push scheduler does not consistently use enough capacity on the network, an administrator / agent may choose to reduce the allocated capacity, potentially resulting in significant changes to the monitored characteristics of the network.

[0042] The push scheduler may be configured to define a transmission capacity of the bandwidth-on-demand network as being greater than the received allocated transmission capacity, store the excess transmission capacity in a buffer associated with the bandwidth-on-demand network, and over-advertise a needed amount of bandwidth to an administrator of the bandwidth-on-demand network. As a result of over-advertising a needed bandwidth, the bandwidth-on-demand network may begin to allocate a larger amount of unused bandwidth to the push scheduler. Furthermore, as a result of defining the transmission capacity as being greater than the received allocated transmission capacity, the push scheduler may ensure that data packets are available for transmission to take advantage of any increases in the received allocated transmission capacity that may occur in the future.

[0043] A further technical challenge associated with pull scheduler systems is that data packets with specified requirements are difficult to fairly allocate for transmission. For example, if a particular type of data packet requires a connection (e.g., a video feed) with a particular latency, the pull scheduler system may bypass or delay the packet until an available network that meets the latency requirement wakes up. In this way, depending on when the data packet enters the input queue, latency-sensitive data packets may be delayed, further exacerbating the latency problem.

[0044] Rate-based congestion control typically adjusts the rate based on delay and loss. Congestion detection needs to be done with a certain time lag to avoid overreacting to spurious congestion events such as delay spikes. Despite the fact that there are more than one congestion indicator, the result is that there is still only one mechanism for adjusting the sending rate. This makes it difficult to simultaneously achieve the goals of high throughput and rapid response to congestion. [Brief description of the drawings]

[0045] In the figures, embodiments are shown by way of example, and it is to be expressly understood that the description and drawings are for illustrative purposes only and as an aid to understanding only.

[0046] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1A] FIG. 1 is a schematic diagram of a pull communication system. [Figure 1B] 1 is a schematic diagram of a push communication system, according to some exemplary embodiments. [Figure 1C] 1 is a flowchart of a method for transmitting data packets to push packets to multiple networks based on availability, according to some example embodiments. [Figure 1D] 11 is a flowchart of another method for transmitting data packets for ordering connections based on connection metadata, according to some example embodiments. [Figure 1E] 5 is a flowchart of a further method for transmitting data packets based on a selected order of a network for allocating packets to a network, according to some example embodiments. [Figure 1F] 1 is a process flow illustrating dimensions used to sort network connections for controlling routing of data packets, according to some example embodiments. [Diagram 2] FIG. 2 is a schematic diagram of scheduling data packets of different priority levels according to some example embodiments; [Diagram 3] 1C is a series of schematic diagrams of the push communication system of FIG. 1B for scheduling data packets of different priority levels according to some exemplary embodiments. [Figure 4A] 1C is a schematic diagram of connection selection with the push communication system of FIG. 1B according to some exemplary embodiments. [Figure 4B]1C is a schematic diagram of data packet selection using the push communication system of FIG. 1B according to some exemplary embodiments. [Figure 4C] FIG. 1C is a schematic diagram of data packet allocation using the push communication system of FIG. 1B according to some exemplary embodiments. [Figure 4D] 1C is a schematic diagram of another data packet allocation using the push communication system of FIG. 1B according to some exemplary embodiments. [Figure 5A] FIG. 1C is a schematic diagram of the push communication system of FIG. 1B scheduling two independent data flows for transmission, according to some exemplary embodiments. [Figure 5B] 1C is a schematic diagram of the push communication system of FIG. 1B scheduling two dependent data flows for transmission according to some exemplary embodiments. [Figure 6A] 4A-4C show corresponding screenshots of an interface for managing data packet transmissions according to an exemplary embodiment. [Figure 6B] 4A-4C show corresponding screenshots of an interface for managing data packet transmissions according to an exemplary embodiment. [Figure 6C] 4A-4C show corresponding screenshots of an interface for managing data packet transmissions according to an exemplary embodiment. [Figure 6D] 4A-4C show corresponding screenshots of an interface for managing data packet transmissions according to an exemplary embodiment. [Figure 6E] 4A-4C show corresponding screenshots of an interface for managing data packet transmissions according to an exemplary embodiment. [Figure 7] 1 is a diagram of a data packet transmission according to some exemplary embodiments. [Figure 8A] 1A-1D are two diagrams of unidirectional data packet transmissions according to some exemplary embodiments. [Figure 8B]5A-5C are two diagrams of data packet transmissions in response to acknowledgments, according to some exemplary embodiments. [Figure 8C] 5A-5C are two further diagrams of data packet transmissions without requiring an acknowledgment for sending further data packets, according to some demonstrative embodiments. [Figure 9A] FIG. 2 is a diagram of in-flight goodput data, according to some exemplary embodiments. [Figure 9B] FIG. 2 is another diagram of in-flight goodput data, according to some example embodiments. [Figure 9C] FIG. 13 is a further illustration of in-flight goodput data, according to some example embodiments. [Figure 10A] 5A-5C are a series of diagrams illustrating data packet transmissions in a connection experiencing congestion, in accordance with some demonstrative embodiments. [Figure 10B] 5A-5C are a series of diagrams illustrating data packet transmissions after adjustment of transmission capacity in response to a connection experiencing congestion, according to some demonstrative embodiments. [Figure 11] 1 is an exemplary schematic diagram of a computing device according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0047] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A system, method and device for transmitting data packets where multiple connections are available for transmission, alternatively referred to as a mixed router or mixed system, is described.

[0048] Existing solutions for transmitting data packets over multiple available connections for transmission are typically implemented as a "pull" type architecture, where data packets are transmitted in response to an event generated or defined by a device or system, such as an internal timer associated with the multiple available connections.

[0049] The internal timer may be responsive to various connection parameters (eg, a timer duration configured to reduce recovery time of a retransmitted packet) but is not responsive to the connection parameters of other available connections.

[0050] As a result of the independence of timer events, pull-based systems may suffer from under-utilization (e.g., timer values ​​that are too long resulting in idle connections) or over-utilization (e.g., data packets may be assigned to unreliable connections despite insufficient performance), which may result in greater latency in delivery of data packets, reduced throughput, unreliable data packet delivery, more expensive data packet delivery, etc.

[0051] A method, system, and device for transmitting data packets according to a push-based architecture for transmitting data packets over a plurality of available connections is proposed herein. The disclosed method, system, and device include a scheduler configured to obtain a subset of operational characteristics (monitored characteristics) of each of a plurality of connections in response to a monitored communication event, and iteratively allocate data packets available for transmission to any of the connections determined to have a transmission capacity based on the monitored characteristics of these connections. In an exemplary embodiment, other monitored characteristics of the network may be used in conjunction with the transmission requirements associated with the data packets to allocate the data packets to a connection that matches the transmission requirements of the data packets.

[0052] Scheduling data packets in a push-based architecture where multiple connections are available has not been used. A typical multi-path system (alternatively referred to as a mixed connection system) assigns packets and flows to connections on a static basis because it does not consider the flow requirements or operational characteristics of the connections. For example, a multi-homed Internet Protocol (IP) router assigns packets to connections purely based on the destination IP address in the packet header matching against the router's destination routing table. It is easier to implement a pull-based architecture when connection assignments do not change frequently. For example, there is no need to handle the case where a connection is assigned a large queue of packets by the push scheduler, but then has to be recalled or reallocated because the connection goes offline.

[0053] The proposed method, system and device receives a data flow comprising one or more data packets from an input network through an input connection.

[0054] The proposed method, system, and device include a three-stage data pipeline that transmits data packets over a mixed set of connections to a remote peer. New data destined for multiple connections is passed to a first pipeline stage, where the new data is queued and then passed to a second pipeline stage associated with each connection of the mixed connections. The second pipeline stage compiles statistics about the connections underlying the second pipeline stage and passes the packets to a third pipeline stage. The second stage may provide congestion control functionality and packet buffering to ensure that the packets are available for the next pipeline stage. The third stage then writes the packets to a network interface, possibly according to pacing or timing hints provided by the second stage.

[0055] Another embodiment may have a single second stage instance that then provides the packet to an instance of a third stage, each of which manages the transmission of the packet over a set of network interfaces that share a physical interface card. There may be other embodiments with different numbers of instances of each of the pipeline stages.

[0056] FIG. 1A is a schematic diagram of a pull communication system 100A, according to some exemplary embodiments.

[0057] When each connection timer (e.g., timers 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively) wakes up, the connection timer invokes the pull scheduler 130 to request data packets for transmission from the data packet input queue 102. The data packets may be requested in bursts, with the burst size nominally reflecting 20 ms (1 / 50 Hz) worth of bytes at the estimated bit rate of each timer's corresponding connection. A limitation associated with such pull timers is that the number of bytes requested at each timer's wakeup does not reflect the complete congestion window (e.g., representing the "capacity") of the connection.

[0058] Since each timer fires independently, the pull scheduler 130 does not have visibility into the overall transmission state to make a globally optimal scheduling decision. For example, the connection 110A may be a terrestrial broadband connection, and the connection 110B may be a bandwidth-on-demand (BoD) satellite connection. The connection 110A may be preferred over the connection 110B as a result of cost, latency, etc., but when the connection 110B timer 108B wakes up to invoke the pull scheduler 130, the pull scheduler 130 does not have visibility into the next wake-up time of the connection 110A timer 108A (since the timers 108A fire independently). The pull scheduler 130 does not have a basis for determining whether the connection 110A can process all of the data packets in the input queue 102 or whether it should allocate more data packets to the connection 110B to complete the transmission.

[0059] When connection 110B is available, pull scheduler 130 may not have a means to determine which connection should be used to transmit which of the data packets available for transmission, because only pull scheduler 130 has visibility into the timer.

[0060] The data packets are transmitted across the connection and then received by a receiver 116 via connection 114. The receiver 116 includes a sequencer 118 for assembling the transmitted data packets and may further transmit the assembled data packets via a local area network (LAN) 120.

[0061] In contrast to pull-based architectures, the proposed system may be able to schedule data packets to specific connections to increase effective connection utilization.

[0062] 1B is a schematic diagram of a push communication system 100B according to some exemplary embodiments. In the exemplary embodiment, the system 100B is designed to aggregate connection types that use multiplexing techniques to provide multiple access (MA) to a shared communication channel. For example, orthogonal codes assigned to each node in a Code Division Multiple Access (CDMA) network allow the nodes to share a set of frequencies and transmit simultaneously without interfering with each other.

[0063] A characteristic of connection types that use these multiplexing techniques is that reallocation of available capacity to nodes in the network is rapid, typically on the order of tens to hundreds of milliseconds. In an exemplary embodiment, system 100B is designed with this characteristic in mind, and reallocates traffic among available WAN connections on this time scale.

[0064] In contrast, connections using Demand Assigned Multiple Access (DAMA) techniques, also known as Bandwidth on Demand (BoD), share a communications channel by having nodes negotiate access to the channel for a fixed period of time. Once allocated, the nodes dedicate the negotiated portion. Thus, sharing of the available capacity occurs sequentially through the nodes using the negotiated portion of the channel.

[0065] With BoD connections, this negotiation and subsequent reallocation of capacity between nodes typically takes on the order of seconds (mid single digits to low double digits) to complete. In an exemplary embodiment, system 100B would require modifications (described herein) to effectively use these types of connections.

[0066] Long reallocation times (with associated packet loss and / or increased latency) are interpreted as congestion in current implementations, and typical congestion control methods reduce usage on connections where congestion is detected to avoid overloading these connections. However, BoD connections actually require the opposite behavior (at least temporarily) in order to request, and potentially obtain, an increased allocation of capacity.

[0067] The system 100B includes an input queue 102 that receives a data flow including one or more data packets from an input network for transmission. Examples of a data flow may be a file transfer, a video file transfer, an audio file transfer, a Voice Over Internet Protocol (VoIP) call, or data packets associated with a Virtual Private Network (VPN).

[0068] Input queue 102 is also responsible for applying a dropping policy to packets when a scheduler (as described herein) cannot service input queue 102 at the same rate that new packets arrive from LAN-side clients, which typically occurs when LAN clients are transmitting at a rate higher than the aggregate WAN transmission capacity.

[0069] System 100B includes three stages, alternatively referred to as pipelines or pipeline stages, for transmitting data packets to remote peers over a mixed set of connections according to a push-based architecture.

[0070] The first stage 104 includes a flow classification engine 104A, a scheduler 104B (hereinafter alternatively referred to as a push scheduler 104), and receives data packets from the input queue 102 and queues the data packets for passing to the second stage 106. In an exemplary embodiment, the first stage 104 includes all of the input queue 102, the flow classification engine 104A, and the scheduler 104B.

[0071] The input queues 102 are responsible for buffering packets received from clients on the sender's LAN side and classifying the data packets into flows (alternatively referred to as mapping data packets to flows), possibly with the aid of or in cooperation with the flow classification engine 104A. The input queues 102 ensure that the data flows are fairly serviced by the scheduler of the first stage 104.

[0072] In an exemplary embodiment, the first stage 104, or the assisting flow classification engine 104A of the first stage 104, monitors or is notified of changes in user-configured settings associated with the allocation of data packets to connections. The user-configured settings include information on how connections should be assigned to priority levels and when those priority levels should be active, as well as user-configurable flow classification rules that may cause packets belonging to a flow to prefer connections with certain monitored characteristics (e.g., packets belonging to a video stream class may prefer connections with low latency, whereas packets belonging to a file download class may prefer connections with lower monetary cost).

[0073] The first stage 104 is periodically notified of current metadata (e.g., operational characteristics) for each of a number of connection monitoring instances (as described herein) corresponding to the WAN connection, including mixed links. This metadata is collected by the monitoring instances 106 and may include the estimated bottleneck bandwidth of the connection, the estimated minimum round trip time, the recent actual round trip time, the current congestion window for the connection, and the receive / loss status for packets previously sent to the next stage. Using this metadata, the first stage 104 may also extend the metadata to track an estimate of the bytes that the first stage 104 has scheduled as in-flight on this connection, and further break down that total into new bytes, retransmitted bytes, or dummy / probing bytes. All of the metadata collected and tracked by the first stage 104 is referred to as the monitored characteristics.

[0074] Using all available monitored characteristics, the first stage 104 will then determine whether to send the packet to the next stage, and if so, which connection should receive the packet. The set of packets provided to a connection monitor instance may include duplicates of packets previously sent (either to the same connection monitor instance or to a different connection monitor instance). Alternatively, the scheduler may determine that a given connection monitor instance is not currently eligible to receive the packet, even if the connection at that instance currently has capacity. The allocation of packets to specific stage 3 instances is discussed elsewhere in this document.

[0075] The second stage 106 includes a connection monitor instance for each of the multiple available connections (e.g., connection monitor instances 106A, 106B, and 106N associated with connections 110A, 110B, and 110N, respectively). The connection monitor instances of the second stage 106 monitor the associated connections and determine or obtain statistics associated with the performance (e.g., latency) of the connections, referred to as operational characteristics. The second stage 106 may provide congestion control functionality and packet buffering to ensure that packets are available for the next stage. The second stage 106 passes the packets to the third stage 108, which includes output queues for each connection (e.g., output queues 108A, 108B, and 108N). The third stage 108 then writes the packets to the network interface, possibly according to pacing or timing hints provided by the second stage 106.

[0076] In some embodiments, the second stage 106 may implement one or more R2RI protocols (e.g., PPPoE (RFC5578), R2CP, DLEP (RFC8175)) to better support the bandwidth allocation process for BoD connections.

[0077] The second stage 106 is responsible for receiving packets from the first stage 104, queuing these packets locally, and providing groups of packets as bursts to the next pipeline stage. The second stage 106 also tracks metadata about its associated connections based on acknowledgements received from peers or recipients.

[0078] The purpose of the queue in the second stage 106 is to ensure that there is always a packet available for the next pipeline stage to write to the network device for transmission. Some embodiments may include priority metadata as part of each packet's information, so that new packets arriving from the first stage 104 may be reordered within the queue to provide preference to higher priority packets.

[0079] To ensure that the second stage 106 has buffered enough packets so that they can be provided to the third stage 108, the second stage 106 may provide the modified metadata to the first stage 104. For example, according to some example embodiments, the second stage 106 may choose to report (i.e., "over-advertise") a larger congestion window so that the first stage 104 provides additional packets that can be buffered at the second stage 106 for when the third stage 108 finishes writing the actual congestion window packets.

[0080] The second stage 106 may also provide modified metadata to the first stage 104 to facilitate changing the allocation of capacity on the BoD connection (i.e., to trigger the BoD connection to allocate higher capacity, causing the first stage 104 to schedule packets on this connection as if the first stage 104 had higher capacity). This modified metadata may include information that breaks down the higher capacity into different requested types / priorities. For example, the higher incremental capacity used to probe the BoD connection is typically made up of redundant / dummy packets since the probe packets are more likely to be dropped by the BoD connection.

[0081] The advertising or over-advertising of capacity from the second stage 106 to the first stage 104 does not necessarily result in immediate probing for higher incremental capacity. The first stage 104 ultimately still makes the decision on how packets are allocated to connections. An embodiment may wait until the CWND on all non-BoD connections is fully consumed (or close to being fully consumed) before instructing the second stage 106 to begin the DAMA reallocation process.

[0082] The queue in the second stage 106 also allows for fast notification of failure to previous stages if something happens to the underlying network connection. Packets provided to the third stage 108 and written to the network must be assumed to be in transit even if the network interface subsequently reports down, meaning that a timeout must occur before those packets can be marked as lost. However, packets still queued in the second stage 106 can be immediately marked as lost once the second stage 106 is notified that the connection is no longer available, which makes them immediately eligible for prioritized retransmission.

[0083] Before being passed to the third stage 108, some embodiments may attempt to explicitly pace the rate of packet transmission by assigning each packet a nominal timestamp that indicates when the next stage should attempt to send that packet over the connection. Such a timestamp may be determined based on the estimated bottleneck bandwidth of the connection (e.g., the connection to which the packet is assigned) and the packet size to prevent network congestion at or before the network bottleneck link.

[0084] FIG. 2 illustrates, at 200, an example embodiment in which the second stage 106 supports prioritized packets. In FIG. 2, the head of the queue is shown on the right and the tail of the queue is shown on the left. In the upper part of FIG. 2 (e.g., first step), the first stage 104 provides a new list of prioritized packets to the connection monitoring instance of the second stage 106. The lower part of FIG. 2 (e.g., second step) shows these new packets being added to the priority queue in priority order, such that all priority 1 packets are dequeued before any priority 2 packets are dequeued. Packets of the same priority are kept in sequence order.

[0085] In an exemplary embodiment not shown, the second stage 106 may have a single connection monitoring instance that then provides packets to instances of a third stage, each of which manages the transmission of packets on a set of network interfaces that share a physical interface card. Other embodiments may exist with different numbers of instances of each of the pipeline stages.

[0086] Feedback in the form of metadata or callbacks may be sent from a stage to a previous stage, or a stage may receive notifications from different parts of the system. Any of these actions may trigger (i.e., push) a stage to send a packet to the next stage.

[0087] The third stage 108 receives packets from the second stage 106 via one or more connection transmission instances (e.g., connection transmission instances 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively) and writes these packets to the network 112. In some embodiments, all connection monitoring instances of the second stage 106 may provide packets to a single connection transmission instance, or each connection monitoring instance may be associated with a connection transmission instance.

[0088] Some embodiments of the third stage 108 may use a queuing mechanism to allow previous stages to provide hints as to when to send a packet. For example, the hint may come in the form of a "nominal transmission time" associated with the packet as metadata. If the next packet to be sent has a nominal transmission time in the future, the third stage 108 will wait until that time before sending the packet. Urgent packets may be flagged with an immediate timestamp that may cause the urgent packet to be sent ahead of other queued packets. To prevent starvation of non-urgent packets, some embodiments may queue urgent packets after non-urgent packets that have nominal transmission times in the past.

[0089] Some embodiments may include a callback function as part of the packet metadata that may be used to trigger a notification to the second stage 106 to indicate that a packet has been sent. Such a notification may be used to trigger a new batch of packets to be pushed from the second stage 106 to the third stage 108.

[0090] 3 shows at 300 the progression over time of an embodiment including a "send time" hint (nominal send time) that requests that packets provided by the second stage 106 not be sent before a specified time. Packets with a "send time" of 0 should be sent as soon as possible. FIG. 3 illustrates examples for all of these cases (e.g., the use of past nominal send times for urgent packets and how the system avoids starvation of non-urgent packets).

[0091] In the state depicted at t=1, the packet queue of the third stage 108 instance contains four packets, three of which are currently eligible for transmission, and the second stage 106 instance is about to provide four new packets, the first of which has sequence 5 (sequence order 5) and should be transmitted immediately.

[0092] At the state depicted at t=2, a new packet has been added to the priority queue of the third stage 108. Note that this embodiment chose to queue the packet with sequence 5 for transmission after the packet having sequence 2 because sequence 2 was already eligible for transmission at the time sequence 5 was queued. In this manner, freezing the packet order at the head of the packet queue prevents starvation, where a constant stream of new urgent packets to be sent immediately may prevent non-immediate packets from being sent even though they may have become eligible for transmission long before the most recent immediately-sent urgent packet arrived.

[0093] In the state depicted at t=3, the third stage 108 has dequeued the first four packets from the output queue and provided these packets to the network interface. Because no packets are currently eligible to be sent until time t=4, the third stage 108 instance would need to wait to send more packets until time advances to t=4 or until a new packet arrives that should be sent immediately or at time t=3, even if there is currently write capacity on the network interface.

[0094] Packets may be passed from one stage to the next in order as each stage determines is optimal.

[0095] 1A, data packets are transmitted across a connection and then received by a receiver 116 via connection 114. The receiver 116 includes a sequencer 118 for assembling the transmitted data packets and may further transmit the assembled data packets via a local area network (LAN) 120.

[0096] In contrast to system 100A, push communication system 100B does not include an independent timer associated with each connection, but instead includes a push scheduler 104 in which scheduling of packets onto connections is performed at a centralized location, has access to monitored characteristics of each connection, and is activated in response to one or more monitored events occurring other than based on a timer. For example, the one or more monitored communication events may include an acknowledgement / negative acknowledgement (ACK / NACK) from a recipient of a previously transmitted data packet, or the arrival of a new data packet at input queue 102, etc.

[0097] In an exemplary embodiment, the entire connection (congestion window) CWND can potentially be consumed or allocated by push scheduler 104, not just the 20 ms of the packet being transmitted.

[0098] With access to the monitored characteristics of each connection, the push scheduler 104 can be configured to optimize packet scheduling decisions. For example, a data flow with specific transmission requirements can be preferentially assigned to a connection that can meet the requirements associated with the data flow, rather than being assigned based in large part on the wake order of a timer. In one illustrative example, a first data flow requires transmission with a latency of less than 50 ms.

[0099] Connections 110A, 110B, and 110N have latencies of 5 ms, 45 ms, and 80 ms, respectively. Using pull scheduler 130, if the 50 Hz timer for connection 110B (45 ms latency) were to wake up first, it would be able to schedule packets for this first data flow onto connection 110B. Using push scheduler 104, the data flow can be preferentially scheduled on connection 110A (5 ms latency) until the CWND of connection 110A is completely consumed before overflowing onto connection 110B (45 ms latency).

[0100] To enable bidirectional communication, a single instance of a multipath endpoint may include all of the components described in Figure 1B, specifically, one or more instances of sender 100 to support upstream communication to other multipath endpoints, and one or more instances of receiver 116 to support downstream communication from other multipath endpoints.

[0101] According to some embodiments, for example, one or more of the available connections are BoD connections, and the push scheduler 104 can allocate more data to the BoD connections to request more capacity from the network based on criteria such as:

[0102] While the CWND of connection 110A is completely consumed (or approaching complete consumption), more packets are available in input queue 102 that are eligible to be transmitted on connection 110B (e.g., based on per-flow rules).

[0103] In an example embodiment, the push communication system 100B can enable more accurate / fair Quality of Service (QoS) and Active Queue Management (AQM) packet dropping rules applied to the input queues 102. Referring back to the previous example, a flow with a 50 ms latency requirement can never utilize an 80 ms connection. Since the push scheduler 104 has a consistent snapshot of the entire system, it can calculate the flow's partial usage and fair share of the total capacity excluding the 80 ms connection.

[0104] A potential advantage of the push-based scheduling approach is that a complete set of metadata about multiple connections, including mixed links, is available to the push scheduler 104 in order to optimally assign packets to target connections.

[0105] FIG. 1C is a flow chart of a method 110C for transmitting data packets using system 100B or other push communication systems according to some example embodiments.

[0106] At block 1130, the method 100C detects when the input queue contains an unallocated data packet available for transmission.

[0107] At block 1132, the method 100C includes allocating one or more of the unallocated data packets to one or more corresponding output queues of a plurality of networks having suitable monitored characteristics.

[0108] At block 1134, the method 100C includes updating the monitored characteristics of the corresponding plurality of networks to reflect the allocated one or more data packets.

[0109] At block 1136, the method 100C communicates the allocated data packets from the output queues corresponding to each network of the plurality of networks to a destination device.

[0110] In an exemplary embodiment, the method for transmitting data packets in system 100B includes a loop of the following steps each time push scheduler 104 is triggered by an external event. 1. Request a data packet from the input queue 102; 2. Assigning received packets to connections for transmission; 3. Repeat steps 1-2 until a) there are no more packets available in the input queue 102, or b) any of the active connections have no more available transmission capacity (e.g., as defined by a congestion window (CWND) metric).

[0111] The described method may be implemented such that the push scheduler 104 does not consider any data flow specific requirements. In such an embodiment, the system 100B allocates received packets based on the ordering of the connections.

[0112] The ordering can be based on monitored characteristics of the connections, which can include, for example, a subset of the operating characteristics estimated by a congestion control algorithm. In one embodiment, the BBRv1 algorithm is used, while other embodiments can include examples such as BBRv2, TCP Reno, TCP LEDBAT, proprietary implementations, etc., for handling specific connection types such as BoD. In the subsequent discussion of the operating characteristics specific to BBRv1, for illustrative purposes only, reference is made to a pipe analogy, where two endpoints are modeled as a series of interconnected pipes, where each pipe segment can have a different diameter and length.

[0113] The performance characteristic may be a measure of the minimum time required for a transmitted packet to reach a destination endpoint (e.g., a receiver) and be acknowledged (e.g., via an ACK message from the receiver to the sender). This duration is termed the "round trip propagation delay" (abbreviated as "RtProp") and is measured in units of time (e.g., milliseconds). In terms of the pipe analogy, this value represents the total length of all pipe segments from the sender to the receiver.

[0114] The operating characteristic may be the maximum rate at which data can travel between endpoints along the pipe segment with the smallest diameter. The smallest pipe diameter, referred to as the "bottleneck" segment, determines the maximum rate at which data can travel between the transmitter and receiver, also referred to as the "bottleneck bandwidth" (abbreviated as "BtlBw"). For example, a technique used to estimate BtlBw is described in an IETF draft: https: / / tools.ietf.org / html / draft-cheng-iccrg-delivery-rate-estimation-00.

[0115] Table 1 shows an exemplary set of WAN connections and characteristics of the WAN connections, an exemplary subset of the operational characteristics that are selected by scheduler 104 to be the monitored characteristics, and the resulting sort order. Table 1: Example WAN connections and sort order for WAN connections [Table 1]

[0116] This exemplary method sorts the connections across the following dimensions, moving on to the next dimension only if the current dimension compares equal across connections. 1. Increase the preference priority level 2. Increase RtProp 3. Decrease BtlBw 4. Increment the interface name.

[0117] Note how the result of the sorting operation keeps the order of connections relatively stable since the sorting dimensions are all parameters that are not expected to change frequently.

[0118] This stable sorting order is intentional, so that packets arriving at an input queue tend to be "sticky" to the same set of connections (those sorted at the top of the list), and will only "overflow" upwards to connections lower in the sorting order if the incoming requests exceed the total CWND of the higher sorted connections.

[0119] Note how the sorting criteria in this example is not flow-specific. For example, an administrator or end user may only want high priority application flows to be transmitted over the metered Cell 1 and Cell 0 connections (an example of a transmission requirement), meaning that those flows would want a resulting sort order that places Cell 1 and Cell 0 lower in the list. Similarly, there may be some flows that have a maximum latency limit, such as a VoIP call where the end user considers the maximum latency limit unusable for interactive conversations if the latency exceeds 150 ms. For these flows, it may be desirable to have a resulting sort order that uses RtProp as the primary sorting dimension. An embodiment of extending the sorting criteria to be flow-specific is discussed in the following paragraphs.

[0120] The scheduler is also responsible for determining the retransmission policy for packets. In an exemplary embodiment, the system 100B is configured such that the scheduler always marks a packet for retransmission if the packet is reported as missing by the receiver. While this is desirable for many flows, other flows may not want this at all (because the additional latency or packets potentially arriving out of order would make the packet unusable by the application), or may only want retransmission attempts within a certain period of time.

[0121] FIG. 1D is a flowchart of another method for transmitting data packets according to some demonstrative embodiments.

[0122] One embodiment of a simple scheduling loop is as follows: 1. Perform any accounting required to update connection metadata and determine the set of active connections. 2. Determine the preferred ordering of connections based on connection metadata (monitored characteristics). 3. While there are packets to send and there is available network capacity for the set of active connections: a. Get the next packet to be sent. b. Queue the packet to be sent on the first connection in the ordering that has available capacity. c. Update the connection metadata (the monitored characteristics) to reflect the scheduled packet. d.Repeat. 4. For each connection that has queued packets, send those packets to the next pipeline stage for that connection.

[0123] Such an embodiment treats all packets as equal for purposes of preferred ordering of connections, where packets can be considered to belong to the same class of flow (even though separate packets may logically belong to separate flows, from the scheduler's point of view there is no distinction between these flows for purposes of connection allocation).

[0124] FIG. 1E is a flowchart of a further method for transmitting data packets according to some example embodiments.

[0125] An embodiment of a scheduler that can distinguish between different classes of flows (i.e., ICMP echo requests / responses vs. video streams vs. file transfers), and even between individual flows within the same class (a video stream between host A and host B vs. a video stream between host A and host C), can be provided as follows. 1. In block 1102, perform any necessary accounting of connection metadata updates (monitored characteristics) and determine the set of active connections. 2. While the active connection set has capacity (block 1114), In block 1103, a selector is constructed based on the currently active set of connections. b. A selector is used to choose a packet for transmission over one of the available connections, in block 1105. If no packet has been selected, the loop is exited. c. In block 1107, determine an ordering of active connections based on packet flows and flow classifications in combination with connection metadata (monitored characteristics). d. In block 1110, queue the packet to be sent on the first connection in the ordering that has available capacity. e. In block 1112, update the connection metadata to reflect the scheduled packet. f.Repeat. 3. In block 1116, for each connection that has packets queued, send those packets to the next pipeline stage for that connection.

[0126] These two sample methods (shown in FIG. 1D and FIG. 1E) are similar, with the method shown in FIG. 1D being a simplified and optimized version of the more general method in FIG. 1E.

[0127] The first method can be transformed into the second method by defining a single flow class and a selector that matches all packets. With this in mind, we discuss the second method (FIG. 1E) in more detail, and the same discussion would be applicable to the first method (FIG. 1D), subject to the restrictions above.

[0128] The first step is to update any pending metadata accounting (monitored characteristics) to determine the set of active connections. In one embodiment, the set of active connections are connections that are not subject to unreliable connection backoff (i.e., connections that are being limited in use due to recent failures to reliably deliver packets).

[0129] Other embodiments may further restrict the set of active connections, for example, by eliminating connections whose metadata (monitored characteristics) do not match any of the currently defined flow class requirements (i.e., assuming that all packets must match one defined flow class, a connection may be considered inactive if it is never selected by any flow class; e.g., if all flow classes specify a maximum RTT of 500 ms, then any connection with an RTT>500 ms may be considered inactive).

[0130] A set of active connections has the capacity to send packets if at least one connection in the set has an available CWND (congestion window), meaning that the number of in-flight bytes currently tracked by the scheduler for this connection does not exceed the CWND determined for this connection. In one embodiment, the CWND for a connection may be the CWND reported by the next pipeline stage, or if throttling or priority routing is in effect, the scheduler may choose to reduce the reported CWND.

[0131] In one embodiment, a connection that is throttled to a particular rate reduces the connection's CWND by the same factor as the throttle rate over the bottleneck bandwidth (or, if the throttle rate is higher than the bandwidth estimate, the CWND is unchanged). Similarly, the connection's CWND may be reduced if it is subject to priority routing (wherein this embodiment supports grouping of connections into a priority level that is activated when the total goodput (throughput excluding overhead) of all connections at a higher priority level falls below a configured threshold for the activated level). Various methods for determining CWND reduction in these scenarios are described below in conjunction with Figures 8 and 9.

[0132] Connections assigned to a priority level that is inactive (the total goodput plus unused bitrate for all connections at higher priority levels meets or exceeds the activation bitrate for the priority level) have an effective CWND of zero, and connections at a priority level that is non-primary have their CWND set to zero if the estimated total goodput bitrate of all connections in the set of currently active priority levels meets or exceeds the activation threshold of the lowest active priority.

[0133] In other embodiments using priority routing, non-primary level connections may be capped at a rate lower than their bottleneck bandwidth so that they only contribute enough to close the gap to the configured activation threshold. This may be done to reduce costs, for example, if the non-primary connections are more expensive connections intended to be used only in emergencies. As with throttling, the CWND is reduced to reflect the capped rate at which these connections should be transmitted.

[0134] The loop in step 2 of the method can run as long as the connections in the active set have capacity.

[0135] Step 2a (block 1103) constructs a selector that is used to find packets that are eligible to be sent on one of the active connections that have active capacity. In some embodiments, the selector consists of a set of flow classes that is equivalent to the set of current active connections that have capacity.

[0136] For example, if a flow class requires a connection with a maximum latency of 500 ms, and all active connections with available capacity have latencies greater than 500 ms, then that flow class will be excluded from the selector. Some embodiments may choose to include flow classes in the selector that would not normally match the criteria of an active connection if the embodiment can determine that there is currently no connection (active or inactive, with or without available capacity) that meets the criteria of such a packet. In that case, the embodiment may choose to violate the match criteria in favor of sending the packet rather than holding it until it times out and is dropped. Other embodiments may choose to enforce a strict match policy whereby packets that match a flow class that does not have an eligible connection are not sent.

[0137] In another embodiment, illustrated by the exemplary sort order in the process flow of FIG. 1F, alternative techniques can be utilized for computationally determining or establishing a sort order for controlling the routing of data packets to various network connections.

[0138] Some or all of the input parameters may be rounded / bucketized before being used in the sort order determination. A technical improvement from rounding and bucketing is to keep the sort order as "stable" as possible so that changes in the sort order do not occur all the time (e.g., without rounding and bucketing, two close values ​​may cause "jitter" by successively having sort changes as their values ​​vary over time).

[0139] Factors that influence the sorting order decision include keeping the sorting order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (which would increase the likelihood of self-induced out-of-order and jitter events) and taking into account both the measured characteristics of the connections and unmeasured external directives.

[0140] Values ​​to round to may include: ●Latency related values ​​(RtProp, RTT) are rounded to the nearest multiple of 50ms, with a floor of 1ms after rounding. ●Throughput related values ​​(BtlBw) are rounded to the nearest digit. Packet loss (worst value among the three sliding windows) is rounded to the nearest digit. Note that the raw values ​​here are restricted to the range [0,100], so the output values ​​are restricted to the set (0,1,2,3,4,5,6,7,8,9,10,20,30,40,50,60,70,80,90,100).

[0141] An example of rounding to the nearest digit is provided in the table below, which shows the raw value, the calculated number of digits of the raw value, and then the resulting rounded value. [Table 2]

[0142] A specific improvement based on improving the networking routing order to control the use of network interfaces is described in some embodiments that utilizes a composite characteristic of round trip time (RTT) and packet loss to predict throughput based on the Mathis formula to establish an upper bound on TCP throughput given RTT, packet loss, and MSS. The Mathis coefficient can be utilized to modify the sorting order for routing networked packets.

[0143] The measured characteristics may include packet loss during three separate sliding windows (500ms, 3s, 30s) (further summarized as the worst value during the three windows), current network round trip time (RTT), bandwidth round trip propagation time (BBRv1) parameters, RtProp (network propagation delay), and BtlBw (throughput estimate). External directives may include priority (PR) rules among others, but may also include flow-specific rules / requirements.

[0144] In some embodiments, the measured properties of BBRv1 differ from the design of these properties in the original BBRv1 IETF draft. For example, to account for embodiments that may be executed on systems that lack consistency in their execution times, the size of the sliding window used to calculate BtlBw may be increased to cover long durations (hiding variability due to execution inconsistencies), but if network loss or latency increase events occur, the window duration may also be immediately shortened since those events indicate that BtlBw may indeed have changed and any long history in the sliding window must be forgotten.

[0145] For similar execution inconsistency reasons, the input parameters used to measure RtProp may be modified to include more than network propagation delay, adding a portion of execution / processing time that may not be trivial and contributes consistently to the actual delay encountered when transmitting a packet and waiting for a response. In some embodiments, this is referred to as "System RtProp" as opposed to "Network RtProp", which consists only of network propagation delay. In these embodiments, "System RtProp" and "Network RtProp" may be used for various calculations depending on the intended purpose. For example, when calculating CWND, "System RtProp" is appropriate so that the pipeline includes enough data in flight to account for processing delays. However, when sorting network connections to make packet routing decisions, "Network RtProp" is more appropriate since the preferred routing decision is independent of system execution delays.

[0146] Packet loss, latency, and throughput are not completely independent variables: combinations of the three affect each other depending on the application's congestion control behavior, because, for example, with TCP, higher packet loss results in longer periods of radio "silence" (underutilized channel) as the sender waits for a DUP ACK or RTO to occur.

[0147] Therefore, some composite attributes are determined and used in the final sorting method. Specifically, Mathis Coefficient: This composite attribute is based on the Mathis Relation, which predicts throughput given RTT and packet loss, and gives an upper bound on TCP throughput given RTT, packet loss, and MSS. When comparing attributes of any two network connections to determine a sorting order for these network connections, this relation shows that the expected throughput is proportional to the inverse of (RTT*sqrt(packet loss)), since MSS can be considered a constant. This relation is used in this example as a practical implementation to establish a coefficient that is used in directly controlling the operation of a network controller or router. For each connection, the system determines the Mathis Coefficient as follows: MAX(rounded Rtt, 1ms)*sqrt(rounded aggregated packet loss). A connection with a smaller Mathis Coefficient value is considered better. Since the calculation is a function of two independently measured characteristics, the relationship between these characteristics determines how the connections compare to each other. The Mathis Coefficient is maintained and stored in a data object, such as a corresponding Mathis Coefficient data value associated with each connection, which may be stored, for example, in an array of data values ​​along with other monitored characteristics.

[0148] For example, at one extreme, two connections with 0% packet loss will always have the same Mathis coefficient, regardless of the actual RTT values ​​of those connections. In this scenario, the Mathis coefficient does not affect the sort order, which is determined by subsequent comparisons of measured and composite attributes.

[0149] At the other extreme, if two connections have 100% packet loss (rounded aggregate packet loss = 1.0), then only the rounded Rtt values ​​of these connections will determine how they compare to each other. In some embodiments, this is undesirable, so connections with 100% packet loss are filtered out from the available selection earlier.

[0150] Between the two extremes, a connection with both a low RTT and low packet loss is likely to compare better than a connection with a higher value. However, a connection with a high percentage of loss can still be compensated for by having a low RTT and can have the same Mathis coefficient as a connection with a low percentage of loss. For example, Connection A: 100 ms RTT, 0.01 (1%) loss => Mathis coefficient = 10 Connection B: 19 ms RTT, 0.25 (25%) loss => Mathis coefficient = 9.5

[0151] In this example, even though connection B has significantly higher loss (25% versus 1%), the lower RTT of connection B compensates for it and makes it match the Mathis coefficient better than connection A. The explanation for this is that the lower RTT of connection B allows it to exchange packet acknowledgments (positive and negative) for lost packets and complete any necessary packet retransmissions faster than connection A, which has a higher RTT, even though connection A has a smaller percentage loss.

[0152] Capacity Factor: This composite attribute takes into account BtlBw and RtProp and is determined as follows: rounded BtlBw / MAX(rounded RtProp, 1ms). The units are "bytes / second per millisecond" and have no physical meaning, but provide a way to normalize the bandwidth a connection can achieve per unit of RtProp. In other words, a connection that can achieve more bandwidth per millisecond of the connection's RtProp is considered more efficient than a connection that achieves less bandwidth. Efficient transmission in this context means a connection that achieves more bandwidth per millisecond of propagation delay (RtProp).

[0153] Once all of the raw input has been processed and the complex properties calculated as described above, the actual sorting mechanism can be run sequentially as follows.

[0154] The transition from one comparison to the next is made only if all of the previous rows are compared as equal. If the comparison is not equal, the rest of the comparison is irrelevant and is skipped. In some embodiments, the selection of the comparison is made on a flow classification basis (e.g., VPN vs. VoiP vs. ping packets vs. FEC packets), and for each different flow, the comparison criteria list can be different (e.g., each can have a different predefined most important characteristic, second most important characteristic, third most important characteristic, etc.). Each of these characteristics can be iterated one after the other, starting from the most important, etc., until the connections themselves can be sorted into an ordered sequence. In some embodiments, depending on the flow classification, the first pre-step is to select a subset of connections that are suitable for transmitting the type of data packets in the flow classification, and then an order is given to them before packet allocation is performed. 1. A connection that is completely down (an aggregate loss of 100%) is considered a worse connection than a connection that has an aggregate loss of less than 100%. 2. PR priority (the higher the priority, the better) 3. Mathis coefficient (the lower the better) 4. Rounded RTT (lower is better) 5. Capacity factor (the higher the better) 6. Rounded RtProp (lower is better) 7. Rounded BtlBw (higher is better) 8. Connection ID (the lower the better)

[0155] A rendering of the sorting mechanism is shown in Figure IF. In an exemplary sorting mechanism, real-time flows first include a comparison of RTT / RtProp, then an equality comparison, then the real-time flows compare reliability, then if equal again, compare transmission rate, etc. Depending on the flow type, there can be different sorting orders. For example, flows wanting throughput may compare transmission rate first. If equal, compare with reliability. If equal again, real-time flows may decide not to evaluate anything else and just choose arbitrarily.

[0156] In some embodiments, flow requirements (e.g. real-time vs. throughput) may dictate the order and number of comparisons. For example, real-time flows may prefer to only compare rounded RTT and rounded RtProp, ignoring all other attributes. Throughput flows may decide to only compare Mathis and capacity factors.

[0157] The techniques described above concern explicit sorting dimensions. There are also implicit sorting dimensions that tend to arise as a by-product of typical application traffic patterns.

[0158] Most application traffic tends to be bursty or adaptive and therefore does not require use of all WAN connections. This means that connections that are currently at the bottom of the sort order may not have traffic carried over them. If this were not the case, their explicit sort dimensions would not be updated, which would tend to maintain their position near the bottom of the sort order.

[0159] In some embodiments, over a period of time, push scheduling can also be configured to implicitly induce a sorting order along one or more implicit sorting dimensions. The implicit sorting dimensions force connections that experience periodic bad events (e.g., latency spikes due to congestion) to bubble to the bottom of the list. Connections with consistent behavior naturally bubble to the top and remain near the top.

[0160] This is generally considered a good attribute of the algorithm, but it can be modified in the future to be influenced by external factors: for example, if a flow has a specific requirement that cannot currently be met by connections near the top of the sort order, it may make sense to generate probing traffic on connections near the bottom to refresh the explicit sorting dimensions of these connections.

[0161] FIG. 4A illustrates an embodiment in which, at 400A, a selector is generated using a predefined flow class and the current connection state.

[0162] In this example, there are three connections, each with a set of characteristics to be monitored, and there are three predefined flow classes, each of which defines match criteria and preferences for sorting connections that meet the match criteria.

[0163] In this embodiment, the selector consists of a set of flow classes whose match criteria match at least one connection that has available capacity. In this example, flow class 1 and flow class 2 both have match criteria that match connection 1 and connection 2, while flow class 3 does not match any connection and is not included in the selector.

[0164] Even though both connection 1 and connection 2 match, connection 2 is excluded from the selector because it does not have an available congestion window and cannot currently send packets. When constructing a selector, an embodiment may determine that flow class 1 and flow class 2 can be included in the selector without having to look only at connection 1 to ensure that connection 2 matches, since flow classes can be included as soon as a matching connection with an available congestion window is found.

[0165] Step 2b (block 1105) uses the selector from step 2a to find a packet on the retransmission or transmission queue that is eligible to be (re)transmitted on an active connection that has capacity. If no such packet is found, the loop ends. This may happen even if some active connections have capacity and there are packets on the input or retransmission queues.

[0166] 4B illustrates at 400B an embodiment with a priority input queue that is an ordered set of flows, where flows on the right side of the queue should send packets before flows on the left side. Each flow has a flow ID and each flow belongs to one of the predefined flow classes. Each flow also contains an ordered queue of packets so that when a flow is selected, the packet at the head of the queue is removed.

[0167] In this example, the selector from Figure 4A is used to evaluate the flows in the input queue from right to left, and the selector picks the first flow that has a flow class listed in the selector. In this case, flows with ID5 and ID2 are evaluated and rejected because they both have flow class 3, which is not one of the flow classes from the selector. Flow 7 is then evaluated and matches because it belongs to flow class 1, which is one of the flow classes in the selector. A packet is then extracted from the head of the packet queue for flow 7 and stamped with the flow metadata.

[0168] Once packets are selected, step 2c (block 1105) builds an ordering of active connections. The ordering chosen is based on rules provided in the flow class definition if the embodiment supports it, or is otherwise a commonly allowed ordering. Some embodiments may use the same ordering for all flows that match the flow class, and some embodiments may make the ordering flow-dependent (e.g., making all packets in a flow sticky to a given connection without requiring all flows to be sticky to the same connection). The ordering may be built as needed at each iteration of the loop in step 2, and some embodiments may choose to cache the ordering and clear the cache if the criteria used to build the ordering change (i.e., with a metadata update).

[0169] One embodiment constructs the ordering disregarding whether a connection currently has capacity, since scheduling packets on a connection may reduce the capacity to zero, or may increase the capacity above its current value of zero if scheduling packets on a different connection causes the metadata mix to change enough to activate a set of connections at a previously unused priority level. Constructing the ordering in this manner allows the ordering to be cached and reused when the only items that change are the effective CWND and in-flight byte count for a connection.

[0170] A flow traversing the system 100B tends to be better behaved if packets of that flow traverse the same WAN connection whenever possible. Therefore, an embodiment may construct the ordering using stable sorting criteria such that orderings made for the same flow will be consistent unless one or more monitored characteristics of the connection change significantly. Thus, operating characteristics such as RTT and RtProp, as well as bandwidth, are poor choices because they may fluctuate over short periods of time, making the sort order unstable.

[0171] The conversion from the "noisy" raw behavior characteristics to the monitored characteristics may be done using statistical filtering methods. Statistical filtering methods may range from traditional measures such as mean, variance, ANOVA, to the field of (quantized) rank order statistics (rank order filtering). For example, the monitored characteristics may be a range of the raw behavior characteristics, where the range may be defined by a specific value (e.g., 1-100 ms), or an order of magnitude, etc. Specific examples follow in the subsequent paragraphs, including the examples shown in Table 2.

[0172] Some embodiments can use machine learning techniques to translate from operational characteristics to monitored characteristics. For example, a trained model can be developed to predict impending failure of a connection, and then an ordering can be constructed that sorts matching connections to the end of the list. Another exemplary model can be used to detect the type of connection (e.g., wired, cellular, WiFi, satellite, BoD, etc.) and use that to form the basis of a new monitored statistic (e.g., susceptibility to buffer bloat) that cannot be easily measured in real time.

[0173] One embodiment uses the following ordering for all packets (later criteria are used only if all previous criteria are equal):

[0174] "Bucketized" drop probability: Connections can be grouped into buckets defined by a range of packet-count-based or size-based (e.g., in bytes) drop probabilities. Connections in buckets with smaller drop probability ranges appear early in the ordering, and connections in buckets with larger drop probability ranges appear later in the ordering.

[0175] Connection priority groups (provided by administrator or end user). Higher priority connections are ranked earlier (primary > secondary > tertiary, etc.)

[0176] "Bucketized" RTT. Latency is rounded to the nearest multiple of 50ms. Bucketing the latency values ​​allows normal variations in latency to be hidden from the ordering, while at the same time allowing large changes in latency to be reflected in the ordering.

[0177] "Capacity Factor". This is an aggregate value measured in "bytes per second per millisecond" and is calculated by rounding the Bottleneck Bandwidth Estimate to the nearest decimal place or to the nearest integer multiple of a decimal place (e.g., 1 to 1, 10 and 11 to 10, 15 to 20, 86000 to 90000) and dividing it by RtProp bucketed to the nearest 50ms. Connections that achieve a higher bandwidth per ms of RtProp are considered better than connections that have a lower bandwidth per ms of RtProp.

[0178] Determining the capacity factor of raw BtlBW and RtProp values ​​is highly variable, but stabilizing the capacity factor by rounding BtlBW to the nearest digit or nearest integer multiple of digits and bucketing RtProp will stabilize the values ​​over time unless significant changes are observed. Table 2 shows some examples of bucketing, rounding, and capacity factor calculations. Note that in this embodiment, the capacity factor is rounded to the nearest integer, and a bucketed RtProp value of 0 is treated as 1 for purposes of the capacity factor calculation.

[0179] "Bucketized" RtProp. Lower values ​​of RtProp bucketed to the nearest 50ms are preferred over higher values.

[0180] "Rounded" BtlBW. Higher values ​​of BtlBW rounded to the nearest digit or to an integer multiple of the nearest digit are preferred over lower values.

[0181] If all of the above criteria compare equal, the connection ID (assigned when the connection was enumerated) is used to deterministically break ties. Other embodiments may use more complex tie-breaking mechanisms, such as hashing attributes of the flows to an integer modulo the number of connections to balance. Table 2: Examples of bucketing, rounding, and capacity factors [Table 3]

[0182] An embodiment that distinguishes between flow types may, for example, modify the above ordering to prioritize high capacity factor connections over lower latency connections for file transfer flows, and low latency connections over high bandwidth connections for video streams, depending on the transmission requirements of the flow type.

[0183] An embodiment that performs strict flow pinning on a connection (or set of connections) may choose to only allow exact matches of connection IDs (possibly excluding other connections from the ordering entirely). Such pinning rules are typically provided by an administrator or end user and are intended to override any ordering decisions determined by monitored characteristics. Thus, even if the monitored characteristics indicate that a connection is unavailable or has failed, the flows will continue with the same ordering and therefore will have no connectivity through the multipath system 100B.

[0184] Some embodiments may create monitored characteristics that are composites of other monitored or operational characteristics, and use these composites to determine ordering. For example, a composite reliability metric may be created that combines bucketized drop probability, (on-the-fly) latency (RtProp) variance, and (on-the-fly) throughput into a weighted metric that represents the overall reliability for a connection. Connections that rank high on this metric will be reserved for flows that require high reliability (high reliability may not mean high speed).

[0185] In general, when a selector is generated that allows a packet to be selected, it should generate an ordering that allows the packet to be scheduled on some of the connections that were used to generate the selector.

[0186] FIG. 4C shows an example of how a packet's flow class can be used to construct an ordering for connections at 400C. In the depicted embodiment, the flow class in the packet's metadata is used to reference the flow class definition. The match criteria of the flow class is used to select which connections should be included in the constructed ordering (in other words, the flow class defines a set of transmission requirements that can be associated with a particular flow class); in this case, connection 1 and connection 2 both match, while connection 3 does not. Note that connection 2 is included even though it does not have an available congestion window (i.e., it is not currently eligible to transmit packets). This facilitates caching the ordering and allowing it to be reused even if the state of the available congestion window should change (e.g., if the scheduler initially blocks transmission on a connection because it has low priority, but then decides to re-enable the connection because the current data mix on the enabled connection does not meet the minimum system throughput).

[0187] Once a set of connections is selected, the ordering criteria from the flow class (e.g., a subset of the transmission requirements associated with the flow class) are used to sort the matching connections into an ordered list. In this example, the first criterion is the capacity factor; both connections have the same capacity factor. The second criterion is the RTT; both connections have the same RTT. The third criterion is the BtlBW (bottleneck bandwidth); connection 2 appears before connection 1 in the ordering because it has a higher value.

[0188] A set of ordering criteria (e.g., transmission requirements) on a flow class allows packets to be scheduled on the available connection that is most optimal for the type of flow they belong to. For example, packets belonging to a "video encoder" flow class may preferentially use a connection with the lowest latency, while packets belonging to a "file transfer" flow may preferentially use a high throughput connection even if the latency is high or highly variable.

[0189] Once ordering has been determined for the connections, Step 2d (block 1107) uses the ordering to assign packets to connections that have available bandwidth. The packets are placed on a transmit queue for the connection.

[0190] 4D illustrates, at 400D, selecting a connection for a given packet using the ordering from FIG. 4C. In this example, the first connection in the ordering is connection 2, but connection 2 is not currently eligible to send the packet because it does not have an available congestion window. Connection 1 is then checked and, since it is able to send the packet, connection 1 is selected and the packet is queued for transmission to the stage 3 pipeline for connection 1.

[0191] The connection metadata (monitored characteristics) of in-flight bytes is then adjusted as part of step 2e. In some embodiments, this includes distinguishing between goodput bytes and retransmit / dummy / probing bytes when tracking CWND consumption for a connection. Such adjustments may impact the goodput and / or potential goodput for a priority group that may activate the next priority level connection in the next iteration of the loop in step 2. Various methods for converting units of volume (CWND bytes) to rate (goodput) are discussed following the description of Figures 8 and 9.

[0192] When step 3 is reached, either the available CWND for each connection is full or there are no packets eligible for transmission. Packets queued in the per-connection output queue in step 2 are then sent to the appropriate instance of the next stage in the pipeline.

[0193] In some embodiments, an explicit objective function can be used to determine the desired tradeoff when there are multiple flows that belong to flow classes with the same priority but with competing requirements.

[0194] 4A-4D show an objective function that implements fairness among flows with the same priority: the selector serves each of the flows that match the flow class in a round-robin fashion.

[0195] A fair implementation of this embodiment uses the deficit round-robin scheduling algorithm described in RFC 8290. A simple overview of this algorithm is that it utilizes a "byte credit" scheme, allocating a quantum of byte credits to each queue in each round of iterations through the queues.

[0196] The number of bytes that can be obtained for transmission from each queue in each round is nominally limited to the number of available credits that the queue has. Fairness naturally arises when the byte credit quantum for each queue is equal.

[0197] In some embodiments, the objective function may allow intentional unfairness to be configured by an administrator or end user. For example, weights may be configured on the matching flows that may ultimately result in unequal byte credit quanta being awarded to queues at each iteration.

[0198] For example, there can be five different weight levels: ●Best (5) ●High(4) ●Standard (3) ●Low(2) ●Minimum (1)

[0199] The value assigned to each weight represents the ratio between the byte credit quanta of these queues. For example, if the quantum corresponding to normal is 30KB, a flow marked as low will receive (2 / 3)*30KB=20KB of quantum in each round. A flow marked as highest will receive (5 / 3)*30KB=50KB.

[0200] Note that the absolute numbers in this example are not important: if the standard quantum were 5KB instead of 30KB, the weights would still scale the relative quantum for the other priority levels appropriately, and the end result would be the same in terms of overall fairness.

[0201] Other embodiments may allow the weights to be any integer number rather than the fixed value of five in the above example, which would allow an administrator or end user to configure greater fairness between flows into the objective function if desired.

[0202] Also, note that, as with all Quality of Service (QoS), these weightings only matter if there is contention for available WAN capacity: no contention means that the data in all queues fits entirely within the available capacity, so the ratio between queue usage is a natural fit since applications generate different volumes of data.

[0203] Other embodiments may have completely different objective functions that do not consider fairness at all. For example, ●Ensure that the maximum number of flows can be satisfied by the available WAN capacity, or maximize the total throughput achieved by the served flows, or • Minimize changes to current bandwidth allocations on BoD connections.

[0204] In some embodiments, the way in which a WAN connection is used can alter the attributes of the WAN connection and make it ineligible to serve certain flow classes. For example, some types of connections may achieve high throughput but at the expense of incurring a higher RTT.

[0205] An embodiment with an objective function that seeks to maximize throughput would choose to serve flows with high throughput requirements, which would increase the WAN connection RTT and make the WAN connection ineligible for servicing flow classes with low latency requirements.

[0206] Conversely, an embodiment with an objective function that seeks to serve the maximum number of flows may choose the opposite trade-off so that low latency flow classes can continue to be served by this WAN connection.

[0207] In some embodiments, the connection sort order of an individual scheduler is maintained on a per-flow basis, rather than globally for all flows.

[0208] This allows flow-specific requirements to influence the sort order so that packets of a flow are assigned to connections in a way that meets the needs of the connections.

[0209] For example, a VoIP flow with a preferred maximum latency tolerance of 150 ms would want a sorting criterion that places connections with an RTT greater than 150 ms at the bottom of the list. Conversely, a TCP flow with no specific latency requirements would instead want to prioritize connection throughput in its sorting criteria, either ignoring RTT entirely or using RTT only as one of many secondary criteria.

[0210] In some embodiments, these flow requirements and preferences are configured in the form of rules that consist of match criteria and resulting behaviors.

[0211] For example, match criteria can take the form of an IP3-tuple (protocol, source IP, and destination IP) or an IP5-tuple (protocol, source IP, source port, destination IP, and destination port). Behavior takes the form of explicit preferences for latency or throughput, with target preferred values ​​for either.

[0212] Examples of other heuristic-based match criteria may include: DSCP tag Any of the other header fields available in the IP (v4 or v6) header Any other header field in the TCP or UDP header The cumulative volume of data transmitted, e.g. TCP flows that exceed a volume threshold, may be matched as "bulk data transfers", whereas those that do not may not be matched as interactive SSH / telnet sessions. ●Cumulative duration of flow Regular expression matching against packet payload Clear-text parameters extracted from the TLS handshake (e.g., SNI, CN, SubjectAltName)

[0213] Examples of other actions that can affect the scheduler sort order include: A particular preference for connection ordering (e.g., a user may prefer to sort expensive connections to the bottom of the list for low-priority flows) Jitter Tolerance ● Maximum latency tolerance The amount of redundancy desired (e.g., FEC) Whether retransmission of lost packets is desired, and if so, whether there is a maximum time limit after which retransmission attempts should stop Whether explicit packet pacing is desired For example, a real-time video application transmitting video at 30 frames per second may transmit video frames exactly 33.3 milliseconds apart and may not want the multipath system 100B to change the pacing. In contrast, bursty applications may benefit from having the scheduler explicitly pace bursts of video at the aggregate WAN rate. Desired throughput (e.g., minimum, target, maximum)

[0214] Rules (both match criteria and actions) can be automatically generated using machine learning techniques that analyze traffic patterns.

[0215] Inputs (features) to the ML method may include: i. The cumulative volume of data transmitted ii. Packet size distribution (histogram) iii. Inter-packet spacing (both pacing and jitter) iv. Packet Frequency and Grouping v. Packet header fields (Ethernet, IP, TCP, UDP, etc.) vi. Packet Contents (Payload) vii. The intended destination of the packet, not just the IP addressing (e.g., SNI, CN, and SubjectAltName fields in the TLS handshake and exchanged certificates) viii. Cumulative duration of flow ix.Time x. Number of concurrent flows to the same destination or from the same source xi. Current or historical predictions (labels) of any concurrent flows to the same destination or from the same source (e.g., some applications open multiple flows, one as the control plane and another as the data plane, and knowing that there are existing control plane flows can help predict data plane flows)

[0216] The predictions (labels) from the ML method will include any of the above behaviors, determined based on the behavior of the training corpus.

[0217] In some embodiments, rules can have the concept of subordinate rules that apply additional match criteria and more specific behaviors while still being governed by a parent rule. For example, Any VPN flow identified by an IP5-tuple may have a rule defined with match criteria and actions. This would be considered the "parent" rule. A VPN will carry many internal flows within the VPN and from the applications it protects. Normally these internal flows would be completely hidden from the match criteria (encrypted by the VPN). However, some VPNs may choose to expose a limited set of information about their internal flows by setting DSCP tags on encrypted packets that correspond to the preferences of the internal (cleartext) packets.

[0218] Dependent rules can be created that match DSCP tags, but the resulting available behavior will still be governed by the parent rule. For example, a. Generally, a VPN requires in-order delivery of packets (they can tolerate out-of-order, but only up to a small window). b. The behavior configured in a subordinate rule that matches DSCP tags must not result in the parent rule delivering packets significantly out of order.

[0219] For example, a dependent rule that requires a maximum latency limit of 150 ms and a second dependent rule that requires a maximum throughput may violate the ordering requirements of the parent rules, since interleaved packets from these two dependent rules may be assigned to connections with significantly different latencies.

[0220] 5A illustrates two independent flows, each with completely independent rules and behaviors. In this example, the SIP flow requires low latency, so the behavior causes scheduler 104B to transmit only packets of the SIP flow on connection 1. Conversely, the FTP flow requires high throughput, so scheduler 104B transmits only packets of the FTP flow on connection 2.

[0221] As packets for these two flows are transmitted over the air and reach sequencer 118, they are put back in order and transmitted independently to their final destinations. In this example, sequencer 118 transmits packets belonging to the SIP flow that have already arrived and are in the correct order, regardless of the state of the FTP flow, whose packets are still in transmission over the air.

[0222] 5B illustrates the concept of dependent flows. In this example, the same SIP and FTP flows exist, but they go through a VPN client for encryption and encapsulation before scheduler 104B sees them.

[0223] Upon emerging from the VPN client, packets from both flows have the same IP5 tuple, so scheduler 104B and sequencer 118 treat this as the parent flow and are constrained by the requirements of the parent flow.

[0224] The presence of subordinate flows (SIP and FTP) is conveyed using DSCP tags, which control how scheduler 104B allocates packets for transmission to connections 1 and 2. In this example, the allocation is the same as in Figure 5A.

[0225] However, the parent flow constraint requires that packets be transmitted from sequencer 118 in the same order that they arrived at scheduler 104B. Thus, in this example, even if packets belonging to a dependent SIP flow have already arrived at sequencer 118, packets belonging to the dependent SIP flow cannot be transmitted until packets belonging to the dependent FTP flow have arrived and been transmitted first.

[0226] Figure 6A, at 600A, is a sample embodiment of a rule management page. The configured rules are displayed in a table, with each row summarizing match criteria and behavior. The order in which the rules are listed is the order in which packets will match the rules. Figure 6B, at 600B, shows the workflow for changing the order of rules.

[0227] FIG. 6C shows how, at 600C, each row in the table can be expanded to show details of the match criteria and behavior, as well as the currently active flows through the system that match this rule.

[0228] Each rule in the table includes edit and delete icons.

[0229] 6D shows the dialog that appears when the Edit button is clicked in 600D. The match criteria and behavior for the rule can be changed in this dialog.

[0230] The main rule management screen has a separate "Add Rule" link, and FIG. 6E shows, at 600E, the modal dialog that appears when clicked.

[0231] In some embodiments, rules are configured on the sender side and automatically pushed to the receiver. By default, rules are symmetric and bidirectional, meaning that when a rule is added and its behavior is configured, a matching pair rule is added transparently, but with the source and destination match criteria swapped. It will be possible to add asymmetric rules (different behavior for either direction of the same flow) via an advanced configuration screen.

[0232] In one embodiment, the scheduler probes various network characteristics of multiple connections (to determine measured characteristics for each connection) and uses the measured characteristics in combination with the transmission requirements to make decisions regarding packet transmissions.

[0233] For example, one such decision is to determine how many packets to transmit on a particular connection and when to transmit those packets.

[0234] Some embodiments are volume-based. Such embodiments operate by determining a limit (e.g., in units of bytes or packets) on the volume of data that may be transmitted on a particular connection. Once enough packets at that volume have been transmitted, transmission stops until some feedback about these packets is received. If feedback is received before the full volume limit has been transmitted, transmission is allowed to continue unstopped as long as the volume of transmitted data for which no feedback has been received does not exceed the determined volume limit.

[0235] In one embodiment, the volume of data is chosen to be the congestion window (CWND) of the connection. Simply put, CWND is the maximum volume of data that can be sent on a connection before receiving any feedback without causing significant congestion on that connection. There are many methods for estimating CWND. It is assumed that the scheduler has access to one such method and can receive an estimate of CWND through that method.

[0236] In some embodiments, the scheduler is required to determine the rate at which data is being transmitted (e.g., in units of bytes per second). This rate can then be used to make other decisions regarding packet transmissions.

[0237] For example, in one embodiment, if the rate of data being transmitted on currently active network connection(s) is less than the rate of data being received from a source application for transmission, the scheduler may decide to activate more connections to increase the aggregate transmission rate to accommodate the application rate.

[0238] In another example, an embodiment may have to meet quality of service (QoS) requirements, which may be expressed in terms of guaranteed transmission rates. If the aggregate rate falls below a required level, the scheduler may decide to activate more connections to increase the aggregate transmission rate to meet the QoS requirements.

[0239] In yet another example, an embodiment may be required to measure goodput (i.e., the rate at which application data successfully passes through the network) for purposes of guaranteeing application performance levels, optimizing performance, reporting, etc.

[0240] In one embodiment, TCP performance may be optimized by pacing packets sent at the aggregate rate achieved by multiple connections, see, e.g., PCT Application No. PCT / CA2020 / 051090.

[0241] In embodiments where a rate is required for some function, a volume-based scheduler requires a method to convert the volume of data being transmitted into a rate.

[0242] The trivial approach of converting volume to rate by dividing the volume by the duration of time transmitted generally produces inaccurate results. Such approaches measure the transmission rate at the transmitter, not the actual rate data is traveling through the network. The actual rate at which data travels through the network is truly determined by the rate at the slowest link of a multi-link network, commonly referred to as the bottleneck of the network.

[0243] The characteristics of the bottleneck link of a network are not available to the sender or receiver, especially knowing that a multi-link network can change the combination of links used at any time transparently to the sender and receiver. However, congestion control methods can generate estimates based on the performance observed at the sender and receiver.

[0244] In some embodiments, the congestion control method may provide an estimate of the transmission rate of the bottleneck link, which may be referred to as the bottleneck bandwidth BtlBw of the network.

[0245] In some embodiments, the network round trip propagation time RtProp may be estimated by sampling the time between when a packet is sent and when feedback about the packet is received.

[0246] A combination of BtlBw and RtProp can be used to determine a network attribute called the bandwidth-delay product BDP, as follows: BDP=BtlBw×RtProp

[0247] The most common unit for BDP is bytes. For these quantities the following units are assumed: (i) bytes per second for BtlBw, (ii) seconds for RtProp, and therefore (iii) bytes for BDP.

[0248] BDP refers to the amount of data that would be present in the network (in-flight) if operating under ideal conditions, assuming that data is always available for transmission. The network would transfer data at the network's nominal rate BtlBw and provide feedback on every packet exactly RtProp time after sending that packet. Figure 7 shows two examples at 700. The top half of Figure 7 illustrates this concept using an example where the BDP consists of 16 equal-sized packets. In practice, the amount and size of packets can be different, as long as the total size is equal to the value calculated using the BDP.

[0249] In some volume-based embodiments, the BDP can be used to estimate various data transfer rates, some examples of which are provided above.

[0250] For the in-flight volume under consideration, assuming ideal conditions, when the volume is less than the BDP, only a fraction of the network's nominal rate, BtlBw, can be achieved. This fraction can be calculated as the ratio of the volume to the BDP. The resulting achieved rate is determined as follows: Achieved rate corresponding to in-flight volume = Volume / BDP x BtlBw

[0251] In practice, networks rarely operate under ideal conditions. For example, packets may be temporarily buffered before being delivered to the receiving end, and feedback may be delayed at the receiver or by the network. In some cases, packets may be lost and never reach the receiver, thereby not triggering feedback. In other cases, the feedback itself may be lost. Loss may be estimated, for example, when feedback of a new packet is received, or based on timer expiration.

[0252] Waiting for delayed feedback (which may or may not arrive) to trigger the next transmission event can artificially limit the achievable rate in the network.

[0253] FIG. 8A illustrates, in diagram 800A, an example of a receiver that aggregates feedback and sends an acknowledgment back to the sender once for every four packets received.

[0254] 8B continues to illustrate how, in diagram 800B, for example, aggregation of acknowledgments results in an overall throughput that is less than the network's capacity. The sender waits a longer time before starting to transmit the next group of packets. Overall, the transmitter ends up transmitting the same amount of packets as in the ideal case of FIG. 7, but over a longer period of time, resulting in a decrease in throughput.

[0255] In one embodiment, this artificial limit is overcome by specifying an additional allowance of data to send even if feedback of a previously transmitted packet has not yet been received. This means that the volume of data in-flight may exceed the BDP. In such a case, however, the network is not actually forwarding data at a higher rate, as application of the achieved rate formula above might suggest. The additional allowance simply allows the sender to maintain a constant rate equal to BtlBw for a longer period of time. Thus, the estimated transfer rate of the volume at hand is capped to BtlBw.

[0256] 8C continues to illustrate how, in diagram 800C, a transmitter may use such additional transmission allowance to continue transmitting packets, for example, before an acknowledgment is received. The volume of data in-flight, 19 packets, exceeds the BDP of 16 packets. However, the actual rate is matched to the network's capabilities, and capping of the calculated rate achieves the same result by using the minimum of the in-flight volume and the BDP.

[0257] In some embodiments, a similar approach can be used to estimate the goodput that could potentially be achieved on a network, hereafter referred to as potential goodput.

[0258] A particular volume of data can be divided into a goodput portion, which contains newly transmitted application data, and another portion, which contains data that cannot be counted as newly transmitted application data, e.g., retransmissions of previously lost data, control data, probing data, etc.

[0259] If the total volume of data is larger than the BDP, the packet and / or feedback delays are assumed as above. We assume that there is no extra space available for additional goodput. The resulting (potential) goodput rate is estimated as follows: Goodput rate = Goodput portion / total volume × BtlBw

[0260] For example, Figure 9A shows a network with a BDP of 16 packets at 900A. To achieve a throughput equal to the network's capacity, the volume put in flight is 19 packets. However, only 14 packets are goodput, resulting in a goodput rate of 7 / 8 of the network's capacity. If the volume of data is less than the BDP, the difference is considered to be extra space that is assumed to be available for additional goodput.

[0261] FIG. 9B shows an example in which, at 900B, only 3 packets are in-flight, resulting in an extra space of 13 packets that can be assumed to be goodput when calculating the potential goodput rate. This extra space must be limited by the CWND that the congestion control method determines. At any point in time, the extra volume that the congestion control allows is the difference between the CWND and the total volume. If the extra space for goodput exceeds this congestion control allowance, the space is simply limited to the congestion control allowance.

[0262] FIG. 9C shows an example in which, at 900C, CWND has been reduced to 11 packets, which limits the extra space assumed to be available for additional goodput to 8 packets.

[0263] The resulting potential goodput rate is estimated as follows: Potential goodput rate = (goodput portion + limited goodput extra space) / BDP × BtlBw

[0264] Continuing with the above example, Figures 9B and 9C show how the above potential goodput formula results in a lower potential goodput rate for the scenario in Figure 9C, as would be expected given the CWND reduction enforced by the congestion control method.

[0265] Although the above technique is described in terms of goodput and potential goodput rates, the technique can be easily modified to calculate the rate or potential rate of any type of data that makes up a portion of the total volume.

[0266] Below is a numerical example of applying the preceding equations to a network having attributes typical of those encountered over an LTE cellular network. BtlBw=10Mbps RtProp=60ms BDP=10Mbps*60ms=75KB CWND=60KB Goodput portion = 25KB Other parts = 10KB Total volume = 25KB + 10KB = 35KB Goodput extra space = BDP - total volume = 75KB - 35KB = 40KB Congestion control allowance = CWND - total volume = 60KB - 35KB = 25KB Limited goodput extra space = Min(goodput extra space, congestion control tolerance) = Min(40KB,25KB)=25KB Potential Goodput Rate = (Goodput portion + limited goodput extra space) / BDPxBtlBw= (25KB+25KB) / 75KBx10Mbps=50 / 75x10Mbps=6.67Mbps

[0267] Systems communicating over IP networks often implement congestion control mechanisms that attempt to achieve fair network utilization by all network nodes that employ similar congestion control mechanisms.

[0268] Congestion control mechanisms often operate by monitoring communications occurring on a network, possibly probing the network as necessary, and deriving network attributes based on the results. A model is created to represent the network, and subsequent communications are guided based on this model. For example, the model may track network attributes such as the current maximum throughput achieved, the current and minimum round-trip times of packets, packet loss events, etc.

[0269] Based on the developed model and the tracked information, the congestion control mechanism can determine when the network is underperforming, indicating that the network may be experiencing congestion. The congestion control mechanism can then take corrective action aimed at reducing or eliminating network congestion and restoring desirable network performance.

[0270] In previous congestion control implementations (eg, TCP), packet loss is used as an indicator of network congestion and corrective action is taken in response to the loss (eg, reducing the CWND).

[0271] In one embodiment, the latency of a packet, i.e., the time required for a packet to traverse the network from a transmitter to a receiver, is an indicator of network performance. High latency packets may be useless to the receiver and may be discarded, resulting in a waste of transmission of these packets. For example, a real-time VoIP application that requires low conversation delay between participants is designed to avoid using packets carrying voice data that are older than the maximum acceptable conversation delay.

[0272] In such an embodiment, the congestion control mechanism may monitor packet latency and attempt to control the latency such that the latency does not exceed an acceptable level.

[0273] In one embodiment, the minimum latency measured for a packet may form a baseline value to which subsequent measurements may be compared, and if the subsequent measurements exceed a threshold that is a function of the baseline value, the congestion control mechanism considers the network to be experiencing congestion.

[0274] In another embodiment, a similar approach uses, instead of latency, the round-trip time of a packet, which is the time that elapses between sending a packet and receiving an acknowledgment therefor.

[0275] In one embodiment, when the congestion control mechanism determines that the network is experiencing congestion based on increased packet delays, the congestion window (CWND) is reduced to limit the amount of data that can be transmitted in the network without receiving any feedback. The purpose is to reduce the volume of data that will be or is currently buffered by the network, which is a common cause of increased packet delays.

[0276] In one embodiment, the congestion window is reduced to the bandwidth-delay product (BDP) of the network. Theoretically, the BDP represents the volume of data that the network should be able to deliver without forming a stationary queue of packets in its buffers, assuming that packets are transmitted by the sender at a constant pace that reflects the network's throughput (BtlBw). This is therefore expected to allow the network to at least partially recover and reduce packet delays back to acceptable levels.

[0277] For example, assume a network with a BDP of 16 packets and a congestion control mechanism that has determined a CWND of 24 packets to allow for acknowledgment aggregation. FIG. 10A illustrates how congestion from other network nodes can cause the sender to dominate the network at 1000A. In this case, the sender does not reduce its congestion window, but puts the entire amount in flight. This causes the network buffer to fill up and the excess packets that cannot be buffered are dropped. Even after the congestion is cleared and the normal draining of the network buffer continues, FIG. 10A illustrates how a stationary queue would have formed inside the network buffer that would increase the round trip time of all future packets and make it easier to fill the network buffer.

[0278] Continuing with this example, FIG. 10B illustrates how a transmitter reducing its congestion window to the BDP at 1000B avoids both packet drops and the formation of stuck queues. The transmitter can determine that congestion is occurring, for example, by detecting that no feedback has been received for a period of time that exceeds a baseline duration for feedback that the transmitter measured in past transmissions. In another example, the transmitter can determine that congestion is occurring by receiving feedback indicating that an amount of previously transmitted packets has not been delivered to the receiver that exceeds the baseline loss encountered in past transmissions. In response to this congestion determination, reducing the congestion window to match the BDP reduces the amount of packets that are put in flight. In turn, this reduces the likelihood of filling up network buffers, which previously led to packet drops. Furthermore, putting only BDP packets in flight allows the network to fully drain the network buffers before new packets from the transmitter arrive again (assuming that network attributes have not changed).

[0279] In another embodiment, the congestion window is reduced to be equal to the volume of packets currently in-flight.

[0280] Once the congestion control mechanism determines that an acceptable level of performance has been restored, the network is considered to no longer be experiencing congestion.

[0281] In one embodiment, a reduction in packet delay below a threshold is an indication that network performance has been restored.

[0282] In one embodiment, the threshold is determined as a function of the baseline delay value.

[0283] In another embodiment, transmitting a particular volume of data without incurring any loss is an indication that network performance has been restored.

[0284] In one embodiment, the volume of data is equal to the BDP.

[0285] In another embodiment, the volume of data is equal to the original value of the congestion window before it was reduced by a congestion control mechanism in response to congestion.

[0286] The push architecture improves system performance for non-BoD connections for the most common application types. Table 3 summarizes the improvements observed experimentally. Table 3: Application performance improvements due to push-based scheduler [Table 4]

[0287] The push-based scheduling method also makes priority routing behavior more intuitive: for example, in one embodiment, the decision to engage lower priority connections depends solely on how the BtlBw of these connections compares to a configured threshold.

[0288] For example, consider the following scenario: wan0 ●Priority: Primary ● Target threshold: Not applicable ●BtlBw: 10Mbps wan1 ●Priority: Secondary ● Target threshold: 15Mbps ●BtlBw: 30Mbps

[0289] Using a pull scheduler, every time wan1's 50Hz timer wakes up and requests a packet, the scheduler offers wan1 a packet for transmission because the scheduler knows that the desired target threshold (15Mbps) is greater than the threshold that wan0 can provide alone (10Mbps).

[0290] However, the scheduler does this regardless of the rate at which packets arrive at the input: for example, if the input rate does not exceed 10Mbps, it would be more optimal if wan1 was not involved, since the traffic could fit entirely on wan0 alone.

[0291] With a push scheduler, wan0 will be sorted to the top of the list, and even if the threshold indicates that wan1 should be active, packets will only be scheduled on wan0 after the wan0 CWND is completely consumed, resulting in less unexpected usage of lower priority connections and improved performance.

[0292] In summary, the scheduler of the multipath system can be implemented as either a pull type (100A) or a push type (100B). The push type is superior because it allows the scheduler to make packet scheduling decisions on a self-determining cadence rather than being driven by the rate at which the target connection is ready to transmit. This self-determining cadence allows it to take into account all factors that are important to the application that generated the packet and / or to the user of the system.

[0293] For example, packets generated by a TCP-based application requiring maximum throughput will preferentially have their packets scheduled onto connections ordered from best Mathis coefficient to worst Mathis coefficient.

[0294] Now consider a more complex example using the same TCP-based application, except that the user has configured multipath system 100B with priority routing rules to prioritize connection 110B over all other connections 110N unless the achievable goodput of 110B drops below 5 Mbps. In a hypothetical scenario in which 110B has high throughput, low latency, but high packet loss, 110B will have a worse Mathis coefficient than all other connections 110N.

[0295] Push-based scheduler 100B could make complex scheduling decisions that respect user preferences to use connection 110B first, and also calculate the achievable goodput for 110B by converting the in-flight CWND of connection 110B to a bit rate and using that to estimate the available goodput for connection 110B. If this availability drops below a configured 5 Mbps threshold, 110B could schedule packets onto the remaining connections 110N ordered from best to worst Mathis coefficient.

[0296] FIG. 11 is a schematic diagram of a computing device 1100 that can be used to implement system 100B, according to one embodiment.

[0297] As depicted, computing device 1100 includes at least one processor 1102 , memory 1104 , at least one I / O interface 1106 , and at least one network interface 1108 .

[0298] Each processor 1102 may be, for example, a microprocessor or microcontroller (e.g., a dedicated microprocessor or microcontroller), a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or various combinations thereof.

[0299] The memory 1104 may include a combination of various types of computer memory located either internally or externally, such as, for example, random access memory (RAM), read only memory (ROM), compact disc read only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read only memory (EPROM), and electrically erasable programmable read only memory (EEPROM), ferroelectric RAM (FRAM), etc.

[0300] Each I / O interface 1106 allows the computing device 1100 to interconnect with one or more input devices, such as a keyboard, mouse, camera, touch screen, and microphone, or one or more output devices, such as a display screen and speakers.

[0301] Each network interface 1108 enables computing device 1100 to communicate with, exchange data with, access and connect to network resources, host applications, and run other computing applications by connecting to a network (or multiple networks) capable of carrying data, including the Internet, Ethernet, Plain Old Telephone Service (POTS) lines, Public Switched Telephone Network (PSTN), Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), coaxial cable, optical fiber, satellite, mobile, wireless (e.g., Wi-Fi, WiMAX), SS7 signaling networks, fixed lines, local area networks, wide area networks, and others including various combinations thereof.

[0302] For simplicity only, one computing device 1100 is shown, however, system 100B may include multiple computing devices 1100. The computing devices 1100 may be the same or different types of devices. The computing devices 1100 may be connected in a variety of ways, including being directly coupled, indirectly coupled via a network, distributed across a large geographic region, connected via a network (which may be referred to as "cloud computing").

[0303] For example, but not limited to, computing device 1100 may be a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a personal data assistant, a mobile phone, a smartphone device, a UMPC tablet, a video display terminal, a game console, or a variety of other computing devices capable of being configured to perform the methods described herein.

[0304] Applicants emphasize that the described embodiments and examples are illustrative and non-limiting: practical implementations of the features may incorporate combinations of some or all of the aspects, and the features described herein should not be taken as an indication of future or existing product plans.

[0305] The terms "connected" or "coupled" can include both direct coupling (wherein the two elements coupled to each other are in contact with each other) and indirect coupling (wherein at least one additional element is located between the two elements).

[0306] Although the embodiments have been described in detail, it is to be understood that various changes, substitutions, and alterations can be made herein without departing from the scope. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification.

[0307] Throughout the foregoing description, numerous references may be made to a packet controller, server, instance, interface, portal, platform, or other system formed from a computing device. It should be understood that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer-readable tangible non-transitory medium. For example, a packet controller may implement the technical solutions of the described embodiments in the form of a software product. The software product may be stored on a non-volatile or non-transitory storage medium, which may be a CD-ROM, a USB flash disk, or a hard disk (removable or otherwise). The software product includes some instructions that enable a computing device (e.g., a packet controller, or a network device) to execute a method provided by an embodiment.

[0308] As can be understood, the embodiments described and illustrated above are intended to be exemplary only.

Claims

1. A packet communications controller coupled to a plurality of network connections, each network connection having a corresponding output queue from a plurality of output queues, the packet communications controller comprising: a processor; monitoring and maintaining, in a data object, a network characteristic associated with each network of the plurality of networks, such that each network of the plurality of networks may be ordered in an ordered sequence based on the network characteristic; When it is determined that a trigger event has occurred, establishing the ordered sequence for network connections of a selected set of network connections based on the network characteristics; assigning one or more of the unallocated data packets of an input queue to one or more corresponding output queues of the plurality of networks by assigning the one or more data packets to consecutive available network connections based on availability and the ordering sequence; updating the monitored characteristics of the corresponding plurality of networks to reflect the assigned one or more data packets; and A packet communications controller including a processor configured to communicate the assigned data packets from the plurality of output queues to a destination device.

2. The packet communications controller is a push scheduler, and the processor further comprises: further configured to measure a transmission capacity of each of the plurality of network connections; The packet communications controller of claim 1 , wherein the controller is configured to allocate the one or more data packets while the transmission capacity of one or more of the plurality of network connections is not exhausted.

3. 2. The packet communications controller of claim 1, wherein the processor is further configured to allocate the one or more data packets upon detecting a new data packet in the input queue or upon receiving an acknowledgment that one or more previous data packets have been received at the destination device.

4. The processor, For each of the plurality of network connections, the monitored characteristic is estimating at least one of a transmission rate, a round trip time, and a round trip propagation time of a bottleneck link; comparing the at least one of an estimated round trip propagation time, the estimated round trip time, and the estimated transmission rate to a respective reference range of at least one of a round trip propagation time, a reference range of a round trip time, and a reference range of a transmission to establish a score value for each of the at least one of the estimated round trip propagation time, the estimated round trip time, and the estimated transmission rate under consideration; 2. The packet communications controller of claim 1, further configured to measure by: ordering said plurality of networks in said ordered sequence based on said score value for each of said plurality of networks.

5. 5. The packet communications controller of claim 4, wherein the estimation of at least one of a transmission rate, a round trip time, and a round trip propagation time of a bottleneck link includes an estimation of a plurality of score values ​​for at least two of the transmission rate, the round trip time, and the round trip propagation time of the bottleneck link, and the ordering sequence is based on a comparison of the score values ​​corresponding to predefined most important characteristics, and if the score values ​​corresponding to the predefined most important characteristics are equal, proceeding to a next most important predefined characteristic for comparison.

6. 6. The packet communications controller of claim 5, wherein the processor is configured to perform a flow classification of the unassigned data packets, and based on the flow classification, the predefined most important characteristic and the predefined second most important characteristic are identified.

7. The processor, measuring modified monitored characteristics, including system overhead processing time, for each of the plurality of network connections, and using either the original characteristics or the modified characteristics, determining at least one of an estimate of the transmission rate, the round trip time, the round trip propagation time, a bandwidth-delay product (BDP), and a congestion window of a bottleneck link; 5. The packet communications controller of claim 4, wherein the ordering of the plurality of networks is further based on the at least one of the estimates of the transmission rate, the round trip time, the round trip propagation time, the BDP, or the congestion window of a bottleneck link.

8. The processor, 8. The packet communications controller of claim 7, further configured to modify the measured characteristics by dynamically adjusting a duration of one or more corresponding sliding window histories based on other measured characteristics including at least one of packet loss in the round trip time or a change in the round trip time for each of the plurality of network connections.

9. The processor, estimating a Mathis coefficient data value indicative of a relative upper limit of achievable throughput for said network connections among said plurality of network connections, taking into account round trip times and packet loss for said network connections; further configured to compare the estimated Mathis coefficient data value to a reference range of Mathis coefficient data values; The packet communications controller of claim 4 , wherein the ordered sequence of the plurality of networks is further based on the comparison of the estimated Mathis coefficient data values ​​to the reference range.

10. 10. A packet communications controller as claimed in claim 7, wherein the processor is configured to perform a flow classification of the unallocated data packets, and based on the flow classification, the predefined most important characteristic and the predefined second most important characteristic are identified.

11. 1. A method for packet communications using a packet communications controller coupled to a plurality of network connections, each network connection having a corresponding output queue from a plurality of output queues, comprising: monitoring and maintaining, in a data object, a network characteristic associated with each network of the plurality of networks, such that each network of the plurality of networks may be ordered in an ordered sequence based on the network characteristic; When it is determined that a trigger event has occurred, establishing the ordered sequence for network connections of a selected set of network connections based on the network characteristics; allocating one or more of the unallocated data packets of an input queue to one or more corresponding output queues of the plurality of networks by allocating the one or more data packets to successive available network connections based on availability and the ordering sequence; updating the monitored characteristics of the corresponding plurality of networks to reflect the assigned one or more data packets; and communicating the assigned data packets from the plurality of output queues to a destination device.

12. The packet communication controller is a push scheduler, and the method further comprises: measuring a transmission capacity of each of the plurality of network connections; 12. The method of claim 11, wherein the allocating of the one or more data packets comprises allocating the one or more data packets to a corresponding one of the plurality of network connections only if the transmission capacity of the corresponding one of the plurality of network connections is not exhausted.

13. 12. The method of claim 11, wherein the allocation of the one or more data packets occurs upon detecting a new data packet in the input queue or upon receiving an acknowledgement that one or more previous data packets have been received at the destination device.

14. For each of the plurality of network connections, the monitored characteristic is estimating at least one of a transmission rate, a round trip time, and a round trip propagation time of a bottleneck link; comparing the at least one of an estimated round trip propagation time, the estimated round trip time, and the estimated transmission rate to a respective reference range of at least one of a round trip propagation time, a reference range of a round trip time, and a reference range of a transmission to establish a score value for each of the at least one of the estimated round trip propagation time, the estimated round trip time, and the estimated transmission rate under consideration; The method of claim 11 , further comprising: ordering the plurality of networks in the ordered sequence based on the score value for each of the plurality of networks.

15. 15. The method of claim 14, wherein the estimation of at least one of a transmission rate, a round trip time, and a round trip propagation time of a bottleneck link includes an estimation of a plurality of score values ​​for at least two of the transmission rate, the round trip time, and the round trip propagation time of the bottleneck link, and the ordering sequence is based on a comparison of the score values ​​corresponding to predefined most important characteristics, and if the score values ​​corresponding to the predefined most important characteristics are equal, proceeding to a next most important predefined characteristic for comparison.

16. 16. The method of claim 15, further comprising: performing a flow classification of the unassigned data packets, wherein the predefined most important characteristic and the predefined second most important characteristic are identified based on the flow classification.

17. measuring modified monitored characteristics, including system overhead processing time, for each of the plurality of network connections; and using either the original characteristics or the modified characteristics; determining at least one of an estimate of the transmission rate, the round trip time, the round trip propagation time, a bandwidth-delay product (BDP), and a congestion window of a bottleneck link; 15. The method of claim 14, wherein the ordered sequence of the plurality of networks is further based on the at least one of the estimates of the transmission rate of a bottleneck link, the round trip time, the round trip propagation time, the BDP, and the congestion window.

18. 20. The method of claim 17, further comprising: for each of the plurality of network connections, modifying the measured characteristics by dynamically adjusting a duration of one or more corresponding sliding window histories based on other measured characteristics including at least one of packet loss in the round trip time or a change in the round trip time.

19. estimating a Mathis coefficient data value indicative of a relative upper limit of achievable throughput for said network connections of said plurality of network connections, taking into account round trip times and packet loss for said network connections; comparing the estimated Mathis coefficient data value to a reference range of Mathis coefficient data values ​​to determine a Mathis coefficient score; The method of claim 14 , wherein the ordered sequence of the plurality of networks is further based on the comparison of the estimated Mathis coefficient data values ​​to the reference range.

20. A computer-readable recording medium having computer-interpretable instructions stored thereon for causing a processor to perform an operation, comprising: The operation, 1. A method for packet communications using a packet communications controller coupled to a plurality of network connections, each network connection having a corresponding output queue from a plurality of output queues, comprising: monitoring and maintaining, in a data object, a network characteristic associated with each network of the plurality of networks, such that each network of the plurality of networks may be ordered in an ordered sequence based on the network characteristic; When it is determined that a trigger event has occurred, establishing the ordered sequence for network connections of a selected set of network connections based on the network characteristics; allocating one or more of the unallocated data packets of an input queue to one or more corresponding output queues of the plurality of networks by allocating the one or more data packets to successive available network connections based on availability and the ordering sequence; updating the monitored characteristics of the corresponding plurality of networks to reflect the assigned one or more data packets; and communicating the assigned data packets from the plurality of output queues to a destination device. A computer-readable recording medium.

Images