System and method for push-type data communication

Abstract

A data packet communication system is described that may be implemented as a physical push-type data packet communication device, such as a router, a gateway, or a controller circuit coupled to a router or gateway adapted to control data packet communication, the data packet communication system adapted to assess network capacity of each of a plurality of networks at the time of a monitored communication event and allocate data packets accordingly.

Description

Technical Field

[0001] Cross-reference This application is a non-provisional application of U.S. Patent Application Nos. 63 / 139286 and 63 / 183952, both entitled SYSTEMS AND METHODS FOR PUSH-BASED DATA COMMUNICATIONS, filed on January 19, 2021 and May 4, 2021, respectively, and claims all benefits including these priorities, which are hereby incorporated by reference in their entirety.

[0002] This application incorporates by reference the entire following applications: PCT Application No. PCT / CA2017 / 051584, filed on December 21, 2017, entitled "PACKET TRANSMISSION SYSTEM AND METHOD".

[0003] PCT Application No. PCT / CA2018 / 051012, filed on August 22, 2018, entitled "SYSTEM AND METHOD FOR ASSESSING COMMUNICATION RESOURCES".

[0004] PCT Application No. PCT / CA2020 / 051090, filed on August 7, 2020, entitled "SYSTEMS AND METHODS FOR MANAGING DATA PACKET COMMUNICATIONS".

[0005] Embodiments of the present disclosure relate to the field of electronic data packet communications, and more specifically, embodiments relate to devices, systems, and methods for push-based data packet communications where multiple network connections are available.

[0006] Introduction Existing data packet communication solutions for transmitting data packets across multiple available network connections rely on static routing logic to determine how data packets are transported through connections. For example, a system could employ a timer for each available connection, using each timer to determine when to pull packets from an input queue for transmission over the connection associated with that timer. These solutions assume that these connections have similar operating characteristics and availability, trading performance for simplicity of implementation, and schedule the next available data packet for transmission over the connection with the most recently timed-out timer. While the timer approach is simple, it can be computationally inefficient in certain situations.

[0007] Communication systems that treat multiple network connections as multiple instances of connections with similar operating characteristics (e.g., LACP, ML-PPP) may lead to inefficient use of multiple connections if the networks have different operating characteristics, as certain networks may be underutilized or overutilized. This inefficiency in use can result in higher latency, less reliable data packet delivery, higher cost data packet delivery, and reduced throughput.

[0008] Connection management solutions also encounter difficulties in handling applications where the available connections or the operational characteristics of those connections change over time. For example, timer-based systems may not accurately reflect the network performance of all other connections because each connection's timer is triggered independently, making them insensitive to changes in network performance.

[0009] A system and method for data communication that utilizes multiple connections more efficiently to transmit data packets with lower latency and increased throughput, thereby enabling more reliable or less expensive data packet delivery, is desirable. [Overview of the project] [Means for solving the problem]

[0010] The systems and methods described herein include push-type data packet communication systems, which may be implemented as physical push-type data packet communication devices, such as routers, gateways, or controller circuits coupled to routers or gateways adapted to control data packet communication. In other embodiments, the system may be implemented as software for use with electronic hardware, such as a communication controller board having a processor, or as software residing in memory as an instruction set executed by electronic hardware. Various embodiments described herein are for operating the push-type data packet communication system based on monitored data communication characteristics (e.g., network traffic) associated with a connection among multiple connections to improve data communication (e.g., latency, reliability, throughput, error correction) across multiple connections. Corresponding computer-readable media (e.g., non-temporary computer-readable media storing machine-interpretable instruction sets for execution on the processor) that operate in conjunction with data storage, computer processors, and computer memory are also provided.

[0011] The embodiments described herein offer technical improvements over alternatives such as the use of static routing tables and pull scheduling (e.g., the use of connection-specific timers that are not necessarily synchronized). For example, the Linux kernel TCP stack uses a "pull" model to reduce the recovery time of retransmitted packets. See: https: / / groups.google.com / g / bbr-dev / c / d14XAcuTBxo / m / Pwui1o3MBAAJ.

[0012] The push-type data packet communication system described herein is adapted to provide a router that can be adapted to connection (link) aggregation for low-granularity packet-type data distribution, which is made possible by real-time measurement of connection characteristics related to mixed connections to remote peers.

[0013] As described in various embodiments herein, a multipath connection aggregation system (alternatively referred to as a mixed connection system) can be modeled as a three-stage data pipeline having a first input queue and scheduling stage, a second connection characteristics monitoring stage (e.g., providing congestion control and packet buffering and encapsulating pacing instructions), and a third transmission stage for delivering packets to a network interface (e.g., according to the pacing instructions provided in the second stage).

[0014] A push-type data packet communication system includes a scheduler configured to queue data packets for transmission in response to monitored communication events (also called trigger events). Exemplary trigger events (other than timers) include ACK / NACK of previously transmitted WAN packets, arrival of new data in the input queue, and lack of feedback over a given period of time. The scheduler allocates data packets to various networks based on the monitored characteristics at the time of the trigger event.

[0015] The monitored characteristics of an individual connection are a subset of the connection's operational characteristics and may include, but are not limited to, estimated throughput, measured and / or expected latency, measured packet loss, and other characteristics that may be derived from measured or expected values. In a clear example, operational characteristics may include estimated throughput, measured and / or expected latency, measured packet loss, and other characteristics that may be derived from measured or expected values, while the monitored characteristics may be limited to estimated throughput.

[0016] In contrast to a pull scheduler, a push scheduler evaluates the monitored characteristics of each of multiple networks at the time of a monitored communication event and allocates data packets to connections with available transmission capacity. In exemplary embodiments, the push scheduler takes into account, in addition to available transmission capacity, data packet (application) requirements, network ordering preferences configured by the administrator, and monitored characteristics of the connection (other than transmission capacity). The push scheduler continues to allocate data packets to available networks if there is available transmission capacity associated with the network (available transmission capacity is continuously updated to reflect allocated data packets) and there are data packets available for transmission. In this way, the push scheduler can utilize the available transmission capacity of multiple networks to a greater extent than a pull scheduler.

[0017] One aspect of a push scheduler is a method of configuring it to computationally determine or establish a sort order for controlling the routing of data packets to various network connections. Factors influencing the determination of the sort order include keeping the sort order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (splitting increases the machine of self-induced out-of-order and jitter events), and taking into account both measurable and unmeasurable external directives of the connections.

[0018] The characteristics measured may include packet loss between three separate sliding windows (500ms, 3s, 30s) (further aggregated as the worst value between the three windows), current network round-trip time (RTT), bandwidth and round-trip propagation time (BBRv1) parameters, RtProp (network propagation delay), and BtlBw (throughput estimate). External directives may include priority (PR) rules, among other things, but may also include flow-specific rules / requirements.

[0019] In some embodiments, the measured properties of BBRv1 differ from those designed in the original BBRv1 IETF draft. For example, considering embodiments that may operate on systems with low system execution time consistency, the size of the sliding window used to compute BtlBw may vary either statically or dynamically. Also, for similar execution consistency reasons, the input parameters used to compute RtProp may be slightly modified.

[0020] Specific improvements based on improving networking routing order to control the use of network interfaces are described in several embodiments that predict throughput based on the Mathis Equation to establish an upper limit on TCP throughput given RTT, packet loss, and MSS by utilizing the combined characteristics of round-trip time (RTT) and packet loss. The Mathis factor can be used to modify the sort order for routing networked packets.

[0021] In some embodiments, push scheduling can also be configured to implicitly cause a sort order along one or more implicit sort dimensions over a certain period of time. The implicit sort dimensions force connections that encounter periodic bad events (e.g., latency spikes due to congestion) to bubble at the bottom of the list. Connections with consistent behavior naturally bubble at the top and remain near the top.

[0022] Pull scheduling does not allow for increased network capacity utilization because pulls occur based on connection-specific timers that are triggered independently of each other. A pull scheduler system includes a shared pull scheduler and timers associated with each network in a group of available networks, each timer having a wake-up value. When a timer wakes up and pulls packets from the pull scheduler, the decision of which packets to schedule is based solely on the monitored characteristics of the current connection, because the potential wake-up times of other connections are unknown. The resulting packet scheduling decisions are locally optimal for each individual connection at the time of the pull event, but may result in a globally suboptimal decision where packets scheduled in an earlier pull event on a certain connection might have been more optimally scheduled on a different connection where the pull request was imminent. The pull scheduler is configured to pull a quantity of packets from the input queue based on the estimated bit rate of the corresponding network and the duration between each timer wake-up (e.g., 1 / 50Hz, i.e., 20ms).

[0023] A technical challenge associated with push schedulers is ensuring that data packets belonging to a particular data flow (e.g., packets containing audio files, video, etc.) are transmitted over a single network or a set of relatively stable networks sharing similar monitored characteristics that can satisfy the flow requirements. For example, transmitting audio packets belonging to a single data flow across two networks with different latencies requires a sequencer (i.e., hardware or software responsible for reconstructing packets after transmission) to buffer and reorder the audio packets, potentially adding latency or failing to deliver the data packets in their original order, resulting in applications perceiving high-latency, inconsistent, and inadequate networking encounters. Push schedulers, by grouping connections with similar monitored characteristics, provide more direct control over assigning data packets to the connection that best satisfies the flow requirements.

[0024] In some embodiments, the proposed push scheduler is configured to order multiple networks based on a comparison between a monitored characteristic (e.g., round-trip time, bottleneck bandwidth, etc.) and a baseline range of network operating characteristics (e.g., a baseline range of transmission), where the range is defined to be insensitive to minor variations in the measured operating characteristics. The use of a range allows the push scheduler to continue responding to changes in the monitored characteristics of the network while building some degree of insensitivity to smaller variations in the monitored characteristics.

[0025] The proposed push scheduler can be configured to allocate data packets to multiple networks based on the ordering of the data packets and to facilitate allocating the data packets belonging to a particular flow to a single or relatively stable set of networks based on the monitored characteristics of the network. In an exemplary embodiment, the push scheduler stores the ordering and continues to transmit packets according to the ordering until the ordering is updated as a result of new or updated monitored characteristics.

[0026] The monitored characteristics can include a capacity factor that is an aggregated value at least partially defined by dividing the bottleneck bandwidth by the round-trip propagation time. The capacity factor can indicate a connection that achieves a higher bandwidth per millisecond of round-trip propagation time, indicating an efficient network. Further, the capacity factor can be determined by estimating the bottleneck bandwidth to the nearest order of magnitude (e.g., 1, 10, 100, 1000, 10,000, etc.) or an integer multiple thereof (e.g., 2, 3...20, 30, etc.) so that the capacity factor responds only to relatively large changes in the measured bottleneck bandwidth value. Similarly, the capacity factor can be determined by reducing the sensitivity of the capacity factor to variations in the measured round-trip propagation time using a bounded round-trip propagation time associated with the measured round-trip propagation time.

[0027] In an exemplary embodiment, the proposed push scheduler may further order the multiple networks based on user input criteria. For example, the ordering may be at least partially based on a user who has assigned a priority to each of the available networks, thereby enabling the user to provide a preference regarding the order in which the networks should be used based on external factors such as cost, information regarding future reliability or availability that cannot be measured or predicted.

[0028] In response to the technical challenge of utilizing a greater amount of existing network transmission capacity, the push scheduler may be configured to identify and / or utilize the transmission requirements associated with data packets queued for transmission by using a flow classification engine (e.g., the system may identify a particular data flow type and receive criteria that can be used to utilize predefined transmission requirements associated with the identified data flow type). By identifying or utilizing the transmission requirements of data packets and comparing these transmission requirements to the monitored characteristics of multiple networks, the push scheduler can allocate data packets to a network that matches the required transmission requirements in order to maximize the amount of data packets transmitted over a network with suitable monitored characteristics. It may still be possible to transmit packets over a network whose monitored characteristics do not match the transmission requirements, but this can result in applications that interpret the "delayed" or out-of-order delivery of packets as network degradation, thereby generating packets that degrade the quality of the service being attempted. Exemplary transmission requirements can include, among other things, the latency required throughout.

[0029] In an exemplary embodiment, the push scheduler may use flow classification engine packets within a flow having a set of transmission requirements assigned to each flow. In another embodiment, each flow inherently has transmission requirements that form the characteristics of the flow. This flow mapping is based on one or more elements within the header of the data packet or associated with the data packet. These flow transmission requirements can then be used to allocate a packet to one of a plurality of connections having equivalent monitored characteristics. In an example where a data packet is classified as a VPN data flow, it has an inherent requirement that the data be delivered in order, and these requirements can include flow transmission requirements that are transmitted over a connection having a particular minimum reliability.

[0030] For example, a TCP / IP header, identified as a TCP / IP 5-tuple (e.g., source and destination IP addresses, source and destination port numbers, and protocol), may determine that a data flow is a virtual private network (VPN) data flow, which is an ordering-sensitive data flow. As a result, the scheduler may receive transmission requirements from a flow classification engine that require all data packets in the VPN data flow to be delivered sequentially, regardless of the presence of other indicators, such as DSCP tags, on individual packets that may conflict with this requirement. For example, some packets may have DSCP tags that require low latency, while others may require high throughput. The scheduler may map (assign) packets to connections that have monitored characteristics that satisfy the specific transmission requirements of the DSCP tags, as long as they do not violate the ordering requirements of the encapsulated VPN data flow.

[0031] In another example, a push scheduler could use a flow classification engine to map packets to flows where transmission requirements prioritize low latency over reliability or throughput (for example, the source might indicate a type of data, such as a live video stream, that can be sensitive to increased latency).

[0032] In a further exemplary embodiment, the push scheduler may assign priority transmission requirements to individual data packets, independently of the transmission requirements derived from flow classification. For example, the push scheduler may be configured to assign the highest priority to packets that are queued for retransmission (i.e., packets that have been previously transmitted and failed).

[0033] A technical challenge associated with push schedulers that implement data flow or data packet priority transmission requirements is that non-urgent data packets can be continuously moved out of the transmission queue, potentially leading to a "starvation" of non-urgent packets in the system.

[0034] A push scheduler can be configured to pace the rate of packet transmission by allocating a nominal wall clock time for transmission to each data packet in response to a hunger for non-urgent packets. In an exemplary embodiment, packets are allocated wall clock time based on an estimated bottleneck bandwidth and the size of the data packet to prevent congestion.

[0035] The network output queue to which data packets are assigned in the pipeline stage may be configured to transmit data packets according to the nominal wall clock time to which the data packets are assigned, without prioritizing priority packets and moving non-urgent packets. For example, at time zero, a non-urgent data packet may be assigned a nominal wall clock time for transmission at time 3. Urgent data packets may be scheduled for transmission at time 3 based on the amount of available transmission capacity on the network, and this available transmission capacity is reduced to accommodate non-urgent data packets.

[0036] Continuing with the example, a push scheduler may be configured to queue non-urgent data packets for transmissions after a nominal wall clock time. For example, if there is network capacity available at time 0 and time 2, the push scheduler may send a non-urgent data packet with a timestamp of time 3 for transmission at time 2. Subsequently, the non-urgent data packet may be rescheduled for transmission at time 3 in response to receiving a packet with a more urgent nominal wall clock time.

[0037] A technical challenge associated with push schedulers is that, by pushing data packets based on estimated available network transmission capacity, the system may lose robustness in the event of network failures or unacceptable levels of network congestion.

[0038] A push-type scheduler may be configured to monitor network operating characteristics associated with each available network and determine whether the monitored characteristics meet a congestion threshold. In an exemplary embodiment, the congestion threshold is established from a baseline profile associated with each network. For example, the minimum latency measured for packets transmitted over a particular network can form a baseline value for the network, and subsequent transmissions along that network can be compared to this baseline value. In another exemplary embodiment, the threshold may be based at least in part on a baseline of packet round-trip times. In other embodiments, the baseline may be calculated using statistical methods more complex than simply taking the minimum latency.

[0039] To increase the robustness of the push scheduler, the scheduler may be configured to reduce the amount of transmission capacity defined as available for transmission, or to update the available transmission capacity in a manner other than following each monitored communication event, in response to detecting a network failure or an unacceptable amount of network congestion. In exemplary embodiments, the network's transmission capacity may only be updated in response to receiving an acknowledgment that previously transmitted data packets have been received (e.g., in response to receiving an ACK). According to some embodiments, for example, the defined transmission capacity is reduced to a theoretical transmission capacity that does not include queues of packets with no movement in the buffer associated with the network. In further exemplary embodiments, the defined transmission capacity is reduced to be equal to the amount of in-flight data packets across a particular network.

[0040] In some exemplary embodiments, one of several networks operates according to the Bandwidth On-Demand (BOD) principle, where a network administrator and / or an automated network monitoring agent allocates available bandwidth for transmission to a transmitter based on previous transmissions or previous requests for bandwidth. This allocation may increase or decrease the allocated bandwidth depending on the administrator's or agent's prediction of subsequent bandwidth usage. During this bandwidth allocation process, packet loss and / or latency may temporarily increase, requiring the scheduler to make more intelligent decisions about allocating packets to connections. For example, a probing packet used to request more bandwidth may be marked with a DSCP tag indicating that the probing packet has a lower priority and is therefore a candidate to be dropped first if necessary.

[0041] A technical challenge associated with push schedulers pushing data packets to bandwidth-on-demand networks is that bandwidth-on-demand networks rarely update the bandwidth available for transmission, making it difficult to utilize available capacity. For example, a bandwidth-on-demand network may have available capacity, but as a result of allocation delays, this capacity may become unused. Alternatively, if the push scheduler fails to consistently utilize sufficient capacity on the network, the administrator / agent may choose to reduce the allocated capacity, potentially resulting in a significant change in the monitored characteristics of the network.

[0042] A push scheduler may be configured to define the transmission capacity of a bandwidth-on-demand network as greater than the received allocated transmission capacity, store the excess transmission capacity in a buffer associated with the bandwidth-on-demand network, and over-advertise the required amount of bandwidth to the bandwidth-on-demand network administrator. As a result of over-advertising the required bandwidth, the bandwidth-on-demand network may begin allocating a larger amount of unused bandwidth to the push scheduler. Furthermore, as a result of defining the transmission capacity as greater than the received allocated transmission capacity, the push scheduler can ensure that data packets are available for transmission to take advantage of any subsequent increases in the received allocated transmission capacity.

[0043] A further technical challenge associated with pull scheduler systems is the difficulty in fairly allocating data packets with specified requirements to transmission. For example, if a certain type of data packet requires a connection with a specific latency (e.g., a video feed), a pull scheduler system may bypass or delay that packet until an available network that meets the latency requirement wakes up. In this way, latency-sensitive data packets are delayed depending on when they enter the input queue, potentially exacerbating latency problems.

[0044] Rate-based congestion control typically adjusts the rate based on delay and loss. Congestion detection needs to be performed with a certain time lag to avoid overreacting to spurious congestion events such as delay spikes. Despite the fact that there are two or more congestion metrics, the result is that there is still only one mechanism for adjusting the transmit rate. This makes it difficult to simultaneously achieve the goals of high throughput and rapid response to congestion. [Brief explanation of the drawing]

[0045] In the figures, embodiments are shown as examples. It should be expressly understood that the descriptions and drawings are for illustrative purposes only and for aiding understanding only.

[0046] Herein, embodiments will be described only by reference to the accompanying drawings, for illustrative purposes only. [Figure 1A] This is a schematic diagram of a pull communication system. [Figure 1B] This is a schematic diagram of a push communication system according to several exemplary embodiments. [Figure 1C] This is a flowchart of a method for transmitting data packets to push packets to multiple networks based on availability, according to several exemplary embodiments. [Figure 1D] This is a flowchart of another method for transmitting data packets to order connections based on connection metadata, according to several exemplary embodiments. [Figure 1E] This is a flowchart of a further method for transmitting data packets based on a selected order of networks for allocating packets to a network, according to several exemplary embodiments. [Figure 1F] This is a process flow illustrating, in several exemplary embodiments, the dimensions used to sort network connections for controlling the routing of data packets. [Figure 2] This is a schematic diagram illustrating the scheduling of data packets with different priority levels, based on several exemplary embodiments. [Figure 3] Figure 1B is a series of schematic diagrams of a push communication system that schedules data packets with different priority levels, according to several exemplary embodiments. [Figure 4A] Figure 1B is a schematic diagram of connection selection with the push communication system, according to several exemplary embodiments. [Figure 4B]Figure 1B is a schematic diagram of data packet selection using the push communication system shown in several exemplary embodiments. [Figure 4C] Figure 1B is a schematic diagram of data packet allocation using the push communication system, based on several exemplary embodiments. [Figure 4D] Figure 1B is a schematic diagram of another data packet allocation using the push communication system, according to several exemplary embodiments. [Figure 5A] Figure 1B is a schematic diagram of a push communication system that schedules two independent data flows for transmission, according to several exemplary embodiments. [Figure 5B] Figure 1B is a schematic diagram of a push communication system that schedules two dependent data flows for transmission, according to several exemplary embodiments. [Figure 6A] A corresponding screenshot of an interface for managing data packet transmission, according to an exemplary embodiment, is shown. [Figure 6B] A corresponding screenshot of an interface for managing data packet transmission, according to an exemplary embodiment, is shown. [Figure 6C] A corresponding screenshot of an interface for managing data packet transmission, according to an exemplary embodiment, is shown. [Figure 6D] A corresponding screenshot of an interface for managing data packet transmission, according to an exemplary embodiment, is shown. [Figure 6E] A corresponding screenshot of an interface for managing data packet transmission, according to an exemplary embodiment, is shown. [Figure 7] This diagram illustrates data packet transmission in several exemplary embodiments. [Figure 8A] Two diagrams illustrating unidirectional data packet transmission in several exemplary embodiments. [Figure 8B]Two diagrams illustrating data packet transmission in response to an acknowledgment, according to several exemplary embodiments. [Figure 8C] Two further diagrams illustrating data packet transmission without requiring an acknowledgment to send further data packets, according to some exemplary embodiments. [Figure 9A] This is a diagram of in-flight goodput data in several exemplary embodiments. [Figure 9B] Another diagram of in-flight goodput data, according to several exemplary embodiments. [Figure 9C] Further diagrams of in-flight goodput data by several exemplary embodiments. [Figure 10A] A series of diagrams illustrating data packet transmission where a connection encounters congestion, according to several exemplary embodiments. [Figure 10B] A series of diagrams illustrating data packet transmission after adjusting transmission capacity in response to connection congestion, according to several exemplary embodiments. [Figure 11] This is an illustrative schematic diagram of a computing device according to several embodiments. [Modes for carrying out the invention]

[0047] Systems, methods, and devices for transmitting data packets are described, which may be referred to as mixed routers or mixed systems, and which may be used as alternatives, as multiple connections are available for transmission.

[0048] Existing solutions for transmitting data packets across multiple available connections are typically implemented as a "pull" architecture, where data packets are transmitted in response to events generated or defined by the device or system, such as internal timers associated with the multiple available connections.

[0049] The internal timer can respond to various connection parameters (e.g., timer duration configured to reduce the recovery time of retransmitted packets), but it does not respond to connection parameters of other available connections.

[0050] As a result of the independence of timer events, pull-type systems can suffer from underutilization (e.g., idle connections if timer values ​​are too long) or overutilization (e.g., data packets may be allocated to unreliable connections despite insufficient performance), which can lead to greater latency, reduced throughput, unreliable data packet delivery, and more expensive data packet delivery.

[0051] Methods, systems, and devices for transmitting data packets according to a push architecture for transmitting data packets over multiple available connections are proposed herein. The disclosed methods, systems, and devices include a scheduler configured to take a subset of the operational characteristics (monitored characteristics) of each of the multiple connections in response to a monitored communication event, and to iteratively allocate data packets available for transmission to any of the connections determined to have transmission capacity based on the monitored characteristics of these connections. In exemplary embodiments, other monitored characteristics of the network may be used in conjunction with the transmission requirements associated with the data packets to allocate data packets to connections that match the transmission requirements of the data packets.

[0052] Scheduling data packets in a push-based architecture where multiple connections are available has not been used. Typical multipath systems (alternatively referred to as mixed connection systems) assign packets and flows to connections on a static basis, as they do not consider the flow requirements or operational characteristics of connections. For example, a multi-homed Internet Protocol (IP) router assigns packets to connections purely on the destination IP address in the packet header that matches against the router's destination routing table. It is simpler to implement a pull-based architecture when connection assignments do not change frequently. For example, there is no need to handle the case where a push scheduler has assigned a large queue of packets to a connection, but the connection goes offline and needs to be recalled or reassigned.

[0053] The proposed method, system, and device receive a data flow containing one or more data packets from an input network through an input connection.

[0054] The proposed method, system, and device include a three-stage data pipeline for transmitting data packets over a mixed set of connections to remote peers. New data destined for multiple connections is passed to a first pipeline stage, where the new data is queued and then passed to a second pipeline stage associated with each connection in the mixed connections. The second pipeline stage compiles statistics about the underlying connections and passes the packets to a third pipeline stage. The second stage may provide congestion control functionality and packet buffering to ensure that packets are available to the next pipeline stage. The third stage then writes the packets to the network interface, possibly according to pacing or timing hints provided by the second stage.

[0055] Another embodiment may have a single second stage instance that then provides packets to an instance of a third stage, each of which manages the transmission of packets on a set of network interfaces sharing a physical interface card. Other embodiments may exist with a different number of instances of each of the pipeline stages.

[0056] Figure 1A is a schematic diagram of a pull communication system 100A according to several exemplary embodiments.

[0057] When each connection timer (e.g., timers 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively) wakes up, the connection timers call the pull scheduler 130 to request data packets for transmission from the input queue 102. Data packets can be requested in bursts, with the burst size nominally reflecting bytes equivalent to 20ms (1 / 50Hz) at the estimated bitrate of the corresponding connection for each timer. A limitation associated with such pull timers is that the number of bytes requested at the wake-up of each timer does not reflect the full congestion window (e.g., representing the "capacity") of the connection.

[0058] Since each timer is activated independently, the pull scheduler 130 does not have visibility into the overall transmission state in order to make a globally optimal scheduling decision. For example, connection 110A may be a terrestrial broadband connection, and connection 110B may be a bandwidth-on-demand (BoD) satellite connection. Connection 110A may be preferable to connection 110B as a result of costs, latency, etc., but when connection 110B timer 108B wakes up and calls the pull scheduler 130, the pull scheduler 130 does not have visibility into the next wake-up time of connection 110A timer 108A (since timer 108A is activated independently). The pull scheduler 130 does not have a basis for determining whether connection 110A can process all the data packets in input queue 102, or whether further data packets should be allocated to connection 110B to complete the transmission.

[0059] If connection 110B is useful, the pull scheduler 130 does not need to have a means to determine which connection should be used to transmit which of the available data packets for transmission, since only the pull scheduler 130 has visibility to the timer.

[0060] The data packets are transmitted across the entire connection and then received by the receiver 116 via connection 114. The receiver 116 includes a sequencer 118 for assembling the transmitted data packets and may further transmit the assembled data packets via the local area network (LAN) 120.

[0061] In contrast to pull architectures, the proposed system may be able to schedule data packets to specific connections in order to increase efficient connection utilization.

[0062] Figure 1B is a schematic diagram of a push communication system 100B in several exemplary embodiments. In the exemplary embodiments, system 100B is designed to aggregate connection types that provide multiple access (MA) to a shared communication channel using multiplexing techniques. For example, orthogonal codes assigned to each node in a code division multiple access (CDMA) network allow those nodes to share a set of frequencies and transmit simultaneously without interfering with each other.

[0063] A characteristic of connection types using these multiplexing techniques is that the reallocation of available capacity to nodes in the network is rapid, typically on the order of tens to hundreds of milliseconds. In an exemplary embodiment, System 100B is designed with this characteristic in mind, reallocating traffic among available WAN connections on this timescale.

[0064] In contrast, connections using the Request Assignment Multiple Access (DAMA) technique, also known as Bandwidth on Demand (BoD), share a communication channel by having nodes negotiate access to the channel for a set period of time. Once allocated, nodes use the negotiated portion exclusively. Thus, the sharing of available capacity occurs sequentially through the nodes using the negotiated portion of the channel.

[0065] Using BoD connections, this negotiation and subsequent reallocation of capacity between nodes typically takes on the order of seconds (mid-single-digit to low-double-digit) to complete. In exemplary embodiments, system 100B would require modifications (described herein) to effectively utilize these types of connections.

[0066] Long reallocation times (with associated packet loss and / or increased latency) are interpreted as congestion in current implementations, and typical congestion control methods reduce usage on connections where congestion is detected to avoid overloading these connections. However, BoD connections actually require the opposite behavior (at least temporarily) to request and, in some cases, obtain increased capacity allocation.

[0067] System 100B includes an input queue 102 that receives data flows containing one or more data packets from an input network for transmission. Examples of data flows may include file transfers, video file transfers, audio file transfers, Voice over Internet Protocol (VoIP) calls, or data packets associated with a virtual private network (VPN).

[0068] The input queue 102 is also responsible for applying a drop policy to packets if the scheduler (as described herein) cannot service the input queue 102 at the same rate as new packets arrive from LAN-side clients. This typically occurs when LAN clients are transmitting at a rate higher than the aggregated WAN transmission capacity.

[0069] System 100B, following a push architecture, includes three stages for transmitting data packets to remote peers via a mixed set of connections, which are alternatively referred to as pipelines or pipeline stages.

[0070] The first stage 104 includes a flow classification engine 104A and a scheduler 104B (hereinafter referred to as the push scheduler 104), which receive data packets from the input queue 102 and queue the data packets to pass them on to the second stage 106. In an exemplary embodiment, the first stage 104 includes the input queue 102, the flow classification engine 104A, and the scheduler 104B.

[0071] The input queue 102 is responsible for buffering packets received from clients on the sender's LAN side and, if necessary, classifying data packets into flows (alternatively referred to as mapping data packets to flows) by using or cooperating with the flow classification engine 104A. The input queue 102 ensures that data flows are served fairly by the scheduler in the first stage 104.

[0072] In exemplary embodiments, the first stage 104, or the assisting flow classification engine 104A of the first stage 104, monitors for or is notified of changes to user configuration settings associated with the assignment of data packets to connections. User configuration settings include information on how connections should be assigned to priority levels when those priority levels should be active, as well as user-configurable flow classification rules that may cause packets belonging to a flow to prefer connections having certain monitored characteristics (for example, packets belonging to the video stream class may prefer connections with low latency, while packets belonging to the file download class may prefer connections with lower monetary costs).

[0073] The first stage 104 is periodically notified of current metadata (e.g., operational characteristics) for each of the multiple connection monitoring instances (as described herein) corresponding to the WAN connection, including mixed links. This metadata is collected by monitoring instance 106 and may include the estimated bottleneck bandwidth of the connection, the estimated minimum round-trip time, the most recent actual round-trip time, the current congestion window for the connection, and the receive / loss status for packets previously sent to the next stage. Using this metadata, the first stage 104 may also extend the metadata to track the estimate of bytes that the first stage 104 has scheduled as in-flight on this connection, and further subdivide the total into new bytes, retransmitted bytes, or dummy / probing bytes. All metadata collected and tracked by the first stage 104 is referred to as the monitored characteristics.

[0074] Next, using all available monitored properties, the first stage 104 determines whether the packet should be sent to the next stage, and if so, which connection should receive the packet. The set of packets provided to a connection monitoring instance may include duplicates of packets previously sent (to either the same connection monitoring instance or a different connection monitoring instance). Alternatively, the scheduler may determine that a given connection monitoring instance is not currently eligible to receive the packet, even if its connection currently has capacity. The allocation of packets to specific stage 3 instances is discussed elsewhere in this document.

[0075] The second stage 106 includes connection monitoring instances for each of the multiple available connections (e.g., connection monitoring instances 106A, 106B, and 106N associated with connections 110A, 110B, and 110N, respectively). The connection monitoring instances in the second stage 106 monitor the associated connections and determine or obtain statistics associated with the connection's performance (e.g., latency), known as operational characteristics. The second stage 106 may provide congestion control functionality and packet buffering to ensure packets are available for the next stage. The second stage 106 passes packets to the third stage 108, which includes output queues for each connection (e.g., output queues 108A, 108B, and 108N). The third stage 108 then writes the packets to the network interface, possibly according to pacing or timing hints provided by the second stage 106.

[0076] In some embodiments, the second stage 106 may implement one or more R2RI protocols (e.g., PPPoE (RFC5578), R2CP, DLEP (RFC8175)) to better support the bandwidth allocation process for the BoD connection.

[0077] The second stage 106 is responsible for receiving packets from the first stage 104, queuing these packets locally, and providing groups of packets as bursts to the next pipeline stage. The second stage 106 also tracks metadata about the associated connection of the second stage 106 based on acknowledgments received from peers or receivers.

[0078] The purpose of the queue in the second stage 106 is to ensure that there are always packets available for the next pipeline stage to write to a network device for transmission. Some embodiments may include priority metadata as part of each packet information, and as a result, new packets arriving from the first stage 104 may be reordered in the queue to provide preference to higher-priority packets.

[0079] To ensure that the second stage 106 buffers enough packets so that packets can be served to the third stage 108, the second stage 106 may provide modified metadata to the first stage 104. For example, according to some exemplary embodiments, the second stage 106 may choose to report a larger congestion window (i.e., "over-advertise") so that the first stage 104 can provide additional packets that can be buffered in the second stage 106 for when the third stage 108 has finished writing the actual congestion window packets.

[0080] The second stage 106 may also provide modified metadata to the first stage 104 to facilitate changes in capacity allocation on the BoD connection (i.e., to cause the first stage 104 to schedule packets on this connection as if it had a higher capacity in order to trigger the BoD connection to allocate a higher capacity). This modified metadata may include information that breaks down the higher capacity into different requested types / priorities. For example, the higher incremental capacity used to probe the BoD connection typically consists of redundant / dummy packets, since the probe packets are likely to be dropped by the BoD connection.

[0081] Advertising or over-advertising capacity from the second stage 106 to the first stage 104 does not necessarily require immediate probing for higher incremental capacity. The first stage 104 still ultimately makes the decision on how packets are allocated to connections. In one embodiment, the system may wait until all CWNDs on non-BoD connections are fully consumed (or nearly fully consumed) before instructing the second stage 106 to begin the DAMA reallocation process.

[0082] The queue in the second stage 106 also allows for rapid notification of failures to previous stages if something happens to the underlying network connection. Packets provided to the third stage 108 and written to the network must be assumed to be in transit even if the network interface subsequently reports down, meaning that a timeout must occur before those packets can be marked as lost. However, packets still queued in the second stage 106 can be marked as lost immediately once the second stage 106 is notified that the connection is no longer available, thereby making these packets immediately eligible for preferred retransmission.

[0083] Before proceeding to the third stage 108, some embodiments may attempt to explicitly pace the rate of packet transmission by assigning a nominal timestamp to each packet indicating when the next stage should attempt to transmit the packet over the connection. Such a timestamp may be determined based on the estimated bottleneck bandwidth of the connection (e.g., the connection to which the packet is assigned) and the packet size, in order to prevent network congestion at or before a network bottleneck link.

[0084] Figure 2 shows an example of an embodiment in which, in 200, the second stage 106 supports preferred packets. In Figure 2, the head of the queue is shown on the right and the tail of the queue is shown on the left. In the upper part of Figure 2 (e.g., the first step), the first stage 104 provides the connection monitoring instance of the second stage 106 with a new list of preferred packets. The lower part of Figure 2 (e.g., the second step) shows that these new packets have been added to the priority queue in order of priority, so that all priority 1 packets are dequeued before any priority 2 packets are dequeued. Packets of the same priority are retained in sequence order.

[0085] In exemplary embodiments not shown, the second stage 106 may have a single connection monitoring instance that then provides packets to instances of the third stage, each of which manages the transmission of packets on a set of network interfaces sharing a physical interface card. Other embodiments may exist with a different number of instances of each of the pipeline stages.

[0086] Feedback in the form of metadata or callbacks may be sent from one stage to the previous stage, or a stage may receive notifications from a different part of the system. Any of these actions may trigger (i.e., push) a stage to send a packet to the next stage.

[0087] The third stage 108 receives packets from the second stage 106 via one or more connection transmission instances (for example, connection transmission instances 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively) and writes these packets to the network 112. In some embodiments, all connection monitoring instances of the second stage 106 may provide packets to a single connection transmission instance, or each connection monitoring instance may be associated with a connection transmission instance.

[0088] Some embodiments of the third stage 108 may use a queuing mechanism to allow the previous stage to provide hints about when a packet should be sent. For example, the hint may come in the form of a “nominal transmit time” associated with the packet as metadata. If the next packet to be sent has a later nominal transmit time, the third stage 108 will wait until that time before sending that packet. Urgent packets may be flagged with an immediate timestamp that allows them to be sent before other queued packets. To prevent starvation of non-urgent packets, some embodiments may queue urgent packets after non-urgent packets with past nominal transmit times.

[0089] Some embodiments may include a callback function as part of the packet metadata, which can be used to trigger a notification to the second stage 106 indicating that a packet has been sent. Such a notification may be used to trigger a new batch of packets to be pushed from the second stage 106 to the third stage 108.

[0090] Figure 3 shows the time progression of an embodiment in 300 that includes a “Transmit Time” hint (nominal transmit time) requiring that packets provided by the second stage 106 not be transmitted before a specified time. Packets with a “Transmit Time” of 0 should be transmitted as soon as possible. Figure 3 illustrates examples for all of these cases (e.g., the use of past nominal transmit times for urgent packets, and how the system avoids starvation of non-urgent packets).

[0091] In the state depicted at t=1, the packet queue of the third instance of stage 108 contains four packets, three of which are currently eligible to be sent, and the second instance of stage 106 is about to provide four new packets, the first of which has sequence 5 (sequence order 5) and should be sent immediately.

[0092] In the state depicted at t=2, the new packet is added to the priority queue of the third stage 108. Note that in this embodiment, since sequence 2 was already eligible to transmit when sequence 5 was queued, it was chosen to queue the packet in sequence 5 to transmit after the packet with sequence 2. Freezing the packet order at the head of the packet queue in this way prevents starvation, where a constant stream of new urgent packets to be transmitted immediately could prevent non-immediate packets from being transmitted even though they may have become eligible to transmit long before the most recent urgent packets arrived.

[0093] In the state depicted at t=3, the third stage 108 dequeued the first four packets from the output queue and provided these packets to the network interface. Currently, there are no packets eligible to send until time t=4, so the third stage 108 instance will have to wait until time t=4, or until new packets arrive that should be sent immediately or at time t=3, even if there is currently write capacity on the network interface, before sending more packets.

[0094] Packets can be passed from one stage to the next in order, at the point when each stage determines it is optimal.

[0095] Similar to Figure 1A, data packets are transmitted across the entire connection and then received by receiver 116 via connection 114. Receiver 116 includes a sequencer 118 for assembling the transmitted data packets, and the assembled data packets may be further transmitted via the local area network (LAN) 120.

[0096] In contrast to system 100A, push communication system 100B does not include an independent timer associated with each connection. Instead, packet scheduling onto the connections is performed at a centralized location and includes a push scheduler 104 that accesses the monitored characteristics of each connection and is activated in response to one or more monitored events occurring outside of the timer. For example, one or more monitored communication events may include acknowledgments / negative acknowledgments (ACK / NACK) from the recipient of a previously transmitted data packet, or the arrival of a new data packet to input queue 102.

[0097] In an exemplary embodiment, not only the 20ms of the transmitted packet, but the entire connection (congestion window) CWND may potentially be consumed or allocated by the push scheduler 104.

[0098] Using access to the monitored characteristics of each connection, the push scheduler 104 may be configured to optimize packet scheduling decisions. For example, data flows with specific transmission requirements may be preferentially assigned to connections that can satisfy the requirements associated with the data flow, rather than being assigned largely based on the timer wake order. In an exemplary case, the first data flow requires transmission with a latency of less than 50 ms.

[0099] Connections 110A, 110B, and 110N have latencies of 5ms, 45ms, and 80ms, respectively. Using the pull scheduler 130, if the 50Hz timer for connection 110B (45ms latency) is woken up first, the timer will be able to schedule packets for this first data flow onto connection 110B. Using the push scheduler 104, data flows can be preferentially scheduled on connection 110A (5ms latency) until the CWND of connection 110A is completely consumed before it overflows onto connection 110B (45ms latency).

[0100] To enable bidirectional communication, a single instance of a multipath endpoint may include all the components shown in Figure 1B. Specifically, it may include one or more instances of sender 100 to support upstream communication to other multipath endpoints, and one or more instances of receiver 116 to support downstream communication from other multipath endpoints.

[0101] According to some embodiments, for example, one or more of the available connections are BoD connections, and the push scheduler 104 can allocate more data to the BoD connections in order to request more capacity from the network based on the following criteria:

[0102] While the CWND of connection 110A is fully consumed (or nearly fully consumed), more packets are available in input queue 102 that are eligible to be transmitted over connection 110B (for example, based on per-flow rules).

[0103] In an exemplary embodiment, the push communication system 100B can enable more accurate and fair quality of service (QoS) and active queue management (AQM) packet dropping rules applied to the input queue 102. Referring again to the previous example, a flow with a 50ms latency requirement can never utilize an 80ms connection. Since the push scheduler 104 has a consistent snapshot of the entire system, it can calculate the partial usage and fair share of the total capacity of the flow excluding the 80ms connection.

[0104] A potential advantage of the push scheduling method is that a complete set of metadata for multiple connections, including mixed links, is available to the push scheduler 104 in order to optimally allocate packets to the target connection.

[0105] Figure 1C is a flowchart of method 110C for transmitting data packets using system 100B or other push communication systems, according to several exemplary embodiments.

[0106] In block 1130, method 100C detects when the input queue contains unassigned data packets available for transmission.

[0107] In block 1132, method 100C includes assigning one or more unassigned data packets to one or more corresponding output queues of a plurality of networks having preferred monitoring characteristics.

[0108] In block 1134, method 100C includes updating the monitored characteristics of a plurality of corresponding networks to reflect one or more assigned data packets.

[0109] In block 1136, method 100C propagates data packets assigned from output queues corresponding to each network of a plurality of networks to a destination device.

[0110] In an exemplary embodiment, a method for transmitting data packets in system 100B includes a loop of the following steps whenever the push scheduler 104 is triggered by an external event. 1. Request data packets from input queue 102, 2. Assign the received packet to a connection for transmission. 3. Repeat steps 1-2 until a) there are no more packets available in input queue 102, or b) there is no more available transmission capacity on any of the active connections (for example, as defined by the Congestion Window (CWND) metric).

[0111] The described method may be implemented such that the push scheduler 104 does not consider any data flow-specific requirements. In such an embodiment, system 100B allocates received packets based on the ordering of connections.

[0112] The ordering can be based on the monitored characteristics of the connection, which may include, for example, a subset of the operating characteristics estimated by a congestion control algorithm. In one embodiment, the BBRv1 algorithm is used, but other embodiments may include examples such as BBRv2, TCP Reno, TCP LEDBAT, and proprietary implementations for handling specific connection types such as BoD. In the subsequent discussion of the operating characteristics specific to BBRv1, for illustrative purposes only, we refer to the pipe analogy, in which two endpoints are modeled as a series of interconnected pipes, each pipe segment may have a different diameter and length.

[0113] Operating characteristics can be a measure of the minimum time required for a transmitted packet to reach its destination endpoint (e.g., a receiver) and be acknowledged (e.g., via an ACK message from the receiver to the transmitter). This duration is called "round-trip propagation delay" (abbreviated as "RtProp") and is measured in units of time (e.g., milliseconds). In terms of pipe analogy, this value represents the total length of all pipe segments from the transmitter to the receiver.

[0114] The operating characteristic may be the maximum rate at which data can travel between endpoints along the pipe segment with the smallest diameter. The smallest pipe diameter, referred to as the “bottleneck” segment, determines the maximum rate at which data can travel between the transmitter and receiver, and this is also referred to as the “bottleneck bandwidth” (abbreviated as “BtlBw”). For example, the methods used to estimate BtlBw are described in the IETF draft: https: / / tools.ietf.org / html / draft-cheng-iccrg-delivery-rate-estimation-00.

[0115] Table 1 shows an exemplary set of WAN connections and WAN connection characteristics, an exemplary subset of operational characteristics selected by the scheduler 104 to be monitored characteristics, and the resulting sort order. Table 1: Exemplary WAN connections and sort order of WAN connections [Table 1]

[0116] This exemplary method sorts connections across the following dimensions and moves to the next dimension only if the current dimension is compared to be equal across connections. 1. Increase the preference priority level. 2. Increase RtProp 3. BtlBw decreases 4. Increase the interface name.

[0117] Since the sort dimension consists of all parameters that are not expected to change frequently, note how the result of the sort operation will keep the order of connections relatively stable.

[0118] Because this stable sort order is intentional, packets arriving at the input queue tend to be "sticky" to the same set of connections (those sorted at the top of the list), and only "overflow" to connections lower in the sort order if the incoming requests exceed the total CWND of the higher-sorted connections.

[0119] Note how the sorting criteria in this example are not flow-specific. For example, an administrator or end-user might want high-priority application flows to be transmitted only through metered cell 1 and cell 0 connections (an example of transmission requirements), meaning those flows would want a resulting sort order that places cell 1 and cell 0 lower in the list. Similarly, some flows may have maximum latency limits, such as VoIP calls where an end-user might consider the maximum latency limit unusable for interactive conversations if the latency exceeds 150ms. For these flows, it might be desirable to have a resulting sort order that uses RtProp as the primary sort dimension. Embodiments that extend the sorting criteria to be flow-specific will be discussed in subsequent paragraphs.

[0120] The scheduler is also responsible for determining the packet retransmission policy. In an exemplary embodiment, system 100B is configured such that the scheduler always marks a packet for retransmission if the packet is reported as missing by the receiver. While this is desirable for many flows, other flows may not want this at all (due to additional latency or potentially out-of-order packets rendering them unusable by the application), or they may only want retransmission attempts within a specific timeframe.

[0121] Figure 1D is a flowchart of another method for transmitting data packets, according to several exemplary embodiments.

[0122] One embodiment of a simple scheduling loop is as follows: 1. Perform any necessary accounting to update connection metadata and determine the set of active connections. 2. Determine the preferred ordering of connections based on connection metadata (monitored characteristics). 3. While there are packets to send and network capacity available for a set of active connections: a. Retrieve the next packet that is sent. b. Queuing packets sent on the first connection in an ordering with available capacity. c. Update connection metadata (monitored characteristics) to reflect scheduled packets. d. Repeat. 4. For each connection with queued packets, send those packets to the next pipeline stage for that connection.

[0123] Such embodiments treat all packets as equivalent for the purpose of preferred ordering of connections. In this case, packets can be considered to belong to the same class of flow (even if separate packets may logically belong to separate flows, from the scheduler's perspective, there is no distinction between these flows for the purpose of connection allocation).

[0124] Figure 1E is a flowchart of further methods for transmitting data packets, according to several exemplary embodiments.

[0125] An embodiment of a scheduler capable of distinguishing between flows of different classes (i.e., ICMP echo request / response versus video stream versus file transfer), and even between individual flows within the same class (video stream between host A and host B versus video stream between host A and host C), may be provided as follows: 1. In block 1102, perform any necessary accounting on connection metadata updates (monitored characteristics) to determine the set of active connections. 2. While the active connection set has capacity (block 1114), a. In block 1103, construct a selector based on the currently active connection set. b. In block 1105, a selector is used to choose which packet to send over one of the available connections. If no packet is selected, the loop is exited. c. In block 1107, the ordering of active connections is determined based on packet flow and flow classification, in combination with connection metadata (monitored characteristics). d. In block 1110, packets sent on the first connection in an ordering with available capacity are queued. e. In block 1112, update the connection metadata to reflect the scheduled packets. f. Repeat. 3. In block 1116, for each connection with queued packets, those packets are sent to the next pipeline stage for that connection.

[0126] These two sample methods (shown in Figures 1D and 1E) are similar, and the method shown in Figure 1D is a simplified and optimized version of the more general method shown in Figure 1E.

[0127] The first method can be transformed into the second method by defining a single flow class and a selector that matches all packets. With this in mind, we will discuss the second method (Figure 1E) in more detail, and the same discussion will be applicable to the first method (Figure 1D), subject to the above limitations.

[0128] The first step is to update any pending metadata accounting (monitored characteristics) to determine the set of active connections. In one embodiment, the set of active connections is the set of connections that are not subject to unreliable connection backoff (i.e., connections whose use is restricted due to recent failures in order to ensure packet delivery).

[0129] Other embodiments may further restrict the set of active connections by, for example, excluding connections whose metadata (monitored characteristics) do not match any of the currently defined flow class requirements (i.e., assuming that all packets must match one defined flow class, a connection may be considered inactive if it is not selected by any flow class; for example, if all flow classes specify an RTT of up to 500ms, connections with an RTT > 500ms may be considered inactive).

[0130] An active set of connections means that at least one connection in the set has the capacity to send packets if it has an available Congestion Window (CWND), and the number of in-flight bytes currently being tracked by the scheduler for this connection does not exceed the CWND determined for this connection. In one embodiment, the CWND for a connection may be the CWND reported by the next pipeline stage, or the scheduler may choose to reduce the reported CWND if throttling or priority routing is enabled.

[0131] In one embodiment, a connection throttled to a specific rate reduces the connection's CWND across the bottleneck bandwidth by the same coefficient as the throttle rate (or, if the throttle rate is higher than the bandwidth estimate, the CWND remains unchanged). Similarly, the CWND of a connection may be reduced if this CWND is subject to priority routing (where this embodiment supports connection grouping to a priority level that is activated when the total good put (throughput excluding overhead) of all connections at a higher priority level falls below a configured threshold for the level to be activated). Various methods for determining CWND reduction in these scenarios are described later in relation to Figures 8 and 9.

[0132] Connections assigned to an inactive priority level (where the total good put + unused bitrate for all connections at higher priority levels meets or exceeds the activation bitrate for that priority level) have a valid CWND of zero, and connections at a non-primary priority level have their CWND set to zero if the estimated total good put bitrate for all connections in the set of currently active priority levels meets or exceeds the activation threshold for the lowest active priority.

[0133] In other embodiments using priority routing, non-primary level connections may be capped at a rate lower than the bottleneck bandwidth of those connections so that they contribute only enough to close the gap to the configured activation threshold. This can be done for cost reduction, for example, if the non-primary connections are more expensive connections intended to be used only in emergencies. Similar to throttling, the CWND is reduced to reflect the capped rate at which these connections should be transmitted.

[0134] The loop in step 2 of the method can run as long as there is capacity in the active set of connections.

[0135] Step 2a (block 1103) constructs a selector used to find packets that are eligible to be sent on one of the active connections that have active capacity. In some embodiments, the selector consists of a set of flow classes that correspond to the current set of active connections that have capacity.

[0136] For example, if a flow class requires a connection with a maximum latency of 500ms, and all active connections with available capacity have a latency greater than 500ms, that flow class would be excluded from the selector. Some embodiments may choose to include flow classes in the selector that would not normally match the criteria for active connections if the embodiment can determine that there are currently no connections (active or inactive, with or without available capacity) that meet the criteria for such packets. In that case, the embodiment may choose to violate the matching criteria that favor sending packets rather than holding them until they time out and are dropped. Other embodiments may choose to enforce a strict matching policy in which packets matching flow classes that do not have eligible connections are not sent.

[0137] In another embodiment, as illustrated by the exemplary sort order in the process flow of Figure 1F, alternative methods can be used to computationally determine or establish a sort order for controlling the routing of data packets to various network connections.

[0138] Some or all of the input parameters can be rounded / bucketed before being used to determine the sort order. The technical improvement from rounding and bucketing is to keep the sort order as "stable" as possible so that changes in the sort order do not always occur (for example, without rounding and bucketing, two closed values ​​could cause "jitter" by having continuous sort changes as their values ​​fluctuate over time).

[0139] Factors influencing the determination of the sort order include keeping the sort order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (splitting increases the machine of self-induced out-of-order and jitter events), and considering both the measurable characteristics of the connections and the unmeasurable external directives.

[0140] The following values ​​may be used for rounding: ●Latency-related values ​​(RtProp, RTT) are rounded, then floored 1ms later, and rounded to the nearest multiple of 50ms. ●Throughput-related values ​​(BtlBw) are rounded to the nearest decimal place. ● Packet loss (the worst value between the three sliding windows) is rounded to the nearest decimal place. Note that since the raw values ​​here are limited to the range [0, 100], the output values ​​are limited to the set (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100).

[0141] An example of rounding to the nearest decimal place is provided in the table below. The raw value, the calculated number of decimal places of the raw value, and then the resulting rounded value are shown. [Table 2]

[0142] Specific improvements based on improving networking routing order to control the use of network interfaces are described in several embodiments that predict throughput based on the Mathis formula to establish an upper limit on TCP throughput given RTT, packet loss, and MSS by utilizing the combined characteristics of round-trip time (RTT) and packet loss. The Mathis coefficient can be used to modify the sort order for routing networked packets.

[0143] The characteristics measured may include packet loss between three separate sliding windows (500ms, 3s, 30s) (further aggregated as the worst value between the three windows), current network round-trip time (RTT), bandwidth round-trip propagation time (BBRv1) parameter, RtProp (network propagation delay), and BtlBw (throughput estimate). External directives may include priority (PR) rules, among other things, but may also include flow-specific rules / requirements.

[0144] In some embodiments, the measured properties of BBRv1 differ from the design of these properties in the original BBRv1 IETF draft. For example, to consider embodiments that may run on systems with inconsistent execution times, the size of the sliding window used to compute BtlBw may be increased to cover longer durations (hide variability due to execution inconsistency), but the window duration may also be immediately shortened, as network loss or latency increase events indicate that BtlBw may indeed be modified and any long history in the sliding window must be disregarded.

[0145] For similar execution inconsistencies, the input parameters used to measure RtProp may be modified to include more than just network propagation delay, adding a portion of execution / processing time, which may not be trivial and may consistently contribute to the actual delay encountered when transmitting packets and waiting for a response. In some embodiments, this is referred to as "system RtProp," as opposed to "network RtProp," which consists only of network propagation delay. In these embodiments, "system RtProp" and "network RtProp" may be used for various calculations depending on the intended purpose. For example, when calculating CWND, "system RtProp" is appropriate so that the pipeline includes enough data in flight to account for processing delay. However, when sorting network connections to determine packet routing, "network RtProp" is more appropriate because the preferred routing decision is independent of system execution delay.

[0146] Packet loss, latency, and throughput are not entirely independent variables. Their combinations influence each other depending on the application's congestion control behavior. For example, with TCP, higher packet loss leads to longer periods of radio "silence" (underutilized channels) as the sender waits for a DUP ACK or RTO to occur.

[0147] Therefore, several composite attributes are determined and used in the final sorting method. Specifically, Mathis coefficient: This composite attribute predicts throughput considering RTT and packet loss, and is based on the Mathis Relation, which gives an upper limit on TCP throughput considering RTT, packet loss, and MSS. Since MSS can be considered a constant when comparing the attributes of any two network connections to determine their sort order, this relation shows that the expected throughput is proportional to the reciprocal of (RTT*sqrt(packet loss)). This relation is used in this example as a practical implementation for establishing a coefficient used when directly controlling the operation of a network controller or router. For each connection, the system determines the Mathis coefficient as follows: MAX(rounded Rtt, 1ms)*sqrt(rounded aggregated packet loss). A connection with a smaller Mathis coefficient value is considered better. Since the calculation is a function of two independently measured properties, the relationship between these properties determines how connections compare to each other. The Mathis coefficient is held and stored in a data object, such as a corresponding Mathis coefficient data value associated with each connection, and may be stored, for example, in an array of data values ​​along with other monitored properties.

[0148] For example, in one extreme case, two connections with 0% packet loss will always have the same Mathis coefficient, regardless of the actual RTT value of these connections. In this scenario, the Mathis coefficient is not affected by the sort order and is determined by a comparison of the subsequently measured attribute and composite attribute.

[0149] In the other extreme case, if two connections have 100% packet loss (rounded aggregate packet loss = 1.0), only the rounded Rtt values ​​of these connections determine how to compare them to each other. In some embodiments, this is undesirable, so connections with 100% packet loss are filtered out from available selection at an earlier stage.

[0150] Between the two extremes, connections with both low RTT and low packet loss are now considered better than connections with higher values. However, connections with a high percentage of loss can still be compensated for by having a low RTT and may have the same Mathis coefficient as connections with a low percentage of loss. For example, Connection A: 100ms RTT, 0.01 (1%) loss => Mathis coefficient = 10 Connection B: 19ms RTT, 0.25 (25%) loss => Mathis coefficient = 9.5

[0151] In this example, even though connection B has significantly higher loss (25% compared to 1%), its lower RTT compensates for this, bringing it to a better Mathis coefficient than connection A. The explanation for this is that connection B's lower RTT allows it to exchange packet acknowledgments (affirmative and negative) for lost packets and complete any necessary packet retransmissions faster than connection A, which has a higher RTT, even though connection A has a smaller percentage of loss.

[0152] Capacity Factor: This composite attribute is determined considering BtlBw and RtProp as follows: rounded BtlBw / MAX(rounded RtProp, 1ms). The unit is "bytes per millisecond" and has no physical meaning, but it provides a way to normalize the bandwidth that a connection can achieve per unit of RtProp. In other words, a connection that can achieve more bandwidth per millisecond of its RtProp is considered more efficient than a connection that achieves less bandwidth. Efficient transmission in this context means a connection that achieves more bandwidth per millisecond of propagation delay (RtProp).

[0153] Once all raw inputs have been processed and the composite properties have been calculated as described above, the actual sorting mechanism can be executed sequentially as follows:

[0154] The transition from one comparison to the next occurs only if all of the previous row are found to be equal. If the comparisons are not equal, the remainder of the comparison are irrelevant and omitted. In some embodiments, the selection of comparisons is based on flow classification (e.g., VPN vs. VoiP vs. ping packets vs. FEC packets), and for each different flow, the comparison criterion list may be different (e.g., each may have different predefined most important characteristics, second most important characteristics, third most important characteristics, etc.). Each of these characteristics can be iterated through, starting with the most important, etc., until the connections themselves can be sorted and placed into an ordered sequence. In some embodiments, depending on the flow classification, a first pre-step is to select a subset of connections that are suitable for transmitting the type of data packets in the flow classification, and then order them before packet allocation is performed. 1. A connection that is completely down (100% aggregate loss) is considered a worse connection than a connection with less than 100% aggregate loss. 2. PR Priority (Higher priority indicates better performance) 3. Mathis coefficient (lower is better) 4. Rounded RTT (lower is better) 5. Capacity coefficient (higher is better) 6. Rounded RtProp (lower is better) 7. Rounded BtlBw (higher is better) 8. Connection ID (lower is better)

[0155] A rendering of the sorting mechanism is shown in Figure 1F. In the exemplary sorting mechanism, the real-time flow first includes a comparison of RTT / RtProp, then an equality comparison, then the real-time flow compares reliability, and then, if still equal, compares transmission rate, etc. Depending on the flow type, different sorting orders are possible. For example, a flow that wants throughput may first compare transmission rate. If equal, compare reliability. If still equal, the real-time flow may decide not to evaluate anything else and simply select arbitrarily.

[0156] In some embodiments, flow requirements (e.g., real-time versus throughput) can determine the order and number of comparisons. For example, a real-time flow might prioritize comparing only rounded RTT and rounded RtProp, ignoring all other attributes. A throughput flow might decide to compare only the Mathis coefficient and the capacity coefficient.

[0157] The methods described above concern explicit sort dimensions. There are also implicit sort dimensions, which tend to arise as a byproduct of typical application traffic patterns.

[0158] Most application traffic tends to be bursty or adaptive, and therefore does not require the use of all WAN connections. This means that connections currently at the bottom of the sort order may not carry traffic. If these connections are not carrying traffic, their explicit sort dimension is not updated, and they tend to maintain their position near the bottom of the sort order.

[0159] In some embodiments, push scheduling can also be configured to implicitly cause a sort order along one or more implicit sort dimensions over a certain period of time. The implicit sort dimensions force connections that encounter periodic bad events (e.g., latency spikes due to congestion) to bubble at the bottom of the list. Connections with consistent behavior naturally bubble at the top and remain near the top.

[0160] While this is generally considered a good attribute of the algorithm, it may be modified in the future to be influenced by external factors. For example, if a flow has certain requirements that cannot currently be met by connections near the top of the sort order, it may make sense to generate probing traffic on connections near the bottom to refresh the explicit sort dimension of those connections.

[0161] Figure 4A illustrates an embodiment in 400A where a selector is generated using a predefined flow class and the current connection state.

[0162] In this example, there are three connections, each with a set of properties to be monitored, and three predefined flow classes, each defining a match criterion and a preference for sorting connections that satisfy the match criterion.

[0163] In this embodiment, the selector consists of a set of flow classes whose match criteria match at least one connection for which there is available capacity. In this example, both flow class 1 and flow class 2 have match criteria that match connection 1 and connection 2, while flow class 3 does not match any connection and is not included in the selector.

[0164] Even if both Connection 1 and Connection 2 match, Connection 2 is excluded from the selector because it does not have an available congestion window and is currently unable to send packets. Since flow classes can be included as soon as a matching connection with an available congestion window is found when constructing the selector, one embodiment may determine that flow classes 1 and 2 can be included in the selector without having to examine only Connection 1 to confirm that Connection 2 matches.

[0165] Step 2b (block 1105) uses the selector from step 2a to find packets on the retransmit or transmit queue that are eligible to be (re)transmitted on an active connection with capacity. If no such packets are found, the loop terminates. This can also occur if several active connections have capacity and packets exist on the input or retransmit queue.

[0166] Figure 4B shows one embodiment of 400B having a priority input queue, which is a set of ordered flows, where flows on the right side of the queue should send packets before flows on the left side. Each flow has a flow ID, and each flow belongs to one of a predefined flow class. Each flow also contains a queue of ordered packets, so when a flow is selected, the packet at the top of the queue is removed.

[0167] In this example, the selector from Figure 4A is used to evaluate the flows in the input queue from right to left, and the selector selects the first flow that has a flow class listed in the selector. In this case, flows with IDs 5 and 2 are evaluated and rejected because both of these flows have flow class 3, which is not one of the flow classes from the selector. Next, flow 7 is evaluated and matches because flow 7 belongs to flow class 1, which is one of the flow classes in the selector. Then, packets are extracted from the head of the packet queue for flow 7 and stamped with flow metadata.

[0168] Once packets are selected, step 2c (block 1105) constructs an ordering of active connections. The selected ordering is, if the embodiment supports it, or otherwise a generally acceptable ordering, based on rules provided in the flow class definition. Some embodiments may use the same ordering for all flows matching a flow class, and some embodiments may make the ordering flow-dependent (e.g., making all packets in a flow sticky to a given connection without requiring all flows to be sticky to the same connection). The ordering may be constructed as needed in each iteration of the loop in step 2, and some embodiments may choose to cache and clear the ordering if the criteria used to construct the ordering change (i.e., with metadata updates).

[0169] One embodiment constructs an ordering system regardless of whether a connection currently has capacity, since scheduling packets on a connection can reduce capacity to zero, or scheduling packets on different connections can increase capacity above the current value of zero if it causes enough changes to the metadata mix to activate a set of connections at a priority level not previously used. Constructing the ordering system in this way makes it possible to cache and reuse the ordering system if the only items that change are the valid CWND and in-flight byte count for each connection.

[0170] Flows traversing System 100B tend to behave better when the packets of that flow traverse the same WAN connection whenever possible. Therefore, in one embodiment, it is preferable to construct the ordering using a stable sorting criterion so that the ordering created for the same flow remains consistent unless one or more monitored characteristics of the connection change significantly. Thus, operating characteristics such as RTT and RtProp, as well as bandwidth, are unsuitable choices because they can fluctuate over short periods and destabilize the sorting order.

[0171] The transformation from "noisy" raw operating characteristics to monitored characteristics can be performed using statistical filtering methods. These methods can range from conventional measures such as mean, variance, and ANOVA to the realm of (quantized) rank-order statistics (rank-order filtering). For example, the monitored characteristics may be a range of raw operating characteristics, which can be defined by a specific value (e.g., 1 to 100 ms) or a number of digits. Specific examples, including those shown in Table 2, follow in the subsequent paragraphs.

[0172] In some embodiments, machine learning techniques can be used to translate operational characteristics into monitored characteristics. For example, a trained model could be developed to predict an impending connection failure, and as a result, an ordering could be constructed that sorts the connections matching the end of a list. Another exemplary model could be used to detect the type of connection (e.g., wired, cellular, WiFi, satellite, BoD, etc.) and use this to form the basis for novel monitored statistics that are not readily measurable in real time (e.g., susceptibility to buffer bloat).

[0173] One embodiment uses the following order for all packets (later criteria are used only if all previous criteria are equal):

[0174] "Bucketed" drop probabilities. Connections can be grouped into buckets defined by a range of drop probabilities based on packet count or size (e.g., in bytes). Connections in buckets with smaller drop probability ranges appear earlier in the ordering, while connections in buckets with larger drop probability ranges appear later.

[0175] Connection priority groups (provided by administrators or end users). Connections with higher priority are ranked earlier (primary > secondary > tertiary, etc.).

[0176] "Bucketed" RTT. Latency is rounded to the nearest multiple of 50ms. Bucketing latency values ​​allows normal latency fluctuations to be hidden from the ordering, while at the same time reflecting large changes in latency in the ordering.

[0177] The "Capacity Factor" is an aggregated value measured in "bytes per second per millisecond," calculated by rounding the bottleneck bandwidth estimate to the nearest decimal place or an integer multiple of the nearest decimal place (e.g., rounding from 1 to 1, from 10 and 11 to 10, from 15 to 20, from 86000 to 90000) and dividing it by the nearest 50ms bucketed RtProp. A connection that achieves a higher bandwidth per ms of RtProp is considered better than a connection with a lower bandwidth per ms of RtProp.

[0178] Determining the capacity coefficients of raw BtlBW and RtProp values ​​is highly variable, but stabilizing the capacity coefficients by rounding BtlBW to the nearest integer or an integer multiple of the nearest integer, and by bucketing RtProp, stabilizes the values ​​over time unless large changes are observed. Table 2 shows some examples of bucketing, rounding, and capacity coefficient calculation. Note that in this embodiment, the capacity coefficients are rounded to the nearest integer, and for the purpose of calculating the capacity coefficients, bucketed RtProp values ​​of 0 are treated as 1.

[0179] "Bucketed" RtProp. A lower value for the nearest 50ms bucketed RtProp is preferred over a higher value.

[0180] "Rounded" BtlBW. A higher value of BtlBW rounded to the nearest number of digits or an integer multiple of the nearest number of digits is preferred over a lower value.

[0181] If all of the aforementioned criteria are compared as equal, the connection ID (assigned when the connection was enumerated) is used to deterministically break the equilibrium. Other embodiments may use a more complex tie-breaking mechanism, for example, hashing the flow's attributes to an integer modulo of the number of connections that balance the equilibrium. Table 2: Examples of bucketing, rounding, and capacity coefficients [Table 3]

[0182] One embodiment of distinguishing between flow types may, for example, modify the above order depending on the transmission requirements of the flow type, prioritizing high-capacity coefficient connections over lower-latency connections for file transfer flows, and prioritizing low-latency connections over high-bandwidth connections for video streams.

[0183] One embodiment of performing strict flow pinning to a connection (or set of connections) may choose to allow only exact matches of connection IDs (and in some cases completely excluding other connections from the ordering). Such pinning rules are typically provided by an administrator or end-user and are intended to override all ordering decisions determined by monitored characteristics. Therefore, even if monitored characteristics indicate that a connection is unavailable or has failed, the flow will continue with the same ordering and thus will no longer have connectivity through the multipath system 100B.

[0184] In some embodiments, a monitored characteristic may be created that is a combination of other monitored or operational characteristics, and these composite materials may be used to determine the ordering. For example, a composite reliability metric may be created that combines bucketed drop probability, (running) latency (RtProp) variance, and (running) throughput to make a weighted metric that represents the overall reliability of a connection. Connections that rank highly in this metric will be reserved for flows that require high reliability (high reliability may not necessarily mean high speed).

[0185] Generally, when a selector is generated that allows a packet to be selected, that packet should generate an ordering that allows the packet to be scheduled on some of the connections used to generate the selector.

[0186] Figure 4C illustrates an example of how packet flow classes can be used to construct connection ordering in the 400C. In the depicted embodiment, flow classes are used in the packet metadata to reference flow class definitions. Flow class matching criteria are used to select which connections should be included in the constructed ordering (in other words, flow classes define a set of transmission requirements that may be associated with a particular flow class), in this case both connection 1 and connection 2 match, while connection 3 does not. Note that connection 2 is included even though it does not have an available congestion window (i.e., is not currently eligible to send packets). This facilitates caching the ordering and allowing its reuse even when the state of the available congestion window should change (for example, if the scheduler initially blocks transmission on a connection due to its low priority, but then decides to re-enable the connection because the current data mix on an active connection does not meet the minimum system throughput).

[0187] Once a set of connections is selected, the matching connections are sorted into an ordered list using ordering criteria from the flow class (e.g., a subset of transmission requirements associated with the flow class). In this example, the first criterion is the capacity factor; both connections have the same capacity factor. The second criterion is RTT; both connections have the same RTT. The third criterion is BtlBW (bottleneck bandwidth); connection 2 has a higher value and therefore appears before connection 1 in the ordering.

[0188] A set of ordering criteria within a flow class (e.g., transmission requirements) allows packets to be scheduled on the most optimal available connection for the type of flow they belong to. For example, packets belonging to the "video encoder" flow class may preferentially use the connection with the lowest latency, while packets belonging to the "file transfer" flow may prefer high-throughput connections, even if they have high or highly variable latency.

[0189] Once the ordering of connections is determined, step 2d (block 1107) uses the ordering to allocate packets to connections with available bandwidth. The packets are placed on the transmit queue for the connections.

[0190] Figure 4D shows that in 400D, a connection is selected for a given packet using the ordering from Figure 4C. In this example, the first connection in the ordering is connection 2, but connection 2 is not currently eligible to send the packet because it does not have an available congestion window. Connection 1 is then checked and is selected because it can send the packet, and the packet is queued to be sent to the stage 3 pipeline for connection 1.

[0191] Next, the connection metadata (monitored characteristics) of the in-flight bytes is adjusted as part of step 2e. In some embodiments, this includes distinguishing between good put bytes and retransmission / dummy / probing bytes when tracking CWND consumption for a connection. Such adjustments may affect good puts and / or potential good puts for a priority group, which may activate connections of the next priority level in the next iteration of the loop in step 2. Various methods for converting from volume units (CWND bytes) to rate (good puts) are discussed following the description of Figures 8 and 9.

[0192] When step 3 is reached, either the available CWNDs for each connection are full, or there are no packets eligible for transmission. Packets queued in the per-connection output queue in step 2 are then sent to the appropriate instance of the next stage of the pipeline.

[0193] In some embodiments, an explicit objective function can be used to determine a desired trade-off when there are multiple flows belonging to a flow class that have the same priority but conflicting requirements.

[0194] The embodiments shown in Figures 4A to 4D illustrate a target function that implements fairness between flows with the same priority. The selector provides services to each flow that matches the flow class in a round-robin fashion.

[0195] A fair implementation in this embodiment uses the underutilized round-robin scheduling algorithm described in RFC 8290. A brief overview of this algorithm is that it utilizes a "byte credit" scheme and allocates a quantum of byte credits to each queue in each round of round iterations across the entire queue.

[0196] The number of bytes that can be obtained for transmission from each queue in each round is nominally limited to the number of available credits the queue possesses. Fairness naturally arises when the byte credit quantum for each queue is equal.

[0197] In some embodiments, the objective function may allow for intentional unfairness to be constructed by an administrator or end-user. For example, weights may be constructed on the matching flow, which could ultimately result in unequal byte credit quantum being given to the queue in each iteration.

[0198] For example, there could be five different weight levels: ●Best (5) ●High(4) ●Standard (3) ●Low(2) ●Minimum (1)

[0199] The value assigned to each weight represents the ratio between the byte credit quantum values ​​of these queues. For example, if the quantum corresponding to standard is 30KB, a flow marked as low would receive (2 / 3)*30KB=20KB of quantum value in each round. A flow marked as highest would receive (5 / 3)*30KB=50KB of quantum value.

[0200] It should be noted that the absolute numbers in this example are not important. If the standard quantum were 5KB instead of 30KB, the weights would still scale appropriately to the relative quantum for other priority levels, and the final result would be the same in terms of overall fairness.

[0201] Other embodiments may allow the weights to be arbitrary integers rather than the five fixed values ​​in the example above. This would allow administrators or end-users to configure greater unfairness between flows in the target function as needed.

[0202] Furthermore, as with all Quality of Service (QoS) settings, it should be noted that these weightings are only important when there is a conflict over available WAN capacity. When there is no conflict, the ratio between queue usages is a natural ratio from applications generating different volumes of data, as it means that all data in the queues fits perfectly within the available capacity.

[0203] Other embodiments may have completely different objective functions that do not consider fairness at all. For example, ●Ensure that the maximum number of flows is met by the available WAN capacity, ● To maximize the total throughput achieved by the serviced flow, or ● Minimize changes to the current bandwidth allocation on the BoD connection.

[0204] In some embodiments, the way a WAN connection is used can modify the attributes of the WAN connection, making it unsuitable for servicing a particular flow class. For example, some types of connections can achieve high throughput but at the cost of causing a higher RTT.

[0205] An embodiment with a target function that aims to maximize throughput would choose to serve flows with high throughput requirements, which would increase the WAN connection RTT and make the WAN connection unsuitable for serving flow classes with low latency requirements.

[0206] Conversely, in an embodiment with a target function that aims to service the maximum number of flows, the opposite trade-off may be selected so that the low-latency flow class can continue to be serviced by this WAN connection.

[0207] In some embodiments, the connection sort order of individual schedulers is maintained on a per-flow basis, rather than globally across all flows.

[0208] This allows flow-specific requirements to influence the sort order, ensuring that flow packets are assigned to connections in a way that meets the connection's needs.

[0209] For example, a VoIP flow with a preferred maximum latency tolerance of 150ms would want a sorting criterion that places connections with RTTs greater than 150ms at the bottom of the list. Conversely, a TCP flow without specific latency requirements would instead prefer that connection throughput be prioritized in the sorting criterion for this TCP flow, either by completely ignoring RTT or using RTT only as one of many secondary criteria.

[0210] In some embodiments, these flow requirements and preferences are structured in the form of rules consisting of matching criteria and resulting behaviors.

[0211] For example, the match criteria can take the form of an IP3 tuple (protocol, source IP, and destination IP) or an IP5 tuple (protocol, source IP, source port, destination IP, and destination port). The behavior takes the form of an explicit preference for latency or throughput, with a preferred target value for either.

[0212] Other examples of heuristic-based matching criteria include: ●DSCP tag ● Any other header field available in the IP (v4 or v6) header ● Either the TCP header or any other header field in the UDP header ●The cumulative volume of transmitted data, such as TCP flows exceeding a volume threshold, may match as a "bulk data transfer," while others may not match as an interactive SSH / telnet session. ● Cumulative duration of the flow ● Regular expression matching for packet payload ● Clear text parameters extracted from the TLS handshake (e.g., SNI, CN, SubjectAltName)

[0213] Other examples of behaviors that may affect the scheduler sort order include: ●Specific preferences regarding connection order (for example, a user might prioritize sorting expensive connections at the bottom of the list for low-priority flows). ● Jitter tolerance ● Maximum latency tolerance ● Desired amount of redundancy (e.g., FEC) ● Whether retransmission of lost packets is desired, and if so, whether there is a maximum time limit during which retransmission attempts should be stopped. ● Whether explicit packet pacing is desired ●For example, a real-time video application that transmits video at 30 frames per second can transmit the video frames precisely at 33.3 millisecond intervals, and the multipath system 100B would not want to change the pacing. ●In contrast, burst applications can benefit from having the scheduler explicitly pace video bursts at the aggregated WAN rate. ● Desired throughput (e.g., minimum, target, maximum)

[0214] Using machine learning techniques to analyze traffic patterns, rules (both matching criteria and behavior) can be automatically generated.

[0215] Possible inputs (features) for ML methods include the following: i. Cumulative volume of transmitted data ii. Packet size distribution (histogram) iii. Interval between packets (both pacing and jitter) iv. Packet frequency and grouping v. Packet header fields (Ethernet, IP, TCP, UDP, etc.) vi. Packet contents (payload) vii. Not only the IP address, but also the intended destination of the packet (e.g., the SNI, CN, and SubjectAltName fields in the TLS handshake and exchanged certificates). viii. Cumulative duration of the flow ix.Time x. Number of simultaneous flows to the same destination or from the same source xi. Predicting the current or historical flow (labeled) of any simultaneous flows to or from the same destination (for example, if several applications open multiple flows, one as the control plane and another as the data plane, knowing that an existing control plane flow exists can be helpful in predicting the data plane flow).

[0216] The predictions (labels) from the ML method would likely be one of the aforementioned behaviors, determined based on the behavior of the training corpus.

[0217] In some embodiments, rules may have the concept of dependent rules that apply further matching criteria and more specific behaviors, while still being managed by parent rules. For example, ●A VPN flow identified by an IP5 tuple may have rules defined by its matching criteria and behavior. This would be considered the "parent" rule. ● A VPN will carry a lot of internal flows from applications protected by the VPN within the VPN. Typically, these internal flows will be completely hidden from the matching criteria (encrypted by the VPN). ●However, some VPNs may choose to expose a limited set of information about internal flows by setting DSCP tags on encrypted packets that correspond to preferences for internal (cleartext) packets.

[0218] While it's possible to create dependent rules that match DSCP tags, the resulting available behavior will still be governed by the parent rule. For example, a. Generally, VPNs require packets to be delivered in order (VPNs can tolerate errors in order, but only to a small degree). b. The behavior of dependent rules that match the DSCP tag must not result in the parent rule delivering packets in a significantly disorganized manner.

[0219] For example, a dependent rule requiring a maximum latency limit of 150ms and a second dependent rule requiring maximum throughput may violate the ordering requirements of the parent rule, as interleaved packets from these two dependent rules may be assigned to connections with significantly different latencies.

[0220] Figure 5A illustrates two independent flows, each with completely independent rules and behaviors. In this example, the SIP flow requires low latency, so the behavior is to have scheduler 104B transmit only SIP flow packets over connection 1. Conversely, the FTP flow requires high throughput, so scheduler 104B transmits only FTP flow packets over connection 2.

[0221] For these two flows, packets are transmitted wirelessly, and when they reach the sequencer 118, these packets are returned in order and transmitted independently to their final destinations. In this example, the sequencer 118 transmits packets that have already arrived and belong to the SIP flow in the correct order, regardless of the state of the FTP flow, in which packets are still being transmitted wirelessly.

[0222] Figure 5B illustrates the concept of dependent flows. In this example, the same SIP and FTP flows exist, but before scheduler 104B sees these flows, they pass through the VPN client for encryption and encapsulation.

[0223] When the packets originate from the VPN client, they have the same IP5 tuple, so the scheduler 104B and sequencer 118 treat this as the parent flow and are constrained by the requirements of the parent flow.

[0224] The presence of dependent flows (SIP and FTP) is communicated using DSCP tags, which control how scheduler 104B allocates packets to connections 1 and 2 for transmission. In this example, the allocation is the same as in Figure 5A.

[0225] However, the parent flow constraint requires that packets be transmitted from the sequencer 118 in the same order in which they arrived at the scheduler 104B. Therefore, in this example, even if packets belonging to a dependent SIP flow have already arrived at the sequencer 118, they cannot be transmitted until packets belonging to a dependent FTP flow arrive and are transmitted first.

[0226] Figure 6A shows a sample implementation of the rule management page in 600A. The configured rules are displayed in a table, with each row summarizing the match criteria and behavior. The order in which the rules are listed is the order in which packets match the rules. Figure 6B shows the workflow for changing the order of rules in 600B.

[0227] Figure 6C shows how each row in the table can be expanded in 600C to show the details of the matching criteria and behavior, as well as the currently active flow through the system that matches this rule.

[0228] Each rule in the table includes edit and delete icons.

[0229] Figure 6D shows the dialog box that appears when the edit button is clicked in 600D. The matching criteria and behavior for the rules can be changed in this dialog box.

[0230] The main rule management screen has a separate [Add Rule] link, and Figure 6E shows the mode dialog that appears when you click it on the 600E.

[0231] In some embodiments, rules are configured on the transmitter side and automatically pushed to the receiver. By default, rules are symmetric and bidirectional; that is, when a rule is added and its behavior is configured, matching pair rules are added transparently, but the source and destination matching criteria are swapped. It may be possible to add asymmetric rules (different behavior for either direction of the same flow) via an advanced configuration screen.

[0232] In one embodiment, the scheduler probes various network characteristics of multiple connections (to determine the characteristics to be measured for each connection) and uses the measured characteristics in combination with transmission requirements to make decisions regarding packet transmission.

[0233] For example, one such decision is to determine the number of packets to transmit over a particular connection and when to transmit those packets.

[0234] Some embodiments are volume-based. Such embodiments operate by determining a limit (e.g., in units of bytes or packets) on the volume of data that can be transmitted over a particular connection. Once enough packets have been transmitted within that volume, transmission stops until some feedback about these packets is received. If feedback is received before the full volume limit is transmitted, transmission can continue without stopping, as long as the volume of transmitted data for which no feedback has been received does not exceed the determined volume limit.

[0235] In one embodiment, the data volume is selected to be the connection's congestion window (CWND). Simply put, the CWND is the maximum volume of data that a connection can send without causing significant congestion before receiving any feedback. Numerous methods exist for estimating the CWND. It is assumed that the scheduler has access to one such method and can receive an estimate of the CWND through that method.

[0236] In some embodiments, the scheduler is required to determine the rate at which data is being transmitted (for example, in units of bytes per second). This rate can then be used to make other decisions regarding packet transmission.

[0237] For example, in one embodiment, if the rate of data being transmitted over currently active network connections is lower than the rate of data being received from the source application for transmission, the scheduler may decide to activate more connections and increase the aggregated transmission rate to accommodate the application rate.

[0238] In another example, one embodiment may need to meet quality of service (QoS) requirements, which can be expressed in terms of a guaranteed transmission rate. If the aggregate rate falls below the required level, the scheduler may decide to activate more connections to increase the aggregate transmission rate in order to meet the QoS requirements.

[0239] In another example, one embodiment may be needed to measure good put (i.e., the rate at which application data successfully passes through the network) for purposes such as ensuring application performance levels, optimizing performance, and reporting.

[0240] In one embodiment, TCP performance can be optimized by pacing the packets being sent at an aggregate rate achieved by multiple connections. See, for example, PCT Application No. PCT / CA2020 / 051090.

[0241] In embodiments where a rate is required for some functions, a volume-based scheduler requires a way to convert the volume of data being transmitted into a rate.

[0242] The trivial approach of converting volume to rate by dividing the volume by the duration of the time over which the volume was transmitted generally yields inaccurate results. Such an approach measures that the transmission rate at the transmitter is moving through the network rather than the actual rate data. The actual rate at which data moves through the network is generally truly determined by the rate on the slowest link of a multi-link network, often referred to as the network bottleneck.

[0243] The characteristics of the network bottleneck link are not available to a transmitter or receiver that knows that a multi-link network can change the combination of links transparently used for the transmitter and receiver at any time. However, congestion control methods can generate an estimate based on the performance observed at the transmitter and receiver.

[0244] In some embodiments, the congestion control method can provide an estimate of the transmission rate of the bottleneck link. This may be referred to as the network bottleneck bandwidth BtlBw.

[0245] In some embodiments, the network round-trip propagation time RtProp can be estimated by sampling the time between when a packet is sent and when feedback regarding the packet is received.

[0246] Using the combination of BtlBw and RtProp, the network attribute called Bandwidth Delay Product (BDP) can be determined as follows. BDP = BtlBw × RtProp

[0247] The most common unit for BDP is bytes. For these quantities, the following units are assumed: (i) for BtlBw, bytes per second, (ii) for RtProp, seconds, and thus (iii) for BDP, bytes.

[0248] BDP means the amount of data that would be present in the network (in - flight) when operating under ideal conditions assuming that data is always available for transmission. This network transfers data at the nominal rate of the network BtlBw and would provide feedback for every packet exactly after the RtProp time to transmit its packet. Figure 7 shows two examples at 700. The upper half of Figure 7 illustrates this concept using an example where BDP consists of 16 equal - size packets. In practice, the amount and size of the packets can vary as long as the total size is equal to the value calculated using BDP.

[0249] In some volume - based embodiments, BDP can be used to estimate various data transfer rates, some of which were described above.

[0250] For the in - flight volume under consideration, assuming ideal conditions, if the volume is smaller than BDP, only a fractional value of the nominal rate of the network, BtlBw, can be achieved. This fractional value can be calculated as the ratio of the volume to BDP. The resulting achieved rate is determined as follows: Achieved rate corresponding to in - flight volume = Volume / BDP × BtlBw

[0251] In reality, networks rarely operate under ideal conditions. For example, packets may be temporarily buffered before being delivered to the receiving end, and feedback may be delayed at the receiver or by the network. In some cases, packets may be lost and never reach the receiving end, thereby preventing feedback from being triggered. In other cases, the feedback itself may be lost. Losses can be estimated, for example, when feedback for a new packet is received or based on timer expiration.

[0252] Waiting for delayed feedback (whether it arrives or not) to trigger the next transmission event can artificially limit the achievable rate on the network.

[0253] Figure 8A shows an example of a receiver in Figure 800A that aggregates feedback and sends an acknowledgment back to the sender once for every four packets received.

[0254] Figure 8B further illustrates how, in Figure 800B, for example, the aggregation of acknowledgments results in overall throughput being less than the network's capacity. The sender waits longer before beginning to send the next packet group. Overall, the sender ultimately sends the same amount of packets as in the ideal case in Figure 7, but over a longer period of time, resulting in reduced throughput.

[0255] In one embodiment, this artificial limitation is overcome by specifying an additional allowance for the data to be transmitted, even if feedback for previously transmitted packets has not yet been received. This means that the volume of in-flight data may exceed BDP. However, in such cases, as the application of the achieved rate formula above may suggest, the network is not actually transferring data at a higher rate. The additional allowance simply allows the transmitter to maintain a constant rate equal to BtlBw over a longer period of time. Thus, the estimated transfer rate of the volume at hand is capped at BtlBw.

[0256] Figure 8C further illustrates how, in Figure 800C, for example, the transmitter uses such additional transmission capacity to continue transmitting packets before an acknowledgment is received. The in-flight data volume, 19 packets, exceeds the BDP of 16 packets. However, the actual rate matches the network capacity, and the calculated rate capping achieves the same result by using the minimum in-flight volume and the BDP.

[0257] In some embodiments, a similar method can be used to estimate what may be achievable on the network, which will hereafter be referred to as potential good puts.

[0258] A specific volume of data can be divided into a good-put portion and another portion. The good-put portion includes newly transmitted application data. The other portion includes data that cannot be counted as newly transmitted application data, such as previously lost data, control data, and probing data.

[0259] If the total volume of data is greater than the BDP, the packet and / or feedback delays are assumed to be as described above. It is assumed that there is no extra space available for additional goodputs. The resulting (potential) goodput rate is estimated as follows: Good input rate = Good input portion / Total volume × BtlBw

[0260] For example, Figure 9A shows a network with 16 BDP packets in 900A. The volume of packets designated as inflight to achieve throughput equal to the network's capacity is 19 packets. However, only 14 packets are good puts, resulting in a good put rate of 7 / 8 of the network's capacity. When the data volume is smaller than the BDP, the difference is considered to be extra space that is assumed to be available for additional good puts.

[0261] Figure 9B shows an example where, in 900B, only 3 packets are in flight, resulting in 13 extra packets that can be assumed to be good puts when calculating the potential good put rate. This extra space must be limited by the Congestion Wavelength (CWND) determined by the congestion control method. At any given time, the additional volume that congestion control allows is the difference between the CWND and the total volume. If the extra space for good puts exceeds this congestion control allowance, the space is simply limited to the congestion control allowance.

[0262] Figure 9C shows an example where the CWND is reduced to 11 packets in the 900C, which limits the extra space that would otherwise be available for additional goodputs to 8 packets.

[0263] The resulting potential good put rate is estimated as follows: Potential good put rate = (Good put portion + Extra space for limited good put) / BDP × BtlBw

[0264] Continuing with the above example, FIGS. 9B and 9C show how the above potential goodput formula results in a lower potential goodput rate for the scenario in FIG. 9C, as expected considering the CWND reduction performed by the congestion control method.

[0265] The above approach is described in terms of goodput and potential goodput rate, but can be easily modified to calculate the rate or potential rate of any type of data that partially constitutes the total volume.

[0266] The following is a numerical example of applying the above formula to a network having attributes typical of those encountered via an LTE cellular network. BtlBw = 10 Mbps RtProp = 60 ms BDP = 10 Mbps * 60 ms = 75 KB CWND = 60 KB Goodput portion = 25 KB Other portion = 10 KB Total volume = 25 KB + 10 KB = 35 KB Extra space for goodput = BDP - Total volume = 75 KB - 35 KB = 40 KB Congestion control allowance = CWND - Total volume = 60 KB - 35 KB = 25 KB Extra space for limited goodput = Min(Extra space for goodput, Congestion control allowance) = Min(40 KB, 25 KB) = 25 KB Potential goodput rate = (Goodput portion + Extra space for limited goodput) / BDP x BtlBw = (25 KB + 25 KB) / 75 KB x 10 Mbps = 50 / 75 x 10 Mbps = 6.67 Mbps

[0267] Systems that communicate over IP networks often implement congestion control mechanisms that aim to achieve fair network usage by all network nodes employing similar congestion control mechanisms.

[0268] Congestion control mechanisms often operate by monitoring communications occurring on a network, sometimes probing the network as needed, and deriving network attributes based on the results. A model representing the network is created, and subsequent communications are guided based on this model. For example, this model can track network attributes such as the current maximum throughput achieved, the current and minimum round-trip times of packets, and packet loss events.

[0269] Based on the created model and tracked information, the congestion control mechanism can determine when the network is underperforming, indicating that the network may be experiencing congestion. The congestion control mechanism can then take corrective actions aimed at reducing or eliminating network congestion and restoring desirable network performance.

[0270] In previous congestion control implementations (e.g., TCP), packet loss is used as an indicator of network congestion, and corrective measures (e.g., reducing the CWND) are taken in response to the loss.

[0271] In one embodiment, packet latency, i.e., the time required for a packet to travel across the network from the transmitter to the receiver, is an indicator of network performance. High-latency packets may, in some cases, be useless to the receiver and may be discarded, resulting in wasted transmission of these packets. For example, real-time VoIP applications that require low conversational latency between participants will avoid using packets that carry voice data older than the maximum acceptable conversational latency.

[0272] In such embodiments, the congestion control mechanism may monitor and control packet latency so that packet latency does not exceed an acceptable level.

[0273] In one embodiment, the minimum latency measured for a packet can form a baseline value, and subsequent measurements can be compared to this baseline value. If subsequent measurements exceed a threshold that is a function of the baseline value, the congestion control mechanism considers the network to be experiencing congestion.

[0274] In another embodiment, a similar technique uses packet round-trip time, which is the time elapsed between sending a packet and receiving an acknowledgment for it, instead of latency.

[0275] In one embodiment, when a congestion control mechanism determines that the network is experiencing congestion based on increased packet delay, the congestion window (CWND) is reduced to limit the amount of data that can be transmitted over the network without receiving any feedback. The objective is to reduce the volume of data that will be buffered or is currently buffered by the network, which is a common cause of increased packet delay.

[0276] In one embodiment, the congestion window is reduced to the network's bandwidth-delay product (BDP). Theoretically, the BDP represents the volume of data the network should be able to deliver without forming a motionless queue of packets in its buffer, assuming packets are transmitted by the transmitter at a constant rate reflecting the network's throughput (BtlBw). This is therefore expected to allow the network to recover at least partially and reduce packet delay back to an acceptable level.

[0277] For example, consider a network with 16 BDPs and a congestion control mechanism that determines a 24-packet CWND to account for acknowledgment aggregation. Figure 10A illustrates how congestion from other network nodes can cause the transmitter to control the network in 1000A. In this case, the transmitter does not reduce its congestion window and puts the entire amount in flight. This fills the network buffer, and any excess packets that cannot be buffered are dropped. Even after the congestion is cleared and normal emptiness of the network buffer continues, Figure 10A illustrates how a stationary queue may have formed within the network buffer, making it easier to increase the round-trip time of all subsequent packets and fill the network buffer.

[0278] Continuing this example, Figure 10B illustrates how a transmitter that reduces its congestion window to BDP in 1000B avoids both packet drops and the formation of stagnant queues. The transmitter can determine congestion is occurring, for example, by detecting that no feedback has been received for a period exceeding the baseline duration for feedback measured by the transmitter in past transmissions. In another example, the transmitter can determine congestion is occurring by receiving feedback indicating that a quantity of packets transmitted before exceeding the baseline loss encountered in past transmissions has not been delivered to the receiver. In response to this congestion determination, reducing the congestion window to match BDP reduces the amount of packets that will be considered in-flight. Consequently, this reduces the likelihood of filling the network buffers, which previously led to packet drops. Furthermore, allowing only BDP packets to be in-flight allows the network to completely empty its buffers before new packets from the transmitter reach the network buffers again (assuming network attributes have not changed).

[0279] In another embodiment, the congestion window is reduced to be equal to the volume of packets currently in flight.

[0280] When the congestion control mechanism determines that an acceptable level of performance has been restored, the network is considered to no longer encounter congestion.

[0281] In one embodiment, a reduction in packet latency below a threshold is an indicator that network performance has been restored.

[0282] In one embodiment, the threshold is determined as a function of the baseline delay value.

[0283] In another embodiment, transmitting a specific volume of data without any loss is an indicator that network performance has been restored.

[0284] In one embodiment, the volume of data is equal to the BDP.

[0285] In another embodiment, the volume of data is equal to the original value of the congestion window before it is reduced by the congestion control mechanism in response to congestion.

[0286] The push architecture improves system performance in non-BoD connections for the most common application types. Table 3 summarizes the experimentally observed improvements. Table 3: Application performance improvements due to push-type schedulers [Table 4]

[0287] The push-based scheduling method also makes priority routing behavior more intuitive. For example, in one embodiment, the decision of whether to involve lower-priority connections depends solely on how the BtlBw of those connections is compared to a configured threshold.

[0288] For example, consider the following scenario: wan0 ●Priority: Primary ●Target threshold: Not applicable ●BtlBw: 10Mbps wan1 ●Priority: Secondary ●Target threshold: 15Mbps ●BtlBw: 30Mbps

[0289] With a pull-type scheduler, each time the 50Hz timer on wan1 wakes up and requests a packet, the scheduler will provide a packet for transmission to wan1 because it knows that the desired target threshold (15Mbps) is greater than the threshold that wan0 can provide on its own (10Mbps).

[0290] However, the scheduler does this regardless of the rate at which packets reach the input. For example, if the input rate does not exceed 10 Mbps, the traffic can fit entirely within wan0 alone, making it more optimal when wan1 is not involved.

[0291] Using a push-type scheduler, wan0 is sorted to the top of the list, and packets are scheduled on wan0 only after the wan0 CWND is fully consumed, even if the threshold indicates that wan1 should be active. As a result, there is less unexpected use of lower-priority connections, improving performance.

[0292] In summary, the scheduler for a multipath system can be implemented as either a pull-type (100A) or push-type (100B) system. The push-type system is superior because the scheduler can make packet scheduling decisions on a self-determining rhythm, rather than being driven by the rate at which the target connection is ready to transmit. This self-determining rhythm allows for consideration of all factors important to the application that generated the packet, and / or the system's users.

[0293] For example, packets generated by TCP-based applications requiring maximum throughput will prioritize being scheduled on connections ordered from best-in-class to worst-in-class Mathis coefficients.

[0294] Furthermore, consider a more complex example using the same TCP-based application, where the user has configured a multipath system 100B with priority routing rules, prioritizing connection 110B over all other connections 110N unless the good put achievable on 110B drops below 5Mbps. In a scenario assuming 110B has high throughput, low latency, but high packet loss, 110B would have a worse Mathis coefficient than all other connections 110N.

[0295] The push-type scheduler 100B makes complex scheduling decisions that respect user preferences to use connection 110B first, and can also calculate the good put achievable on 110B by converting the in-flight CWND of connection 110B to a bitrate and using it to estimate the available good put on connection 110B. If this availability falls below the configured 5Mbps threshold, 110B can schedule packets on the remaining connections 110N, ordered from the best Mathis coefficient to the worst Mathis coefficient.

[0296] Figure 11 is a schematic diagram of a computing device 1100 that may be used to implement system 100B according to one embodiment.

[0297] As described, the computing device 1100 includes at least one processor 1102, memory 1104, at least one I / O interface 1106, and at least one network interface 1108.

[0298] Each processor 1102 may be, for example, a microprocessor or microcontroller (e.g., a dedicated microprocessor or microcontroller), a digital signal processing (DSP) processor, an integrated circuit, a field-programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or various combinations thereof.

[0299] The memory 1104 may include, for example, a combination of various types of computer memory located either internally or externally, such as random access memory (RAM), read-only memory (ROM), compact disk read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electro-erasable programmable read-only memory (EEPROM), and ferroelectric RAM (FRAM).

[0300] Each I / O interface 1106 enables the computing device 1100 to interconnect with one or more input devices such as a keyboard, mouse, camera, touchscreen, and microphone, or with one or more output devices such as a display screen and speakers.

[0301] Each network interface 1108 enables the computing device 1100 to communicate with other components, exchange data with other components, access and connect to network resources, provide applications, and run other computing applications by connecting to networks (or multiple networks) capable of carrying data, including the Internet, Ethernet, Plain Old Telephone Service (POTS) lines, Public Switched Telephone Networks (PSTN), Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), Coaxial Cable, Optical Fiber, Satellite, Mobile, Wireless (e.g., Wi-Fi, WiMAX), SS7 Signaling Communications Networks, Fixed Lines, Local Area Networks, Wide Area Networks, and other various combinations thereof.

[0302] For the sake of simplification, only one computing device 1100 is shown, but system 100B may include multiple computing devices 1100. The computing devices 1100 may be the same or different types of devices. The computing devices 1100 may be connected in a variety of ways, including being directly connected, indirectly connected via a network, distributed across a wide geographical area, or connected via a network (which may be referred to as “cloud computing”).

[0303] For example, but not limited to, computing device 1100 could be a server, network appliance, set-top box, embedded device, computer expansion module, personal computer, laptop, personal data assistant, mobile phone, smartphone device, UMPC tablet, video display terminal, game console, or various other computing devices that can be configured to perform the methods described herein.

[0304] The applicant would like to emphasize that the embodiments and examples described herein are illustrative and non-limiting. Practical implementations of the features may involve combinations of some or all of the embodiments, and the features described herein should not be considered indicators of future or existing product plans.

[0305] The terms "connected" or "joined" can include both direct joining (two elements joined together are in contact with each other) and indirect joining (at least one additional element is located between the two elements).

[0306] While embodiments are described in detail, it should be understood that various changes, substitutions, and modifications can be made without departing from the scope of this specification. Furthermore, the scope of this application is not intended to be limited to specific embodiments of the processes, machines, manufactures, composition of materials, means, methods, and steps described herein.

[0307] Throughout the above description, numerous references may be made to packet controllers, servers, instances, interfaces, portals, platforms, or other systems formed from computing devices. It should be understood that the use of such terms is considered to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer-readable, tangible, non-temporary medium. For example, a packet controller may implement the technical solutions of the described embodiments in the form of a software product. The software product may be stored on a non-volatile or non-temporary storage medium, which may be a CD-ROM, USB flash disk, or hard disk (removable or otherwise). The software product includes several instructions that enable a computer device (e.g., a packet controller or network device) to perform the methods provided by the embodiments.

[0308] For the purposes of this matter, the examples described and illustrated above are intended to be illustrative only.

Claims

1. A packet communication controller coupled to a plurality of network connections, each having a corresponding output queue from a plurality of output queues, wherein the packet communication controller is a processor, The network characteristics associated with each of the plurality of networks are monitored and maintained so that each of the plurality of networks can be ordered in a sequential order based on the network characteristics. When it is determined that a trigger event has occurred, Based on the aforementioned network characteristics, the sequence is established for the network connections of a selected set of network connections. Assign one or more data packets from the unassigned data packets in the input queue to one or more corresponding output queues of the plurality of networks by assigning the one or more data packets to an available network connection based on availability and the sequence of the data packets, The monitored characteristics of the corresponding network are updated to reflect the one or more data packets that have been allocated, and Includes a processor configured to propagate the assigned data packets from the plurality of output queues to a destination device, The aforementioned processor, For each of the aforementioned network connections, the characteristics to be monitored are as follows: To estimate at least one of the following: the transmission rate of the bottleneck link, the round-trip time, and the round-trip propagation time, To establish a score value for each of the estimated round-trip propagation time, estimated round-trip time, and estimated transmission rate under consideration by comparing at least one of the estimated round-trip propagation time, estimated round-trip time, and estimated transmission rate with each of the reference ranges of at least one of the round-trip propagation time, estimated round-trip time, and estimated transmission rate, The measurement is further configured to be performed by ordering the plurality of networks in the sequential sequence based on the score value for each of the plurality of networks, A packet communications controller wherein the estimation of at least one of the transmission rate, round-trip time, and round-trip propagation time of a bottleneck link includes estimations of a plurality of score values ​​for at least two of the transmission rate, round-trip time, and round-trip propagation time of the bottleneck link, the sequential sequence is based on a comparison of the score values ​​corresponding to a predefined most important characteristic, and if the score values ​​corresponding to the predefined most important characteristics are equal, the sequence proceeds to a predefined next most important characteristic for comparison.

2. The packet communication controller is a push scheduler, and the processor determines whether the monitored characteristics of the plurality of networks are suitable. It is further configured to measure the transmission capacity of each of the aforementioned multiple network connections, The packet communication controller according to claim 1, wherein the controller is configured to allocate one or more data packets while the transmission capacity of one or more of the plurality of network connections is not exhausted.

3. The packet communication controller according to claim 1, further configured to allocate the one or more data packets when the processor detects a new data packet in the input queue or when it receives an acknowledgment that one or more previous data packets have been received by the destination device.

4. The packet communication controller according to claim 1, wherein the processor is configured to perform flow classification of the unassigned data packets, and based on the flow classification, the predefined most important characteristic and the predefined next most important characteristic are identified.

5. The aforementioned processor, For each of the aforementioned network connections, the modified characteristics to be monitored, including system overhead processing time, are measured, and either the original characteristics or the modified characteristics are used. Determine at least one of the estimated values ​​of the transmission rate, round-trip time, round-trip propagation time, bandwidth delay product (BDP), and congestion window of the bottleneck link. The packet communication controller according to claim 1, wherein the order of the plurality of networks is further based on at least one of the estimated values ​​of the transmission rate of the bottleneck link, the round trip time, the round trip propagation time, the BDP, or the congestion window.

6. The aforementioned processor, The packet communication controller according to claim 5, further configured to modify the measured characteristics by dynamically adjusting the duration of one or more corresponding sliding window histories based on at least one of the plurality of network connections, which includes packet loss or the change in round-trip time.

7. The aforementioned processor, Considering the round-trip time and packet loss of the aforementioned network connection, a Math coefficient data value is estimated that represents the relative upper limit of the throughput achievable by the aforementioned network connection among the multiple network connections. The system is further configured to compare the estimated Math coefficient data values ​​with a reference range of Math coefficient data values. The packet communication controller according to claim 1, wherein the ordered sequence of the plurality of networks is further based on the comparison of the estimated Math coefficient data value with the reference range.

8. The packet communication controller according to any one of claims 5 to 7, wherein the processor is configured to perform flow classification of the unassigned data packets, and based on the flow classification, the predefined most important characteristic and the predefined next most important characteristic are identified.

9. A method for packet communication using a packet communication controller coupled to multiple network connections, each having a corresponding output queue from multiple output queues, The network characteristics associated with each of the plurality of networks are monitored and maintained so that each of the plurality of networks can be ordered in a sequential order based on the network characteristics. When it is determined that a trigger event has occurred, Based on the aforementioned network characteristics, the sequence of network connections is established for a selected set of network connections. Assigning one or more data packets from the unassigned data packets in the input queue to one or more corresponding output queues of the plurality of networks by assigning the one or more data packets to an available network connection based on availability and the sequence of the data packets, Updating the monitored characteristics of the corresponding network to reflect the one or more data packets that have been allocated, This includes transmitting the assigned data packets from the plurality of output queues to the destination device, For each of the aforementioned network connections, the characteristics to be monitored are as follows: To estimate at least one of the following: the transmission rate of the bottleneck link, the round-trip time, and the round-trip propagation time, To establish a score value for each of the estimated round-trip propagation time, estimated round-trip time, and estimated transmission rate under consideration by comparing at least one of the estimated round-trip propagation time, estimated round-trip time, and estimated transmission rate with each of the reference ranges of at least one of the round-trip propagation time, estimated round-trip time, and estimated transmission rate, The measurement further includes ordering the plurality of networks in the sequential sequence based on the score value for each of the plurality of networks, A method wherein the estimation of at least one of the transmission rate, round-trip time, and round-trip propagation time of a bottleneck link includes estimations of a plurality of score values ​​for at least two of the transmission rate, round-trip time, and round-trip propagation time of the bottleneck link, wherein the sequential sequence is based on a comparison of the score values ​​corresponding to a predefined most important characteristic, and if the score values ​​corresponding to the predefined most important characteristics are equal, the comparison proceeds to the predefined next most important characteristic.

10. The packet communication controller is a push scheduler, and the method is This includes measuring the transmission capacity of each of the aforementioned plurality of network connections. The method of claim 9, wherein the allocation of one or more data packets is allocated to the corresponding connection among the plurality of network connections only if the transmission capacity of the corresponding connection among the plurality of network connections has not been exhausted.

11. The method according to claim 9, wherein when a new data packet is detected in the input queue, or when an acknowledgment is received that one or more previous data packets have been received by the destination device, the allocation of one or more data packets is performed.

12. The method according to claim 9, further comprising performing a flow classification of the unassigned data packets, wherein the predefined most important characteristic and the predefined next most important characteristic are identified based on the flow classification.

13. For each of the aforementioned network connections, the modified characteristics to be monitored, including system overhead processing time, are measured, and either the original characteristics or the modified characteristics are used. Determine at least one of the estimated values ​​of the transmission rate, round-trip time, round-trip propagation time, bandwidth delay product (BDP), and congestion window of the bottleneck link. The method according to claim 9, wherein the sequential sequence of the plurality of networks is further based on at least one of the estimates of the transmission rate of the bottleneck link, the round trip time, the round trip propagation time, the BDP, and the congestion window.

14. The method according to claim 13, further comprising modifying the measured characteristics by dynamically adjusting the duration of one or more corresponding sliding window histories based on at least one of the plurality of network connections, which includes packet loss or the change in round trip time.

15. Considering the round-trip time and packet loss of the aforementioned network connection, estimate a Math coefficient data value that indicates the relative upper limit of the throughput achievable by the aforementioned network connection among the multiple network connections. The method further includes comparing the estimated Math coefficient data values ​​with a reference range of Math coefficient data values ​​to determine the Math coefficient score, The method according to claim 9, wherein the ordered sequence of the plurality of networks is further based on the comparison of the estimated Math coefficient data values ​​with the reference range.

16. A computer-readable recording medium that stores computer-interpretable instructions that cause a processor to perform an action, The aforementioned operation, A method for packet communication using a packet communication controller coupled to multiple network connections, each having a corresponding output queue from multiple output queues, The network characteristics associated with each of the plurality of networks are monitored and maintained so that each of the plurality of networks can be ordered in a sequential order based on the network characteristics. When it is determined that a trigger event has occurred, Based on the aforementioned network characteristics, the sequence of network connections is established for a selected set of network connections. Assigning one or more data packets from the unassigned data packets in the input queue to one or more corresponding output queues of the plurality of networks by assigning the one or more data packets to an available network connection based on availability and the sequence of the data packets, Updating the monitored characteristics of the corresponding network to reflect the one or more data packets that have been allocated, This includes transmitting the assigned data packets from the plurality of output queues to the destination device, For each of the aforementioned network connections, the characteristics to be monitored are as follows: To estimate at least one of the following: the transmission rate of the bottleneck link, the round-trip time, and the round-trip propagation time, To establish a score value for each of the estimated round-trip propagation time, estimated round-trip time, and estimated transmission rate under consideration by comparing at least one of the estimated round-trip propagation time, estimated round-trip time, and estimated transmission rate with each of the reference ranges of at least one of the round-trip propagation time, estimated round-trip time, and estimated transmission rate, The measurement further includes ordering the plurality of networks in the sequential sequence based on the score value for each of the plurality of networks, A computer-readable recording medium wherein the estimation of at least one of the transmission rate, round-trip time, and round-trip propagation time of a bottleneck link includes estimations of a plurality of score values ​​for at least two of the transmission rate, round-trip time, and round-trip propagation time of the bottleneck link, the sequential sequence is based on a comparison of the score values ​​corresponding to a predefined most important characteristic, and if the score values ​​corresponding to the predefined most important characteristics are equal, the sequence proceeds to a predefined next most important characteristic for comparison.

Images