Systems and methods for communicating using hybrid wide area network

Abstract

An apparatus for coordinating data communications across wide area network (WAN) connections, the apparatus comprising a processor coupled with a computer memory and a non-transitory computer readable storage medium, the processor configured to: monitor, for each WAN connection, latency and packet loss probability characteristics of packets transmitted over the WAN connection; identifying, for each packet having a delay / jitter preference among the plurality of packets being routed, an adjusted target delay (ATL) based at least on a per-packet deadline to transmit the packet to a target endpoint; grouping using the per-packet ATL and the plurality of WAN connection characteristics to establish a plurality of layered packets such that each WAN connection is grouped into a corresponding layered packet of the plurality of layered packets; and transmitting the packet using one or more selected WAN connections, the one or more selected WAN connections being selected using at least the plurality of layered packets.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional U.S. application No. 63 / 350,654, entitled “Systems and Methods for Communications Using Blended Wide Area Networks,” filed on June 9, 2022, and claims all the benefits of said U.S. application. This application is incorporated by reference.

[0003] This application is related to the following applications, and the following applications are incorporated by reference in their entirety:

[0004] PCT Application No. PCT / CA2017 / 051584, filed on December 21, 2017, and entitled “PACKET TRANSMISSION SYSTEM AND METHOD”.

[0005] PCT Application No. PCT / CA2021 / 050732, filed on May 28, 2021, and entitled “SYSTEMS AND METHODS FOR DATA TRANSMISSION ACROSS UNRELIABLE CONNECTIONS”

[0006] PCT application No. PCT / CA2022 / 050077, filed on January 19, 2022, entitled “SYSTEMS AND METHODS FOR PUSH-BASED DATA COMMUNICATIONS”. Technical Field

[0007] Embodiments of the present disclosure relate to the field of electronic data packet communications, and more particularly, embodiments relate to apparatus, systems, and methods for push-based data packet communications in the context of multiple network connections (eg, wide area network connections). Background Art

[0008] Existing packet communication solutions for transmitting packets over multiple available network connections rely on static routing logic to determine how to deliver packets over the connections. For example, a system may use a timer for each of the available connections, and the corresponding timer is used to determine when to pull a packet from an input queue for transmission over the connection associated with the timer. These solutions assume that the connections have similar operating characteristics and availability, trading performance for implementation simplicity, thereby scheduling the next available packet for transmission over the connection with the most recently expired timer. While the timer approach is simple, it may compromise computational efficiency in some cases.

[0009] If multiple networks have different operating characteristics, a communication system (e.g., LACP, ML-PPP) that treats multiple network connections as multiple connection instances with similar operating characteristics may result in inefficient use of the multiple connections because some networks may be underutilized or overutilized. This inaccuracy in use may result in longer delays in delivering packets, reduced reliability of packet delivery, higher packet delivery costs, reduced throughput, etc.

[0010] Connection management solutions also experience difficulty in handling applications where the available connections or their operating characteristics vary over time. For example, a timer-based system may be insensitive to changing network performance because the timer for each connection is started independently and may not have accurate knowledge of the network performance of all other connections.

[0011] What are desired are systems and methods for data communications that more efficiently utilize multiple connections to transmit data packets with reduced latency, increased throughput, more reliable data packet delivery, or lower data packet delivery costs. Summary of the invention

[0012] Providing a reliable communication mechanism that can satisfy different and competing communication requirements by various applications running on a networked device is technically challenging. The communication requirements from each application compete with each other for limited communication resources, and computational methods are required for controlling the coordination and use of limited communication resources. These challenges are further complicated in the case of multiple wide area networks (WANs), where a control device (such as a network router) is coupled to multiple WAN connections and is configured to control how packets are communicated across one or more interfaces coupled to the multiple WAN connections.

[0013] For example, a network router may be an electronic device or onboard processor having routing logic or circuitry configured to control packet communications (eg, packet delivery and / or packet reception) as required by various client applications.

[0014] The network router can be used in practical embodiments, such as controlling the communication of a mobile or fixed communication center, such as a portable media car (e.g., for broadcasting) or a data center facility (e.g., an emergency dispatch center). In these cases, the network router utilizes multiple WANs to improve uptime and reliability, such as where a single connection may be prone to communication failures (e.g., high demand periods after an emergency or denial of service attack) or communication challenges (e.g., when the car is driving around, the spectrum characteristics of the connection to each cellular tower or satellite change). Because multiple WANs are used, there are different options and available communication pathways, and additional methods for sorting and / or possibly reordering / regenerating sequences at the recipient endpoint are required. For example, a WAN can be used to intelligently distribute packets between them to control data packets corresponding to a specific media broadcast to communicate separately across multiple WANs, and as described herein, redundant packets may also be sent across connections at the expense of efficiency to improve overall reliability. Examples of using multiple WANs may include routers with multiple SIM slots, multiple wired Ethernet connection ports, and / or multiple satellite connections, etc.

[0015] Packet communications may be requested by different types of client applications, and it is important to note that these communications may have different types of communication requirements, namely (i) requirements that emphasize throughput (e.g., bulk file transfers), or (ii) requirements that emphasize latency / jitter (e.g., real-time uses such as VoIP, video conferencing). There may be multiple types of client applications that have communication needs at or around the same time (e.g., media trucks at a sporting event running applications simultaneously, uploading files for post-production over a few hours, sending low-quality data feeds for quick previews, and holding video conferences for sideline reporters), and these client communications compete with each other for limited, constrained resources.

[0016] As described herein, latency / jitter-intensive requirements are particularly difficult to meet because their packet flows are coupled to real-time delivery deadlines, which require packets to arrive in a timely manner with minimal jitter / loss. When packets do not arrive in a timely manner or when jitter or loss occurs, there may be problems such as stuttering, freezing, and pauses in voice and video. In contrast, applications where flows have throughput-intensive requirements may be able to handle latency and jitter, for example, by using reordering and buffers.

[0017] In one suggested method, an apparatus for coordinating data communications across multiple wide area network connections is proposed, the apparatus being configured as a packet routing device having a processor coupled to a computer memory and a non-transitory computer-readable storage medium.

[0018] The device is configured to monitor, for each of the multiple WAN connections, the latency, packet loss rate, and throughput characteristics of packets transmitted on the WAN connection; for each packet with a latency / jitter preference among the multiple packets being routed, identify an adjusted target latency (ATL) based at least on a per-packet deadline for communicating the packet to a target endpoint; group using the per-packet ATL and the multiple WAN connection characteristics to establish a plurality of layered packets such that each of the multiple WAN connections is grouped into a corresponding layered packet among the multiple layered packets; and communicate the packets using one or more selected WAN connections selected from the multiple WAN connections, the one or more selected WAN connections being selected using at least the multiple layered packets.

[0019] In one aspect, WAN connection latency and packet loss characteristics include estimated delivery time (EDT).

[0020] The wide area network connections belonging to each of the plurality of layered packets are considered to be functionally equivalent in order to select the wide area network connection for communication, and the plurality of layered packets may include at least a first layer (layer 1), a second layer (layer 2), and a third layer (layer 3), the layers being based on the per-packet ATL and the plurality of wide area network connection characteristics EDT and one-way propagation delay (T prop The number of layers is not limited to three, and in a variant, the plurality of stratified groups includes further layers that subdivide layer 1, layer 2, and layer 3 based on one or more predefined connection priority rules, so that a new set of layers is formed.

[0021] The selection of the selected WAN connection from the available WAN connections is performed iteratively, for example, from the available WAN connections in layer 1, and then layer 2, and finally layer 3, and the communication of the packet can use the selected WAN connection selected from the plurality of WAN connections. For example, the scheduling of the packet can be performed on the plurality of connections selected from the plurality of WAN connections for preemptive retransmission.

[0022] Each of the WAN connections in each layer can be assigned a layer-based score, and scheduling of the packets on multiple connections is performed to achieve a cumulative score greater than or equal to a predefined target score, and in a variation, if scheduling of the packets on multiple connections on better (e.g., alternate, higher, lower) layers fails to achieve the cumulative score greater than or equal to the predefined target score, the connection priority rules are ignored. In some embodiments, a useful bypass mechanism is provided. Connections designated as unreliable or not having recent monitoring characteristics can be assigned a layer-based score of 0, regardless of their corresponding layer level, to encourage the use of more reliable and / or recent characteristics. Such connections are still used for packet transmission in order to measure their updated monitoring characteristics, which will determine whether they have become reliable again.

[0023] Selecting the selected WAN connection among the available WAN connections in the layered grouping may include pseudo-randomly selecting an available WAN connection, so that for a data stream corresponding to a packet, an initial selection of the selected WAN connection includes an initial pseudo-random selection, and for future packets corresponding to the data stream, the selection of the selected WAN connection is maintained, for example until a connection dormancy period threshold has been reached or an explicit stream termination notification has occurred. This helps the WAN connection to be more "sticky", so that the system is biased to use a specific stream (if available) for a consistent WAN connection within a specific time. The technical benefit of having a certain level of mandatory consistency is that it reduces the inefficiency associated with instantiating or starting different connections for use. For example, this can be used to avoid a scenario in which the system constantly selects between two or more different connections based on a small amount of improvement, thereby causing data communication to be inefficient due to the non-ideality introduced by opening and closing different connections for use.

[0024] The device can segment and / or classify data streams into (i) throughput preference streams, and (ii) real-time streams; and only packets from the real-time streams are identified as packets with latency / jitter preference, so that grouped layers are established for packets from the throughput preference streams based on the same comprehensive characteristics based at least on the characteristics measured by one or more of the multiple wide area network connections.

[0025] In some embodiments, the selected WAN connection is selected among the available WAN connections in the tiered grouping not using pseudo-random selection, but rather based on the connection with the shortest backlog in the tiered grouping. The backlog is measured in time units (typically milliseconds) and is defined as the volume of in-flight (transmitted but unacknowledged) bytes divided by the transmission rate (throughput) of the WAN connection. When a flow requires more bandwidth than any single connection can provide on its own, selecting the connection with the shortest backlog minimizes the delay experienced by the flow.

[0026] In some embodiments, the device may track the per-packet deadline of packets sent on the first wide area network connection that are approaching the deadline but have not yet been acknowledged; and use a packet with an available congestion window (CWND) and a sufficiently short T prop and a second wide area network connection with a backlog value, performing early retransmission on the packet that is approaching the deadline but has not been confirmed, so as to communicate the packet that is approaching the deadline but has not been confirmed to the corresponding target endpoint, or determining that there is no available congestion window (CWND) and a sufficiently short T prop and backlog value to communicate the packet; and in response to the determination, using one or more outdated wide area network connections to perform early retransmission of the packet that is approaching the deadline but has not yet been acknowledged. Additionally and / or simultaneously, early retransmission can be performed, for example, on the first wide area network connection originally used to transmit the packet.

[0027] In another embodiment, a method is proposed, that is, before communicating the packets using the selected wide area network connection, priorities are established between flows to which the packets belong. The priorities are translated into per-flow defined packet queues, and packets are unloaded from the defined packet queues based on a number of modified differential round robin (DRR) scheduling states.

[0028] Corresponding methods and non-transitory computer-readable media are contemplated. As described in various embodiments herein, the system can be implemented as a networking controller that operates as a physical networking device for controlling communications across multiple network interfaces and / or network connections. The networking controller can be implemented, for example, as a specific hardware device, or as a chipset or set of embedded logic gates integrated within another hardware device or operating on a computer server or software.

[0029] The applied use relates to controlling networking connections in situations where the connection is unreliable, there are a large number of connections to choose from, but the networking resources are limited and / or there may be sudden load spikes or outages. For example, the networking method can be used for emergency dispatch control, media broadcasting (e.g., in rural areas or congested areas).

[0030] This can be useful in situations where there are multiple demanding applications running together, or even where there is a single non-demanding application but the connection itself is failing or experiencing problems.

[0031] Real-world scenarios include sports broadcasting, live event broadcasting, public safety, high uptime / critical infrastructure, etc. The method helps leverage a combination of connections to help improve the overall delivery and communication characteristics of data packets, and is particularly beneficial if the connections have different networking characteristics that vary over time (e.g., with congestion), vary with the location of the device (e.g., the device is mobile), and also vary with the price and / or standard of service delivery (e.g., a best-effort contention connection shared across multiple users). BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In the accompanying drawings, embodiments are shown by way of example. It should be expressly understood that the description and drawings are for illustrative purposes only and as an aid to understanding.

[0033] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0034] Figure 1A is a schematic diagram of a pull communication system according to some example embodiments;

[0035] Figure 1B is a schematic diagram of a push communication system according to some example embodiments;

[0036] Figure 1C is a flow chart of a method for transmitting data packets to push packets to multiple networks based on availability according to some example embodiments;

[0037] Figure 1D is a flow chart of another method for transmitting packets to order connections based on connection metadata according to some example embodiments;

[0038] Figure 1E is a flow chart of another method for transmitting data packets to assign packets to networks based on a selected ordering of the networks according to some example embodiments;

[0039] Figure 1F is a process flow illustrating dimensions for sequencing network connections to control routing of data packets, according to some example embodiments;

[0040] Figure 2 is a schematic diagram of scheduling packets of different priority levels according to some example embodiments;

[0041] Figure 3 is a method of scheduling packets of different priority levels according to some example embodiments Figure 1B A series of schematic diagrams of a push communication system;

[0042] Figure 4A is the use according to some example embodiments Figure 1BSchematic diagram of connection selection for a push communication system, taking into account connection and flow categories;

[0043] Figure 4B is the use according to some example embodiments Figure 1B Schematic diagram of data packet selection in a push communication system;

[0044] Figure 4C is the use according to some example embodiments Figure 1B Schematic diagram of data packet allocation of a push communication system;

[0045] Figure 4D is the use according to some example embodiments Figure 1B Another schematic diagram of data packet allocation of a push communication system;

[0046] Figure 5A is a method of scheduling two independent data streams for transmission according to some example embodiments Figure 1B Schematic diagram of a push communication system;

[0047] Figure 5B is a method of scheduling two independent data streams for transmission according to some example embodiments Figure 1B Schematic diagram of a push communication system;

[0048] Figures 6A to 6E each showing a corresponding screenshot of an interface for managing data packet transmission according to an example embodiment; and

[0049] Figure 7 is an illustration of data packet transmission according to some example embodiments;

[0050] Fig. 8A Included are two diagrams showing illustrations of unidirectional data packet transmission according to some example embodiments;

[0051] Figure 8B includes two diagrams showing illustrations of data packet transmission occurring in response to an acknowledgement according to some example embodiments;

[0052] Figure 8C are two additional illustrations of data packet transmission occurring without requiring an acknowledgement to send additional data packets in accordance with some example embodiments;

[0053] Fig. 9A is a graphical illustration of actual throughput data in flight according to some example embodiments;

[0054] Fig. 9B is another illustration of actual throughput data in flight according to some example embodiments;

[0055] Fig. 9Cis an additional illustration of actual throughput data in flight according to some example embodiments;

[0056] Fig. 10A is a series of diagrams illustrating packet transmission where a connection experiences congestion according to some example embodiments;

[0057] Fig. 10B is a series of diagrams illustrating transmission of data packets following adjustment of transmission capacity in response to a connection experiencing congestion, according to some example embodiments;

[0058] Fig.11 is an example block schematic diagram illustrating an overall hybrid connectivity system operable as a network gateway according to some example embodiments;

[0059] Fig. 12A is a schematic diagram of a per-stream communication system according to some example embodiments;

[0060] Fig. 12B is a schematic diagram of a per-stream communication system according to some example embodiments;

[0061] Fig.13A is a diagram illustrating the calculation of estimated time delivery according to some example embodiments;

[0062] Fig. 13B is a diagram illustrating the calculation of estimated time delivery according to some example embodiments;

[0063] Fig.14 is a schematic diagram illustrating a wide area network connection in different stratified packets showing three different layers based on estimated time delivery according to some example embodiments;

[0064] Fig.15A is a diagram illustrating an example of pseudo-random hashing and flow viscosity according to some example embodiments;

[0065] Fig. 15B is a diagram illustrating stickiness expiration after idle time and after a TCP FIN exchange according to some example embodiments;

[0066] Fig. 15C is a diagram showing how pseudo-random selection helps keep packets of a single flow on the same connection and also how increased demand traverses multiple layers according to some example embodiments;

[0067] Fig.16 is a diagram showing an example of multiple connections sequenced into multiple layers for sending many packets according to some example embodiments;

[0068] Fig.17 is a diagram illustrating an example of early retransmission according to some example embodiments;

[0069] Fig.18 is a diagram showing an example of multiple connections for multiple layers for sequencing of a number of packets for sending a throughput flow according to some example embodiments;

[0070] Fig.19 An example of a 2-level Deficit Weighted Round Robin (DWRR) queue management algorithm is shown in accordance with some example embodiments;

[0071] Fig. 20 is a block diagram of a per-flow packet receiver according to some example embodiments;

[0072] Fig.21 is a block diagram of a per-stream communication system workflow according to some example embodiments;

[0073] Fig. 22 is a block diagram illustrating the workflow of real-time streaming according to some example embodiments;

[0074] Fig.23 is a block diagram illustrating a workflow for accounting for proactive retransmission of in-flight packets, according to some example embodiments;

[0075] Fig.24 is an illustration of a per-stream communication system implementation according to some example embodiments;

[0076] Fig.25 is an illustration of a per-stream communication system implementation according to some example embodiments;

[0077] Fig.26 is an illustration of a per-stream communication system implementation according to some example embodiments;

[0078] Fig. 27 is an example schematic diagram of a computing device according to an example embodiment; and

[0079] Fig.28 is a diagram depicting a physical computer server rack, according to some example embodiments.

[0080] Fig.29 is a diagram depicting practical variations using a computing device with reduced computing power, according to some example embodiments. DETAILED DESCRIPTION

[0081] Systems, methods and apparatus for transmitting data packets when multiple transport connections are available are described, alternatively referred to as hybrid routers or hybrid systems. The method will be expanded below.

[0082] Existing solutions for transmitting data packets over multiple available transport connections are typically implemented as a "pull" based architecture, where data packets are transmitted in response to events generated or defined by a device or system, such as internal timers associated with the multiple available connections.

[0083] The internal timer may be responsive to individual connection parameters (eg, a timer duration configured to reduce the recovery time of a retransmitted packet), but not to connection parameters of other available connections.

[0084] Due to the independence of timer events, pull-based systems may be underutilized (e.g., resulting in idle connections when the timer value is too long) or overutilized (e.g., packets may be allocated to less reliable connections despite poor performance), which may result in longer delays in delivering packets, reduced throughput, reduced packet delivery reliability, higher packet delivery costs, etc.

[0085] The methods, systems and apparatus proposed herein transmit data packets according to a push-based architecture to transmit the data packets over multiple available connections. The disclosed methods, systems and apparatus include a scheduler configured to retrieve a subset of operating characteristics (monitored characteristics) of each of a plurality of connections in response to a monitored communication event, and iteratively allocate data packets available for transmission to any of the determined connections based on their monitored characteristics to have transmission capacity. In an example embodiment, other monitored characteristics of the network can be combined with the transmission requirements associated with the data packets to allocate the data packets to the connections that match the transmission requirements of the data packets.

[0086] The use of a push-based architecture to schedule packets in situations where multiple connections are available is something that has not been previously used. Typical multipath systems (alternatively referred to as hybrid connection systems) assign packets and flows to connections on a static basis because the system does not take into account flow requirements or operational characteristics of the connections. For example, a multi-homed Internet Protocol (IP) router assigns packets to connections purely based on the destination IP address in the packet header, thereby matching its destination routing table. It is simpler to implement a pull-based architecture when connection assignments do not change frequently. For example, there is no need to handle situations where a push scheduler has assigned a large queue of packets to a connection, but now must recall or reallocate because the connection has gone offline.

[0087] The proposed method, system and apparatus receive a data stream, including one or more data packets, from an input network via an input connection.

[0088] The proposed method, system and apparatus include a three-stage data pipeline that transmits data packets to a remote peer over a hybrid set of connections. New data intended for multiple connections is passed to a first pipeline stage where it is queued and then passed to a second pipeline stage associated with each connection in the hybrid connection. The second pipeline stage compiles statistics for its underlying connections and passes packets to a third pipeline stage. The second stage may provide congestion control functions and packet buffering to ensure that packets are available for the next pipeline stage. The third stage may then write packets to a network interface based on a pacing or timing hint provided by the second stage.

[0089] Another embodiment may have a single second-stage instance that then provides packets to third-stage instances, each of which manages packet transmission on a set of network interfaces that share a physical interface card. Other embodiments may have different numbers of instances of each pipeline stage in the pipeline stage.

[0090] Figure 1A is a schematic diagram of a pull communication system 100A, according to some example embodiments.

[0091] When each connection timer (e.g., timers 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively) wakes up, the connection timer invokes the pull scheduler 130 and requests packets from the input queue 102 of packets for transmission. Packets may be requested in bursts, with the burst size nominally reflecting 20 milliseconds (1 / 50 Hz) worth of bytes at the estimated bit rate of the corresponding connection for each timer. A limitation associated with such pull timers is that the number of bytes requested each time the timer wakes up does not reflect the full congestion window (representing, for example, "capacity") of the connection.

[0092] Because each timer is started independently, the pull scheduler 130 does not have visibility into the overall transmission status and cannot make globally optimal scheduling decisions. For example, connection 110A may be a terrestrial broadband connection and connection 110B may be a bandwidth on demand (BoD) satellite connection. Connection 110A may be preferred over connection 110B due to cost, latency, etc., however, when connection 110B timer 108B wakes up and calls into the pull scheduler 130, the pull scheduler 130 does not have visibility into the next wake-up time of connection 110A timer 108A (because timer 108A is started independently). The pull scheduler 130 does not have a basis for determining whether connection 110A will be able to process all of the packets in input queue 102, or whether connection 110B should be assigned additional packets to complete the transmission.

[0093] If connection 110B does help, then pull scheduler 130 may have no way of determining which connection should be used to transmit which packets available for transmission, since the pull scheduler only includes visibility into the timer.

[0094] The data packets are transmitted across the connection and are subsequently received by a receiver 116 via connection 114. The receiver 116 includes a sequencer 118 for assembling the transmitted data packets and may further transmit the assembled data packets via a local area network (LAN) 120.

[0095] Compared to pull-based architectures, in one embodiment, the proposed system is adapted to schedule data packets to specific connections to improve effective connection utilization. However, not all architectures considered necessarily need to be pull-based.

[0096] Figure 1B is a schematic diagram of a push communication system 100B according to some example embodiments. In an example embodiment, the system 100B is designed as a general connection type using multiplexing technology to provide multiple access (MA) to a shared communication channel. For example, in a code division multiple access (CDMA) network, orthogonal codes assigned to each node allow those nodes to share a set of frequencies and transmit simultaneously without interfering with each other.

[0097] The characteristic of the connection types using these multiplexing techniques is that available capacity is quickly reallocated to nodes in the network, typically on the order of tens to hundreds of milliseconds. In the exemplary embodiment, the system 100B is designed with this characteristic in mind, thereby reallocating traffic between available WAN connections based on this time scale.

[0098] In contrast, connections using Demand Assigned Multiple Access (DAMA) technology, also known as Bandwidth on Demand (BoD), share a communication channel by having nodes negotiate access to the channel for time periods. Once assigned, nodes can use the negotiated portion exclusively. Thus, sharing of available capacity is achieved by nodes using the negotiated channel portions in turn.

[0099] In the case of BoD connections, this negotiation and subsequent reallocation of capacity between nodes typically takes on the order of seconds to complete (reallocation times, typically mid-single digits to low double digits). In an example embodiment, system 100B will require changes (described herein) to effectively use these types of connections.

[0100] In current implementations, long reassignment times (and associated packet loss and / or latency increases) are interpreted as congestion, and typical congestion control methods reduce the use of the connection where congestion is detected to avoid overloading the connection. However, BoD connections actually require the opposite behavior (at least temporarily) in order to request and possibly obtain increased capacity allocations.

[0101] System 100B includes an input queue 102 that receives a data stream (including one or more packets) from an input network for transmission. Examples of data streams may be file transfers, video file transfers, audio file transfers, Voice over Internet Protocol (VoIP) calls, or packets associated with a Virtual Private Network (VPN).

[0102] Input queue 102 is also responsible for applying a loss policy to packets if the scheduler (as described herein) fails to service input queue 102 at the same rate that new packets arrive from LAN side clients. This typically happens if the transmission rate of the LAN clients is higher than the total WAN transmission capacity.

[0103] The system 100B includes three stages, alternatively referred to as pipelines or pipeline stages, for transmitting data packets to remote peers through a hybrid set of connections according to a push-based architecture.

[0104] The first stage 104 includes a flow classification engine 104A, a scheduler 104B (hereinafter alternatively referred to as a push scheduler 104), and receives packets from the input queue 102 and queues the packets for delivery to the second stage 106. In an exemplary embodiment, the first stage 104 includes all of the input queue 102, the flow classification engine 104A, and the scheduler 104B.

[0105] Input queue 102 buffers packets received from clients on the LAN side of the sender, possibly by utilizing or cooperating with flow classification engine 104A, and is responsible for classifying packets into flows (alternatively referred to as mapping packets into flows). Input queue 102 ensures that the scheduler of first stage 104 serves data flows fairly.

[0106] In an example embodiment, the first stage 104 or its auxiliary flow classification engine 104A monitors or is notified of changes to user-configured settings associated with assigning packets to connections. The user-configured settings include information about how connections should be assigned to priority levels, when those priority levels should become active, and user-configurable flow classification rules that can cause packets belonging to a flow to prefer connections with certain monitored characteristics (e.g., packets belonging to a video stream classification may prefer connections with low latency, while packets belonging to a file download classification may prefer connections with lower monetary costs).

[0107] The first level 104 periodically learns current metadata (e.g., operational characteristics) for each of a plurality of connection monitoring instances (as described herein) corresponding to a WAN connection containing a hybrid link. This metadata is collected by the monitoring instance 106 and may include an estimated bottleneck bandwidth of the connection, an estimated shortest round trip time, a most recent actual round trip time, the current congestion window of the connection, and the receive / loss status of packets previously sent to the next level. The first level 104 may also use this metadata to extend the metadata to track an estimate of the bytes it has scheduled as in-flight on this connection, and further break down the total into new bytes, retransmitted bytes, or false / probe bytes. All metadata collected and tracked by the first level 104 is referred to as monitored characteristics (e.g., operational characteristics).

[0108] The first level 104 will then use all available monitored characteristics to determine whether the packet should be sent to the next level, and if so, which connections should receive the packet. The set of packets provided to a connection monitoring instance may include duplicates of packets previously sent (to the same connection monitoring instance, or to a different connection monitoring instance). Alternatively, the scheduler may determine that a given connection monitoring instance is not currently eligible to receive a packet, even if the instance's connections currently have capacity. The allocation of packets to specific level 3 instances is discussed elsewhere in this document.

[0109] The second stage 106 includes a connection monitoring instance for each of a plurality of available connections (e.g., connection monitoring instances 106A, 106B, and 106N associated with connections 110A, 110B, and 110N, respectively). The connection monitoring instances of the second stage 106 monitor the associated connections and determine or retrieve statistics associated with the performance (e.g., latency) of the connections, referred to as operating characteristics. The second stage 106 may provide congestion control functions and packet buffering to ensure that packets are available for the next stage. The second stage 106 passes the packets to the third stage 108, which includes an output queue for each connection (e.g., output queues 108A, 108B, and 108N). The third stage 108 may write the packets to the network interface according to a pace or timing prompt provided by the second stage 106.

[0110] In some embodiments, the second stage 106 may implement one or more router to radio interface (R2RI) protocols (eg, PPPoE (RFC 5578), R2CP, DLEP (RFC 8175)) to better support the bandwidth allocation process for BoD connections.

[0111] The second stage 106 is responsible for receiving packets from the first stage 104, queuing the packets locally, and providing groups of packets as bursts to the next pipeline stage. The second stage 106 also tracks metadata for its associated connection based on acknowledgments received from peers or recipients.

[0112] The purpose of the queues in the second stage 106 is to ensure that there are always packets available for the next pipeline stage to be written to the network device for transmission. Some embodiments may include priority metadata as part of each packet information so that new packets arriving from the first stage 104 can be reordered in the queue to provide preference for higher priority packets.

[0113] In order for the second stage 106 to have enough buffered packets to ensure that packets can be provided to the third stage 108, the second stage 106 may provide altered metadata to the first stage 104. For example, according to some example embodiments, the second stage 106 may choose to report a larger congestion window (i.e., "over-advertise") so that the first stage 104 provides additional packets that can be buffered in the second stage 106 for when the third stage 108 finishes writing the actual congestion window packets. In other words, in some embodiments, the second stage 106 over-advertises the congestion window to account for the processing overhead / delay caused by the entire pipeline 104 / 106 / 108 in order to ensure that the network always remains filled with in-flight bytes.

[0114] In some embodiments, it is known that the pipeline stage or the network itself uses compression techniques that effectively increase the number of bytes in flight that the network can support. In this scenario, "over-advertising" the congestion window to the first stage 104 is also used to ensure that the network always remains filled with bytes in flight.

[0115] The second stage 106 may also provide modified metadata to the first stage 104 to facilitate a change in the allocation of capacity in the BoD connection (i.e., to cause the first stage 104 to schedule packets on this connection as if it had higher capacity in order to trigger the BoD connection to allocate the higher capacity). This modified metadata may include information that breaks down the higher capacity into different requested types / priorities. For example, a higher incremental capacity for probing a BoD connection typically contains redundant / spurious packets because the probing packets are likely to be lost by the BoD connection before the DAMA reallocation process is completed.

[0116] Advertising or over-advertising capacity from the second stage 106 to the first stage 104 need not necessarily immediately result in the detection of higher incremental capacity. The first stage 104 still ultimately makes the decision on how to allocate packets to the connections. One embodiment may wait until the CWNDs on all non-BoD connections are fully consumed (or nearly fully consumed) before instructing the second stage 106 to begin the DAMA reallocation process.

[0117] If a problem occurs with the underlying network connection, the queues in the second stage 106 can also quickly notify the previous stage of a failure. Packets that have been provided to the third stage 108 and written to the network must be assumed to be in transit, even if the network interface subsequently reports a downtime, which means that timeouts must occur before those packets can be marked as lost. However, once the second stage 106 is notified that the connection is no longer available, packets still queued in the second stage 106 can be immediately marked as lost, which makes the packets immediately eligible for priority retransmission on an alternative connection.

[0118] Some embodiments may attempt to explicitly pace the rate at which packets are transmitted by assigning each packet a nominal timestamp that indicates when the next stage should attempt to send the packet over a connection before being passed to the third stage 108. Such timestamps may be determined based on the estimated bottleneck bandwidth of the connection (e.g., the connection to which the packet is assigned) and the packet size in order to prevent network congestion at or before a network bottleneck link.

[0119] Figure 2 An example of an embodiment in which the second stage 106 supports prioritized packets is shown at 200. Figure 2 The head of the queue appears on the right, and the tail of the queue appears on the left. Figure 2 In the top portion (eg, first step) of the process, the first stage 104 provides a list of new prioritized packets to the second stage 106 connection monitoring instance. Figure 2 The bottom portion of (e.g., step 2) shows that these new packets have been added to the priority queue in priority order, so that all priority 1 packets will be dequeued before any priority 2 packets are dequeued. Packets at the same priority level are maintained in sequential order.

[0120] In an example embodiment not shown, the second stage 106 may have a single connection monitoring instance that then provides packets to instances of the third stage, each of which manages packet transmission on a set of network interfaces that share a physical interface card. Other embodiments may have different numbers of instances of each pipeline stage in the pipeline stage.

[0121] Feedback in the form of metadata or callbacks (e.g., the callbacks may be a mechanism for sending metadata) may be sent from one level to the previous level, or a level may receive notifications from different parts of the system. Any of these actions may trigger (i.e., push) a level to send a package to the next level.

[0122] The third stage 108 receives packets from the second stage 106 and writes the packets to the network 112 via one or more connection transport instances (e.g., connection transport instances 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively). In some embodiments, all second stage 106 connection monitoring instances provide packets to a single connection transport instance, or each connection monitoring instance can be associated with a connection transport instance.

[0123] Some embodiments of the third level 108 may use a queuing mechanism to allow a previous level to provide a hint about when a packet should be sent. For example, the hint may be in the form of a "nominal send time" associated with the packet as metadata. If the nominal send time of the next packet to be sent is in the future, the third level 108 will wait until after that time to send the packet. Critical packets may be marked with an immediate timestamp, which may cause the critical packets to be sent ahead of other queued packets. To prevent starvation of non-critical packets, some embodiments may queue critical packets after non-critical packets whose nominal send times have passed.

[0124] Some embodiments may include a callback function as part of the packet metadata when a packet moves through the pipeline from the second stage 106 to the third stage 108. This callback may be used to trigger a notification on the second stage 106 to indicate that the third stage 108 has transmitted the packet. Such a notification may be used to trigger a new batch of packets to be pushed from the second stage 106 to the third stage 108.

[0125] Figure 3 An embodiment of the progression over time is shown at 300, where the second stage 106 provides a packet with a "send at" hint (nominal send time) requesting that the packet not be sent before a specified time. Packets with a "send at" time of 0 should be sent as soon as possible. Figure 3 Examples are shown for all of these situations (eg, using nominal send times in the past for critical packets, and how the system avoids starvation of non-critical packets).

[0126] In the state depicted at t=1, the packet queue of the third level 108 instance contains 4 packets, three of which are currently eligible to be sent, and the second level 106 instance is providing 4 new packets, the first of which has sequence 5 (sequence order 5) and should be sent immediately.

[0127] In the state depicted at t=2, a new packet has been added to the third level 108 priority queue. Note that this embodiment has chosen to queue the packet with sequence 5 for transmission after the packet with sequence 2, because sequence 2 was already eligible to be transmitted at the time sequence 5 was queued. Freezing the order of packets at the head of the packet queue in this manner prevents starvation, i.e., where a continuous flow of new urgent packets to be sent immediately can prevent non-immediate packets from being sent, even though the non-immediate packets may have become eligible to be sent long before the latest urgent packet to be sent immediately arrived.

[0128] In the state depicted at t=3, the third stage 108 has dequeued the first four packets from the output queue and has provided the first four packets to the network interface. Prior to time t=4, there are currently no packets eligible to be sent, so even if there is currently write capacity on the network interface, the third stage 108 instance will need to wait to send more packets until the time advances to t=4, or until a new packet arrives that should be sent immediately or at time t=3.

[0129] Packets may be passed from one stage to the next in a time order determined by each stage as optimal.

[0130] Similar to Figure 1A , the data packets are transmitted across the connection and are subsequently received by a receiver 116 via connection 114. The receiver 116 includes a sequencer 118 for assembling the transmitted data packets and may further transmit the assembled data packets via a local area network (LAN) 120.

[0131] Compared to the system 100A, the push communication system 100B does not include an independent timer associated with each connection - instead, packets are scheduled onto the connections in a centralized location; includes a push scheduler 104 that accesses monitored characteristics of each connection, and includes a push scheduler 104 that is activated in response to the occurrence of one or more monitored events rather than based on a timer. For example, the one or more monitored communication events may include an acknowledgement / negative acknowledgement (ACK / NACK) of a previously transmitted data packet by a recipient, or the arrival of a new data packet at the input queue 102, etc.

[0132] In an example embodiment, the push scheduler 104 may potentially consume or allocate the entire connection (congestion window) CWND instead of transmitting only 20 millisecond packets.

[0133] By accessing the monitored characteristics of each connection, the push scheduler 104 can be configured to optimize packet scheduling decisions. For example, a data stream with a specific transmission requirement can be preferentially allocated to a connection that can meet the requirements associated with the data stream, rather than being subject to allocation based primarily on a timer-based wake-up order. In an illustrative example, a first data stream requires a transmission latency of less than 50 milliseconds.

[0134] Connections 110A, 110B, and 110N have latencies of 5 ms, 45 ms, and 80 ms, respectively. Using the pull scheduler 130, if the 50 Hz timer of connection 110B (45 ms latency) wakes up first, the timer will have allowed packets of this first data flow to be scheduled onto connection 110B. Using the push scheduler 104, the data flow can be scheduled preferentially onto connection 110A (5 ms latency) until the CWND of connection 110A is completely consumed, and then overflow onto connection 110B (45 ms latency).

[0135] To enable bidirectional communication, a single instance of a multipath endpoint can include Figure 1B Specifically, the instances may include one or more instances of the sender 100 to support upstream communications to other multipath endpoints, and one or more instances of the receiver 116 to support downstream communications from other multipath endpoints.

[0136] For example, according to some embodiments, one or more of the available connections are BoD connections, and the push scheduler 104 may cause more data to be assigned to the BoD connections based on criteria such as the CWND of connection 110A being fully consumed (or nearly fully consumed) in order to request more capacity from the network; while more packets that are eligible for transmission on connection 110B (e.g., based on per-flow rules) are available at the input queue 102.

[0137] In an example embodiment, the push communication system 100B can enable more accurate / fair quality of service (QoS) and active queue management (AQM) packet loss rules to be applied at the input queue 102. Referring again to the previous example, a flow with a latency requirement of 50 milliseconds will never be able to utilize an 80 millisecond connection. Since the push scheduler 104 has a consistent snapshot of the entire system, it can calculate the partial usage and fair share of the total capacity for flows excluding the 80 millisecond connection.

[0138] One potential advantage of a push-based scheduling approach is that a full set of metadata for multiple connections containing mixed links is available to the push scheduler 104 to allocate packets to target connections in an optimal manner.

[0139] Figure 1Cis a flow chart of a method 110C for transmitting a data packet using the system 100B or other push communication system, according to some example embodiments.

[0140] At block 1130, method 100C senses when an input queue includes unallocated packets available for transmission.

[0141] At block 1132, method 100C includes assigning one or more of the unassigned packets to one or more corresponding output queues of the plurality of networks having suitable monitored characteristics.

[0142] At block 1134, method 100C includes updating monitored characteristics of a corresponding plurality of networks to reflect the allocated one or more data packets.

[0143] At block 1136, method 100C transmits the allocated data packets from the output queue corresponding to each network of the plurality of networks to the destination device.

[0144] In an example embodiment, each time an external event triggers the push scheduler 104, the method for transmitting a data packet using the system 100B includes a loop of the following steps:

[0145] 1. Request a data packet from the input queue 102;

[0146] 2. Allocate received packets to the connection for transmission; and

[0147] 3. Repeat steps 1 to 2 until there are: a) no more packets available at the input queue 102; or b) no more transmission capacity available on any active connection (eg, as defined by the congestion window (CWND) metric).

[0148] The method may be implemented such that the push scheduler 104 does not take into account any data stream specific requirements. In such embodiments, the system 100B distributes received packets based on the ordering of the connections.

[0149] The sorting may be based on monitored characteristics of the connection, which may include, for example, a subset of the operational characteristics estimated by the congestion control algorithm. In one embodiment, the BBRv1 algorithm is used, but other embodiments may include instances (e.g., BBRv2, TCP Reno, TCP LEDBAT), proprietary implementations to handle specific connection types such as BoD, etc. In the subsequent discussion of the operational characteristics of BBRv1, reference is made to a pipeline analogy, in which two endpoints are modeled as a series of interconnected pipelines in which each pipeline segment may have a different diameter and length, which is for illustrative purposes only.

[0150] The operational characteristic may be a measure of the minimum time required for a transmitted packet to reach the destination endpoint (e.g., the receiver) and be acknowledged (e.g., by an ACK message from the receiver to the sender). This duration is referred to as the "round trip propagation delay" (abbreviated "RtProp") and is measured in time units (e.g., milliseconds). In terms of the pipeline analogy, this value represents the total length of all pipeline segments from the sender to the receiver and back.

[0151] The operating characteristic may be the maximum rate at which data can move between endpoints along the pipeline segment with the smallest diameter. The smallest pipeline diameter, referred to as the "bottleneck" segment, determines the maximum rate at which data can move between the sender and the receiver, and the bottleneck segment is also referred to as the "bottleneck bandwidth" (abbreviated "BtlBw"). For example, a method for estimating BtlBw is described in an IETF draft: https: / / tools.ietf.org / html / draft-cheng-iccrg-delivery-rate-estimation-00.

[0152] Table 1 shows an example of a set of WAN connections and their characteristics, an example subset of operational characteristics that the scheduler 104 selects as characteristics to become monitored, and the resulting ranking order:

[0153] Table 1: Example WAN connections and their sort order

[0154]

[0155] This example method orders the joins by the following dimensions, moving to the next dimension only if the current dimension compares equal across the joins:

[0156] 1. Raise the priority level of preferences;

[0157] 2. Improve RtProp;

[0158] 3. Reduce BtlBw; and

[0159] 4. Add the interface name.

[0160] Note how the results of the sort operation remain relatively stable in the join order, because the sort dimensions are all parameters that are expected to change infrequently.

[0161] This stable sort order is intentional so that packets arriving at the input queue will tend to "stick" to the same set of connections (the connections sorted at the top of the list), only "overflowing" to lower-order connections when the input demand exceeds the total CWND of those connections sorted higher.

[0162] Note how the sorting criteria in this example are not specific to a particular flow. For example, an administrator or end user may only wish to transmit high priority application flows (instances of transmission requirements) over metered cell1 and cell0 connections, which means that those flows would want a resulting sort order that would place cell1 and cell0 lower in the list. Similarly, there may be some flows that have maximum latency constraints, such as VoIP calls, where the end user considers them unusable for interactive conversations if the latency exceeds 150 milliseconds. For these flows, a resulting sort order that uses RtProp as the primary sorting dimension may be desirable. Subsequent paragraphs will discuss embodiments that extend the sorting criteria to specific flows.

[0163] The scheduler is also responsible for deciding the retransmission strategy for packets. In an example embodiment, the system 100B is configured so that if a packet is reported as missed by the recipient, the scheduler always marks the packet for retransmission. This is desirable for many flows, but other flows may not want this at all (because of additional delays or packets may arrive out of order making the application unusable), or may only want to attempt retransmissions within a certain time period.

[0164] Figure 1D is a flowchart of another method for transmitting a data packet according to some example embodiments.

[0165] An example of a simple dispatch loop is as follows:

[0166] 1. Make any required considerations for connection metadata updates and determine the active connection set.

[0167] 2. Determine a preferred connection ranking based on connection metadata (monitored characteristics, such as monitored operational characteristics).

[0168] 3. When there are packets to send and there is available network capacity in the active connection set:

[0169] a. Get the next packet to send.

[0170] b. Queue the packet to be sent on the first connection in the order that has available capacity.

[0171] c. Update the connection metadata (monitored characteristics) to reflect the scheduled packets.

[0172] d. Repeat.

[0173] 4. For each connection that has queued packets, send those packets to the next pipeline stage for that connection.

[0174] For the purpose of preference sorting of connections, such embodiments treat all packets as equivalent. In this case, packets can be considered to belong to the same flow category (even though different packets may logically belong to different flows, but from the perspective of the scheduler, there is no distinction between these flows for the purpose of connection allocation).

[0175] Figure 1E is a flowchart of an additional method of transmitting a data packet according to some example embodiments.

[0176] An embodiment of a scheduler capable of distinguishing between different flow categories (i.e., ICMP echo request / response, versus video flow, versus file transfer), and even distinguishing between individual flows within the same classification (video flow between hosts A and B, versus video flow between hosts A and C) can be provided as follows.

[0177] 1. At block 1102, any required consideration is made for connection metadata updates (monitored characteristics) and the active connection set is determined.

[0178] 2. When the active connection set has capacity (block 1114):

[0179] a. At block 1103, a selector is constructed based on the current active connection set.

[0180] b. At block 1105, a selector is used to select a packet to be sent over one of the available connections. If no packet is selected, the loop is exited.

[0181] c. At block 1107, the ranking of active connections is determined based on the flow and flow classification of the packets in conjunction with the connection metadata (the monitored characteristics).

[0182] d. At block 1110, the packet is queued for transmission on the first connection in the order that has available capacity.

[0183] e. At block 1112, the connection metadata is updated to reflect the scheduled packet.

[0184] f. Repeat.

[0185] 3. At block 1116, for each connection that has queued packets, send those packets to the next pipeline stage for that connection.

[0186] These two sampling methods (shown in Figure 1D and 1E ), which is similar to the method in Figure 1D The method in Figure 1E A simplified and optimized version of the more general method in .

[0187] The first approach can be converted to the second approach by defining a single flow class and defining a selector that matches all packets. For this reason, the second approach ( Figure 1E ), and the same discussion will apply to the first approach ( Figure 1D ), subject to the restrictions set out above.

[0188] The first step is to update any pending metadata considerations (monitored characteristics) to determine the active connection set. In one embodiment, the active connection set is the connection that is not subject to unreliable connection fallback (i.e., because of a recent failure to reliably deliver packets, the use of which is therefore limited).

[0189] Other embodiments may further restrict the set of active connections by, for example, eliminating connections whose metadata (monitored characteristics) do not match any of the currently defined flow class requirements (i.e., assuming that all packets must match one of the defined flow classes, then a connection may be considered inactive if it will never be selected by any flow class; for example, if all flow classes specify a maximum RTT of 500 milliseconds, then connections with RTT>500 milliseconds may be considered inactive).

[0190] If at least one connection in the active connection set has an available CWND (congestion window), which means that the number of in-flight bytes currently tracked by the scheduler for this connection does not exceed the CWND that has been determined for this connection, then the set has capacity to send packets. In one embodiment, the CWND of the connection can be the CWND as reported by the next pipeline stage, or the scheduler can choose to reduce the reported CWND if throttling or priority routing is in effect.

[0191] In one embodiment, the CWND of a connection that is throttled to a certain rate is reduced by the same factor as the throttling rate at the bottleneck bandwidth (or the CWND is unchanged if the throttling rate is higher than the bandwidth estimate). Similarly, if a connection is subject to priority routing (where embodiments support grouping of connections into priority classes that are activated when the total actual throughput of all connections in a higher priority class (excluding overhead throughput such as protocol headers, control packets, and retransmissions) drops below a configured threshold for the class to be activated), the CWND of the connection may be reduced. Various methods for determining CWND reduction in these scenarios will be referenced later. Figures 8A-8C and 9A-9C are described.

[0192] The effective CWND assigned to connections in non-active priority levels (for which the total actual throughput plus the unused bit rate of all connections in higher priority levels reaches or exceeds the activation bit rate of the priority level) is zero; and the CWND of connections in non-primary priority levels is set to zero if the estimated total actual throughput bit rate of all connections in the current active priority level set reaches or exceeds the activation threshold of the lowest active priority level.

[0193] In other embodiments using priority routing, connections in non-primary levels can be capped at a rate below their bottleneck bandwidth so that their contribution is just enough to close the gap with the configured activation threshold. This can be done to save costs, for example, when the non-primary connections are more expensive connections intended to be used only during emergencies. Similar to throttling, the CWND is reduced to reflect the capped rate at which these connections should be sent.

[0194] The loop in step 2 of the method can be performed as long as the connections in the active set have capacity.

[0195] Step 2a (shown in Figure 1E 1003 in the block 1104) constructs a selector for finding packets eligible to be sent on one of the active connections with active capacity. In some embodiments, the selector is composed of a set of flow categories that are compatible with the current set of active connections with capacity.

[0196] For example, if the maximum delay of the stream class requirement connection is 500 milliseconds, and the delay of the active connection with available capacity all exceeds 500 milliseconds, the stream class will be excluded from the selector so. If the embodiment can determine that there is no connection (active or inactive, with or without available capacity) that meets the standard of this type of bag at present, some embodiments can choose to include in the selector the stream class that will usually not match the standard of the active connection with capacity. In this case, the embodiment can choose to violate the matching criteria so that the bag is sent, rather than retaining the bag until it times out and is lost. Other embodiments can choose to enforce strict matching strategy, in which the bag that matches the stream class with the connection that is not qualified will never be sent.

[0197] In another embodiment, shown in Figure 1F The example sort order in the process flow of FIG. 4 may be computationally determined or established using alternative methods to control the routing of data packets to various network connections.

[0198] Before being used to determine the sort order, some or all of the input parameters may be rounded / bucketed. The technical improvement of rounding and bucketing is to keep the sort order as "stable" as possible so that the sort order does not change constantly (for example, without rounding and bucketing, two similar values ​​may fluctuate over time, causing the order to change constantly, resulting in "jitter").

[0199] Factors influencing its decision include: keeping the sort order as "stable" as possible so that packets are not split unnecessarily across multiple WAN connections (splits increase the chance of spontaneous out-of-sequence and jitter events); and considering both measured characteristics of the connection and unmeasured external instructions.

[0200] Values ​​that may be rounded include the following:

[0201] Latency-related values ​​(RtProp, RTT) are rounded to the nearest multiple of 50 milliseconds, with a rounded floor limit of 1 millisecond.

[0202] • Throughput related values ​​(BtlBw) are rounded to the "nearest order of magnitude".

[0203] The packet loss rate (worst value in the 3 sliding windows) is rounded to the nearest order of magnitude. Note that the raw values ​​here are restricted to the range [0,100], so the output values ​​are restricted to the set (0,1,2,3,4,5,6,7,8,9,10,20,30,40,50,60,70,80,90,100).

[0204] Examples of rounding to the nearest order of magnitude are provided in the following table. The original value, its calculated order of magnitude, and then the resulting rounded value are shown.

[0205] Original value (range) Order of magnitude Rounded value 0 1 0 1 1 1 2 1 2 10-14 10 10 15-24 10 20 85000-94999 10000 90000 95000-99999 10000 100000 100000-149999 100000 100000 150000-200000 100000 200000

[0206] In some embodiments, specific improvements based on improving the order of networking routing to control the use of network interfaces are described, wherein the throughput is predicted based on the Mathis Equation using the combined characteristics of round-trip time (RTT) and packet loss rate to establish an upper limit for TCP throughput under a given RTT, packet loss rate, and MSS. The Mathis factor can be used to modify the sorting order used to route networking packets.

[0207] The measured characteristics may include: packet loss rate over 3 independent sliding windows (500 ms, 3 s, 30 s), which is further summarized as the worst value over the three windows; current network round trip time (RTT); Bandwidth Round Trip Propagation Time (BBRv1) parameters: RtProp (network propagation delay); and BtlBw (throughput estimate). External instructions may include priority routing (PR) rules, etc., but may also include flow specific rules / requirements.

[0208] In some embodiments, the measured characteristics of BBRv1 differ from its design in the original BBRv1 IETF draft. For example, to account for embodiments that may be run on systems whose execution times are less consistent, the size of the sliding window used to calculate BtlBw may be increased to cover long durations (hiding variability caused by execution inconsistencies), but the window duration may also be immediately shortened if a network loss or latency increase event occurs, since those events indicate that BtlBw may have actually changed and any long history in the sliding window must be forgotten.

[0209] For similar reasons of inconsistent execution, the input parameters used to measure RtProp can be changed to include a portion that exceeds network propagation delays, thereby adding execution / processing time, as said time may be non-trivial and continue to contribute to actual delays when transmitting packets and waiting for responses. In some embodiments, this is referred to as "System RtProp" rather than "Network RtProp" which consists solely of network propagation delays. In these embodiments, "System RtProp" and "Network RtProp" can be used for different calculations, depending on the intended purpose. For example, when calculating CWND, "System RtProp" is appropriate so that the pipeline contains enough in-flight data to account for processing delays. However, when sorting network connections to determine packet routing, "Network RtProp" is more appropriate because the preferred routing decision does not depend on system execution delays.

[0210] Packet loss, latency, and throughput are not completely independent variables. Depending on the application's congestion control behavior, all three combine to affect each other. This is because, for example, with TCP, higher packet loss results in longer "silent" periods in the air (underutilized channel) while the sender waits for duplicate acknowledgments (DUP ACKs) or retransmission timeouts (RTOs) occur.

[0211] Therefore, some comprehensive features were identified and used in the final sorting method. Specifically:

[0212] Mathis Factor: This comprehensive characteristic takes into account RTT and packet loss rate to predict throughput, and is based on the Mathis Relation that gives an upper limit on TCP throughput for a given RTT, packet loss rate, and MSS. When comparing the characteristics of any two network connections to determine their rank order, the MSS can be considered a constant, so this relationship indicates that the expected throughput is proportional to the inverse of (RTT*sqrt(packet loss rate (PacketLoss))). In this example, the relationship is used as a practical implementation to establish a coefficient for directly controlling the operation of a network controller or router. For each connection, the system determines the Mathis Factor as: maximum (rounded Rtt (RoundedRtt), 1 millisecond) * sqrt (rounded summarized packet loss rate (RoundedSummarizedPacketLoss)). Connections with smaller Mathis Factor values ​​are considered better. The calculation is a function of two independently measured characteristics, so the relationship between the characteristics determines how the connections will compare to each other. The Mathis coefficients may be maintained and stored in a data object, such as a corresponding Mathis coefficient data value associated with each connection, and stored with other monitored characteristics, such as in an array of data values.

[0213] For example, in one extreme case, two connections with 0% packet loss will always have the same Maxis coefficient - their actual RTT value is not important. In this scenario, the Maxis coefficient will not affect the sorting order - the sorting order will be determined by comparing the subsequently measured characteristics with the combined characteristics.

[0214] At the other extreme, if two connections have a packet loss rate of 100% (rounded aggregated packet loss rate = 1.0), only their rounded Rtt values ​​will determine how they compare to each other. In some embodiments, this is undesirable, so connections with a packet loss rate of 100% are filtered out from the available choices at an early stage.

[0215] In between the two extremes, connections with both low RTT and low packet loss will perform better than those with higher values. However, connections with a high percentage of loss can still compensate by having a low RTT and have the same Mathis coefficient as connections with a low percentage of loss. For example:

[0216] Connection A: 100 ms RTT, 0.01 (1%) loss rate => Mathis coefficient = 10

[0217] Connection B: 19 ms RTT, 0.25 (25%) loss rate => Mathis coefficient = 9.5

[0218] In this example, although connection B has a significantly higher loss rate (25% versus 1%), its lower RTT compensates and gives it a better Mathis coefficient than connection A. The explanation for this phenomenon is that although connection A has a lower loss rate percentage, connection B's lower RTT allows it to exchange packet acknowledgments (positive and negative) for lost packets and complete any needed packet retransmissions more quickly than connection A could do using its higher RTT.

[0219] Capacity Factor: This composite characteristic takes into account BtlBw and RtProp, and is determined as: Rounded BtlBw (RoundedBtlBw) / Maximum (Rounded RtProp (RoundedRtProp), 1 millisecond). The units are "Bytes per Second per Millisecond", which have no physical meaning, but provide a way to normalize the bandwidth that a connection can achieve per unit of RtProp. In other words, a connection that can achieve more bandwidth per millisecond with its RtProp is considered more efficient than a connection that achieves less bandwidth. In this context, efficient transport means a connection that achieves more bandwidth per millisecond of propagation delay (RtProp).

[0220] Once all raw inputs have been processed and the composite features calculated as described above, the actual sorting mechanism can be performed sequentially as follows:

[0221] Moving from one comparison to the next only occurs if the first few rows all compare equal. If the comparisons are not equal, the remaining comparisons are irrelevant and are skipped. In some embodiments, the selection of comparisons is made on the basis of the flow classification (e.g., VPN, relative to VoiP, relative to ping packets, relative to FEC packets), and the list of comparison criteria can be different for each different flow (e.g., each flow can have a different predefined most important characteristic, second most important characteristic, third most important characteristic, etc.). Each of these characteristics can be iterated in sequence starting with the most important until the connection itself can be sorted and put into a sequential sequence. In some embodiments, depending on the flow classification, the first pre-step is to select a subset of connections that are suitable for sending data packet types in the flow classification, and then assign an order to them before performing packet distribution.

[0222] 1. A connection that is completely down (aggregated loss rate of 100%) is considered worse than a connection with an aggregate loss rate of < 100%.

[0223] 2.PR priority (the higher the priority, the better)

[0224] 3. Mathis coefficient (the lower the better)

[0225] 4. Rounded RTT (lower is better)

[0226] 5. Capacity Factor (the higher the better)

[0227] 6. Rounded RtProp (lower is better)

[0228] 7. Rounded BtlBw (higher is better)

[0229] 8. Connection ID (the lower the better)

[0230] Figure 1F A rendering of the sorting mechanism is shown in . In an example sorting mechanism, the real-time stream first includes a comparison of RTT / RtProp, and then if equal, it will then compare reliability, and then if also equal, compare transmission rate, etc. Depending on the stream type, there may be a different sorting order. For example, a stream that requires throughput may compare transmission rate first. If equal, then compare reliability. If also equal, the stream may decide not to do other evaluations, and then simply choose arbitrarily.

[0231] In some embodiments, the flow requirements (e.g., real-time, versus throughput) may specify the order and number of comparisons. For example, a real-time flow may prefer to compare only rounded RTT and rounded RtProp, ignoring all other characteristics. A throughput flow may decide to compare only the Mathis coefficient and the capacity factor.

[0232] The above approaches involve explicit sorting dimensions. There are also implicit sorting dimensions that often emerge as a byproduct of typical application traffic patterns.

[0233] Most application traffic tends to be bursty or adaptive, so it does not need to use all of the WAN connection. This means that a connection that is currently at the bottom of the sort order may have no traffic on it. If the connection does not do this, its explicit sort dimension will not be updated, so the connection's position tends to remain near the bottom of the sort order.

[0234] In some embodiments, the push schedule can also be configured to implicitly generate a sort order along one or more implicit sorting dimensions within a time period. The implicit sorting dimensions force connections that have experienced periodic bad events (e.g., latency spikes caused by congestion) to bubble to the bottom of the list. Connections that behave consistently naturally bubble up and remain near the top.

[0235] This is generally considered a good property of the algorithm, but may be modified in the future due to external factors. For example, if a flow has specific requirements that cannot be satisfied by connections currently near the top of the sort order, it may make sense to generate probing traffic on connections near the bottom to refresh their explicit sorting dimensions.

[0236] Figure 4A An embodiment in which a selector is generated using predefined flow classes and current connection status is depicted at 400A.

[0237] In this example, there are 3 connections each having a monitored set of characteristics, and there are 3 predefined flow classes, where each predefined flow class defines matching criteria and a preference for ordering connections that meet the matching criteria.

[0238] In this embodiment, the selector consists of a set of flow classes whose matching criteria match at least one connection with available capacity. In this example, flow class 1 and flow class 2 both have matching criteria that match connection 1 and connection 2, while flow class 3 does not match a connection and is not included in the selector.

[0239] Although both connection 1 and connection 2 match, connection 2 is excluded from the selector because it has no available congestion window and cannot currently send packets. When building the selector, flow classes can be included as long as a matching connection with an available congestion window can be found, so an embodiment may only check connection 1 to determine that flow class 1 and flow class 2 can be included in the selector without checking connection 2 for a match.

[0240] Step 2b (shown in Figure 1E 1105 in step 2a) uses the selector in step 2a to find a packet on the retransmit queue or the send queue that is eligible to be (re)sent on an active connection with capacity. If no such packet is found, the loop is exited. This may occur even if some active connections have capacity and there are packets in the input queue or the retransmit queue.

[0241] Figure 4B Shown at 400B is an embodiment with a priority input queue, which is an ordered stream set, wherein the stream on the right side of the queue should send a packet before the stream on the left side. Each stream has a stream ID, and each stream belongs to one of predefined stream categories. Each stream also contains an ordered packet queue so that if a stream is selected, the packet at the head of the queue is removed.

[0242] In this instance, Figure 4AThe selector in is used to evaluate the flows in the input queue from right to left, and it selects the first flow that has the flow category listed in the selector. In this case, flows with IDs 5 and 2 are evaluated and rejected because they both have flow category 3, which is not one of the flow categories in the selector. Then, flow 7 is evaluated and is a match because it belongs to flow category 1, which is one of the flow categories in the selector. The packet is then extracted from the head of the packet queue for flow 7 and stamped with the flow metadata.

[0243] Once the package has been selected, step 2c (shown in Figure 1E 1107 in step 2) to construct an ordering of active connections. The selected ordering will be based on the rules provided in the definition of the flow class (if the embodiment supports the rules), or based on a generally acceptable ordering performed in other ways. Some embodiments may use the same ordering for all flows that match the flow class, and some embodiments may make the ordering flow-dependent (e.g., making all packets in a flow sticky to a given connection without requiring all flows to have stickiness to the same connection). The ordering may be constructed as needed at each iteration of the loop in step 2, or some embodiments may choose to cache the ordering and clear the cache when the criteria used to construct the ordering have changed (i.e., when the metadata is updated).

[0244] One embodiment constructs the ordering regardless of whether the connection currently has capacity, since capacity may be reduced to zero due to scheduling packets on a connection, or increased above the current value of zero if scheduling packets on different connections changes the metadata mix sufficiently to activate a set of connections in a previously unused priority level. Constructing the ordering in this way allows it to be cached and reused if the only items that change are the effective CWND and the byte count in flight for the connection.

[0245] Flows traversing system 100B tend to perform better when packets of the flow traverse the same WAN connection whenever possible. Therefore, embodiments should construct rankings using stable ranking criteria so that the ranking created for the same flow is consistent unless the monitored characteristics of one or more of the connections change significantly. Therefore, operational characteristics such as RTT and RtProp and bandwidth are poor choices because they may fluctuate over short periods of time, making the ranking order unstable.

[0246] Using statistical filtering methods, the "noisy" raw operating characteristics can be converted to monitored characteristics. The statistical filtering methods can range from traditional measures such as mean, variance and ANOVA to the field of (quantified) rank order statistics (rank order filtering). For example, the monitored characteristic can be a range of raw operating characteristics, which can be defined by specific values ​​(e.g., 1-100 milliseconds) or orders of magnitude, etc. The following paragraphs are specific examples, including the examples shown in Table 2.

[0247] Some embodiments may use machine learning techniques to convert operational characteristics into monitored characteristics. For example, a trained model may be developed to predict impending connection failures, and from there construct a ranking that sorts matching connections to the end of a list. Another example model may be used to detect the connection type (e.g., wired, cellular, WiFi, satellite, BoD, etc.), and use the connection type to form the basis for a new monitored statistic (e.g., susceptibility to buffer bloat) that cannot be easily measured in real time.

[0248] One embodiment uses the following ordering for all packets (later criteria are used only if all previous criteria are equal):

[0249] "Bucketed" loss probabilities. Connections can be grouped into buckets defined by ranges of loss probabilities based on packet count or based on size (e.g., in bytes). Connections in buckets with smaller ranges of loss probabilities appear first in the ordering, and connections in buckets with larger ranges of loss probabilities appear later in the ordering.

[0250] Connection priority groups (such as provided by an administrator or end user). Connections with higher priorities are ranked higher (level 1 > level 2 > level 3, etc.).

[0251] "Bucketed" RTT. Latency is rounded to the nearest multiple of 50 milliseconds. Bucketing the latency values ​​allows normal variations in latency to be hidden in the ranking, while still allowing significant variations in latency to be reflected in the ranking.

[0252] As described above, for example, the "capacity factor" is a total value measured in "bytes per second per millisecond" and is calculated by rounding the bottleneck bandwidth estimate to the nearest order of magnitude or integer multiple of the nearest order of magnitude (e.g., 1 is rounded to 1, 10 and 11 are rounded to 10, 15 is rounded to 20, 86000 is rounded to 90000), and then dividing by the RtProp bucketed to the nearest 50 milliseconds. Connections that achieve higher bandwidth per millisecond RtProp are considered better than connections with lower bandwidth per millisecond RtProp.

[0253] Determining the capacity factor from the raw BtlBW and RtProp values ​​is highly variable, but unless large changes are observed, the values ​​are stabilized over a period of time by rounding BtlBW to the nearest order of magnitude or the nearest integer multiple of the order of magnitude and binning RtProp. Table 2 shows several examples of binning, rounding, and capacity factor calculations. Note that in this example, the capacity factor is rounded to the nearest integer and a binned RtProp value of 0 is treated as 1 for the purpose of the capacity factor calculation.

[0254] "Bucketed" RtProp. Lower values ​​of RtProp that are bucketed to the nearest 50 milliseconds are preferred over higher values.

[0255] "Rounded" BtlBW. Higher values ​​of BtlBW rounded to the nearest order of magnitude or integer multiple of the nearest order of magnitude are preferred over lower values.

[0256] If all the aforementioned criteria compare equal, the connection ID (assigned when enumerating the connections) is used to deterministically break the tie.Other embodiments may use more complex tie-breaking mechanisms, such as hashing the characteristics of the flow into an integer modulo the number of tied connections.

[0257] Table 2: Examples of bucketing, order of magnitude rounding, and capacity factors

[0258]

[0259] Embodiments that differentiate between stream types may, for example, modify the above-described ordering to prefer high capacity factor connections over lower latency connections for file transfer streams, and to prefer lower latency connections over high bandwidth connections for video streams, based on the transmission requirements of the stream type.

[0260] Embodiments that perform strict stream fixation on a connection (or set of connections) may choose to allow only complete matches on the connection ID (possibly excluding other connections from the ordering entirely). Such fixed rules are typically provided by an administrator or end user, and are intended to override all ordering decisions determined by the monitored characteristics. Therefore, even if the monitored characteristics indicate that the connection is unavailable or has expired, the stream will continue to use the same ordering, and will therefore no longer be connected through the multipath system 100B.

[0261] Some embodiments may create monitored characteristics that are composites of other monitored characteristics or operational characteristics, using these composites to determine ranking. For example, a composite reliability metric may be created that combines bucketed loss probability, (running) latency (RtProp) variance, and (running) throughput into a weighted metric representing the overall reliability of the connection. Connections that rank higher on this metric will be reserved for flows that require high reliability (which may not imply high speed).

[0262] In general, if a generated selector allows a packet to be selected, then the packet should generate an ordering that allows the packet to be scheduled on some connection used to generate the selector.

[0263] Figure 4C An example of how a packet's flow category ("flow category") may be used to construct a connection ordering is shown at 400C. In the depicted embodiment, the flow category in the packet's metadata is used to look up the flow category definition. The matching criteria for the flow category is used to select which connection should be included in the constructed ordering (alternatively stated, the flow category defines a set of transmission requirements that may be associated with a particular flow category), in which case both connection 1 and connection 2 match, while connection 3 does not match. Note that connection 2 is included even though it does not have an available congestion window (i.e., it is not currently eligible to send packets). This facilitates cache ordering and allows the ordering to be reused even if the state of the available congestion window should change (e.g., if the scheduler initially prevents data from being sent on the connection because it has a lower priority, but then decides to re-enable the connection because the data combination on the currently enabled connection does not meet the minimum system throughput).

[0264] Once the set of connections has been selected, the matched connections are sorted into a sorted list using the sorting criteria for the flow class (e.g., a subset of the transmission requirements associated with the flow class). In this example, the first criterion is the capacity factor. Both connections have the same capacity factor. The second criterion is RTT, and both connections have the same RTT. The third criterion is BtlBW (bottleneck bandwidth), and connection 2 has a higher value and therefore appears before connection 1 in the sorting.

[0265] The set of sorting criteria (e.g., transmission requirements) within a flow class allows packets to be scheduled onto the best available connection for the type of flow they belong to. For example, packets belonging to a "video encoder" flow class may prefer connections with the lowest latency, while packets belonging to a "file transfer" flow may prefer high throughput connections even if the latency is high or highly variable.

[0266] Once the order of connections has been determined, step 2d (shown in Figure 1EThe block 1110 in FIG. 11 uses the sorting to assign packets to connections with available bandwidth. The packets are placed on the send queue of the connection.

[0267] Figure 4D At 400D, it is shown Figure 4C The order in which a connection is selected for a given packet. In this example, the first connection in the order is connection 2, but it does not have a congestion window available, so it is not currently eligible to send packets. Then, connection 1 is checked, and it is able to send packets, so it is selected, and the packet is queued to be sent to connection 1's 3-stage pipeline.

[0268] Then, as part of step 2e (shown in Figure 1E 11), the connection metadata (the monitored characteristic) of the bytes in flight is adjusted. In some embodiments, this includes distinguishing actual throughput bytes from retransmitted / false / probe bytes when tracking CWND consumption for a connection. Such adjustments may affect the actual throughput and / or potential actual throughput of the priority group, which may cause connections in the next priority level to be activated in the next iteration of the loop in step 2. Various methods for converting from volume units (number of CWND bytes) to rate (actual throughput) are then described for Figures 8A-8C and discussed in the description of 9A-9C.

[0269] Once step 3 has been reached, the available CWND for each connection has been filled, or there are no more packets eligible for transmission. The packets queued to the per-connection output queue in step 2 are then sent to the appropriate instance of the next stage of the pipeline.

[0270] In some embodiments, when there are multiple flows with the same priority, but belonging to flow classes with conflicting requirements, an explicit objective function may be used to determine the desired tradeoff.

[0271] Described in Figures 4A-4D The embodiment in shows an objective function that implements fairness among flows at the same priority. The selector serves each flow that matches its flow class in a round-robin manner.

[0272] The fair implementation in this example uses the Deficit Round Robin scheduling algorithm described in RFC 8290. A brief overview of this algorithm is that it utilizes a "byte credit" scheme, allocating a quantum of byte credits to each queue in each of its iterations over the queues.

[0273] The number of bytes that can be retrieved from each queue for transmission in each round is nominally capped by the number of credits available to it. Fairness naturally emerges when each queue has an equal quantum of byte credits.

[0274] In some embodiments, the objective function may allow an administrator or end user to configure intentional unfairness. For example, weights may be configured on matching flows, which may ultimately result in unequal byte credits given to queues at each iteration.

[0275] For example, there might be 5 different weight levels:

[0276] Superlatives (5)

[0277] Advanced (4)

[0278] Intermediate (3)

[0279] Low level (2)

[0280] Lowest level (1)

[0281] The value assigned to each weight will represent the ratio between its byte integration quanta. For example, if the quantum corresponding to "medium" is 30KB, then the stream marked as "low" will receive (2 / 3)*30KB=20KB of quantum in each round. The stream marked as the highest level will receive (5 / 3)*30KB=50KB.

[0282] Note that the absolute numbers in this example are not important. If the mid-level quantum was 5KB instead of 30KB, the weights would still appropriately adjust the relative quanta of the other priority levels, and the end result would be the same in terms of overall fairness.

[0283] Other embodiments may allow the weights to be arbitrary integers, rather than the fixed values ​​of 5 in the previous example. This would allow an administrator or end user to configure even higher unfairness between flows in the objective function if desired.

[0284] Note also that, like all Quality of Service (QoS), these weight settings are only important when there is contention for the available WAN capacity. If there is no contention, it means that the data in all queues fits exactly into the available capacity, so the ratio between their usage is the natural ratio for applications generating different data volumes.

[0285] Other embodiments may have completely different objective functions that do not consider fairness at all. For example:

[0286] Ensure the maximum number of flows can be accommodated with available WAN capacity;

[0287] Maximize the total throughput achieved by the served flows; or

[0288] Minimize changes to current bandwidth allocations on BoD connections.

[0289] In some embodiments, the way a WAN connection is used can change its characteristics, making it ineligible to serve certain classes of flows. For example, some connection types can achieve very high throughput, but at the expense of a higher RTT.

[0290] Embodiments whose objective function seeks to maximize throughput will choose to serve flows with high throughput requirements, which will cause the WAN connection RTT to increase and make it ineligible to serve flow classes with low latency requirements.

[0291] Conversely, an embodiment whose objective function seeks to serve the maximum number of flows may choose the opposite tradeoff so that the WAN connection can continue to serve the low latency flow class.

[0292] In some embodiments, different scheduler connection sorting orders are maintained on a per-flow basis rather than globally for all flows.

[0293] This allows flow-specific requirements to influence the sorting order so that packets for a flow are distributed to connections in a way that meets its needs.

[0294] For example, a VoIP flow that prefers a maximum latency tolerance of 150 milliseconds would require a sorting criterion that places connections with RTTs exceeding 150 milliseconds at the bottom of the list. Conversely, a TCP flow that does not have a specific latency requirement would instead need to prioritize connection throughput in its sorting criterion, either completely ignoring RTT or using it only as one of many secondary criteria.

[0295] In some embodiments, these flow requirements and preferences are configured in the form of rules consisting of matching criteria and corresponding actions.

[0296] For example, the matching criteria may take the form of an IP 3-tuple (protocol, source IP, and destination IP) or an IP 5-tuple (protocol, source IP, source port, destination IP, and destination port). The behavior may take the form of an explicit preference for latency or throughput, each with a target preference value.

[0297] Other examples of heuristic-based matching criteria may include:

[0298] οDSCP tag

[0299] o Any of the other header fields available in the IP (v4 or v6) header

[0300] o Any field in the other header fields in the TCP or UDP header

[0301] o The cumulative volume of data transferred - for example, TCP flows that exceed a volume threshold can be matched as "bulk data transfers", while TCP flows that do not exceed a volume threshold can be matched as interactive SSH / telnet sessions

[0302] Cumulative duration of the flow

[0303] οRegular expression matching against packet payload

[0304] ο Plaintext parameters extracted from the TLS handshake (e.g., SNI, CN, SubjectAltName)

[0305] Examples of other behaviors that may affect the scheduler sort order include:

[0306] o Specific preferences for connection order (e.g., a user may prefer to order expensive connections to the bottom of the list of low priority flows)

[0307] οJitter tolerance

[0308] οMaximum delay tolerance

[0309] o The desired amount of redundancy (e.g., FEC)

[0310] o Whether retransmission of lost packets is desired, and if so, whether there is a maximum time limit after which retransmission attempts should cease

[0311] οWhether explicit packet pacing is expected

[0312] i. For example, a real-time video application that transmits video at 30 frames per second may transmit its video frames exactly 33.3 milliseconds apart and would not want the multipath system 100B to change pace.

[0313] ii. In contrast, bursty applications may benefit from the scheduler explicitly pacing their bursts at the total WAN rate.

[0314] o Expected throughput (e.g., minimum, target, maximum)

[0315] Rules (both matching criteria and behaviors) can be automatically generated using machine learning techniques that analyze traffic patterns.

[0316] Inputs (features) to ML methods can include:

[0317] i. Cumulative volume of data transferred

[0318] ii. Packet size distribution (histogram)

[0319] iii. Inter-packet spacing (both pacing and jitter)

[0320] iv. Frequency and grouping of packets

[0321] v. Packet header fields (Ethernet, IP, TCP, UDP, etc.)

[0322] vi. Contents of the packet (payload)

[0323] vii. The intended destination of the packet, not just the IP address (e.g., SNI, CN, and SubjectAltName fields in TLS handshakes and exchanged certificates)

[0324] viii. Cumulative duration of the flow

[0325] ix. Time of day

[0326] x. The number of concurrent flows to the same destination or from the same source

[0327] xi. Current or historical predictions (labels) for any concurrent flows to the same destination or from the same source (e.g., some applications open multiple flows, one for control plane and another for data plane, and knowing that there are existing control plane flows can help predict data plane flows)

[0328] The predictions (labels) of the ML method will include any of the aforementioned behaviors determined based on the behavior of the training corpus.

[0329] In some embodiments, rules can have the concept of subordinate rules that apply additional matching criteria and more specific behavior while still being governed by their parent rule. For example:

[0330] o A VPN flow identified by an IP 5-tuple can have a rule defined with a match criteria and an action. This will be considered a "parent" rule.

[0331] oVPN will carry many internal flows from applications protected by VPN therein. Typically, these internal flows will be completely hidden (encrypted by VPN) from the matching criteria.

[0332] o However, some VPNs may choose to expose a limited set of information about internal flows by setting a DSCP tag on the encrypted packets that corresponds to the preference of the inner (plaintext) packets.

[0333] You can create subordinate rules that match on DSCP tags, but the resulting available actions will still be governed by the parent rule. For example:

[0334] a. Typically, VPNs require packets to be delivered in order (the packets can tolerate mis-ordering, but only within a small window).

[0335] b. The actions configured on subordinate rules that match DSCP tags must not cause the parent rule to deliver packets significantly out of order.

[0336] For example, a subordinate rule requesting a maximum latency limit of 150 milliseconds and a second subordinate rule requesting maximum throughput may violate the ordering requirements of the parent rule because interleaved packets from these two subordinate rules may be assigned to connections with completely different latencies.

[0337] Figure 5A Two independent flows are shown, where the rules and behaviors on each flow are completely independent. In this example, the SIP flow requires low latency, so the behavior causes the scheduler 104B to transmit its packets only on connection 1. Conversely, the FTP flow requires high throughput, so the scheduler 104B transmits its packets only on connection 2.

[0338] As the packets of these two flows are transmitted over the air and arrive at the sequencer 118, the packets are re-sequenced and transmitted independently to the final destination. In this example, the sequencer 118 is transmitting the packets belonging to the SIP flow that have arrived and are in the correct order, regardless of the state of the FTP flow whose packets are still transmitted over the air.

[0339] Figure 5B The concept of dependent flows is shown. In this example, there is the same SIP flow and FTP flow, but before the scheduler 104B sees it, it is encrypted and encapsulated by the VPN client.

[0340] From the VPN client, packets from both flows have the same IP 5-tuple, so scheduler 104B and sequencer 118 treat this as the parent flow and are bound by its requirements.

[0341] The presence of dependent flows (SIP and FTP) is communicated using DSCP tags, and they control how scheduler 104B allocates packets to connections 1 and 2 for transmission. Figure 5A The same as in.

[0342] However, the parent flow constraints require that packets be transmitted from sequencer 118 in the same order as they arrived at scheduler 104B. Thus, in this example, even though packets belonging to the dependent SIP flow have arrived at sequencer 118, they cannot be transmitted until packets belonging to the dependent FTP flow arrive and are transmitted first.

[0343] Fig. 6A At 600A is a sample embodiment of a rule management page. The configured rules are displayed in a table, where each row summarizes the matching criteria and behavior. The order in which the rules are listed is the order in which packets match the rules. Figure 6B The workflow for changing the order of rules is shown at 600B.

[0344] Figure 6C At 600C is shown how each row in the table can be expanded to show details of the matching criteria and actions, as well as the active flows currently passing through the system that match this rule.

[0345] Each rule in the table includes Edit and Delete icons.

[0346] Fig.6D The dialog window that appears when clicking the edit button is shown at 600D. The matching criteria and behavior of the rule can be changed in this dialog box.

[0347] There is a separate Add Rule link on the Manage Rules main screen, and Fig. 6E The modal dialog box that appears when the link is clicked is shown at 600E.

[0348] In some embodiments, rules will be configured on the transmitter side and will be automatically pushed to the receiver. By default, rules will be symmetric and bidirectional. This means that when adding a rule and configuring its behavior, the matching pair rule will be added transparently, but the source and destination matching criteria will be swapped. Through the advanced configuration screens, it will be possible to add asymmetric rules (different behavior in either direction for the same flow).

[0349] In one embodiment, the scheduler probes various network characteristics of a plurality of connections (to determine measured characteristics of each connection) and uses the measured characteristics in conjunction with the transmission requirements to make decisions regarding packet transmissions.

[0350] For example, one such decision is determining how many packets to transmit on a particular connection and when to transmit those packets.

[0351] Some embodiments are volume based. Such embodiments operate by determining a limit on the volume of data (e.g., in bytes or packets) that can be transmitted on a particular connection. Once enough packets have been transmitted for the volume, transmission stops until some feedback is received about those packets. If feedback is received before the full volume limit has been transmitted, transmission can continue without stopping as long as the volume of data sent for which feedback has not yet been received does not exceed the determined volume limit.

[0352] In one embodiment, the data volume is selected as the congestion window (CWND) of the connection. In short, the CWND is the maximum volume of data that can be sent on a connection without causing significant congestion on the connection before any feedback is received. There are multiple methods for estimating the CWND. It is assumed that the scheduler has access to one such method and is able to receive a CWND estimate through it.

[0353] In some embodiments, the scheduler needs to determine the transmission rate of data (e.g., in bytes per second). The rate can then be used to make other decisions about packet transmission.

[0354] For example, in one embodiment, if the data rate transmitted on the current active network connection is lower than the transmission rate of data received from the source application, the scheduler may decide to activate more connections to increase the total transmission rate to accommodate the application rate.

[0355] In another example, an embodiment may need to meet quality of service (QoS) requirements, which may be expressed as a guaranteed transmission rate. If the total rate drops below the required level, the scheduler may decide to activate more connections to increase the total transmission rate to meet the QoS requirements.

[0356] In yet another example, embodiments may require measuring actual throughput (ie, the rate at which application data successfully traverses the network) for purposes of ensuring application performance levels, optimizing performance, reporting, and the like.

[0357] In one embodiment, TCP performance can be optimized by pacing the transmitted packets at the total rate achieved by multiple connections. See, e.g., PCT Application No. PCT / CA2020 / 051090.

[0358] In embodiments where some functions require a rate, a volume-based scheduler requires a method for converting the volume of data transferred into a rate.

[0359] Simple methods of converting volume to rate by dividing the volume by the duration of the transmission usually produce inaccurate results. Such methods measure the transmission rate of the sender, not the actual rate at which data moves across the network. The actual rate at which data moves across the network is in fact determined by the rate of the slowest link of the multi-link network (often referred to as the network bottleneck).

[0360] The characteristics of the network bottleneck link are not available to the sender or receiver, especially given that multi-link networks can change the combination of used links at any time and in a manner transparent to the sender and receiver. However, congestion control methods are able to generate estimates based on the performance observed at the sender and receiver.

[0361] In some embodiments, the congestion control method can provide an estimate of the transmission rate of the bottleneck link. This is sometimes referred to as the bottleneck bandwidth BtlBw of the network.

[0362] In some embodiments, the round trip propagation time RtProp of the network may be estimated by sampling the time between when a packet is sent and when feedback about the packet is received.

[0363] The combination of BtlBw and RtProp can be used to determine a network characteristic known as the bandwidth-delay product BDP as follows:

[0364] BDP=BtlBw x RtProp

[0365] The most common unit of BDP is bytes. Henceforth, the following units are assumed for these quantities: (i) BtlBw, bytes per second; (ii) RtProp, seconds; and correspondingly (iii) BDP, bytes.

[0366] BDP represents the amount of data the network will have (in flight) if it operates under ideal conditions, assuming that data is always available for sending. The network will transmit data at its nominal rate BtlBw and provide feedback on each packet exactly after the RtProp time of sending the packet. Figure 7 Two examples at 700 are shown. Figure 7 The upper part of shows this aspect using an example where the BDP consists of 16 packets of equal size. In practice, the amount and size of packets can be different as long as the total size is equal to the value calculated using the BDP.

[0367] In some volume-based embodiments, the BDP may be used to estimate various data transfer rates, some examples of which were previously described.

[0368] For the volume of data in flight under consideration, if the volume is less than the BDP, assuming ideal conditions, only a fraction of the network nominal rate BtlBw can be achieved. This fraction can be calculated as the ratio of the volume to the BDP. The resulting achieved rate is determined as:

[0369] Achieved rate corresponding to volume in flight = Volume / BDP x BtlBw

[0370] In practice, networks rarely operate under ideal conditions. For example, packets may be temporarily buffered before being delivered to the receiving end, or feedback may be delayed at the receiving end or delayed by the network. In some cases, packets may be lost and never make it to the receiving end, thus not triggering any feedback. In other cases, the feedback itself may be lost. For example, the loss may be inferred when feedback for a newer packet is received or based on the expiration of a timer.

[0371] Waiting for delayed feedback (which may or may not arrive) to trigger the next transmission event can artificially limit the achievable rate on the network.

[0372] Fig. 8AAn example of a receiver aggregating feedback and sending an acknowledgment back to the sender for every 4 packets received is shown at diagram 800A.

[0373] Figure 8B Figure 800B continues to illustrate how, for example, aggregated ACKs can cause the total throughput to fall below the network capacity. The sender waits longer before starting to send the next set of packets. Overall, the sender ends up sending a smaller number of packets than the network capacity. Figure 7 The ideal situation is the same as in , but over a longer period of time, resulting in lower throughput.

[0374] In one embodiment, this artificial limitation is overcome by specifying an additional data transmission allowance, even when feedback for previously transmitted packets has not yet been received. This means that the amount of data in flight may exceed the BDP. However, in such cases, the network is not actually transmitting data at a higher rate, as the application of the rate equations achieved above may suggest. The additional allowance only allows the sender to maintain a constant rate equal to BtlBw for a longer period of time. Therefore, the estimated transmission rate of the current volume is capped at BtlBw.

[0375] Figure 8C Continuing at figure 800C, it is demonstrated how, for example, the sender can use such additional send allowance to continue sending packets before receiving an acknowledgment. The in-flight data volume of 19 packets exceeds the BDP of 16 packets. However, the actual rate matches the network capacity, and the same result is achieved by capping the calculated rate using the minimum of the in-flight volume and the BDP.

[0376] In some embodiments, a similar approach may be used to estimate the actual throughput that may be achievable on the network (referred to as potential actual throughput from here on).

[0377] A specific data volume can be split into an actual throughput portion, which contains newly transmitted application data, and another portion, which contains data that cannot be counted as newly transmitted application data, such as retransmissions of previously lost data, control data, probing data, etc.

[0378] If the complete data volume is larger than the BDP, packet and / or feedback delays are assumed as before. No additional space is assumed for additional actual throughput. The resulting (potential) actual throughput rate is estimated as follows:

[0379] Actual throughput rate = Actual throughput fraction / Full volume x BtlBw

[0380] For example, Fig. 9AAt 900A, a network with a BDP of 16 packets is shown. The volume put into flight is 19 packets in order to achieve a throughput equal to the network capacity. However, only 14 packets are actual throughput, resulting in an actual throughput rate of 1 / 2 of the network capacity. 7 / 8. If the data volume is less than the BDP, the difference is treated as extra space assumed to be available for additional real throughput.

[0381] Fig. 9B An example is shown at 900B where only 3 packets are in flight, resulting in an extra headroom of 13 packets that can be assumed as actual throughput when calculating the potential actual throughput rate. This extra headroom must be limited by the CWND determined by the congestion control method. At any time, the additional volume allowed by congestion control is the difference between the CWND and the full volume. If the actual throughput extra headroom exceeds this congestion control allowance, it is directly limited to the allowance.

[0382] Fig. 9C An example is shown at 900C where the CWND has been reduced to 11 packets, which would limit the additional space available for additional actual throughput to 8 packets.

[0383] The resulting potential actual throughput rate is estimated as follows:

[0384] Potential actual throughput rate = (actual throughput portion + limited actual throughput extra space) / BDP x BtlBw

[0385] Continuing with the above example, Fig. 9B and Fig. 9C shows how the potential practical throughput equation above leads to Fig. 9C The potential actual throughput rate for the scenario in is lower, which is expected given the CWND reduction enforced by the congestion control method.

[0386] Although the above method is described for actual throughput and potential actual throughput rates, the method can be easily modified to calculate rates or potential rates for any type of data that partially constitutes a complete volume.

[0387] The following is a numerical example that applies the previously described equations on a network having characteristics typical of those experienced on an LTE cellular network.

[0388] BtlBw=10Mbps

[0389] RtProp = 60 ms

[0390] BDP = 10Mbps*60ms = 75KB

[0391] CWND=60KB

[0392] Actual throughput portion = 25KB

[0393] Other parts = 10KB

[0394] Full size = 25KB + 10KB = 35KB

[0395] Actual throughput additional space = BDP-full volume = 75KB-35KB = 40KB

[0396] Congestion control allowance = CWND - complete size = 60KB - 35KB = 25KB

[0397] Actual throughput limited by extra space =

[0398] Minimum value (actual throughput extra space, congestion control allowance) =

[0399] Minimum value (40KB, 25KB) = 25KB

[0400] Potential actual throughput rate =

[0401] (Actual throughput portion + limited actual throughput extra space) / BDP x BtlBw =

[0402] (25KB+25KB) / 75KB x 10Mbps=50 / 75x 10Mbps=6.67Mbps

[0403] Systems communicating over IP networks typically implement a congestion control mechanism that attempts to achieve fair network utilization by all network nodes employing similar congestion control mechanisms.

[0404] Congestion control mechanisms typically operate by monitoring the traffic occurring on the network, possibly probing the network when necessary, and drawing conclusions about network characteristics based on the results. A model representing the network is created, and future communications are directed based on this model. For example, the model may track network characteristics such as current throughput and maximum throughput achieved, current and minimum packet round trip times, packet loss events, etc.

[0405] Based on the created model and the tracked information, the congestion control mechanism can determine when the network is performing poorly, indicating that the network may be experiencing congestion. The congestion control mechanism can then take corrective actions aimed at reducing or removing network congestion and restoring desired network performance.

[0406] In conventional congestion control embodiments (eg, TCP), packet loss rate is used as an indication of network congestion, and corrective actions (eg, reducing CWND) are taken in response to the losses.

[0407] In one embodiment, the latency of a packet, i.e., the time required for a packet to traverse the network from the sender to the receiver, is an indicator of network performance. Packets with high latency may be of no use to the receiver and are lost, making their transmission wasteful. For example, a real-time VoIP application requiring low conversational delay between participants would be of no use to packets carrying voice data later than the maximum acceptable conversational delay.

[0408] In such embodiments, the congestion control mechanism may monitor the latency of packets and attempt to control the latency so that it does not exceed acceptable levels.

[0409] In one embodiment, the lowest delay measured for a packet may form a baseline value, and future measurements may be compared thereto. If future measurements exceed a threshold that varies with the baseline value, the congestion control mechanism deems the network to be experiencing congestion.

[0410] In another embodiment, a similar approach uses the round trip time of a packet (ie, the time elapsed between sending a packet and receiving an acknowledgment for the packet) instead of latency.

[0411] In one embodiment, when the congestion control mechanism determines that the network is experiencing congestion based on increased packet delay, the congestion window (CWND) is reduced to limit the volume of data that can be sent on the network without receiving any feedback. The purpose is to reduce the volume of data that the network will or is currently buffering, which is a common cause of increased packet delay.

[0412] In one embodiment, the congestion window is reduced to the bandwidth delay product (BDP) of the network. In theory, the BDP represents the volume of data that the network should be able to deliver without forming a static queue of packets in its buffers, assuming that packets are transmitted by the sender at a constant pace that reflects the throughput (BtlBw) of the network. Therefore, it is expected that this will allow the network to at least partially recover and reduce packet delays back to acceptable levels.

[0413] For example, assume that the network has a BDP of 16 packets, and the congestion control mechanism determines a CWND of 24 packets to account for acknowledgment aggregation. Fig. 10AHow congestion from other network nodes can cause a sender to overwhelm the network is shown at 1000A. In this case, the sender does not reduce its congestion window and puts the full amount into flight. This fills the network buffers, and the extra packets that cannot be buffered are lost. Even after the congestion is cleared and normal draining of the network buffers continues, Fig. 10A It shows how static queues will form inside the network buffer, which increases the round trip time of all future packets and makes it easier to fill the network buffer.

[0414] Continuing with the example, Fig. 10B At 1000B, it is shown how the sender that reduces its congestion window to BDP avoids both packet loss and formation of static queues. For example, the sender can determine that congestion is occurring by detecting that the time period in which no feedback is received exceeds the baseline duration of the feedback measured in the past transmission. In another example, the sender can determine that congestion is occurring by receiving feedback indicating that the amount of previously sent packets that have exceeded the baseline loss experienced in the past transmission has not yet been delivered to the receiver. In response to this congestion determination, reducing the congestion window to match the BDP reduces the amount of the packet that will be placed in flight. This correspondingly reduces the probability of filling the network buffer that previously caused packet loss. In addition, only placing the BDP packet in flight allows the network to completely exhaust its buffer (assuming that the network characteristics have not changed) before the new packet from the sender arrives at its buffer again.

[0415] In another embodiment, the congestion window is reduced to be equal to the volume of packets currently in flight.

[0416] When the congestion control mechanism determines that acceptable performance levels have been restored, the network is considered to no longer be experiencing congestion.

[0417] In one embodiment, a decrease in packet delay below a threshold indicates that network performance has recovered.

[0418] In one embodiment, the threshold is determined as a function of a baseline delay value.

[0419] In another embodiment, transmitting a particular volume of data without incurring any losses indicates that network performance has been restored.

[0420] In one embodiment, the data volume is equal to the BDP.

[0421] In another embodiment, the data volume is equal to the original value of the congestion window before the congestion control mechanism reduces the congestion window in response to congestion.

[0422] For the most common application types, the push-based architecture improves system performance over non-BoD connections. Table 3 summarizes the improvements that have been experimentally observed.

[0423] Table 3: Application performance improvements resulting from the push-based scheduler

[0424]

[0425]

[0426] The push-based scheduling approach also makes priority routing behavior more intuitive. For example, in one embodiment, the decision whether to join with a lower priority connection depends only on the comparison of its BtlBw with the configured threshold.

[0427] For example, consider the following scenario:

[0428] wan0

[0429] Priority: Level 1

[0430] Target threshold: Not applicable

[0431] BtlBw: 10Mbps

[0432] wan1

[0433] οPriority: Level 2

[0434] οTarget threshold: 15Mbps

[0435] οBtlBw: 30Mbps

[0436] Using the pull-based scheduler, each time wan1's 50Hz timer wakes up and requests a packet, the scheduler provides it with a packet for transmission because it finds that the desired target threshold (15Mbps) is greater than what wan0 alone can supply (10Mbps).

[0437] However, the scheduler does this regardless of the rate at which packets arrive at the input. For example, if the input rate does not exceed 10 Mbps, it would be better to not join wan1, since the traffic can be handled entirely by wan0 alone.

[0438] Using a push-based scheduler, wan0 will be sorted to the top of the list, and even if the threshold indicates that wan1 should be active, packets will only be scheduled onto it after the wan0 CWND is fully consumed. As a result, there will be less unintended usage of lower priority connections, improving performance.

[0439] In summary, the scheduler of the multipath system can be implemented as pull-based (100A) or push-based (100B). Push-based is superior because the scheduler can make packet scheduling decisions at a self-determined pace, rather than being driven by the rate at which the target connection is ready to transmit. This self-determined pace allows it to take into account all factors that are important to the application generating the packet and / or the system user.

[0440] For example, packets generated by a TCP-based application that requires maximum throughput will prefer its packets to be scheduled onto connections that are ordered from best to worst Mathis coefficients.

[0441] Also considered is a more complex example using the same TCP-based application, but where the user configures a priority routing rule in the multipath system 100B to prefer connection 110B over all other connections 110N unless the actual throughput achievable on 110B drops below 5 Mbps. In a hypothetical scenario where 110B has high throughput, low latency, but high packet loss, 110B will have a worse Mathis coefficient than all other connections 110N.

[0442] The push based scheduler 100B will be able to make complex scheduling decisions to respect the user preference to use connection 110B first, but also calculate the actual throughput achievable on it by converting 110B's in-flight CWND to a bit rate and using that bit rate to estimate the actual throughput available to it. If this availability drops below the configured 5Mbps threshold, it will schedule packets onto the remaining connections 110N sorted from best to worst Mathis coefficient.

[0443] As described above, the measured connection characteristics may include: packet loss rate over 3 independent sliding windows (500 milliseconds, 3 seconds, 30 seconds), which is further summarized as the worst value over the three windows; current network round trip time (RTT); bandwidth and round trip propagation time (BBRv1) parameters: RtProp (network propagation delay); and BtlBw (throughput estimate). External instructions may include priority routing (PR) rules, etc., but may also include flow specific rules / requirements.

[0444] In some embodiments, the measured characteristics of BBRv1 differ from its design in the original BBRv1 IETF draft. For example, the size of the sliding window used to calculate BtlBw may vary in a static or dynamic manner to allow for embodiments running on systems that may be less consistent in their execution times. For similar reasons of execution consistency, the input parameters used to calculate RtProp may also be slightly modified.

[0445] Before being used to determine the sort order, some or all of the input parameters may be rounded / bucketed / quantified. The technical improvement of rounding and bucketing is to keep the sort order as "stable" as possible so that the sort order does not change constantly (for example, without rounding and bucketing, two similar values ​​may fluctuate over time, causing the order to change constantly, resulting in "jitter").

[0446] Factors influencing its decision include: keeping the sort order as "stable" as possible so that packets are not split unnecessarily across multiple WAN connections (splits increase the chance of spontaneous out-of-sequence and jitter events); and considering both measured characteristics of the connection and unmeasured external instructions.

[0447] Values ​​that may be rounded include the following:

[0448] Latency-related values ​​(RtProp, RTT) are rounded to the nearest multiple of 50 milliseconds, with a rounded floor limit of 1 millisecond.

[0449] • Throughput related values ​​(BtlBw) are rounded to the "nearest order of magnitude".

[0450] The packet loss rate (worst value in the 3 sliding windows) is rounded to the nearest order of magnitude. Note that the raw values ​​here are restricted to the range [0,100], so the output values ​​are restricted to the set (0,1,2,3,4,5,6,7,8,9,10,20,30,40,50,60,70,80,90,100).

[0451] Examples of rounding to the nearest order of magnitude are provided in the following table. The original value, its calculated order of magnitude, and then the resulting rounded value are shown.

[0452]

[0453]

[0454] In some embodiments, specific improvements based on improving the order of networking routing to control the use of network interfaces are described, wherein the throughput is predicted based on the Mathis Equation using the combined characteristics of round-trip time (RTT) and packet loss rate to establish an upper limit for TCP throughput under a given RTT, packet loss rate, and MSS. The Mathis coefficient can be used to modify the sorting order used to route networking packets.

[0455] The measured characteristics may include: packet loss rate over 3 independent sliding windows (500 ms, 3 s, 30 s), which is further summarized as the worst value over the three windows; current network round trip time (RTT); Bandwidth Round Trip Propagation Time (BBRv1) parameters: RtProp (network propagation delay); and BtlBw (throughput estimate). External instructions may include priority routing (PR) rules, etc., but may also include flow specific rules / requirements.

[0456] In some embodiments, the measured characteristics of BBRv1 differ from its design in the original BBRv1 IETF draft. For example, to account for embodiments that may be run on systems whose execution times are less consistent, the size of the sliding window used to calculate BtlBw may be increased to cover long durations (hiding variability caused by execution inconsistencies), but the window duration may also be immediately shortened if a network loss or latency increase event occurs, since those events indicate that BtlBw may have actually changed and any long history in the sliding window must be forgotten.

[0457] For similar reasons of inconsistent execution, the input parameters used to measure RtProp can be changed to include a portion that exceeds network propagation delays, thereby adding execution / processing time, as said time may be non-trivial and continue to contribute to actual delays when transmitting packets and waiting for responses. In some embodiments, this is referred to as "System RtProp" rather than "Network RtProp" which consists solely of network propagation delays. In these embodiments, "System RtProp" and "Network RtProp" can be used for different calculations, depending on the intended purpose. For example, when calculating CWND, "System RtProp" is appropriate so that the pipeline contains enough in-flight data to account for processing delays. However, when sorting network connections to determine packet routing, "Network RtProp" is more appropriate because the preferred routing decision does not depend on system execution delays.

[0458] Packet loss rate, latency and throughput are not completely independent variables. Depending on the application's congestion control behavior, all three combine to affect each other. This is because, for example, with TCP, higher packet loss rates lead to longer "silent" periods in the air (underutilized channel) while the sender waits for a DUP ACK or RTO to occur.

[0459] Therefore, some comprehensive features were identified and used in the final sorting method. Specifically:

[0460] Mathis coefficient: This composite characteristic takes into account RTT and packet loss rate to predict throughput, and is based on the Mathis relationship that gives an upper limit on TCP throughput for a given RTT, packet loss rate, and MSS. When comparing the characteristics of any two network connections to determine their rank order, the MSS can be considered a constant, so this relationship indicates that the expected throughput is proportional to the inverse of (RTT*sqrt(packet loss rate)). In this example, the relationship is used as a practical implementation to establish coefficients for directly controlling the operation of a network controller or router. For each connection, the system determines the Mathis coefficient as: maximum (rounded Rtt, 1 millisecond)*sqrt(rounded aggregated packet loss rate). Connections with smaller Mathis coefficient values ​​are considered better. The calculation is a function of two independently measured characteristics, so the relationship between the characteristics determines how the connections will compare to each other. The Mathis coefficient can be maintained and stored in a data object, such as a corresponding Mathis coefficient data value associated with each connection, and stored with other monitored characteristics in, for example, an array of data values.

[0461] For example, in one extreme case, two connections with 0% packet loss will always have the same Maxis coefficient - their actual RTT value is not important. In this scenario, the Maxis coefficient will not affect the sorting order - the sorting order will be determined by comparing the subsequently measured characteristics with the combined characteristics.

[0462] At the other extreme, if two connections have a packet loss rate of 100% (rounded aggregated packet loss rate = 1.0), only their rounded Rtt values ​​will determine how they compare to each other. In some embodiments, this is undesirable, so connections with a packet loss rate of 100% are filtered out from the available choices at an early stage.

[0463] In between the two extremes, connections with both low RTT and low packet loss will perform better than those with higher values. However, connections with a high percentage of loss can still compensate by having a low RTT and have the same Mathis coefficient as connections with a low percentage of loss. For example:

[0464] Connection A: 100 ms RTT, 0.01 (1%) loss rate => Mathis coefficient = 10

[0465] Connection B: 19 ms RTT, 0.25 (25%) loss rate => Mathis coefficient = 9.5

[0466] In this example, although connection B has a significantly higher loss rate (25% versus 1%), its lower RTT compensates and gives it a better Mathis coefficient than connection A. The explanation for this phenomenon is that although connection A has a lower loss rate percentage, connection B's lower RTT allows it to exchange packet acknowledgments (positive and negative) for lost packets and complete any needed packet retransmissions more quickly than connection A could do using its higher RTT.

[0467] Capacity Factor: This composite characteristic takes into account BtlBw and RtProp, and is determined as: rounded BtlBw / max(rounded RtProp, 1 ms). The units are "bytes per second per millisecond", which have no physical meaning, but provide a way to normalize the bandwidth a connection can achieve per unit of RtProp. In other words, a connection that can achieve more bandwidth per millisecond with its RtProp is considered more efficient than a connection that achieves less bandwidth. In this context, efficient transport means a connection that achieves more bandwidth per millisecond of propagation delay (RtProp).

[0468] Once all raw inputs have been processed and the composite features calculated as described above, the actual sorting mechanism can be performed sequentially as follows:

[0469] Moving from one comparison to the next will only occur if the first few rows all compare equal. If the comparisons are not equal, the remaining comparisons are irrelevant and are skipped. In some embodiments, the selection of comparisons is made on the basis of flow classification (e.g., VPN, relative to VoIP, relative to ping packets, relative to FEC packets), and the list of comparison criteria can be different for each different flow (e.g., each flow can have a different predefined most important characteristic, second most important characteristic, third most important characteristic, etc.). Each of these characteristics can be iterated in sequence starting with the most important until the connection itself can be sorted and put into a sequential sequence. In some embodiments, depending on the flow classification, the first pre-step is to select a subset of connections that are suitable for sending data packet types in the flow classification, and then assign an order to them before performing packet distribution.

[0470] 1. A connection that is completely down (aggregated loss rate of 100%) is considered worse than a connection with an aggregate loss rate of < 100%.

[0471] 2.PR priority (the higher the priority, the better)

[0472] 3. Mathis coefficient (the lower the better)

[0473] 4. Rounded RTT (lower is better)

[0474] 5. Capacity factor (the higher the better)

[0475] 6. Rounded RtProp (lower is better)

[0476] 7. Rounded BtlBw (higher is better)

[0477] 8. Connection ID (lower the better)

[0478] Figure 1F A rendering of the sorting mechanism is shown in . In an example sorting mechanism, the real-time streams first include a comparison of RTT / RtProp, and then if equal, they will then compare reliability, and then if also equal, compare transmission rate, etc. Depending on the stream type, there may be a different sorting order. For example, a stream that requires throughput may compare transmission rate first. If equal, then compare reliability. If also equal, the stream may decide not to do other evaluations, and then simply choose arbitrarily.

[0479] In some embodiments, the flow requirements (e.g., real-time, versus throughput) may specify the order and number of comparisons. For example, a real-time flow may prefer to compare only rounded RTT and rounded RtProp, ignoring all other characteristics. A throughput flow may decide to compare only the Mathis coefficient and the capacity factor.

[0480] Most application traffic tends to be bursty or adaptive, so it does not need to use all of the WAN connection. This means that a connection that is currently at the bottom of the sort order may have no traffic on it. If the connection does not do this, its explicit sort dimension will not be updated, so the connection's position tends to remain near the bottom of the sort order.

[0481] This is generally considered a good property of the algorithm, but may be modified by external factors. For example, if a flow has specific requirements that cannot be satisfied by connections currently near the top of the sort order, it may make sense to generate probing traffic on connections near the bottom to refresh their explicit sorting dimensions.

[0482] For example, as explained in further detail below, in some embodiments, these connections may be scored 0 in the layer grouping, so that these connections may be used to send packets in any case and scheduling may continue. For example, early retransmissions are sent for outdated and unreliable connections to refresh the connection and check whether the connection has become reliable.

[0483] Fig.11A hybrid connection overall system 1100 is shown, which is configured to utilize an improved scheduling method in the transmission portion of the system and a buffering system at the receiving end and to sequence packets. As shown, in one embodiment, the components in the system are hardware components that are configured to interoperate with each other. In another embodiment, the hardware components are not discrete components, and more than one component can be implemented on a specific hardware component (e.g., a computer chip that performs the functions of two or more of the components).

[0484] In some embodiments, the hardware components reside on the same platform (e.g., the same printed circuit board), and the system 1100 is a single device that can be transported, connected to a data center / field-carryable device (e.g., a ruggedized mobile transmitter), etc. In another embodiment, the components are dispersed and may not all be located in close proximity, but communicate electronically via telecommunications (e.g., processing and control are performed by components residing in a distributed resource environment (e.g., the cloud) rather than locally).

[0485] Providing hybrid connectivity is particularly desirable in mobile scenarios where signal quality, network availability, network quality, etc. are suboptimal (e.g., professional news gathering / video creation may occur in locations without robust network infrastructure). Thus, in some embodiments, the device described herein is a ruggedized network controller device that is suitable for operation in a remote or mobile scenario, such as, for example, a single-person portable (e.g., worn in a backpack by a news technician), vehicle portable (e.g., pushed along a media cart), or vehicle portable (e.g., coupled to a media van, truck, boat, helicopter, or airplane).

[0486] Alternatively, the device may be located in a communication center or other type of centralized communication facility that coordinates connections with additional endpoints. For example, in this example, the device may be located, coupled to, or reside in a communication device such as a communication relay station (e.g., a satellite station, a relay transmitter, a broadcast translator, a translator, a repeater, a supplemental station), which may coordinate communications with various endpoints across multiple different channels, such as television stations, radio stations, data communication stations, personal computers, mobile devices, etc.

[0487] Multiple different data connections 1106 (e.g., "paths") are shown, representing one or more networks (or network channels), labeled as Connection 1, Connection 2, Connection N. Multiple data connections / paths may exist across a single network or multiple data connections that may use one or more networks.

[0488] These data connections may include various types of technologies, and each technology may face different communication environments, such as having different types of available communication capabilities, array architectures, polarizations, multipath propagation mechanisms, spectrum interference, frequency ranges, modulations, etc., and therefore, the connections may all have different levels of communication reliability.

[0489] The system 1100 may be configured to communicate with various endpoints 1102, 1110 or applications that need not possess any information about the multiple paths / connections 1106 used to request and receive data (e.g., the endpoints 1102, 1110 may operate independently of the paths or connections 1106). For example, received data may be reconstructed such that the original transmission may be regenerated from contributions from the different paths / connections 1106 (one example use scenario would be to regenerate video by way of a receiver configured to be plugged into a server rack at a data center facility, thereby integrating with existing broadcast infrastructure to provide improved networking capabilities).

[0490] The system 1100 receives input (data streams) from a source endpoint 1102 and schedules the delivery of improved packets across various connections 1106, and then sorts the packets at the other end 1108 of the system before transmission to the destination endpoint application 1110. By doing so, the system 1100 is configured to increase bandwidth to approach the sum of the maximum bandwidths of the various available paths. The system 1100 also provides improved reliability compared to using a single connection, which is an important consideration in time-limited, highly sensitive scenarios such as news gathering at a live event while the event is occurring. In these events, there may be high signal congestion (e.g., sporting events) or unreliability across one or more of the paths (e.g., news coverage after a natural disaster).

[0491] Multiple connections can be bound together to operate as a single connection, and different methods can be utilized to coordinate routing, so that a connection can be used multiple times to send the same packet, etc.

[0492] In various embodiments, both scheduler 1160 and sequencer 1162 may be provided by a cloud computing implementation, or may be provided at an endpoint (before an application at the endpoint consumes the data), or may be provided in various combinations thereof.

[0493] System 1100 may be tuned to optimize and or prioritize performance, best latency, best throughput, minimum jitter (variation in latency on a packet flow between two systems), connection cost, combination of connections for a particular flow, etc. (e.g., if system 1100 has information that a transmission (data flow) is of content type X, then system 1100 may be configured to only use data connections with similar latency, while content type Y may allow for a wider combination of data connections (or require greater net capacity that can only be achieved through a combination of data connections)). This tuning may be provided generally to the system or specifically to each flow (or to a set of flows based on location, owner of the origin or endpoint or combination thereof, time of transmission, set of available communication links, security required for transmission, etc.).

[0494] Although in some embodiments only one gateway may be required, the system 1100 can typically be bidirectional in which each gateway 1104, 1108 will typically have a scheduler 1160 and sequencer 1162 to handle TCP traffic (or UDP traffic, or a combination of TCP and UDP traffic, or any type of general Internet Protocol (IP) traffic).

[0495] The system 1100 can be used in a variety of scenarios, such as as a failover (e.g., for disaster recovery) or supplement to an existing Internet connection (e.g., a VoIP phone system or an enterprise network connection), thereby seamlessly adding an additional network (or path) to replace a downed primary Internet connection, or to supplement a saturated primary Internet connection by bonding with a higher-cost network. In the case of a failover, there may be a coordinated failure across many communication channels (e.g., a large-scale denial of service attack or a major solar / geomagnetic storm), and then the system 100 will need to use communication resources efficiently because the available communication resources are scarce.

[0496] Another purpose of system 1100 is to provide a way to maximize the use of high-cost (usually sunk cost), high-reliability data connections (such as satellite) by allowing traffic to be offloaded to other data connections with different characteristics.

[0497] In some embodiments, system 1100 is a network gateway configured to route data flows across multiple network connections.

[0498] Fig.11An overview of a system with two gateways 1104 and 1108 is provided, each containing a buffer manager 1150, an operation engine 1152, a connection controller 1154, a flow classification engine 1156 (responsible for flow identification and classification), a scheduler 1160, a sequencer 1162, and a network characteristic monitoring unit 1161, and linked by N data connections 1106, wherein each gateway is connected to a specific endpoint 1102, 1110. Reference letters A and B are used to distinguish components of each of the two gateways 1104 and 1108.

[0499] Each gateway 1104 and 1108 is configured to include multiple network interfaces for transmitting data over multiple network connections, and is a device (e.g., including configured hardware, software, or embedded firmware) that includes a processor configured to: monitor time-varying network transmission characteristics of the multiple network connections; parse at least one packet in a data packet stream to identify a data flow category of the data flow, wherein the data flow category defines or is otherwise associated with at least one network interface requirement of the data flow; and route packets in the data flow across the multiple network connections based on the data flow category and the time-varying network transmission characteristics.

[0500] The buffer manager 1150 is configured to set up buffers within the gateway suitable for more efficiently managing traffic (both individual flows and combinations of multiple simultaneous flows through the system). In some embodiments, the buffer manager 1150 is a discrete processor. In other embodiments, the buffer manager 1150 is a computing unit provided by way of a processor configured to perform activities such as buffer management 1150.

[0501] The operation engine 1152 is configured to apply one or more deterministic methods and / or logical operations based on received input data sets (e.g., feedback information, network congestion information, transmission characteristics) to inform the system of constraints to be applied to bound connections, the constraints being for each user / client, destination / server, connection (e.g., latency, throughput, cost, jitter, reliability), stream type / requirement (e.g., FTP, versus HTTP, versus streaming video). For example, the operation engine 1152 can be configured to restrict certain types of streams to a particular connection or set of data connections based on cost in one instance, but reliability and low latency may be more important for different users or stream types. For example, different conditions, triggers, methods may be utilized based on one or more elements of known information. For example, the operation engine 1152 may be provided on the same or different processor as the buffer manager 1150.

[0502] Operation engine 1152 may be configured to generate, apply, or otherwise manipulate or use one or more rule sets that determine logical operations by which to control routing over N data connections 1106 .

[0503] The flow classification engine 1156 is configured to evaluate each data stream for transmission received by the multi-path gateway 1104, and is configured to apply a flow classification method to determine the type of traffic being sent and its requirements (if not already known). In some embodiments, deep packet inspection technology is suitable for performing the determination. In another embodiment, the evaluation is based on a heuristic method or a data stream that has been marked or labeled when generated. In another embodiment, the evaluation is based on rules provided by the user / administrator of the system. In another embodiment, a combination of methods is used. The flow classification engine 1156 is configured to interoperate with one or more network interfaces and can be implemented using electronic circuits or processors.

[0504] The scheduler 1160 is configured to perform determinations, i.e., as to which packets should be sent over which connections 1106. The scheduler 1160 can be considered an improved QoS engine. In some embodiments, the scheduler 1160 is implemented using one or more processors or separate chips or configuration circuits (such as comparator circuits or FPGAs). The scheduler 1160 can include a series of logic gates that confirm to perform determinations.

[0505] While a typical QoS engine manages a single connection, the QoS engine (or in this case the scheduler 1160) can be configured to perform flow identification and classification, with the end result being that the QoS engine reorders packets before they are sent out on a connection.

[0506] In contrast, while the scheduler 1160 is configured to perform flow identification, classification, and packet reordering, the scheduler 1160 in some embodiments is further configured to perform the determination of which connection to send a packet to so that the transmission characteristics of the data flow are improved and / or the policy set by the user / administrator for the flow (or the policy stated in various rules) is met. For example, the scheduler 1160 can modify the network interface operating characteristics by transmitting a set of control signals to the network interface to turn the network interface on or off, or to indicate which interface should be used to route data. The control signal can be a set of instructions indicating specific characteristics of the desired routing (such as packet timing, network interface reservation for specific types of traffic, etc.).

[0507] For example, consider 2 connections with the following characteristics:

[0508] Connection 1: Round Trip Time (RTT), 1 millisecond; estimated bandwidth, 0.5 Mbps; and

[0509] Connection 2: RTT, 30 ms; estimated bandwidth, 10 Mbps.

[0510] Scheduler 1160 may attempt to reserve connection 1 exclusively for Domain Name System (DNS) traffic (small packets, low latency). In this example, there may be too much DNS traffic to reach the capacity of connection 1, so that scheduler 1160 may be configured to overflow traffic to connection 2, but scheduler 11160 may selectively do so based on other determinations or factors (e.g., if scheduler 1160 is configured to provide fairness determinations, scheduler 1160 may be configured to first overflow traffic from IP addresses that have sent a large amount of DNS traffic in the past X seconds).

[0511] For example, the scheduler 1160 may be configured to handle the determination based on a process or method operating in conjunction with one or more processors or similar implementations in hardware (e.g., an FPGA). The scheduler 160 may be configured to operate under the control of the operation engine 152 to deassemble the data stream into packets and then route the packets into buffers (managed by the buffer manager 150) that feed the packets to the data connections according to rules designed to optimize packet delivery while taking into account the characteristics of the data connections.

[0512] In some embodiments, the scheduler 1160 need not be configured to communicate packets in the correct order, but rather to communicate packets across different connections to meet or exceed desired QoS / QoE metrics (some of which may be defined by the network controller and others may be user / client defined). In situations where packets may be communicated out of order, the sequencer 162 and buffer manager may be utilized to reorder received packets.

[0513] A continuous burst of packets is transmitted across network interfaces, and a bandwidth estimate for a first network interface is generated based on the timestamp of when packets in the continuous burst of packets are received at a receiving node and the size of the packets. The estimate is then used to route packets in a data stream of continuous packets across a set of network connections based on the generated bandwidth of the first network interface. As described below, in some embodiments, the bandwidth estimate is generated based on the timestamps of packets in the burst that are not merged with the initial packet or the final packet in the burst, and a lower bandwidth value can be estimated, and a higher bandwidth value can be estimated (e.g., by packet replacement). The packets sent can be test packets, test packets that "piggyback" data packets, or mixed packets. In the case where data packets are used for "piggybacking", some embodiments include marking such data packets to increase redundancy (e.g., to enhance tolerance for lost packets, especially for packets used for bandwidth testing purposes).

[0514] In some embodiments, consecutive packets may be received in order, or within an acceptable order deviation, so that the sequencer 1162 can reorder the packets for consumption. In some embodiments, the sequencer 1162 is a physical hardware device that can be incorporated into a broadcast or networking infrastructure that receives signals and generates an output signal that reassembles the signals. For example, the physical hardware device can be a rack-mounted device that acts as a first stage of signal reception and reassembly.

[0515] The sequencer 1162 is configured to sort the received packets and transmit them to the application at the endpoint in an acceptable order so as to reduce unnecessary packet re-requests or error correction of other streams. In some embodiments, the order is consistent with the original order. In other embodiments, the order is within an acceptable order range so that the sequencer 1162 can still reassemble the data stream. For example, the sequencer 1162 may include a buffer or other mechanism for smoothing the received stream, and in some embodiments, the sequencer is configured to control the transmission of confirmations and the storage of packets based on the monitored transmission characteristics of multiple network connections and the uneven distribution in the data stream of received consecutive packets.

[0516] For example, a sequencer 1162 may be provided on a processor or implemented in hardware (eg, a field programmable gate array) under the control of the operation engine 1152 and configured to reassemble a data stream from received data packets extracted from a buffer.

[0517] The sequencer 1162 is configured on a per-flow basis to hide latency differences between multiple connections that would be unacceptable for each flow.

[0518] The operation engine 1152 may operate as an aggregator of information provided by other components (including 1154) and direct the sequencer 1162 through one or more control signals, ie, instruct the sequencer 1162 how it should operate on a given stream.

[0519] When a system configured for a protocol such as TCP receives packets, the system is typically configured to expect (but not require) packets to arrive in order. However, the system is configured to establish a time limit (usually some multiple of the round trip time or RTT) as to when it expects out-of-order packets to arrive. The system may also be configured to retransmit missed packets before the time limit based on heuristics (e.g., fast retransmissions triggered by three DUP ACKs).

[0520] In the event that packets arrive at the sequencer 1162 on connections with significantly different latencies, the sequencer 1162 (on a per-flow basis) can be configured to buffer packets until the age (latency) of the packets is approximately the same and then forward the packets to the destination. This would be done, for example, if the flow requires consistent latency and low jitter.

[0521] Sequencer 1162 is not necessarily required to provide reliable, strictly in-order delivery of packets, and in some embodiments it is configured to provide the necessary packets so that a system using the protocol (e.g., an application on top of TCP or UDP) does not prematurely determine that a packet has been lost by the network.

[0522] In some embodiments, sequencer 1162 is configured to monitor (based on data maintained by operations engine 1152) the latency variation (jitter) and packet loss rate of each data connection to predict which data connections are likely to delay packets beyond expectations for the flow based on the connection reliability (which means that endpoints 1102 and 1110 will consider that packet loss rate and invoke their error correction routines).

[0523] For example, for out-of-sequence situations, sequencer 1162 can utilize larger jitter buffers on connections that exhibit larger delay variations. For packet retransmission, sequencer 1162 can be configured to immediately request lost packets over the "best" (most reliable, lowest delay) connection.

[0524] In some embodiments, a network controller device (e.g., a router) is described that includes a processor coupled to a computer memory and a data storage device, the processor being configured to: receive one or more data sets indicating monitored network communication characteristics; maintain a hierarchical representation of multiple connections in a data structure stored on the data storage device, the multiple connections being separated into multiple groups, each group being established based at least on a minimum probability associated with successful data packet communication across one or more of the multiple connections residing within the group; and control multiple communications of a data packet so that the data packet is sent at least once across one or more of the multiple connections, so that the total of the multiple communications results in the transmission of the data packet satisfying a target probability threshold.

[0525] In some embodiments, receipt of a data packet defines reliability with respect to a target probability threshold, and in such aspects, multiple communications result in receipt of the data packet satisfying the target probability threshold.

[0526] On the other hand, multiple groups are organized into multiple corresponding layers, each layer representing the number of times a packet must be transmitted across any corresponding connection of the layer to reach a target probability threshold.

[0527] In another aspect, the multiple communications include retransmissions of connections across layers in the multiple layers, wherein the number of retransmissions is based on the number of times a packet would have to be transmitted across corresponding connections in the layers to reach a target probability threshold for the corresponding layer. In various embodiments, retransmissions may be performed on the same connection, but at different times (i.e., different bursts) to prevent loss of all retransmissions due to a temporary reduction in network reliability (i.e., burst loss due to buffer saturation of an intermediate host in the path).

[0528] On the other hand, retransmissions are performed across different connections of the layers.

[0529] On the other hand, multiple communications involve retransmissions across connections at different layers in multiple layers.

[0530] Due to the configured priority routing preferences, a connection may not participate in a transmission event, which may disable a lower priority connection because the higher priority connection has sufficient bandwidth to handle the priority data flow. In the absence of other traffic, this limits the system's knowledge of lower priority connections, and the scheduler 1160 or connection manager 1154 must account for outdated observations of connection characteristics.

[0531] This document describes embodiments that can expand upon the methods and systems described above.

[0532] In some embodiments, new scheduling methods (e.g., algorithms) can be introduced to solve the following three problems that arise when sorting connections into an absolute preference order by their characteristics (as described above): These algorithms are implemented on physical computing controllers and / or circuits.

[0533] A) Each flow with the same preference uses the connections in the same order. Flows may compete with each other unnecessarily on each connection, increasing the probability that the flow's traffic will be split between multiple connections. When possible, the preference is to keep a flow's traffic sticky to a single connection only to avoid unnecessary reassembly of its packets on the other side.

[0534] B) Less frequently used are connections that are further back in the absolute preference order, and are therefore more likely to have outdated properties.

[0535] C) Connections with very similar measured / combined characteristics may end up frequently swapping places in the absolute preference order even after rounding / bucketing, causing packets from the flow to be unnecessarily split between connections.

[0536] Additionally, technically challenging situations that are difficult to meet are real-time applications with real-time deadlines, such as video chats, voice calls, where packets may need to be received on time and with minimal jitter / loss due to the deadlines. In some embodiments, both latency and jitter requirements may be specified. For example, in order for voice and video to be smooth, the voice and video need to come in at a specific frame rate, otherwise the user may experience freezes and pauses.

[0537] In a non-limiting example, the types of streams considered may include those generated by streaming video applications. Using these applications, a sender captures and encodes video at a given frame rate, which is typically constant (e.g., 50 frames per second), but may be variable in some cases. The sender typically packages these encoded video frames into UDP packets that include a timestamp indicating the relative capture time of each video and delivers them to the receiver. For example, for a frame rate of 50fps, if the timestamp of the first video frame is t=0, the timestamp of the next video frame will be t=20 milliseconds (1 second divided by 50 frames per second).

[0538] In these streaming video applications, the sender then transmits the UDP packets to the receiver, which needs to receive the packets in a time-critical manner. This is because at the moment the receiver starts playing back the first video frame, all subsequent frames (consisting of one or more UDP packets) must be received within 20 milliseconds of the previous frame, otherwise the video played back to the viewer will be perceived as having artifacts (video stuttering or freezes), providing a poor viewing experience.

[0539] From a networking perspective, the delivery of UDP packets usually occurs on a best-effort basis, meaning that there is enough variability that the UDP packets that make up each video frame may not be consistently delivered to the receiver within 20 milliseconds after the previous video frame. For example, this may be due to queuing delays caused by contention for available capacity at one or more network links between the sender and the receiver. The receiver usually attempts to handle this problem by storing the video frames in a jitter buffer, which is sized to be large enough to handle the predicted changes in delivery time. For example, a jitter buffer storing two 50fps video frames will be able to hide network variability in the delivery time of the UDP packets by up to 40 milliseconds. However, a larger jitter buffer results in a longer glass-to-glass delay (the delay between the time a video frame is captured and the time it is played back). When setting the size of its jitter buffer, the receiver must make a trade-off decision to balance the perceived playback smoothness (making the video frames exactly 20 milliseconds apart) relative to the overall delay experienced by the viewer.

[0540] For this streaming video example, having multiple connections presents both opportunities and challenges in that there are multiple network paths between the sender and the receiver. The queuing delay and resulting variability in the delivery time of UDP packets on each connection will generally be uncorrelated, so the sender may use scheduling techniques to transmit each UDP packet on one or more connections, resulting in video frames always arriving at the receiver before their respective deadlines, allowing the receiver to use a smaller jitter buffer and reduce the overall latency experienced by the viewer.

[0541] Throughput-based applications, such as bulk data delivery and large file downloads including FTP, may pose different challenges because latency and jitter are less of a concern. Latency may not be an issue as long as the connection provides throughput. For example, for a file transfer, if there is an initial pause before the transfer begins, this may generally be acceptable.

[0542] Previous schedulers may have ordered the connections in an absolute order from best to worst, which presents a technical problem because it is not necessarily good for every application to use the connections in the same order, especially when the applications have competing requirements (e.g., throughput, versus real-time). The best connection for real-time purposes may be different from the best connection for throughput purposes. Using the connections in a different order allows the load to be split to reduce contention for the first connection, making it more likely to meet the real-time flow requirements. In some embodiments, it may be good to consider some connections to be "functionally equivalent", and the embodiments consider keeping the flow sticky to one of several "functionally equivalent" connections. For example, if there is stickiness, there may be less need to switch between connections or to reorder packets on the other (receiving) side or put the packets back in order.

[0543] Consider a real-time application with constraints and multiple WAN-connected hybrid routers whose packets need to be routed to meet the constraints set by the applications. There may be multiple applications with different requirements running simultaneously. What is needed is to achieve high performance while balancing the various issues. This technical challenge is further complicated when many applications are running and competing for bandwidth, where the different needs of all applications must be met. In many cases, some applications will be real-time, some will require throughput, and since WAN capacity is limited, it is technically challenging to balance the various tradeoffs (e.g., bandwidth, latency, jitter).

[0544] According to some embodiments, an improved approach is to group connections into layers, where each connection in a layer is "functionally equivalent" from a flow preference perspective.

[0545] Fig. 12A is a schematic diagram of a per-stream communication system according to some example embodiments.

[0546] In the embodiment shown in diagram 1200A, device 1202 can receive input packets 1204, and then in 1206, the input packets are sorted into streams and prioritized based on their requirements. Then, the available connections can be sorted into connection layers 1208 based on a combination of stream requirements, measured / comprehensive connection characteristics, and any predefined rules (e.g., connection priority rules specified by an administrator). As shown, connection layers 1208 include layers 1, 2, 3 to N. Connection layers 1208 may have variations, and in a variant embodiment, layers are further subdivided into sublayers. Other layers, different layers, alternative layers, more layers, fewer layers are all possible. Then, input packets 1204 can be transmitted on the determined connection 1210 corresponding to the appropriate layer and parameters.

[0547] Fig. 12B is a schematic diagram of a per-flow communication system 1200B according to some example embodiments. At layer creation 1212, a connection and requirements processor 1214 takes into account various requirements such as, but not limited to, connection monitoring, flow characteristics, packet characteristics, administrator configuration, location, etc., and multiple connections.

[0548] In some embodiments, the connection and requirement processor 1214 may group multiple connections into layers, such as layer 1, layer 2, layer 3, ..., layer N as shown in 1200B. Layer creation 1212 may occur at any point before flow ordering, as shown in Fig. 12A In some embodiments, the communication system 1200B can be included in the communication system 1200A, for example, the layer creation 1212 occurs in the flow ordering and packet scheduling 1206.

[0549] For example, during layer creation 1212, connections sorted into layers by connection and requirement processor 1214 may then transmit packet streams. Connection scheduler 1216 may schedule packet transmissions by iterating through layer levels and iterating through connections sorted into a single layer level. Packet transmission and scheduling methods will be further described herein.

[0550] Functionally, tiers may group "functionally equivalent" connections as described above, however, in these described embodiments, "functional equivalence" has been expanded. From a system perspective, tiers may be managed using database records or metadata that is updated or appended in a dynamic manner. For example, each connection may have a tier assignment that is tracked and updated by a corresponding database value stored in a database table (e.g., a routing table) stored in a coupled computer memory (e.g., a non-transitory computer readable medium) that is referenced or traversed to obtain tier information that is then used in the connection control methods described herein.

[0551] According to some embodiments, the following definition may describe "functional equivalence":

[0552] For flows with throughput preference, the layers may be based on the same (Mathis coefficient, capacity coefficient) composite characteristic pair calculated from the raw or rounded / bucketed / quantified measured characteristics. In some embodiments, the composite characteristics may also be rounded / bucketed. In some other embodiments, additional rounding / bucketing / filtering may be introduced to the measured characteristics and the composite characteristics to account for edge cases. For example, without this additional filtering, a connection with very high loss probability (98+%) and low latency may produce a better Mathis coefficient than a connection with lower loss rate but much higher latency.

[0553] One interpretation of grouping / layering connections based on unique (Maxis coefficient, capacity coefficient) pairs for flows with throughput preference is that these coefficients measure the throughput capacity of a connection and how efficiently it can use that throughput. Recall that the Maxis coefficients predict throughput based on the connection RTT and packet loss rate (a high packet loss rate means that the connection will have to retransmit packets that have been lost, and if the RTT is large, retransmissions take longer). Recall that the capacity coefficient is a measure of the transmission rate that a connection can achieve per millisecond of its RTT (in general, higher throughput and lower RTT are preferred). The following table describing the characteristics of several example connections is given:

[0554]

[0555] The "functional equivalence" grouping / layering of these 3 connections is that A and B belong to the same layer because they are both reliable, high performance connections. Neither has any loss (and therefore can use its throughput efficiently because no retransmissions will be needed), and both achieve similar throughput per millisecond of latency. They are "functionally equivalent" in that if a throughput-preferred flow is scheduled on either connection, the flow will be able to fully utilize the capacity of the connection. C intuitively belongs to a different "functionally equivalent" layer because of its high loss rate compared to B. The actual throughput achieved on C will be much lower than B because many packets will need to be retransmitted.

[0556] For flows with real-time preference, the tiers may be based on packet connections based on the probability that the connection will be able to deliver packets of the flow before the deadline for each packet.

[0557] By comparing the characteristics of the available connections to the real-time requirements of the streams, it is possible to observe how and why different connections can be classified as "functionally equivalent" for these types of streams. For example, if the target latency requirement for a real-time video chat application is 150 milliseconds, it means that the video sender expects each of its UDP packets to be delivered to the video receiver before the age of each packet reaches 150 milliseconds, where the age is measured from the time the sender initially transmits each packet. This is a common requirement for video chat applications, where the overall round-trip latency must be kept to 300 milliseconds or less for human participants to perceive the chat as interactive. The following table describing the characteristics of several example connections is given:

[0558] connect One-way delay Packet loss rate A 50 ms 0％ B 80 ms 0％ C 50 ms 10％ D 80 ms 5％ E 200 ms 0％ F 300 ms 0％

[0559] The "functionally equivalent" grouping / layering of these 6 connections is that A and B belong to the same layer because they both have a one-way latency of less than the application target of 150 ms, and because they experience 0% packet loss, every packet transmitted will arrive with its measured one-way latency. From the application's perspective, they are "functionally equivalent" because no matter which connection is used to transmit its UDP packet, it will arrive before the application's 150 ms deadline.

[0560] Similarly, it is clear that E and F belong to the same "functionally equivalent" layer, because although their one-way delays are different, both have application goals greater than 150 milliseconds. No matter which one is used to transmit the UDP packet, it is unlikely to arrive in time to meet the deadline.

[0561] Finally, it is less intuitively obvious where C and D belong, except that it is obvious that they do not necessarily belong to either of the other two functionally equivalent layers. Their one-way latency is less than 150 milliseconds, but because they experience significant packet loss rates, many transmitted packets will be lost, requiring them to be retransmitted, which may result in missed deadlines. The following is a more detailed and formal approach to grouping / layering connections by "functional equivalence" of real-time application flows.

[0562] The probability that a connection will be able to deliver a packet before its deadline is calculated based on a comprehensive property of packets transmitted on the connection named "Estimated Delivery Time" (EDT), which is defined as the expected value of the delivery time of a packet after retransmission due to lost packets.

[0563] Fig.13A is a diagram 1300A showing the calculation of EDT according to some example embodiments. In some embodiments, Fig.13AAs shown, this can be calculated as the sum of the probabilities of successful delivery over the retransmissions multiplied by the delivery time of the retransmissions. Assuming a uniform probability P of packet loss on each retransmission drop 13103, constant one-way propagation delay T prop 13101, Response timeout after retransmission of unconfirmed packet T rto 13102, this can be Fig.13A The determination shown is:

[0564]

[0565] Other variations are possible.

[0566] In some embodiments, the receiver may be able to quickly identify when a packet is lost and transmit a NACK. Fig. 13B is a diagram 1300B illustrating the calculation of estimated time delivery, according to some example embodiments. Fig. 13B A variation is shown in FIG. 1 , where the propagation delay T of the ACK or NACK packet is rprop The duration of 13204 may vary, and the total round-trip propagation delay of the packet and response is T rtprop =T prop +T rprop The duration of ACK or NACK may be different. rdrop 13205 is lost, and the probability is the probability that the packet is lost in the reverse direction. Taking this into account, T in the above equation rto You can use P rdrop T rto +(1-P rdrop )T rtprop Substitute, so that the EDT is calculated as:

[0567]

[0568] In some embodiments, the simplifying assumption T prop =T rprop and P drop =P rdrop To estimate EDT.

[0569] In some embodiments, the EDT can be estimated by observing the actual time that elapses from when a packet is first scheduled for delivery on a connection until the packet is successfully delivered, but this is only for packets that are retransmitted exclusively on a connection. There are also possible variations of EDT derivation, reflecting a more complex or simpler approach. For example, in a variation, there are different P values ​​for forward and reverse. drop , T prop and T rpropIt may not be symmetric, and / or ACK packets may be lost.

[0570] Fig.14 is a diagram 1400 showing wide area network connections in different tiered groups based on estimated time delivery according to some example embodiments. In diagram 1400, given specific packet characteristics, connections may be grouped into appropriate tiers as described below.

[0571] For real-time flows (those with a latency / jitter preference, expressed as a nominal “target latency”), a layer can adjust the latency based on the per-packet deadline of the flow (called the adjusted target latency or ATL) and the EDT and TTL of the connection. prop Relationships between features. For example:

[0572] ο Layer 1: T prop ≤ATL&&EDT≤ATL

[0573] Layer 2: T prop ≤ATL&&EDT>ATL

[0574] ο Layer 3: T prop >ATL&&EDT>ATL

[0575] The explanation of each of these layers is:

[0576] Layer 1: Even when taking into account packet loss and retransmissions, the one-way latency of the connection is low enough to meet the deadlines.

[0577] Layer 2: The one-way latency of the connection is low enough to meet the deadline, except for packet loss and retransmission.

[0578] o Layer 3: The one-way delay of the connection is not low enough to meet the deadline, both with and without retransmission.

[0579] ATL can be a dynamic value that changes for each packet and changes as the packet ages: ATL = Packet Deadline - Current Time, where: Packet Deadline = T PacketReceived +NominalTargetLatency.

[0580] In some embodiments, there may be fewer than 3 layers, depending on the total number of connections and their characteristics in relation to the flow cutoff time. prop and P dropmay be very small, which means that their EDT composite characteristics are also very small. Alternatively, the ATL may be very large because the real-time application is not interactive. For example, unlike a Video over Internet Protocol (VoIP) call which is interactive and has an ATL < 150 milliseconds, a broadcast video application may have an ATL < 8 seconds. In both scenarios, all connections can be classified as Layer 1 because they are "functionally equivalent" in terms of EDT being small enough to meet ATL, but they may not be equivalent in terms of efficiency.

[0581] Conversely, there may be some embodiments where there are more than 3 layers, where efficiency is a concern, for example. Consider two connections where both of their EDTs satisfy layer 1 (T prop ≤ATL&&EDT≤ATL) required relationship, but a connected P drop is large, which means it may need to retransmit each packet several times to reach its EDT. drop If is zero, each packet can reach its EDT with only a single transmission.

[0582] In this example scenario, where it is desirable to optimize efficiency, additional restrictions may be added to the tier membership to include, for example, drop The measured characteristics are:

[0583] ο Layer 1: T prop ≤ATL&&EDT≤ATL&&P drop <0.02

[0584] Layer 2: T prop ≤ATL&&EDT≤ATL&&P drop ≥0.02

[0585] ο Layer 3: T prop ≤ATL&&EDT>ATL&&P drop <0.1

[0586] ο Layer 4: T prop ≤ATL&&EDT>ATL&&P drop ≥0.1

[0587] Layer 5: T prop >ATL&&EDT>ATL&&P drop <0.35

[0588] ο Layer 6: T prop >ATL&&EDT>ATL&&P drop ≥0.35

[0589] In some embodiments, the tiers may be recalculated based on events that occur. For example, if a WAN connection has updated measured characteristics, its corresponding tier membership may be adjusted based on the new characteristics. The adjustment may include updating corresponding records or data values ​​on a stored reference table or database table used for routing control.

[0590] For real-time streaming, in some embodiments, layer recalculation may be done on a per-packet basis, since each packet may have a different ATL.

[0591] There may be more than two types of stream preferences (in addition to throughput and latency), and therefore, there may be more than two tiering / sequencing methods. In some embodiments, both may be used in combination (high throughput streams with low latency are also required, such as high-resolution real-time video).

[0592] In some embodiments, the tiering may incorporate only a subset of available connections. In one embodiment, the time of day is taken into account because it affects the expected use or demand on some connections. For example, during regular working hours, there may be a high probability that a connection becomes congested and provides poor performance. Such connections will be excluded from the tiering of mission-critical connections. In another embodiment, administrator configuration is taken into account because there may be factors that cannot be automatically detected or measured that may affect the tiering and must be manually configured. For example, a network administrator may exclude a connection due to regular maintenance at a connection provider, due to high excess charges, or due to other administrative reasons.

[0593] In another embodiment, advanced or complex connection characteristics are considered. For example, if it is determined that the throughput of a connection has a directly proportional relationship to latency (e.g., the connection can send at a very high throughput, but the latency will increase significantly, or the connection can maintain very low latency as long as a certain throughput is not exceeded), then if it is known that this connection is currently serving a high throughput flow, then the connection can be excluded from tiering for real-time flows with strict latency requirements.

[0594] In another embodiment, location is considered due to functional or policy requirements. For example, a data residency requirement may dictate that certain data is only transmitted within a network having a specified region.

[0595] Connections that can be routed to networks outside the region can then be excluded from the tier. In another example, a service with geo-locked restrictions can only service requests received from specific geographic locations. Connections that are known to route / egress through locations outside of these locations (e.g., where a concentrator providing bound connections is installed) can then be excluded from the tier. In another embodiment, security requirements can be considered. For example, unencrypted network connections (such as those not provided through a VPN) can be excluded from the tier.

[0596] In some embodiments, the layering can be extended by intentionally incorporating more connections into one or more layers. This can be achieved in more than one way, including but not limited to relaxing the layering requirements so that more connections qualify to belong to the same layer; after the layering is completed, combining subsets of the resulting layers; moving connections from one layer to other layers, etc. In some embodiments, security / obfuscation requirements that may originate from administrator configuration (management configuration) are taken into account when extending the layers. For example, obfuscation can be achieved by including more connections in the first layer so that the user's packets and / or flows are distributed across multiple networks, thereby preventing a man-in-the-middle attacker who can eavesdrop on a single network from intercepting all of the user's packets and / or flows.

[0597] In some embodiments, tiering can be completely manual based on administrator policy. In one embodiment, tiering is based on the cost of using the connection, so that connections of similar cost are grouped into the same tier, and the tiers are traversed from lowest cost to highest cost. For example, an administrator can use connection prioritization (also known as priority routing) in which low-cost connections are designated as level one, medium-cost connections are designated as level two, and high-cost connections are designated as level three, and throughput activation thresholds are configured so that the additional cost is justified when the total service drops below a specific bit rate. In another example, a modified version of connection prioritization can use different types or combinations of activation or deactivation thresholds, such as total usage during a billing period, total cost during a billing period, number of connected users, priority of connected users, etc.

[0598] In some embodiments, determining functionally equivalent WAN connections and the resulting tiering / grouping of said connections is based on machine learning or other classification-like pattern recognition techniques. For example, a model is trained using features / characteristics including measured / composite characteristics of the connections, where labels are provided directly or indirectly by application or user feedback.

[0599] According to some embodiments, each layer may be iterated from best to worst when looking for a connection on which to schedule packets for a flow. Within a layer, iteration may occur on each connection, but the starting connection for each flow may be selected in a pseudo-random manner (e.g., based on a hash of the flow tuple). The initial connection selected in each layer may always remain the same, so if the membership of the layer changes (e.g., new connections are added or removed), the starting connection may remain the same (the flow remains sticky to the first connection selected).

[0600] Fig.15A Provided to demonstrate examples of pseudo-random hashing and flow viscosity according to some embodiments. Fig. 15B Provided to illustrate stickiness expiration after idle time and after a TCP FIN exchange according to some embodiments. Fig. 15C Provided to illustrate how pseudo-random selection helps keep packets of a single flow on the same connection and also how increasing demand traverses multiple layers. Other variations are possible.

[0601] Fig.15A An example 1500A is shown where a new connection D is added to a layer that already has connections A, B, and C while the TCP and UDP flows are active.

[0602] Prior to adding connection D, the TCP stream was pseudo-randomly hashed to the connection at index 3 in the layer (i.e., connection C), and the UDP stream was pseudo-randomly hashed to the connection at index 2 in the layer (i.e., connection B). When connection D is added in 1502, it is placed at index 2, thereby replacing connection B and connection C with index 3 and index 4, respectively. Although the TCP stream and the UDP stream are pseudo-randomly hashed to index 2 and index 3, respectively, the TCP stream and the UDP stream remain sticky to the connection from which they originally started.

[0603] In some embodiments, the system can be configured so that the selected connection can eventually be forgotten based on idle time. For example, given a tier, a connection can be selected in a pseudo-random manner as a preferred connection. This connection can be selected for a flow (e.g., a video call), and packets belonging to the flow can be made sticky to the connection. Sticky connections are useful because using the same connection when possible can help improve overall communication characteristics.

[0604] However, the stream may eventually become idle because, for example, a video call ends. The system may not have explicit notification that the stream has ended, but may infer it after not seeing packets after a certain period of time, and at which time the sticky connection preference may be expired / reset.

[0605] Continuing with the above example, Fig. 15BExample 1500B is shown where a UDP flow resets a sticky connection after an idle period and is subsequently remapped.

[0606] Assume that both the TCP and UDP flows become idle for more than a minute at 1520. This causes the UDP flow to be considered ended and its sticky connection preference (connection B) is forgotten. At 1522, the TCP and UDP flows are resumed again and UDP is remapped after expiration. Here, TCP remains sticky because a FIN packet has not been detected. For example, if the UDP flow is started again, assuming the hash function produces the same index 2, the sticky connection preference is now determined again and becomes connection D, which happens to be at index 2 in the layer this time.

[0607] In some embodiments, a sticky connection may be forgotten based on different factors. For example, some protocols, such as the Transmission Control Protocol (TCP), have an explicit stream termination sequence (FIN exchange). Connections may be forgotten based on identifying such state transitions.

[0608] Continuing with the above example, Fig. 15B An example of a sticky connection being reset and subsequently remapped for a TCP flow after a FIN exchange is detected at 1524 is shown. Assume that the TCP flow has completed and both sides of the connection have exchanged FINs. This causes the TCP flow to be considered finished and its sticky connection preference (connection C) to be forgotten. In some embodiments, the TCP connection may then be established again, with a new handshake at 1524 and may be remapped. For example, if the TCP flow is started again, assuming the hash function produces the same index 3, the sticky connection preference is now determined again and becomes connection B which happens to be at index 3 in the layer this time.

[0609] In some embodiments, connections may be intentionally forgotten for security / obfuscation purposes. For example, packets may be intentionally split between connections to prevent an attacker eavesdropping on a single connection from capturing all packets associated with a flow.

[0610] Forgetting preferred connections after idle periods can be important to avoid the "over-stickiness" problem, which is a situation where many long-lived (but bursty) flows end up sticky to the same connection and competing for capacity on the same preferred connection, requiring their traffic to be split among multiple connections and reassembled at the receiver.

[0611] According to some embodiments, pseudo-random selection of preferred connections for starting iterations on "functionally equivalent" connections solves various technical challenges. Specifically, a flow is more likely to remain entirely on one connection because it will not compete with other flows for the same connection in the same order, which would make it more likely that packets would overflow onto the next connection. In addition, all connections in the active layer will tend to have a fresh characteristic because they are constantly being used by different flows.

[0612] Fig. 15C An example 1500C of two flows sending packets on multiple connections in multiple layers is shown. Assume that each connection can only send 6 packets at a time to avoid congestion. In the upper half, each of flow 1 and flow 2 has 4 packets to send.

[0613] Without the pseudo-random selection at 1540, the two flows shown in the example as Flow 1 and Flow 2 always select the first connection in Layer 1 (Connection A) as their starting connection, and packets for both flows are eventually split between Connection A and Connection B. Fig. 15C In the lower half of , with a pseudo-random selection at 1542, each flow may select a different starting connection.

[0614] For example, if flow 1 chooses connection A, and flow 2 chooses connection B: all packets for flow 1 end up only on connection A, and all packets for flow 2 end up only on connection B. In other words, each flow assigns all of its packets to a different connection.

[0615] When demand on one flow increases and requires the scheduler to iterate all of its traffic over multiple connections / layers, the other flow is not affected. In this example, demand on flow 2 increases and gets 5 additional packets to send. The flow is iterated over all connections in layer 1 and if all connections fill their capacity with packets, it is moved to layer 2. More specifically, the 5 additional packets are distributed as follows: 2 on connection B in layer 1, 2 on connection A in layer 1, and 1 on connection C in layer 2. It is observed that despite splitting flow 2 across 3 connections in 2 layers, this did not affect flow 1 which continued to have all of its packets on 1 connection, connection A in layer 1.

[0616] In some embodiments, sticky preference connections are only remembered for the first layer, because once more than one layer is used, packets are by definition split across multiple connections, and thus will need to continue to be reordered at the receiver.

[0617] In some embodiments, a preferred connection may be forgotten (and a new connection selected) if it no longer exists in the first layer. For example, its measured characteristics and resulting EDT may cause it to be moved to another layer.

[0618] The pseudo-random selection of the preferred connection can be done in many different ways. In some embodiments, the characteristics of the flow can be hashed (destination IP, port, etc.), or a random number can be generated, and in both cases, the value modulo the number of connections in the layer becomes the index of the selected connection.

[0619] In some embodiments, the selection may be made in a sequence, e.g., the first stream is assigned to the first connection in the layer as its sticky connection preference, the second stream is assigned to the second connection, and so on, wrapping around to the first connection each time the number of connections in the layer is reached.

[0620] In some embodiments, sticky selection can be performed with a bias based on available capacity. For example, selection can be based on the least currently used connections, which can be determined based on factors such as the number of flows that have selected each connection as their preference or the current unused capacity of each connection.

[0621] In some embodiments, an estimate of the flow throughput requirement can be used as part of the selection of a preferred sticky connection so that the connection selected is the one least likely to cause the flow to need to overflow to other connections. Although the connections in a layer are functionally equivalent, they are not identical, so overflowing packets onto multiple connections may result in out-of-order packets that need to be reordered at the receiver. The likelihood of this happening is reduced when the flow uses only one connection.

[0622] In some embodiments, an initial connection is selected in a layer, and then the connections in the layer are iterated in order from the shortest backlog to the longest backlog ("shortest backlog scheduling"). The backlog is measured in time units (usually milliseconds) and is defined as the volume of in-flight (transmitted but not acknowledged) bytes divided by the transfer rate of the connection. Iterating in this order minimizes the delay experienced by a flow when it requires more bandwidth than any single connection can provide on its own.

[0623] For example, consider the following scenario:

[0624] Applications that transmit packets at a rate of 1 packet per 1 millisecond

[0625] Two identical WAN connections, each of which:

[0626] Capable of transmitting 1 packet every 2 milliseconds

[0627] One-way delay is 5 milliseconds

[0628] CWND (congestion window) is 5 packets

[0629] Using sticky connection selection, packets generated by the application (1 every 1 millisecond) are scheduled onto connections C1 and C2 and transmitted as follows:

[0630] t = 0 ms: At t = 0 ms, Packet 1 is scheduled for transmission on C1 (Remaining CWND = 4)

[0631] t = 1 ms: At t = 2 ms, Packet 2 is scheduled for transmission on C1 (remaining CWND = 3)

[0632] t = 2 ms: At t = 4 ms, Packet 3 is scheduled for transmission on C1 (remaining CWND = 2)

[0633] t = 3 ms: At t = 6 ms, Packet 4 is scheduled for transmission on C1 (remaining CWND = 1)

[0634] t = 4 ms: At t = 8 ms, Packet 5 is scheduled for transmission on C1 (remaining CWND = 0)

[0635] t = 5 ms: At t = 5 ms, Packet 6 is scheduled for transmission on C2 (remaining CWND = 4)

[0636] The receiver receives the out-of-order packets as follows:

[0637] Packet 1: Received via C1 at t = 5 ms

[0638] Packet 2: Received via C1 at t = 7 ms

[0639] Packet 3: Received via C1 at t = 9 ms

[0640] Packet 6: Received via C2 at t = 10 ms

[0641] Packet 4: Received via C1 at t = 11 ms

[0642] Packet 5: Received via C1 at t = 13 ms

[0643] Once the receiver reorders the packets back into the correct sequence and releases them to the destination, the destination experiences high jitter (packets are no longer consistently 1 millisecond apart) and overall latency that is longer than the 5 millisecond one-way latency on each connection:

[0644] Packet 1: released after reordering at t=5 ms

[0645] Packet 2: released after reordering at t=7 ms

[0646] Packet 3: released after reordering at t=9 ms

[0647] Packet 4: released after reordering at t=11 ms

[0648] Packet 5: released after reordering at t=13 ms

[0649] Packet 6: released after reordering at t=13 ms

[0650] Conversely, performing connection selection and packet scheduling based on minimizing the backlog will result in packets being scheduled onto C1 and C2 and transmitted as follows:

[0651] t=0 ms: At t=0 ms, schedule packet 1 for transmission on C1 (remaining CWND=4)

[0652] t=1 ms: At t=1 ms, schedule packet 2 for transmission on C2 (remaining CWND=4)

[0653] t = 2 ms: At t = 2 ms, packet 3 is scheduled for transmission on C1 (remaining CWND = 3)

[0654] t=3 ms: At t=3 ms, schedule packet 4 for transmission on C2 (remaining CWND=3)

[0655] t = 4 ms: At t = 4 ms, schedule packet 5 for transmission on C1 (remaining CWND = 2)

[0656] t=5 ms: At t=5 ms, schedule packet 6 for transmission on C2 (remaining CWND=2)

[0657] The receiver receives the packets in order with no additional jitter (the inter-packet interval remains consistent at 1 ms), and the application experiences a delay equal to the 5 ms one-way delay on each connection:

[0658] Packet 1: Received and released via C1 at t = 5 ms

[0659] Packet 2: Received and released by C2 at t = 6 ms

[0660] Packet 3: Received and released via C1 at t = 7 ms

[0661] Packet 6: Received and released by C2 at t = 8 ms

[0662] Packet 4: Received and released via C1 at t = 9 ms

[0663] Packet 5: Received and released by C2 at t = 10 ms

[0664] In some embodiments, the shortest backlog scheduling is instead reduced to polling between each connection in the layer. In cases where the connections are known to have the same characteristics, this approach is easier to implement and results in similar performance.

[0665] In some embodiments, connection priority (CP) rules or priority routing can be specified by the administrator to express preferences for the order in which connections are used. In previous solutions, these rules were global, applying to all flows, and also had the ability to limit bandwidth usage on low priority connections. Consider an example set of connections with a CP configuration as follows:

[0666] Connection A: Level 1

[0667] Connection B: Level 2 / Threshold = 10 Mbps

[0668] According to these previous embodiments, connection B will not become active unless the capacity of connection A drops below 10 Mbps, and once active, connection B will only contribute the difference to bring the aggregate total capacity back to 10 Mbps.

[0669] According to some current embodiments, the granularity of CP rules can be the same as for a single flow. Limiting bandwidth usage with lower priority may also be removed, because a high priority for one flow may be a low priority for another flow, so trying to limit bandwidth usage would conflict with the requirements of both flows. For example, assume that flow 1 has the same CP rules as described in the previous example, but flow 2 has the opposite CP rules:

[0670] Connection B: Level 1

[0671] Connection A: Level 2 / Threshold = 10 Mbps

[0672] Flow 1 can consume the full capacity of connection A because A is its primary, and flow 2 can consume the full capacity of connection B because B is its primary. Neither can be restricted as they could using global CP rules.

[0673] In some embodiments, for example, the bandwidth limiting functionality is replaced by a global setting that limits bandwidth usage per WAN connection (regardless of flow or CP rules) or a bandwidth limit applied to matching flows.

[0674] CP rules may affect the construction of layers, because if two connections are at different CP levels, they may be placed in different layers even if they are "functionally equivalent" when all their other characteristics are considered. For example, consider the following

[0675]

[0676]

[0677] Given ATL's 100ms packet, in the absence of connection priority rules, the layer construction would be:

[0678] Layer 1: A, B (T prop ≤ATL&&EDT≤ATL)

[0679] Layer 2: C, D (T prop ≤ATL&&EDT>ATL)

[0680] Layer 3: E, F (T prop >ATL&&EDT>ATL)

[0681] However, using the connection priority rules, even if T prop and EDT features still satisfy the constraints, but each change of priority results in the creation of a new layer:

[0682] Layer 1: A(T prop ≤ATL&&EDT≤ATL&&CP=Level 1)

[0683] Layer 2: C(T prop ≤ATL&&EDT>ATL&&CP=Level 1)

[0684] Layer 3: E(T prop >ATL&&EDT>ATL&&CP=Level 1)

[0685] Layer 4: B(T prop ≤ATL&&EDT≤ATL&&CP=Secondary)

[0686] Layer 5: D(T prop ≤ATL&&EDT>ATL&&CP=Secondary)

[0687] Layer 6: F(T prop >ATL&&EDT>ATL&&CP=Secondary)

[0688] Additionally, if one connection is "functionally better" than another connection, it may be placed in a less preferred tier because it belongs to a lower priority CP level. For example, although connection B has a lower T prop and EDT values, but due to the connection priority rules, the above connection E is placed in layer 3, while connection B is placed in layer 4.

[0689] In some embodiments, scheduling methods are introduced to help ensure that packets belonging to real-time streams arrive before their deadlines, but this is balanced against bandwidth usage (e.g., initial implementations might simply broadcast real-time packets over all available connections).

[0690] In some embodiments, real-time packets may be scheduled onto multiple connections to maximize the probability of the packets arriving before the deadline while balancing bandwidth usage. As a non-limiting example, according to one embodiment:

[0691] For example, to assign a score to a packet transmission on each layer:

[0692] ο Tier 1: Rating = 3

[0693] ο Layer 2: Rating = 2

[0694] ο Layer 3: Rating = 1

[0695] Schedule packets on multiple connections as needed so that their cumulative score is greater than or equal to a predefined target score. For example, the target value can be a parameter such as 3.

[0696] Connections that have no fresh statistics or are unreliable (close to 100% loss probability), regardless of their tier, may contribute score = 0. This may have the intentional side effect of refreshing statistics on these connections if / when packets are acknowledged.

[0697] Fig.16 An example 1600 is shown with various connections 1602 sorted into various layers 1604 for sending a number of packets 1606. Similar to the above, layer 1 connections are scored 3, layer 2 connections are scored 2, and layer 3 connections are scored 1. Unreliable connections or connections with outdated statistics are scored 0 regardless of which layer they belong to. For each incoming packet, the connections are iterated from layer 1 to layer 3 (in other words, from connections A to F). Each packet is sent on the connection it spans, and its score is accumulated until the target score of 3 is reached or exceeded. This produces the following exemplary results:

[0698] Packets 1 to 4 reach the target score immediately after being sent on the first connection (connection A). At this point, the capacity of connection A is fully consumed.

[0699] Packet 5 reaches the target score after being sent on 3 connections: Layer 1: Connection B, Layer 2: Connection C, and Layer 2: Connection D. They have accumulated scores of 0, 2, and 2, respectively, for a total score of 4, which exceeds the target score of 3. Note that Layer 1: Connection B has a score of 0 because it is unreliable. At this point, the capacity of Connection D is fully consumed.

[0700] Packet 6 reaches the target score after being sent on 4 connections: Layer 1: Connection B, Layer 2: Connection C, Layer 2: Connection E, and Layer 3: Connection F. They have accumulated scores of 0, 2, 0, and 1, respectively, for a total score of 3, which meets the target score. Note that Layer 2: Connection E has a score of 0 because its statistics are not fresh. At this point, the capacity of Layer 1: Connection B and Layer 3: Connection F is fully consumed.

[0701] Packet 7 does not reach the target score even after being sent on 2 connections: 2: Connection C, and Layer 2: Connection E. It accumulates scores of 2 and 0, respectively, for a total score of 2, which does not reach the target score of 3. At this point, the capacity of Layer 2: Connection C, and Layer 2: Connection E is fully consumed. At this point, the capacity of all connections has also been fully consumed, so all sending stops. Unless Packet 7 belongs to a packet that has been confirmed as received, in the next round of sending, Packet 7 continues to be sent when more connection capacity becomes available (e.g., packets are confirmed as received or new connections have been added).

[0702] Packet 8 does not achieve the target score because all connection capacity is fully consumed and the packet is not sent on any connection. When more connection capacity becomes available, sending packet 8 may occur in the next round of sending.

[0703] Some embodiments may have different predefined target values ​​on a per-flow basis, such as a very high priority flow that wishes to broadcast its packets on all connections, or other methods that allow the administrator to control the tradeoff between bandwidth usage and reaching deadlines.

[0704] In some embodiments, to reach the target score, the CP rules may be ignored, assuming that the user will prefer to reach the deadline, even if activating a lower priority will incur additional costs. For example, if the cumulative score does not reach the target after iterating all connections in all higher priority layers (which the CP rules consider active), iterating over connections in lower priority layers (which the CP rules consider inactive) may continue.

[0705] In some embodiments, preemptive scheduling can take into account predictions of upcoming position changes (e.g., buses, airplanes, etc. moving on predictable paths with predictable changes in measured characteristics). For example, a current layer 1 connection may soon become layer 3, and vice versa, so in some embodiments, the target score can be adjusted in advance and more redundancy can be transmitted closer to the transition.

[0706] The described embodiments present a technical improvement over existing solutions because preemptive retransmission is only applied to flows with deadlines (real-time applications), and the flow deadline and connection EDT can be considered when deciding whether preemptive retransmission is needed, rather than just the connection loss probability. Throughput flows or other similar types without deadlines will not incur the bandwidth cost and overhead of preemptive retransmission.

[0707] In other embodiments, there may be early retransmissions of packets that are approaching their deadlines but are still in flight and have not yet been acknowledged (including preemptive retransmissions). Such embodiments may balance meeting deadlines with efficient use of connections (cost—dollars, bandwidth, compute). In some embodiments, there may be per-flow rules that allow administrators to influence this balance. For example, if a flow is so important that early retransmissions of packets can be broadcast on all connections regardless of the additional costs that may be incurred, then the "cost" will not be important. As a non-limiting example:

[0708] In one embodiment, if a packet has a deadline and it has not been acknowledged as received, and the reliable connection with the lowest latency, available capacity, and freshness statistics has just enough time to span the packet for one additional time (taking into account some variations, such as time to handle overhead), an early retransmission of this packet is sent on this connection. This is done as a last resort to span the packet in time before its deadline. Fig.17 Example 1700 of such an embodiment (Embodiment 1) is shown in which an unacknowledged packet originally sent on connection A at t=0 msec with a deadline of 150 msec has an early retransmission sent once more on connection A at t=150-50-15=85 msec, where 50 msec is an assumed time difference and 15 msec is the known delay of connection A.

[0709] In one embodiment, the time difference is a fixed amount, such as 50 milliseconds, that can account for some variation in processing time and actual connection delay.

[0710] In another embodiment, the time difference is a variable function of the connection type or measured characteristic. For example, if the round trip time of the connection has a statistically large standard deviation, the time difference of early retransmission may be increased. In another example, satellite connections are known to have relatively stable processing times compared to, for example, cellular connections, and may therefore be assigned a lower time difference when considering early retransmissions.

[0711] In one embodiment, if this packet has been previously sent using the reliable connection with the lowest latency, available capacity, and freshness statistics, then an early retransmission of the packet will be sent using another reliable connection with sufficiently low latency, freshness statistics, and available capacity to make the packet cross before its deadline. This is done to increase the diversity of connections over which the same packet is sent, and because the first connection selected is considered likely to be unreliable because it has not yet delivered its copy of the packet. Fig.17 An example 1700 of such an embodiment (Embodiment 2) is shown in which, since connection A has already been used to originally send the packet, it is avoided and an early retransmission of the packet is sent on connection D at time t = 150 - 50 - 20 = 80 milliseconds, where 50 milliseconds is the assumed time difference and 20 milliseconds is the known delay of connection D.

[0712] In one embodiment, early retransmissions of unacknowledged packets that are approaching their deadlines are sent on some or all connections that are unreliable, have no fresh statistics, or both. Such connections may have become reliable, or now have more favorable characteristics (e.g., they may have become less congested because they have not been used for a period of time), and therefore may be able to have packets spanned before their deadlines. The number of such connections used for early retransmissions may depend on the balance between meeting deadlines and effectively using the previously mentioned network connections. Fig.17 An example 1700 of such an embodiment (Embodiment 3) is shown, in which early retransmissions are sent on all unreliable and / or outdated connections. Each transmission time is calculated similarly to the previous embodiment in this figure. For connection F, the calculation results in a time in the past (t = 150-50-150 = -50 milliseconds); therefore, the early retransmission on connection F is sent simultaneously with the earliest early retransmission, that is, the early retransmission of connection C in this example (t = 75 milliseconds).

[0713] In one embodiment, more than one of the above-described early retransmission methods is combined. For example, when serving a very important flow whose deadlines should be met as much as possible at any cost, an embodiment may combine all of the above-described methods. Thus, early retransmissions are sent on: the original connection on which the packet was sent; other reliable connections with appropriate latency and EDT, available capacity, and freshness statistics; and all other connections that are unreliable or have outdated statistics. Fig.17 An example 1700 of such an embodiment (Embodiment 4) is shown, which combines the early retransmissions of Embodiments 2 and 3.

[0714] In one embodiment, the above-described early retransmission methods are considered sequentially (in any order) until at least one or more connections for sending early retransmissions of the current packet are found.

[0715] Early retransmissions may ignore CP rules, assuming that early retransmissions are the last chance to meet the packet deadline, so all available connections must be considered.

[0716] In some embodiments, a modified scheduling method is introduced for packets belonging to flows with throughput preference, assuming that the method has a higher tolerance to delay / jitter / loss.

[0717] According to an embodiment, the scheduling algorithm is similar to that described above for real-time flows, except that each layer has a contribution score of 3 (except for outdated or unreliable connections that still have a contribution score of 0). Therefore, packets belonging to the throughput flow may be transmitted only once (excluding outdated or unreliable connections).

[0718] Fig.18 is a diagram of an example 1800 showing a plurality of connections 1802 sequenced into a plurality of layers 1804 for sending a number of packets 1806 for a throughput flow, according to some example embodiments. Fig.18 Shows Fig.16 A variation of the example in , but this time for the throughput stream. The categories of tiers 1804 and connections 1802 are the same, except that all tiers 1804 now have a score of 3. Unreliable connections or connections 1802 with outdated statistics continue to have a score of 0, regardless of which tier they belong to. This produces the following exemplary results:

[0719] Packets 1 to 4 reach the target score immediately after being sent on the first connection (connection A). At this point, the capacity of connection A is fully consumed.

[0720] Packets 5 and 6 reach the target score after being sent on 2 connections: Layer 1: Connection B and Layer 2: Connection C. They each accumulate scores of 0 and 3, respectively, for a total score of 3, which reaches the target score of 3. Note that Layer 1: Connection B has a score of 0 because it is unreliable. At this point, the capacity of Connection B is fully consumed.

[0721] • Packet 7 reaches the target score immediately after being sent on layer 2: connection C. At this point, the capacity of connection C is fully consumed.

[0722] • At layer 2: Packet 8 reaches the target score immediately after being sent on connection D. At this point, the capacity of connection D is fully consumed.

[0723] Packet 9 reaches the target score after being sent on 2 connections: Layer 2: Connection E and Layer 3: Connection F. It has accumulated scores of 0 and 3 respectively, for a total score of 3, which reaches the target score of 3. Note that Layer 2: Connection E has a score of 0 because its statistics are not fresh. At this point, the capacity of Connection F is fully consumed.

[0724] Even after being sent on 1 connection, packet 10 does not reach the target score: 2: Connection E. It accumulates a score of 0 and does not reach the target score of 3. At this point, the capacity of layer 2: Connection E is fully consumed. At this point, the capacity of all connections has also been fully consumed, so all sending stops. Unless packet 10 belongs to a packet that has been confirmed as received, in the next round of sending, continue to send packets 10 when more connection capacity becomes available (e.g., packets are confirmed as received or new connections are added).

[0725] In some embodiments, priorities between flows may be implemented at a level prior to packet scheduling (input queues). A packet queue may be created for each flow entering the system, and the queues may be serviced according to a Deficit Round Robin (DRR) and / or Deficit Weighted Round Robin (DWRR) queue scheduler algorithm (described in RFC 8290), which attempts to be fair to all queues by default. The described embodiments introduce a technical improvement as an additional modification that allows administrators to configure intentional unfairness.

[0726] DRR / DWRR has the concept of quantum (i.e., the target number of bytes to serve from each queue in each round). Typically, this quantum is equal for all queues, but the modification allows this to be modified by weights (specified as rules by administrator configuration). In some embodiments, weights can be specified by other means, such as automatically determined based on application type, time of day, machine learning, etc.

[0727] For example:

[0728] ο Stream 1: Weight = 2

[0729] ο Stream 2: Weight = 5

[0730] ο Stream 3: Weight = 1

[0731] In this example, stream 1 receives 2 times the nominal quantum per round, stream 2 receives 5 times the nominal quantum, and stream 3 receives 1 times the nominal quantum. In some embodiments, quanta can be scaled proportionally, such as by multiples, while retaining relative weights. For example, nominal quantum*k, multiplied by 2 / 8, 5 / 8, 1 / 8, respectively, where nominal quantum and k are parameters. In some embodiments, weights can be assigned to rules, and all streams matching the rule share a common quantum pool.

[0732] In the above example, k would be k=1, and the nominal quantum would have a default value, which may be manually specified by administrator configuration, or automatically specified based on application type, time of day, machine learning, etc.

[0733] In some embodiments, the concept of queue importance levels is introduced. Queues of higher importance are usually served before those of lower importance. In some embodiments, if there is only enough capacity to serve high importance queues, lower importance queues may be starved. Queues of the same importance level may split capacity according to their assigned weights.

[0734] This embodiment improves upon the existing DRR / DWRR because it describes a multi-level DRR / DWRR priority queue. The characteristic of the conventional DRR / DWRR queue is that each flow "i" will achieve a minimum long-term average data rate:

[0735]

[0736] Where Q i is the quantum of flow "i", and R is the total WAN capacity available.

[0737] In some of the described multi-level DRR / DWRR queue embodiments, flows predefined as having higher "importance" are allowed to starve flows of lower "importance" such that a given flow "i" in priority level "M" will achieve a minimum long-term average data rate:

[0738]

[0739] Where Q i is the quantum of flow "i" among N flows in priority level M, with priority levels 0 to M-1 all having higher priority than M, and R j is the WAN capacity used by all flows in a particular priority level.

[0740] Fig.19An example of an embodiment 1900 with a 2-level DWRR queue management algorithm is shown in accordance with some example embodiments. DWRR level 1 is more important than DWRR level 2. DWRR level 1 has 3 flows with weights as described above, i.e., flows 1, 2, and 3 are 2, 5, and 1, respectively. DWRR level 2 has a single flow 4 with a weight of 2. Assume that there are 3 connections in this example with a total capacity of 16 packets per second. Assume that the packet rates of the flows are as follows: flow 1 generates 4 packets per second, flow 2 generates 10 packets per second, flow 3 generates 3 packets per second, and flow 4 generates 3 packets per second. Since DWRR level 1 is more important than DWRR level 2, DWRR level 1 will always be served before DWRR level 2 as long as it has available packets from any of its flows. In this example, the flows of DWRR level 1 generate a total of 17 packets per second, which is higher than the total capacity of the connections; in other words, DWRR level 1 will always have available packets, causing DWRR level 2 to become starved and never be served.

[0741] Fig.19 The table 1902 at the bottom of the diagram shows the number of packets seen by each flow at the input and output over time.

[0742] Flows 1 and 2 in DWRR level 1 are of high importance and generate packets proportional to their quanta; therefore, the number of output packets always catches up with the number of input packets.

[0743] Flow 3 in DWRR level 1 is of higher importance and is served; however, since it generates more packets than its quantum share, it forms a backlog of packets, which is seen as the number of input packets growing faster than the number of output packets.

[0744] Flow 4 in DWRR level 2 is of lower importance and is not served because the packets generated by the higher importance DWRR level 1 are sufficient to consume the full capacity of the available connection. Flow 4 is starved and this is seen as the number of input packets increasing while the number of output packets remains at 0. If flow 4 were at DWRR level 1, some of its packets would be served before DWRR returns to serve flow 1, which means that flow 4 would not be completely starved.

[0745] This embodiment introduces technical improvements to the above sequencer to operate more efficiently on a per-stream basis. The sequencer can operate using the following measurement characteristics:

[0746] a) Perform statistically filtered measurements on the one-way latency of each WAN connection.

[0747] b) The set of WAN connections that a particular flow is actively using.

[0748] Fig. 20 is a block diagram of a per-flow packet receiver 2000 according to some example embodiments.

[0749] The per-flow packet receiver maintains an estimate of the one-way delay of the WAN connection at 2001, and an estimate of the set of WAN connections that have recently been used for a particular flow at 2002. These can be used to determine the appropriate time to release packets from the flow output queue at 2003 to minimize jitter in the propagation time of packets in the flow through the system.

[0750] The basic implementation measures and stores both characteristics on a per-flow basis. The described embodiment provides an improvement over the observation in (a) above, namely that the statistically filtered measurement of the one-way delay of each WAN connection does not vary on a per-flow basis, so the estimate of the one-way delay of the WAN connection at 2001 can be shared among all flows.

[0751] In some per-stream packet receiver embodiments, the one-way delay estimator may accept tuning parameters that adjust the quality of the estimate based on the expected characteristics of the stream or WAN connection. The tuning parameters may include low-pass filter parameters and the duration after which a single packet measurement is lost. In this embodiment, there may be more than one estimate of the one-way delay per WAN connection, and these estimates may be shared among all streams using the same tuning parameters.

[0752] In some embodiments, the estimated one-way delay of the WAN connection at 2001 may be used to calculate the EDT of the WAN connection.

[0753] The above-described embodiments, including improvements to layering, scheduling, and sequencing (collectively referred to as "real-time streaming support"), have been tested in simulated scenarios that mimic real-world usage patterns and have been shown to provide quantitative improvements in jitter and latency. Specifically, a common scenario is the use of a real-time video chat application, such as Zoom, over a multi-WAN router containing a wired broadband connection (e.g., consumer cable or DSL) and a 3xLTE connection. TM , Microsoft Teams TM or Google Meet TM The connection priority (CP) rule is used to set the wired broadband connection as the primary one and the LTE connection as the secondary one.

[0754] The scenario then simulates a sudden failure of the wired broadband link (e.g., 100% packet loss and / or 0 bandwidth capacity) and measures the resulting impact on packet loss, latency, and jitter. The results are as follows:

[0755] Example Packet loss rate Jitter (average) Jitter(max) Delay (average) Delay (maximum) Live streaming support 0％ 797 ms 1158 ms 40.7 ms 62 ms No live streaming support 0％ 986 ms 2070 ms 51.1 ms 351.2 ms

[0756] The improvement of jitter and latency comes from preemptive retransmission (target scoring system) and early retransmission logic. In this scenario, when the wired broadband connection suddenly fails, the in-flight packets lost on the wired broadband connection have been preempted or retransmitted in advance on the LTE connection. In contrast, embodiments without real-time streaming support must wait for the full RTO (retransmission timeout) to occur before retransmitting the lost packets.

[0757] Fig.21 is a block diagram of a per-stream communication system workflow 2100 according to some example embodiments. For a user connecting a smartphone via WiFi to a multi-LTE WAN router (such as the one described above), the experience may be similar to making a video call over a wired broadband connection.

[0758] According to one embodiment, the per-stream communication system workflow 2100 may follow such a process as outlined below, but in other embodiments, more or fewer steps may be included and may be performed in a different order.

[0759] In the exemplary workflow 2100 shown, starting from 2110, a target score is assigned to a packet based on the requirements of the flow to which the packet belongs. Then, in 2111, the connections can be tiered based on the flow requirements. For example, for real-time flows, the per-packet deadline ATL and the characteristics of the connection (such as EDT and T prop ) is used to build layers. For throughput flows, layers can be built based on unique Mathis and capacity factor pairs.

[0760] Next, at 2112, scores may be assigned for packet transmissions occurring on connections at each layer. Connections with no fresh statistics (stale connections) or unreliable connections (connections with a loss probability approaching 100%) contribute a score of 0 regardless of the layer. At 2113, the packet scheduling process begins, iterating through each layer from best to worst, and iterating through each connection in each layer, transmitting packets on multiple connections as needed to achieve a cumulative score greater than or equal to the predefined target score previously assigned in 2110.

[0761] A connection within the currently selected layer may then be selected in a pseudo-random manner at 2120. If the current layer is also the highest (best) layer, the selected connection will be remembered as a sticky preference connection. If this connection still exists in the best layer, future packets belonging to this flow will always start from that connection (rather than selecting a connection in a pseudo-random manner).

[0762] Next, the packet may be transmitted in 2121 and the cumulative score may be updated. Next, it may be determined in 2122 whether the target score has been reached. If not, a check may be made at 2123 whether any more available connections occur. If so, at 2124, iteration may occur at the next connection in the current layer, thereby exhausting all available connections in the current layer before moving on to the next layer.

[0763] If at 2125 a connection is still available in the current layer, the process can continue from 2121, transmitting on the next connection in the layer, and updating the score accordingly. If all connections in the layer have been exhausted, it can iterate to the next layer, and the process can continue at 2120, selecting a connection in this new layer in a pseudo-random manner.

[0764] Once the target score has been reached in 2122, it may be determined at 2126 whether there are any packets remaining to be transmitted. If so, the entire process will begin again for the next packet at 2110. If all available packets have been transmitted at 2126, packet scheduling is stalled at 2130 until more packets arrive for transmission.

[0765] It is also possible that the packet has not yet reached its target score at 2122, but there are no available connections in any layer at 2123, where the lack of availability may be due to reasons such as all connections having consumed their available capacity (in-flight packets are greater than or equal to their congestion windows), or connection priority rules preventing some connections from being activated. If this is the case, packet scheduling is stalled at 2130 until an event occurs such as connection capacity becoming available due to in-flight packets being acknowledged. This allows the packet to begin the recovery scheduling and transmission process again at 2110, but using part of its accumulated score.

[0766] As a non-limiting example implementation that can follow the workflow shown in 2100, consider a multi-LTE WAN router including X connections, and connection 1 will be equivalent to the old single LTE connection. In view of this, there are now connections 2 to X. A user can start a voice call, which has a delay / jitter sensitive flow (packet 1, packet 2, packet N). The flow itself may also have a given delay / jitter requirement. Consider 150 ms maximum delay, 30 ms maximum jitter, and 1% maximum packet loss rate, respectively.

[0767] In some embodiments, a flow may consist of packets with multiple different latency / jitter requirements, but similar to a standard router, the flows can be identified and separated based on fields in the standard IP / TCP / UDP header ("SrcIP, SrcPort, DstIP, DstPort, Protocol, DSCP" tuple). Typically, the application is aware of this router standard behavior, and therefore, if it has different latency / jitter requirements for different packets, it will create different flows for them.

[0768] Now, consider the initial starting assumptions of the example implementation through the exemplary workflow 2100. Recall that, as described above, according to some embodiments, a score can be assigned to packet transmissions on each layer. In this example, layer 1 is given a score of 3, layer 2 is given a score of 2, and layer 3 is given a score of 1. Regardless of the layer, connections that do not have fresh statistics or unreliable connections contribute a score of 0. Packets can then be scheduled on multiple connections as needed so that the packet cumulative score is greater than or equal to a predefined target score. In this example, a target score of 3 is considered. It is also assumed that the nominal target latency requirement of the application is 150 milliseconds and that the initial characteristics of the available connections are as follows:

[0769]

[0770]

[0771] Next, consider Packet 1 and its tiering. Recall that, as described above, according to some embodiments, tiers are based on the per-packet deadline of the stream (referred to as ATL) and the EDT and T prop Given a nominal target latency of 150 ms, and assuming that packet 1 ATL is also 150 ms, the resulting layers are as follows:

[0772] oPackage 1:

[0773] i. Layer 1: Connect A, B, C

[0774] ii. Layer 2: F, G

[0775] iii. Layer 3: J, K

[0776] iv.Layer 4:D,E

[0777] v. Layer 5: H, I

[0778] vi. Layer 6: L, M

[0779] As described above, iteration may start at layer 1. In this example, assume that the first transmission of this flow is chosen in a pseudo-random manner to be sticky to connection B. An attempt is made to transmit on B, but it does not currently have available capacity. Iterate through the layers to connection C, which transmitted once, but is stale, so its contribution to the goal is 0. Iterate to connection A, which transmitted once, and A is fresh, and its contribution to the goal is 3.

[0780] Next consider packet 2, where the layers are the same, but the characteristics of B have changed so that it now has some capacity available. B has a sticky connection preference, so packet 2 is transmitted once on B, which is still fresh, and contributes 3 to the goal.

[0781] For packet 3, assume that connections B and C become unavailable (100% packet loss). B and C are moved to layer 2 (EDT becomes infinite, T prop Still the last measured value), and it is marked as outdated. B's sticky connection preference is no longer in layer 1, so it is forgotten. Transmit once on the only remaining layer 1 connection A (fresh, with a contribution of 3 to the target), and select A as the new sticky connection preference.

[0782] Package 3:

[0783] vii.Layer 1:A

[0784] viii. Layer 2: B, C, F, G

[0785] ix. Layer 3: J, K

[0786] x. Layer 4: D, E

[0787] xi.Layer 5:H,I

[0788] xii. Layer 6: L, M

[0789] For packet 4, assume that connection A becomes unavailable (100% packet loss). A is moved to layer 2 (EDT becomes infinite, T prop is still the last measured value) and is marked as outdated. Layer 1 is empty, so the sticky connection preference is completely forgotten, and iteration starts from layer 2. Assume that B is selected in a pseudo-random manner from layer 2 as the initial connection to start the iteration, so packet 2 is transmitted once on B, which is outdated and its contribution to the target is 0.

[0790] Package 4:

[0791] xiii. Layer 1:

[0792] xiv. Layer 2: A, B, C, F, G

[0793] xv. Layer 3: J, K

[0794] xvi. Layer 4: D, E

[0795] xvii. Layer 5: H, I

[0796] xviii. Layer 6: L, M

[0797] Iterate to the next connection C and transmit once (outdated, contribution to the target is 0).

[0798] Iterate to the next connection F and transfer once (fresh, contribution to the target is 2).

[0799] Iterating to the next connection G, there is no available capacity, so it is skipped.

[0800] Iterate to the next connection A, transmit once (outdated, contribution to the target is 0).

[0801] Now, layer 2 is exhausted, and the goal is still not reached, so iterate to layer 3. For layers greater than 1, there is no sticky connection preference, so a pseudo-random starting selection of J is assumed. Packet 4 is transmitted once on J (outdated, with a contribution of 0 to the goal).

[0802] Iterate to the next connection K, and transmit once (fresh, with a contribution of 1 to the target, resulting in a cumulative score of 3).

[0803] For the last packet 5, assume that connection K becomes unavailable (100% packet loss). K stays in layer 3 (EDT becomes infinite, T prop is the last measured value, the target latency is still > 150 ms) and is marked as out of date.

[0804] Package 5:

[0805] xix. Layer 1:

[0806] xx. Layer 2: A, B, C, F, G

[0807] xxi. Layer 3: J, K

[0808] xxii.Layer 4:D,E

[0809] xxiii. Layer 5: H, I

[0810] xxiv. Layer 6: L, M

[0811] It is assumed that all of the stale connections that were previously transmitted for packet 4 are still stale. Typically, this is not actually the case, as those stale transmissions that occurred with packet 4 would refresh the connection statistics, allowing updated characteristics to be measured on each of the stale connections. However, this example will assume this is the case in order to show how to use secondary priorities to achieve the goal.

[0812] Repeating all the same iterations in package 4, the inventors found that layer 3 had been exhausted and the target score was still not reached. Therefore, iterated to layer 4. There was no sticky connection preference for layer 4, so a pseudo-random starting selection of E was assumed.

[0813] Transmit once on E (outdated, contribution to target is 0).

[0814] Iterate to the next connection D, transmitting once (stale, contribution to the goal is 0). Then, layer 4 is exhausted, and the goal is still not reached, so iterate to layer 5. No sticky connection preference exists for layer 5, so a pseudo-random starting choice of H is assumed. Transmit once on H (fresh, contribution to the goal is 2, cumulative score is 4).

[0815] It is assumed that there are no more packets available, so scheduling and transmission are suspended until an event occurs (eg, a new packet is received at the input, a negative acknowledgment is received for a transmitted packet, etc.).

[0816] Also consider that given all the outdated and preempted packet transmissions described above, there may be many duplicate and out-of-order packets. In some embodiments, reordering and de-duplication may occur, and the endpoint may receive packets in order.

[0817] According to one embodiment, implementation of a per-stream communication system may follow such a process as outlined above, but in other embodiments, more or fewer steps may be included, and possibly performed in a different order.

[0818] Consider another non-limiting example implementation for a flow with a throughput maximization requirement (as opposed to a real-time flow in the previous example) that can follow the exemplary workflow 2100. Recall that, as described above, according to some embodiments, a score can be assigned to packet transmissions on each layer. In this example, the throughput flow configures all layers to contribute a score of 3. Connections that do not have fresh statistics or unreliable connections contribute a score of 0 regardless of the layer.

[0819] In this example, the target score for the throughput flow is also 3, meaning that a single packet transmission on any fresh and reliable connection is sufficient to achieve the target. The threshold for activating secondary connections is also configured to be 10 Mbps, meaning that the total capacity of all primary connections must drop below 10 Mbps before a secondary connection will become available for transmission.

[0820] The initial state of available connections is assumed to be as follows:

[0821]

[0822] Next, consider the transmission of Packet 1 and its stratification. Recall that, as described above, according to some embodiments, the stratification of throughput flows is based on a unique tuple of rounded Maxis coefficient, rounded capacity factor, and connection priority. Given that smaller rounded Maxis coefficient values ​​are better, and larger rounded capacity factor values ​​are better, this yields the following stratification:

[0823] Package 1:

[0824] Layer 1: Connect A and E

[0825] Layer 2: B

[0826] Layer 3: F

[0827] Layer 4: C, D

[0828] As described above, iterations may start at level 1. In this example, assume that the first transmission of this flow is chosen in a pseudo-random manner to be sticky to connection A. A transmission occurs on A, but it is out of date, so its contribution to the goal is zero.

[0829] Now, layer 1 is exhausted and the goal is still not reached, so iterate to layer 2. There is no sticky connection preference for layers greater than 1, and there is only one connection in this layer, so pseudo-random selection provides choice B by default. Capacity is available, so the transfer occurs. The connection is also fresh, so its contribution to the cumulative score is 3, thus reaching the desired goal.

[0830] Next, consider the transmission of packet 2, which belongs to the same flow. Assume that all connection characteristics remain the same as for packet 1, except that the measured characteristics of A are updated due to the transmission of packet 1 so that they are no longer out-of-date. The flow has a sticky connection preference for A, so packet 2 is transmitted on it, achieving the target score of 3.

[0831] Next, consider the transmission of packet 3 belonging to the same flow. Assume that connection A has now become unreliable and obsolete, causing its rounded Mathis coefficient to increase to a maximum value of 5,000.

[0832]

[0833]

[0834] Connection A now has a unique tuple, causing it to be placed in its own layer:

[0835] Package 3:

[0836] Layer 1: Connection E

[0837] Layer 2: B

[0838] Layer 3:A

[0839] Layer 4: F

[0840] Layer 5: C, D

[0841] Iteration starts at layer 1. Connection A no longer exists in layer 1, so its sticky connection preference is forgotten. Transmission is attempted on E, and it becomes the new sticky connection preference, but it has no capacity available, so it is skipped. Iteration proceeds to layer 2.

[0842] Sticky connection preference exists only for layer 1, and there is only one connection in layer 2, so pseudo-random selection provides choice B by default. Capacity is available, so the transfer occurs. The connection is also fresh, so its contribution to the cumulative score is 3, achieving the desired goal.

[0843] Next, consider the transmission of packet 4 belonging to the same flow. Assume that connection B has now become unreliable, causing its rounded Mathis coefficient to increase to a maximum value of 5,000, and it becomes obsolete.

[0844]

[0845] This makes its tuple the same as connection F, so it is now in the same layer:

[0846] Package 4:

[0847] xxv. Layer 1: Connection E

[0848] xxvi. Layer 2: A

[0849] xxvii. Layer 3: B, F

[0850] xxviii. Layer 4: C, D

[0851] Iteration starts at layer 1. Connection E is a sticky connection preference and it exists in layer 1, but it has no capacity available, so it is skipped. Iteration proceeds to layer 2. Only layer 1 has a sticky connection preference, and there is only one connection in layer 2, so pseudo-random selection provides choice A by default. It is out-of-date and unreliable, so even if a transfer occurs, its contribution to the cumulative score is 0.

[0852] Iteration proceeds to layer 3. Sticky connection preference exists only for layer 1, so connection F is chosen first assuming pseudo-random connection selection. Transfer occurs, but since it is stale, its contribution to the target score is 0. Iteration proceeds within the layer to connection B. It is stale and unreliable, so it also contributes to the target score after transfer.

[0853] The iteration stops without going to layer 4 because all primary connections have been exhausted and the connection priority rules are configured to activate secondary connections only when the total fresh reliable capacity drops below 10 Mbps. Connection E still reaches this threshold, it just does not currently have capacity.

[0854] Packet 4 has no more available connections for transmission, but it has not yet reached its target score of 3. The system will wait until an event occurs that may allow additional transmissions to occur. For example, the acknowledgment of an in-flight packet on connection E will increase its available capacity; the failure of connection E will meet the threshold for activating a secondary connection; the measured characteristics of any of the other primary connections are updated so that they are no longer out-of-date and / or unreliable.

[0855] Assume that an event occurs that causes the capacity of connection E to drop to 8 Mbps, which also reduces its capacity factor:

[0856]

[0857] The layers remain unchanged and the iteration steps repeat from the beginning again, but this time instead of stopping at layer 3, the iteration proceeds to layer 4 because the total available primary capacity has dropped below the configured threshold of 10 Mbps (only 8 Mbps is available on connection E).

[0858] Only layer 1 has sticky connection preference, so assuming pseudo-random connection selection, connection D is selected first from layer 4. The transfer occurs, but since it is stale, its contribution to the target score is 0. Iteration proceeds within the layer to connection C. It is fresh, so its contribution to the packet score is 3. The target is reached, and the transfer of packet 4 is completed.

[0859] It is assumed that there are no more packets available, so scheduling and transmission are suspended until an event occurs (eg, a new packet is received at the input, a negative acknowledgment is received for a transmitted packet, etc.).

[0860] Also consider that given all the outdated packet transmissions described above, there may be many duplicate and out-of-order packets. In some embodiments, reordering and de-duplication may occur, and the endpoint may receive packets in sequence.

[0861] According to one embodiment, implementation of a per-stream communication system may follow such a process as outlined above, but in other embodiments, more or fewer steps may be included, and possibly performed in a different order.

[0862] Fig. 22 is a block diagram illustrating a workflow 2200 that may occur in accordance with some example embodiments. Fig.21 2121 and 2122, especially for real-time streams. These streams can trigger early retransmission of the packet when the deadline of the packet is about to arrive, but it has not yet been confirmed.

[0863] According to one embodiment, the exemplary workflow 2200 may follow such a process as outlined below, but in other embodiments, more or fewer steps may be included and may be performed in a different order.

[0864] In the exemplary workflow 2200 shown, starting at 2210, the transmitted packet (which occurs at Fig.21 2121), check whether this is the first transmission of the packet. If not, it means that the packet has been transmitted previously on a better connection, so the early retransmission qualification check will have been completed. In this way, the packet returns to Fig.21 Of which 2122.

[0865] However, if this is the first transmission of the packet, a check occurs at 2211 whether there is enough time before the packet deadline to complete an early retransmission attempt. In some embodiments, this check consists of three parts:

[0866] The best-case duration of this first transmission and its corresponding ACK (this will typically be the connection RTT)

[0867] The worst case duration for which a retransmission attempt succeeds (this will typically be the connection EDT) after the best case duration has elapsed without receiving an ACK.

[0868] A predefined "overhead" value to account for system processing time (in some embodiments, this is a constant of 50 milliseconds).

[0869] If the sum of these three parts exceeds the current packet ATL, it is marked as not eligible for early retransmission at 2212. In both cases (eligible and ineligible), the packet proceeds to the return Fig.21 Step 2122 in .

[0870] As a non-limiting example implementation (which may follow the workflow as shown in 2200), consider Packet 1 from a real-time stream, with an ATL of 200 milliseconds, scheduled for its first transmission on Connection A, where the RTT is 25 milliseconds and the EDT is 100 milliseconds. At 2211, the sum of the three parts (RTT+EDT+50 milliseconds=25+100+50=175 milliseconds) is less than the ATL, so the packet qualifies for retransmission. The explanation for this comparison is that in a nominal scenario, Connection A is expected to deliver Packet 1 and the ACK within its RTT of 25 milliseconds. If 25 milliseconds have elapsed, and an ACK or NACK has not been received, the system may begin to suspect that the transmission has been lost, and consider an early retransmission in order to meet the deadline. In order to reserve enough time for this early retransmission to be effective, there must be an additional EDT+processing time overhead reserved before the Packet 1 deadline.

[0871] In this instance, there is sufficient time so that packet 1 qualifies for consideration for early retransmission and it proceeds back to step 2122.

[0872] Next, consider that package 1 enters the workflow again, because it needs multiple transmissions to reach its target score as described in workflow 2100. This time, at 2210, package 1 is not on its first transmission, so it proceeds directly back to 2122.

[0873] Next, consider packet 2 from the same real-time stream coming in at 2210, with an ATL of 150 milliseconds. Assume that it is also scheduled for its first transmission on connection A, with the same RTT and EDT parameters as packet 1. This time, the sum of the three parts (175 milliseconds) is greater than the ATL, so packet 2 is marked as ineligible for retransmission at 2212. It then proceeds back to the 2100 workflow at 2122.

[0874] According to one embodiment, implementation of a per-stream communication system may follow such a process as outlined above, but in other embodiments, more or fewer steps may be included, and possibly performed in a different order.

[0875] Fig.23 is a block diagram illustrating a workflow for considering early retransmission of in-flight packets according to some example embodiments. Specifically, Fig.23 is a block diagram illustrating a workflow 2300 for considering early retransmission of in-flight packets (those packets that have been transmitted and have reached their target score via workflows 2100 and 2200, but have not yet been acknowledged as received or lost).

[0876] According to one embodiment, the exemplary workflow 2300 may follow such a process outlined below, but in other embodiments, more or fewer steps may be included and may be performed in a different order.

[0877] In the illustrated exemplary workflow 2300, starting at 2310, an iteration of the in-flight packets belonging to the real-time stream occurs in ascending order of the packet deadline.

[0878] At 2311, packets with a deadline in the past are skipped, and at 2312, packets that have previously been retransmitted early are skipped. Any packet marked as ineligible for early retransmission as part of workflow 2200 is also skipped at 2312.

[0879] If the packet has not been skipped, its ATL is compared with the EDT of all available connections at 2320. For all fresh connections, a connection that has the best EDT less than the packet's ATL and is not the first connection initially used to transmit the packet is returned as a candidate for early retransmission at 2321. If no such candidate exists, a plurality of candidates consisting of all the stale connections plus the first connection initially used to transmit the packet (if it meets the EDT < ATL requirement) is returned at 2322.

[0880] At 2323, the candidate with the worst EDT is checked according to the packet deadline. If the deadline has become too close such that this candidate would need to transmit the packet now for it to reach the destination in time, the packet is scheduled for immediate early retransmission on all candidates at 2324. In this exemplary embodiment, the definition of "too close" is a predefined constant of 50 milliseconds, meaning that the current time plus the connection EDT is less than 50 milliseconds from the packet deadline.

[0881] Otherwise, the packet's deadline is not close enough to authorize early retransmission, and the system prefers to wait longer in case the initial transmission of the packet through workflow 2200 is successfully delivered.

[0882] The iteration continues to the next in-flight packet at 2313 (if one is available), otherwise the early retransmission workflow ends at 2314 and resumes later. In this exemplary embodiment, for the packet with the earliest deadline, workflow 2300 is triggered by a timer set according to the above "too close" deadline. The workflow is also always triggered immediately before workflow 2100.

[0883] As a non - limiting example implementation (which may follow the workflow shown in 2300), consider Packet 1, which is the only packet in - flight, where the future deadline is T = 1000 milliseconds and the current time is t = 100 milliseconds. Assume the following connections exist in the system, and the first connection on which Packet 1 was initially transmitted is C:

[0884] connect EDT Fresh / Stale A 150 ms Fresh B 300 ms Outdated C 50 ms Fresh D 195 ms Fresh

[0885] At 2310, Packet 1 is the only packet in - flight, so it is selected for iteration.

[0886] At 2311, it is determined that Packet 1 does not have a past deadline (current time is t = 100 milliseconds and the deadline is T = 1000 milliseconds).

[0887] At 2312, it is determined that the packet is eligible for early re - transmission (not previously marked ineligible by workflow 2200) and has not been re - transmitted early.

[0888] At 2320, the ATL is 900 milliseconds (T = 1000 milliseconds minus t = 100 milliseconds), so all four connections A, B, C, and D satisfy the (EDT < ATL) requirement. However, B does not satisfy the freshness requirement, and C is the first connection used to transmit Packet 1, so it is also excluded. This leaves connections A and D as the only eligible connections.

[0889] Connection A has the best EDT of the two, so it is returned as the only eligible candidate at 2321.

[0890] At 2323, connection A does not satisfy the "packet deadline approaching" requirement. Recall that in this exemplary embodiment, it is (now + EDT+50 milliseconds) >= packet deadline, or (t = 100 milliseconds + 150 milliseconds+50 milliseconds)=300 milliseconds, which is not >= T = 1000 milliseconds.

[0891] Therefore, Packet 1 is not re - transmitted early. At 2313, more packets are iterated over, but there are no remaining packets, so the early re - transmission workflow terminates at 2314.

[0892] In this exemplary embodiment, the timer is set to restart workflow 2300 at t = 800 milliseconds. If Packet 1 is still in - flight at that time, this is considered the time when the deadline for Packet 1 is considered close enough to require early re - transmission on connection A.

[0893] Assume this timer starts, Packet 1 is still in - flight, but the connection characteristics have now changed such that connection A is obsolete:

[0894]

[0895]

[0896] The previous steps 2310, 2311, and 2312 remain unchanged.

[0897] At 2320, the ATL is 200 milliseconds (T = 1000 milliseconds minus t = 800 milliseconds), so only connecting A, C, and D satisfies the (EDT < ATL) requirement. However, A no longer meets the freshness requirement, and C was the first connection initially used to transmit Packet 1, so it is also excluded. This leaves connection D as the only eligible connection, so it is returned as a candidate at 2321.

[0898] At 2323, connection D satisfies the "packet deadline approaching" requirement. Recall that in this exemplary embodiment, it is (now + EDT + 50 milliseconds) >= packet deadline, or (t = 800 milliseconds + 195 milliseconds + 50 milliseconds) = 1045 milliseconds, which is >= T = 1000 milliseconds.

[0899] As a result, at 2324, Packet 1 is retransmitted early on connection D.

[0900] The next iteration of the packet occurs at 2313. Assume there are no other in - flight packets, so the early retransmission workflow terminates at 2314.

[0901] Assume there is now an in - flight Packet 2 with a deadline of T = 2000 milliseconds, initially transmitted on connection C, and the current time is t = 1900 milliseconds. Assume the connection characteristics have not changed since Packet 1:

[0902] connect EDT Fresh / Stale A 150 ms Outdated B 300 ms Outdated C 50 ms Fresh D 195 ms Fresh

[0903] Assume Packet 2 passes the checks at levels 2311 and 2312 and thus moves to 2320. The ATL is 100 milliseconds (T = 2000 milliseconds minus t = 1900 milliseconds). Only connection C satisfies the EDT < ATL requirement, however it is also the first connection used to transmit Packet 2.

[0904] As a result, 2320 has no option but to proceed to 2322, where it returns all the outdated connections plus the first connection in the initial transmission. This means that all of connections A, B, and C are returned.

[0905] At 2323, the connection with the worst EDT is B, and it meets the "Packet Deadline Close" requirement. Recall that in this exemplary embodiment, it is (now + EDT + 50 ms) >= Packet Deadline, or (t = 1900 ms + 300 ms + 50 ms) = 2250 ms, which is >= T = 2000 ms.

[0906] As a result, packet 2 is retransmitted in advance on all candidates, i.e., connections A, B, and C. At 2313, the next packet iteration occurs. Assuming there are no other packets in flight, the early retransmission workflow is terminated at 2314.

[0907] According to one embodiment, implementation of a per-stream communication system may follow such a process as outlined above, but in other embodiments, more or fewer steps may be included, and possibly performed in a different order.

[0908] Fig.24 is an illustration of a per-stream communication system implementation 2400 according to some example embodiments.

[0909] During an emergency, first responders should not need to worry about the connection of their communications, nor should they be IT experts. First responders and public safety agencies rely on advanced technology to improve situation awareness and response times. Transmitting real-time video and data from the scene requires a lot of bandwidth, and relying on a single connection can make the organization vulnerable.

[0910] Emergency situations may occur in a variety of environments, including those where network conditions are unreliable but a reliable connection is still needed. This connection needs to provide appropriate connection security, diversity, redundancy, latency, and bandwidth required for communications such as real-time video and data. These communications typically have latency / jitter requirements, and with current technology, available WAN connections alone cannot always meet real-time requirements. The reason for this failure may be due to connection overload (contention / competition for connection capacity), or due to connection failure / inconsistency (e.g., fiber cut, wireless interference, equipment failure). Connection contention can be anywhere along the path (first mile / middle mile / last mile), anywhere where demand exceeds the capacity of a particular network link.

[0911] The above-described embodiments use multiple WAN connections to be able to meet these real-time requirements despite the unreliability of individual WANs.

[0912] Consider the exemplary implementation 2400 of the embodiment, such as Fig.24In this example, a police officer 2404 can be seen through body camera 2402. For example, police officer 2402 may be conducting a pursuit in a spectrally challenging environment such as an apartment building, and body camera 2404 may be relaying real-time live video and audio of the situation to a 911 dispatch center.

[0913] As seen in 2400, its connection to network 2414 may have been overloaded, or it may have experienced a failure of the primary connection 2406. According to one of the above embodiments, router 2408 is a multi-LTE WAN router and has multiple connections 2410. Router 2408 may be able to maintain reliable communication with the dispatch center through multiple other connections. Data packets for video and audio communications can be transmitted according to the processes described in the above embodiments, and latency and jitter requirements can be ensured to be met so that the video feed is clear to the dispatch center (including, for example, dispatch worker 2412).

[0914] Similarly, since connection failures may occur with any type of connection, the dispatch center itself may be connected to the network using a multi-WAN router 2408 that supports multiple wired and wireless broadband connections 2410 (e.g., fiber, cable, DSL, LEO satellite, LTE, etc.). According to the above embodiments, connection priority rules may be used to set unmetered connections as primary (e.g., fiber, cable, DSL, LEO satellite) and metered, more expensive connections (e.g., LTE) as secondary. Data packets for video and audio communications may be transmitted according to the processes described in the above embodiments, and latency and jitter requirements may be ensured to be met even if congestion or failures are experienced simultaneously on multiple WAN connections at the dispatch center. This typically occurs at dispatch centers in areas with limited unmetered connection options, such as rural areas that only have access to low-quality DSL lines and oversubscribed LEO satellite services.

[0915] Other public safety-related use cases may include mobile command, where reliable connectivity is needed to enable a connected command vehicle to send and receive data from cameras, sensors, and edge devices, for example. First responders must be able to quickly and reliably get the information they need to make life-saving decisions.

[0916] As public safety agencies around the world increase their use of real-time video, the amount of video generated and consumed by law enforcement agencies is also increasing rapidly. The described embodiments present a solution to this increased need while requiring high standards of reliability, thereby implementing multiple different connections for sending data packets, a process for sorting / tiering these connections, and sending packets to ensure that the packets arrive in a timely manner while adhering to strict requirements.

[0917] The embodiments described may provide for the use of fleet connectivity, where real-time video can be sent back to a command center and enable a "mobile precinct" to help make appropriate response decisions, or retrieve important data from the scene to keep the team safe and enable faster responses. For example, a mobile camera can capture a license plate and send it to the edge for immediate analysis to alert the police whether action should be taken.

[0918] Real-time video applications, such as those described in example embodiment 2400, can also ensure that central command has real-time video to support decision making during sporting events, parades, protests, and other events that attract large, mobile crowds.

[0919] The described embodiments may enable unmanned aerial systems, or drones, to provide real-time video from the safety of the skies to improve situational awareness for law enforcement officers and fire commanders.

[0920] Fig.25 is an illustration of a per-stream communication system implementation 2500 according to some example embodiments.

[0921] As VoIP grows, broadcast needs, processes, and deliverables continue to evolve. For example, the embodiments may provide a single service for uninterrupted live broadcasts from remote locations. Some embodiments may be easy to use and integrate into broadcast workflows, and may be suitable for different environmental uses (e.g., vehicle-mounted, backpack-worn, integrated into drones, etc.).

[0922] Broadcast communications typically have latency / jitter requirements, and with current technology, available WAN connections themselves are not always able to meet real-time requirements. The reason they cannot meet the requirements may be due to connection overload (contention / competition for connection capacity), or due to connection failure / inconsistency (e.g., fiber cut, wireless interference, equipment failure). Connection contention can be anywhere along the path (first mile / middle mile / last mile), anywhere where demand exceeds the capacity of a particular network link.

[0923] The above-described embodiments use multiple WAN connections to be able to meet these real-time requirements despite the unreliability of individual WAN connections.

[0924] Consider the Fig.25 Reporter 2502 may be covering breaking news of a live event, where reporter 2502 is filming with broadcast camera 2504 at a remote broadcast location, and the primary connection 2508 becomes very slow due to overload and contention because of the large crowd at the live event all using their cell phones 2506. A slow connection is obviously not conducive to good broadcast production.

[0925] Reporter 2502 may need to transmit live video and make a VoIP call with the main broadcast anchor to receive his video and audio and talking points at the remote location via a teleprompter. Low latency and jitter are required to ensure that the remote team and reporter 2502 stay in sync with the central production department 2514.

[0926] Router 2510 may allow this low latency return video and teleprompt feed to be sent to the live production team on site and central production and may allow them to simultaneously view these feeds on tablets and mobile devices. According to one of the above embodiments, router 2510 is a multi-LTE WAN router.

[0927] The router 2510 may be able to maintain reliable communication with the central production department through multiple other connections 2512. The data packets of the video communication can be transmitted according to the process described in the above embodiment, and the delay and jitter requirements can be ensured to be met so that the video feed is clear for broadcast.

[0928] The router 2510 can provide a reliable connection to the network 2516, which can provide high throughput wireless Internet for the remote team with reporters 2502, which can improve their production efficiency by providing access to cloud services, online resources, media assets, and newsroom systems at any location. The router 2510 can also be a portable and / or vehicle-mounted device. The embodiments can provide a remote connection, where media assets and newsroom systems can be accessed remotely, files can be transferred back and forth quickly, and communication with field personnel can be more convenient.

[0929] Fig.26 is an illustration of a per-stream communication system implementation 2600 according to some example embodiments. Fig.26 Another broadcast implementation example is shown.

[0930] Storms, floods and other extreme weather outbreaks may interrupt broadcast operations, and damage or cuts to fiber optic cables may take days or weeks to locate and repair. As shown in 2600, the main connection 2602 has been damaged and cut due to severe weather events and is no longer able to transmit data.

[0931] With multiple LTE WAN connections 2606, router 2604 can provide a portable and permanent backup connectivity solution. As shown in 2600, this can allow broadcast 2608 to remain on the air and keep remote teams connected to the network 2610 during emergency weather events. In such cases, for public safety, in the event of a large natural disaster such as an earthquake, it may be very important that the news broadcast remains active to provide the public with accurate and up-to-date live information. For example, in the event of damage to the broadcast facility or studio 2612, the embodiments can keep the broadcast 2608 on the air and maintain content distribution capabilities.

[0932] Other example broadcast implementations and use cases of the described embodiments may be for remote contributions, such as Fig.25 As shown and described above, embodiments may also assist in remote production situations, where the production activity itself occurs at a remote location.

[0933] For example, the described embodiments may also allow for the distribution of content to network affiliates, group television stations, or other broadcasters and media organizations in a cost-effective manner.

[0934] Fig. 27 is a schematic diagram of a computing device 2700 that can be used to implement the systems 1200A and / or 1200B according to one embodiment.

[0935] As shown, computing device 2700 includes at least one processor 2702 , memory 2704 , at least one I / O interface 2706 , and at least one network interface 2708 .

[0936] Each processor 2702 can be, for example, a microprocessor or microcontroller (e.g., a special purpose microprocessor or microcontroller), a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or any combination thereof.

[0937] Memory 2704 may include various types of computer memory located internally or externally, such as a suitable combination of random access memory (RAM), read-only memory (ROM), compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), ferroelectric RAM (FRAM), etc.

[0938] Each I / O interface 2706 enables the computing device 2700 to be interconnected with one or more input devices such as a keyboard, a mouse, a camera, a touch screen, and a microphone, or with one or more output devices such as a display screen and a speaker.

[0939] Each network interface 2708 enables the computing device 2700 to communicate with other components, exchange data with other components, access and connect to network resources, service applications, and execute other computing applications by connecting to a network (or multiple networks) capable of carrying data, including the Internet, Ethernet, plain old telephone service (POTS) lines, public switched telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, optical fiber, satellite, mobile network, wireless network (e.g., Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network, wide area network, etc., including various combinations of these networks.

[0940] For simplicity only, one computing device 2700 is shown, but the system 1200A and / or 1200B may include multiple computing devices 2700. The computing devices 2700 may be the same or different types of devices. The computing devices 2700 may be connected in various ways, including directly coupled, indirectly coupled through a network, and distributed over a wide geographic area and connected through a network (which may be referred to as "cloud computing").

[0941] For example and without limitation, computing device 2700 may be a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop computer, a personal data assistant, a cellular telephone, a smart phone device, a UMPC tablet computer, a video display terminal, a game console, or various other computing devices that can be configured to perform the methods described herein.

[0942] Fig.28 is a diagram 2800 depicting a physical computer server rack according to some example embodiments. In the computer server rack shown in 2800, there are multiple networked computing components, including rack-mounted computing devices, such as networking routers, switches, hubs, gateways, etc. In this example, the components interoperate with each other to control routing, such as by establishing and / or periodically updating routing tables and / or routing rules. In another embodiment, the components interoperate according to routing tables and / or routing rules to control routing by controlling network connections, routing data packets, etc.

[0943] As described herein, the method is a technology, a computing solution that can be implemented in the form of a physical data router or other networking device configured to control the communication routing of data packets. The device may include one or more processors operating in conjunction with a computer memory, and the one or more processors may be coupled to a data storage device. Corresponding methods, and non-transitory computer-readable media (e.g., floppy disks, solid-state storage devices, hard drives) having machine-interpretable instructions stored thereon (e.g., software that, when executed by a computer processor, causes the processor to perform the methods described herein in various embodiments) are described.

[0944] Fig.29 is a diagram depicting practical variations using a computing device with reduced computing power, according to some example embodiments.

[0945] exist Fig.29 2900 shows a variant embodiment of the per-stream system described herein. In this variant, the system is adapted to use a computing device 2904 (e.g., a Raspberry Pi 2904) with reduced computing power at one or the other side of the communication. TM Device or Arduino TM device) to operate.

[0946] In this example embodiment, computationally expensive or computationally resource-intensive operations (e.g., processing time, speed, power, memory / storage requirements) are performed on the side with more available resources, while the other side is assigned to perform basic operations that require only the minimum computational resources.

[0947] This is useful when practical situations require asymmetric levels of investment and computing power, due to, for example, the availability of computing devices in a particular area or the difficulty of transporting expensive and heavy equipment to such areas (e.g., emergency areas, remote areas, correspondents in areas with poor transportation infrastructure).

[0948] Thus, asymmetric computing scenarios are possible and contemplated, and the system may be adapted to operate in these types of configurations as well.

[0949] Specifically, the side with more resources can perform computationally expensive operations and communicate the results over the air to the other side, such as network models for each connection (e.g., capacity, cwnd, loss rate, RTO, all of which require data collection over time, mathematically demanding statistics and filtering operations).

[0950] A negotiation protocol may optionally be implemented where both sides can communicate their capabilities and jointly agree on who will compute what and communicate it to the other side.

[0951] Specifically, each side may be associated with a data structure advertising corresponding capabilities, and a scoring / ranking algorithm may be utilized to perform an automated mutual agreement process.

[0952] In another variation, a discovery protocol is utilized where each side understands what the other side is capable of, and the transmission of packets is managed so that even if the network allows high bit rates, the other side is not computationally overwhelmed in terms of packet processing. In practice, the discovery protocol can be implemented using a data process running on both, e.g., asynchronously or synchronously, to maintain an up-to-date list of available computing power. This can be useful in situations where devices draw resources from a shared pool of capacity, such as in a distributed computing system where resources can be dynamically allocated and dispatched.

[0953] Therefore, if Fig.29 As shown, the dynamic allocation of computing tasks for network routing can be shared between two devices, so that the device with more computing power can be used to control routing and send routing instructions to the other device in the form of data messages and packets. Specifically, the device with more computing power can be used to manage the layer structure and establish priority routing, while maintaining and / or sensing the priority of various network connections. The device with more computing power can manage the comprehensive characteristics representing the estimated delivery time and apply various scheduling methods to solve the problems that arise when sorting connections by characteristics.

[0954] Specifically, the layered approach allows functional equivalence to be established based on various connections defined as belonging to a specific layer, such as a specific approach on a per-flow basis (e.g., a different approach for a flow with a throughput preference versus a real-time flow that may have a latency / jitter preference). This distinction can be used to control the retransmission approach / reception probability approach at a finer granularity, such that, for example, a preemptive retransmission approach is only applicable to real-time flows.

[0955] Thus, the more powerful device can manage and track the ATL value as a dynamic value that changes on a per-packet basis and over time (e.g., as the packet ages). The more powerful device then identifies the connection to schedule the packet for delivery and sends routing instructions accordingly, and additionally, connection priority rules and logic flows can be utilized to establish granular control methods or can be used to influence how layers are constructed and / or how layers are maintained.

[0956] During the scheduling method, the device with greater computing power performs score assignment, scheduling, and controls the communication of any device across the various network connections.

[0957] In addition, in some embodiments, the occurrence of early retransmission of packets can be controlled by a device with stronger computing power.In some example cases, when sending on all outdated connections and / or original connections, each packet deadline can be maintained and early retransmission can be controlled.

[0958] By controlling which device performs which activities, multiple devices with reduced computing power can be considered, and the load can be shared or completely transferred to one device (e.g., a device with more resources). This can also be used to manage other types of computing characteristics, such as device wear / lifetime usage management, thermal stages, etc. Therefore, from the perspective of computing load balancing, asymmetric computing can be a useful variation to solve technical problems caused by differences in computing characteristics or differences in computing environments. Although the examples described focus on completely transferring the load to a device with more computing power, this does not necessarily need to happen. Instead, the load of a particular activity can be balanced between two devices.

[0959] The terms “connected” or “coupled to” may include direct coupling (where two elements coupled to each other are in contact with each other) and indirect coupling (where at least one additional element is positioned between the two elements).

[0960] Although the embodiments have been described in detail, it should be understood that various changes, substitutions and modifications may be made herein without departing from the scope. In addition, the scope of the present application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, components, methods and steps described in this specification.

[0961] As those skilled in the art will readily appreciate from this disclosure, currently existing or later to be developed processes, machines, manufactures, compositions of matter, components, methods, or steps may be utilized that perform substantially the same functions as those in the corresponding embodiments described herein or achieve substantially the same results as those in the corresponding embodiments described herein. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufactures, compositions of matter, components, methods, or steps.

[0962] As can be appreciated, the examples described and illustrated above are intended to be exemplary only.

Claims

1. An apparatus for coordinating data communications across multiple wide area network connections, the apparatus comprising: a processor coupled to a computer memory and a non-transitory computer-readable storage medium, the processor configured to: For each of the plurality of WAN connections, monitoring at least one of a delay, a packet loss rate, and a throughput characteristic of packets transmitted on the WAN connection; identifying, for each packet of the plurality of packets being routed having a latency / jitter preference, an adjusted target latency (ATL) based at least on a per-packet deadline for delivering the packet to a target endpoint; grouping using the per-packet ATL and the plurality of WAN connection characteristics to establish a plurality of layered packets such that each WAN connection in the plurality of WAN connections is grouped into a corresponding layered packet in the plurality of layered packets; and The packet is transmitted using one or more selected wide area network connections selected from the plurality of wide area network connections, the one or more selected wide area network connections being selected using at least the plurality of hierarchical packets.

2. The apparatus of claim 1, wherein the WAN connection latency and packet loss characteristics include an estimated delivery time (EDT) generated based on the following relationship: in: P drop is defined as the probability of packet loss; T prop is defined as the one-way propagation delay; and T rtprop is defined as the round trip propagation delay.

3. The apparatus of claim 1, wherein the wide area network connections belonging to each of the plurality of stratified groups are considered to be functionally equivalent to select the one or more selected wide area network connections.

4. The apparatus according to claim 1, wherein the plurality of layered packets include at least a first layer (layer 1), a second layer (layer 2), and a third layer (layer 3), wherein the layers are based on the ATL of each packet and the plurality of WAN connection characteristics EDT and T prop The relationship between is defined as: Layer 1: T prop ≤ATL&&EDT≤ATL Layer 2: T prop ≤ATL&&EDT>ATL Layer 3: T prop >ATL&&EDT>ATL.

5. The apparatus of claim 4, wherein the plurality of stratified packets include additional layers that subdivide layer 1, layer 2, and layer 3 based on one or more predefined connection priority rules, such that the result is a new set of layers from layer 1 to layer N. 6 . The apparatus of claim 4 , wherein selecting the one or more selected WAN connections among available WAN connections iterates from the available WAN connections in layer 1 , and then layer 2 , and finally layer 3 .

7. The apparatus of claim 6, wherein said transmitting said packet using said one or more selected wide area network connections selected from said plurality of wide area network connections comprises scheduling said packet on a plurality of connections selected from said plurality of wide area network connections for preemptive retransmission.

8. The apparatus of claim 7, wherein the wide area network connections in each tier are assigned a tier-based score, and scheduling the packets over the plurality of connections is performed to achieve an aggregate score greater than or equal to a predefined target score.

9. The apparatus of claim 5, wherein connection priority rules are ignored if scheduling the packet over multiple connections on a higher layer fails to achieve the cumulative score that is greater than or equal to a predefined target score.

10. The apparatus of claim 8, wherein connections designated as unreliable or having no recent monitoring characteristics are assigned a tier-based score of 0, regardless of their corresponding tier level.

11. The apparatus of claim 6, wherein selecting the selected wide area network connection among the available wide area network connections in the stratified grouping comprises selecting an available wide area network connection in a pseudo-random or algorithmic manner.

12. The apparatus of claim 11, wherein the initial selection of the selected wide area network connection for a data flow corresponding to a packet comprises an initial pseudo-random selection, and the selection of the selected wide area network connection is maintained for future packets corresponding to the data flow.

13. The apparatus of claim 12, wherein said selection of said selected wide area network connection is maintained only until a connection dormancy period threshold has been reached or an explicit stream termination notification has occurred, after which further packets on said data stream utilize further initial pseudo-random selections.

14. The apparatus of claim 1, wherein the processor is further configured to: Data flows are classified into (i) throughput-preferred flows, and (ii) real-time flows; and only packets from the real-time flows are identified as delay / jitter-preferred packets.

15. The apparatus of claim 14, wherein for packets from the throughput preferred flow, grouped layers are established based on a same composite characteristic pair based at least on one or more measured characteristics of the plurality of wide area network connections.

16. The apparatus of claim 1, wherein the processor is further configured to: tracking a per-packet deadline for packets sent over the first wide area network connection, the packets approaching the deadline but not yet acknowledged; Using a congestion window (CWND) that is available and T is short enough to deliver the packets that are approaching the deadline but have not yet been acknowledged to the corresponding target endpoint. prop The second wide area network connection with the backlog value performs early retransmission of the packets that are approaching the deadline but have not yet been acknowledged.

17. The apparatus of claim 1, wherein the processor is further configured to: tracking a per-packet deadline for packets sent over the first wide area network connection, the packets approaching the deadline but not yet acknowledged; Determine that there is no available congestion window (CWND) and T short enough to transmit the packet. prop and an available non-stale second WAN connection with a backlog value; and In response to the determination, early retransmission of the packets that are approaching the deadline but have not yet been acknowledged is performed using one or more outdated wide area network connections.

18. The apparatus of claim 17, wherein the processor is further configured to: Preemptive retransmission of the packet is performed over the first wide area network connection originally used to transmit the packet.

19. The apparatus of claim 10, wherein for packets associated with a throughput preferred flow, each tier is assigned the same tier-based score that matches the predefined target score.

20. The apparatus of claim 1, wherein the processor is further configured to: Prior to transmitting the packets using the selected wide area network connection, priorities defining a queue of packets for the flow are established between packets within the flow.

21. An apparatus according to claim 20, wherein the packet queues are unloaded according to a multi-level deficit weighted round-robin (DWRR) scheduling method, wherein the queues with higher importance levels are able to starve the queues with lower importance levels, thereby preventing the queues with lower importance levels from being served, regardless of the weights assigned to the queues with lower importance levels.

22. The apparatus of claim 1, wherein the ATL is a dynamic value that changes per packet.

23. The apparatus of claim 22, wherein the ATL varies as the packet ages.

24. The apparatus of claim 23, wherein the ATL is defined as a packet deadline time value minus a current time value.

25. The apparatus of claim 24, wherein the current deadline time value of the packet is equal to the original reception time (T PacketReceived ) plus the nominal target latency requirement of the flow.

26. The device of claim 1, wherein the device is a network router.

27. The device of claim 1, wherein the device is a physical chipset residing within a network router.

28. The apparatus of claim 26, wherein the network router resides within a data center and is configured to control the transmission of data packets transmitted through one or more coupled endpoint devices.

29. The apparatus of claim 28, wherein the data packets include both packets having a throughput preference and packets having a delay / jitter preference.

30. The apparatus of claim 28, wherein the network router controls communications for a media broadcast station or an emergency dispatch center.

31. A method for coordinating data communications across multiple wide area network connections, the method comprising: For each of the plurality of WAN connections, monitoring a delay and a packet loss rate characteristic of packets transmitted on the WAN connection; identifying, for each packet of the plurality of packets being routed having a latency / jitter preference, an adjusted target latency (ATL) based at least on a per-packet deadline for delivering the packet to a target endpoint; grouping using the per-packet ATL and the plurality of WAN connection characteristics to establish a plurality of layered packets such that each WAN connection in the plurality of WAN connections is grouped into a corresponding layered packet in the plurality of layered packets; as well as The packet is transmitted using one or more selected wide area network connections selected from the plurality of wide area network connections, the one or more selected wide area network connections being selected using at least the plurality of hierarchical packets.

32. The method of claim 31 , wherein the wide area network connection latency and packet loss characteristics include an estimated delivery time (EDT) generated based on the following relationship: in: P drop is defined as the probability of packet loss; T prop is defined as the one-way propagation delay; and T rtprop is defined as the round trip propagation delay.

33. The method of claim 31, wherein the wide area network connections belonging to each stratified grouping of the plurality of stratified groups are considered functionally equivalent to select the wide area network connection for communication.

34. The method of claim 31, wherein the plurality of layered packets include at least a first layer (layer 1), a second layer (layer 2), and a third layer (layer 3), the layers being based on the ATL of each packet and the plurality of WAN connection characteristics EDT and TDT. prop The relationship between is defined as: Layer 1: T prop ≤ATL&&EDT≤ATL Layer 2: T prop ≤ATL&&EDT>ATL Layer 3: T proo >ATL&&EDT>ATL.

35. The method of claim 34, wherein the plurality of stratified groups include additional layers that subdivide layer 1, layer 2, and layer 3 based on one or more predefined connection priority rules, such that the result is a new set of layers from layer 1 to layer N.

36. The method of claim 34, wherein selecting the one or more selected WAN connections iterates from the available WAN connections in layer 1, and then layer 2, and finally layer 3.

37. The method of claim 36, wherein said transmitting said packet using said one or more selected wide area network connections selected from said plurality of wide area network connections comprises scheduling said packet for preemptive retransmission on a plurality of connections selected from said plurality of wide area network connections.

38. The method of claim 37, wherein the wide area network connections in each tier are assigned a tier-based score, and scheduling the packets over the plurality of connections is performed to achieve an aggregate score greater than or equal to a predefined target score.

39. The method of claim 35, wherein connection priority rules are ignored if scheduling the packet over multiple connections on a higher layer fails to achieve the cumulative score greater than or equal to a predefined target score.

40. The method of claim 38, wherein connections designated as unreliable or having no recent monitoring characteristics are assigned a tier-based score of 0, regardless of their corresponding tier level.

41. The method of claim 36, wherein selecting the one or more selected wide area network connections among the available wide area network connections in the stratified grouping comprises selecting the available wide area network connections in a pseudo-random manner.

42. The method of claim 41, wherein the initial selection of the selected wide area network connection for a data flow corresponding to a packet comprises an initial pseudo-random selection, and the selection of the selected wide area network connection is maintained for future packets corresponding to the data flow.

43. A method according to claim 42, wherein the selection of the one or more selected wide area network connections is maintained only until a connection dormancy period threshold has been reached or an explicit stream termination notification has occurred, after which further packets on the data stream utilize further initial pseudo-random selections.

44. The method of claim 31, comprising: Data flows are classified into (i) throughput-preferred flows, and (ii) real-time flows; and only packets from the real-time flows are identified as delay / jitter-preferred packets.

45. The method of claim 44, wherein for packets from the throughput preferred flow, grouped tiers are established based on a same composite characteristic pair based at least on one or more measured characteristics of the plurality of wide area network connections.

46. ​​The method of claim 31, comprising: tracking a per-packet deadline for packets sent over the first wide area network connection, the packets approaching the deadline but not yet acknowledged; Using a congestion window (CWND) that is available and T is short enough to deliver the packets that are approaching the deadline but have not yet been acknowledged to the corresponding target endpoint. prop The second wide area network connection with the backlog value performs early retransmission of the packets that are approaching the deadline but have not yet been acknowledged.

47. The method of claim 31, comprising: tracking a per-packet deadline for packets sent over the first wide area network connection, the packets approaching the deadline but not yet acknowledged; Determine that there is no available congestion window (CWND) and T short enough to transmit the packet. prop and an available non-stale second WAN connection with a backlog value; and In response to the determination, early retransmission of the packets that are approaching the deadline but have not yet been acknowledged is performed using one or more outdated wide area network connections.

48. The method of claim 47, comprising: Preemptive retransmission of the packet is performed over the first wide area network connection originally used to transmit the packet.

49. The method of claim 40, wherein for packets associated with a throughput preferred flow, each tier is assigned the same tier-based score that matches the predefined target score.

50. The method of claim 41, comprising: Prior to transmitting the packets using the selected wide area network connection, priorities defining a queue of packets for the flow are established between packets within the flow.

51. A method according to claim 50, wherein the packet queues are unloaded according to a multi-level deficit weighted round robin (DWRR) scheduling method, wherein the queues with higher importance levels are able to starve the queues with lower importance levels, thereby preventing the queues with lower importance levels from being served, regardless of the weight assigned to the queues with lower importance levels.

52. The method of claim 31, wherein the ATL is a dynamic value that changes per packet.

53. The method of claim 52, wherein the ATL changes as the packet ages.

54. The method of claim 53, wherein the ATL is defined as a packet deadline time value minus a current time value.

55. The method of claim 54, wherein the current deadline time value of the packet is equal to the original reception time (T PacketReceived ) plus the nominal target latency requirement of the flow.

56. The method of claim 31, wherein the method is performed on a network router.

57. The method of claim 31, wherein the method is performed by a physical chipset residing within a network router.

58. The method of claim 56, wherein the network router resides within a data center and is configured to control the transmission of data packets transmitted via one or more coupled endpoint methods.

59. The method of claim 58, wherein the data packets include both packets having a throughput preference and packets having a delay / jitter preference.

60. The method of claim 58, wherein the network router controls communications for a media broadcast station or an emergency dispatch center.

61. A non-transitory computer readable medium storing a set of machine interpretable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 31 to 60.

62. The apparatus of claim 6, wherein selecting the one or more selected WAN connections among the available WAN connections in the layered grouping comprises selecting the WAN connection with the shortest backlog, wherein the backlog is measured in time units and is defined as the number of bytes in flight divided by the connection bit rate.

63. The apparatus of claim 12, wherein the selection of the one or more selected wide area network connections is maintained only for a period of time permitted by administrative configuration.

64. The method of claim 36, wherein selecting the one or more selected WAN connections among the available WAN connections in the layered grouping comprises selecting the WAN connection with the shortest backlog, wherein the backlog is measured in time units and is defined as the number of bytes in flight divided by the connection bit rate.

65. The method of claim 42, wherein the selection of the one or more selected wide area network connections is maintained only for a period of time permitted by administrative configuration.

Images