System and method for communication using a mixed wide area network

Abstract

A device for coordinating data communication across wide area network (WAN) connections, including a processor coupled to a computer memory and a non-transitory computer-readable storage medium, wherein the processor monitors the latency property and packet loss property of packets transmitted over each WAN connection, and for each packet having a latency / jitter preference within a plurality of routed packets, identifies an adjusted target latency (ATL) based at least on a per-packet deadline for communication of the packet to a target endpoint, groups the per-packet ATL and a plurality of WAN connection properties to establish a plurality of hierarchical groupings such that each WAN connection is grouped into a corresponding hierarchical grouping of the plurality of hierarchical groupings, and communicates packets using one or more selected WAN connections selected using at least the plurality of hierarchical groupings.

Description

Technical Field

[0001] Cross-reference This application is a non-provisional application of U.S. Patent Application No. 63 / 350,654, filed on June 9, 2022, entitled "Systems and Methods for Communications Using Blended Wide Area Networks", and claims all priority rights. This application is incorporated by reference.

[0002] This application relates to the following applications, which are incorporated by reference in their entirety:

[0003] PCT Application No. PCT / CA2017 / 051584, filed on December 21, 2017, entitled "PACKET TRANSMISSION SYSTEM AND METHOD".

[0004] PCT Application No. PCT / CA2021 / 050732, filed on May 28, 2021, entitled "SYSTEMS AND METHODS FOR DATA TRANSMISSION ACROSS UNRELIABLE CONNECTIONS".

[0005] PCT Application No. PCT / CA2022 / 050077, filed on January 19, 2022, entitled "SYSTEMS AND METHODS FOR PUSH-BASED DATA COMMUNICATIONS".

[0006] Embodiments of the present disclosure relate to the field of electronic data packet communication. More specifically, embodiments relate to devices, systems, and methods for push-based data packet communication where multiple network connections (e.g., wide area network connections) are available.

[0007] Introduction Existing data packet communication solutions for the transmission of data packets via multiple available network connections rely on static routing logic to determine how to transmit data packets through the connections. For example, a system can employ a timer for each available connection and use each timer to determine when to pull a packet from an input queue for transmission over the connection associated with the timer. These solutions assume that these connections have similar operating characteristics and availability in exchange for simplicity of implementation, and schedule the next available data packet for transmission over the connection with the timer that most recently expired. The timer approach is simple but can be computationally inefficient in certain situations.

[0008] Communication systems (e.g., LACP, ML-PPP) that treat multiple network connections as multiple instances of connections with similar operating characteristics can lead to sub-optimal use of the multiple networks because a particular network may be under-utilized or over-utilized, and the multiple networks may have different operating characteristics. This imprecision in use can result in greater latency in data packet delivery, less reliable data packet delivery, more expensive data packet delivery, reduced throughput, etc.

[0009] Connection management solutions also encounter difficulties in handling applications where the available connections or the operating characteristics of these connections change over time. For example, a timer-based system may be insensitive to changes in network performance because the timers for each connection fire independently, and may not accurately view the network performance of all other connections.

[0010] Systems and methods for data communication that more efficiently utilize multiple connections to transmit data packets with lower latency times, increased throughput, and enable more reliable or less expensive data packet delivery are desirable.

SUMMARY OF THE INVENTION

[0011] It is technically difficult to provide a reliable communication mechanism that can meet different competing communication requirements by various applications operating on networked devices. The communication requirements from each application compete with each other for finite communication resources, and computational approaches are required to control the coordination and use of the limited communication resources. These issues are further compounded in multiple WAN scenarios where a control device such as a network router is coupled to multiple wide area network (WAN) connections and is configured to control how packets are communicated through one or more interfaces coupled to the multiple WAN connections.

[0012] A network router can be, for example, an electronic device having routing logic or circuitry, or an on-board processor, configured to control packet communication (e.g., packet delivery and / or packet reception) as required by various client applications.

[0013] A network router can be used in practical embodiments such as controlling the communication of mobile or fixed communication centers such as portable media tracks (e.g., for broadcasting) or data center facilities (e.g., emergency dispatch centers). For example, when connections are individually affected by communication failures (e.g., high demand periods following emergencies or denial-of-service attacks) or communication challenges (e.g., as the track travels around, the spectral characteristics of connecting to various cellular towers or satellites change), multiple WANs can be utilized by the network router in these situations to provide improved uptime and reliability. Since multiple WANs are used, there are different options and available paths for communication, and additional approaches are needed to sequence the sequences at the receiver's endpoint and / or potentially re-sequence / regenerate the sequences. For example, data packets corresponding to a specific media broadcast can be individually controlled for communication across multiple WANs, the WANs can be used intelligently to distribute the packets among them, and potentially also, as described herein, pre-emptively transmit redundant packets across the connections to improve overall reliability at the expense of efficiency. Exemplary uses of multiple WANs can include, among other things, routers having multiple SIM slots, multiple wired Ethernet connection ports, and / or multiple satellite connections.

[0014] Packet communications can be requested from different types of client applications, and it is important to note that these communications can have different types of communication requirements, namely, (i) requirements that emphasize throughput (e.g., bulk file transfer), or (ii) requirements that emphasize latency / jitter (e.g., real-time usage such as VoIP, video conferencing). There can be multiple types of client applications that request communications simultaneously or almost simultaneously (e.g., the media track of a sports event is running an application to upload files for post-production over several hours, sending a low-quality data feed for quick preview, and conducting a video conference for a sideline reporter), and these client communications are competing with each other for finite and limited resources.

[0015] As described herein, requirements that emphasize latency / jitter are particularly difficult to meet because their packet flows are tied to real-time delivery deadlines and require packets to arrive on time with minimal jitter / loss. If a packet does not arrive on time, or is accompanied by jitter or loss, for example, if the audio and video are not smooth, there may be, among other things, speech clipping, freezes, and pauses. In contrast, an application with a flow having a throughput-emphasizing requirement may be able to handle latency and jitter, for example, through the use of re-sequencing and buffering.

[0016] In the proposed approach, a device is proposed for coordinating data communications across multiple wide area network connections, and the device is configured as a packet routing device having a processor coupled to a computer memory and a non-transitory computer-readable storage medium.

[0017] For each of the multiple wide area network connections, the device monitors the latency property, packet loss property, and throughput property of the packets transmitted over the wide area network connection, and for each packet having a latency / jitter preference within the multiple routed packets, identifies an adjusted target latency (ATL) based at least on a per-packet deadline for communication of the packet to a target endpoint, and groups the per-packet ATL and the multiple wide area network connection properties to establish multiple hierarchical groupings such that each of the multiple wide area network connections is grouped into a corresponding hierarchical grouping of the multiple hierarchical groupings, and communicates the packets using one or more selected wide area network connections selected from the multiple wide area network connections, at least using the multiple hierarchical groupings.

[0018] In one aspect, the latency property and packet loss property of the wide area network connection include an estimated delivery time (EDT).

[0019] The wide area network connections belonging to each of the multiple hierarchical groupings can be considered functionally equivalent for the selection of the wide area network connection for communication, and the multiple hierarchical groupings can include at least a first layer (layer 1), a second layer (layer 2), and a third layer (layer 3) defined based on the relationship between the per-packet ATL and the multiple wide area network connection properties, EDT, and one-way propagation delay (T prop )). The number of layers is not limited to three, and in a variant, the multiple hierarchical groupings include additional layers that subdivide layers 1, 2, and 3 based on one or more predefined connection priority rules such that the result is a new set of layers.

[0020] The selection of a selected wide area network connection from among the available wide area network connections can be iterated from, for example, the available wide area network connections in layer 1, then layer 2, and finally layer 3, and packet communication can use the selected wide area network connection selected from among the plurality of wide area network connections. For example, packet scheduling can be performed on a plurality of connections selected from among the plurality of wide area network connections for preemptive retransmission.

[0021] Each of the wide area network connections in each layer can be assigned a layer-based score, and packet scheduling on a plurality of connections is performed to achieve a cumulative score above a predefined target score. In a variant, if packet scheduling on a plurality of connections in a better (e.g., alternative, higher, lower) layer cannot achieve a cumulative score above the predefined target score, the connection priority rules are ignored. In some embodiments, a useful bypass mechanism is provided. Connections designated as unreliable or having no recently monitored properties can be assigned a layer-based score of 0 regardless of their corresponding layer level to encourage the use of more reliable and / or recent properties. Such connections are still used for packet transmission to measure their updated monitored properties to determine whether they have become reliable again.

[0022] The selection of a selected wide area network connection from among the available wide area network connections in hierarchical grouping may include selecting the available wide area network connections pseudo-randomly, such that the initial selection of the selected wide area network connection for a data flow corresponding to a packet includes an initial pseudo-random selection, and for future packets corresponding to the data flow, the selection of the selected wide area network connection is maintained, for example, until a connection downtime threshold is reached or until an explicit flow termination notification occurs. This helps the wide area network connection to be more "sticky" such that the system, if available, uses a consistent wide area network connection for a particular flow at a particular time to bias it. The technical benefit of having a forced level of consistency is to reduce the inefficiencies associated with the instantiation or initiation of different connections for use. For example, this can be used to avoid scenarios that cause inefficient data communication due to non-idealities introduced by the system continuously selecting between two or more different connections based on minor improvements and turning different connections on and off for use.

[0023] The device can segment and / or classify a data flow into (i) a throughput preference flow and (ii) a real-time flow, and only packets from the real-time flow are identified as packets having latency / jitter preferences, such that for packets from the throughput preference flow, the grouped layers are established based on the same composite property pair based on at least one or more measured properties of a plurality of wide area network connections.

[0024] In some embodiments, the selection of a selected wide area network connection from among the available wide area network connections in hierarchical grouping is not made using pseudo-random selection. Instead, it is selected based on the connection in hierarchical grouping that has the shortest backlog. The backlog is measured in time units (typically milliseconds) and is defined as the volume of in-flight (transmitted but unacknowledged) bytes divided by the transmission rate (throughput) of the wide area network connection. By selecting the connection with the shortest backlog, when a flow requires more bandwidth than any single connection can provide alone, the latency experienced by the flow is minimized.

[0025] In some embodiments, the device tracks the per-packet deadline of packets transmitted on a first wide area network connection that is approaching its deadline but has not been acknowledged, and communicates packets that are approaching their deadline but have not been acknowledged to the corresponding target endpoint in a T that is short enough. prop And, together with the backlog value, use a second wide area network connection having an available congestion window (CWND) to perform an early retransmission of packets that are approaching their deadline but have not been acknowledged, or a T that is short enough to communicate the packets. prop And, together with the backlog value, determine that there is no available non-old second wide area network connection having an available congestion window (CWND), and in response to the determination, use one or more old wide area network connections to perform an early retransmission of packets that are approaching their deadline but have not been acknowledged. Additionally, and / or simultaneously, for example, an early retransmission can be performed on the first wide area network connection used to initially transmit the packets.

[0026] In a further embodiment, an approach is proposed in which priorities are established among flows to which packets belong before communicating the packets using a selected wide area network connection. The priorities are translated into defining packet queues for each flow and unloading packets from the defined packet queues according to a number of modified deficit round robin (DRR) scheduling states.

[0027] Corresponding methods, and non-transitory computer-readable media are contemplated. As described in various embodiments herein, the system may be implemented as a networking controller that operates as a physical networking device for controlling communications across multiple network interfaces and / or network connections. The networking controller may be implemented, for example, as a specific hardware appliance, or integrated within another hardware device, or as an integrated set of chip sets or logic gates operating on a computer server or software.

[0028] The application relates to the control of network connections in situations where the connection is unreliable, where there are multiple connections to choose from but network resources are limited, and / or where there may be sudden load spikes or downtime. For example, the networking approach may be useful in relation to emergency dispatch control, media broadcasts (e.g., local or overcrowded areas).

[0029] This may also be useful when multiple demanding applications are being run together, or when there is a non-demanding application for a single request, but a failure or problem occurs with the connection itself.

[0030] Practical scenarios include, among others, sports broadcasts, live event broadcasts, public safety, high-usage hours / critical infrastructure. This approach helps improve the overall delivery and communication characteristics of data packets by leveraging combinations of connections, especially when the connections change over time (e.g., as congestion varies), as the device's location changes (e.g., as the device is mobile), and as the price and / or criteria for service delivery change (e.g., as competing best-effort type connections are shared among multiple users), which is beneficial when the changing networking characteristics are different.

[0031] In the figures, embodiments are illustrated by way of example. It should be explicitly understood that the specification and drawings are for illustrative purposes only and for the sole purpose of aiding understanding.

[0032] Here, embodiments are described by reference to the accompanying drawings only by way of example.

Brief Description of the Drawings

[0033]

Figure 1A

Figure 1B

Figure 1C

Figure 1D

Figure 1E

Figure 1F

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 5A

Figure 5B

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 7

Figure 8A

Figure 8B

Figure 8C

Figure 9A

Figure 9B

Figure 9C

Figure 10A

Figure 10B

Figure 11

Figure 12A

Figure 12B

Figure 13A

Figure 13B

Figure 14

Figure 15A

Figure 15B

Figure 15C

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

[0034] A system, method, and device for transmitting data packets are described, in which multiple connections for transmission are available and which is alternatively referred to as a hybrid router or a hybrid system. The described approach is extended as follows.

[0035] Existing solutions for transmitting data packets over multiple available connections for transmission are typically implemented as a "pull" type architecture, where data packets are transmitted in response to events generated or defined by a device or system, such as internal timers associated with the multiple available connections.

[0036] The internal timer can respond to various connection parameters (e.g., timer duration configured to reduce the recovery time of retransmitted packets), but not to the connection parameters of other available connections.

[0037] As a result of the independence of the timer events, pull-type systems can suffer from underutilization (e.g., when the timer value is too long, resulting in idle connections), or overutilization (e.g., data packets can be assigned to low-reliability connections despite insufficient performance), which can result in greater latency in data packet delivery, reduced throughput, low-reliability data packet delivery, more expensive data packet delivery, etc.

[0038] A method, system, and device for transmitting data packets according to a push-type architecture for transmitting data packets over multiple available connections are proposed herein. The disclosed method, system, and device acquire a subset of the operating characteristics (monitored characteristics) of each of the multiple connections in response to monitored communication events, and based on the monitored characteristics of these connections, include a scheduler configured to repeatedly allocate data packets available for transmission to any of the connections determined to have transmission capacity. In an exemplary embodiment, other monitored characteristics of the network can be used in conjunction with the transmission requirements associated with the data packets to assign the data packets to a connection that matches the transmission requirements of the data packets.

[0039] Scheduling data packets in a push-type architecture where multiple connections are available has not been used before. A typical multi-path system (alternatively referred to as a hybrid connection system) assigns packets and flows to connections on a static basis without considering the flow requirements or operating characteristics of the connections. For example, a multi-homed Internet Protocol (IP) router assigns packets to connections purely based on the destination IP address in the packet header that matches the router's destination routing table. If the connection assignment does not change frequently, it is easier to implement a pull-type architecture. For example, there is no need to handle the case where a large queue of packets has been assigned to a connection by a push scheduler but the connection has gone offline and needs to be recalled or re-assigned.

[0040] The proposed method, system, and device receive a data flow including one or more data packets from an input network through an input connection.

[0041] The proposed method, system, and device include a three-stage data pipeline for transmitting data packets via a set of hybrid connections to a remote peer. New data destined for multiple connections is passed to a first pipeline stage where it is queued and then passed to a second pipeline stage associated with each of the hybrid connections. The second pipeline stage aggregates statistics regarding the connections underlying the second pipeline stage and passes the packets to a third pipeline stage. The second stage may provide congestion control functionality and packet buffering to ensure that packets are available for the next pipeline stage. The third stage then writes the packets to a network interface, optionally according to pacing or timing hints provided by the second stage.

[0042] Another embodiment may have a single second-stage instance that then provides packets to instances of the third stage, each of the third stage managing packet transmission with a set of network interfaces that share a physical interface card. Other embodiments may exist with different numbers of instances for each of the pipeline stages.

[0043] FIG. 1A is a schematic diagram of a pull communication system 100A according to some exemplary embodiments.

[0044] When each connection timer (e.g., timers 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively) wakes up, the connection timer calls the pull scheduler 130 and requests data packets for transmission from the input queue 102 of data packets. The data packets can be requested in bursts, and the burst size nominally reflects the number of bytes equivalent to 20 ms (1 / 50 Hz) at the estimated bit rate of the corresponding connection of each timer. A limitation associated with such pull timers is that the number of bytes requested upon wake-up of each timer does not reflect the full convergence window (e.g., representing "capacity") of the connection.

[0045] Since each timer activates independently, the pull scheduler 130 does not have visibility into the overall transmission state in order to make globally optimal scheduling decisions. For example, connection 110A could be a terrestrial broadband connection and connection 110B could be a bandwidth on demand (BoD) satellite connection. Connection 110A may be preferable to connection 110B as a result of cost, latency, etc., but when connection 110B timer 108B wakes up and invokes the pull scheduler 130, the pull scheduler 130 does not have visibility into the next wake-up time of connection 110A timer 108A (since timer 108A activates independently). The pull scheduler 130 does not have a basis for determining whether connection 110A can process all of the data packets in input queue 102 or whether additional data packets should be allocated to connection 110B to complete transmission.

[0046] If connection 110B is useful, the pull scheduler 130 may not have means for determining which of the available data packets for transmission should be transmitted using which connection since only the pull scheduler 130 has visibility into the timers.

[0047] Data packets are transmitted across the connection and then received by receiver 116 via connection 114. Receiver 116 includes a sequencer 118 for assembling the transmitted data packets and may further transmit the assembled data packets via local area network (LAN) 120.

[0048] In contrast to pull architectures, the proposed system of one embodiment is adapted to schedule data packets for specific connections to increase efficient connection utilization. However, not all architectures contemplated need to be pull-based.

[0049] Figure 1B is a schematic diagram of a push communication system 100B according to some exemplary embodiments. In the exemplary embodiments, system 100B is designed to aggregate connection types that provide multiple access (MA) to a shared communication channel using multiplexing techniques. For example, orthogonal codes assigned to each node in a code division multiple access (CDMA) network enable those nodes to share a set of frequencies and transmit simultaneously without interfering with each other.

[0050] The properties of connection types using these multiplexing techniques are such that the reallocation of available capacity to nodes in the network is rapid, typically on the order of tens to hundreds of milliseconds. In the exemplary embodiments, system 100B is designed with this property in mind and reallocates traffic among available WAN connections on this time scale.

[0051] In contrast, connections that use demand-assigned multiple access (DAMA) techniques, also known as bandwidth on demand (BoD), share the communication channel by having nodes negotiate access to the channel for a period of time. Once allocated, the node uses the negotiated portion exclusively. Thus, sharing of available capacity occurs sequentially through the nodes using the negotiated portions of the channel.

[0052] With BoD connections, this negotiation and subsequent reallocation of capacity between nodes typically takes on the order of seconds to complete (reallocation time, typically in the mid-single digits to the first half of the double digits). In the exemplary embodiments, system 100B will require modifications (described herein) to effectively use these types of connections.

[0053] Long reallocation times (associated with increased packet loss and / or latency) are interpreted as congestion in the current implementation, and typical congestion control methods reduce the usage for the connections where congestion is detected to avoid overloading these connections. However, BoD connections actually require increased allocations and, in some cases, need to behave oppositely (at least temporarily) to obtain them.

[0054] System 100B includes an input queue 102 that receives a data flow including one or more data packets from an input network for transmission. Examples of data flows can be data packets associated with file transfers, video file transfers, audio file transfers, Voice over Internet Protocol (VoIP) calls, or Virtual Private Networks (VPNs).

[0055] Input queue 102 is also responsible for applying a drop policy to packets when a scheduler (as described herein) cannot service input queue 102 at the same rate at which new packets arrive from the LAN-side client. This typically occurs when the LAN client is transmitting at a rate higher than the aggregated WAN transmission capacity.

[0056] System 100B includes three stages for transmitting data packets to a remote peer via a mixed set of connections according to a push-type architecture, and these stages are alternatively referred to as pipelines or pipeline stages.

[0057] The first stage 104 includes a flow classification engine 104A, a scheduler 104B (hereinafter alternatively referred to as a push scheduler 104), receives data packets from input queue 102, and queues the data packets for passing to the second stage 106. In an exemplary embodiment, the first stage 104 includes all of input queue 102, flow classification engine 104A, and scheduler 104B.

[0058] The input queue 102 buffers the packets received from the client on the LAN side of the transmitter and classifies the data packets into flows (alternatively referred to as mapping the data packets to flows) by using, in some cases, the flow classification engine 104A or in cooperation with the flow classification engine 104A. The input queue 102 ensures that the data flows are fairly serviced by the scheduler of the first stage 104.

[0059] In an exemplary embodiment, the first stage 104, or the assist flow classification engine 104A of the first stage 104, monitors or is notified of changes to the user configuration settings associated with the assignment of data packets to connections. The user configuration settings include information on how connections should be assigned to priority levels when those priority levels are to become active, as well as user-configurable flow classification rules that can preferentially select connections with certain monitored characteristics for packets belonging to a flow (e.g., packets belonging to the video stream class may prefer a connection with low latency, while packets belonging to the file download class may prefer a connection with a lower monetary cost).

[0060] The first stage 104 is periodically notified of current metadata (e.g., operating characteristics) for each of a plurality of connection monitoring instances corresponding to a WAN connection that includes mixed links (as described herein). This metadata is collected by the monitoring instance 106 and can include the estimated bottleneck bandwidth of the connection, the estimated minimum round-trip time, the most recent actual round-trip time, the current congestion window for the connection, and the receive / loss status for packets previously sent to the next stage. Using this metadata, the first stage 104 can also extend the metadata to track the estimate of the bytes the first stage 104 has scheduled as inflight on this connection and further subdivide that total into new bytes, retransmitted bytes, or dummy / probing bytes. All of the metadata collected and tracked by the first stage 104 is referred to as monitored characteristics (e.g., operating characteristics).

[0061] Then, using all available monitored characteristics, the first stage 104 will determine whether to send a packet to the next stage and, if so, which connection should receive the packet. The set of packets provided to a connection monitoring instance can include duplicates of packets previously sent (to either the same connection monitoring instance or a different connection monitoring instance). Alternatively, the scheduler can determine that a given connection monitoring instance is not currently eligible to receive a packet even if the instance's connection currently has capacity. The assignment of packets to specific stage 3 instances is considered elsewhere in this document.

[0062] The second stage 106 includes connection monitoring instances (e.g., connection monitoring instances 106A, 106B, and 106N associated with connections 110A, 110B, and 110N respectively) for each of the plurality of available connections. The connection monitoring instances of the second stage 106 monitor the associated connections and determine or obtain statistics related to the performance (e.g., latency) of the connections, which are called operating characteristics. The second stage 106 may provide congestion control functionality and packet buffering to ensure that packets are available for the next stage. The second stage 106 passes the packets to a third stage 108 that includes output queues for each connection (e.g., output queues 108A, 108B, and 108N). The third stage 108 then writes the packets to the network interface, optionally according to pacing or timing hints provided by the second stage 106.

[0063] In some embodiments, the second stage 106 may implement one or more Router to Radio Interface (R2RI) protocols (e.g., PPPoE (RFC5578), R2CP, DLEP (RFC8175)) to better support the bandwidth allocation process for BoD connections.

[0064] The second stage 106 is responsible for receiving packets from the first stage 104, locally queuing these packets, and providing groups of packets as bursts to the next pipeline stage. The second stage 106 also tracks metadata for the associated connections of the second stage 106 based on acknowledgment responses received from peers or receivers.

[0065] The purpose of the queue in the second stage 106 is to ensure that there are always packets available for the next pipeline stage to be written to the network device for transmission. Some embodiments may include priority metadata as part of each packet's information, such that new packets arriving from the first stage 104 can be reordered within the queue to give preference to higher-priority packets.

[0066] To ensure that the second stage 106 buffers enough packets such that packets can be provided to the third stage 108, the second stage 106 may provide modified metadata to the first stage 104. For example, according to some exemplary embodiments, the second stage 106 may choose to report a larger congestion window (i.e., "over-advertise") such that the first stage 104 provides additional packets that can be buffered at the second stage 106 by the time the third stage 108 completes writing the actual congestion window packets. In other words, in some embodiments, the second stage 106 over-advertises the congestion window to account for the processing overhead / delay incurred by the entire pipeline 104 / 106 / 108 to ensure that the network always remains filled with in-flight bytes.

[0067] In some embodiments, the pipeline stage or the network itself is known to employ compression techniques, which effectively increases the number of in-flight bytes that can be supported by the network. "Over-advertising" the congestion window to the first stage 104 is also used in this scenario to ensure that the network always remains filled with in-flight bytes.

[0068] The second stage 106 may also provide modified metadata to the first stage 104 to facilitate changing the capacity allocation in the BoD connection (i.e., to trigger the BoD connection to allocate a higher capacity, cause the first stage 104 to schedule packets on this connection as if the first stage 104 had a higher capacity). This modified metadata may include information that decomposes the higher capacity into different requested types / priorities. For example, the higher incremental capacity used to probe the BoD connection typically consists of redundant / dummy packets because there is a high likelihood that the probe packets will be dropped by the BoD connection until the DAMA reallocation process is complete.

[0069] Advertising or over - advertising capacity from the second stage 106 to the first stage 104 does not necessarily immediately result in probing for a higher incremental capacity. The first stage 104 ultimately still makes a decision on how packets are allocated to the connection. One embodiment may wait until all CWNDs on non - BoD connections are fully consumed (or nearly fully consumed) before instructing the second stage 106 to start the DAMA reallocation process.

[0070] The queue in the second stage 106 also enables fast notification of a failure to the previous stage if something happens to the underlying network connection. Packets provided to the third stage 108 and written to the network must be assumed to be in transit even if the network interface subsequently reports being down, meaning that a timeout must occur before they can be marked as lost. However, packets still queued in the second stage 106 can be immediately marked as lost as soon as the second stage 106 is notified that the connection is no longer available, thereby making these packets immediately eligible for prioritized re - transmission on an alternative connection.

[0071] Before being passed to the third stage 108, in some embodiments, the rate of packet transmission may be explicitly paced by assigning to each packet a nominal timestamp indicating when the next stage should attempt to send the packet over the connection. Such a timestamp may be determined based on the estimated bottleneck bandwidth of the connection (e.g., the connection to which the packet is assigned) and the packet size, at or before the network bottleneck link, to prevent network congestion.

[0072] FIG. 2 shows an example of one embodiment that supports packets prioritized by the second stage 106 at 200. In FIG. 2, the head of the queue is shown on the right side and the tail of the queue is shown on the left side. In the upper part of FIG. 2 (e.g., the first step), the first stage 104 provides a new list of prioritized packets to the connection monitoring instance of the second stage 106. The lower part of FIG. 2 (e.g., the second step) shows that these new packets are added to the priority queue in order of priority, such that all packets of priority 1 are dequeued before any packets of priority 2 are dequeued. Packets of the same priority are maintained in sequence order.

[0073] In an exemplary embodiment not shown, the second stage 106 may then have a single connection monitoring instance that provides packets to an instance of the third stage, each of these instances managing the transmission of packets with a set of network interfaces that share a physical interface card. Other embodiments may exist with different numbers of instances for each of the pipeline stages.

[0074] Feedback in the form of metadata or callbacks (which can be the mechanisms used to send metadata) can be sent from a stage to the previous stage, or a stage can receive notifications from different parts of the system. Any of these actions can trigger (i.e., push) the stage that sends packets to the next stage.

[0075] The third stage 108 receives packets from the second stage 106 via one or more connection transmission instances (e.g., connection transmission instances 108A, 108B, and 108N associated with connections 110A, 110B, and 110N respectively), and writes these packets to the network 112. In some embodiments, all connection monitoring instances of the second stage 106 provide packets to a single connection transmission instance, or each connection monitoring instance can be associated with a connection transmission instance.

[0076] Some embodiments of the third stage 108 may use a queuing mechanism to enable providing hints regarding when the previous stage should send packets. For example, the hint can occur in the form of a "nominal transmission time" associated with the packet as metadata. If the next packet to be sent has a later nominal transmission time, the third stage 108 will wait until that time before sending that packet. Emergency packets can be flagged with an immediate timestamp that causes them to be sent before other queued packets. To prevent starvation of non-emergency packets, some embodiments may queue emergency packets after non-emergency packets with past nominal transmission times.

[0077] In some embodiments, a callback function may be included as part of the packet metadata when a packet moves through the pipeline from a second stage 106 to a third stage 108. This callback can be used to trigger a notification to the second stage 106 to indicate that the third stage 108 has transmitted the packet. Such a notification can be used to trigger a new batch of packets pushed from the second stage 106 to the third stage 108.

[0078] Figure 3 shows, at 300, the progression over time of an embodiment that includes a "send time" hint (nominal transmission time) that requires a packet provided by the second stage 106 not to be transmitted before a specified time. Packets with a "send time" of 0 should be transmitted as soon as possible. Figure 3 illustrates examples for all of these cases (e.g., the use of past nominal transmission times for emergency packets and how the system avoids starvation of non-emergency packets).

[0079] In the state depicted at t = 1, the packet queue of an instance of the third stage 108 contains four packets, three of which are currently eligible for transmission, and an instance of the second stage 106 is about to provide four new packets, the first of which has sequence 5 (sequence order 5) and should be transmitted immediately.

[0080] In the state depicted at t = 2, new packets are added to the priority queue of the third stage 108. Note that in this embodiment, when sequence 5 was queued, sequence 2 was already eligible to be transmitted, so it was chosen to queue the packets in sequence 5 to be transmitted after the packets having sequence 2. In this way, freezing the order of packets at the head of the packet queue can prevent starvation when it can prevent non-immediate packets from being transmitted, even though a constant stream of new urgent packets to be transmitted immediately may become eligible to be transmitted long before the latest immediately transmitted urgent packet arrives.

[0081] In the state depicted at t = 3, the third stage 108 dequeued the first four packets from the output queue and provided these packets to the network interface. Currently, since there are no packets eligible to be transmitted until time t = 4, the third stage 108 instance will need to wait to transmit more packets until time progresses to t = 4, or until a new packet arrives that is to be transmitted immediately or at time t = 3, even if there is currently write capacity on the network interface.

[0082] Packets can be passed in order from one stage to the next when each stage determines it is optimal.

[0083] Similar to FIG. 1A, data packets are transmitted across the connection and then received by the receiver 116 via the connection 114. The receiver 116 includes a sequencer 118 for assembling the transmitted data packets and may further transmit the assembled data packets via a local area network (LAN) 120.

[0084] In contrast to the system 100A, the push communication system 100B does not include an independent timer associated with each connection. Instead, packet scheduling onto the connection is performed in a centralized location and includes a push scheduler 104 that accesses the monitored characteristics of each connection, and the push scheduler 104 is activated in response to one or more monitored events that occur other than based on a timer. For example, one or more monitored communication events may include an acknowledgement / negative acknowledgement (ACK / NACK) from the receiver of a previously transmitted data packet, or the arrival of a new data packet at the input queue 102.

[0085] In an exemplary embodiment, not only the 20 ms during which a packet is being transmitted, but the entire connection (congestion window) CWND can potentially be consumed or allocated by the push scheduler 104.

[0086] Using access to the monitored characteristics of each connection, the push scheduler 104 can be configured to optimize packet scheduling decisions. For example, data flows with specific transmission requirements can be preferentially allocated to connections that can meet the requirements associated with the data flow, rather than being allocated mostly based on the wake-up order of timers. In an illustrative example, a first data flow requires a transmission with a latency of less than 50 ms.

[0087] Connections 110A, 110B, and 110N have latencies of 5 ms, 45 ms, and 80 ms, respectively. Using the pull scheduler 130, if the 50 Hz timer for connection 110B (45 ms latency) wakes up first, that timer would be able to schedule packets for this first data flow onto connection 110B. Using the push scheduler 104, the data flow can be preferentially scheduled onto connection 110A (5 ms latency) until the CWND of connection 110A is completely consumed before overflowing onto connection 110B (45 ms latency).

[0088] To enable two-way communication, a single instance of the multipath endpoint can include all of the components described in FIG. 1B. Specifically, it can include one or more instances of the transmitter 100 to support upstream communication to other multipath endpoints and one or more instances of the receiver 116 to support downstream communication from other multipath endpoints.

[0089] According to some embodiments, for example, one or more of the available connections are BoD connections, and the push scheduler 104 can allocate more data to the BoD connection to request more capacity from the network based on criteria such as, for example, one of the connections 110A has its CWND completely consumed (or approaching complete consumption) while more packets are available in the input queue 102 eligible to be transmitted on connection 110B (e.g., based on per-flow rules).

[0090] In an exemplary embodiment, the push communication system 100B can enable more accurate / fair quality of service (QoS) and active queue management (AQM) packet dropping rules to be applied to the input queue 102. Referring back to the previous example, a flow with a 50 ms latency requirement can never utilize an 80 ms connection. Since the push scheduler 104 has a consistent snapshot of the entire system, it can calculate a fair share of the partial usage and total capacity of the flows excluding the 80 ms connection.

[0091] A potential advantage of the push-based scheduling approach is that a complete set of metadata for multiple connections, including mixed links, is available to the push scheduler 104 to optimally allocate packets to the target connection.

[0092] Figure 1C is a flowchart of a method 110C for transmitting data packets using system 100B or other push communication systems, according to some exemplary embodiments.

[0093] In block 1130, method 100C detects when the input queue contains unassigned data packets available for transmission.

[0094] In block 1132, method 100C includes assigning one or more of the unassigned data packets to one or more corresponding output queues of a plurality of networks having suitable monitored characteristics.

[0095] In block 1134, method 100C includes updating the monitored characteristics of the corresponding plurality of networks to reflect the assigned one or more data packets.

[0096] In block 1136, method 100C transmits the assigned data packets from the output queues corresponding to each network of the plurality of networks to the destination device.

[0097] In an exemplary embodiment, the method for transmitting data packets in system 100B includes a loop of the following steps each time the push scheduler 104 is triggered by an external event. 1. Request data packets from input queue 102, 2. Assign the received packets to a connection for transmission, 3. Repeat steps 1-2 until a) there are no more packets available in input queue 102, or b) there is no available transmission capacity in any of the active connections (such as defined by the congestion window (CWND) metric).

[0098] The method described can be implemented such that the push scheduler 104 does not consider any data flow specific requirements. In such an embodiment, the system 100B assigns received packets based on the ordering of the connections.

[0099] The ordering can be based on monitored characteristics of the connections and can include, for example, a subset of the operating characteristics estimated by a congestion control algorithm. In one embodiment, the BBRv1 algorithm is used, although other embodiments can include examples such as BBRv2, TCP Reno, TCP LEDBAT, custom implementations, etc. for handling specific connection types such as BoD. In subsequent discussions regarding the operating characteristics specific to BBRv1, for illustrative purposes only, reference is made to a pipe analogy where two endpoints are modeled as a series of interconnected pipes, each pipe segment having a different diameter and length.

[0100] The operating characteristics can be a measure of the minimum time required for transmitted packets to reach the destination endpoint (e.g., the receiver) and be acknowledged (e.g., via an ACK message from the receiver to the transmitter). This duration is referred to as the "round-trip propagation delay" (abbreviated as "RtProp") and is measured in units of time (e.g., milliseconds). From the perspective of the pipe analogy, this value represents the total length of all pipe segments from the transmitter to the receiver.

[0101] The operating characteristics can be the maximum rate at which data can move between endpoints along the pipe segment with the smallest diameter. The smallest pipe diameter, referred to as the "bottleneck" segment, determines the maximum rate at which data can move between the transmitter and the receiver, which is also referred to as the "bottleneck bandwidth" (abbreviated as "BtlBw"). For example, the techniques used to estimate BtlBw are described in the IETF draft: https: / / tools.ietf.org / html / draft-cheng-iccrg-delivery-rate-estimation-00.

[0102] Table 1 shows an exemplary set of WAN connections and properties of the WAN connections, an exemplary subset of the operating characteristics selected by the scheduler 104 to be the characteristics to be monitored, and the resulting sort order. [Table 1]

[0103] This exemplary method sorts the connections across the following dimensions and moves to the next dimension only if the current dimension is compared as equal between connections. 1. Increase the preference priority level, 2. Increase the RtProp, 3. Decrease the BtlBw, and 4. Increase the interface name.

[0104] Note how the result of the sort operation keeps the order of the connections relatively stable for all parameters where the sort dimensions are not expected to change frequently.

[0105] Since this stable sort order is intentional, packets arriving at the input queue tend to be "sticky" with respect to the same set of connections (sorted at the top of the list), and the sort order "overflows" up to a connection with a lower sort only if the input request exceeds the total CWND of the connections sorted higher.

[0106] Note that the sorting criteria in this example are not flow-specific. For example, an administrator or end user may only want high-priority application flows to be transmitted via the measured Cell 1 connection and Cell 0 connection (an example of transmission requirements), meaning that those flows would prefer the resulting sort order that places Cells 1 and 0 lower in the list. Similarly, there may be some flows with a maximum latency limit, e.g., a VoIP call where the end user considers the maximum latency limit unusable for an interactive conversation if the latency exceeds 150 ms. For these flows, it may be desirable to have the resulting sort order that uses RtProp as the primary sort dimension. Embodiments that extend the sorting criteria to be flow-specific are considered in subsequent paragraphs.

[0107] The scheduler is also responsible for determining the packet retransmission policy. In an exemplary embodiment, System 100B is configured such that if a packet is reported as missing by the receiver, the scheduler always marks the packet for retransmission. This is desirable for many flows, but other flows may not want this at all (additional latency or packets arriving out of order may make the packets unusable by the application), or may only want retransmission attempts within a certain period.

[0108] Figure 1D is a flowchart of another method for transmitting data packets according to some exemplary embodiments.

[0109] One embodiment of a simple scheduling loop is as follows. 1. Perform any accounting required to update the connection metadata and determine the set of active connections. 2. Determine the preferred ordering of the connections based on the connection metadata (monitored characteristics such as monitored operating characteristics). 3. While there are packets to send and network capacity available in the set of active connections: a. Obtain the next packet to be sent. b. Queue the packet to be sent on the first connection in the ordering that has available capacity. c. Update the connection metadata (monitored characteristics) to reflect the scheduled packet. d. Repeat. 4. For each connection having queued packets, send those packets to the next pipeline stage for that connection.

[0110] Such an embodiment treats all packets as equal for the purpose of the preferred ordering of connections. In this case, packets can be considered to belong to the same class of flow (even though distinct packets may logically belong to distinct flows, there is no distinction between these flows for the purpose of connection assignment from the perspective of the scheduler).

[0111] Figure 1E is a flowchart of a further method of transmitting data packets according to some exemplary embodiments.

[0112] An embodiment of a scheduler that can distinguish between different classes of flows (i.e., ICMP echo request / response vs. video stream vs. file transfer) and even between individual flows within the same class (video stream between host A and host B vs. video stream between host A and host C) can be provided as follows. 1. In block 1102, perform any necessary accounting for updates to the connection metadata (monitored characteristics) and determine the set of active connections. 2. While the set of active connections has capacity (block 1114), a. In block 1103, construct a selector based on the currently active set of connections. b. In block 1105, use a selector to select a packet to transmit over one of the available connections. If no packet is selected, exit the loop. c. In block 1107, based on the packet flow and flow classification, in combination with connection metadata (monitored characteristics), determine the ordering of the active connections. d. In block 1110, queue the packets to be transmitted on the first connection in the ordering that has available capacity. e. In block 1112, update the connection metadata to reflect the scheduled packets. f. Repeat. 3. In block 1116, for each connection having queued packets, transmit those packets to the next pipeline stage for that connection.

[0113] These two sample methods (the methods shown in FIGS. 1D and 1E) are similar, and the method shown in FIG. 1D is a simplified and optimized version of the more general method in FIG. 1E.

[0114] The first method can be converted to the second method by defining a single flow class and a selector that matches all packets. In view of this, the second method (FIG. 1E) will be considered in more detail, and the same considerations will apply to the first method (FIG. 1D) subject to the above limitations.

[0115] The first step is to update any pending metadata accounting (monitored characteristics) to determine the set of active connections. In one embodiment, the set of active connections is the connections that are not subject to unreliable connection backoff (i.e., connections whose use is restricted due to recent failures in order to ensure packet delivery).

[0116] Other embodiments may further restrict the set of active connections, for example, by eliminating connections whose metadata (monitored characteristics) do not match any of the currently defined flow class requirements (i.e., assuming that all packets must match one defined flow class, if a connection is not selected by any flow class, the connection may be considered inactive; for example, if all flow classes specify an RTT of up to 500 ms, a connection with an RTT > 500 ms may be considered inactive).

[0117] The set of active connections has the capacity to send packets if at least one connection in the set has an available CWND (congestion window), meaning that the number of in - flight bytes currently being tracked by the scheduler for this connection does not exceed the CWND determined for this connection. In one embodiment, the CWND for a connection can be the CWND reported by the next pipeline stage, and if slotting or priority routing is enabled, the scheduler may choose to reduce the reported CWND.

[0118] In one embodiment, a connection that is throttled to a particular rate reduces the CWND of the connection by the same factor as the throttle rate across the bottleneck bandwidth (or, if the throttle rate is higher than the bandwidth estimate, the CWND is not changed). Similarly, the CWND of a connection can be reduced if this CWND is subject to priority routing (where this embodiment supports grouping connections to a priority level that is activated when the total goodput (throughput excluding overhead) of all connections at a higher priority level drops below a configured threshold for the level at which activation occurs). Various methods for determining CWND reduction in these scenarios are described later in connection with FIGS. 8A - 8C and FIGS. 9A - 9C.

[0119] A connection assigned to a non-active priority level (where the total goodput + unused bitrate for all connections at higher priority levels meets or exceeds the activation bitrate for the priority level) has a zero effective CWND, and a connection at a non-primary priority level has its CWND set to zero if the estimated total goodput bitrate of all connections within the currently active set of priority levels meets or exceeds the lowest active priority activation threshold.

[0120] In other embodiments using priority routing, non-primary level connections can be capped at a rate lower than their bottleneck bandwidth such that they only contribute enough to close the gap to the configured activation threshold. This can be done, for example, for cost reduction if the non-primary connections are more expensive connections intended to be used only in emergencies. Similar to throttling, the CWND is reduced to reflect the capped rate at which these connections should be sent.

[0121] The loop in step 2 of the method can operate as long as the connections within the active set have capacity.

[0122] Step 2a (block 1103 shown in FIG. 1E) constructs a selector used to find packets eligible to be sent on one of the active connections having active capacity. In some embodiments, the selector consists of a set of flow classes equivalent to the current set of active connections having capacity.

[0123] For example, if a flow class requires a connection with a maximum latency of 500 ms and all available active connections have a latency exceeding 500 ms, that flow class will be excluded from the selector. In some embodiments, if the embodiment can determine that there are no existing connections (active or inactive, regardless of available capacity) that meet the criteria for such packets, the embodiment may choose to include in the selector a flow class that would not normally match the criteria for an active connection. In that case, the embodiment may choose to violate the matching criteria that would support sending the packet rather than holding the packet until it times out and is dropped. Other embodiments may choose to enforce a strict matching policy where packets that match a flow class without a qualified connection are never sent.

[0124] In another embodiment, shown in an exemplary sort order in the process flow of FIG. 1F, alternative techniques can be utilized to calculate, determine, or establish a sort order for controlling the routing of data packets to various network connections.

[0125] Some or all of the input parameters can be rounded / bucketed before being used in the sort order determination. The technical improvement from rounding and bucketing is to keep the sort order as "stable" as possible so that sort order changes do not always occur (e.g., without rounding and bucketing, two close values can cause "jitter" by having consecutive sort changes as their values vary over time).

[0126] Factors affecting the determination of the sort order include keeping the sort order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (splitting increases the mechanism for self-induced out-of-order events and jitter events), and taking into account both the measured properties of the connections and unmeasured external directives.

[0127] Values for rounding can include the following: ● Latency related values (RtProp, RTT) are floored 1 ms after rounding and rounded to the nearest multiple of 50 ms. ● Throughput related value (BtlBw) is rounded to the "nearest digit". ● Packet loss (the worst value among three sliding windows) is rounded to the nearest digit. Note that since the raw value here is limited to the range [0, 100], the output value is limited to the set (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100).

[0128] An example of rounding to the nearest digit is provided in the following table. The raw value, the calculated digit of the raw value, and then the resulting rounded value are shown.

Table 2

[0129] Specific improvements based on improving the networking routing order to control the use of network interfaces are described in some embodiments that utilize the combined properties of round-trip time (RTT) and packet loss to predict throughput based on the Mathis equation for establishing the upper limit of given TCP throughput for RTT, packet loss, and MSS. The Mathis coefficient can be used to correct the sorting order for routing networked packets.

[0130] The properties to be measured can include packet loss (further aggregated as the worst value among three windows) during three separate sliding windows (500 ms, 3 s, 30 s), the current network round-trip time (RTT), the bandwidth round-trip propagation time (BBRv1) parameter, RtProp (network propagation delay), and BtlBw (throughput estimate). External directives can include, among other things, priority routing (PR) rules, but can also include flow-specific rules / requirements.

[0131] In some embodiments, the measured properties of BBRv1 are different from the design of these characteristics in the original BBRv1 IETF draft. For example, to account for embodiments that may run on systems lacking consistency in the execution time of the embodiment, the size of the sliding window used to calculate BtlBw can be increased to cover a longer duration (hiding variability due to execution non-consistency), but if network loss or latency increase events occur, those events can indicate that BtlBw can actually change and any long history in the sliding window must be forgotten, so the window duration can also be immediately shortened.

[0132] For similar reasons of execution non-uniformity, the input parameters used to measure RtProp are changed to include more than just network propagation delay and may add a portion of the execution / processing time, which can be non-trivial and consistently contribute to the actual delay encountered when transmitting packets and waiting for responses. In some embodiments, this is referred to as "system RtProp" as opposed to "network RtProp" which consists only of network propagation delay. In these embodiments, "system RtProp" and "network RtProp" can be used for various calculations depending on the intended purpose. For example, "system RtProp" is appropriate when calculating CWND to include sufficient data in-flight for the pipeline to account for processing delays. However, "network RtProp" is more appropriate when sorting network connections to determine packet routing, as the preferred routing decision is independent of system execution delay.

[0133] Packet loss, latency, and throughput are not completely independent variables. Their three combinations affect each other according to the congestion control behavior of the application. This is because, for example, when using TCP, higher packet loss results in a longer period of wireless "silence" (underutilized channels) while the transmitter waits for duplicate acknowledgment (DUP ACK) or retransmission timeout (RTO) to occur.

[0134] Therefore, several composite attributes are determined and used in the final sorting method. Specifically,

[0135] Mathis Coefficient: This composite property is based on the Mathis Relation, which predicts throughput considering RTT and packet loss and gives the upper bound of TCP throughput considering RTT, packet loss, and MSS. When comparing the properties of any two network connections to determine their sort order, since MSS can be considered a constant, this relation shows that the expected throughput is proportional to the reciprocal of (RTT * sqrt(packet loss)). In this example, this relation is used as a practical implementation for establishing coefficients used when directly controlling the operation of a network controller or router. For each connection, the system determines the Mathis coefficient as follows: MAX(rounded Rtt, 1ms) * sqrt(rounded aggregated packet loss). Connections with smaller Mathis coefficient values are considered better. Since the calculation is a function of two independently measured properties, the relationship between these characteristics determines how connections compare to each other. The Mathis coefficient is held and stored in a data object such as the corresponding Mathis coefficient data value associated with each connection and can be stored, for example, in an array of data values along with other monitored characteristics.

[0136] For example, in one extreme case, two connections with 0% packet loss will always have the same Mathis coefficient, and the actual RTT values of these connections are irrelevant. In this scenario, the Mathis coefficient does not affect the sort order and is determined by comparing subsequent measured properties with the composite property.

[0137] In the other extreme case, when two connections have 100% packet loss (rounded aggregated packet loss = 1.0), only the rounded Rtt values of these connections determine how they compare to each other. In some embodiments, this is undesirable, so connections with 100% packet loss are filtered out from the available choices at an earlier stage.

[0138] Between two extremes, a connection having both low RTT and low packet loss comes to be compared as being better than a connection having higher values. However, a connection with a high percentage of loss can still be compensated by having a low RTT and can have the same Mathis coefficient as a connection with a low percentage of loss. For example,

[0139] Connection A: 100 ms RTT, 0.01 (1%) loss => Mathis coefficient = 10

[0140] Connection B: 19 ms RTT, 0.25 (25%) loss => Mathis coefficient = 9.5

[0141] In this example, even though Connection B has significantly higher loss (25% versus 1%), the lower RTT of Connection B compensates for it and rivals a better Mathis coefficient than Connection A. The explanation for this is that the lower RTT of Connection B exchanges packet acknowledgment responses (positive and negative) for the lost packets and, even though Connection A has a smaller percentage of loss, is faster than Connection A with a higher RTT and enables any necessary packet retransmission to be completed.

[0142] Capacity coefficient: This composite property takes into account BtlBw and RtProp and is determined as follows: Rounded BtlBw / MAX(rounded RtProp, 1 ms). The unit is "bytes / second per millisecond" and has no physical meaning, but provides a way to normalize the bandwidth that a connection can achieve per unit of RtProp. In other words, a connection that can achieve more bandwidth per millisecond of its RtProp is considered more efficient than a connection that achieves less bandwidth. Efficient transmission in this context means a connection that achieves more bandwidth per millisecond of propagation delay (RtProp).

[0143] Once all of the raw input has been processed and the composite properties have been calculated as described above, the actual sorting mechanism can operate continuously as follows.

[0144] The transition from one comparison to the next occurs only if all of the previous lines are compared as equal. If the comparison is not equal, the remainder of the comparison is irrelevant and is omitted. In some embodiments, the selection of comparisons is done on a flow classification basis (e.g., VPN vs. VoIP vs. ping packet vs. FEC packet), and for each different flow, the list of comparison criteria can be different (e.g., each can have different pre-defined most important characteristics, second most important characteristics, third most important characteristics, etc.). Each of these characteristics can be iterated through one after the other, starting with the most important, until the connection itself can be sorted and placed into an ordered sequence. In some embodiments, depending on the flow classification, the first pre-step is to select a subset of the connections that are suitable for transmitting the type of data packet in the flow classification, and then an order is assigned to them before packet allocation is performed. 1. A connection that is completely down (100% aggregated loss) is considered a worse connection than a connection that has less than 100% aggregated loss. 2. PR priority (higher priority is better) 3. Mathis coefficient (lower is better) 4. Rounded RTT (lower is better) 5. Capacity coefficient (higher is better) 6. Rounded RtProp (lower is better) 7. Rounded BtlBw (higher is better) 8. Connection ID (lower is better)

[0145] The rendering of the sorting mechanism is shown in FIG. 1F. In an exemplary sorting mechanism, the real-time flow first includes a comparison of RTT / RtProp, then an equality comparison, then the real-time flow compares reliability, and then, if still equal, compares things like transmission speed. Depending on the flow type, there can be different sorting orders. For example, a flow that desires throughput may first compare the transmission rate. If equal, it compares by reliability. If still equal, the real-time flow may decide to evaluate nothing else and simply make an arbitrary selection.

[0146] In some embodiments, flow requirements (e.g., real-time vs. throughput) can determine the order and number of comparisons. For example, a real-time flow may prefer to compare only the rounded RTT and the rounded RtProp, ignoring all other properties. A throughput flow may decide to compare only the Mathis coefficient and the capacity coefficient.

[0147] The above-described approach relates to explicit sort dimensions. There are also implicit sort dimensions that tend to arise as a byproduct of typical application traffic patterns.

[0148] Since most application traffic tends to be bursty or adaptive, it does not require the use of all WAN connections. This means that connections that are currently at the bottom of the sort order may not have traffic transmitted over these connections. If these connections are not like this, since the explicit sort dimensions of these connections are not updated, these connections tend to maintain their position near the bottom of the sort order.

[0149] In some embodiments, over a certain period of time, push scheduling can also be configured to implicitly cause a sorting order along one or more implicit sort dimensions. The implicit sort dimension forces connections that have encountered periodic bad events (e.g., latency spikes due to congestion) to bubble to the bottom of the list. Connections with consistent behavior naturally bubble to the top and stay near the top.

[0150] This is generally considered a good property of the algorithm, but can be modified to be affected by external factors in the future. For example, if the flow has specific requirements that cannot be satisfied by connections near the top of the current sort order, it may make sense to generate probing traffic with connections near the bottom to refresh the explicit sort dimension of these connections.

[0151] Figure 4A depicts an embodiment in which a selector is generated at 400A using predefined flow classes and the current connection state.

[0152] In this example, there are three connections each having a set of monitored characteristics, and three predefined flow classes each defining a match criterion and a preference for sorting connections that meet the match criterion.

[0153] In this embodiment, the selector consists of a set of flow classes that match at least one connection having available capacity for the match criterion. In this example, both Flow Class 1 and Flow Class 2 have match criteria that match Connections 1 and 2, while Flow Class 3 does not match any connection and is not included in the selector.

[0154] Even if both Connection 1 and Connection 2 match, Connection 2 is excluded from the selector because it does not have an available congestion window and cannot currently transmit packets. When constructing the selector, since a matching connection with an available congestion window can be included in the flow class as soon as it is found, one embodiment may determine that it can include Flow Class 1 and Flow Class 2 in the selector without examining only Connection 1 and confirming that Connection 2 matches.

[0155] Step 2b (sub-block 1105 shown in FIG. 1E) uses the selector from Step 2a to find packets on the retransmission queue or transmission queue that are eligible to be (re)transmitted on an active connection with capacity. If such packets cannot be found, the loop ends. This can occur even if several active connections have capacity and there are packets on the input queue or retransmission queue.

[0156] FIG. 4B shows, in 400B, an embodiment having a priority input queue that is a set of ordered flows, where the flow on the right side of the queue should transmit packets before the flow on the left side. Each flow has a flow ID, and each flow belongs to one of the pre-defined flow classes. Each flow also includes an ordered queue of packets, so that when a flow is selected, the packet at the head of the queue is removed.

[0157] In this example, a selector from Figure 4A is used to evaluate the flows in the input queue from right to left, and the selector selects the first flow that has a flow class listed in the selector. In this case, the flows with ID5 and ID2 are evaluated and rejected because both of these flows have flow class 3 which is not one of the flow classes from the selector. Then flow 7 is evaluated and flow 7 matches because it belongs to flow class 1 which is one of the flow classes in the selector. Then packets are extracted from the head of the packet queue of flow 7 and stamped with flow metadata.

[0158] When a packet is selected, step 2c (block 1107 shown in Figure 1E) constructs an ordering of the active connections. The ordering selected is based on the rules provided in the definition of the flow class, in an order that is generally acceptable if the embodiment supports it or otherwise. Some embodiments may use the same ordering for all flows that match a flow class, and some embodiments may make the ordering flow-dependent (e.g., making all packets within a flow sticky to a given connection without requiring all flows to be sticky to the same connection). The ordering may be constructed as needed in each iteration of the loop in step 2, or some embodiments may choose to cache the ordering and clear the cache if the criteria used to construct the ordering change (i.e., on metadata updates).

[0159] One embodiment constructs an ordering regardless of whether a connection currently has capacity, since scheduling packets on a connection can reduce the capacity to zero, or scheduling packets on different connections can increase the capacity above the current value of zero if it causes the metadata mix to change enough to activate a set of connections at a previously unused priority level. By constructing the ordering in this way, if the only items that change are the effective CWND and in - flight byte count for a connection, the ordering can be cached and reused.

[0160] Flows across the system 100B tend to behave better when the packets of that flow cross the same WAN connection whenever possible. Thus, one embodiment should construct the ordering using a stable sorting criterion so that the ordering created for the same flow is consistent as long as one or more of the monitored characteristics of the connection do not change significantly. Thus, operational characteristics such as RTT and RtProp, and bandwidth are poor choices since they can vary over short periods and destabilize the sort order.

[0161] The conversion from "noisy" raw operational characteristics to monitored characteristics can occur using statistical filtering methods. Statistical filtering methods can range from previous measures such as mean, variance, ANOVA to the field of (quantized) rank - order statistics (rank - order filtering). For example, the monitored characteristic can be a range of raw operational characteristics, which can be defined by specific values (e.g., 1 - 100 ms) or number of digits, etc. Specific examples are included in the subsequent paragraphs, including the example shown in Table 2.

[0162] Some embodiments can use machine learning techniques to convert from operational characteristics to monitored characteristics. For example, a trained model can be developed to predict imminent connection failures, and as a result, construct an ordering that sorts connections matching at the end of a list. Another exemplary model can be used to detect the type of connection (e.g., wired, cellular, WiFi, satellite, BoD, etc.) and use it to form the basis for new monitored statistics (e.g., susceptibility to buffer bloat) that are not easily measurable in real time.

[0163] One embodiment uses the following order for all packets (the later criteria are only used if all previous criteria are equal).

[0164] "Bucketized" drop probability. Connections can be grouped into buckets defined by a range of drop probabilities based on packet count or size (e.g., in bytes). Connections within a bucket with a smaller drop probability range appear first in the ordering, and connections within a bucket with a larger drop probability range appear later in the ordering.

[0165] Connection priority group (provided by an administrator or end user). Higher priority connections are ranked earlier (primary > secondary > tertiary, etc.)

[0166] "Bucketized" RTT. Latency is rounded to the nearest multiple of 50 ms. Bucketizing the latency values allows normal variations in latency to be hidden from the ordering while allowing significant changes in latency to be reflected in the ordering.

[0167] As described above, the "capacity factor" is an aggregated value measured in "bytes per second per millisecond", and the estimated bottleneck bandwidth is rounded to the nearest order of magnitude or an integer multiple of the nearest order of magnitude (e.g., rounding 1 to 1, rounding 10 and 11 to 10, rounding 15 to 20, rounding 86000 to 90000), and it is calculated by dividing this by, for example, RtProp bucketed to the nearest 50 ms. A connection achieving a higher bandwidth per millisecond of RtProp is considered better than a connection having a lower bandwidth per millisecond of RtProp.

[0168] Determining the capacity factor for raw BtlBW and RtProp values is highly variable, but stabilizing the capacity factor by rounding the BtlBW to the nearest order of magnitude or an integer multiple of the nearest order of magnitude and bucketizing the RtProp stabilizes the value over time, provided no large changes are observed. Table 2 shows some examples of bucketizing, rounding, and capacity factor calculation. Note that in this embodiment, the capacity factor is rounded to the nearest integer, and for the purpose of calculating the capacity factor, a bucketized RtProp value of 0 is treated as 1.

[0169] "Bucketized" RtProp. Lower values of RtProp bucketed to the nearest 50 ms are preferred over higher values.

[0170] "Rounded" BtlBW. Higher values of BtlBW rounded to the nearest order of magnitude or an integer multiple of the nearest order of magnitude are preferred over lower values.

[0171] When all of the above criteria are compared as equal, the connection ID (assigned when the connection was enumerated) is used to break the tie deterministically. Other embodiments may use a more complex tiebreaking mechanism, such as hashing to the integer modulo of the number of connections balancing the flow's properties.

Table 3

[0172] One embodiment of differentiating between flow types may, for example, modify the above order according to the transmission requirements of the flow type, giving priority to connections with a high capacity factor over connections with lower latency for file transfer flows, and giving priority to low latency connections over high bandwidth connections for video streams.

[0173] One embodiment of performing strict flow pinning to a connection (or set of connections) may choose to allow only exact matches of connection IDs (and in some cases, completely exclude other connections from the ordering). Such pinning rules are typically intended to override all ordering decisions determined by characteristics provided and monitored by an administrator or end user. Thus, even if the monitored characteristic indicates that a connection is unavailable or has failed, the flow will continue with the same ordering, and thus will not have connectivity through the multipath system 100B.

[0174] Some embodiments may create monitored characteristics that are composites of other monitored or operational characteristics and use these composites to determine the ordering. For example, a composite reliability metric can be created that combines the bucketized drop probability, the (in - flight) latency (RtProp) variance, and the (in - flight) throughput into a weighted metric representing the overall reliability of a connection. Connections that rank highly in this metric will be ensured for flows that require high reliability (high reliability may not mean high speed).

[0175] Generally, when a selector is generated that enables a packet to be selected, that packet should generate an ordering that enables the packet to be scheduled on some of the connections used to generate the selector.

[0176] Figure 4C shows an example of how the flow class of a packet (the "flow class") can be used to construct the order of connections at 400C. In the depicted embodiment, the flow class in the packet metadata is used to reference a flow class definition. The match criteria of the flow class are used to select which connections should be included in the constructed ordering (in other words, the flow class defines a set of transmission requirements that can be associated with a particular flow class), in this case, both connection 1 and connection 2 match, while connection 3 does not. Note that connection 2 is included even though it does not have an available congestion window (i.e., is not currently eligible to send packets). This facilitates caching the ordering and reusing the ordering even if the state of the available congestion window changes (e.g., if the scheduler initially blocks transmission on a connection due to its low connection priority, but then decides to reactivate the connection because the current data mix on the active connections does not meet the minimum system throughput).

[0177] Once a set of connections is selected, ordering criteria from the flow class (e.g., a subset of the transmission requirements associated with the flow class) are used to sort the matching connections into an ordered list. In this example, the first criterion is the capacity factor. Both connections have the same capacity factor. The second criterion is the RTT, and both connections have the same RTT. The third criterion is the BtlBW (bottleneck bandwidth), and since connection 2 has a higher value, it appears before connection 1 in the ordering.

[0178] A set of ordering criteria in a flow class (e.g., transmission requirements) enables scheduling packets on available connections that are optimal for the type of flow to which they belong. For example, packets belonging to the "video encoder" flow class can preferentially use the connection with the lowest latency, while packets belonging to the "file transfer" flow can prefer high-throughput connections even if the latency is high or highly variable.

[0179] Once an ordering for the connections is determined, step 2d (block 1110 shown in FIG. 1E) uses the ordering to allocate packets to connections with available bandwidth. The packets are placed on a transmission queue for the connection.

[0180] FIG. 4D shows, in 400D, selecting a connection for a given packet using the ordering from FIG. 4C. In this example, the first connection in the ordering is connection 2, but connection 2 does not have an available congestion window and is thus not currently eligible to transmit the packet. Next, connection 1 is checked and since it can transmit the packet, connection 1 is selected and the packet is queued to be transmitted on the stage 3 pipeline for connection 1.

[0181] Next, the in-flight byte connection metadata (monitored characteristics) is adjusted as part of step 2e (block 1112 shown in FIG. 1E). In some embodiments, this includes distinguishing goodput bytes from retransmit / dummy / probing bytes when tracking CWND consumption for the connection. Such adjustments can affect the goodput and / or potential goodput for a priority group, which can activate the next priority level of connections in the next iteration of the loop in step 2. Various methods for converting from units of volume (CWND bytes) to rate (goodput) are considered following the descriptions for FIGS. 8A - 8C and FIGS. 9A - 9C.

[0182] Upon reaching Step 3, for each connection, either the available CWND is full or there are no packets eligible for transmission. Packets queued in the output queue for each connection in Step 2 are then sent to the appropriate instance of the next stage of the pipeline.

[0183] In some embodiments, an explicit objective function can be used to determine the desired trade-off when there are multiple flows belonging to flow classes that have the same priority but competing requirements.

[0184] The embodiments described in FIGS. 4A - 4D show an objective function that implements fairness among flows with the same priority. The selector services each flow that matches the flow class in a round-robin-like fashion.

[0185] The fair implementation in this embodiment uses the deficit round-robin scheduling algorithm described in RFC8290. A brief overview of this algorithm is that it utilizes a "byte credit" scheme and assigns a quantum of byte credits to each queue in each round of iteration over the entire queue.

[0186] The number of bytes that can be obtained for transmission from each queue in each round is nominally limited by the number of available credits the queue has. When the byte credit quantum for each queue is equal, fairness naturally occurs.

[0187] In some embodiments, the objective function may allow for the intentional configuration of unfairness by an administrator or end user. For example, weights can be configured on matching flows, which can ultimately result in unequal byte credit quanta given to the queues in each iteration.

[0188] For example, there can be five different weight levels: ● Highest (5) ● High (4) ● Standard (3) ● Low (2) ● Minimum (1)

[0189] The values assigned to each weight represent the ratio between the byte credit quantum of these queues. For example, if the quantum corresponding to the standard is 30KB, then for a flow marked as low, it will receive a quantum of (2 / 3)*30KB = 20KB in each round. For a flow marked as highest, it will receive (5 / 3)*30KB = 50KB.

[0190] Note that the absolute numbers in this example are not important. If the standard quantum were 5KB instead of 30KB, the weights would still scale the relative quanta for the other priority levels appropriately, and the final result would be the same in terms of overall fairness.

[0191] Other embodiments may allow the weights to be arbitrary integers rather than the five fixed values in the above example. This would allow an administrator or end user to configure a greater unfairness between flows in the objective function, if desired.

[0192] Also, as with all Quality of Service (QoS), note that these weight settings are only important if there is contention for the available WAN capacity. In the absence of contention, it means that all the data within all queues fits completely within the available capacity, so the ratio between queue usage amounts is the natural ratio from applications generating different volumes of data.

[0193] Other embodiments may have completely different objective functions that do not consider fairness at all. For example, ● Guarantee that the maximum number of flows is satisfied by the available WAN capacity, ● Maximize the total throughput achieved by the serviced flows, or ●Minimize changes to the current bandwidth allocation on the BoD connection.

[0194] In some embodiments, the way the WAN connection is used can be modified, and the WAN connection can be made unqualified to serve a particular flow class. For example, some types of connections can achieve high throughput but at the expense of causing a higher RTT.

[0195] In one embodiment with an objective function that attempts to maximize throughput, it would be selected to serve flows with high throughput requirements, which would increase the WAN connection RTT and make the WAN connection unqualified to serve flow classes with low latency requirements.

[0196] Conversely, in one embodiment with an objective function that attempts to serve the maximum number of flows, the opposite trade-off can be selected so that the low latency flow class can continue to be served by this WAN connection.

[0197] In some embodiments, the connection sorting order of individual schedulers is maintained per flow rather than globally for all flows.

[0198] This allows flow-specific requirements to influence the sorting order so that packets of a flow are assigned to a connection in a way that meets the needs of the connection.

[0199] For example, a VoIP flow with a preferred maximum latency tolerance of 150 ms would prefer a sorting criterion that places connections with an RTT exceeding 150 ms at the bottom of the list. Conversely, a TCP flow without specific latency requirements would prefer to prioritize connection throughput in the sorting criterion for this TCP flow, either by completely ignoring the RTT or using the RTT only as one of many secondary criteria.

[0200] In some embodiments, these flow requirements and preferences are configured in the form of rules consisting of match criteria and resulting behavior.

[0201] For example, the match criteria can take the form of an IP3 tuple (protocol, source IP, and destination IP) or an IP5 tuple (protocol, source IP, source port, destination IP, and destination port). The behavior takes the form of an explicit preference for latency or throughput and has a target preferred value for either.

[0202] Examples of other heuristic-based match criteria can include the following: ○ DSCP tag ○ Any of the other header fields available in the IP (v4 or v6) header ○ Any of the other header fields within the TCP header or UDP header ○ Cumulative volume of transmitted data, e.g., a TCP flow exceeding a volume threshold may match as "bulk data transfer" whereas those that do not may not match as an interactive SSH / telnet session ○ Cumulative duration of the flow ○ Regular expression match on the packet payload ○ Clear text parameters extracted from the TLS handshake (e.g., SNI, CN, SubjectAltName)

[0203] Examples of other operations that can affect the scheduler sort order can include the following: ○ Specific preference for connection order (e.g., the user may prefer to sort expensive connections to the bottom of the list for low-priority flows) ○ Jitter tolerance ○ Maximum latency tolerance ○ Desired amount of redundancy (e.g., FEC) ○ Whether retransmission of lost packets is desired and, if so, whether there is a maximum time limit at which retransmission attempts should stop ○ Whether explicit packet pacing is desired i. For example, in a real-time video application that transmits video at 30 frames per second, the video frames of the video can be transmitted at exactly 33.3 ms intervals, and the multi-path system 100B would not be expected to change the pacing. ii. In contrast, for a burst application, it can benefit from having the scheduler explicitly pace the bursts of video at the aggregated WAN rate. ○ Desired throughput (e.g., minimum, target, maximum)

[0204] Using machine learning techniques to analyze traffic patterns, rules (both match criteria and actions) can be automatically generated.

[0205] Inputs (features) to the ML method can include the following: i. Cumulative volume of data transmitted ii. Packet size distribution (histogram) iii. Inter-packet intervals (both pacing and jitter) iv. Packet frequency and grouping v. Packet header fields (Ethernet, IP, TCP, UDP, etc.) vi. Packet content (payload) vii. Not only IP addressing, but also the intended destination of the packet (e.g., SNI, CN, and SubjectAltName fields in the TLS handshake and exchanged certificates) viii. Cumulative duration of the flow ix. Time x. Number of concurrent flows to the same destination or from the same source xi. Prediction (label) of the current or history of any simultaneous flows to the same destination or from the same source (e.g., some applications open multiple flows, one as a control plane and another as a data plane. Knowing the existence of an existing control plane flow can help predict the data plane flow).

[0206] Predictions (labels) from the ML method would include any of the aforementioned behaviors determined based on the behavior of the training corpus.

[0207] In some embodiments, the rules can have the concept of subordinate rules that apply additional matching criteria and more specific behaviors while still being managed by a parent rule. For example, ○ A VPN flow identified by an IP5 tuple can have rules defined by matching criteria and actions. This would be considered a "parent" rule. ○ The VPN will carry many internal flows from the applications it protects within the VPN. Typically, these internal flows will be completely hidden from the matching criteria (encrypted by the VPN). ○ However, some VPNs may choose to expose a limited set of information about internal flows by setting DSCP tags on encrypted packets corresponding to the preference for internal (clear text) packets.

[0208] Subordinate rules that match the DSCP tag can be created, but the resulting available behaviors will still be managed by the parent rule. For example, the following are some cases. a. Generally, a VPN is required to deliver packets in order (a VPN can tolerate out-of-order delivery, but only up to a small window). b. The behavior composed of subordinate rules that match the DSCP tag should not result in the parent rule delivering packets in a significantly out-of-order manner.

[0209] For example, a dependent rule that requires a maximum latency limit of 150 ms and a second dependent rule that requires maximum throughput may violate the ordering requirements of the parent rule because the interleaved packets from these two dependent rules may be assigned to a connection that has significantly different latencies.

[0210] Figure 5A illustrates two independent flows where each rule and behavior is completely independent. In this example, for the SIP flow, since it requires low latency, the behavior is to have the scheduler 104B transmit only the packets of the SIP flow on connection 1. Conversely, for the FTP flow, since it requires high throughput, the scheduler 104B transmits only the packets of the FTP flow on connection 2.

[0211] When the packets for these two flows are transmitted wirelessly and reach the sequencer 118, these packets are returned in order and transmitted independently to their final destinations. In this example, the sequencer 118 transmits the packets belonging to the SIP flow that have already arrived and are in the correct order, regardless of the state of the FTP flow whose packets are still being transmitted via wireless.

[0212] Figure 5B illustrates the concept of a dependent flow. In this example, the same SIP and FTP flows exist, but before the scheduler 104B sees these flows, these flows pass through the VPN client for encryption and encapsulation.

[0213] When coming out of the VPN client, the packets from both flows have the same IP5 tuple, so the scheduler 104B and the sequencer 118 treat this as a parent flow and are constrained by the requirements of the parent flow.

[0214] The presence of the dependent flows (SIP and FTP) is communicated using DSCP tags, which control how the scheduler 104B allocates packets to connections 1 and 2 for transmission. In this example, the allocation is the same as that in FIG. 5A.

[0215] However, the parent flow constraint requires that packets be transmitted from the sequencer 118 in the same order in which they reach the scheduler 104B. Thus, in this example, even if packets belonging to the dependent SIP flow have already reached the sequencer 118, packets belonging to the dependent SIP flow cannot be transmitted until packets belonging to the dependent FTP flow have arrived and are transmitted first.

[0216] FIG. 6A shows, at 600A, a sample embodiment of a rule management page. The configured rules are displayed in a table, with each row summarizing the match criteria and behavior. The order in which the rules are listed is the order in which packets match the rules. FIG. 6B shows, at 600B, a workflow for changing the order of the rules.

[0217] FIG. 6C shows, at 600C, how each row in the table can be expanded to show the details of the match criteria and behavior, as well as the currently active flows passing through the system that match this rule.

[0218] Each rule in the table includes edit and delete icons.

[0219] FIG. 6D shows, at 600D, a dialog window that appears when the edit button is clicked. The match criteria and behavior for the rule can be changed in this dialog.

[0220] The main rule management screen has a separate [Add Rule] link, and FIG. 6E shows, at 600E, a mode dialog that appears when clicked.

[0221] In some embodiments, the rules are configured on the transmitter side and are automatically pushed to the receiver. By default, the rules are symmetric and bi-directional. That is, when a rule is added and the behavior of the rule is configured, the matching pair rule is transparently added, but the match criteria for the source and destination are swapped. It would be possible to add asymmetric rules (different behaviors for either direction of the same flow) via an advanced configuration screen.

[0222] In one embodiment, the scheduler probes the various network characteristics of multiple connections (to determine the measured characteristics for each connection) and uses the measured characteristics in combination with the transmission requirements to make decisions regarding packet transmission.

[0223] For example, one such decision is to determine the number of packets to transmit on a particular connection and when to transmit these packets.

[0224] Some embodiments are volume-based. Such embodiments operate by determining a limit (e.g., in units of bytes or packets) on the volume of data that can be transmitted on a particular connection. When enough packets have been transmitted for that volume, transmission stops until some feedback regarding these packets is received. If feedback is received before the full volume limit is transmitted, transmission can continue without stopping as long as the volume of transmitted data for which feedback has not been received does not exceed the determined volume limit.

[0225] In one embodiment, the volume of data is selected to be the congestion window (CWND) of the connection. Briefly, the CWND is the maximum volume of data that can be transmitted on a connection without causing significant congestion on that connection before receiving any feedback. There are a number of methods for estimating the CWND. It is assumed that the scheduler can access one such method and receive an estimated value of the CWND through that method.

[0226] In some embodiments, the scheduler is required to determine the rate at which data is being transmitted (e.g., in units of bytes per second). This rate can then be used to make other decisions regarding packet transmission.

[0227] For example, in one embodiment, if the rate of data being transmitted on the currently active network connection(s) is less than the rate of data being received from the source application for transmission, the scheduler can determine to activate more connections to increase the aggregate transmission rate to match the application rate.

[0228] In another example, an embodiment may need to meet quality of service (QoS) requirements that can be expressed in terms of a guaranteed transmission rate. If the aggregate rate falls below the required level, the scheduler can determine to activate more connections to increase the aggregate transmission rate to meet the QoS requirements.

[0229] In yet another example, an embodiment may be required to measure goodput (i.e., the rate at which application data successfully passes through the network) for purposes such as guaranteeing application performance levels, optimizing performance, and reporting.

[0230] In one embodiment, TCP performance can be optimized by pacing the packets being sent at an aggregated rate achieved by multiple connections. See, for example, PCT Application No. PCT / CA2020 / 051090.

[0231] In embodiments where a rate is required for some functions, a volume-based scheduler requires a way to convert the volume of data being transmitted into a rate.

[0232] The trivial technique of converting volume to rate by dividing the volume by the duration of the time over which the volume was transmitted generally results in inaccurate results. Such a technique measures that the transmission rate at the transmitter, rather than the actual rate data, is moving through the network. The actual rate at which data moves through the network is generally truly determined by the rate on the slowest link of a multi-link network, generally referred to as the network bottleneck.

[0233] The characteristics of the network bottleneck link are not available to a transmitter or receiver that knows that a multi-link network can change the combination of links that are transparently used for the transmitter and receiver at any time. However, congestion control methods can generate estimates based on the performance observed at the transmitter and receiver.

[0234] In some embodiments, the congestion control method can provide an estimate of the transmission rate of the bottleneck link. This may be referred to as the network bottleneck bandwidth BtlBw.

[0235] In some embodiments, the network round-trip propagation time RtProp can be estimated by sampling the time between when a packet is sent and when feedback regarding the packet is received.

[0236] Using the combination of BtlBw and RtProp, the following can be used to determine a network property called the bandwidth delay product (BDP).

[0237] BDP = BtlBw × RtProp

[0238] The most common unit for BDP is bytes. For these quantities, the following units are assumed: (i) bytes per second for BtlBw, (ii) seconds for RtProp, and thus (iii) bytes for BDP.

[0239] BDP means the amount of data that would be present in the network (in - flight) when operating under ideal conditions, assuming that the data is always available for transmission. This network transfers data at the nominal rate of the network BtlBw and will provide feedback on every packet exactly after the RtProp time it takes to transmit that packet. Figure 7 shows two examples at 700. The upper half of Figure 7 illustrates this aspect using an example where the BDP consists of 16 equal - size packets. In practice, the amount and size of the packets can vary as long as the total size is equal to the value calculated using the BDP.

[0240] In some volume - based embodiments, the BDP can be used to estimate various data transfer rates, some of which were described above.

[0241] For the in - flight volume under consideration, assuming ideal conditions, if the volume is less than the BDP, only a fractional value of the nominal rate of the network, BtlBw, can be achieved. This fractional value can be calculated as the ratio of the volume to the BDP. The resulting achievable rate is determined as follows:

[0242] Achievable rate corresponding to the in - flight volume = Volume / BDP × BtlBw

[0243] In reality, it is rare for a network to operate under ideal conditions. For example, packets may be temporarily buffered before being delivered to the receiving end, or feedback may be delayed at the receiver or by the network. In some cases, packets are lost and never reach the receiving side, thereby preventing feedback from being triggered. In other cases, the feedback itself may be lost. Losses can be estimated, for example, when feedback for new packets is received or based on timer expiration.

[0244] Waiting for delayed feedback (which may or may not arrive) to trigger the next transmission event can artificially limit the achievable rate in the network.

[0245] FIG. 8A shows an example of a receiver that aggregates feedback and returns an acknowledgment to the transmitter once every four received packets in FIG. 800A.

[0246] FIG. 8B continues to illustrate in FIG. 800B how, for example, the aggregation of acknowledgments results in the overall throughput being less than the capacity of the network. The transmitter waits a longer time before starting to transmit the next packet group. Overall, the transmitter eventually transmits the same amount of packets as in the ideal case of FIG. 7, but over a longer period, resulting in a throughput degradation.

[0247] In one embodiment, this artificial limitation is overcome by specifying an additional allowance of data to send even when feedback for previously sent packets has not yet been received. This means that the in - flight data volume can exceed the BDP. However, in such cases, as the above - achieved rate formula application might suggest, the network is not actually transferring data at a higher rate. The additional allowance simply enables the transmitter to maintain a constant rate equal to BtlBw over a longer period. Thus, the estimated transfer rate of the local volume is capped at BtlBw.

[0248] FIG. 8C in FIG. 800C subsequently illustrates how such additional transmission allowance is used, for example, by the transmitter to continue sending packets before an acknowledgment is received. The in - flight data volume, 19 packets, exceeds the BDP of 16 packets. However, the actual rate matches the network's capacity, and capping the calculated rate using the minimum of the in - flight volume and the BDP achieves the same result. In some embodiments, a similar approach can be used to subsequently estimate what is hereinafter referred to as the potential goodput, which is the goodput that can be achieved on the network in some cases.

[0249] A particular volume of data can be divided into a goodput portion and other portions. The goodput portion includes newly transmitted application data. The other portions include re - transmissions of data that cannot be counted as newly transmitted application data, such as previously lost data, control data, probing data, etc.

[0250] When the total volume of data is greater than the BDP, the packet and / or feedback delay is assumed as above. Assume there is no extra space available for additional goodput. The resulting (potential) goodput rate is estimated as follows:

[0251] Goodput rate = Goodput part / Total volume × BtlBw

[0252] For example, FIG. 9A shows a network having a BDP of 16 packets at 900A. To achieve a throughput equal to the capacity of the network, the in - flight volume is 19 packets. However, only 14 packets are goodput, resulting in a goodput rate of 7 / 8 of the network capacity. When the volume of data is smaller than the BDP, the difference is considered to be the extra space assumed to be available for additional goodput.

[0253] FIG. 9B shows an example at 900B where only 3 packets are in - flight, resulting in an extra space of 13 packets that can be assumed to be goodput when calculating the potential goodput rate. This extra space must be limited by the CWND determined by the congestion control method. At any given time, the additional volume allowed by the congestion control is the difference between the CWND and the total volume. If the extra space for goodput exceeds this congestion control allowance, the space is simply limited to the congestion control allowance.

[0254] FIG. 9C shows an example at 900C where the CWND is reduced to 11 packets, which limits the extra space assumed to be available for additional goodput to 8 packets.

[0255] The resulting potential goodput rate is estimated as follows:

[0256] Potential goodput rate = (Goodput part + Extra space for limited goodput) / BDP × BtlBw

[0257] Continuing with the above example, FIGS. 9B and 9C show how the above potential goodput formula results in a lower potential goodput rate for the scenario in FIG. 9C, which is as expected considering the CWND reduction performed by the congestion control method.

[0258] The above approach is described in terms of goodput and potential goodput rate, but this approach can be easily modified to calculate the rate or potential rate of any type of data that partially constitutes the total volume.

[0259] The following is a numerical example of applying the above formula to a network having properties typical of what is encountered via an LTE cellular network. BtlBw = 10 Mbps RtProp = 60 ms BDP = 10 Mbps * 60 ms = 75 KB CWND = 60 KB Goodput portion = 25 KB Other portion = 10 KB Total volume = 25 KB + 10 KB = 35 KB Extra space for goodput = BDP - Total volume = 75 KB - 35 KB = 40 KB Congestion control allowance = CWND - Total volume = 60 KB - 35 KB = 25 KB Extra space for limited goodput = Min(Extra space for goodput, Congestion control allowance) = Min(40 KB, 25 KB) = 25 KB Potential goodput rate = (Goodput portion + Extra space for limited goodput) / BDP × BtlBw = (25 KB + 25 KB) / 75 KB × 10 Mbps = 50 / 75 × 10 Mbps = 6.67 Mbps

[0260] Systems that communicate over an IP network often implement a congestion control mechanism that attempts to achieve fair network utilization by all network nodes that employ a similar congestion control mechanism.

[0261] The congestion control mechanism often operates by monitoring the communications occurring on the network, and in some cases, probing the network as needed and deriving network properties based on the results. A model representing the network is created, and subsequent communications are guided based on this model. For example, this model can track network properties such as the current maximum throughput achieved, the current and minimum round-trip times of packets, packet loss events, and the like.

[0262] Based on the created model and the tracked information, the congestion control mechanism can determine when the network is underperforming, indicating that the network may be experiencing congestion. The congestion control mechanism can then take corrective measures aimed at reducing or eliminating network congestion and restoring desirable network performance.

[0263] In previous congestion control embodiments (e.g., TCP), packet loss has been used as an indicator of network congestion, and corrective measures (e.g., reducing CWND) have been taken in response to the loss.

[0264] In one embodiment, the latency of a packet, i.e., the time required for the packet to pass through the network from the transmitter to the receiver, is an indicator of network performance. Packets with high latency may, in some cases, be useless to the receiver and be discarded, potentially wasting the transmission of these packets. For example, real-time VoIP applications that require low conversation latency between participants are designed not to use packets that carry voice data older than the maximum allowable conversation latency.

[0265] In such an embodiment, the congestion control mechanism may monitor and attempt to control the latency of packets so that the latency of the packets does not exceed an acceptable level.

[0266] In one embodiment, the minimum latency measured for a packet can form a baseline value, and subsequent measured values can be compared to this baseline value. If a subsequent measured value exceeds a threshold that is a function of the baseline value, the congestion control mechanism considers that the network is experiencing congestion.

[0267] In another embodiment, a similar approach uses the round-trip time of a packet, which is the time elapsed between sending the packet and receiving an acknowledgment response regarding it, instead of latency.

[0268] In one embodiment, when the congestion control mechanism determines that the network is experiencing congestion based on increased packet delay, the congestion window (CWND) is reduced to limit the volume of data that can be transmitted on this network without receiving any feedback. The goal is to reduce the volume of data that will be buffered by the network or that is currently being buffered, which is a common cause of increased packet delay.

[0269] In one embodiment, the congestion window is reduced to the bandwidth-delay product (BDP) of the network. Theoretically, the BDP represents the volume of data that the network should be able to deliver without forming a queue of packets with no movement within its buffer, assuming that the packets are transmitted by the transmitter at a constant pace reflecting the network's throughput (BtlBw). Therefore, this is expected to enable the network to at least partially recover and reduce the packet delay back to an acceptable level.

[0270] For example, assume a network with a BDP of 16 packets and a congestion control mechanism that determines a CWND of 24 packets to account for acknowledgment aggregation. FIG. 10A illustrates, in 1000A, how congestion from other network nodes can cause the network to throttle the transmitter. In this case, the transmitter does not reduce its congestion window and inflights the full amount. This causes the network buffer to fill up and excess packets that cannot be buffered are dropped. Even after the congestion is relieved and normal drainage of the network buffer continues, FIG. 10A illustrates how a motionless queue that would increase the round-trip time of all subsequent packets and more easily fill the network buffer would be formed inside the network buffer.

[0271] Continuing with this example, FIG. 10B illustrates, in 1000B, how a transmitter that reduces its congestion window to the BDP can avoid both packet drops and the formation of a motionless queue. The transmitter can determine that congestion is occurring, for example, by detecting that no feedback has been received for a period exceeding a baseline duration for feedback measured in past transmissions. In another example, the transmitter can determine that congestion is occurring by receiving feedback indicating that the amount of packets previously transmitted that exceeded a baseline loss encountered in past transmissions has not been delivered to the receiver. In response to this determination of congestion, reducing the congestion window to match the BDP reduces the amount of packets that will be in flight. Consequently, this reduces the likelihood of filling the network buffer that would have otherwise led to packet drops. Further, inflighting only BDP packets allows the network to fully drain the network buffer before new packets from the transmitter reach the network buffer again (assuming no network properties have changed).

[0272] In another embodiment, the congestion window is reduced to be equal to the volume of packets currently in flight.

[0273] When the congestion control mechanism determines that an acceptable level of performance has been restored, the network is considered to have stopped experiencing congestion.

[0274] In one embodiment, a reduction in packet delay below a threshold is an indicator that network performance has been restored.

[0275] In one embodiment, the threshold is determined as a function of the baseline delay value.

[0276] In another embodiment, transmitting a certain volume of data without incurring any loss is an indicator that network performance has been restored.

[0277] In one embodiment, the volume of that data is equal to the BDP.

[0278] In another embodiment, the volume of that data is equal to the original value of the congestion window before being reduced by the congestion control mechanism in response to congestion.

[0279] The push-type architecture improves system performance in non-BoD connections of the most common application types. Table 3 summarizes the experimentally observed improvements.

Table 4

[0280] The push-type scheduling method also makes the priority routing behavior more intuitive. For example, in one embodiment, the decision of whether to involve lower-priority connections depends only on how the BtlBw of these connections is compared to a configured threshold.

[0281] For example, consider the following scenario:

[0282] wan0 ● Priority: Primary ● Target Threshold: Not applicable ● BtlBw: 10 Mbps

[0283] wan1 ○ Priority: Secondary ○ Target Threshold: 15 Mbps ○ BtlBw: 30 Mbps

[0284] When using a pull-type scheduler, every time the 50 Hz timer of wan1 wakes up and requests a packet, the scheduler knows that the desired target threshold (15 Mbps) is greater than the threshold (10 Mbps) that wan0 can provide alone. Therefore, the scheduler provides packets for transmission to wan1.

[0285] However, the scheduler does this regardless of the rate at which packets arrive at the input. For example, if the input rate does not exceed 10 Mbps, the traffic can be completely handled by wan0 alone, so it would be more optimal if wan1 were not involved.

[0286] When using a push-type scheduler, wan0 will be sorted to the top of the list, and packets will be scheduled on wan0 only after the wan0 CWND is completely consumed, even if the threshold indicates that wan1 should be active. As a result, the unexpected use of lower-priority connections is reduced, and performance is improved.

[0287] In summary, the scheduler of the multipath system can be implemented as either a pull type (100A) or a push type (100B). Since the scheduler can make packet scheduling decisions based on self-determined rhythms rather than being driven by the rate at which the target connection is ready to transmit, the push type is superior. This self-determined rhythm makes it possible to consider all factors that are important to the application that generated the packet and / or the user of the system.

[0288] For example, packets generated by a TCP-based application that requires maximum throughput will prefer to be scheduled on connections that are ordered from the best Mathis coefficient to the worst Mathis coefficient.

[0289] Also, consider a more complex example using the same TCP-based application, but where the user has configured the multipath system 100B with a priority routing rule to prefer connection 110B over all other connections 110N as long as the goodput achievable at 110B does not drop below 5 Mbps. In a scenario where 110B has high throughput, low latency, but high packet loss, 110B will have a worse Mathis coefficient than all other connections 110N.

[0290] The push type scheduler 100B can make complex scheduling decisions that respect the user preference to use connection 110B first, and can also calculate the goodput achievable at 110B by converting the in-flight CWND of connection 110B to a bit rate and using that to estimate the available goodput of connection 110B. If this availability drops below the configured 5 Mbps threshold, 110B can schedule packets on the remaining connections 110N that are ordered from the best Mathis coefficient to the worst Mathis coefficient.

[0291] As described above, the measured properties of a connection can include packet loss (further aggregated as the worst value among three windows) between three separate sliding windows (500 ms, 3 s, 30 s), the current network round-trip time (RTT), the bandwidth round-trip propagation time (BBRv1) parameter, RtProp (network propagation delay), and BtlBw (throughput estimate). External directives can include, among other things, priority routing (PR) rules, but can also include flow-specific rules / requirements.

[0292] In some embodiments, the measured properties of BBRv1 are different from these designs in the original BBRv1 IETF draft. For example, considering embodiments that can operate on systems with low system runtime consistency, the size of the sliding window used to calculate BtlBw can vary, either statically or dynamically. Also, for similar reasons of execution consistency, the input parameters used to calculate RtProp can be slightly modified.

[0293] Some or all of the input parameters can be rounded / bucketed / quantized before being used in the sort order determination. The technical improvement from rounding and bucketing is to keep the sort order as "stable" as possible so that sort order changes do not always occur (e.g., without rounding and bucketing, two close values can cause "jitter" by having continuous sort changes as their values vary over time).

[0294] Factors affecting the determination of the sort order include keeping the sort order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (splitting increases the machinery of self-induced out-of-order events and jitter events), and considering both the measured properties of the connection and the unmeasured external directives.

[0295] Values for rounding can include the following: ● Latency-related values (RtProp, RTT) are floored 1 ms after rounding and rounded to the nearest multiple of 50 ms. ● Throughput-related value (BtlBw) is rounded to the "nearest digit". ● Packet loss (the worst value among three sliding windows) is rounded to the nearest digit. Note that since the raw value here is limited to the range [0, 100], the output value is limited to the set (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100).

[0296] An example of rounding to the nearest digit is provided in the following table. The raw value, the calculated digit of the raw value, and then the resulting rounded value are shown.

Table 5

[0297] Specific improvements based on improving the networking routing order to control the use of network interfaces are described in some embodiments that utilize the combined properties of round-trip time (RTT) and packet loss to predict throughput based on the Mathis equation for establishing an upper limit of given TCP throughput for RTT, packet loss, and MSS. The Mathis coefficient can be used to modify the sort order for routing networked packets.

[0298] The properties to be measured can include packet loss (further aggregated as the worst value among three windows) during three separate sliding windows (500 ms, 3 s, 30 s), the current network round-trip time (RTT), the bandwidth round-trip propagation time (BBRv1) parameter, RtProp (network propagation delay), and BtlBw (throughput estimate). External directives can include, among other things, priority routing (PR) rules, but can also include flow-specific rules / requirements.

[0299] In some embodiments, the measured properties of BBRv1 are different from the design of these characteristics in the original BBRv1 IETF draft. For example, to account for embodiments that may be run on a system lacking consistency in execution time, the size of the sliding window used to calculate BtlBw can be increased to cover a longer duration (to hide variability due to execution non-consistency), but if network loss or latency increase events occur, those events can indicate that BtlBw can actually change and any long history in the sliding window must be forgotten, so the window duration can also be immediately shortened.

[0300] For similar reasons of execution non-uniformity, the input parameters used to measure RtProp are changed to include more than just network propagation delay and may add a portion of the execution / processing time, which may be non-trivial and consistently contribute to the actual delay encountered when transmitting packets and waiting for responses. In some embodiments, this is referred to as "system RtProp" as opposed to "network RtProp" which consists only of network propagation delay. In these embodiments, "system RtProp" and "network RtProp" can be used for various calculations depending on the intended purpose. For example, "system RtProp" is appropriate when calculating CWND to include sufficient data in-flight for the pipeline to account for processing delays. However, "network RtProp" is more appropriate when sorting network connections to determine packet routing as the preferred routing decision is independent of system execution delay.

[0301] Packet loss, latency, and throughput are not completely independent variables. Their three combinations affect each other depending on the congestion control behavior of the application. This is because, for example, with TCP, higher packet loss results in a longer period of wireless "silence" (underutilized channel) while the transmitter waits for DUP ACK or RTO to occur.

[0302] Accordingly, several composite properties are determined and used in the final sorting approach. Specifically,

[0303] Mathis Coefficient: This composite property is based on the Mathis Relation, which predicts throughput considering RTT and packet loss and gives the upper bound of TCP throughput considering RTT, packet loss, and MSS. When comparing the properties of any two network connections to determine their sort order, since MSS can be considered a constant, this relation shows that the expected throughput is proportional to the reciprocal of (RTT * sqrt(packet loss)). In this example, this relation is used as a practical implementation form to establish the coefficients used when directly controlling the operation of a network controller or router. For each connection, the system determines the Mathis coefficient as follows: MAX(rounded Rtt, 1ms) * sqrt(rounded aggregated packet loss). Connections with smaller Mathis coefficient values are considered better. Since the calculation is a function of two independently measured properties, the relationship between these characteristics determines how the connections compare to each other. The Mathis coefficient is held and stored in data objects such as the corresponding Mathis coefficient data value associated with each connection and can be stored, for example, in an array of data values along with other monitored characteristics.

[0304] For example, in one extreme case, two connections with 0% packet loss will always have the same Mathis coefficient, and the actual RTT values of these connections are irrelevant. In this scenario, the Mathis coefficient does not affect the sort order and is determined by comparing the subsequent measured properties with the composite property.

[0305] In the other extreme case, when two connections have 100% packet loss (rounded aggregated packet loss = 1.0), only the rounded Rtt values of these connections determine how they compare to each other. In some embodiments, this is undesirable, so connections with 100% packet loss are filtered from the available options at an earlier stage.

[0306] Between two extremes, a connection having both low RTT and low packet loss comes to be compared as being better than a connection having higher values. However, a connection with a high percentage of loss can still be compensated by having a low RTT and can have the same Mathis coefficient as a connection with a low percentage of loss. For example,

[0307] Connection A: 100ms RTT, 0.01 (1%) loss => Mathis coefficient = 10

[0308] Connection B: 19ms RTT, 0.25 (25%) loss => Mathis coefficient = 9.5

[0309] In this example, even though Connection B has significantly higher loss (25% versus 1%), the lower RTT of Connection B compensates for it and rivals a better Mathis coefficient than Connection A. The explanation for this is that the lower RTT of Connection B exchanges packet acknowledgments (positive and negative) for the lost packets and, even though Connection A has a smaller percentage of loss, is faster than Connection A with a higher RTT and enables any necessary packet retransmissions to be completed.

[0310] Capacity coefficient: This composite property takes into account BtlBw and RtProp and is determined as follows: Rounded BtlBw / MAX(rounded RtProp, 1ms). The unit is "bytes / second per millisecond" and has no physical meaning, but provides a way to normalize the bandwidth that a connection can achieve per unit of RtProp. In other words, a connection that can achieve more bandwidth per millisecond of its RtProp is considered more efficient than a connection that achieves less bandwidth. Efficient transmission in this context means a connection that achieves more bandwidth per millisecond of propagation delay (RtProp).

[0311] Once all of the raw input has been processed and the composite properties have been calculated as described above, the actual sorting mechanism can operate continuously as follows.

[0312] The transition from one comparison to the next occurs only if all of the previous rows are compared as equal. If the comparison is not equal, the remainder of the comparison is irrelevant and is omitted. In some embodiments, the selection of comparisons is done on a flow classification basis (e.g., VPN vs. VoIP vs. ping packet vs. FEC packet), and for each different flow, the list of comparison criteria can be different (e.g., each can have different predefined most important characteristics, second most important characteristics, third most important characteristics, etc.). Each of these characteristics can be iterated through one after the other from the most important, etc., until the connection itself can be sorted and placed into an ordered sequence. In some embodiments, depending on the flow classification, the first pre-step is to select a subset of connections that are suitable for transmitting the type of data packet in the flow classification, and then an order is assigned to them before packet allocation is performed. 1. A connection that is completely down (100% aggregated loss) is considered a worse connection than a connection with less than 100% aggregated loss. 2. PR priority (higher priority is better) 3. Mathis coefficient (lower is better) 4. Rounded RTT (lower is better) 5. Capacity coefficient (higher is better) 6. Rounded RtProp (lower is better) 7. Rounded BtlBw (higher is better) 8. Connection ID (lower is better)

[0313] The rendering of the sorting mechanism is shown in Figure 1F. In an exemplary sorting mechanism, the real-time flow first includes a comparison of RTT / RtProp, then includes an equality comparison, then the real-time flow compares reliability, and then, if still equal, compares things like transmission speed. Depending on the flow type, there can be different sorting orders. For example, a flow that desires throughput may first compare the transmission rate. If equal, it compares by reliability. If still equal, the real-time flow may decide to evaluate nothing else and simply make an arbitrary selection.

[0314] In some embodiments, the flow requirements (e.g., real-time vs. throughput) can determine the order and number of comparisons. For example, a real-time flow may prefer to compare only the rounded RTT and the rounded RtProp, ignoring all other properties. A throughput flow may decide to compare only the Mathis coefficient and the capacity coefficient.

[0315] Most application traffic tends to be bursty or adaptive and thus does not require the use of all WAN connections. This means that the connections that are currently at the bottom of the sort order may not have traffic transmitted over these connections. If these connections were not like this, since the explicit sort dimensions of these connections are not updated, these connections tend to maintain their position near the bottom of the sort order.

[0316] This is generally regarded as a good property of the algorithm, but can be modified to be affected by external factors. For example, if a flow has specific requirements that cannot be met by the connections near the top of the current sort order, it may make sense to generate probing traffic on the connections near the bottom to refresh the explicit sort dimensions of these connections.

[0317] For example, as will be described in more detail below, in some embodiments, these connections can be given a score of 0 in layer grouping, so that these connections can be used to send packets and scheduling can continue. For example, send early retransmissions on old or unreliable connections to refresh the connection and check whether the reliability of the connection has increased.

[0318] FIG. 11 illustrates a hybrid connection aggregation system 1100 configured to utilize an improved scheduling approach on the transmission portion of the system and a buffering system on the receiving end using packet sequencing. In one embodiment, the components in the system as illustrated are hardware components configured to interact with each other. In another embodiment, the hardware components are not separate components, but rather two or more components can be implemented on a particular hardware component (e.g., a computer chip that performs two or more of the functions of the components).

[0319] In some embodiments, the hardware components reside on the same platform (e.g., the same printed circuit board), and the system 1100 is a single device that is transportable and connectable to a data center / field portable device (e.g., a rugged mobile transmitter), etc. In another embodiment, the components are distributed and not all are located in close proximity; rather, they communicate electronically through telecommunications (e.g., not executed locally, but rather the processing and control are performed by components residing in a distributed resource environment (e.g., the cloud)).

[0320] The provision of mixed connectivity is particularly desirable in mobile scenarios where signal quality, network availability, quality networks, etc. are not optimal (e.g., professional news gathering / video creation can be done in locations without a strong network infrastructure). Thus, in some embodiments, the devices described herein are ruggedized network controller devices adapted for operation in remote or mobile scenarios, which can be, for example, man-portable (e.g., worn in a backpack by a journalist engineer), cart-portable (e.g., pushed along a media cart), or vehicle-portable (e.g., coupled to a media van, truck, boat, helicopter, or airplane).

[0321] Alternatively, the device can be located in a communication hub or other type of centralized communication facility that coordinates connections to additional endpoints. For example, in this instance, the device can be located in, coupled to, or present in a communication facility such as a communication relay site (e.g., a satellite station, a repeater transmitter, a broadcast translator, a rebroadcaster, a repeater, a complementary station) that can coordinate communication across several different channels to various endpoints, which can be, among other things, a television station, a radio station, a data communication station, a personal computer, a mobile device.

[0322] A plurality of different data connections 1106 (e.g., "paths") representing one or more networks (or network channels) are shown and labeled as connection 1, connection 2.. connection N. There can be multiple data connections / paths across a single network and multiple data connections that can use one or more networks.

[0323] These data connections can include various types of technologies, each facing different communication situations, such as having different types of available communication power, array architectures, polarizations, multipath propagation mechanisms, spectral interference, frequency ranges, modulations, etc., and thus the connections can all have different levels of communication reliability.

[0324] System 1100 can be configured to communicate with various endpoints 1102, 1110 or applications, which do not necessarily have any information regarding the multiple paths / connections 1106 used to request and receive data (e.g., endpoints 1102, 1110 can function independently of paths or connections 1106). For example, the received data can be reconstructed such that the original transmission can be regenerated from the contributions of different paths / connections 1106 (an exemplary use scenario is the playback of video by a receiver configured to slot into a server rack in a data center facility in integration with an existing broadcast infrastructure to provide improved network functionality).

[0325] System 1100 receives an input (data flow) from source endpoint 1102, schedules the delivery of improved data packets across various connections 1106, and then sequences the data packets at the other end of system 1108 before transmitting them to destination endpoint application 1110. By doing so, system 1100 is configured to increase the bandwidth to approach the sum of the maximum bandwidths of the various available paths. Compared to using a single connection, system 1100 also provides improved reliability, which can be an important consideration in time - limited, highly confidential scenarios, such as collecting news at a live event when an event is occurring. In these events, there can be high signal congestion (such as at a sports event), or low reliability across one or more paths (such as reporting news after a natural disaster).

[0326] Multiple connections can be bonded together to act as a single connection, and different approaches can be utilized to coordinate routing such that connections can be used multiple times to, among other things, send the same packet.

[0327] In various embodiments, both the scheduler 1160 and the sequencer 1162 can be provided from a cloud computing implementation, or at an endpoint (before data is consumed by an application at the endpoint), or in various combinations thereof.

[0328] System 1100 can be adjusted, among other things, to optimize and / or prioritize performance, best latency, best throughput, minimum jitter (variation in latency of packet flow between two systems), cost of connections, combination of connections for a particular flow, etc. (e.g., if system 1100 has information that a transmission (data flow) is of content type X, system 1100 can be configured to use only data connections having similar latency, while content type Y can allow for a broader mix of data connections (or may require a larger net capacity that can only be achieved with a combination of data connections)). This adjustment can be provided to the system generally or per flow (or set of flows based on, for example, location, owner of any of the source or endpoint, or combinations thereof, transmission time, set of available communication links, security required for the transmission, etc.).

[0329] System 1100 can generally be bi-directional in that each gateway 1104, 1108 generally has a scheduler 1160 and a sequencer 1162 for handling TCP traffic (or UDP traffic, or a combination of TCP and UDP traffic, or any type of general Internet Protocol (IP) traffic), although in some embodiments only one gateway may be required.

[0330] System 1100 can be utilized for various scenarios, such as failover (e.g., for disaster recovery) or as a supplement for existing Internet connections (e.g., VoIP phone systems or corporate connections to the web), whereby an additional network (or path) is seamlessly added to replace a dropped primary Internet connection or complement a saturated primary Internet connection by bonding with a costly network. In a failover situation, coordinated failures (e.g., large-scale denial-of-service attacks or large-scale solar storms / geomagnetic storms) can occur across many communication channels, and then System 100 will require efficient use of communication resources as available communication resources are at a premium.

[0331] As another use of System 1100, there may be provided a means to maximize the use of high-cost (often sunk cost), high-reliability data connections, such as satellites, by enabling offloading of traffic to other data connections having different properties.

[0332] In some embodiments, System 1100 is a network gateway configured to route data flows across multiple network connections.

[0333] FIG. 11 provides an overview of a system having two gateways 1104 and 1108, each including a buffer manager 1150, an operation engine 1152, a connection controller 1154, a flow classification engine 1156 (involved in flow identification and classification), a scheduler 1160, a sequencer 1162, and a network characteristic monitoring unit 1161, linked by N data connections 1106, and each gateway is connected to specific endpoints 1102, 1110. Reference characters A and B are used to distinguish the components of each of the two gateways 1104 and 1108.

[0334] Each of gateways 1104 and 1108 is configured to include a plurality of network interfaces for transmitting data over a plurality of network connections, monitoring the time-varying network transmission characteristics of the plurality of network connections, and analyzing at least one packet of the data flow of the packet to identify a data flow class, wherein the data flow class defines at least one network interface requirement for the data flow or, otherwise, is associated therewith, analyzing, and routing packets within the data flow over the plurality of network connections based on the data flow class and the time-varying network transmission characteristics, and includes a processor configured to perform the above in a device (e.g., including configured hardware, software, or embedded firmware).

[0335] Buffer manager 1150 is configured to set a buffer within the gateway adapted to more efficiently manage traffic (both of individual flows and combinations of multiple simultaneous flows passing through the system). In some embodiments, buffer manager 1150 is a separate processor. In other embodiments, buffer manager 1150 is a computing unit provided by a processor configured to perform buffer management 1150 among other activities.

[0336] The operation engine 1152 is configured to apply one or more deterministic methods and / or logical operations based on the received input data set (e.g., feedback information, network congestion information, transmission characteristics), and notify the system about the constraints applied to the bonded connections for each of the user / client, destination / server, connection (e.g., latency, throughput, cost, jitter, reliability), flow type / requirement (e.g., FTP vs. HTTP vs. streaming video). For example, the operation engine 1152 can be configured to limit a particular type of flow to a particular connection or set of data connections based on the cost in one instance, while reliability and low latency may be more important for different users or flow types. Different conditions, triggers, methods can be utilized depending on, for example, one or more elements of the known information. The operation engine 1152 can be provided, for example, on the same or a different processor as the buffer manager 1150.

[0337] The operation engine 1152 can be configured to generate, apply, or otherwise operate or use one or more rule sets that determine the logical operations by which the routing on the N data connections 1106 is controlled.

[0338] The flow classification engine 1156 is configured to evaluate each data flow received by the multipath gateway 1104 for transmission, and apply a flow classification approach to determine the type of traffic being sent and its requirements if not known. In some embodiments, deep packet inspection techniques are adapted to perform the determination. In another embodiment, the evaluation is based on a heuristic method or data flow that is marked or labeled when generated. In another embodiment, the evaluation is based on rules provided by the user / administrator of the system. In another embodiment, a combination of methods is used. The flow classification engine 1156 is configured to interact with one or more network interfaces and can be implemented using an electronic circuit or a processor.

[0339] The scheduler 1160 is configured to perform a determination as to which packet should be sent to which connection 1106. The scheduler 1160 can be regarded as an improved QoS engine. In some embodiments, the scheduler 1160 is implemented using one or more processors, or a stand-alone chip or configured circuit such as a comparator circuit or an FPGA. The scheduler 1160 may include a series of logic gates verified for performing the determination.

[0340] A typical QoS engine manages a single connection, but the QoS engine (or, in this case, the scheduler 1160) can be configured to perform flow identification and classification, and the final result is that the QoS engine will reorder the packets before they are sent on one connection.

[0341] In contrast, the scheduler 1160 is configured to perform flow identification, classification, and packet reordering, but in some embodiments, the scheduler 1160 is further configured to perform a determination as to on which connection to send a packet and / or to satisfy a policy set for the flow by the user / administrator (or as set by various rules) in order to improve the transmission characteristics of the data flow. The scheduler 1160 can change the network interface operating characteristics, for example, by transmitting a set of control signals to the network interface to indicate that they should be switched on or off, or that they should be used for routing data. The control signals can be a set of instructions indicating specific characteristics of the desired routing, such as packet timing, reservation of the network interface for a particular type of traffic.

[0342] For example, consider two connections having the following characteristics.

[0343] Connection 1: A round-trip time (RTT) of 1 ms and an estimated bandwidth of 0.5 Mbps.

[0344] Connection 2: 30 ms RTT, estimated bandwidth of 10 Mbps.

[0345] Scheduler 1160 may attempt to reserve connection 1 for Domain Name System (DNS) traffic (small packets, low latency). In this example, there may be more DNS traffic than can fit into connection 1, and scheduler 1160 may be configured to overflow traffic to connection 2, although scheduler 11160 can selectively do so based on other criteria or factors (e.g., if scheduler 1160 is configured to provide fair arbitration, scheduler 1160 may configure the first overflow traffic from an IP address that has already sent a significant amount of DNS traffic in the past X seconds).

[0346] Scheduler 1160 may be configured to process decisions based on, for example, a process or method operating in conjunction with a similar implementation in one or more processors or hardware (e.g., FPGA). Scheduler 160 may be configured to operate under the control of operation engine 152, which decomposes data streams into data packets and then routes the data packets to a buffer (managed by buffer manager 150) that supplies the data packets to a data connection according to rules that attempt to optimize packet delivery while considering the characteristics of the data connection.

[0347] In some embodiments, scheduler 1160 need not be configured to communicate packets in the correct order, but rather is configured to communicate packets across diverse connections so as to meet or exceed desired QoS / QoE metrics, some of which may be defined by the network controller and others of which may be defined by the user / customer. If packets can be communicated out of order, sequencer 162 and the buffer manager may be utilized to reorder received packets.

[0348] Continuous bursts of packets are transmitted over a network interface, and an estimated value of the bandwidth of the first network interface is generated based on a timestamp recorded when the packets within the continuous burst of packets are received at the receiving node and the size of the packets. The estimated value is then utilized to route packets within the data flow of sequential packets for the entire set of network connections based on the generated bandwidth of the first network interface. As described below, in some embodiments, the estimated value of the bandwidth is generated based on the timestamps of packets within the burst that are not combined with the initial or final packets within the burst, and a lower bandwidth value can be estimated, and a higher bandwidth value can be estimated (e.g., through packet replacement). The packets transmitted can be test packets, test packet "piggybacking" on data packets, or hybrid packets. When data packets are used for "piggybacking", in some embodiments, flagging such data packets is included to increase redundancy (e.g., to enhance tolerance to lost packets, particularly for packets used for bandwidth testing purposes).

[0349] In some embodiments, sequential packets can be received in order or within an acceptable deviation from the order such that the sequencer 1162 can rearrange the packets for consumption. In some embodiments, the sequencer 1162 can be a physical hardware device incorporated into a broadcast infrastructure or a networking infrastructure that receives signals and generates an output signal that is the reassembled signal. For example, the physical hardware device can be a rack-mounted appliance that functions as a first stage of signal reception and reassembly.

[0350] Sequencer 1162 is configured to order the received packets and transmit them to the application at the endpoint in an acceptable order in order to reduce unnecessary packet re-requests or other error corrections for the flow. In some embodiments, this order follows the original order. In other embodiments, the order is within an acceptable error range such that Sequencer 1162 can still reassemble the data flow. Sequencer 1162 may include, for example, a buffer or other mechanism for smoothing the received flow and, in some embodiments, is configured to control packet acknowledgments and storage transmissions based on monitoring the transmission characteristics of multiple network connections and the non-uniform distribution in the reception of the data flow of sequential packets.

[0351] Sequencer 1162 may be provided, for example, on a processor or implemented in hardware (e.g., a field programmable gate array) provided under the control of operation engine 1152 that is configured to reassemble the data flow from received data packets extracted from a buffer.

[0352] Sequencer 1162 is configured to hide latency differences between a plurality of unacceptable connections for each flow.

[0353] Operation engine 1152 is operable as an aggregator of information provided by other components (including 1154) and instructs Sequencer 1162 through one or more control signals indicating how Sequencer 1162 should operate in a given flow.

[0354] When a system configured for a protocol such as TCP receives a packet, the system is generally configured to expect (but not require) that packets arrive in order. However, the system is configured to establish a timeout when out-of-order packets are expected to arrive (usually several times the round-trip time or RTT). The system may also be configured to retransmit missing packets (e.g., fast retransmission triggered by three DUP ACKs) earlier than a heuristically based limited time.

[0355] When packets arrive at sequencer 1162 on a connection having significantly different latencies, sequencer 1162 may be configured to buffer the packets (per flow) until they are approximately the same age (delay) before sending the packets to the destination. For example, this is done if the flow has requirements for consistent latency and low jitter.

[0356] Sequencer 1162 does not necessarily provide reliable, strictly ordered delivery of data packets, and in some embodiments, the system using the protocol (e.g., an application on top of TCP or UDP) is configured to provide what is necessary so that the system does not early determine that packets are being lost by the network.

[0357] In some embodiments, sequencer 1162 monitors the latency variation (jitter) of each data connection (based on data maintained by operation engine 1152) along with packet loss, and is configured to predict which data connections are likely to delay packets beyond what is expected by the flow (i.e., meaning that endpoints 1102 and 1110 would consider them lost and call an error correction routine).

[0358] For different situations, the sequencer 1162 may utilize, for example, a larger jitter buffer on the connection that exhibits greater latency variation. For packet retransmission, the sequencer 1162 may be configured to immediately request lost packets via the "best" (most reliable and lowest latency) connection.

[0359] In some embodiments, a network controller device (e.g., a router) is described, the device including a processor coupled to computer memory and data storage, the processor receiving one or more data sets indicative of monitored network communication characteristics, and maintaining, in a data structure stored in the data storage, a hierarchical representation of a plurality of connections separated into a plurality of groups, each group established based at least on a minimum probability associated with the successful communication of data packets across one or more of the connections present within the group, controlling a plurality of communications of the data packet such that the data packet is transmitted at least once across one or more of the plurality of connections, thereby controlling the plurality of communications of the data packet such that the plurality of communications converge and the transmission of the data packet meets a target probability threshold.

[0360] In some embodiments, with respect to the target probability threshold, the receipt of the data packet defines reliability, and in such a manner, the receipt of the data packet meets the target probability threshold by way of the plurality of communications.

[0361] In another aspect, the plurality of groups are organized into a plurality of corresponding layers, each layer representing the number of times the data packet must be transmitted via any corresponding connection of the layer in order to achieve the target probability threshold.

[0362] In another aspect, the plurality of communications includes retransmissions across connections of a layer among the plurality of layers, and the number of retransmissions is based on the number of times a data packet must be transmitted across the corresponding connection of the layer in order to achieve the target probability threshold of the corresponding layer. In a variant embodiment, the retransmissions can be performed on the same connection but at different times (i.e., separate bursts) in order to prevent all retransmission losses due to a temporary degradation of network reliability (i.e., burst losses due to buffer saturation on an intermediate host in the path).

[0363] In another aspect, the retransmissions are performed across different connections of the layer.

[0364] In another aspect, the plurality of communications includes retransmissions across connections of different layers among the plurality of layers.

[0365] A connection may not participate in a transmission event due to a configured priority routing preference, such that a lower priority connection can be deactivated because a higher priority connection has sufficient bandwidth to handle the prioritized data flow. In the absence of other traffic, this limits the system's knowledge about the lower priority connection, and the scheduler 1160 or the connection manager 1154 must consider old observations of the connection properties.

[0366] Embodiments are described herein that can be extended on the methods and systems described above.

[0367] In some embodiments, a new scheduling approach (e.g., algorithm) can be introduced to address the following three problems that arise when sorting connections in an absolutely preferred order by the properties of the connection (as described above). These algorithms are implemented in a physical computing controller and / or circuitry. A) The connections are used in the same order for all flows with the same preference. Flows may unnecessarily compete with each other on each connection, and it becomes more likely that traffic will be split among multiple connections. Whenever possible, the preference keeps the traffic of a flow sticky to only a single connection and avoids unnecessary work for recombining its packets on the other side. B) Connections further down in the absolutely preferred order are likely to have old properties because they are less frequently used. C) Connections with very similar measurement / composite properties will frequently swap places in the absolutely preferred order even after rounding / bucketing, which may cause packets from a flow to be unnecessarily split among connections.

[0368] In addition, technically difficult situations that are difficult to satisfy are real - time applications with real - time deadlines such as video chats and voice calls. Due to the deadlines, it may be necessary to minimize jitter / loss and receive packets on time. In some embodiments, both latency and jitter requirements may be imposed. For example, for audio and video to be smooth, they need to come in at a certain frame rate; otherwise, the user may experience freezes and pauses.

[0369] In a non-limiting example, the flow types for consideration can include flow types generated by streaming video applications. In these applications, the transmitter typically captures and encodes video at a given frame rate that is constant (e.g., 50 frames per second) but can be variable in some cases. The transmitter packetizes these encoded video frames for transfer to the receiver in UDP packets that typically include a timestamp representing the relative capture time of each video. For example, in the case of a 50 fps frame rate, if the very first video frame has a timestamp of t = 0, the next video frame will have a timestamp of t = 20 ms (dividing 1 second by 50 frames per second).

[0370] In these streaming video applications, the transmitter then transmits the UDP packets to the receiver, which needs to receive them in a time-critical manner. This is because at the moment the receiver starts playing back the first video frame, all subsequent frames (consisting of one or more UDP packets) must be received within 20 ms of the previous frame; otherwise, the playback to the viewer will be recognized as having artifacts (jerky or stuttering video) and will provide an inadequate viewing experience.

[0371] From a networking perspective, the delivery of UDP packets is typically done on a best-effort basis, meaning that there is sufficient variability such that UDP packets that make up each video frame may not be consistently delivered to the receiver within 20 ms from the previous video frame. This can be due to, for example, queuing delays resulting from contention for available capacity on one or more network links between the transmitter and the receiver. The receiver typically attempts to handle this by storing the video frames in a jitter buffer that is large enough to handle the predicted variations in delivery time. For example, a jitter buffer that stores two 50 fps video frames could hide up to 40 ms of network variability in the delivery time of UDP packets. However, a larger jitter buffer results in a longer glass-to-glass latency (the delay from when a video frame is captured until it is played back). When determining the size of the jitter buffer, the receiver must make a trade-off decision to balance the perceived smoothness of playback (rendering video frames at exactly 20 ms intervals) against the overall latency experienced by the viewer.

[0372] The opportunity and challenge presented by itself with respect to having multiple connections for this streaming video example is that there are multiple network paths between the transmitter and the receiver. Since the queuing delays and resulting variability in UDP packet delivery times on each connection are typically uncorrelated, it is possible for the transmitter to use scheduling techniques to transmit each UDP packet on one or more connections, resulting in the video frames arriving at the receiver consistently before their respective deadlines, and enabling the receiver to use a smaller jitter buffer and reduce the overall latency experienced by the viewer.

[0373] Throughput-based applications such as bulk data delivery including FTP and downloading large-capacity files do not focus on latency and jitter, and thus may pose different problems. As long as the connection provides throughput, there may be no problem with latency. For example, in the case of file transfer, if there is an initial pause before the transfer starts, this may often be acceptable.

[0374] Previous schedulers may sort connections in an absolute order from best to worst, which presents a technical problem because all applications using connections in the same order may not necessarily be good, especially when there are competing requirements (e.g., throughput vs. real-time). For real-time purposes, the best connection may be different from the best connection for throughput purposes. Using connections in a different order has a high probability of splitting the load and reducing the competition for the first connection, and meeting the real-time flow requirements. Treating some connections as "functionally equivalent" may be good in some embodiments, and the described embodiments consider keeping the flow sticky to one of several "functionally equivalent" connections. For example, if it is sticky, there may be less need to shift between connections or reorder packets / put them back in order on the other (receiving) side.

[0375] Consider a constrained real-time application and a plurality of WAN connection hybrid routers that need to route packets to meet the constraints set by the application. Multiple applications with varying requirements can be executed simultaneously. There is a need to achieve high performance while balancing various issues. This technical problem becomes even more complex when many applications are running and competing for bandwidth, and the different requirements of all applications must be met. In many situations, some applications are real-time, some require throughput, and due to the constrained WAN capacity, it is technically difficult to balance various trade-offs (e.g., bandwidth, latency, jitter).

[0376] According to some embodiments, the improvement is to group the connections into layers, and all connections within a layer are "functionally equivalent" from the perspective of flow preference.

[0377] FIG. 12A is a schematic diagram of a per-flow communication system according to some exemplary embodiments. The per-flow communication system is

[0378] In the embodiment shown in FIG. 1200A, device 1202 may receive input packet 1204, and then the input packet 1204 may be sorted into flows and prioritized based on their requirements at 1206. Next, the available connections may be classified into connection layer 1208 based on a combination of flow requirements, measured / composite connection properties, and any predefined rules (e.g., connection priority rules specified by an administrator). As shown, connection layer 1208 includes layer 1, layer 2, layer 3, up to maximum layer N. Variations of connection layer 1208 are possible, and in alternative embodiments, the layers may be further subdivided into sub-layers. Other, different, alternative, more, or fewer layers are possible. Then, input packet 1204 may be transmitted on the determined connection 1210 corresponding to the appropriate layer and parameters.

[0379] Figure 12B is a schematic diagram of a per-flow communication system 1200B according to some exemplary embodiments. In layer creation 1212, among other things, various requirements such as connection monitoring, flow properties, packet properties, administrator configuration, location, and multiple connections are considered by connection and requirement processor 1214.

[0380] In some embodiments, connection and requirement processor 1214 may group multiple connections into layers, as shown in 1200B as layers 1, layer 2, layer 3, …, layer N. Layer creation 1212 may occur at any point prior to flow sorting, as shown at 1206 in FIG. 12A. In some embodiments, communication system 1200B may be included in layer creation 1212 that occurs within communication system 1200A, for example, at flow sorting and packet scheduling 1206.

[0381] Next, during layer creation 1212, the connections sorted into layers by connection and requirement processor 1214 may transmit, for example, a flow of packets. Connection scheduler 1216 may schedule packet transmission by iterating through the layer levels and by iterating through the connections sorted to a single layer level. Packet transmission and scheduling approaches are further described herein.

[0382] Functionally, a layer may be described above as grouping "functionally equivalent" connections, but in these described embodiments, "functional equivalence" is extended. From a system perspective, a layer can be managed through the use of database records or metadata that are dynamically updated or added. For example, each connection can have a layer assignment that is tracked and updated through corresponding database values stored in a database table such as a routing table stored in a coupled computer memory such as a non-transitory computer-readable medium, and the non-transitory computer-readable medium is referenced or traversed to obtain layer information used in the connection control approach described herein.

[0383] The following definitions may describe "functional equivalence" according to some embodiments.

[0384] For flows with throughput preference, the layers may be based on the same (Mathis coefficient, capacity coefficient) composite property pair calculated from the measured properties, where the measured properties are raw or rounded / bucketized / quantized. In some embodiments, the composite properties may be similarly rounded / bucketized. In some other embodiments, additional rounding / bucketizing / filtering of the measured properties and composite properties may be introduced to account for edge cases. For example, without this additional filtering, a connection with a very high drop probability (98+%) and low latency may result in a better Mathis coefficient compared to a connection with low loss but much higher latency.

[0385] The explanation for grouping / hierarchizing connections based on the unique (Mathis coefficient, capacity coefficient) pair of a flow with throughput preference is that these factors measure the throughput capacity of the connection and how efficiently it can use that throughput. Recall that the Mathis coefficient predicts throughput based on connection RTT and packet loss (high packet loss means the connection has to retransmit lost packets, which takes time when the RTT is large). Recall that the capacity coefficient is a measure of the transmission rate that a connection can achieve per millisecond of its RTT (generally, the idea that higher throughput and lower RTT are preferred). A table describing some exemplary connection properties is shown below.

Table 6

[0386] The "functionally equivalent" grouping / hierarchization of these three connections is that they belong to the same layer because both A and B are reliable and high-performance connections. Both have no loss (therefore, no retransmission is required, and their throughput can be used efficiently), and both achieve a similar throughput per millisecond of latency. They are "functionally equivalent" in that when a throughput-prioritized flow is scheduled on either connection, the capacity of the connection can be maximally utilized. C intuitively belongs to a different "functionally equivalent" layer because it has a higher loss rate compared to B. Since many packets need to be retransmitted, the actual throughput achieved by C is much lower than that of B.

[0387] For flows with real-time preferences, the layer can be based on grouping connections based on the probability that the flow's packets can be delivered before the deadline of each packet.

[0388] By comparing the characteristics of the available connections with the real-time requirements of the flow, an explanation can be observed regarding how and why different connections can be categorized as "functionally equivalent" for these types of flows. For example, a real-time video chat application has a target latency requirement of 150 ms, which means that the video transmitter hopes that each of its UDP packets is delivered to the video receiver before the aging of each packet reaches 150 ms, and the aging is measured from when the transmitter first sends each packet. This is a common requirement for video chat applications, and for the chat to be recognized as interactive by human participants, the overall round-trip latency needs to remain below 300 ms. A table explaining the properties of some exemplary connections is shown below.

Table 7

[0389] The "functional equivalence" grouping / hierarchization of these six connections is that A and B belong to the same layer because both have a one-way latency smaller than the 150 ms application target, and because they experience 0% packet loss, all packets transmitted arrive with their measured one-way latency. They are "functionally equivalent" from the application's perspective because they arrive before the 150 ms deadline of the application, regardless of the connection used to transmit their UDP packets.

[0390] Similarly, it is clear that E and F belong to the same "functionally equivalent" layer because, although their one-way latencies are different, both are greater than the 150 ms application target. Regardless of which is used to transmit the UDP packet, it is impossible for them to arrive within the time limit to meet the deadline.

[0391] Finally, it is not intuitively obvious where C and D belong, except that it is clear that they do not necessarily belong to either of the other two functionally equivalent layers. Their one-way latencies are smaller than the 150 ms application target, but they experience significant packet loss, so many of the transmitted packets are lost and need to be retransmitted, which can result in missing the deadline. A more detailed and formal way to group / hierarchize connections by "functional equivalence" of real-time application flows is as follows.

[0392] The probability that a connection can deliver a packet before its deadline is calculated based on a composite property called the "Estimated Delivery Time" (EDT) of the packets transmitted on the connection, which is defined as the expected value of the delivery time of the packets after retransmission due to dropped packets.

[0393] Figure 13A is a diagram 1300A showing the calculation of EDT according to some exemplary embodiments. In some embodiments, this can be calculated as the sum of the probabilities of successful delivery in retransmission, multiplied by the delivery time of that retransmission, as shown in Figure 13A. This is a uniform probability, P, that a packet is dropped each time it is retransmitted, as shown in Figure 13A drop 13103, a constant one-way propagation delay, T prop 13101, the response timeout after a retransmitted unacknowledged packet, T rto 13102 can be assumed and determined

Number

[0394] Other variations are possible

[0395] In some embodiments, the receiver may be able to quickly identify when a packet is lost and transmit a NACK. Figure 13B is a diagram 1300B showing the calculation of the estimated time delivery according to some exemplary embodiments. Variations are shown in Figure 13B, with different durations, T rprop 13204, the propagation delay of the ACK or NACK packet, and T rtprop =T prop +T rprop , i.e., there may be a total round-trip propagation delay for the packet and the response. The ACK or NACK may be lost with a different probability, P rdrop 13205, i.e., the probability that the packet is dropped in the reverse direction. Considering these, the T in the above formula rto is P rdrop T rto +(1 - P rdrop )T rtprop can be replaced, and the EDT calculated as follows is obtained

Number

[0396] In some embodiments, the EDT is Tprop = T rprop and is, and P drop = P rdrop can be estimated under the simplifying assumption that it is.

[0397] In some embodiments, the EDT can be estimated by observing the actual time elapsed from the time when a packet is scheduled to be first delivered on a connection until that packet is successfully delivered, but only for packets that are exclusively retransmitted on one connection. Variations in EDT derivation are also possible, reflecting more complex or simpler approaches. For example, in a variation, different P drop exist in the forward and reverse directions, and T prop and T rprop may not be symmetric, and / or ACK packets may also potentially be dropped.

[0398] FIG. 14 is a schematic diagram 1400 showing wide area network connections in different hierarchical groupings based on the estimated time distribution according to some exemplary embodiments. In FIG. 1400, given specific packet properties, the connections can be grouped into appropriate layers given the approach described below.

[0399] For real-time flows (those with latency / jitter preference, represented as a nominal "target latency"), the layer can be based on the relationship between the deadline per packet of the flow (referred to as the adjusted target latency or ATL) and the EDT and T prop properties of the connection. For example, as follows. ○ Layer 1: T prop ≦ ATL && EDT ≦ ATL ○ Layer 2: T prop ≦ ATL && EDT > ATL ○ Layer 3: T prop > ATL && EDT > ATL

[0400] An explanation for each of these layers is as follows. ○ Layer 1: The one-way latency of the connection is low enough to meet the deadline even when packet loss and retransmission are considered. ○ Layer 2: The one-way latency of the connection is low enough to meet the deadline, but not so when packet loss and retransmission are considered. ○ Layer 3: Regardless of the presence of retransmission, the one-way latency of the connection is not low enough to meet the deadline.

[0401] ATL can be a dynamic value that varies for each packet and changes according to the aging of the packet. ATL = PacketDeadline - CurrentTime, where: PacketDeadline = T PacketReceived + NominalTargetLatency.

[0402] In some embodiments, less than three layers may exist depending on the total number of connections and the relationship between their properties and the flow deadline. For example, for all connections, T prop and P drop may be very small, meaning that their EDT composite properties are also very small. Alternatively, since the real-time application is not interactive, ATL may be very large. For example, unlike an interactive Voice over Internet Protocol (VoIP) call with an ATL of less than 150 ms, a broadcast video application may have an ATL of less than 8 seconds. In both of these scenarios, all connections are "functionally equivalent" in that the EDT is small enough to meet the ATL, but may not be equivalent in terms of efficiency, so all connections can be categorized into Layer 1.

[0403] Conversely, there may be some embodiments with more than three layers, for example, when efficiency is considered. Considering two connections (T prop ≤ ATL && EDT ≤ ATL) where both EDTs meet the required relationship for Layer 1, one connection has a large P drop which means that several retransmissions may be required for each packet to achieve the EDT. The other connection has zero Pdrop When having it, the EDT can be achieved with only a single transmission per packet.

[0404] In this example scenario, which is desirably optimized for efficiency, P drop Further constraints on layer membership can be added to include measurement properties such as ○ Layer 1: T prop ≤ ATL && EDT ≤ ATL && P drop <0.02 ○ Layer 2: T prop ≤ ATL && EDT ≤ ATL && P drop ≥ 0.02 ○ Layer 3: T prop ≤ ATL && EDT > ATL && P drop <0.1 ○ Layer 4: T prop ≤ ATL && EDT > ATL && P drop ≥ 0.1 ○ Layer 5: T prop > ATL && EDT > ATL && P drop <0.35 ○ Layer 6: T prop > ATL && EDT > ATL && P drop ≥ 0.35

[0405] In some embodiments, the layer can be recalculated based on the events that occur. For example, when a WAN connection updates a measurement property, its corresponding layer membership can be adjusted based on the new property. The adjustment can include updating the corresponding record or data value on a stored reference table or database table used for routing control.

[0406] In the case of real-time flow, in some embodiments, since each packet can have a different ATL, layer recalculation can be performed for each packet.

[0407] There can be more than three types of flow preferences (other than throughput and latency), and thus, there can be more than three ways of stratifying / sorting. In some embodiments, both combinations can be used (also, for high throughput flows that require low latency, e.g., high resolution real-time video).

[0408] In some embodiments, stratification can incorporate only a subset of the available connections. In one embodiment, time is considered because it can affect the predicted usage or demand on some connections. For example, during normal business hours, connections are likely to be congested and may provide insufficient performance. Such connections are excluded from stratification for mission-critical connectivity. In another embodiment, administrator configuration is considered because there can be factors that can affect stratification and that must be manually configured and are not automatically detectable or measurable. For example, a network administrator can exclude a connection due to scheduled maintenance at the connection provider, due to high overage fees, or for other administrative reasons.

[0409] In another embodiment, advanced or complex connection properties are considered. For example, if a connection is determined to have a directly proportional relationship between its throughput and latency (e.g., the connection can transmit at a very high throughput, but the latency will substantially increase, or the connection can maintain a very low latency as long as a certain throughput is not exceeded), this connection can be excluded from stratification for real-time flows with strict latency requirements if it is known that this connection is currently servicing a high throughput flow.

[0410] In another embodiment, location is considered due to functional or policy requirements. For example, data residency requirements can obligate that certain data be transmitted only within a network having a specified region.

[0411] Subsequently, connections that can be routed to a network outside the area can be excluded from the stratification. In another example, a service with geolocking restrictions may provide the service only for requests received from specific geographical locations. Connections known to route / exit through locations outside these positions (e.g., a concentrator providing a bonded connection is installed at such a position) can then be excluded from the stratification. In another embodiment, security requirements can be considered. For example, unencrypted network connections not provided via a VPN can be excluded from the stratification.

[0412] In some embodiments, the stratification can be extended by intentionally incorporating more connections with one or more layers. This can be achieved in two or more ways including, but not limited to, relaxing the stratification requirements so that more connections are eligible to belong to the same layer, combining subsets of the resulting layers after the stratification is complete, moving connections from one layer to another. In some embodiments, security / obfuscation requirements that may arise from an administrative configuration (management configuration) are considered when extending the layers. For example, obfuscation can be achieved by including more connections in the first layer such that a user's packets and / or flows are spread across multiple networks, preventing a man-in-the-middle attacker who can eavesdrop on a single network from intercepting all of that user's packets and / or flows.

[0413] In some embodiments, the stratification can be completely manual based on an administrator policy. In one embodiment, the stratification is based on the cost of the connections such that connections of similar cost are grouped into the same layer and the layers are traversed from lowest to highest cost. For example, the administrator can use a connection priority (also known as priority routing) where low-cost connections are designated as primary, medium-cost connections are designated as secondary, high-cost connections are designated as tertiary, and the throughput activation threshold is configured such that the extra cost is justified when the aggregated service drops below a certain bitrate. In another example, a modified version of the connection priority can use different types or combinations of activation or deactivation thresholds such as the total usage during the billing period, the total cost during the billing period, the number of connected users, the priority of the connected users, etc.

[0414] In some embodiments, the determination of the functionally equivalent WAN connections and the resulting stratification / grouping is based on machine learning or other similar classes of pattern recognition techniques. For example, a model trained using attributes / features including the measured / composite properties of the connections, using labels provided directly or indirectly by feedback from the application or user.

[0415] According to some embodiments, when finding a connection to schedule packets for a flow, the iteration can occur across each layer from best to worst. Within a layer, the iteration can occur across all connections, but the starting connection can be selected pseudo-randomly for each flow (e.g., based on a hash of the flow tuple). Since the first connection selected in each layer can be persisted, the starting connection can remain the same if the members of the layer change (e.g., if a new connection is added or removed) (the flow can remain sticky to the first connection selected).

[0416] FIG. 15A is provided to illustrate examples of pseudo-random hashing and flow stickiness according to some exemplary embodiments. FIG. 15B is provided to show expiration of stickiness after idle time and after TCP FIN exchange according to some embodiments. FIG. 15C is provided to show how pseudo-random selection helps keep packets of a single flow on the same connection and also how increased demands cross multiple layers. Other variations are possible.

[0417] FIG. 15A shows example 1500A of a new connection D added to a layer that already has connections A, B, and C while TCP flows and UDP flows are active.

[0418] Before connection D is added, the TCP flow is pseudo-randomly hashed to the connection at layer index 3, i.e., connection C, and the UDP flow is pseudo-randomly hashed to the connection at layer index 2, i.e., connection B. When connection D is added at 1502, connection D is placed at index 2, thus repositioning connections B and C to indices 3 and 4, respectively. Despite the pseudo-random hashing of the TCP and UDP flows to indices 2 and 3, the TCP and UDP flows remain sticky to the connections where they first started.

[0419] In some embodiments, the system may be configured such that a selected connection may ultimately be forgotten based on an idle time. For example, given a layer, a connection may be preferably selected pseudo-randomly. This connection may be selected for a flow (e.g., a video call), and packets belonging to the flow may be kept sticky to the flow. Sticky connections are useful because they may help improve overall communication characteristics by using the same connection as much as possible.

[0420] However, for example, since a video call has ended, the flow may ultimately become idle. The system may not explicitly notify that the flow has ended, but the flow can be inferred at the point when after a certain period, packets for the flow are no longer visible and the sticky connection preference expires / resets.

[0421] Continuing with the above example, FIG. 15B shows Example 1500B where the sticky connection is reset after a period of idle time for a UDP flow and remapped later.

[0422] Assume that both the TCP flow and the UDP flow are idle for 1 minute or more at 1520. As a result, the UDP flow is considered to have ended and its sticky connection preference (Connection B) is forgotten. At 1522, the TCP flow and the UDP flow are restarted, and UDP is remapped after expiration. Here, since no FIN packet has been detected, TCP remains sticky. For example, assuming that the hash function generates the same index 2, when the UDP flow is restarted, the sticky connection preference is determined again and becomes Connection D, and Connection D happens to be at index 2 in the layer at that time.

[0423] In some embodiments, the sticky connection can be forgotten based on different factors. For example, some protocols such as the Transmission Control Protocol (TCP) have an explicit flow termination sequence (FIN exchange). The connection can be forgotten based on recognizing such a state transition.

[0424] Continuing with the above example, FIG. 15B shows an example where a sticky connection is reset and remapped later after a FIN exchange is detected at 1524 for a TCP flow. Assume that the TCP flow is completed and both sides of the connection perform a FIN exchange. This causes the TCP flow to be considered terminated and its sticky connection preference (connection C) to be forgotten. The TCP connection can then be re-established with a new handshake at 1524 and, in some embodiments, remapped. For example, assuming the hash function generates the same index 3, when the TCP flow is restarted, the sticky connection preference is determined again and becomes connection B, and connection B happens to be at index 3 in the layer at that time.

[0425] In some embodiments, a connection can be intentionally forgotten for security / obfuscation purposes. For example, packets can be intentionally split between connections to prevent an attacker eavesdropping on a single connection from capturing all packets related to the flow.

[0426] Forgetting the preferred connection after an idle period can be important to avoid the "over-sticky" problem where many long-lived (but bursty) flows become sticky to the same connection, compete for capacity on the same preferred connection, and their traffic needs to be split between multiple connections and recombined at the receiver.

[0427] According to some embodiments, the pseudo-random selection of a preferred connection to start an iteration via "functionally equivalent" connections addresses various technical challenges. Specifically, since the flow does not compete with other flows for the same connection in the same order, it is likely to stay entirely on one connection, which is likely to cause packet overflow to the next connection. Further, all connections in the active layer tend to have fresh properties since they are continuously used by different flows.

[0428] Figure 15C shows an example 1500C of two flows that send packets through multiple connections in multiple layers. Assume that only 6 packets can be sent on each connection at a time to avoid congestion. In the upper half, each of Flow 1 and Flow 2 has 4 packets to send.

[0429] At 1540, without using pseudo-random selection, both flows, shown as Flow 1 and Flow 2 in the example, always select the first connection in Layer 1, Connection A, as their starting connection, and the packets of both flows are ultimately split between both Connection A and Connection B. At 1542 in the lower half of Figure 15C, using pseudo-random selection, each flow can select a different starting connection.

[0430] For example, if Flow 1 selects Connection A and Flow 2 selects Connection B, all the packets of Flow 1 end only on Connection A, and all the packets of Flow 2 end only on Connection B. In other words, for each flow, all its packets are assigned to different connections.

[0431] If the requirements of one flow increase and the scheduler needs to repeatedly schedule all traffic across multiple connections / layers, the other flow is not affected. In this example, the requirements of Flow 2 increase, and Flow 2 acquires 5 additional packets to send. This iterates across all the connections in Layer 1 and moves to Layer 2 if all connections are filled with packets up to their capacity. More specifically, the 5 additional packets are distributed as follows: 2 on Connection B in Layer 1, 2 on Connection A in Layer 1, and 1 on Connection C in Layer 2. Observe that Flow 2, despite being split across 3 connections in 2 layers, does not affect Flow 1, which continues to have all its packets on one connection, namely Connection A in Layer 1.

[0432] In some embodiments, the sticky preferred connection is only remembered for the first layer because, if two or more layers are used, by definition, the packets are split across multiple connections, and thus packet re-sequencing is required at the receiver.

[0433] In some embodiments, if the preferred connection no longer exists in the first layer, it may be forgotten (and a new connection may be selected). For example, the measured property and the resulting EDT may have caused a move to another layer.

[0434] The pseudo-random selection of the preferred connection can be done in many different ways. In some embodiments, the properties of the flow can be hashed (destination IP, port, etc.) or a random number can be generated, and in both cases, the value modulo the number of connections in the layer becomes the index of the selected connection.

[0435] In some embodiments, the selection can be done in sequence. For example, the first flow is assigned the first connection in the layer as its sticky connection preference, the second flow is assigned the second connection, and so on, returning to the first connection whenever the number of connections in the layer is reached.

[0436] In some embodiments, the sticky selection can be done by biasing based on the available capacity. For example, the selection can be based on the currently least used connection, which can be determined based on factors such as the number of flows that have selected each connection as a preference or the current unused capacity of each connection.

[0437] In some embodiments, an estimated value of the throughput requirement of a flow can be used as part of a preferred sticky connection selection such that the selected connection is most likely to result in flows that do not interfere with other connections. Although connections in a layer are functionally equivalent, they are not identical, so packet overflow to multiple connections can result in out-of-order packets that require resequencing at the receiver. If a flow uses only one connection, the likelihood of this occurring is reduced.

[0438] In some embodiments, the first connection in a layer and subsequent iterations through the connection in the layer occur in order of connection to the shortest to the longest backlog (“shortest backlog scheduling”). Backlog is measured in time units (typically milliseconds) and is defined as the volume of in-flight (transmitted but unacknowledged) bytes divided by the transmission rate of the connection. By iterating in this order, latency experienced by a flow is minimized if a single connection requires more bandwidth than can be provided alone.

[0439] For example, consider the following scenario: ● An application that transmits packets at a rate of 1 packet per 1 ms ● Two identical WAN connections, each: ● Can transmit 1 packet every 2 ms ● Has a one-way latency of 5 ms ● Has a CWND (congestion window) of 5 packets

[0440] Using sticky connection selection, the packets (one per 1 ms) generated by the application are scheduled on connections C1 and C2 and transmitted as follows. ● t = 0 ms: Packet 1 scheduled for transmission on C1 @ t = 0 ms (remaining CWND = 4) ● t = 1 ms: Packet 2 scheduled for transmission on C1 @ t = 2 ms (remaining CWND = 3) ● t = 2 ms: Packet 3 scheduled for transmission on C1 @ t = 4 ms (remaining CWND = 2) ● t = 3 ms: Packet 4 scheduled for transmission on C1 @ t = 6 ms (remaining CWND = 1) ● t = 4 ms: Packet 5 scheduled for transmission on C1 @ t = 8 ms (remaining CWND = 0) ● t = 5 ms: Packet 6 scheduled for transmission on C2 @ t = 5 ms (remaining CWND = 4)

[0441] The receiver receives the packets in an arbitrary order as follows. ● Packet 1: Received via C1 @ t = 5 ms ● Packet 2: Received via C1 @ t = 7 ms ● Packet 3: Received via C1 @ t = 9 ms ● Packet 6: Received via C2 @ t = 10 ms ● Packet 4: Received via C1 @ t = 11 ms ● Packet 5: Received via C1 @ t = 13 ms

[0442] When the receiver reorders the packets into the correct sequence and releases them to the destination, the destination experiences both a high degree of jitter (the packets are no longer consistently spaced at 1 ms intervals) and an overall latency longer than the 5 ms one-way latency on each connection. ● Packet 1: Released after reordering @ t = 5 ms ● Packet 2: Released after reordering @ t = 7 ms ● Packet 3: Released after reordering @ t = 9 ms ● Packet 4: Released after reordering @ t = 11 ms ● Packet 5: Released after reordering @ t = 13 ms ● Packet 6: Released after reordering @ t = 13 ms

[0443] Conversely, connection selection and packet scheduling based on the minimum backlog result in packets being scheduled and transmitted on C1 and C2 as follows. ● t = 0ms: Packet 1 @ t = 0ms scheduled for transmission on C1 (remaining CWND = 4) ● t = 1ms: Packet 2 @ t = 1ms scheduled for transmission on C2 (remaining CWND = 4) ● t = 2ms: Packet 3 @ t = 2ms scheduled for transmission on C1 (remaining CWND = 3) ● t = 3ms: Packet 4 @ t = 3ms scheduled for transmission on C2 (remaining CWND = 3) ● t = 4ms: Packet 5 @ t = 4ms scheduled for transmission on C1 (remaining CWND = 2) ● t = 5ms: Packet 6 @ t = 5ms scheduled for transmission on C2 (remaining CWND = 2)

[0444] The receiver receives the packets in order without additional jitter (maintaining a consistent 1ms inter-packet interval placement), and the latency experienced by the application is equal to the 5ms one-way latency on each connection. ● Packet 1: Received and released via C1 @ t = 5ms ● Packet 2: Received and released via C2 @ t = 6ms ● Packet 3: Received and released via C1 @ t = 7ms ● Packet 6: Received and released via C2 @ t = 8ms ● Packet 4: Received and released via C1 @ t = 9ms ● Packet 5: Received and released via C2 @ t = 10ms

[0445] In some embodiments, instead, the shortest backlog scheduling is simplified to round robin among each connection in the layer. When it is known that the connections have the same characteristics, this method is easy to implement and provides similar performance.

[0446] Connection priority (CP) rules, or priority routing, in some embodiments, can be specified by an administrator to express preferences about the order in which connections are to be used. In previous solutions, these rules were global and applied to all flows and also had the function of restricting the use of bandwidth for lower-priority connections. Consider an example of a set of connections using CP configuration as follows. ● Connection A: primary ● Connection B: secondary / threshold = 10 Mbps

[0447] According to these prior embodiments, connection B does not become active unless the capacity of connection A drops below 10 Mbps, and when it does become active, connection B only provides the difference to bring the total aggregated capacity back to 10 Mbps.

[0448] According to some current embodiments, the CP rules can be as granular as individual flows. Since the high priority of one flow can be the low priority of another flow, trying to limit the use of low-priority bandwidth can conflict with the requirements of both flows, so limiting the use of low-priority bandwidth can also be removed. For example, assume that flow 1 has the same CP rules as described in the previous example, but flow 2 has the reverse CP rules. ● Connection B: primary ● Connection A: secondary / threshold = 10 Mbps

[0449] Flow 1 may consume the full capacity of connection A because connection A is its primary, and flow 2 may consume the full capacity of connection B because B is its primary. Since either could have global CP rules, it cannot be restricted.

[0450] In some embodiments, the bandwidth limiting function is replaced with, for example, the bandwidth usage per WAN connection (regardless of flow or CP rule), or a global setting that limits the bandwidth limit applied to matching flows.

[0451] CP rules can affect layer construction because they can be placed in separate layers even if, considering all other properties, two connections are "functionally equivalent" when they are at different CP levels. For example, consider the following [Table 8]

[0452] Given a packet with an ATL of 100 ms, in the absence of connection priority rules, the layer structure is as follows. ● Layer 1: A, B (T prop ≤ ATL && EDT ≤ ATL) ● Layer 2: C, D (T prop ≤ ATL && EDT > ATL) ● Layer 3: E, F (T prop > ATL && EDT > ATL)

[0453] However, in connection priority rules, each change in priority results in the creation of a new layer even if the T prop and EDT properties still satisfy the constraints. ● Layer 1: A (T prop ≤ ATL && EDT ≤ ATL && CP = primary) ● Layer 2: C (T prop ≤ ATL && EDT > ATL && CP = primary) ● Layer 3: E (T prop > ATL && EDT > ATL && CP = primary) ● Layer 4: B (T prop ≤ ATL && EDT ≤ ATL && CP = secondary) ● Layer 5: D (T prop ≤ ATL && EDT > ATL && CP = secondary) ● Layer 6: F (T prop>ATL && EDT > ATL && CP = secondary)

[0454] In addition, if a connection is "functionally better" than another connection, it may be placed in a less preferred layer because it has a lower priority CP level. For example, connection E above is placed in layer 3, and connection B has a lower T prop and EDT values, but connection B is placed in layer 4 due to the connection priority rules.

[0455] In some embodiments, a scheduling method is introduced to help ensure that packets belonging to the real-time flow arrive before their deadlines, while balancing this with the amount of bandwidth used (e.g., a naive implementation could simply broadcast real-time packets on all available connections).

[0456] In some embodiments, real-time packets can be scheduled across multiple connections while balancing the use of bandwidth and maximizing the likelihood that they will arrive before their deadlines. As a non-limiting example, according to one embodiment,

[0457] assign a score to packet transmission on each layer, for example, as follows. ○ Layer 1: Score = 3 ○ Layer 2: Score = 2 ○ Layer 3: Score = 1

[0458] Schedule the packet on the number of connections necessary to bring the cumulative score above a pre-defined target score. The target can be a parameter such as, for example, 3.

[0459] Connections with no fresh or unreliable statistical information (close to 100% drop probability) can be given a score = 0 regardless of the layer. This can have the intended side effect of refreshing the statistical information on these connections when the packet is acknowledged.

[0460] Figure 16 shows an example 1600 having various connections 1602 sorted into various layers 1604 used to send several packets 1606. Similar to the above, the layer 1 connection obtains 3, the layer 2 connection obtains 2, and the layer 3 connection obtains 1. Connections with unreliable connections or old statistical information obtain 0 regardless of which layer they belong to. For each input packet, the connections are iterated from layer 1 to layer 3, in other words, from connection A to F. Each packet is sent on the connections it encounters and accumulates their scores until it reaches or exceeds a target score of 3. This results in the following exemplary results.

[0461] Packets 1 to 4 reach the target score immediately after being sent on the first connection, which is connection A. At this point, the capacity of connection A is completely consumed.

[0462] Packet 5 reaches the target score after being sent on three connections. Layer 1: connection B, Layer 2: connection C, and Layer 2: connection D. Scores of 0, 2, and 2 are accumulated respectively, resulting in a total score of 4, which exceeds the target score of 3. Note that since Layer 1: connection B is unreliable, it obtains 0. At this point, the capacity of connection D is completely consumed.

[0463] Packet 6 reaches the target score after being sent on four connections. Layer 1: connection B, Layer 2: connection C, Layer 2: connection E, and Layer 3 connection F. Scores of 0, 2, 0, and 1 are accumulated respectively, resulting in a total score of 3, which meets the target score. Note that since Layer 2: connection E has stale statistical information, it obtains 0. At this point, the capacity of Layer 1: connection B and Layer 3: connection F is completely consumed.

[0464] Even after being transmitted on two connections, Packet 7 does not reach the target score. Layer 2: Connection C and Layer 2: Connection E. It accumulates scores 2 and 0 respectively, resulting in a total score of 2, which is less than the target score of 3. At this point, the capacity of Layer 2: Connection C and Layer 2: Connection E is completely consumed. Even at this point, since the capacity of all connections is completely consumed, all transmissions stop. Unless Packet 7 is among the packets that have been acknowledged as received, when more connection capacity becomes available (for example, a packet that has been acknowledged as received or a packet for which a new connection has been added), the transmission of Packet 7 resumes in the next round of transmission.

[0465] Packet 8 does not reach the target score because all connection capacity is completely consumed and it is not transmitted on any connection. The transmission of Packet 8 can be carried out in the next round of transmission when more connection capacity becomes available.

[0466] Some embodiments may have different predefined target values for each flow. For example, a super-high-priority flow that desires to broadcast the packet on all connections, or other ways that allow an administrator to control the trade-off between the usage amount of bandwidth and meeting deadlines.

[0467] In some embodiments, under the assumption that the user may prefer the deadline to be achieved even if it incurs an additional cost to activate a lower priority, the CP rule can be ignored to reach the target score. For example, after iterating over all connections in all higher-priority layers (considered active by the CP rule), if the cumulative score has not reached the target, the iteration can continue within the connections of the lower-priority layer (considered inactive by the CP rule).

[0468] In some embodiments, preemptive scheduling may take into account predictions regarding future position changes (e.g., a bus, an airplane moving on a predictable path with a predictable change in a measured property). For example, since a current layer 1 connection may immediately become a layer 3 connection and vice versa, in some embodiments, the target score may be adjusted in advance and more redundancy may be transmitted when approaching a transition.

[0469] The described embodiments present a technical improvement over existing solutions, as preemptive retransmission is applied only to flows with deadlines (real-time applications), and flow deadlines and connection EDTs may be considered when determining whether preemptive retransmission is necessary, not only the connection drop probability. Throughput flows without deadlines or other similar types do not incur the bandwidth cost and overhead of preemptive retransmission.

[0470] In other embodiments, there may be early retransmission (including preemptive retransmission) of packets that are approaching their deadline but are still in-flight and have not yet been acknowledged. Such embodiments may balance meeting the deadline and efficient use of the connection (cost - dollars, bandwidth, computation). In some embodiments, there may be per-flow rules that allow an administrator to influence this balance. For example, if a flow is very important, "cost" is not an issue, and thus early retransmission of packets may be broadcast on all connections regardless of the additional cost incurred. Non-limiting examples:

[0471] In one embodiment, if a packet has a deadline, has not been acknowledged as received, and there is a reliable connection with minimum latency, available capacity, and fresh statistical information that has enough time to pass the packet over one additional time (while considering some variance such as time for processing overhead), an early retransmission of this packet is sent over this connection. This is done as a last resort to pass the packet in time before the deadline. FIG. 17 shows an example 1700 of Embodiment 1 which is such an embodiment, where an unacknowledged packet first sent on connection A at t = 0 ms with a 150 ms deadline has an early retransmission sent again on connection A at time t = 150 - 50 - 15 = 85 ms, where 50 ms is the assumed time variance and 15 ms is the known latency of connection A.

[0472] In one embodiment, the time variance is a fixed amount, for example, 50 ms, that can account for some variance in processing time and actual connection latency.

[0473] In another embodiment, the time variance is a variable amount that is a function of the connection type or measured property. For example, if the round-trip time of the connection has a statistically large standard deviation, the time variance for early retransmission can increase. In another example, satellite connections are known to have a relatively stable processing time compared to, for example, cellular connections, and thus can be assigned a lower time variance when considering early retransmissions.

[0474] In one embodiment, if this reliable connection with the lowest latency, available capacity, and fresh statistical information has already been used to send this packet, then to send an early retransmission of the packet, another reliable connection with a sufficiently low latency, fresh statistical information, and available capacity to pass the packet before its deadline is used. This is done to increase the diversity of the connections over which the same packet is sent, because the initially selected connection is potentially considered unreliable as it has not delivered a copy of the packet. FIG. 17 shows an example 1700 of Embodiment 2 which is such an embodiment, where connection A has already been used to send the packet initially and is thus avoided, and the early retransmission of the packet is sent on connection D at time t = 150 - 50 - 20 = 80 ms, with 50 ms being the assumed time dispersion and 20 ms being the known latency of connection D.

[0475] In one embodiment, the early retransmission of an unacknowledged packet approaching its deadline is sent on some or all of the connections that are either unreliable, do not have fresh statistical information, or both. Such connections may have become more reliable or now have more beneficial properties (e.g., having been unused for a while and thus potentially having less congestion), and can thus potentially pass the packet before its deadline. The number of such connections used for the early retransmission may depend on a balance between meeting the deadline and the efficient use of the aforementioned network connections. FIG. 17 shows an example 1700 of Embodiment 3 which is such an embodiment, where the early retransmission is sent on all unreliable and / or old connections. Each transmission time is calculated in the same way as in the previous embodiment of this figure. For connection F, the calculation results in a past time (t = 150 - 50 - 150 = -50 ms). Thus, the early retransmission on connection F is sent opportunistically at the same time as the earliest early retransmission on connection C (t = 75 ms) in this example.

[0476] In one embodiment, two or more of the above early retransmission approaches are combined. For example, one embodiment may combine all of the above approaches when serving a very important flow that should meet its deadline at any cost. Thus, early retransmission is sent on the original connection from which the packet was sent, other reliable connections with appropriate latency and EDT, available capacity and fresh statistical information, and all other connections with unreliable or old statistical information. FIG. 17 shows an example 1700 of Embodiment 4 which is such an embodiment, combining the early retransmissions of Embodiments 2 and 3.

[0477] In one embodiment, the above early retransmission approaches are considered sequentially (in any order) until at least one or more connections are found to send an early retransmission of the packet at hand.

[0478] Since early retransmission may ignore the CP rule under the assumption that the early retransmission is the last chance to meet the packet deadline, it is necessary to consider all available connections.

[0479] In some embodiments, a modified scheduling method for packets belonging to a flow with throughput preference is introduced, assuming a higher tolerance for delay / jitter / loss.

[0480] According to one embodiment, the scheduling algorithm is similar to the above description for real-time flows, except that each layer gives a score of 3 (excluding old or unreliable connections that still give a score of 0). Thus, packets belonging to a throughput flow can be transmitted only once (excluding old or unreliable connections).

[0481] FIG. 18 shows an example 1800 having various connections 1802 sorted into various layers 1804 used to send several packets 1806 for a throughput flow according to some exemplary embodiments. FIG. 18 shows a variation of the example of FIG. 16, but this time it is a throughput flow. The various layers 1804 and connections 1802 are the same except that all layers 1804 now have a score of 3. Connections 1802 with unreliable connections or old statistical information continue to acquire 0 regardless of which layer they belong to. As a result, the following exemplary results are obtained. ● Packets 1-4 reach the target score immediately after being sent on the first connection, which is connection A. At this point, the capacity of connection A is completely consumed. ● Packets 5 and 6 reach the target score after being sent on two connections. Layer 1: connection B and Layer 2: connection C. They accumulate scores of 0 and 3 respectively, resulting in a total score of 3, which meets the target score of 3. Note that since connection B in Layer 1 is unreliable, it acquires 0. At this point, the capacity of connection B is completely consumed. ● Packet 7 reaches the target score immediately after being sent on Layer 2: connection C. At this point, the capacity of connection C is completely consumed. ● Packet 8 reaches the target score immediately after being sent on Layer 2: connection D. At this point, the capacity of connection D is completely consumed. ● Packet 9 reaches the target score after being sent on two connections: Layer 2: connection E and Layer 3: connection F. Scores of 0 and 3 are accumulated respectively, resulting in a total score of 3, which meets the target score of 3. Note that connection E in Layer 2 acquires 0 because its statistical information is not fresh. At this point, the capacity of connection F is completely consumed. ● Even after being transmitted on one connection, Packet 10 does not reach the target score: Layer 2: Connection E. It accumulates a score of 0 and is less than the target score of 3. At this point, the capacity of Layer 2: Connection E is completely consumed. Even at this point, since the capacity of all connections is completely consumed, all transmissions stop. Unless Packet 10 is among the packets for which a confirmation response has been received, when more connection capacity becomes available (e.g., a packet for which a confirmation response has been received or a packet for which a new connection has been added), the transmission of Packet 10 resumes in the next round of transmission.

[0482] In some embodiments, the priority between flows can be implemented at a stage before packet scheduling (input queue). One packet queue can be created for each flow entering the system, and the queues are served according to the deficit round robin (DRR) and / or deficit weighted round robin (DWRR) queue scheduling algorithms (described in RFC8290) that attempt to be fair to all queues by default. The described embodiments introduce technical improvements as additional modifications that allow for the intentional unfairness to be configured by an administrator.

[0483] DRR / DWRR has the concept of a quantum (i.e., the target number of bytes to serve from each queue in each round). Usually, this quantum is equal for all queues, but the described modifications allow this to be modified by weights (specified as rules by administrator configuration). In some embodiments, the weights can be specified by other means, such as being automatically determined based on, among other things, the type of application, time, and machine learning.

[0484] For example, as follows. ○ Flow #1: Weight = 2 ○ Flow #2: Weight = 5 ○ Flow #3: Weight = 1

[0485] In this example, Flow #1 receives twice the nominal quantum per round, Flow #2 receives five times the nominal quantum, and Flow #3 receives one times the nominal quantum. In some embodiments, the quantum can be scaled, for example, by a multiple while maintaining relative weights. For example, multiply 2 / 8, 5 / 8, 1 / 8 by NominalQuantum * k, where NominalQuantum and k are parameters. In some embodiments, the weights can be specified in the rules, and all flows that match the rules share a common pool of quanta.

[0486] In the above example, k = 1, and NominalQuantum has a default value that can be specified manually by an administrator configuration or, inter alia, automatically based on the application type, time, or machine learning.

[0487] In some embodiments, the concept of queue importance levels is introduced. Higher importance queues are typically serviced before lower importance queues. In some embodiments, lower importance queues can be starved if there is only enough capacity to service higher importance queues. Queues with the same importance level can divide the capacity according to the assigned weights.

[0488] This embodiment describes a multi-level DRR / DWRR priority queue, thus improving existing DRR / DWRR. Conventional DRR / DWRR queues are described by the property that each flow "i" achieves the next minimum long-term average data rate.

Number

[0489] where Q i is the quantum of flow "i" and R is the total available WAN capacity.

[0490] In some of the described embodiments of the multi-level DRR / DWRR queue, a given flow "i" within priority level "M" that is pre-defined to have a higher "importance" so as to achieve the following minimum long-term average data rate can starve flows with a lower "importance".

Number

[0491] Where Q i is the quantum of flow "i" among the N flows within priority level M, and all of priority levels 0 to M-1 have a higher priority than M, and R j is the WAN capacity used by all of the flows within a particular priority level.

[0492] FIG. 19 shows an example of embodiment 1900 having a two-level DWRR queue management algorithm according to some exemplary embodiments. DWRR level 1 has a higher importance than DWRR level 2. DWRR level 1 has three flows with weights as described above, i.e., 2, 5, and 1 for flows 1, 2, and 3 respectively. DWRR level 2 has a single flow 4 with a weight of 2. In this example, assume there are three connections with an aggregate capacity of 16 packets per second. Assume the packet rates of the flows are as follows: flow 1 generates 4 packets per second, flow 2 generates 10 packets per second, flow 3 generates 3 packets per second, and flow 4 generates 3 packets per second. Since DWRR level 1 has a higher importance than DWRR level 2, DWRR level 1 is always serviced before DWRR level 2 as long as it has packets available from any of its flows. In this example, the flows of DWRR level 1 together generate a total of 17 packets per second, which is higher than the aggregate capacity of the connection. In other words, DWRR level 1 always has available packets, starves DWRR level 2, and is never serviced.

[0493] Table 1902 at the bottom of FIG. 19 shows the number of packets for each flow over time at the input and output.

[0494] Flows 1 and 2 at DWRR level 1 are highly important, generate packets in proportion to their quanta, and thus the number of output packets always catches up with the number of input packets.

[0495] Flow 3 at DWRR level 1 is more important and is served, but since it generates more packets than its shared quantum, it forms a packet backlog, which is seen as the number of input packets growing faster than the number of output packets.

[0496] Flow 4 at DWRR level 2 is less important and is not served because the more important DWRR level 1 generates enough packets to consume the full capacity of the available connections. Flow 4 is starved, which is considered the number of input packets always increasing while the number of output packets remains at 0. If Flow 4 were at DWRR level 1, some of its packets would have been served before DWRR returned to serve Flow 1, i.e., Flow 4 would not have been completely starved.

[0497] This embodiment introduces technical improvements to the above-described sequencer for more efficient operation per flow. The sequencer can use the following measurement properties to operate: a) Statistically filtered measurements of the one-way latency of each WAN connection. b) The set of WAN connections actively used by a particular flow.

[0498] FIG. 20 is a block diagram of a per-flow packet receiver 2000 according to some exemplary embodiments.

[0499] The per-flow packet receiver maintains, in 2001, an estimated value of the one-way latency of the WAN connection and, in 2002, a set of the most recently used WAN connections for a particular flow. These can be used to determine, in 2003, the appropriate time to release a packet from the flow output queue in order to minimize the jitter in the propagation time of the packets within that flow across the system.

[0500] In a basic implementation, both of these properties are measured and stored per flow. The described embodiment provides an improvement from the observation that, in (a) above, the statistically filtered measurements of the one-way latency of each WAN connection do not change per flow, so the estimated value of the one-way latency of the WAN connection in 2001 can be shared across all flows.

[0501] In some embodiments of the per-flow packet receiver, the one-way latency estimator can accept a tuning parameter that adjusts the quality of the estimated value based on the expected characteristics of the flow or the WAN connection. The tuning parameter can include a low-pass filter parameter and the duration after which individual packet measurements are discarded. In this embodiment, there can be two or more estimated values of the one-way latency of each WAN connection, and these estimated values can be shared across all flows that use the same tuning parameter.

[0502] In some embodiments, the estimated one-way latency of the WAN connection in 2001 can be used to calculate the EDT of the WAN connection.

[0503] The above-described embodiments, including improvements to stratification, scheduling, and sequencing (collectively, "real-time flow support"), have been tested in simulated scenarios that mimic real-world usage patterns and have been shown to provide quantitative improvements to jitter and latency. Specifically, typical scenarios are real-time video chat applications such as Zoom (trademark), Microsoft Teams (trademark), or Google Meet (trademark), used via a multi-WAN router that includes a wired broadband connection (e.g., consumer cable or DSL) and a 3xLTE connection. Connection priority (CP) rules are used to set the wired broadband connection as primary and the LTE connection as secondary.

[0504] The scenario then simulates a sudden failure of the wired broadband link (e.g., 100% packet loss and / or 0 bandwidth capacity) and measures the resulting impact on packet loss, latency, and jitter. The results are as follows: [Table 9]

[0505] The improvements to jitter and latency are due to the preemptive retransmission (target scoring system) and early retransmission logic. In this scenario, when the wired broadband connection suddenly fails, in-flight packets lost on the wired broadband connection are preemptively or early retransmitted on the LTE connection. In contrast, embodiments without real-time flow support must wait for a full RTO (retransmission timeout) to occur before retransmitting lost packets.

[0506] Figure 21 is a block diagram of a workflow 2100 of a per-flow communication system according to some exemplary embodiments. For a user having a smartphone connected via WiFi to a multi-LTE WAN router as described above, this experience may be similar to a video call via a wired broadband connection.

[0507] The per-flow communication system workflow 2100 according to one embodiment may follow such a process as outlined below, although in other embodiments, it may include more or fewer steps and may be executed in various orders.

[0508] In the exemplary workflow 2100 shown, starting at 2110, a target score is assigned to a packet based on the requirements of the flow to which the packet belongs. Next, at 2111, the connections may be tiered based on the flow requirements. For example, in the case of a real-time flow, per-packet deadlines ATL, as well as connection properties such as EDT and T prop are used to build the tiers. In the case of a throughput flow, the tier structure may be based on a unique pair of Mathis coefficient and capacity coefficient.

[0509] Next, at 2112, the scores may be assigned to the packet transmissions occurring on each tier's connections. Connections without fresh statistics (old connections) or unreliable connections (connections close to a 100% drop probability) are given a score of 0 regardless of the tier. At 2113, the packet scheduling process is started, iterating from best to worst across each tier and across each connection within each tier, and transmitting packets on the number of connections necessary to achieve a cumulative score above a pre-defined target score previously assigned at 2110.

[0510] Next, the connections within the currently selected layer can be selected pseudo-randomly at 2120. If the current layer is also the highest (best) layer, the selected connection is remembered as a sticky preferred connection. Future packets belonging to this flow will start from this connection always (instead of randomly selecting one pseudo-randomly) if still present in the best layer.

[0511] Next, the packet can be transmitted at 2121 and the cumulative score can be updated. Next, it can be determined at 2122 whether the target score has been reached. If not, at 2123, it is checked whether available connections occur. If yes, at 2124, the iteration can occur up to the next connection in the current layer, exhausting all available connections in the current layer before moving to the next layer.

[0512] At 2125, if the connection is still available in the current layer, the process can continue from 2121, transmit on the next connection in the layer, and update the score accordingly. If all connections in the layer have been exhausted, it can iterate to the next layer and the process can continue at 2120 by pseudo-randomly selecting a connection in this new layer.

[0513] If the target score is reached at 2122, it can be determined at 2126 whether there are remaining packets to be transmitted. If so, for the next packet, the whole process is restarted at 2110. If all available packets have been transmitted at 2126, then the packet scheduling is paused at 2130 until more packets arrive for transmission.

[0514] In 2122, the packet has not yet reached its target score, but in 2123, there may be no available connections in any of the layers. In that case, the lack of availability may be due to all connections having consumed their available capacity (in - flight packets being above their congestion window) or connection - priority rules preventing some connections from being activated. In this case, packet scheduling pauses at 2130 until an event occurs, such as the connection capacity becoming available due to in - flight packets being acknowledged. This allows the packet to start again at 2110 and resume the scheduling and transmission process, but with its partial cumulative score.

[0515] As a non - limiting exemplary implementation, consider following the workflow as shown in 2100, where a multi - LTE WAN router includes X connections, and connection 1 is equivalent to an old single LTE connection. Taking this into account, connections 2 through X exist. A user may initiate a voice call, and the voice call has a flow (packets 1, packets 2, packets N) that is latency / jitter - sensitive. The flow itself can also have a given latency / jitter requirement. Consider a maximum latency of 150 ms, a maximum jitter of 30 ms, and a maximum packet loss of 1% respectively.

[0516] In some embodiments, a flow can consist of packets with multiple different latency / jitter requirements, but similar to a standard router, a flow can be identified and separated based on the fields of a standard IP / TCP / UDP header (the "SrcIP, SrcPort, DstIP, DstPort, Protocol, DSCP" tuple). Typically, since an application knows this standard router property, if it is interested in the different latency / jitter requirements of different packets, it creates separate flows for them.

[0517] Next, consider the initial start assumptions for an exemplary implementation that walks through the exemplary workflow 2100. As noted above, recall that according to some embodiments, scores can be assigned to packet transmissions on each layer. In this example, layer 1 gives a score of 3, layer 2 gives a score of 2, and layer 3 gives a score of 1. Connections with no fresh or reliable statistics are given a score of 0 regardless of the layer. Packets can then be scheduled on the number of connections necessary to bring the packet cumulative score above a predefined target score. In this example, consider a target score of 3. Also assume that the application has a nominal target latency requirement of 150 ms and has the following initial properties with respect to the available connections. [Table 10]

[0518] Next, consider packet 1 and its layering. As noted above, recall that according to some embodiments, the layer is based on the relationship between the deadline for each packet of the flow (referred to as ATL) and the EDT and T prop properties of the connection. Assuming a nominal target latency of 150 ms, assume that the ATL of packet 1 is also 150 ms, resulting in the following layers. ○ Packet 1: i. Layer 1: Connections A, B, C ii. Layer 2: F, G iii. Layer 3: J, K iv. Layer 4: D, E v. Layer 5: H, I vi. Layer 6: L, M

[0519] As described above, the iteration can start at layer 1. In this example, assume that the first transmission of this flow is pseudo-randomly selected to be sticky to connection B. A transmission is attempted on B, but there is no currently available capacity. When iterating over the layers for connection C, one transmission is made, but since it is old, a 0 is given to the target. Iterating for connection A, one transmission is made, and A is fresh and gives a 3 to the target.

[0520] Considering the next packet 2, here the layer is the same, but the properties of B are changed so that there is now some available capacity. Since there is a sticky connection preference for B, packet 2 is transmitted once on B, and B is still fresh and gives a 3 to the target.

[0521] For packet 3, assume that connections B and C become unavailable (100% packet loss). B and C are moved to layer 2 (EDT becomes infinite and T prop is still the last measured value), and they are marked as old. Since the sticky connection preference for B is no longer in layer 1, it is forgotten. One transmission is made on the only remaining layer 1 connection A (fresh, gives a 3 to the target), and A is selected as the new sticky connection preference.

[0522] Packet 3: vii. Layer 1: A viii. Layer 2: B, C, F, G ix. Layer 3: J, K x. Layer 4: D, E xi. Layer 5: H, I xii. Layer 6: L, M

[0523] For packet 4, assume that connection A becomes unavailable (100% packet loss). A is moved to layer 2 (EDT becomes infinite and T prop(which is still the last measured value), is marked as old. Since layer 1 is empty, the sticky connection preference is completely forgotten and iteration starts in layer 2. Assuming that B is pseudo-randomly selected from layer 2 as the initial connection to start the iteration, packet 2 is transmitted once on B, and B, being old, gives 0 to the target.

[0524] Packet 4: xiii. Layer 1: xiv. Layer 2: A, B, C, F, G xv. Layer 3: J, K xvi. Layer 4: D, E xvii. Layer 5: H, I xviii. Layer 6: L, M

[0525] Iterate and transmit once for the next connection C (old, give 0 to the target).

[0526] Iterate and transmit once for the next connection F (fresh, give 2 to the target).

[0527] Iterate for the next connection G, but skip it as there is no available capacity.

[0528] Iterate and transmit once for the next connection A (old, give 0 to the target).

[0529] Layer 2 is exhausted and the target is not yet satisfied, so iterate for layer 3. Since there is no sticky connection preference for layers greater than 1, assume a pseudo-random start selection of J. Transmit packet 4 once on J (old, give 0 to the target).

[0530] Iterate and transmit once for the next connection K (fresh, give 1 to the target, and the cumulative score becomes 3).

[0531] For the last packet 5, assume that connection K becomes unavailable (100% packet loss). K remains at layer 3 (EDT becomes infinite and T prop is the last measured value and is still above the target latency of 150 ms) and is marked as old.

[0532] Packet 5: xix. Layer 1: xx. Layer 2: A, B, C, F, G xxi. Layer 3: J, K xxii. Layer 4: D, E xxiii. Layer 5: H, I xxiv. Layer 6: L, M

[0533] Assume that all previous old connections transmitted for packet 4 are still old. Typically, this is not actually the case because those old transmissions that occurred with packet 4 refresh the connection statistics and allow updated properties to be measured on each of the old connections. However, for the purpose of showing how secondary priorities can be used to meet the target, assume that this is the case.

[0534] Repeating all the same iterations from packet 4 reaches the exhausted layer 3 and the target score is still not met. Therefore, iterate over layer 4. Assume a pseudo-random start selection of E since there is no sticky connection preference for layer 4.

[0535] Transmit once on E (old, gives 0 to the target).

[0536] Iterate over the next connection D and transmit once (old, gives 0 to the target). Since layer 4 is exhausted and the target is still not met, iterate over layer 5. Assume a pseudo-random start selection of H since there is no sticky connection preference for layer 5. Transmit once on H (fresh, gives 2 to the target, cumulative score of 4).

[0537] Assuming there are no more available packets, scheduling and transmission are interrupted until an event occurs (for example, a new packet is received at the input, a negative acknowledgment response for a transmitted packet is received, etc.).

[0538] Also, consider that in all of the above cases of old preemptive packet transmissions, there may be many duplicates and packets in random order. In some embodiments, resequencing and duplicate elimination may occur, and the endpoint may receive packets in order.

[0539] The implementation of a per-flow communication system according to one embodiment may follow such a process as outlined above, but in other embodiments, it may include more or fewer steps and may be executed in various orders.

[0540] (In contrast to the real-time flow of the previous example) Consider another non-limiting exemplary implementation that may follow the exemplary workflow 2100 for a flow with throughput maximization requirements. As noted above, recall that in some embodiments, a score may be assigned to packet transmissions on each layer. In this example, the throughput flow configures all layers to give a score of 3. Connections without fresh or reliable statistics are given a score of 0 regardless of the layer.

[0541] In this example, the target score for the throughput flow is also 3, meaning that a single packet transmission on any fresh and reliable connection is sufficient to reach the target. The threshold for activating a secondary connection is also configured at 10 Mbps, meaning that the aggregate capacity of all primary connections must drop below 10 Mbps before the secondary connection becomes available for transmission.

[0542] Regarding the initial state of available connections, assume the following.

Table 11

[0543] Next, consider the transmission of packet 1 and its iteration. As noted above, according to some embodiments, recall that the layer for the throughput flow is based on a unique tuple of a rounded Mathis coefficient, a rounded capacity coefficient, and a connection priority. Considering that the smaller the value of the rounded Mathis coefficient, the better, and the larger the value of the rounded capacity coefficient, the better, this results in the following layer. Packet 1: Layer 1: Connections A, E Layer 2: B Layer 3: F Layer 4: C, D

[0544] As explained above, the iteration can start at layer 1. Assume that in this example, the first transmission of this flow is pseudo-randomly selected to be sticky to connection A. The transmission occurs on A, but it is old, so it gives 0 to the target.

[0545] Here, layer 1 is exhausted and the target is still not satisfied, so iterate for layer 2. Since there is no sticky connection preference for layers greater than 1 and there is only one connection in this layer, the pseudo-random selection provides a default selection of B. Since capacity is available, a transmission occurs. Since the connection is also fresh, a cumulative score of 3 is given and the desired target is reached.

[0546] Next, consider the transmission of packet 2 belonging to the same flow. Assume that all connection properties remain the same as those of packet 1, except that A is no longer old due to the transmission of packet 1 updating the measurement properties. Since the flow has a sticky connection preference for A, packet 2 is transmitted thereon and meets the target score of 3.

[0547] Next, consider the transmission of packet 3 belonging to the same flow. Assume that connection A becomes unreliable, old, and the rounded Mathis coefficient increases to the maximum value of 5,000. [Table 12]

[0548] Connection A has a unique tuple and is thus placed in its own layer. Packet 3: Layer 1: Connection E Layer 2: B Layer 3: A Layer 4: F Layer 5: C, D

[0549] The iteration starts at layer 1. Since connection A no longer exists in layer 1, the sticky connection preference for layer 1 is forgotten. Transmission is attempted on E and becomes the new sticky connection preference, but is skipped because there is no available capacity. The iteration proceeds to layer 2.

[0550] The sticky connection preference only exists for layer 1 and there is only one connection in layer 2, so the pseudo-random selection provides the default choice of B by default. Since capacity is available, transmission occurs. Since the connection is also fresh, a score of 3 is given to the cumulative score and the desired target is reached.

[0551] Next, consider the transmission of packet 4 belonging to the same flow. Assume that connection B becomes unreliable, the rounded Mathis coefficient increases to the maximum value of 5,000, and it becomes old. [Table 13]

[0552] As a result, the same tuple as connection F is obtained, so they exist in the same layer.

[0553] Packet 4: xxv. Layer 1: Connection E xxvi. Layer 2: A xxvii. Layer 3: B, F xxviii. Layer 4: C, D

[0554] The iteration starts at Layer 1. Connection E is a sticky connection preference and exists in Layer 1, but is skipped because there is no available capacity. The iteration proceeds to Layer 2. Since the sticky connection preference only exists for Layer 1 and there is only one connection in Layer 2, the pseudo-random selection provides the selection of A by default. Because it is old and unreliable, even if transmission occurs, 0 is given to the cumulative score.

[0555] The iteration proceeds to Layer 3. Since the preference for sticky connections only exists for Layer 1, assume that the selection of the pseudo-random connection first selects connection F. Transmission occurs, but because it is old, 0 is given to the target score. The iteration proceeds to connection B within the layer. Because it is old and unreliable, it also contributes to the target score after transmission.

[0556] Since all primary connections are exhausted, the iteration stops and does not proceed to Layer 4. The connection priority rule is configured to activate secondary connections only when the total fresh and reliable capacity drops below 10 Mbps. This threshold is still met by connection E, but there is no current capacity.

[0557] Packet 4 does not have more connections available for transmission, but has not yet met its target score of 3. The system waits until an event occurs that allows further transmission. For example, an acknowledgment of an in-flight packet on connection E that increases the available capacity, a failure of connection E that meets the threshold for activating secondary connections, or updated measurement properties of any of the other primary connections such that they are no longer old and / or unreliable.

[0558] Assume that an event occurs where the capacity of connection E drops below 8 Mbps and the capacity factor also decreases.

Table 14

[0559] The layers remain the same and the iterative steps are repeated from the beginning. However, this time, instead of stopping at layer 3, it proceeds to layer 4 because the available aggregated primary capacity dropped below the set threshold of 10 Mbps (only 8 Mbps is available in connection E).

[0560] Since sticky connection preference exists only for layer 1, assume that the selection of pseudo-random connection first selects connection D from layer 4. Transmission occurs, but since it is old, it is given a target score of 0. The iteration proceeds to connection C within the layer. Since it is fresh, it is given a packet score of 3. The target is met and the transmission of packet 4 is completed.

[0561] Assuming there are no more available packets, scheduling and transmission are interrupted until an event occurs (e.g., a new packet is received at the input, a negative acknowledgment response for a transmitted packet is received, etc.).

[0562] Also, consider that in all cases of the above-described old packet transmissions, there may be many duplicates and packets in random order. In some embodiments, re-sequencing and duplicate elimination may occur, and the endpoint may receive packets in order.

[0563] The implementation of a per-flow communication system according to one embodiment may follow such a process as outlined above, but in other embodiments, it may include more or fewer steps and may be executed in various orders.

[0564] FIG. 22 is a block diagram showing a workflow 2200 that may occur between 2121 and 2122 of FIG. 21, specifically for real-time flows, according to some exemplary embodiments. These flows have the ability to trigger early retransmission of packets when the deadline is approaching, but have not yet been acknowledged.

[0565] An exemplary workflow 2200 according to one embodiment may follow such a process as outlined below, but in other embodiments, may include more or fewer steps and may be executed in various orders.

[0566] In the exemplary workflow 2200 shown, starting at 2210, the transmitted packet (generated at 2121 in FIG. 21) checks whether this is the first transmission of the packet. If not, since this means that a previous transmission of the packet on a better connection already exists, the early retransmission eligibility check will already be complete. Accordingly, the packet returns to 2122 in FIG. 21.

[0567] However, if it is the first transmission of the packet, at 2211, a check is performed if there is sufficient time before the packet deadline to complete an early retransmission attempt. In some embodiments, this check consists of three components.

[0568] The best-case duration of this first transmission and the corresponding ACK (which is typically the connection RTT)

[0569] The worst-case duration of an early retransmission attempt to succeed after the best-case duration has elapsed without receiving an ACK (which is typically the connection EDT)

[0570] A predefined "overhead" value taking into account system processing time (in some embodiments, this is a constant of 50 ms)

[0571] If the sum of these three components exceeds the current packet ATL, at 2212, it is marked as ineligible for early retransmission. In both cases (eligible and ineligible), the packet returns to step 2122 in FIG. 21.

[0572] As a non-limiting, exemplary implementation that can follow the workflow as shown in 2200, consider Packet 1 from a real-time flow with a 200 ms ATL, scheduled for its first transmission on Connection A with an RTT of 25 ms and an EDT of 100 ms. At 2211, the sum of the three components (RTT + EDT + 50 ms = 25 + 100 + 50 = 175 ms) is less than the ATL, so the packet is eligible for retransmission. The explanation for this comparison is that in the nominal scenario, it is expected that Connection A will deliver Packet 1 and ACK it within its 25 ms RTT. If 25 ms elapses and no ACK or NACK has been received yet, the system may begin to suspect that the transmission has been lost and may consider an early retransmission to meet the deadline. For there to be enough time for this early retransmission to be effective, there must be additional EDT + processing time overhead remaining before the deadline of Packet 1.

[0573] In this example, since there is enough time, Packet 1 is eligible to be considered for early retransmission and returns to Step 2122.

[0574] Next, consider that Packet 1 needs multiple transmissions to reach its target score, as explained by workflow 2100, so Packet 1 enters the workflow again. This time, at 2210, since Packet 1 is not on its first transmission, it returns directly to 2122.

[0575] Next, consider that Packet 2 from the same real-time flow enters at 2210 with a 150 ms ATL. Also assume that it is scheduled for its first transmission on Connection A with the same RTT and EDT parameters as Packet 1. This time, since the sum of the three components (175 ms) is greater than the ATL, Packet 2 is marked as ineligible for early retransmission at 2212. Next, at 2122, it returns to workflow 2100.

[0576] The implementation of the per-flow communication system according to one embodiment may follow such a process as outlined above, but in other embodiments, it may include more or fewer steps and may be executed in various orders.

[0577] FIG. 23 is a block diagram showing a workflow for considering early retransmission of in-flight packets according to some exemplary embodiments. In particular, FIG. 23 is a block diagram showing a workflow 2300 for considering early retransmission of in-flight packets (packets that are transmitted through workflows 2100 and 2200, meet their target scores, but have not yet been acknowledged as received or lost).

[0578] An exemplary workflow 2300 according to one embodiment may follow such a process as outlined below, but in other embodiments, it may include more or fewer steps and may be executed in various orders.

[0579] In the exemplary workflow 2300 shown, starting at 2310, iterations via in-flight packets belonging to the real-time flow occur in ascending order of packet deadlines.

[0580] Packets with past deadlines are skipped at 2311, and packets that have already been early retransmitted previously are skipped at 2312. Any packet marked as ineligible for early retransmission as part of workflow 2200 is also skipped at 2312.

[0581] If the packet is not skipped, at 2320, its ATL is compared with the EDT of all available connections. For all fresh connections, the one with the best EDT that is smaller than the packet ATL and is not the first connection used to transmit the packet is returned as a candidate for early retransmission at 2321. If no such candidate exists, (when the EDT < ATL requirement is met) a plurality of candidates consisting of all old connections and the first connection used to transmit the packet are returned at 2322.

[0582] In 2323, candidates with the worst EDT are checked against the packet deadline. If, due to the deadline being very close, the packet needs to be transmitted for this candidate to reach the destination in time, the packet is scheduled to be immediately retransmitted early for all candidates in 2324. In this exemplary embodiment, the definition of "very close" is a predefined constant of 50 ms, meaning that there is less than 50 ms between the current time + the connected EDT and the packet deadline.

[0583] Otherwise, the packet is not close enough to its deadline and does not guarantee early retransmission, and the system prefers to wait longer if the packet was successfully delivered by the initial transmission that occurred via workflow 2200.

[0584] If there are available ones, the iteration continues with the next in - flight packet at 2313; otherwise, the early retransmission workflow ends at 2314 and resumes later. In this exemplary embodiment, workflow 2300 is triggered by a timer set according to the "very close" deadline described above for the packet with the earliest deadline. The workflow is also always triggered just before workflow 2100.

[0585] As a non - limiting exemplary implementation that can follow the workflow as shown in 2300, consider packet 1, which is the only in - flight packet with a future deadline of T = 1000 ms where the current time is t = 100 ms. Assume that there is a next connection in the system and the first connection on which packet 1 was initially transmitted is C.

Table 15

[0586] In 2310, since packet 1 is the only in - flight packet, it is selected for iteration.

[0587] In 2311, Packet 1 is determined to have no past deadline (current time is t = 100ms, deadline is T = 1000ms).

[0588] In 2312, the packet is determined to be eligible for early retransmission (not marked as ineligible beforehand by workflow 2200) and has not been early retransmitted yet.

[0589] In 2320, since the ATL is 900ms (T = 1000ms - t = 100ms), all four of connections A, B, C, and D satisfy the (EDT < ATL) requirement. However, B does not satisfy the freshness requirement, and C was the first connection used to transmit Packet 1, so C is also excluded. As a result, connections A and D remain as the only eligible ones.

[0590] Since connection A has the best EDT out of the two, connection A is returned as the only eligible candidate in 2321.

[0591] In 2323, connection A does not satisfy the "packet deadline is near" requirement. In this exemplary embodiment, recall that (now + EDT + 50ms) >= PacketDeadline, or (t = 100ms + 150ms + 50ms) = 300ms, which is not >= T = 1000ms.

[0592] As a result, Packet 1 is not early retransmitted. Iterations occur over more packets in 2313, but since there are none remaining, the early retransmission workflow ends in 2314.

[0593] In this exemplary embodiment, the timer is set to resume workflow 2300 at t = 800ms, which is a time when, if Packet 1 is still in flight at that point, the deadline of Packet 1 is considered close enough to require early retransmission on connection A.

[0594] Assume that this timer is triggered and Packet 1 is still in flight, but the connection properties are changed so that connection A becomes old.

Table 16

[0595] The previous steps 2310, 2311, and 2312 remain the same.

[0596] At 2320, since the ATL is 200 ms (T = 1000 ms - t = 800 ms), only connections A, C, and D satisfy the (EDT < ATL) requirement. However, A no longer meets the freshness requirement, and C was the first connection used to transmit Packet 1, so C is also excluded. As a result, connection D remains as the only remaining eligible connection and is returned at 2321 as a candidate.

[0597] At 2323, connection D meets the "Packet deadline is approaching" requirement. In this exemplary embodiment, recall that (now + EDT + 50 ms) >= PacketDeadline, or (t = 800 ms + 195 ms + 50 ms) = 1045 ms, which is >= T = 1000 ms.

[0598] As a result, Packet 1 is early retransmitted on connection D at 2324.

[0599] The iteration to the next packet occurs at 2313. Assuming no other packets are in flight, the early retransmission workflow ends at 2314.

[0600] Next, assume that Packet 2 is in flight, has a deadline of T = 2000 ms, was initially transmitted on connection C, and the current time is t = 1900 ms. Assume that the connection properties have not been changed since Packet 1.

Table 17

[0601] Assume that Packet 2 passes the checks at Stages 2311 and 2312 and moves to 2320. The ATL is 100 ms (T = 2000 ms - t = 1900 ms). Only connection C meets the EDT < ATL requirement and is also the first connection used to transmit Packet 2.

[0602] As a result, 2320 has no option but to proceed to 2322, where it returns all the old connections plus the first connection from the initial transmission. This means that all of connections A, B, and C are returned.

[0603] At 2323, the connection with the worst EDT is B, which meets the "packet deadline approaching" requirement. In this exemplary embodiment, recall that (now + EDT + 50 ms) >= PacketDeadline, or (t = 1900 ms + 300 ms + 50 ms) = 2250 ms, which is >= T = 2000 ms.

[0604] As a result, Packet 2 is early retransmitted on all candidates, connections A, B, and C. The iteration to the next packet occurs at 2313. Assuming no other packets are in - flight, the early retransmission workflow ends at 2314.

[0605] The implementation of a per - flow communication system according to one embodiment may follow such a process as outlined above, but in other embodiments, it may include more or fewer steps and may be executed in various orders.

[0606] FIG. 24 is a diagram of an implementation 2400 of a per - flow communication system according to some exemplary embodiments.

[0607] In an emergency, first responders do not need to worry about establishing a connection, nor do they need to be IT experts. First responders and law enforcement agencies rely on advanced technologies to improve situation awareness and response times. Transmitting real-time video and data from the scene requires a large amount of bandwidth, and relying on a single connection can make an organization vulnerable.

[0608] Emergencies can occur in a variety of environments, including unreliable network conditions where reliable connectivity is still required. This connection needs to provide appropriate connection security, diversity, redundancy, latency, and bandwidth required for communications such as real-time video and data. These communications often have latency / jitter requirements, and with current technologies, available WAN connections alone cannot consistently meet real-time requirements. The reason for not being able to meet the requirements can be due to connection overload (competition / contention for connection capacity) or connection failures / inconsistencies (e.g., cut fiber, wireless interference, failed equipment). Connection contention can be anywhere along the path (first / middle / last mile), anywhere where the capacity of a particular network link is exceeded.

[0609] The above-described embodiments utilize multiple WAN connections to be able to meet these real-time requirements despite the unreliability of individual WANs.

[0610] Consider an exemplary implementation 2400 of an embodiment as shown in FIG. 24. In this example, police officer 2402 can be seen using body camera 2404. Police officer 2402 may be in pursuit, for example, in a spectrally challenging environment such as an apartment, and body camera 2404 can relay real-time live video and audio of the situation to, for example, a 911 dispatch center.

[0611] As can be seen in 2400, their connections to network 2414 may be overloaded or they may have suffered a failure of main connection 2406. Router 2408 is a multi-LTE WAN router according to one of the above embodiments and has multiple connections 2410. Router 2408 may be able to maintain reliable communication with the dispatch center via a plurality of other connections. Data packets for video and audio communication may be transmitted according to the processes described in the above embodiments, for example, ensuring that latency and jitter requirements are met so that the video feed is clear for a dispatch center including dispatch worker 2412.

[0612] Similarly, since connection failures can occur with any type of connection, the dispatch center itself can be connected to the network using a multi-WAN router 2408 that supports multiple wired and wireless broadband connections 2410 (e.g., fiber, cable, DSL, LEO satellite, LTE, etc.). According to the above embodiments, connection priority rules can be used to set un-metered connections as primary (e.g., fiber, cable, DSL, LEO satellite) and metered, more expensive connections (e.g., LTE) as secondary. Data packets for video and audio communication may be transmitted according to the processes described in the above embodiments, ensuring that latency and jitter requirements are met even if the dispatch center experiences congestion or failure simultaneously on multiple WAN connections. This commonly occurs in dispatch centers in areas where there are limited options for un-metered connections, such as areas that may only have access to low-quality DSL lines and oversubscribed LEO satellite services.

[0613] Other public safety related use cases can include movement commands, e.g., reliable connections that are required to enable connected command vehicles to send and receive data from cameras, sensors, and edge devices. First responders must be able to obtain the information necessary to make life-saving decisions quickly and reliably.

[0614] As police departments around the world increase their use of real-time video, the amount of video produced and consumed by law enforcement agencies is growing rapidly. The described embodiments present solutions to this increased demand that simultaneously require a high level of reliability, enable multiple different connections for sending data packets, sort / hierarchize these connections while complying with strict requirements, send the packets, and the processing to ensure the packets arrive in a timely manner.

[0615] The described embodiments can provide for the use of fleet connections that enable "movement zones" to help make appropriate response decisions or obtain important data on-site in order to send live video back to a command center, keep teams safe, and respond more quickly. For example, a mobile camera can capture a license plate and immediately send it to the edge for analysis to alert whether a police officer needs to take action.

[0616] Real-time video applications, such as those described in the exemplary implementation 2400, can also ensure that the central command has real-time video to support decision-making during sports events, parades, protests, and other events that attract large moving crowds.

[0617] The described embodiments can enable an unmanned aerial system, or drone, to provide real-time video from the air to improve situational awareness for law enforcement agencies and fire chiefs.

[0618] FIG. 25 is a diagram of an implementation 2500 of a per-flow communication system according to some exemplary embodiments.

[0619] As VoIP grows, the needs, processes, and deliverables of broadcasting continue to evolve. The described embodiments may provide, for example, a single service for uninterrupted live broadcasts from remote locations. Some embodiments may be user-friendly, integrable into a broadcast workflow, and may be versatile for different environmental uses (e.g., mounted on a vehicle, worn in a backpack, integrated into a drone, etc.).

[0620] Broadcast communications typically have latency / jitter requirements, and with current technology, available WAN connections alone cannot consistently meet real-time requirements. The reasons for not being able to meet the requirements may be due to connection overloading (competition / contention for connection capacity) or connection impairments / mismatches (e.g., cut fiber, wireless interference, faulty equipment). Connection contention can be anywhere along the path (first / middle / last mile), anywhere where the need exceeds the capacity of a particular network link.

[0621] The above-described embodiments utilize multiple WAN connections to meet these real-time requirements, despite the lack of reliability of individual WAN connections.

[0622] Consider the exemplary implementation shown in FIG. 25. Reporter 2502 may be reporting a breaking news of a live event at a remote broadcast location where reporter 2502 is filming with broadcast camera 2504, and due to the fact that a large crowd at the live event is all using their mobile phones 2506, main connection 2508 has become very slow due to overload and contention. A slow connection clearly does not help with good broadcast production.

[0623] The reporter 2502 may need to transmit real-time video, be present on a VoIP call with the main broadcast anchor, and receive topics via their video and audio, as well as a teleprompter at a remote location. Low latency and jitter are required to ensure that the remote team remains synchronized with the central production 2514 along with the reporter 2502.

[0624] The router 2510 may be able to transmit this low-latency return video and teleprompt feed to the live broadcast production team on-site, along with the central production, and may enable them to view these feeds simultaneously on tablets and mobile devices. The router 2510 is a multi-LTE WAN router according to one of the above-described embodiments.

[0625] The router 2510 may be able to maintain a reliable communication with the central production via a plurality of other connections 2512. Data packets for video communication may be transmitted according to the processes described in the above embodiments, ensuring that latency and jitter requirements are met so that the video feed is clear for broadcast.

[0626] The router 2510 may provide a reliable connection to the network 2516, which may provide high-throughput wireless Internet to the remote team with the reporter 2502, which may improve their productivity by providing access to cloud services, online resources, media assets, and newsroom systems from anywhere. The router 2510 may similarly be a portable and / or in-vehicle device. The described embodiments may provide remote connectivity such that media assets and newsroom systems may be remotely accessed, files may be quickly transferred back and forth, and easier communication with on-site personnel is enabled.

[0627] Figure 26 is a diagram of an implementation 2600 of a per-flow communication system according to some exemplary embodiments. Figure 26 shows another example of a broadcast implementation.

[0628] The routes of storms, floods, and other extreme weather can interrupt broadcast operations, while damage or cuts to fiber optic cables can take days or weeks to locate and repair. As shown in 2600, the main connection 2602 has been damaged and cut due to severe weather events and can no longer transmit data.

[0629] The router 2604 can provide a portable and permanent backup connectivity solution due to multiple LTE WAN connections 2606. This can, as shown in 2600, keep the broadcast 2608 on air and keep a remote team connected to the network 2610 during an emergency weather event. In such situations, it can be very important for public safety for news broadcasts to remain active in order to provide the public with accurate and up-to-date information live, such as when a major natural disaster like an earthquake occurs. The described embodiments can keep the broadcast 2608 on air and maintain content delivery capabilities, for example, if damage occurs to a broadcast facility or studio 2612.

[0630] Other exemplary broadcast implementations and use cases of the described embodiments are shown in FIG. 25 and can be for remote contributions, as described above. The embodiments can also be useful in the case of remote production where the production activity itself is being produced remotely.

[0631] The described embodiments can also enable cost-effective delivery of content to, for example, network-related companies, group stations, or other broadcast stations and media organizations.

[0632] FIG. 27 is a schematic diagram of a computing device 2700 that can be used to implement the system 1200A and / or 1200B according to one embodiment.

[0633] As shown, computing device 2700 includes at least one processor 2702, memory 2704, at least one I / O interface 2706, and at least one network interface 2708.

[0634] Each processor 2702 can be, for example, a microprocessor or a microcontroller (e.g., a dedicated microprocessor or microcontroller), a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read only memory (PROM), or various combinations thereof.

[0635] Memory 2704 can include various combinations of computer memories, located either internally or externally, such as random access memory (RAM), read only memory (ROM), compact disk read only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read only memory (EPROM), and electrically erasable programmable read only memory (EEPROM), ferroelectric RAM (FRAM), etc.

[0636] Each I / O interface 2706 enables computing device 2700 to interconnect with one or more input devices such as a keyboard, mouse, camera, touch screen, and microphone, or one or more output devices such as a display screen and speaker.

[0637] Each network interface 2708 enables the computing device 2700 to communicate with other components, exchange data with other components, access and connect to network resources, provide applications, and execute other computing applications by connecting to a network (or networks) that can carry data including the Internet, Ethernet, Plain Old Telephone Service (POTS) lines, Public Switched Telephone Network (PSTN), Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), coaxial cable, fiber optic, satellite, mobile, wireless (e.g., Wi-Fi, WiMAX), SS7 signaling network, landline, local area network, wide area network, and others including various combinations thereof.

[0638] For simplicity only, one computing device 2700 is shown, but system 1200A and / or 1200B may include multiple computing devices 2700. The computing devices 2700 may be of the same or different types. The computing devices 2700 may be connected in various ways including being directly connected, indirectly connected via a network, distributed over a wide geographic area, and connected via a network (which may be referred to as "cloud computing").

[0639] For example, without limitation, the computing device 2700 may be a server, network appliance, set-top box, embedded device, computer expansion module, personal computer, laptop, personal data assistant, mobile phone, smartphone device, UMPC tablet, video display terminal, game console, or various other computing devices capable of being configured to execute the methods described herein.

[0640] FIG. 28 is FIG. 2800 depicting a physical computer server rack according to some exemplary embodiments. The computer server rack shown in 2800 includes a plurality of networked computing components including rack-mounted computing appliances such as networked routers, switches, hubs, gateways, etc. In this example, the components interact with each other to control routing, for example, by establishing and / or periodically updating routing tables and / or routing rules. In another embodiment, the components interact to control routing by controlling network connections, routing data packets, etc., according to routing tables and / or routing rules.

[0641] As described herein, an approach is a technical computing solution that can be implemented in the form of a physical data router or other networking device configured to control the routing of data packet communications. The device can include one or more processors operating in conjunction with computer memory, and the one or more processors can be coupled to data storage. A corresponding method, and a non-transitory computer-readable medium (e.g., a disk, solid-state storage device, hard disk drive) storing machine-interpretable instructions (e.g., software that, when executed by a computer processor, causes the processor to execute the methods described herein in various embodiments).

[0642] FIG. 29 is a diagram depicting a practical variation using a computing device with reduced computing capabilities according to some exemplary embodiments.

[0643] In FIG. 29, reference numeral 2900 shows an embodiment of a variant of the system for each flow described herein. In this variant, the system is adapted to operate using a computing device 2904 with reduced computing capabilities (e.g., a Raspberry Pi™ device or an Arduino™ device, etc.) on one or the other side of the communication.

[0644] In this exemplary embodiment, computationally expensive or resource-intensive operations (e.g., processing time, speed, power, memory / storage requirements) are performed on the side with more available resources, while the other side is assigned to perform basic operations that require only minimal computing resources.

[0645] This is useful in practical situations where the asymmetric levels of investment and computing power are required, for example, due to the availability of computing devices in a particular region or the difficulty of transporting expensive and heavy equipment to such regions (e.g., responders in an emergency zone, remote areas, areas with little transport infrastructure).

[0646] Accordingly, an asymmetric computing scenario is possible and contemplated, and the system can be adapted to operate with these types of configurations.

[0647] In particular, the side with more resources can perform computationally expensive operations and wirelessly transmit the results, such as the network model for each connection (e.g., capacity, cwnd, loss rate, RTO that requires data collection over time, mathematically required statistical operations, and filtering operations), to the other side.

[0648] The negotiation protocol can be implemented optionally, and both sides can communicate their capabilities and mutually agree on who will compute what and communicate it to the other side.

[0649] In particular, each side can be associated with a data structure that advertises the corresponding capabilities, and an automatic mutual agreement process can be executed using a scoring / ranking algorithm.

[0650] In another variant, a discovery protocol is utilized, and each side learns what the other is capable of so that even if the network enables a high bit rate, the other side is not overwhelmed computationally from a packet processing perspective, and manages the transmission of packets. The discovery protocol can be practically implemented using data processes that are executed both asynchronously or synchronously, for example, to keep a list of available computing capabilities up to date. This can be useful when devices draw resources from a pool of shared capabilities in a distributed computing system where resources can be dynamically allocated and assigned.

[0651] Thus, as shown in FIG. 29, the dynamic allocation of computing tasks for network routing can be shared between two devices such that a device with more computing capabilities can be used to control the routing and send routing instructions to the other in the form of data messages and packets. In particular, a device with more computing capabilities can be used to manage the layer structure and establish priority routing while maintaining and / or sensing the priorities of various network connections. A device with more computing capabilities can manage a composite property representing the estimated delivery time and apply various scheduling approaches to address issues that arise when sorting connections by the property.

[0652] In particular, the hierarchical approach enables the establishment of functional equivalence for connections belonging to a particular layer, based on various definitions such as a concrete approach based on a per-flow basis (e.g., different approaches for flows with throughput preference as opposed to real-time flows which may have latency / jitter preference). This distinction can be used, for example, to more finely control the probability of a retransmission approach / reception approach such that a preemptive retransmission approach is only applied to real-time flows.

[0653] Accordingly, a device having more computing power can manage and track the ATL value as a dynamic value that changes per packet and over time, for example, as the packet ages. The device having more computing power then identifies connections for scheduling packets for delivery, transmits routing instructions accordingly, and in addition, connection priority rules and logical flows can be utilized to establish a more granular approach for control or used to affect how the layers are constructed and / or maintained.

[0654] During the scheduling approach, a device having more computing power performs score assignment, scheduling, and controls communication by any device across various network connections.

[0655] Furthermore, in some embodiments, early retransmission of packets can be controlled to occur by a device having more computing power. A per-packet deadline can be maintained and early retransmission can be controlled in certain exemplary situations such as transmitting on all old connections, and / or on the first connection.

[0656] By controlling which device performs which activity, the reduced computing capabilities of multiple devices can be taken into account, the load can be shared, or it can be fully passed to one device (e.g., a device with more resources). This can also be used, inter alia, to manage other types of computing characteristics such as device wear / lifetime usage management, heat levels, etc. Thus, asymmetric computing can be a useful variant from the perspective of computing load balancing to address technical problems arising from differences in computing characteristics or computing environments. The examples described mainly have the load being fully shifted to a device with more computing capabilities, but this does not necessarily have to occur. Instead, the load for a specific activity can be balanced between two devices.

[0657] The terms "connected" or "coupled" can include both direct coupling (where two elements coupled to each other are in contact with each other) and indirect coupling (where at least one additional element is located between the two elements).

[0658] Although embodiments have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope. Further, the scope of this application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, compositions of matter, means, methods, and steps described herein.

[0659] As will be readily understood by those skilled in the art, processes, machines, manufactures, compositions of matter, means, methods, or steps of substances, whether currently existing or later developed, that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein can be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufactures, compositions of matter, means, methods, or steps.

[0660] As can be understood, it is intended that the embodiments described and illustrated above are merely exemplary.

Claims

1. A device for coordinating data communication across multiple wide area network connections, comprising: a processor coupled to a computer memory and a non-transitory computer-readable storage medium, wherein the processor: for each of the multiple wide area network connections of the multiple wide area network connections, monitors at least one of a latency property, a packet loss property, and a throughput property of packets transmitted on the wide area network connection; for each packet having a latency / jitter preference within a plurality of routed packets, identifies an adjusted target latency (ATL) based at least on a per-packet deadline for communication of the packet to a target endpoint; groups the per-packet ATLs and the multiple wide area network connection properties to establish the multiple hierarchical groupings such that each of the multiple wide area network connections is grouped into a corresponding hierarchical grouping of multiple hierarchical groupings; is configured to communicate the packets using one or more selected wide area network connections selected from the multiple wide area network connections, using at least the multiple hierarchical groupings.

2. The latency property and the packet loss property of the wide area network connection include an estimated delivery time (EDT) and are generated based on the following relationship: 【Number 1】 where P ｄｒｏｐ is defined as the packet drop probability, T ｐｒｏｐ is defined as a one-way propagation delay, T ｒｔｐｒｏｐ is defined as the round-trip propagation delay, The device according to claim 1.

3. The device according to claim 1, wherein the wide area network connections belonging to each of the multiple hierarchical groupings are considered to be functionally equivalent for the selection of the one or more selected wide area network connections.

4. The plurality of hierarchical groupings includes at least a first layer (layer 1), a second layer (layer 2), and a third layer (layer 3) defined based on the relationship between the ATL for each packet, the plurality of wide area network connection properties, EDT, and T ｐｒｏｐ ​ Layer 1: T ｐｒｏｐ ≤ ATL && EDT ≤ ATL Layer 2: T ｐｒｏｐ ≤ ATL && EDT > ATL Layer 3: T ｐｒｏｐ > ATL && EDT > ATL The device according to claim 1.

5. The device according to claim 4, wherein the multiple hierarchical groupings include additional layers that subdivide layers 1, 2, and 3 based on one or more predefined connection priority rules such that the result is a new set of layers from layer 1 to layer N.

6. The selection of the one or more selected wide area network connections from among the available wide area network connections is repeated from the available wide area network connections in layer 1, then layer 2, and finally layer 3, the device of claim 4.

7. The communication of the packet using the one or more selected wide area network connections selected from the plurality of wide area network connections includes scheduling the packet on a plurality of connections selected from the plurality of wide area network connections for preemptive retransmission, the device of claim 6.

8. The wide area network connections in each layer are assigned a layer-based score, and the scheduling of the packet on a plurality of connections is performed to achieve a cumulative score above a predefined target score, the device of claim 7.

9. If the scheduling of the packet on a plurality of connections on a higher layer cannot achieve the cumulative score above the predefined target score, the connection priority rules are ignored, the device of claim 5.

10. Connections designated as unreliable or having no recently monitored properties are assigned a layer-based score of 0, regardless of their corresponding layer level, the device of claim 8.

11. The selection of the selected wide area network connection from among the available wide area network connections in hierarchical grouping includes selecting the available wide area network connections pseudo-randomly or algorithmically, the device of claim 6.

12. The initial selection of the selected wide area network connection for the data flow corresponding to the packet includes an initial pseudo-random selection, and for future packets corresponding to the data flow, the selection of the selected wide area network connection is maintained, the device of claim 11.

13. The selection of the selected wide area network connection is maintained only until a connection pause period threshold is reached or an explicit flow end notification occurs, at which point further packets on the data flow utilize a further initial pseudo-random selection, the device of claim 12.

14. The processor is further configured to classify the data flow into (i) a throughput preference flow and (ii) a real-time flow, and only packets from the real-time flow are identified as packets having a latency / jitter preference, the device of claim 1.

15. For packets from the throughput preference flow, the grouped layers are established based on the same composite property pair based on at least one or more measurement properties of the plurality of wide area network connections, the device of claim 14.

16. The processor is tracking the per-packet deadline of packets transmitted on a first wide area network connection that are approaching their deadline but have not been acknowledged, A T that is short enough to communicate the packet that is approaching its deadline but has not been acknowledged to the corresponding target endpoint ｐｒｏｐ The device of claim 1, further configured to perform an early retransmission of the packet that is approaching its deadline but has not been acknowledged, using a second wide area network connection having an available congestion window (CWN) along with the backlog value.

17. The processor is tracking the per-packet deadline of packets transmitted on a first wide area network connection that are approaching their deadline but have not been acknowledged, A T that is short enough to communicate the packet ｐｒｏｐ and, together with the backlog value, determine that there is no available non-old second wide area network connection having an available congestion window (CWN), and in response to the determination, further configured to perform an early retransmission of the packets that are approaching their deadline but have not been acknowledged using one or more older wide area network connections, the device of claim 1.

18. The processor is further configured to perform an early retransmission of the packet on the first wide area network connection used to initially transmit the packet, the device of claim 17.

19. For packets associated with the throughput preference flow, each layer is assigned the same layer-based score that matches the predefined target score, the device of claim 10.

20. The processor is further configured to establish a priority among the packets within the flow that defines a packet queue for the flow before communicating the packet using the selected wide area network connection, the device of claim 1.

21. The packet queue is unloaded according to a multi-level deficit weighted round robin (DWRR) scheduling approach, and the queue with a higher importance level starves the queue with a lower importance level, preventing the queue with the lower importance level from being serviced regardless of the weight assigned to the queue with the lower importance level. The device according to claim 20.

22. The ATL is a dynamic value that changes for each packet. The device according to claim 1.

23. The ATL changes as the packet ages. The device according to claim 22.

24. The ATL is defined as a packet deadline time value minus the current time value. The device according to claim 23.

25. The current expiration time value of the packet is equal to the original reception time (T ＰａｃｋｅｔＲｅｃｅｉｖｅｄ ) of the packet added to the nominal target latency requirement of the flow, the device according to claim 24.

26. The device is a network router. The device according to claim 1.

27. The device is a physical chipset existing within a network router. The device according to claim 1.

28. The network router exists within a data center and is configured to control the communication of data packets communicated by one or more connected endpoint devices. The device according to claim 26.

29. The data packets include both packets having a throughput preference and packets having a latency / jitter preference. The device according to claim 28.

30. The network router controls the communication of a media broadcast station or an emergency dispatch center. The device according to claim 28.

31. A method for coordinating data communication across a plurality of wide area network connections, For each wide area network connection of the plurality of wide area network connections, monitoring the latency property and the packet loss property of the packets transmitted over the wide area network connection; For each packet having a latency / jitter preference among the plurality of routed packets, identifying an adjusted target latency (ATL) based at least on a per-packet deadline for the communication of the packet to a target endpoint; Grouping the per-packet ATL and the plurality of wide area network connection properties to establish the plurality of hierarchical groupings such that each of the plurality of wide area network connections of the plurality of wide area network connections is grouped into a corresponding hierarchical grouping of a plurality of hierarchical groupings. Communicating the packet using one or more selected wide area network connections selected from the plurality of wide area network connections, at least using the plurality of hierarchical groupings. A method comprising.

32. The latency property and packet loss property of the wide area network connection include an estimated delivery time (EDT) and are generated based on the following relationship. 【Number 2】 where P ｄｒｏｐ is defined as the packet drop probability, T ｐｒｏｐ is defined as a one-way propagation delay, T ｒｔｐｒｏｐ is defined as the round-trip propagation delay The method according to claim 31.

33. The method according to claim 31, wherein the wide area network connections belonging to each hierarchical grouping of the plurality of hierarchical groupings are considered to be functionally equivalent for the selection of the wide area network connection for communication.

34. The plurality of hierarchical groupings includes at least a first layer (layer 1), a second layer (layer 2), and a third layer (layer 3) defined based on the relationship between the ATL for each packet and the plurality of wide area network connection properties, EDT, and T ｐｒｏｐ and is defined based on the relationship between the ATL for each packet and the plurality of wide area network connection properties, EDT, and T Layer 1: T ｐｒｏｐ ≤ ATL && EDT ≤ ATL Layer 2: T ｐｒｏｐ ≤ ATL && EDT > ATL Layer 3: T ｐｒｏｐ > ATL && EDT > ATL The method according to claim 31.

35. The method according to claim 34, wherein the plurality of hierarchical groupings include additional layers that subdivide layers 1, 2, and 3 based on one or more predefined connection priority rules such that the result is a new set of layers from layer 1 to layer N.

36. The method according to claim 34, wherein the selection of the one or more selected wide area network connections from among the available wide area network connections is repeated from the available wide area network connections in layer 1, then layer 2, and finally layer 3.

37. The method according to claim 36, wherein the communication of the packet using the one or more selected wide area network connections selected from the plurality of wide area network connections includes scheduling the packet on a plurality of connections selected from the plurality of wide area network connections for preemptive retransmission.

38. The wide area network connection in each layer is assigned a layer-based score, and the scheduling of the packets on multiple connections is performed to achieve a cumulative score above a predefined target score. The method according to claim 37.

39. If the scheduling of the packets on multiple connections on a higher layer cannot achieve the cumulative score above a predefined target score, the connection priority rules are ignored. The method according to claim 35.

40. Connections designated as unreliable or having no recently monitored properties are assigned a layer-based score of zero, regardless of their corresponding layer level. The method according to claim 38.

41. The selection of the one or more selected wide area network connections from the available wide area network connections in the hierarchical grouping includes randomly selecting the available wide area network connections in a pseudo-random manner. The method according to claim 36.

42. The initial selection of the selected wide area network connection for the data flow corresponding to the packet includes an initial pseudo-random selection, and for future packets corresponding to the data flow, the selection of the selected wide area network connection is maintained. The method according to claim 41.

43. The selection of the one or more selected wide area network connections is maintained only until a connection idle period threshold is reached or an explicit flow end notification occurs, at which point further packets on the data flow utilize a further initial pseudo-random selection. The method according to claim 42.

44. Including classifying the data flow into (i) a throughput preference flow and (ii) a real-time flow, and only packets from the real-time flow are identified as packets having a latency / jitter preference. The method according to claim 31.

45. For packets from the throughput preference flow, the grouped layers are established based on the same composite property pair based on at least one or more measured properties of the multiple wide area network connections. The method according to claim 44.

46. Tracking the per-packet deadlines of packets transmitted over a first wide area network connection that is approaching its deadline but has not been acknowledged, and Communicate the packet that is approaching its deadline but has not been acknowledged to the corresponding target endpoint with a T that is short enough. ｐｒｏｐ And, with the backlog value, perform an early retransmission of the packet that is approaching its deadline but has not been acknowledged using a second wide area network connection having an available congestion window (CWN). The method according to claim 31, comprising. **Claim 47** Tracking the per-packet deadlines of packets transmitted over a first wide area network connection that is approaching its deadline but has not been acknowledged, and A T that is short enough to communicate the packet ｐｒｏｐ and, along with the backlog value, determining that there is no available non-old second wide area network connection having an available congestion window (CWN) In response to the determination, using one or more older wide area network connections to perform an early retransmission of the packet that is approaching its deadline but has not been acknowledged; The method according to claim 31, comprising. **Claim 48** Performing an early retransmission of the packet on the first wide area network connection used to initially transmit the packet, The method according to claim 47, comprising. **Claim 49** For packets associated with a throughput preference flow, each layer is assigned the same layer-based score that matches the pre-defined target score, the method according to claim 40. **Claim 50** Before communicating the packet using the selected wide area network connection, establishing a priority among the packets within the flow that defines a packet queue for the flow, The method according to claim 41. **Claim 51** The packet queue is unloaded according to a multi-level deficit weighted round robin (DWRR) scheduling approach, and the queue with a higher importance level starves the queue with a lower importance level, preventing the queue with the lower importance level from being serviced regardless of the weight assigned to the queue with the lower importance level, the method according to claim 50. **Claim 52** The ATL is a dynamic value that changes for each packet, the method according to claim 31. **Claim 53** The ATL changes as the packet ages, the method according to claim 52. **Claim 54** The ATL is defined as a packet deadline time value minus the current time value, the method according to claim 53. **Claim 55** The current expiration time value of the packet is equal to the sum of the original reception time (T ＰａｃｋｅｔＲｅｃｅｉｖｅｄ ) of the packet and the nominal target latency requirement of the flow. The method according to claim 54 **Claim 56** The method is executed on a network router, the method according to claim 31. **Claim 57** The method is executed by a physical chipset present within a network router, the method according to claim 31. **Claim 58** The method according to claim 56, wherein the network router is present within a data center and is configured to control the communication of data packets communicated by one or more connected endpoint methods.

59. The method according to claim 58, wherein the data packets include both packets having a throughput preference and packets having a latency / jitter preference.

60. The method according to claim 58, wherein the network router controls the communication of a media broadcast station or an emergency dispatch center.

61. A non-transitory computer-readable medium storing a machine-interpretable instruction set, wherein when the machine-interpretable instruction set is executed by a processor, the processor is caused to execute the method according to any one of claims 31 to 60.

62. The selection of the one or more selected wide area network connections from among the available wide area network connections in hierarchical grouping includes selecting the wide area network connection having the shortest backlog, and the backlog is defined as the number of in-flight bytes measured in time units and divided by the connection bit rate. The device according to claim 6.

63. The selection of the one or more selected wide area network connections is maintained only as long as the administrative configuration permits. The device according to claim 12.

64. The selection of the one or more selected wide area network connections from among the available wide area network connections in hierarchical grouping includes selecting the wide area network connection having the shortest backlog, and the backlog is defined as the number of in-flight bytes measured in time units and divided by the connection bit rate. The method according to claim 36.

65. The selection of the one or more selected wide area network connections is maintained only as long as the administrative configuration permits. The method according to claim 42.

Images