Systems and methods for communicating using a hybrid wide area network

Abstract

An apparatus for coordinating data communications across wide area network (WAN) connections, the apparatus comprising a processor coupled with a computer memory and a non-transitory computer-readable storage medium, the processor configured to: monitor, for each WAN connection, latency and packet loss rate characteristics of packets transmitted on the WAN connection; identify, for each packet of a plurality of packets being routed having a latency / jitter preference, an adjusted target latency (ATL) based at least on a per-packet deadline for delivery of the packet to a target endpoint; group, using the per-packet ATLs and the plurality of WAN connection characteristics, to establish a plurality of hierarchical groupings, such that each WAN connection is grouped into a corresponding hierarchical grouping of the plurality of hierarchical groupings; and transmit the packets using one or more selected WAN connections, the one or more selected WAN connections selected using at least the plurality of hierarchical groupings.

Description

[0001] Cross-reference of related applications

[0002] This application is non-provisional U.S. Application No. 63 / 350,654, filed June 9, 2022, entitled "Systems and Methods for Communications Using Blended Wide Area Networks," and claims protection for all benefits of that U.S. application. This application is incorporated herein by reference.

[0003] This application relates to the following applications, which are incorporated herein by reference in their entirety:

[0004] PCT application No. PCT / CA2017 / 051584 entitled “PACKET TRANSMISSION SYSTEM AND METHOD”, filed on 21 December 2017.

[0005] PCT application No. PCT / CA2021 / 050732, entitled “Systems and methods for data transfer across unreliable connections”, filed on May 28, 2021.

[0006] PCT application No. PCT / CA2022 / 050077, entitled “Systems and methods for push-based data communications”, filed on January 19, 2022. Technical Field

[0007] Embodiments of this disclosure relate to the field of electronic data packet communication, and more specifically, to apparatus, systems, and methods for push-based data packet communication in the context of multiple network connections (e.g., wide area network connections). Background Technology

[0008] Existing packet communication solutions for transmitting packets across multiple available network connections rely on static routing logic to determine how to deliver packets across connections. For example, a system might use timers for each available connection, with corresponding timers determining when to pull packets from the input queue for transmission on the connection associated with the timer. These solutions assume connections have similar operational characteristics and availability, trading performance for implementation simplicity, thus scheduling the next available packet for transmission based on the connection with the most recently expired timer. While the timer approach is simple, it can compromise computational efficiency in certain situations.

[0009] If multiple networks have different operating characteristics, communication systems that treat multiple network connections as multiple connection instances with similar operating characteristics (e.g., LACP, ML-PPP) may lead to reduced efficiency in the use of multiple connections, as some networks may be underutilized or overutilized. This inaccuracy in usage can result in longer latency when delivering data packets, reduced reliability of data packet delivery, higher data packet delivery costs, and reduced throughput.

[0010] Connection management solutions also face challenges when dealing with applications where available connections or their operational characteristics change over time. For example, timer-based systems may be insensitive to changing network performance because each connection's timer starts independently, so the timer may not accurately know the network performance of all other connections.

[0011] The desired approach is a system and method for data communication that utilizes multiple connections more efficiently to transmit data packets with lower latency and increased throughput, making data packet delivery more reliable or less costly. Summary of the Invention

[0012] Providing a reliable communication mechanism capable of meeting diverse and competing communication requirements across various applications running on networked devices is technically challenging. Communication requirements from each application compete for limited communication resources, necessitating computational methods for coordinating and controlling the use of these limited resources. These challenges are further complicated in the context of multiple wide area networks (WANs), where control devices (such as network routers) are coupled to multiple WAN connections and configured to control how packets communicate across one or more interfaces coupled to said multiple WAN connections.

[0013] For example, a network router can be an electronic device or an onboard processor with routing logic or circuitry, the onboard processor being configured to control packet communication (e.g., packet delivery and / or packet reception) as required by various client applications.

[0014] Network routers can be used in practical implementations, such as controlling communications at mobile or fixed communication centers, such as portable media vehicles (e.g., for broadcasting) or data center facilities (e.g., emergency dispatch centers). In these cases, network routers utilize multiple WANs to improve uptime and reliability, where individual connections may be susceptible to communication failures (e.g., during emergencies or periods of high demand following denial-of-service attacks) or communication challenges (e.g., changes in spectral characteristics connecting to various cellular towers or satellites as the vehicle travels). Because multiple WANs are used, different options and available communication pathways exist, requiring additional methods for sequencing and / or possibly reordering / regenerating sequences at the receiving endpoint. For example, WANs can be used to intelligently distribute packets among themselves to control data packets corresponding to a specific media broadcast for individual communication across multiple WANs, and, as described herein, redundant packets can be preemptively sent across connections at the expense of efficiency to improve overall reliability. Examples of using multiple WANs can include routers with multiple SIM slots, multiple wired Ethernet ports, and / or multiple satellite connections.

[0015] Packet communication can be requested by different types of client applications, and it is important to note that these communications may have different types of communication requirements, namely (i) throughput-oriented requirements (e.g., bulk file transfers) or (ii) latency / jitter-oriented requirements (e.g., real-time use, such as VoIP, video conferencing). Multiple types of client applications may have communication needs at the same time or around the same time (e.g., a media van at a sporting event running applications simultaneously, uploading files for post-production over several hours, sending low-quality data feeds for quick previews, and holding video conferences for sideline reporters), and these client communications may compete for limited, constrained resources.

[0016] As discussed in this paper, latency / jitter requirements are particularly difficult to meet because the packet stream is coupled with a real-time delivery deadline, which requires packets to arrive on time with minimal jitter / loss. When packets fail to arrive on time or experience jitter or loss, issues such as stuttering, freezing, and pauses may occur, such as audio and video playback problems. In contrast, applications with throughput requirements in streaming may be able to handle latency and jitter, for example, by using reordering and buffering.

[0017] In one proposed approach, an apparatus for coordinating data communication across multiple wide area network connections is proposed, the apparatus being configured to have a packet routing device coupled to a processor and a non-transitory computer-readable storage medium.

[0018] The apparatus is configured to, for each of the plurality of WAN connections, monitor the latency, packet loss rate, and throughput characteristics of packets transmitted over the WAN connection; for each packet having a latency / jitter preference among the routed packets, identify an adjusted target latency (ATL) at least based on a per-packet deadline for communicating the packet to a target endpoint; group packets using the per-packet ATL and the characteristics of the plurality of WAN connections to establish a plurality of hierarchical packets, such that each of the plurality of WAN connections is grouped into a corresponding hierarchical packet within the plurality of hierarchical packets; and communicate the packets using one or more selected WAN connections selected from the plurality of WAN connections, the one or more selected WAN connections being selected using at least the plurality of hierarchical packets.

[0019] On the one hand, WAN connection latency and packet loss rate characteristics include estimated delivery time (EDT).

[0020] The WAN connection belonging to each of the plurality of layered packets is considered functionally equivalent in order to select the WAN connection for communication, and the plurality of layered packets may include at least a first layer (layer 1), a second layer (layer 2), and a third layer (layer 3), the layers being based on the per-packet ATL and the plurality of WAN connection characteristics EDT and one-way propagation delay (T). prop The relationships between the layers are defined. The number of layers is not limited to three, and in one variation, the plurality of hierarchical groups include additional layers that subdivide layer 1, layer 2, and layer 3 based on one or more predefined connection priority rules, thereby forming a new set of layers.

[0021] The selection of the chosen WAN connection from the available WAN connections is performed iteratively, for example, from available WAN connections in Layer 1, then Layer 2, and finally Layer 3, and packet communication can use the chosen WAN connection selected from the plurality of WAN connections. For example, packet scheduling can be performed on multiple connections selected from the plurality of WAN connections for preemptive retransmission.

[0022] Each WAN connection in each layer can be assigned a layer-based score, and packet scheduling across multiple connections is performed to achieve a cumulative score greater than or equal to a predefined target score. In a variant, if packet scheduling across multiple connections at better (e.g., alternating, higher, lower) layers fails to achieve the cumulative score greater than or equal to the predefined target score, connection priority rules are ignored. In some embodiments, a useful bypass mechanism is provided. Connections designated as unreliable or lacking recently monitored characteristics can be assigned a layer-based score of 0, regardless of their corresponding layer level, to encourage the use of more reliable and / or recently monitored characteristics. Such connections are still used for packet transmission to measure their updated monitoring characteristics, which will determine whether they have become reliable again.

[0023] Selecting a WAN connection from available stratified packets may include pseudo-random selection of available WAN connections, such that for a data stream corresponding to a packet, the initial selection of the chosen WAN connection includes an initial pseudo-random selection, and for future packets corresponding to the data stream, the selection of the chosen WAN connection is maintained, for example, until a connection dormancy period threshold has been reached or an explicit flow termination notification has occurred. This helps WAN connections become more "sticky," causing the system to be biased to use a consistent WAN connection for a specific flow (if available) within a given timeframe. The technical benefit of having a certain level of enforced consistency lies in its reduction of the inefficiencies associated with instantiating or initiating different connections for use. For example, this can be used to avoid scenarios where the system continuously selects between two or more different connections based on minor improvements, resulting in inefficient data communication due to the non-ideals introduced by opening and closing different connections for use.

[0024] The device can segment and / or classify data streams into (i) throughput-biased streams and (ii) real-time streams; and only packets from real-time streams are identified as packets with latency / jitter bias, such that packets from throughput-biased streams are grouped into layers based on the same comprehensive characteristics based at least on one or more measured characteristics of the plurality of wide area network connections.

[0025] In some embodiments, the selection of a chosen WAN connection from the available tiered packets is not based on pseudo-random selection, but rather on the connection with the shortest backlog among the tiered packets. Backlog is measured in time units (typically milliseconds) and is defined as the volume of bytes in flight (transmitted but not acknowledged) divided by the WAN connection's transmission rate (throughput). When a flow requires more bandwidth than any single connection can provide on its own, selecting the connection with the shortest backlog minimizes the latency experienced by the flow.

[0026] In some embodiments, the apparatus can track the deadline of each packet sent on the first wide area network connection that is approaching its deadline but has not yet been acknowledged; and use a congestion window (CWND) with an available timeout and a sufficiently short timeout. prop The second wide area network connection to the backlog value performs early retransmission on packets that are approaching their deadline but have not yet been acknowledged, in order to communicate the packets that are approaching their deadline but have not yet been acknowledged to the corresponding target endpoint, or determine that there is no available congestion window (CWND) and a sufficiently short T. prop The backlog value is used to communicate the packet via an available, non-outdated second WAN connection; and in response to the determination, early retransmission is performed on the packet that is approaching its deadline but has not yet been acknowledged using one or more outdated WAN connections. Additionally and / or simultaneously, early retransmission may be performed, for example, on the first WAN connection initially used to transmit the packet.

[0027] In another embodiment, a method is proposed in which priorities are established among the flows to which the packet belongs before the packet is communicated using the selected WAN connection. The priorities are translated into a packet queue defined for each flow, and packets are offloaded from the defined packet queues based on a modified differential round-robin (DRR) scheduling state.

[0028] Corresponding methods and non-transitory computer-readable media have been considered. As described in the various embodiments herein, the system can be implemented as a network controller that operates as a physical networking device for controlling communication across multiple network interfaces and / or network connections. The network controller can be implemented, for example, as a specific hardware device, or as a chipset or set of embedded logic gates integrated into another hardware device or operating on a computer server or software.

[0029] The application involves controlling network connectivity in situations where connections are unreliable, a large number of alternative connections exist, but network resources are limited, and / or there may be sudden load spikes or outages. For example, the networking method can be used in situations involving emergency dispatch control or media broadcasting (e.g., in rural or congested areas).

[0030] This can be useful in situations where multiple demanding applications are running together, or even a single non-demanding application exists, but the connection itself is malfunctioning or experiencing problems.

[0031] Real-world scenarios include sports broadcasting, live event broadcasting, public safety, and high uptime / critical infrastructure. The method helps leverage combinations of connections to improve overall packet delivery and communication characteristics, and is particularly beneficial if the connections have varying networking characteristics that change over time (e.g., with congestion), with device location (e.g., the device is mobile), and with the price and / or standards of service delivery (e.g., best-effort contention connections shared across multiple users). Attached Figure Description

[0032] The accompanying drawings illustrate embodiments by way of example. It should be clearly understood that the descriptions and drawings are for illustrative purposes only and are intended to aid in understanding.

[0033] Embodiments will now be described by way of example only with reference to the accompanying drawings, in which:

[0034] Figure 1A This is a schematic diagram of a pull communication system based on some example embodiments;

[0035] Figure 1B This is a schematic diagram of a push communication system according to some example embodiments;

[0036] Figure 1C This is a flowchart of a method for transmitting data packets to push packets to multiple networks based on availability, according to some example embodiments;

[0037] Figure 1D This is a flowchart of another method for transmitting data packets to order connections based on connection metadata, according to some example embodiments;

[0038] Figure 1E This is a flowchart of another method for transmitting data packets and distributing packets to a network based on a network-selected order, according to some example embodiments;

[0039] Figure 1F This illustrates, according to some example embodiments, a process flow for ordering network connections to control packet routing.

[0040] Figure 2 This is a schematic diagram illustrating the scheduling of data packets of different priority levels according to some example embodiments;

[0041] Figure 3 It is based on some example embodiments to schedule packets of different priority levels. Figure 1B A series of schematic diagrams of a push communication system;

[0042] Figure 4A This is based on the use of some example embodiments. Figure 1BA schematic diagram of connection selection in a push communication system, taking into account connection and stream type;

[0043] Figure 4B This is based on the use of some example embodiments. Figure 1B A schematic diagram of data packet selection in a push communication system;

[0044] Figure 4C This is based on the use of some example embodiments. Figure 1B A schematic diagram of data packet allocation in a push communication system;

[0045] Figure 4D This is based on the use of some example embodiments. Figure 1B A diagram illustrating another data packet allocation method in a push communication system;

[0046] Figure 5A This involves scheduling two independent data streams for transmission, based on some example embodiments. Figure 1B A schematic diagram of a push communication system;

[0047] Figure 5B This involves scheduling two independent data streams for transmission, based on some example embodiments. Figure 1B A schematic diagram of a push communication system;

[0048] Figures 6A to 6E Each shows a corresponding screenshot of an interface for managing packet transmission according to an example embodiment; and

[0049] Figure 7 This is a diagram illustrating data packet transmission based on some example embodiments;

[0050] Figure 8A Includes two diagrams illustrating one-way data packet transmission according to some example embodiments;

[0051] Figure 8B Includes two illustrations illustrating the packet transmission that occurs in response to an acknowledgment, according to some example embodiments;

[0052] Figure 8C These are two additional illustrations of packet transmissions that occur without the need for acknowledgment to send additional packets, based on some example embodiments.

[0053] Figure 9A This is a graphical representation of actual throughput data during flight based on some example embodiments;

[0054] Figure 9B This is another illustration of actual throughput data during flight based on some example embodiments;

[0055] Figure 9CThis is an additional illustration of actual throughput data during flight based on some example embodiments;

[0056] Figure 10A These are a series of diagrams illustrating packet transmission experiencing congestion, based on some example embodiments;

[0057] Figure 10B These are a series of illustrations showing packet transmission after adjusting transmission capacity in response to connection congestion, based on some example embodiments;

[0058] Figure 11 This is an example block diagram illustrating a hybrid connectivity system that can operate as a network gateway, based on some example embodiments;

[0059] Figure 12A This is a schematic diagram of a per-stream communication system according to some example embodiments;

[0060] Figure 12B This is a schematic diagram of a per-stream communication system according to some example embodiments;

[0061] Figure 13A The diagram illustrates the calculation of estimated delivery time according to some example embodiments;

[0062] Figure 13B The diagram illustrates the calculation of estimated delivery time according to some example embodiments;

[0063] Figure 14 This is a schematic diagram illustrating wide area network connections in different hierarchical packets with three different layers based on estimated time delivery, according to some example embodiments.

[0064] Figure 15A The illustrations are based on some example embodiments that show examples of pseudo-random hashing and flow viscosity;

[0065] Figure 15B This is a diagram illustrating sticky expiration after idle time and after TCP FIN exchange, based on some example embodiments;

[0066] Figure 15C The illustrations, based on some example embodiments, show how pseudo-random selection helps keep packets of a single stream on the same connection and how additional requirements can traverse multiple layers.

[0067] Figure 16 This is a diagram illustrating an example of multiple connections ordered into multiple layers for sending a number of packets, based on some example embodiments.

[0068] Figure 17 This is an illustration of an example of early retransmission based on some example embodiments;

[0069] Figure 18 This is a diagram illustrating, according to some example embodiments, an instance of multiple connections ordered across multiple layers for sending a throughput stream;

[0070] Figure 19 Examples of a two-level differential weighted round-robin (DWRR) queue management algorithm according to some example embodiments are shown;

[0071] Figure 20 This is a block diagram of the per-flow packet receiver according to some example embodiments;

[0072] Figure 21 This is a block diagram of the workflow of a per-stream communication system according to some example embodiments;

[0073] Figure 22 This is a block diagram illustrating the workflow of a real-time stream according to some example embodiments;

[0074] Figure 23 This is a block diagram illustrating a workflow that takes into account the early retransmission of packets in flight, based on some example embodiments.

[0075] Figure 24 This is an illustration of a per-stream communication system implementation based on some example embodiments;

[0076] Figure 25 This is an illustration of a per-stream communication system implementation based on some example embodiments;

[0077] Figure 26 This is an illustration of a per-stream communication system implementation based on some example embodiments;

[0078] Figure 27 This is an example schematic diagram of a computing device according to an example embodiment; and

[0079] Figure 28 This is an illustration of a physical computer server rack based on some example embodiments.

[0080] Figure 29 The illustration depicts a practical variation using a computing device with reduced computing power, based on some example embodiments. Detailed Implementation

[0081] Systems, methods, and apparatus for transmitting data packets when multiple transport connections are available are described, and are alternatively referred to as hybrid routers or hybrid systems. The methods described below will be expanded upon.

[0082] Existing solutions for transmitting data packets over multiple available transport connections are typically implemented as a "pull"-based architecture, where data packets are transmitted in response to events generated or defined by the device or system, such as internal timers associated with multiple available connections.

[0083] Internal timers can respond to individual connection parameters (e.g., the duration of a timer configured to reduce the recovery time of retransmitted packets), but not to connection parameters of other available connections.

[0084] Due to the independence of timer events, pull-based systems may be underutilized (e.g., resulting in idle connections when timer values ​​are too long) or overutilized (e.g., potentially allocating packets to less reliable connections despite poor performance), which can lead to longer packet delivery latency, reduced throughput, lower packet delivery reliability, and higher packet delivery costs.

[0085] The methods, systems, and apparatuses proposed herein transmit data packets according to a push-based architecture to transmit data packets over multiple available connections. The disclosed methods, systems, and apparatuses include a scheduler configured to, in response to monitored communication events, retrieve a subset of operational characteristics (monitored features) for each of a plurality of connections, and iteratively allocate data packets available for transmission to any of the determined connections based on their monitored features, to achieve transmission capacity. In example embodiments, other monitored network features may be combined with transmission requirements associated with the data packets to allocate data packets to connections that match the transmission requirements of the data packets.

[0086] Using a push-based architecture to schedule packets when multiple connections are available is previously unused. Typical multipath systems (which can be alternatively called hybrid connection systems) allocate packets and flows to connections on a static basis, as these systems do not consider flow requirements or the operational characteristics of the connections. For example, a multihomed Internet Protocol (IP) router allocates packets to connections purely based on the destination IP address in the packet header, matching it to its destination routing table. Implementing a pull-based architecture is simpler when connection allocation does not change frequently. For example, there is no need to handle situations where a push scheduler has already allocated a large queue of packets to connections, but because those connections are offline, a recall or reallocation must now be performed.

[0087] The proposed methods, systems, and apparatus receive data streams, including one or more data packets, from an input network via an input connection.

[0088] The proposed methods, systems, and apparatus include a three-stage data pipeline that transmits data packets to remote peers through a mixed set of connections. New data intended for multiple connections is passed to a first pipeline stage, where it is queued and subsequently passed to a second pipeline stage associated with each connection in the mixed pipeline. The second pipeline stage compiles statistics for its underlying connections and passes packets to a third pipeline stage. The second stage may provide congestion control and packet buffering to ensure packets are available for the next pipeline stage. The third stage may then write packets to the network interface according to a pacing or timing cues provided by the second stage.

[0089] Another embodiment may have a single second-level instance, which then provides packets to third-level instances, each of which manages packet transmission on a set of network interfaces shared by a physical interface card. Other embodiments may have varying numbers of instances for each pipeline level.

[0090] Figure 1A This is a schematic diagram of a pull communication system 100A according to some example embodiments.

[0091] When each connection timer (e.g., timers 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively) wakes up, the connection timer invokes the pull scheduler 130 and requests data packets from the packet input queue 102 for transmission. Data packets can be requested in a burst manner, where the burst size nominally reflects the value of bytes in 20 milliseconds (1 / 50 Hz) at the estimated bit rate for the corresponding connection for each timer. A limitation associated with such pull timers is that the number of bytes requested each time a timer wakes up does not reflect the full congestion window (represented, for example, "capacity") of the connection.

[0092] Because each timer starts independently, the pull scheduler 130 lacks visibility into the overall transmission state and cannot make globally optimal scheduling decisions. For example, connection 110A could be a terrestrial broadband connection, and connection 110B could be a bandwidth-on-demand (BoD) satellite connection. Connection 110A might be superior to connection 110B due to cost, latency, etc. However, when connection 110B's timer 108B wakes up and is invoked into the pull scheduler 130, the pull scheduler 130 lacks visibility into the next wake-up time of connection 110A's timer 108A (because timer 108A starts independently). The pull scheduler 130 lacks a basis for determining whether connection 110A will be able to process all packets in the input queue 102, or whether connection 110B should be assigned additional packets to complete the transmission.

[0093] If connection 110B is indeed helpful, then pull scheduler 130 may not determine which connection should be used to transmit which data packets are available for transmission, because the pull scheduler only includes visibility of timers.

[0094] The data packets are transmitted across the connection and are subsequently received by receiver 116 via connection 114. Receiver 116 includes sequencer 118 for assembling the transmitted data packets and can further transmit the assembled data packets via local area network (LAN) 120.

[0095] Compared to pull-based architectures, in one embodiment, the proposed system is adapted to schedule packets to specific connections to improve effective connection utilization. However, not all architectures considered necessarily need to be pull-based.

[0096] Figure 1B This is a schematic diagram of a push communication system 100B according to some example embodiments. In the example embodiments, system 100B is designed to use a total connection type of multiplexing technology to provide multiple access (MA) to a shared communication channel. For example, in a code division multiple access (CDMA) network, the orthogonal codes assigned to each node allow those nodes to share a set of frequencies and transmit simultaneously without interfering with each other.

[0097] The characteristic of connection types using these multiplexing techniques is that available capacity is rapidly redistributed to nodes in the network, typically within tens to hundreds of milliseconds. In the example embodiment, system 100B is designed with this characteristic in mind, thereby redistributing traffic among available WAN connections based on this timescale.

[0098] In contrast, connections using Allocated Multiple Access on Demand (DAMA) technology (also known as Bandwidth on Demand (BoD)) share communication channels by having nodes negotiate access to the channel within a time period. Once allocated, a node can exclusively use its negotiated portion. Thus, available capacity is shared by nodes sequentially using the negotiated portion of the channel.

[0099] In the case of BoD connections, this negotiation and subsequent redistribution of capacity between nodes typically takes about a few seconds (redistribution time, typically a medium single-digit to low double-digit number) to complete. In the example embodiment, system 100B will require changes (described herein) to effectively utilize these types of connections.

[0100] In the current implementation, long redistribution times (and associated packet loss rates and / or increased latency) are interpreted as congestion, and typical congestion control methods reduce the use of connections where congestion is detected to avoid overloading those connections. However, BoD connections actually require the opposite behavior (at least temporarily) in order to request and potentially receive increased capacity allocation.

[0101] System 100B includes an input queue 102 that receives data streams (including one or more data packets) from an input network for transmission. Examples of data streams may be file transfers, video file transfers, audio file transfers, Voice over Internet Protocol (VoIP) calls, or data packets associated with a Virtual Private Network (VPN).

[0102] Input queue 102 is also responsible for applying packet loss policies if the scheduler (as described herein) fails to serve input queue 102 at the same rate as new packets arriving from LAN-side clients. This typically occurs if the LAN client's transmission rate exceeds the total WAN transmission capacity.

[0103] System 100B comprises three levels, alternatively referred to as a pipeline or pipeline level, for transmitting data packets to remote peers via a hybrid set of connections according to a push-based architecture.

[0104] The first stage 104 includes a stream classification engine 104A and a scheduler 104B (which may alternatively be referred to as push scheduler 104 below), and receives data packets from the input queue 102 and queues the data packets for delivery to the second stage 106. In an example embodiment, the first stage 104 includes all of the input queue 102, the stream classification engine 104A, and the scheduler 104B.

[0105] Input queue 102 may buffer packets received from clients on the sender's LAN side by utilizing or cooperating with the flow classification engine 104A, and is responsible for classifying packets into flows (or alternatively referred to as mapping packets to flows). Input queue 102 ensures that the scheduler of first stage 104 serves data flows fairly.

[0106] In an example embodiment, the first level 104 or its auxiliary stream classification engine 104A monitors or is notified of changes to user-configured settings associated with allocating packets to connections. The user-configured settings include information about how connections should be allocated to priority levels, when those priority levels should become proactive, and user-configurable stream classification rules that may cause packets belonging to a stream to prefer connections with certain monitored characteristics (e.g., packets belonging to a video stream classification might prefer connections with low latency, while packets belonging to a file download classification might prefer connections with lower monetary costs).

[0107] Level 104 periodically learns current metadata (e.g., operational characteristics) for each of the multiple connection monitoring instances (as described herein) corresponding to a WAN connection containing hybrid links. This metadata is collected by monitoring instance 106 and may include estimated connection bottleneck bandwidth, estimated minimum round-trip time, most recent actual round-trip time, current congestion window of the connection, and the reception / loss status of packets previously sent to the next level. Level 104 may also use this metadata to expand the metadata to track estimates of the bytes it has scheduled as in-flight on this connection, and further subdivide the total number into new bytes, retransmitted bytes, or dummy / probe bytes. All metadata collected and tracked by Level 104 is referred to as monitored characteristics (e.g., operational features).

[0108] The first level 104 then uses all available monitored characteristics to determine whether a packet should be sent to the next level, and if so, which connections should receive the packet. The set of packets provided to the connection monitoring instance may include duplicates of packets previously sent (to the same connection monitoring instance or to different connection monitoring instances). Alternatively, the scheduler may determine that a given connection monitoring instance is not currently eligible to receive packets, even if the instance's connections currently have capacity. Packet allocation to specific level 3 instances is discussed elsewhere in this document.

[0109] Level 2 106 includes a connection monitoring instance for each of the multiple available connections (e.g., connection monitoring instances 106A, 106B, and 106N associated with connections 110A, 110B, and 110N, respectively). The connection monitoring instances of Level 2 106 monitor the associated connections and determine or retrieve statistics known as operational characteristics associated with the performance (e.g., latency) of the connections. Level 2 106 may provide congestion control functions and packet buffering to ensure packets are available for the next level. Level 2 106 passes packets to Level 3 108, which includes output queues for each connection (e.g., output queues 108A, 108B, and 108N). Level 3 108 may write packets to the network interface according to a pacing or timing cues provided by Level 2 106.

[0110] In some embodiments, the second level 106 may implement one or more routers to a wireless interface (R2RI) protocol (e.g., PPPoE (RFC 5578), R2CP, DLEP (RFC 8175)) to better support the bandwidth allocation process for BoD connections.

[0111] Level 2 106 is responsible for receiving packets from Level 1 104, queuing the packets locally, and providing packet groups as bursts to the next pipeline level. Level 2 106 also tracks metadata of its associated connections based on acknowledgments received from peers or receivers.

[0112] The purpose of the queue in the second stage 106 is to ensure that there are always packets available for the next pipeline stage to be written to the network device for transmission. Some embodiments may include priority metadata as part of the information for each packet, allowing new packets arriving from the first stage 104 to be reordered in the queue to provide a preference for higher-priority packets.

[0113] To ensure that the second stage 106 has sufficient buffered packets to provide to the third stage 108, the second stage 106 may provide modified metadata to the first stage 104. For example, according to some example embodiments, the second stage 106 may choose to report a larger congestion window (i.e., “over-promote”), causing the first stage 104 to provide additional packets, which can be buffered in the second stage 106 for the third stage 108 to complete writing the actual congestion window packets. In other words, in some embodiments, the second stage 106 over-promotes the congestion window to account for processing overhead / latency caused by the entire pipeline 104 / 106 / 108, in order to ensure that the network is always populated with bytes in flight.

[0114] In some embodiments, compression techniques are known to be employed at the pipeline level or the network itself, which effectively increase the number of in-flight bytes the network can support. In this scenario, a 104-level "overspreading" congestion window is also employed to ensure the network is always populated with in-flight bytes.

[0115] Level 2 106 can also provide Level 1 104 with modified metadata to facilitate changes in capacity allocation within the BoD connection (i.e., to make Level 1 104 schedule packets on this connection as if it had higher capacity, thus triggering the BoD connection to allocate higher capacity). This modified metadata may include information that breaks down higher capacity into different requested types / priorities. For example, higher incremental capacity used to probe the BoD connection typically includes redundant / spoof packets because probe packets are likely to be lost by the BoD connection before the DAMA reassignment process is complete.

[0116] Promoting or over-promoting capacity from Level 2 106 to Level 1 104 does not necessarily need to immediately result in the detection of higher incremental capacity. Level 1 104 ultimately still makes the decision on how to allocate packets to connections. One embodiment may wait until all CWNDs on non-BoD connections are fully consumed (or nearly fully consumed) before instructing Level 2 106 to begin the DAMA reallocation process.

[0117] If a problem occurs with the underlying network connection, the queue in level 2 106 can quickly notify the previous level of the failure. Packets already provided to level 3 108 and written to the network must be assumed to be in transit, even if the network interface subsequently reports a failure; this means a timeout must occur before those packets can be marked as lost. However, once level 2 106 is notified that the connection is no longer available, packets still queued in level 2 106 can be immediately marked as lost, making them immediately eligible for priority retransmission on the alternative connection.

[0118] Before being passed to the third level 108, some embodiments may attempt to explicitly pace the rate of packet transmission by assigning a nominal timestamp to each packet, indicating when the next level should attempt to send the packet over the connection. Such timestamps can be determined based on estimated bottleneck bandwidth and packet size of the connection (e.g., the connection to which the packet is assigned) to prevent network congestion at or before the network bottleneck link.

[0119] Figure 2 An example of an embodiment in which the second level 106 supports priority packets is shown at position 200. Figure 2 In the diagram, the head of the queue appears on the right, and the tail of the queue appears on the left. Figure 2 In the top part (e.g., the first step), the first level 104 provides a list of new priority packets to the second level 106 connecting monitoring instances. Figure 2 The bottom part (e.g., step two) shows that these new packets have been added to the priority queue in priority order, such that all priority 1 packets will be dequeued before any priority 2 packets. Packets of the same priority are kept in sequential order.

[0120] In an example embodiment not shown, the second level 106 may have a single connection monitoring instance, which then provides packets to instances of the third level, each of which manages packet transmission on a set of network interfaces shared by a physical interface card. Other embodiments may have varying numbers of instances for each pipeline level.

[0121] Feedback, in the form of metadata or callbacks (e.g., a mechanism for sending metadata), can be sent from one level to the previous level, or a level can receive notifications from different parts of the system. Any of these actions can trigger (i.e., push) a level to send a packet to the next level.

[0122] The third level 108 receives packets from the second level 106 and writes the packets into the network 112 through one or more connection transport instances (e.g., connection transport instances 108A, 108B, and 108N associated with connections 110A, 110B, and 110N, respectively). In some embodiments, all second-level 106 connection monitoring instances provide packets to a single connection transport instance, or each connection monitoring instance may be associated with a connection transport instance.

[0123] Some embodiments of the third level 108 may use a queuing mechanism to allow the preceding level to provide hints about when packets should be sent. For example, the hint may be in the form of a "nominal send time" associated with the packet as metadata. If the nominal send time of the next packet to be sent is in the future, the third level 108 will wait until after that time before sending the packet. Urgent packets may be marked with an immediate timestamp, which may cause the urgent packet to be sent earlier than other queued packets. To prevent starvation of non-urgent packets, some embodiments may queue urgent packets after non-urgent packets whose nominal send time has passed.

[0124] As packets move through the pipeline from level 2 106 to level 3 108, some embodiments may include a callback function as part of the packet metadata. This callback can be used to trigger a notification on level 2 106 indicating that a packet has been transmitted to level 3 108. Such a notification can then be used to trigger the push of a new batch of packets from level 2 106 to level 3 108.

[0125] Figure 3 An embodiment of the process over time is shown at 300, wherein the second level 106 provides a "Sent at" prompt (nominal sending time) requesting that the packet not be sent before a specified time. The packet with a "Sent at" time of 0 should be sent as soon as possible. Figure 3 Examples of all these scenarios are shown (e.g., using past nominal sending times for critical packets, and how the system avoids starvation for non-critical packets).

[0126] In the state depicted at t=1, the packet queue of the third-level instance 108 contains 4 packets, three of which are currently eligible to be sent, and the second-level instance 106 is providing 4 new packets, the first of which has sequence 5 (sequence order 5) and should be sent immediately.

[0127] In the state depicted at t=2, the new packet has been added to the third-level 108 priority queue. Note that this embodiment has chosen to send the packet with sequence 5 after the packet with sequence 2, because sequence 2 was already eligible to be sent at the time sequence 5 was queued. Freezing the order of packets at the head of the packet queue in this way prevents starvation, i.e., the continuous flow of new urgent packets to be sent immediately prevents non-immediate packets from being sent, even if such non-immediate packets may have become eligible to be sent long before the most recent immediately sent urgent packet arrives.

[0128] In the state depicted at t=3, the third-level 108 has dequeued the first four packets from the output queue and provided them to the network interface. Before time t=4, there are currently no packets eligible to be sent; therefore, even if there is write capacity available on the network interface, the third-level 108 instance will need to wait to send more packets until time advances to t=4, or until new packets that should be sent immediately or at time t=3 arrive.

[0129] Packets can be passed from one level to the next in the optimal time order determined by each level.

[0130] Similar to Figure 1A The data packets are transmitted across the connection and are subsequently received by receiver 116 via connection 114. Receiver 116 includes sequencer 118 for assembling the transmitted data packets and can further transmit the assembled data packets via local area network (LAN) 120.

[0131] Compared to system 100A, push communication system 100B does not include an independent timer associated with each connection—instead, packets are scheduled to connections in a centralized location; it includes a push scheduler 104 that accesses monitored characteristics of each connection, and includes a push scheduler 104 that is activated in response to the occurrence of one or more monitored events rather than based on timers. For example, the one or more monitored communication events may include a receiver's acknowledgment / negative acknowledgment (ACK / NACK) of a previously transmitted data packet, or the arrival of a new data packet at input queue 102, etc.

[0132] In the example embodiment, the push scheduler 104 may potentially consume or dispatch the entire connection (congestion window) CWND instead of just transmitting 20 milliseconds of packets.

[0133] By accessing the monitored characteristics of each connection, the push scheduler 104 can be configured to optimize packet scheduling decisions. For example, data streams with specific transmission requirements can be preferentially allocated to connections that can meet the requirements associated with the data stream, rather than following a primarily timer-based wake-up order. In an illustrative example, the first data stream requires a transmission latency of less than 50 milliseconds.

[0134] The latency of connections 110A, 110B, and 110N is 5 milliseconds, 45 milliseconds, and 80 milliseconds, respectively. Using pull scheduler 130, if the 50Hz timer of connection 110B (45 millisecond latency) wakes up first, the timer will already allow packets of this first data stream to be scheduled to connection 110B. Using push scheduler 104, the data stream can be preferentially scheduled to connection 110A (5 millisecond latency) until the CWND of connection 110A is completely consumed, and then overflow to connection 110B (45 millisecond latency).

[0135] To enable bidirectional communication, a single instance of a multipath endpoint can include Figure 1B All components described herein. Specifically, the instances may include one or more instances of sender 100 to support upstream communication to other multipath endpoints, and one or more instances of receiver 116 to support downstream communication from other multipath endpoints.

[0136] For example, according to some embodiments, one or more of the available connections are BoD connections, and push scheduler 104 may cause more data to be allocated to BoD connections based on criteria such as the CWND of connection 110A being fully consumed (or nearly fully consumed) in order to request more capacity from the network; while more packets eligible to be transmitted on connection 110B (e.g., based on per-stream rules) are available at input queue 102.

[0137] In the example embodiment, the push communication system 100B enables the application of more accurate / fair Quality of Service (QoS) and Active Queue Management (AQM) packet loss rules at the input queue 102. Referring again to the previous example, a flow with a latency requirement of 50 milliseconds will never be able to utilize an 80 millisecond connection. Because the push scheduler 104 has a consistent snapshot of the entire system, it can calculate a fair share of the partial usage and total capacity of flows excluding 80 millisecond connections.

[0138] One potential advantage of the push-based scheduling approach is that the complete set of metadata for multiple connections containing mixed links can be used by the push scheduler 104 to optimally distribute packets to target connections.

[0139] Figure 1CThis is a flowchart of a method 110C for transmitting data packets using system 100B or other push communication systems, according to some example embodiments.

[0140] At box 1130, method 100C senses when the input queue includes unassigned data packets that can be used for transmission.

[0141] At box 1132, method 100C includes assigning one or more unassigned packets to one or more corresponding output queues in a plurality of networks having suitable monitored characteristics.

[0142] At box 1134, method 100C includes updating the monitored characteristics of the corresponding plurality of networks to reflect one or more assigned data packets.

[0143] At box 1136, method 100C transmits the assigned data packets from the output queue corresponding to each of the multiple networks to the destination device.

[0144] In the example embodiment, each time an external event triggers the push scheduler 104, the method for transmitting data packets using system 100B includes a loop of the following steps:

[0145] 1. Request a data packet from input queue 102;

[0146] 2. Distribute the received packets to the connection for transmission; and

[0147] 3. Repeat steps 1 to 2 until either: a) there are no more available packets at input queue 102; or b) there is no more available transmission capacity on any active connection (e.g., as defined by the congestion window (CWND) metric).

[0148] The method can be implemented such that the push scheduler 104 does not consider any data stream-specific requirements. In such embodiments, system 100B allocates received packets based on the order of connections.

[0149] The ordering can be based on monitored characteristics of the connection, which may include, for example, a subset of operational characteristics estimated by a congestion control algorithm. In one embodiment, the BBRv1 algorithm is used, but other embodiments may include instances (such as BBRv2, TCP Reno, TCP LEDBAT), proprietary implementations, or solutions for specific connection types such as BoD. In the subsequent discussion of the operational characteristics of BBRv1, a pipeline analogy is referenced, where two endpoints are modeled as a series of interconnected pipelines, each pipeline segment in which may have a different diameter and length; this is for illustrative purposes only.

[0150] Operational characteristics can be a measure of the shortest time required for a transmitted packet to reach its destination endpoint (e.g., the receiver) and be acknowledged (e.g., via an ACK message from the receiver to the sender). This duration is called the "round-trip propagation delay" (abbreviated as "RtProp") and is measured in units of time (e.g., milliseconds). In pipeline analogy, this value represents the total length of all pipeline segments from the sender to the receiver and back.

[0151] Operational characteristics can be defined as the maximum rate at which data can move between the endpoints of a pipeline segment with the smallest diameter. The smallest pipeline diameter, referred to as the "bottleneck" segment, determines the maximum rate at which data can move between the sender and receiver; this bottleneck segment is also known as the "bottleneck bandwidth" (abbreviated as "BtlBw"). For example, methods for estimating BtlBw are described in an IETF draft: https: / / tools.ietf.org / html / draft-cheng-iccrg-delivery-rate-estimation-00.

[0152] Table 1 shows instances of WAN connection sets and their characteristics, an example subset of operational characteristics selected by scheduler 104 to become monitored, and the resulting sorting order:

[0153] Table 1: Example WAN Connections and Their Sorting Order

[0154]

[0155] This example method sorts the connections according to the following dimensions, moving to the next dimension only if the connections are equal in the current dimension:

[0156] 1. Increase the priority level of preferences;

[0157] 2. Improve RtProp;

[0158] 3. Reduce BtlBw; and

[0159] 4. Add an interface name.

[0160] Please note how the sorting operation maintains the relative stability of the join order, since all sorting dimensions are parameters that are not expected to change frequently.

[0161] This stable sorting order is intentional, so that packets arriving at the input queue will tend to "stick" to the same set of connections (the connections sorted at the top of the list), and only "overflow" to lower-sorted connections when the input demand exceeds the total CWND of the higher-sorted connections.

[0162] Note how the sorting criteria in this example are not specific to any particular flow. For example, an administrator or end user might only want to transmit high-priority application flows (the instance of transmission requirement) via metered cell1 and cell0 connections. This means those flows would prefer a sorting order that places cell1 and cell0 lower in the list. Similarly, there might be flows with maximum latency limits, such as VoIP calls, where end users consider them unusable for interactive conversations if the latency exceeds 150 milliseconds. For these flows, a sorting order that uses RtProp as the primary sorting dimension might be desirable. Subsequent paragraphs will discuss examples of extending the sorting criteria to specific flows.

[0163] The scheduler is also responsible for determining the packet retransmission strategy. In an example embodiment, system 100B is configured such that if a packet is reported as missed by the receiver, the scheduler always marks the packet for retransmission. This is desirable for many flows, but other flows may not want this at all (because additional delays or packets may arrive out of order, making the packets unusable for the application), or may only want to attempt retransmission within a certain time period.

[0164] Figure 1D This is a flowchart of another method for transmitting data packets according to some example embodiments.

[0165] An example of a simple scheduling loop is as follows:

[0166] 1. Take any necessary considerations for connection metadata updates and determine the active connection set.

[0167] 2. Determine preferred connection rankings based on connection metadata (monitored characteristics, such as monitored operational characteristics).

[0168] 3. When there is a packet to be sent and there is available network capacity in the active connection group:

[0169] a. Get the next packet to be sent.

[0170] b. Queue the packets to be sent on the first connection with available capacity in the sorting.

[0171] c. Update connection metadata (monitored characteristics) to reflect scheduled packets.

[0172] d. Repetition.

[0173] 4. For each connection with queued packets, send those packets to the next pipeline stage of that connection.

[0174] For connection preference ordering purposes, such embodiments treat all packets as equivalent. In this case, packets can be considered to belong to the same flow category (even though different packets may logically belong to different flows, from the scheduler's perspective, for connection allocation purposes, there is no distinction between these flows).

[0175] Figure 1E This is a flowchart of another method for transmitting data packets according to some example embodiments.

[0176] An embodiment of the scheduler capable of distinguishing different stream categories (i.e., ICMP echo requests / responses, relative to video streams, relative to file transfers), and even distinguishing individual streams within the same category (video streams between hosts A and B, relative to video streams between hosts A and C), can be provided as follows.

[0177] 1. At box 1102, make any necessary considerations for connection metadata updates (monitored features) and determine the active connection set.

[0178] 2. When the active join set has capacity (box 1114):

[0179] a. At box 1103, construct a selector based on the current active join set.

[0180] b. At box 1105, use the selector to select the packet to be sent through one of the available connections. If no packet is selected, exit the loop.

[0181] c. At box 1107, the order of active connections is determined based on packet flow and flow classification, combined with connection metadata (monitored characteristics).

[0182] d. At box 1110, queue the packets to be sent on the first connection with available capacity in the sorting.

[0183] e. At box 1112, update the connection metadata to reflect the scheduled packets.

[0184] f. Repeat.

[0185] 3. At box 1116, for each connection with queued packets, send those packets to the next pipeline stage of the connection.

[0186] These two sampling methods (shown in) Figure 1D and 1E The method is similar to that shown in [the text]. Figure 1D The method in Figure 1E A simplified and optimized version of the more general method in [the text].

[0187] The first method can be transformed into the second method by defining a single flow category and a selector that matches all packets. Therefore, the second method will be discussed in more detail. Figure 1E And the same discussion will apply to the first method ( Figure 1D However, it is subject to the limitations mentioned above.

[0188] The first step is to update any pending metadata considerations (monitored characteristics) to determine the active connection set. In one embodiment, the active connection set is the set of connections that have not been subjected to unreliable connection backhaul (i.e., using limited connections due to recent failures in reliable packet delivery).

[0189] Other embodiments may further restrict the set of active connections by, for example, eliminating connections whose metadata (monitored characteristics) do not match any requirements of the currently defined flow class (i.e., assuming all packets must match a defined flow class, then a connection can be considered non-active if it will never be selected by any flow class; for example, if all flow classes specify a maximum RTT of 500 milliseconds, then a connection with an RTT > 500 milliseconds can be considered non-active).

[0190] If at least one connection in the active connection set has an available CWND (Congestion Window), meaning that the number of in-flight bytes currently tracked by the scheduler for this connection does not exceed the CWND already determined for this connection, then the set has the capacity to send packets. In one embodiment, the CWND of a connection can be the CWND reported by the next pipeline stage, or the scheduler can choose to reduce the reported CWND if throttling or priority routing is effective.

[0191] In one embodiment, the CWND of a connection throttled to a certain rate decreases by the same factor as the throttling rate on the bottleneck bandwidth (or if the throttling rate is higher than the bandwidth estimate, the CWND remains unchanged). Similarly, if a connection undergoes priority routing (where embodiments support grouping connections into priority levels, which are activated when the total actual throughput of all connections in a higher priority level (excluding overhead throughput such as protocol headers, control packets, and retransmissions) drops below a configured threshold for the level to be activated), the CWND of that connection may decrease. Various methods for determining CWND reduction in these scenarios will be referenced subsequently. Figures 8A-8C It is described in conjunction with 9A-9C.

[0192] The effective CWND for connections assigned to non-active priority levels (the total actual throughput of all connections in higher priority levels plus the unused bit rate reaching or exceeding the activation bit rate of the priority level) is zero; and the CWND for connections in non-first-level priority levels is set to zero if the estimated total actual throughput bit rate of all connections in the current active priority level set reaches or exceeds the activation threshold of the lowest active priority.

[0193] In other embodiments using priority routing, connections in non-primary tiers can be capped at rates below their bottleneck bandwidth, so that their contribution is only sufficient to narrow the gap with the configured activation threshold. This can be done, for example, to save costs when non-primary tier connections are more expensive connections intended for use only during emergency situations. Similar to throttling, CWND is reduced to reflect the upper limit rate at which these connections should be sent.

[0194] The loop in step 2 of the method can be executed as long as the active set of connections has capacity.

[0195] Step 2a (shown in) Figure 1E Box 1103 in the diagram constructs a selector for finding packets eligible to be sent on one of the active connections with active capacity. In some embodiments, the selector consists of a set of flow categories compatible with the current set of active connections with capacity.

[0196] For example, if a flow class requires a maximum connection latency of 500 milliseconds, and all active connections with available capacity have latency exceeding 500 milliseconds, then that flow class will be excluded from the selector. If an embodiment can determine that no connection currently meets the criteria for such packets (active or non-active, with or without available capacity), some embodiments may choose to include flow classes in the selector that would typically not match the criteria for active connections with capacity. In this case, the embodiment may choose to violate the matching criteria to send the packet, rather than holding the packet until it times out and is lost. Other embodiments may choose to enforce a strict matching policy in which packets matching a flow class with an ineligible connection are never sent.

[0197] In another embodiment, shown in Figure 1F The example sorting order in the process flow can be determined or established computationally using alternative methods in order to control the routing of data packets to various network connections.

[0198] Before determining the sort order, some or all of the input parameters can be rounded / bucketed. The technical improvement of rounding and bucketing is to keep the sort order as "stable" as possible, so that the sort order does not change constantly (for example, without rounding and bucketing, two similar values ​​may cause the sort order to change constantly due to the fluctuation of their values ​​over time, resulting in "jitter").

[0199] Factors influencing its decision-making include: maintaining the sorting order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (splitting increases the likelihood of spontaneous out-of-order and jitter events); and considering both measured characteristics of the connection and unmeasured external commands.

[0200] Values ​​rounded to the nearest whole number can include the following:

[0201] Latency-related values ​​(RtProp, RTT) are rounded to the nearest multiple of 50 milliseconds, with a lower limit of 1 millisecond after rounding.

[0202] • The throughput-related value (BtlBw) is rounded to the "closest order of magnitude".

[0203] • The packet loss rate (the worst value in the 3 sliding windows) is rounded to the nearest order of magnitude. Note that the original values ​​here are limited to the range [0,100], so the output values ​​are limited to the set (0,1,2,3,4,5,6,7,8,9,10,20,30,40,50,60,70,80,90,100).

[0204] The table below provides examples of rounding to the nearest order of magnitude. It shows the original value, its calculated order of magnitude, and then the resulting rounded value.

[0205] Original value (range) Order of magnitude value after rounding 0 1 0 1 1 1 2 1 2 10-14 10 10 15-24 10 20 85000-94999 10000 90000 95000-99999 10000 100000 100000-149999 100000 100000 150000-200000 100000 200000

[0206] In some embodiments, specific improvements are described for controlling network interface usage based on improved network routing order, wherein throughput is predicted based on the Mathis Equation using the combined characteristics of round-trip time (RTT) and packet loss rate to establish an upper limit on TCP throughput for a given RTT, packet loss rate, and MSS. The Mathis factor can be used to modify the order in which network packets are routed.

[0207] The measured characteristics may include: packet loss rate over three independent sliding windows (500 milliseconds, 3 seconds, and 30 seconds), which is further summarized as the worst value over the three windows; current network round-trip time (RTT); bandwidth round-trip propagation time (BBRv1) parameters: RtProp (network propagation delay); and BtlBw (throughput estimate). External instructions may include priority routing (PR) rules, but may also include flow-specific rules / requirements.

[0208] In some embodiments, the measured characteristics of BBRv1 differ from its design in the original BBRv1 IETF draft. For example, considering embodiments running on systems whose execution times may be inconsistent, the size of the sliding window used to compute BtlBw can be increased to cover long durations (hiding variability caused by execution inconsistencies), but the window duration can also be shortened immediately if a network loss or increased latency event occurs, as those events indicate that BtlBw may have actually changed, and any long-term history within the sliding window must be forgotten.

[0209] For similar reasons of inconsistency, the input parameters used to measure RtProp can be changed to include a portion exceeding network propagation delay, thus adding a component of execution / processing time, as this time may be non-negligible and continuously contributes to actual latency while transmitting packets and waiting for responses. In some embodiments, this is referred to as "system RtProp" rather than "network RtProp," which consists solely of network propagation delay. In these embodiments, "system RtProp" and "network RtProp" can be used for different calculations, depending on the intended purpose. For example, "system RtProp" is suitable when calculating CWND, ensuring the pipeline contains sufficient in-flight data to account for processing latency. However, "network RtProp" is more suitable when sorting network connections to determine packet routes, since preferred routing decisions do not depend on system execution latency.

[0210] Packet loss rate, latency, and throughput are not entirely independent variables. They interact and combine depending on the application's congestion control behavior. This is because, for example, with TCP, a higher packet loss rate leads to a longer air "silent period" (underutilized channel) when the sender waits for duplicate acknowledgments (DUP ACK) or retransmission timeouts (RTO).

[0211] Therefore, some comprehensive characteristics were determined and used in the final sorting method. Specifically:

[0212] Mathis Factor: This composite characteristic takes into account RTT and packet loss rate to predict throughput and is based on the Mathis Relation, which gives an upper limit to TCP throughput given RTT, packet loss rate, and MSS. When comparing the characteristics of any two network connections to determine their ranking, MSS can be considered a constant; therefore, this relationship indicates that the expected throughput is proportional to the reciprocal of (RTT * sqrt(Packet Loss)). In this example, this relationship is used as a practical implementation to establish a factor for directly controlling the operation of a network controller or router. For each connection, the system determines the Mathis Factor as: maximum value (Rounded Rtt, 1 ms) * sqrt(Rounded Summarized Packet Loss). Connections with smaller Mathis Factor values ​​are considered better. The calculation is a function of two independently measured characteristics, therefore the relationship between these characteristics determines how connections will be compared to each other. The Mathis coefficients can be maintained and stored in data objects, such as the corresponding Mathis coefficient data values ​​associated with each connection, and stored together with other monitored characteristics in, for example, an array of data values.

[0213] For example, in an extreme case, two connections with a 0% packet loss rate will always have the same Massis coefficient—their actual RTT value is irrelevant. In this scenario, the Massis coefficient does not affect the sorting order—which will be determined by comparing subsequently measured characteristics and combined characteristics.

[0214] In another extreme case, if two connections have a packet loss rate of 100% (rounded summed packet loss rate = 1.0), then only their rounded Rtt values ​​will determine how they are compared to each other. In some embodiments, this is undesirable, and therefore connections with a packet loss rate of 100% are filtered out from the available options at an earlier stage.

[0215] Between the two extremes, connections with both low RTT and low packet loss rate are better than those with higher values. However, connections with a high percentage of packet loss can still be compensated for by having a low RTT and will have the same Massis coefficient as connections with a low percentage of packet loss. For example:

[0216] Connection A: 100 ms RTT, 0.01 (1%) loss rate => Mathis coefficient = 10

[0217] Connection B: 19 ms RTT, 0.25 (25%) dropout rate => Mathis coefficient = 9.5

[0218] In this example, although connection B has a significantly higher loss rate (25% vs. 1%), its lower RTT compensates for this, resulting in a better Mathis coefficient compared to connection A. This phenomenon can be explained by the fact that, despite connection A's lower percentage loss rate, connection B's lower RTT allows it to exchange packet acknowledgments (positive and negative) for lost packets and completes any necessary packet retransmissions much faster than connection A could with its higher RTT.

[0219] Capacity Factor: This comprehensive characteristic takes into account BtlBw and RtProp and is determined as: Rounded BtlBw / Maximum RtProp (Rounded RtProp), 1 millisecond. The unit is "bytes per millisecond," which has no physical meaning but provides a way to normalize the bandwidth achievable per unit of RtProp for a connection. In other words, a connection that achieves more bandwidth per millisecond of RtProp is considered more efficient than one that achieves less bandwidth. In this context, efficient transmission means a connection that achieves more bandwidth per millisecond of propagation delay (RtProp).

[0220] Once all the original inputs have been processed and the comprehensive properties have been calculated as described above, the actual sorting mechanism can be executed sequentially as follows:

[0221] Moving from one comparison to the next only occurs if all the first few comparisons are equal. If the comparisons are not equal, the remaining comparisons are irrelevant and skipped. In some embodiments, the selection of comparisons is based on flow classification (e.g., VPN, relative to VoiP, relative to ping packets, relative to FEC packets), and the list of comparison criteria can be different for each different flow (e.g., each flow can have different predefined most important, second most important, third most important, etc.). Each of these characteristics can be iterated sequentially from the most important until the connections themselves can be sorted and placed in a sequential sequence. In some embodiments, depending on the flow classification, the first preparatory step is to select a subset of connections suitable for the packet types in the sending flow classification, and then assign them an order before performing packet allocation.

[0222] 1. Connections that are completely paralyzed (with a totaled loss rate of 100%) are considered worse than connections with a totaled loss rate of <100%.

[0223] 2. PR Priority (the higher the priority, the better)

[0224] 3. Massis coefficient (the lower the better)

[0225] 4. RTT after rounding (lower is better)

[0226] 5. Capacity Factor (the higher the better)

[0227] 6. RtProp, rounded down (lower is better)

[0228] 7. BtlBw (rounded to the nearest whole number) (the higher the better)

[0229] 8. Connection ID (lower is better)

[0230] Figure 1F The image shows a rendering of the sorting mechanism. In one example sorting mechanism, the real-time stream first includes a comparison of RTT / RtProp, and then if they are equal, it will then compare reliability, and then if that is also equal, it will compare transmission rate, etc. Depending on the stream type, there may be different sorting orders. For example, a stream that requires throughput might first compare transmission rates. If they are equal, it will then compare reliability. If that is also equal, the stream might decide not to do any further evaluation and then simply choose arbitrarily.

[0231] In some embodiments, the streaming requirement (e.g., real-time, relative to throughput) can specify the order and number of comparisons. For example, a real-time streaming stream might prefer to compare only rounded RTT and rounded RtProp, ignoring all other characteristics. A throughput streaming stream might decide to compare only the Massis coefficient and the capacity coefficient.

[0232] The methods described above involve explicit sorting dimensions. There are also implicit sorting dimensions, which often appear as a byproduct of typical application traffic patterns.

[0233] Most application traffic tends to be bursty or adaptive, so it doesn't need to utilize all WAN connections. This means that a connection currently at the bottom of the sort order may not be carrying any traffic. If a connection doesn't do this, its explicit sorting dimension won't be updated, so the connection's position tends to remain near the bottom of the sort order.

[0234] In some embodiments, push scheduling can also be configured to implicitly generate a sorting order along one or more implicit sorting dimensions over a time period. Implicit sorting dimensions force connections that have experienced periodic adverse events (e.g., latency spikes caused by congestion) to bubble up to the bottom of the list. Connections behaving consistently bubble up naturally and remain near the top.

[0235] This is generally considered a good characteristic of the algorithm, but it may be modified in the future due to external factors. For example, if the flow has specific requirements that cannot be satisfied by connections near the top of the current sort order, it may make sense to generate probe flow on connections near the bottom to refresh its explicit sort dimension.

[0236] Figure 4A An example of generating a selector using predefined flow categories and current connection states is depicted at 400A.

[0237] In this example, there are three connections, each with a monitored set of features, and three predefined flow categories, each defining a matching criterion and a preference for ranking connections that meet the matching criterion.

[0238] In this embodiment, the selector consists of a set of flow categories whose matching criteria match at least one connection with available capacity. In this example, both flow category 1 and flow category 2 have matching criteria that match connections 1 and 2, while flow category 3 does not match any connection and is not included in the selector.

[0239] Although both connection 1 and connection 2 match, connection 2 is excluded from the selector because it has no available congestion window and cannot currently send packets. When constructing the selector, a flow category can be included as long as a matching connection with an available congestion window can be found; therefore, one embodiment may only check connection 1 to determine if flow category 1 and flow category 2 can be included in the selector, without needing to check if connection 2 matches.

[0240] Step 2b (shown in) Figure 1E In box 1105, the selector from step 2a is used to search the retransmission queue or the send queue for packets eligible to be (re)transmitted on an active connection with capacity. If no such packet is found, the loop exits. This can happen even if some active connections have capacity and packets are present in the input queue or the retransmission queue.

[0241] Figure 4B An embodiment with a priority input queue is shown at 400B, which is an ordered set of streams, wherein the stream on the right side of the queue should send packets before the stream on the left side. Each stream has a stream ID, and each stream belongs to one of a predefined stream category. Each stream also contains an ordered queue of packets such that if a stream is selected, the packet at the head of the queue is removed.

[0242] In this example, Figure 4AThe selector in the algorithm evaluates the streams in the input queue from right to left, and selects the first stream with the stream category listed in the selector. In this case, streams with IDs 5 and 2 are evaluated and rejected because they both have stream category 3, which is not one of the stream categories in the selector. Then, stream 7 is evaluated and is a match because it belongs to stream category 1, which is one of the stream categories in the selector. A packet is then extracted from the head of the packet queue of stream 7 and stamped with stream metadata.

[0243] Once the package has been selected, step 2c (shown in...) Figure 1E Box 1107 in step 2 constructs the ordering of active connections. The selected ordering will be based on rules provided in the definition of the flow class (if the embodiment supports said rules), or on a generally acceptable ordering in other ways. Some embodiments may use the same ordering for all flows matching the flow class, and some embodiments may make the ordering flow-dependent (e.g., making all packets in a flow sticky to a given connection, without requiring all flows to be sticky to the same connection). The ordering may be constructed as needed at each iteration of the loop in step 2, or some embodiments may choose to cache the ordering and clear the cache when the criteria used to construct the ordering have changed (i.e., when metadata is updated).

[0244] One embodiment builds the ordering without considering whether the connection currently has capacity, because capacity may drop to zero due to packet scheduling on the connection, or increase above the current zero value if scheduling packets on different connections sufficiently alters the metadata mix to activate a set of connections in a previously unused priority level. Building the ordering in this way allows it to be cached and reused if the only changing items are the connection's valid CWND and the byte count in flight.

[0245] When packets of a flow traverse the same WAN connection at all possible times, the flow traversing System 100B tends to perform better. Therefore, embodiments should use stable sorting criteria to construct the sorting, ensuring that the sorting created for the same flow is consistent unless the monitored characteristics of one or more connections in the connection change significantly. Thus, operational characteristics such as RTT and RtProp, as well as bandwidth, are poor choices because they can fluctuate over short periods, making the sorting order unstable.

[0246] Statistical filtering methods can be used to transform raw, "noisy" operating characteristics into monitored characteristics. These methods can range from traditional measures such as mean, variance, and ANOVA to (quantitatively quantified) ranking statistics (ranking filtering). For example, the monitored characteristic can be a range of raw operating characteristics, defined by specific values ​​(e.g., 1-100 milliseconds) or orders of magnitude. Subsequent paragraphs provide specific examples, including those shown in Table 2.

[0247] Some implementations can use machine learning techniques to translate operational characteristics into monitored characteristics. For example, a trained model can be developed to predict impending connection failures, and thereby construct a sort that ranks matching connections to the end of a list. Another example model can be used to detect connection types (e.g., wired, cellular, WiFi, satellite, BoD, etc.) and use said connection types to form the basis for new monitored statistics (e.g., susceptibility to buffer bloat) that cannot be easily measured in real time.

[0248] One implementation uses the following sorting for all packets (the later criterion is used only if all previous criteria are equal):

[0249] "Bucketed" loss probabilities. Connections can be grouped into buckets defined by a range of loss probabilities based on packet count or size (e.g., in bytes). Connections in buckets with smaller loss probability ranges appear first in the sorting, and connections in buckets with larger loss probability ranges appear later in the sorting.

[0250] Connection priority groups (e.g., provided by administrators or end users). Higher priority connections are ranked higher (Level 1 > Level 2 > Level 3, etc.).

[0251] "Bucketed" RTT. Latency is rounded to the nearest multiple of 50 milliseconds. Bucketing latency values ​​allows normal variations in latency to be hidden during sorting, while still allowing significant variations in latency to be reflected in the sorting.

[0252] As described above, for example, the "capacity factor" is a total value measured in "bytes per second per millisecond," calculated by rounding the bottleneck bandwidth estimate to the nearest order of magnitude or an integer multiple thereof (e.g., 1 rounds to 1, 10 and 11 round to 10, 15 rounds to 20, 86000 rounds to 90000), and then dividing by the RtProp that is bucketed to the nearest 50 milliseconds. Connections achieving higher bandwidth per millisecond of RtProp are considered better than connections with lower bandwidth per millisecond of RtProp.

[0253] The capacity factor, determined based on the original BtlBW and RtProp values, is highly variable. However, unless large changes are observed, the value is stabilized over a period of time by rounding BtlBW to the nearest order of magnitude or an integer multiple thereof and bucketing RtProp. Table 2 shows several examples of bucketing, rounding, and capacity factor calculation. Note that in this embodiment, the capacity factor is rounded to the nearest integer, and for the purposes of capacity factor calculation, a bucketed RtProp value of 0 is treated as 1.

[0254] "Bottled" RtProp. Compared to higher values, the preference is for lower RtProp values ​​that are binned to the closest 50 milliseconds.

[0255] "Rounded BtlBW". Compared to lower values, a higher BtlBW value that is rounded to the nearest order of magnitude or an integer multiple of the nearest order of magnitude is preferred.

[0256] If all the aforementioned criteria are equal, the tie is deterministically broken using the connection ID (assigned during connection enumeration). Other embodiments may use more complex tie-breaking mechanisms, such as hashing the stream's properties as an integer modulo the number of tie-breaking connections.

[0257] Table 2: Examples of binning, order of magnitude rounding, and capacity factor

[0258]

[0259] Implementations that differentiate between stream types may, for example, modify the sorting described above to, based on the transmission requirements of the stream type, favor high-capacity connections over low-latency connections for file transfer streams, and favor low-latency connections over high-bandwidth connections for video streams.

[0260] Implementations that enforce strict flow fixing on connections (or sets of connections) may allow only exact matches on the connection ID (potentially excluding other connections from the ordering altogether). Such fixing rules are typically provided by an administrator or end user and are designed to overturn all ordering decisions determined by monitored characteristics. Therefore, even if monitored characteristics indicate that a connection is unavailable or has failed, the flow will continue to use the same ordering and will therefore no longer connect through the multipath system 100B.

[0261] Some embodiments may create monitored characteristics that are composites of other monitored characteristics or operational characteristics, and then use these composites to determine ranking. For example, a composite reliability metric may be created that combines bucketed loss probability, (in-run) latency (RtProp) variance, and (in-run) throughput into a weighted metric representing the overall reliability of a connection. Connections ranked higher on this metric will be reserved for flows that require high reliability (which may not necessarily mean high speed).

[0262] Generally, if the generated selector allows the selection of packets, then the packets should generate an order that allows the packets to be scheduled on a certain connection used to generate the selector.

[0263] Figure 4C An example of how to construct connection ordering using packet flow categories (“flow categories”) is shown at 400C. In the depicted embodiment, the flow category in the packet's metadata is used to look up the flow category definition. The matching criteria for the flow category are used to select which connection should be included in the constructed ordering (alternatively stated as the flow category defining the set of transport requirements that can be associated with a particular flow category), in which case both connection 1 and connection 2 match, while connection 3 does not. Note that connection 2 is included even though it does not have an available congestion window (i.e., it is not currently eligible to send packets). This facilitates buffered ordering and allows the ordering to be reused even if the state of the available congestion window should change (e.g., if the scheduler initially blocks data from being sent on a connection because it has a lower priority, but then decides to re-enable the connection because the combination of data on currently enabled connections does not meet the minimum system throughput).

[0264] Once the connection set has been selected, the matching connections are sorted into a sorted list using a sorting criterion based on the flow class (e.g., a subset of transport requirements associated with the flow class). In this example, the first criterion is capacity factor. Both connections have the same capacity factor. The second criterion is RTT, and both connections have the same RTT. The third criterion is BtlBW (bottleneck bandwidth), and connection 2 has a higher value, therefore appearing before connection 1 in the sorting.

[0265] The sorting criteria set within a stream category (e.g., transmission requirements) allows packets to be scheduled onto the most available connection that is best suited to the type of stream they belong to. For example, packets belonging to the "video encoder" stream category may preferentially use connections with the lowest latency, while packets belonging to the "file transfer" stream may prefer high-throughput connections, even if the latency is high or highly variable.

[0266] Once the order of the connections has been determined, step 2d (shown in...) Figure 1EBox 1110 in the diagram uses sorting to allocate packets to connections with available bandwidth. Packets are placed on the connection's send queue.

[0267] Figure 4D It is shown at 400D. Figure 4C The sorting is used to select connections for a given packet. In this example, the first connection in the sort is connection 2, but it does not have an available congestion window, so it is not currently eligible to send packets. Connection 1 is then checked, and it is able to send packets, so it is selected, and packets are queued to be sent to the 3-stage pipeline of connection 1.

[0268] Then, as part of step 2e (shown in) Figure 1E In box 1112), the connection metadata (monitored characteristics) of the bytes in flight is adjusted. In some embodiments, this includes distinguishing actual throughput bytes from retransmission / dummy / probe bytes when tracking the CWND consumption of a connection. Such adjustments may affect the actual throughput and / or potential actual throughput of a priority group, which may cause connections in the next priority level to be activated in the next iteration of the loop in step 2. Various methods for converting from volume units (CWND bytes) to rate (actual throughput) are subsequently discussed in the section on... Figures 8A-8C This will be discussed in the descriptions of 9A-9C.

[0269] Once step 3 has been reached, the available CWNDs for each connection have either been filled or there are no longer any packets eligible for transmission. The packets queued to the output queue of each connection in step 2 are then sent to the appropriate instance in the next stage of the pipeline.

[0270] In some embodiments, when there are multiple flows with the same priority but belonging to a flow category with conflicting requirements, an explicit objective function can be used to determine the desired trade-off.

[0271] Described in Figures 4A-4D The example illustrates the objective function for ensuring fairness among flows with the same priority. The selector serves each flow that matches its flow category in a round-robin fashion.

[0272] The fair implementation in this example uses the differential round-robin scheduling algorithm described in RFC 8290. In brief, this algorithm utilizes a "byte integral" scheme, allocating a certain number of byte integrals to each queue in each iteration of its iterations over the queues.

[0273] In each round, the number of bytes retrieved from each queue for transmission is nominally capped at its available integral. Fairness naturally occurs when the byte integral quantum is equal for each queue.

[0274] In some embodiments, the objective function may allow administrators or end users to configure intentional unfairness. For example, weights can be configured on the matching stream, which could ultimately result in unequal byte integral quantum values ​​being given to the queue on each iteration.

[0275] For example, there might be 5 different weight levels:

[0276] • Superlative level (5)

[0277] Advanced (4)

[0278] Intermediate (3)

[0279] • Low level (2)

[0280] • Lowest level (1)

[0281] The value assigned to each weight will represent the ratio between its byte integral quanta. For example, if the quantum corresponding to "medium" is 30KB, then the stream labeled "low" will receive (2 / 3)*30KB = 20KB of quantum in each round. The stream labeled "highest" will receive (5 / 3)*30KB = 50KB.

[0282] Please note that the absolute numbers in this example are not important. If the intermediate quantum were 5KB instead of 30KB, the weights would still appropriately adjust the relative quantumes of other priority levels, and the final result would be the same in terms of overall fairness.

[0283] Other implementations may allow weights to be any integer, instead of the five fixed values ​​in the previous example. This would allow administrators or end users to configure even higher levels of unfairness between streams (if needed) in the objective function.

[0284] Also note that, as with all Quality of Service (QoS) settings, these weightings are only important when there is contention for available WAN capacity. If there is no contention, it means that the data in all queues is perfectly within the available capacity, so the ratio between their usage is a natural ratio for applications generating different data volumes.

[0285] Other embodiments may have entirely different objective functions that do not consider fairness at all. For example:

[0286] • Ensure that the maximum number of flows is met with the available WAN capacity;

[0287] • Maximize the total throughput achieved by serving the streams; or

[0288] • Minimize changes to the current bandwidth allocation on the BoD connection.

[0289] In some embodiments, the way a WAN connection is used can alter its characteristics, making it ineligible to serve certain streaming categories. For example, some connection types can achieve very high throughput, but at the cost of higher RTT (Round-Trip Time).

[0290] An implementation of the objective function that aims to maximize throughput will select services for flows with high throughput requirements, which will lead to an increase in WAN connection RTT and disqualify it from serving flow categories with low latency requirements.

[0291] Conversely, implementations that aim to serve the maximum number of flows may choose the opposite trade-off, allowing the WAN connection to continue serving low-latency flow categories.

[0292] In some embodiments, the different scheduler connection ordering is maintained on a per-flow basis, rather than globally for all flows.

[0293] This allows stream-specific requirements to influence the sorting order, so that packets of the stream are allocated to the connection in a way that meets its needs.

[0294] For example, a VoIP stream with a preferred maximum latency tolerance of 150 milliseconds would need to prioritize connections with an RTT exceeding 150 milliseconds in its sorting criteria. Conversely, a TCP stream without specific latency requirements would need to prioritize connection throughput in its sorting criteria, thus ignoring RTT entirely or using it only as one of many secondary criteria.

[0295] In some embodiments, these flow requirements and preferences are configured in the form of rules, which consist of matching criteria and corresponding behaviors.

[0296] For example, matching criteria can take the form of IP 3-tuples (protocol, source IP, and destination IP) or IP 5-tuples (protocol, source IP, source port, destination IP, and destination port). Behavior can take the form of explicit preferences for latency or throughput, each with a target preference value.

[0297] Other examples of heuristic-based matching criteria may include:

[0298] οDSCP tag

[0299] any other header fields available in the IP (v4 or v6) header.

[0300] any field in other header fields of the TCP or UDP header.

[0301] The cumulative volume of data transmitted—for example, TCP streams exceeding a volume threshold can be matched as “bulk data transfers,” while TCP streams within the volume threshold can be matched as interactive SSH / telnet sessions.

[0302] cumulative duration of the flow

[0303] Regular expression matching for packet payload

[0304] plaintext parameters extracted from the TLS handshake (e.g., SNI, CN, SubjectAltName)

[0305] Other instances that may affect the scheduler's sorting order include:

[0306] ο Specific preferences regarding connection order (e.g., a user might prefer to sort expensive connections to the bottom of a list of low-priority flows).

[0307] jitter tolerance

[0308] Maximum delay tolerance

[0309] ο Expected redundancy (e.g., FEC)

[0310] Is it expected that lost packets will be retransmitted, and if so, is there a maximum time limit after which retransmission attempts should stop?

[0311] Do you expect explicit package pace?

[0312] i. For example, a real-time video application that transmits video at 30 frames per second may transmit its video frames at exactly 33.3 millisecond intervals and will not want the multipath system 100B to change its pace.

[0313] ii. In contrast, bursty applications may benefit from the scheduler explicitly pacing its bursts at the total WAN rate.

[0314] ο Expected throughput (e.g., minimum, target, maximum)

[0315] Rules (matching criteria and behavior) can be automatically generated using machine learning techniques that analyze traffic patterns.

[0316] The input (features) for ML methods can include:

[0317] i. The cumulative volume of the transmitted data

[0318] ii. Packet size distribution (histogram)

[0319] iii. Spacing between private rooms (both pace and trembling)

[0320] iv. Packet frequency and grouping

[0321] v. Packet header fields (Ethernet, IP, TCP, UDP, etc.)

[0322] vi. Package contents (payload)

[0323] vii. The intended destination of the packet, not just the IP address (e.g., the SNI, CN, and SubjectAltName fields in the TLS handshake and certificate exchange).

[0324] viii. Cumulative duration of the stream

[0325] ix. Time of day

[0326] x. Number of concurrent flows heading to the same destination or originating from the same source.

[0327] xi. Current or historical predictions (labels) for any concurrent flows heading to the same destination or originating from the same source (e.g., some applications open multiple flows, one as the control plane and another as the data plane; the existence of an existing control plane flow can help predict the data plane flow).

[0328] The predictions (labels) made by the ML method will include any of the behaviors mentioned above, determined based on the behavior of the training corpus.

[0329] In some embodiments, a rule may have the concept of a subordinate rule, which applies additional matching criteria and more specific behaviors while still being governed by its parent rule. For example:

[0330] A VPN flow identified by an IP 5-tuple can have rules defined by matching criteria and behavior. This will be considered the "parent" rule.

[0331] The VPN will carry many internal streams from the VPN-protected applications. Typically, these internal streams will be completely hidden from the matching criteria (encrypted by the VPN).

[0332] However, some VPNs may choose to expose a limited set of information about the internal flow by setting DSCP tags on encrypted packets that correspond to the preferences of the internal (plaintext) packets.

[0333] You can create subordinate rules that match the DSCP tag, but the resulting behavior will still be governed by the parent rule. For example:

[0334] a. Typically, VPNs require packets to be delivered in sequence (the packets can tolerate misordering, but only within a small window).

[0335] b. Behaviors configured on subordinate rules that match DSCP tags must not cause significant disorder in the delivery of packets from the parent rule.

[0336] For example, a subordinate rule requesting a maximum latency limit of 150 milliseconds and a second subordinate rule requesting maximum throughput may violate the parent rule's ordering requirements because interleaved packets from these two subordinate rules may be assigned to connections with completely different latencies.

[0337] Figure 5A Two independent flows are shown, each with completely independent rules and behaviors. In this example, the SIP flow requires low latency, so its behavior causes scheduler 104B to transmit its packets only on connection 1. Conversely, the FTP flow requires high throughput, so scheduler 104B transmits its packets only on connection 2.

[0338] When packets from both streams are transmitted over the air and arrive at sequencer 118, the packets are rearranged and transmitted independently to their final destinations. In this example, sequencer 118 is transmitting packets belonging to the SIP stream that have arrived and are in the correct order, regardless of the state of the FTP stream whose packets are still in the air.

[0339] Figure 5B This illustrates the concept of a dependent stream. In this example, the same SIP and FTP streams exist, but they are encrypted and encapsulated by the VPN client before being seen by the scheduler 104B.

[0340] From the VPN client, packets from both flows have the same IP 5-tuple, so scheduler 104B and sequencer 118 treat this as the parent flow and are bound by its requirements.

[0341] The presence of slave streams (SIP and FTP) enables communication using DSCP tags, and its control scheduler 104B determines how packets are allocated to connections 1 and 2 for transmission. In this example, allocation is as follows... Figure 5A The same as in.

[0342] However, the parent flow constraint requires that packets be transmitted from sequencer 118 in the same order as they arrive at scheduler 104B. Therefore, in this example, even if a packet belonging to a subordinate SIP flow has arrived at sequencer 118, it cannot be transmitted until a packet belonging to a subordinate FTP flow arrives and is transmitted first.

[0343] Figure 6A At 600A is a sample example of the rules management page. The configured rules are displayed in a table, where each row summarizes the matching criteria and behavior. The rules are listed in the order in which packages match those rules. Figure 6B The workflow for changing the rule order is shown at 600B.

[0344] Figure 6C At 600C, it is shown how each row in the table can be expanded to show details of the matching criteria and behaviors, as well as the active flow currently passing through the system that matches this rule.

[0345] Each rule in the table includes edit and delete icons.

[0346] Figure 6D The dialog box that appears when the edit button is clicked is shown at 600D. The matching criteria and behavior of the rule can be changed in this dialog box.

[0347] There is a separate "Add Rule" link on the main management rules screen, and Figure 6E A modal dialog box that appears when the link is clicked is shown at 600E.

[0348] In some embodiments, rules are configured on the transmitter side and automatically pushed to the receiver. By default, rules are symmetric and bidirectional. This means that when adding rules and configuring their behavior, matching pairs of rules are added transparently, but the source and destination matching criteria are interchanged. Asymmetric rules (with different behaviors in either direction) can be added via the advanced configuration screen.

[0349] In one embodiment, the scheduler probes various network characteristics of multiple connections (to determine the measured characteristics of each connection) and combines the measured characteristics with transmission requirements to make decisions about packet transmission.

[0350] For example, one such decision is determining how many packets to transmit on a particular connection and when to transmit those packets.

[0351] Some embodiments are volume-based. Such embodiments operate by determining a limit on the volume of data (e.g., in bytes or packets) that can be transmitted over a particular connection. Once enough packets have been transmitted in terms of that volume, transmission stops until some feedback is received regarding those packets. If feedback is received before the full volume limit has been transmitted, transmission can continue without stopping as long as the volume of data already transmitted for which feedback has not yet been received does not exceed the determined volume limit.

[0352] In one embodiment, the data volume is selected as the connection's congestion window (CWND). Simply put, the CWND is the maximum data volume that can be transmitted on the connection without causing significant congestion before any feedback is received. Several methods exist for estimating the CWND. It is assumed that the scheduler accesses one such method and is able to receive a CWND estimate through it.

[0353] In some embodiments, the scheduler needs to determine the data transmission rate (e.g., bytes per second). This rate can then be used to make other decisions regarding packet transmission.

[0354] For example, in one embodiment, if the data rate transmitted on the current active network connection is lower than the data rate received from the source application, the scheduler may decide to activate more connections to increase the overall transmission rate, thereby adapting to the application rate.

[0355] In another instance, an implementation may need to meet Quality of Service (QoS) requirements, which can be represented by a guaranteed transmission rate. If the overall rate drops below the required level, the scheduler may decide to activate more connections to increase the overall transmission rate, thereby meeting the QoS requirements.

[0356] In yet another instance, an implementation may need to measure actual throughput (i.e., the rate at which application data successfully passes through the network) to ensure application performance levels, optimize performance, and report, etc.

[0357] In one embodiment, TCP performance can be optimized by pacing the transmitted packets at the total rate achieved by multiple connections. See, for example, PCT application PCT / CA2020 / 051090.

[0358] In some implementations where the functions require rate, the volume-based scheduler needs a method to convert the volume of transmitted data into rate.

[0359] The simple method of converting volume to rate by dividing the volume by the transmission duration often yields inaccurate results. Such methods measure the sender's transmission rate, not the actual rate at which data moves across the network. The actual rate at which data moves across the network is actually determined by the rate of the slowest link in a multi-link network (often referred to as the network bottleneck).

[0360] The characteristics of network bottleneck links are not available to either the sender or receiver, especially since it is known that multi-link networks can change the combination of used links at any time and in a manner transparent to both the sender and receiver. However, congestion control methods can generate estimates based on performance observed at both the sender and receiver.

[0361] In some embodiments, congestion control methods can provide an estimate of the transmission rate of the bottleneck link. This is sometimes referred to as the network's bottleneck bandwidth, BtlBw.

[0362] In some embodiments, the round-trip propagation time RtProp of the network can be estimated by sampling the time between sending a packet and receiving feedback about the packet.

[0363] The combination of BtlBw and RtProp can be used to determine network characteristics known as the Bandwidth-Delay Product (BDP) as follows:

[0364] BDP = BtlBw x RtProp

[0365] The most commonly used unit for BDP is bytes. Hereafter, the following units will be assumed for these quantities: (i) BtlBw, bytes per second; (ii) RtProp, seconds; and correspondingly (iii) BDP, bytes.

[0366] BDP represents the amount of data the network will have when operating under ideal conditions (in flight), assuming data is always available for transmission. The network will transmit data at its nominal rate BtlBw and provide accurate feedback on each packet after the RtProp time of transmission. Figure 7 Two instances at 700 are shown. Figure 7 The upper part of the diagram illustrates this point using an example where BDP consists of 16 packets of equal size. In reality, the number and size of packets can vary as long as the total size equals the value calculated using BDP.

[0367] In some volume-based embodiments, BDP can be used to estimate various data transmission rates, some examples of which have been previously described.

[0368] For the data volume under consideration during flight, if the volume is smaller than BDP, assuming ideal conditions, only a small fraction of the network's nominal rate BtlBw can be achieved. This small fraction can be calculated as the ratio of volume to BDP. The achieved rate is then determined as:

[0369] The achieved velocity corresponding to the volume in flight = volume / BDP x BtlBw

[0370] In reality, networks rarely operate under ideal conditions. For example, packets may be temporarily buffered before being delivered to the receiver, or feedback may be delayed at the receiver or delayed by the network. In some cases, packets may be lost and never reach the receiver, thus failing to trigger any feedback. In other cases, the feedback itself may be lost. For example, loss can be inferred upon receiving feedback of a newer packet or based on a timer expiring.

[0371] Waiting for delayed feedback (which may or may not arrive) to trigger the next transmission event can artificially limit the achievable rate on the network.

[0372] Figure 8AFigure 800A shows an example of a receiver summarizing feedback and sending an acknowledgment back to the sender every four packets received.

[0373] Figure 8B Continuing with the demonstration at Figure 800B, for example, summarizing how this resulted in total throughput falling below network capacity. The sender waited a longer time before starting to send the next group of packets. Overall, the number of packets ultimately sent by the sender was... Figure 7 The ideal situation is the same, but the time period is longer, resulting in lower throughput.

[0374] In one embodiment, this artificial limitation is overcome by specifying an additional data transmission allowance, even before feedback on previously transmitted packets has been received. This means that the amount of data in flight may exceed BDP. However, in such cases, the network is not actually transmitting data at a higher rate, as the application of the rate equation already reached above might suggest. The additional allowance only allows the sender to maintain a constant rate equal to BtlBw over a longer period of time. Therefore, the upper limit of the estimated transmission rate for the current volume is BtlBw.

[0375] Figure 8C Continuing at Figure 800C, the demonstration illustrates, for example, how a sender can use such additional transmit allowances to continue sending packets before receiving an acknowledgment. The in-flight data volume of 19 packets exceeds the BDP of 16 packets. However, the actual rate matches the network capacity, and the same result is achieved by capping the calculated rate using the minimum in-flight volume and BDP.

[0376] In some embodiments, a similar approach may be used to estimate the actual throughput that can be achieved on the network (hereafter referred to as potential actual throughput).

[0377] A specific data volume can be divided into an actual throughput portion and another portion. The actual throughput portion contains newly transmitted application data. The other portion contains data that cannot be counted as newly transmitted application data, such as retransmissions of previously lost data, control data, probe data, etc.

[0378] If the total data volume is larger than BDP, then packet and / or feedback latency are assumed as previously described. It is assumed that no additional space is available for further actual throughput. The resulting (potential) actual throughput rate is estimated as follows:

[0379] Actual throughput rate = Actual throughput portion / Total volume x BtlBw

[0380] For example, Figure 9AA BDP network with 16 packets is shown at 900A. The volume of packets dropped into flight is 19 to achieve throughput equal to the network capacity. However, only 14 packets are actually used for throughput, resulting in an actual throughput rate of [missing value]% of the network capacity. 7 / 8. If the data volume is smaller than BDP, the difference will be considered as additional space assumed to be available for further actual throughput.

[0381] Figure 9B An example is shown at 900B where only 3 packets are in flight, resulting in an assumption of an additional 13 packets for the actual throughput when calculating the potential actual throughput rate. This additional space must be limited by the CWND determined by the congestion control method. At any given time, the additional volume allowed by congestion control is the difference between the CWND and the full volume. If the additional space for actual throughput exceeds this congestion control allowance, it is directly limited to said allowance.

[0382] Figure 9C An example is shown at 900C, where CWND has been reduced to 11 packets, which assumes that the additional space available for further actual throughput is limited to 8 packets.

[0383] The potential actual throughput rate is estimated as follows:

[0384] Potential Actual Throughput Rate = (Actual Throughput Portion + Limited Actual Throughput Additional Space) / BDP x BtlBw

[0385] Continuing with the above example, Figure 9B and Figure 9C This illustrates how the aforementioned potential actual throughput equation leads to Figure 9C The potential actual throughput rate in the scenario is low, which is in line with expectations given the reduction in CWND enforced by the congestion control method.

[0386] Although the above method is described in relation to actual throughput and potential actual throughput rate, the method can be easily modified to calculate the rate or potential rate of any type of data that partially constitutes the complete volume.

[0387] The following is a numerical example in which the previously described equations are applied to a network exhibiting those characteristics typical of those experienced on LTE cellular networks.

[0388] BtlBw = 10Mbps

[0389] RtProp = 60 milliseconds

[0390] BDP = 10Mbps * 60 milliseconds = 75KB

[0391] CWND = 60KB

[0392] Actual throughput = 25KB

[0393] Other parts = 10KB

[0394] Full size = 25KB + 10KB = 35KB

[0395] Actual throughput additional space = BDP - full size = 75KB - 35KB = 40KB

[0396] Congestion control allowance = CWND - Full size = 60KB - 35KB = 25KB

[0397] Additional space for limited actual throughput =

[0398] Minimum value (actual throughput extra capacity, congestion control allowance) =

[0399] Minimum value (40KB, 25KB) = 25KB

[0400] Potential actual throughput rate =

[0401] (Actual throughput portion + additional space for restricted actual throughput) / BDP x BtlBw =

[0402] (25KB+25KB) / 75KB x 10Mbps=50 / 75x 10Mbps=6.67Mbps

[0403] Systems that communicate over IP networks typically implement a congestion control mechanism that attempts to achieve fair network utilization across all network nodes that employ similar congestion control mechanisms.

[0404] Congestion control mechanisms typically operate by monitoring communications occurring on the network, potentially probing the network as needed, and deriving network characteristics based on the results. A model representing the network is created, and this model guides future communications. For example, the model might track network characteristics such as current and maximum throughput, current and minimum round-trip times, packet loss rates, etc.

[0405] Based on the created model and the tracked information, the congestion control mechanism can determine when the network is performing poorly, thus indicating that the network may be experiencing congestion. The congestion control mechanism can then take corrective action aimed at reducing or removing network congestion and restoring the desired network performance.

[0406] In conventional congestion control implementations (e.g., TCP), packet loss rate is used as an indicator of network congestion, and corrective actions (e.g., reducing CWND) are generated in response to the loss.

[0407] In one embodiment, packet latency, i.e., the time it takes for a packet to traverse the network from the sender to the receiver, is a metric for network performance. Packets with high latency may be useless to the receiver and be lost, resulting in wasted transmission. For example, real-time VoIP applications that require low dialogue latency between participants would be useless for packets carrying voice data later than the acceptable maximum dialogue latency.

[0408] In such embodiments, the congestion control mechanism can monitor packet latency and attempt to control the latency so that it does not exceed an acceptable level.

[0409] In one embodiment, the lowest latency measured for a packet can be used to establish a baseline value, which future measurements can be compared to. If a future measurement exceeds a threshold that varies with the baseline value, the congestion control mechanism considers the network to be experiencing congestion.

[0410] In another embodiment, a similar method uses the round-trip time of the packet (i.e., the time elapsed between sending the packet and receiving an acknowledgment about the packet) instead of latency.

[0411] In one embodiment, when the congestion control mechanism determines that the network is experiencing congestion based on increased packet latency, the congestion window (CWND) is reduced to limit the amount of data that can be sent over the network without receiving any feedback. The aim is to reduce the amount of data the network will be buffering or is currently buffering, which is a common cause of increased packet latency.

[0412] In one embodiment, the congestion window is reduced to the network's bandwidth-delay product (BDP). Theoretically, BDP represents the volume of data the network should be able to deliver without forming a packet stagnation queue within its buffer, assuming packets are transmitted by the sender at a constant pace reflecting the network's throughput (BtlBw). Therefore, this is expected to allow the network to recover at least partially and reduce packet latency back to an acceptable level.

[0413] For example, suppose the network has a BDP of 16 packets and the congestion control mechanism determines a CWND of 24 packets to account for acknowledgment summaries. Figure 10AAt 1000A, it was demonstrated how congestion from other network nodes can overwhelm the sender's network. In this case, the sender did not reduce its congestion window and released the full amount of data in flight. This filled the network buffer, and additional packets that could not be buffered were lost. Even after the congestion was cleared and normal depletion of the network buffer continued, Figure 10A It demonstrates how a stationary queue can be formed within the network buffer, which increases the round-trip time for all future packets and makes it easier to fill the network buffer.

[0414] Continuing with the example, Figure 10B Section 1000B demonstrates how a sender reducing its congestion window to BDP can avoid both packet loss and the formation of stagnant queues. For example, a sender can determine congestion is occurring by detecting a period of time during which no feedback has been received exceeding the baseline duration of feedback measured in past transmissions. In another instance, a sender can determine congestion is occurring by receiving feedback indicating that the amount of previously sent packets exceeding the baseline loss experienced in past transmissions has not yet been delivered to the receiver. In response to this congestion determination, reducing the congestion window to match the BDP reduces the amount of packets that will be dropped into flight. This correspondingly reduces the likelihood of previously filled network buffers causing packet loss. Furthermore, dropping only BDP packets into flight allows the network to completely exhaust its buffers before new packets from the sender arrive at its buffers again (assuming no change in network characteristics).

[0415] In another embodiment, the congestion window is reduced to be equal to the packet volume in the current flight.

[0416] When the congestion control mechanism determines that an acceptable level of performance has been restored, the network is considered to no longer be experiencing congestion.

[0417] In one embodiment, a packet latency reduction below a threshold indicates that network performance has recovered.

[0418] In one embodiment, the threshold is determined to vary with the baseline delay value.

[0419] In another embodiment, transmitting a specific volume of data without causing any loss indicates that network performance has been restored.

[0420] In one embodiment, the data volume is equal to BDP.

[0421] In another embodiment, the data volume is equal to the original value of the congestion window before the congestion control mechanism reduces the congestion window to cope with congestion.

[0422] For the most common application types, push-based architecture improves system performance for non-BoD connections. Table 3 summarizes the improvements that have been observed experimentally.

[0423] Table 3: Application performance improvements resulting from push-based schedulers

[0424]

[0425]

[0426] Push-based scheduling methods also make priority routing behavior more intuitive. For example, in one embodiment, the decision of whether to engage with a lower-priority connection depends solely on a comparison of its BtlBw with a configured threshold.

[0427] For example, consider the following scenario:

[0428] wan0

[0429] Priority: Level 1

[0430] • Target threshold: Not applicable

[0431] • BtlBw: 10Mbps

[0432] wan1

[0433] Priority: Level 2

[0434] Target threshold: 15Mbps

[0435] οBtlBw: 30Mbps

[0436] Using a pull-based scheduler, the scheduler provides packets for transmission each time wan1 wakes up from its 50Hz timer and requests packets, because the scheduler finds that the desired target threshold (15Mbps) is greater than the threshold (10Mbps) that wan0 alone can supply.

[0437] However, the scheduler does this regardless of the rate at which packets arrive at the input. For example, if the input rate does not exceed 10 Mbps, it would be better not to engage wan1, as the traffic can be handled entirely by wan0.

[0438] Using a push-based scheduler, wan0 will be prioritized at the top of the list, and even if a threshold indicates that wan1 should be proactive, packets will only be scheduled for wan0 after its CWND has been fully consumed. Therefore, there will be less unintended use of lower-priority connections, thus improving performance.

[0439] In summary, the scheduler for a multipath system can be implemented as either pull-based (100A) or push-based (100B). Push-based is superior because the scheduler can make packet scheduling decisions at its own pace, rather than being driven by the rate at which the target connection is ready to transmit. This self-pacing allows it to consider all factors important to the application generating the packets and / or the system users.

[0440] For example, packets generated by TCP-based applications that require maximum throughput will prefer to have their packets scheduled onto connections ordered from best to worst by the Massis coefficient.

[0441] A more complex instance is also considered, using the same TCP-based application but where the user has configured priority routing rules in the multipath system 100B, favoring connection 110B over all other connections 110N, unless the actual throughput achievable on 110B drops below 5 Mbps. In the hypothetical scenario where 110B has high throughput, low latency, but high packet loss rate, 110B will have a worse Mathis coefficient than all other connections 110N.

[0442] The push-based scheduler 100B will be able to make complex scheduling decisions that respect user preferences for prioritizing connection 110B, and will also calculate the actual throughput achievable on 110B by converting its in-flight CWND to a bit rate and using that bit rate to estimate its actual available throughput. If this availability drops below a configured 5Mbps threshold, it will schedule packets to the remaining connections 110N, sorted by their Massis coefficients from best to worst.

[0443] As described above, the measured connectivity characteristics may include: packet loss rate over three independent sliding windows (500 milliseconds, 3 seconds, and 30 seconds), which is further summarized as the worst value over the three windows; current network round-trip time (RTT); bandwidth and round-trip propagation time (BBRv1) parameters: RtProp (network propagation delay); and BtlBw (throughput estimate). External instructions may include priority routing (PR) rules, but may also include flow-specific rules / requirements.

[0444] In some embodiments, the measured characteristics of BBRv1 differ from its design in the original BBRv1 IETF draft. For example, considering embodiments running on systems that may be inconsistent in their execution time, the size of the sliding window used to calculate BtlBw may vary statically or dynamically. For similar reasons of execution consistency, the input parameters used to calculate RtProp may also be slightly modified.

[0445] Before determining the sorting order, some or all of the input parameters can be rounded / bucketed / quantified. The technical improvement of rounding and bucketing is to keep the sorting order as "stable" as possible, so that the sorting order does not change constantly (for example, without rounding and bucketing, two similar values ​​may cause the sorting to change constantly due to the fluctuation of their values ​​over time, resulting in "jitter").

[0446] Factors influencing its decision-making include: maintaining the sorting order as "stable" as possible so that packets are not unnecessarily split across multiple WAN connections (splitting increases the likelihood of spontaneous out-of-order and jitter events); and considering both measured characteristics of the connection and unmeasured external commands.

[0447] Values ​​rounded to the nearest whole number can include the following:

[0448] Latency-related values ​​(RtProp, RTT) are rounded to the nearest multiple of 50 milliseconds, with a lower limit of 1 millisecond after rounding.

[0449] • The throughput-related value (BtlBw) is rounded to the "closest order of magnitude".

[0450] • The packet loss rate (the worst value in the 3 sliding windows) is rounded to the nearest order of magnitude. Note that the original values ​​here are limited to the range [0,100], so the output values ​​are limited to the set (0,1,2,3,4,5,6,7,8,9,10,20,30,40,50,60,70,80,90,100).

[0451] The table below provides examples of rounding to the nearest order of magnitude. It shows the original value, its calculated order of magnitude, and then the resulting rounded value.

[0452]

[0453]

[0454] In some embodiments, specific improvements are described for controlling network interface usage based on improved network routing order, wherein throughput is predicted based on the Mathis Equation using the combined characteristics of round-trip time (RTT) and packet loss rate to establish an upper limit on TCP throughput for a given RTT, packet loss rate, and MSS. The Mathis coefficient can be used to modify the order in which network packets are routed.

[0455] The measured characteristics may include: packet loss rate over three independent sliding windows (500 milliseconds, 3 seconds, and 30 seconds), which is further summarized as the worst value over the three windows; current network round-trip time (RTT); bandwidth round-trip propagation time (BBRv1) parameters: RtProp (network propagation delay); and BtlBw (throughput estimate). External instructions may include priority routing (PR) rules, but may also include flow-specific rules / requirements.

[0456] In some embodiments, the measured characteristics of BBRv1 differ from its design in the original BBRv1 IETF draft. For example, considering embodiments running on systems whose execution times may be inconsistent, the size of the sliding window used to compute BtlBw can be increased to cover long durations (hiding variability caused by execution inconsistencies), but the window duration can also be shortened immediately if a network loss or increased latency event occurs, as those events indicate that BtlBw may have actually changed, and any long-term history within the sliding window must be forgotten.

[0457] For similar reasons of inconsistency, the input parameters used to measure RtProp can be changed to include a portion exceeding network propagation delay, thus adding a component of execution / processing time, as this time may be non-negligible and continuously contributes to actual latency while transmitting packets and waiting for responses. In some embodiments, this is referred to as "system RtProp" rather than "network RtProp," which consists solely of network propagation delay. In these embodiments, "system RtProp" and "network RtProp" can be used for different calculations, depending on the intended purpose. For example, "system RtProp" is suitable when calculating CWND, ensuring the pipeline contains sufficient in-flight data to account for processing latency. However, "network RtProp" is more suitable when sorting network connections to determine packet routes, since preferred routing decisions do not depend on system execution latency.

[0458] Packet loss rate, latency, and throughput are not entirely independent variables. They interact and combine depending on the application's congestion control behavior. This is because, for example, with TCP, a higher packet loss rate leads to longer air "silent periods" (underutilized channels) while the sender waits for DUP ACK or RTO.

[0459] Therefore, some comprehensive characteristics were determined and used in the final sorting method. Specifically:

[0460] Mathis coefficient: This composite characteristic takes into account RTT and packet loss rate to predict throughput and is based on the Mathis relation, which gives an upper limit to TCP throughput given RTT, packet loss rate, and MSS. When comparing the characteristics of any two network connections to determine their ranking, MSS can be considered a constant; therefore, this relation indicates that the expected throughput is proportional to the reciprocal of (RTT * sqrt(packet loss rate)). In this example, this relation is used as a practical implementation to establish a coefficient for directly controlling the operation of a network controller or router. For each connection, the system determines the Mathis coefficient as: maximum value (rounded RTT, 1 millisecond) * sqrt(rounded summed packet loss rate). Connections with smaller Mathis coefficient values ​​are considered better. The calculation is a function of two independently measured characteristics, and therefore the relationship between these characteristics determines how connections will be compared to each other. The Mathis coefficient can be maintained and stored in data objects, such as the corresponding Mathis coefficient data values ​​associated with each connection, and stored along with other monitored characteristics in, for example, an array of data values.

[0461] For example, in an extreme case, two connections with a 0% packet loss rate will always have the same Massis coefficient—their actual RTT value is irrelevant. In this scenario, the Massis coefficient does not affect the sorting order—which will be determined by comparing subsequently measured characteristics and combined characteristics.

[0462] In another extreme case, if two connections have a packet loss rate of 100% (rounded summed packet loss rate = 1.0), then only their rounded Rtt values ​​will determine how they are compared to each other. In some embodiments, this is undesirable, and therefore connections with a packet loss rate of 100% are filtered out from the available options at an earlier stage.

[0463] Between the two extremes, connections with both low RTT and low packet loss rate are better than those with higher values. However, connections with a high percentage of packet loss can still be compensated for by having a low RTT and will have the same Massis coefficient as connections with a low percentage of packet loss. For example:

[0464] Connection A: 100 ms RTT, 0.01 (1%) loss rate => Mathis coefficient = 10

[0465] Connection B: 19 ms RTT, 0.25 (25%) dropout rate => Mathis coefficient = 9.5

[0466] In this example, although connection B has a significantly higher loss rate (25% vs. 1%), its lower RTT compensates for this, resulting in a better Mathis coefficient compared to connection A. This phenomenon can be explained by the fact that, despite connection A's lower percentage loss rate, connection B's lower RTT allows it to exchange packet acknowledgments (positive and negative) for lost packets and completes any necessary packet retransmissions much faster than connection A could with its higher RTT.

[0467] Capacity Factor: This comprehensive characteristic takes into account BtlBw and RtProp and is determined as: rounded BtlBw / maximum value (rounded RtProp, 1 millisecond). The unit is "bytes per millisecond," which has no physical meaning but provides a way to normalize the bandwidth achievable per unit of RtProp for a connection. In other words, a connection that achieves more bandwidth per millisecond of RtProp is considered more efficient than one that achieves less bandwidth. In this context, efficient transmission means a connection that achieves more bandwidth per millisecond of propagation delay (RtProp).

[0468] Once all the original inputs have been processed and the comprehensive properties have been calculated as described above, the actual sorting mechanism can be executed sequentially as follows:

[0469] Moving from one comparison to the next only occurs if all the first few comparisons are equal. If the comparisons are not equal, the remaining comparisons are irrelevant and skipped. In some embodiments, the selection of comparisons is based on flow classification (e.g., VPN, relative to VoIP, relative to ping packets, relative to FEC packets), and the list of comparison criteria can be different for each different flow (e.g., each flow can have different predefined most important, second most important, third most important, etc.). Each of these characteristics can be iterated sequentially from the most important until the connections themselves can be sorted and placed in a sequential sequence. In some embodiments, depending on the flow classification, the first preparatory step is to select a subset of connections suitable for the packet types in the sending flow classification, and then assign them an order before performing packet allocation.

[0470] 1. Connections that are completely paralyzed (with a totaled loss rate of 100%) are considered worse than connections with a totaled loss rate of <100%.

[0471] 2. PR Priority (the higher the priority, the better)

[0472] 3. Massis coefficient (the lower the better)

[0473] 4. RTT after rounding (lower is better)

[0474] 5. Capacity factor (the higher the better)

[0475] 6. RtProp, rounded down (lower is better)

[0476] 7. BtlBw (rounded to the nearest whole number) (the higher the better)

[0477] 8. Connection ID (lower is better)

[0478] Figure 1F The image shows a rendering of the sorting mechanism. In one example sorting mechanism, the real-time stream first includes a comparison of RTT / RtProp, and if they are equal, it then compares reliability, and then, if that is also equal, compares transmission rate, etc. Depending on the stream type, there may be different sorting orders. For example, a stream that requires throughput might first compare transmission rates. If they are equal, it then compares reliability. If that is also equal, the stream might decide not to do any further evaluation and then simply choose arbitrarily.

[0479] In some embodiments, the streaming requirement (e.g., real-time, relative to throughput) can specify the order and number of comparisons. For example, a real-time streaming stream might prefer to compare only rounded RTT and rounded RtProp, ignoring all other characteristics. A throughput streaming stream might decide to compare only the Massis coefficient and the capacity coefficient.

[0480] Most application traffic tends to be bursty or adaptive, so it doesn't need to utilize all WAN connections. This means that a connection currently at the bottom of the sort order may not be carrying any traffic. If a connection doesn't do this, its explicit sorting dimension won't be updated, so the connection's position tends to remain near the bottom of the sort order.

[0481] This is generally considered a good characteristic of algorithms, but it can be modified due to external factors. For example, if a flow has specific requirements that cannot be satisfied by connections currently near the top of the sort order, it may make sense to generate probe flow on connections near the bottom to refresh its explicit sort dimension.

[0482] For example, as explained in further detail below, in some embodiments, these connections may receive a score of 0 in the layer packetization, allowing them to be used for packet transmission under any circumstances and to continue scheduling. For instance, early retransmissions may be sent to outdated and unreliable connections to refresh the connections and check whether they have become reliable.

[0483] Figure 11A hybrid connectivity system 1100 is illustrated, configured to utilize an improved scheduling method in the transmission portion of the system and a buffering system for packet ordering at the receiving end. As shown, in one embodiment, the components in the system are hardware components configured to interoperate with each other. In another embodiment, the hardware components are not discrete components, and more than one component may be implemented on a particular hardware component (e.g., a computer chip performing the functions of two or more components).

[0484] In some embodiments, the hardware components reside on the same platform (e.g., the same printed circuit board), and system 1100 is a single device that can be transported and connected to a data center / field-portable device (e.g., a rugged mobile transmitter), etc. In another embodiment, the components are distributed and may not all be located in a nearby location, but rather communicate electronically via telecommunications (e.g., processing and control are performed by components residing in a distributed resource environment (e.g., the cloud), rather than locally).

[0485] In mobile scenarios where signal quality, network availability, and network quality are suboptimal (e.g., professional news gathering / video creation may occur in locations without robust network infrastructure), hybrid connectivity is particularly desirable. Therefore, in some embodiments, the device described herein is a rugged network controller device adapted to operate in remote or mobile scenarios, such as single-person portability (e.g., worn in a backpack by a news technician), vehicle portability (e.g., pushed along a media vehicle), or vehicle portability (e.g., coupled to a media van, truck, boat, helicopter, or aircraft).

[0486] Alternatively, the device may be located in a communication center or other centralized communication facility that coordinates connections with other endpoints. For example, in this instance, the device may be located, coupled to, or reside in communication equipment such as a communication relay station (e.g., satellite station, relay transmitter, broadcast translator, repeater, repeater, supplementary station) that can coordinate communication with various endpoints across multiple different channels, such as television stations, radio stations, data communication stations, personal computers, mobile devices, etc.

[0487] Multiple distinct data connections 1106 (e.g., "paths") representing one or more networks (or network channels) are illustrated, labeled as connection 1, connection 2, connection N. Multiple data connections / paths may exist across a single network or multiple data connections that may use one or more networks.

[0488] These data connections can include various types of technologies, and each technology may face different communication environments, such as different types of available communication capabilities, array architectures, polarization, multipath propagation mechanisms, spectral interference, frequency ranges, modulation, etc., and therefore, the connections may all have different levels of communication reliability.

[0489] System 1100 can be configured to communicate with various endpoints 1102, 1110, or applications that do not need to have any information about the multiple paths / connections 1106 used to request and receive data (e.g., endpoints 1102, 1110 can operate independently of paths or connections 1106). For example, received data can be reconstructed so that the original transmission can be regenerated from the contributions of different paths / connections 1106 (an example use case would be regenerating video by a receiver configured to be inserted into a server rack at a data center facility, thereby integrating with existing broadcast infrastructure to provide improved networking capabilities).

[0490] System 1100 receives input (data stream) from source endpoint 1102 and schedules the delivery of improved data packets across various connections 1106, then sorts the packets at the other end of the system 1108 before transmitting them to the destination endpoint application 1110. By doing so, system 1100 is configured to increase bandwidth to approximate the sum of the maximum bandwidth of the various available paths. System 1100 also provides improved reliability compared to using a single connection, which is an important consideration in time-constrained, highly sensitive scenarios such as news gathering at the scene of an event while it is unfolding. In these events, there may be severe signal congestion (e.g., sporting events) or unreliability across one or more paths (e.g., news reporting after a natural disaster).

[0491] Multiple connections can be bound together to operate as a single connection, and different methods can be used to coordinate routing, allowing a connection to be used multiple times to send the same packet, etc.

[0492] In various embodiments, both scheduler 1160 and sequencer 1162 may be provided by a cloud computing implementation, or they may be provided at the endpoint (before the application consumes data at the endpoint), or they may be provided in various combinations thereof.

[0493] System 1100 can be adjusted to optimize and / or prioritize performance, optimal latency, optimal throughput, minimum jitter (the variation in latency on packet streams between two systems), connection costs, and combinations of connections for specific streams, etc. (For example, if system 1100 has information that the transmission (data stream) belongs to content type X, then system 1100 can be configured to use only data connections with similar latency, while content type Y may allow for a wider combination of data connections (or require a larger net capacity that can only be achieved through combinations of data connections)). This adjustment can be provided generally to the system or specifically to each stream (or to the set of streams based on location, owner of the origin or endpoint or a combination thereof, transmission time, available set of communication links, security required for transmission, etc.).

[0494] Although in some embodiments only one gateway may be required, system 1100 can typically be bidirectional, in which each gateway 1104, 1108 will typically have a scheduler 1160 and an orderer 1162 to handle TCP traffic (or UDP traffic, or a combination of TCP and UDP traffic, or any type of general Internet Protocol (IP) traffic).

[0495] System 1100 can be used in various scenarios, such as as a failover (e.g., for disaster recovery) or supplement to existing internet connections (e.g., VoIP phone systems or enterprise network connections), seamlessly adding additional networks (or paths) to replace a lost primary internet connection, or supplementing a saturated primary internet connection by binding to a more expensive network. In the event of a failover, coordinated failures may occur across many communication channels (e.g., large-scale denial-of-service attacks or major solar storms / geomagnetic storms), and System 1100 will then need to utilize communication resources efficiently due to the scarcity of available communication resources.

[0496] Another use of System 1100 is to provide a way to maximize the use of high-cost (often sunk cost), high-reliability data connections (such as satellites) by allowing traffic to be offloaded to other data connections with different characteristics.

[0497] In some embodiments, system 1100 is a network gateway configured to route data flows across multiple network connections.

[0498] Figure 11An overview of a system with two gateways 1104 and 1108 is provided. Each gateway contains a buffer manager 1150, an operation engine 1152, a connection controller 1154, a flow classification engine 1156 (responsible for flow identification and classification), a scheduler 1160, a sequencer 1162, and a network characteristic monitoring unit 1161, and is linked through N data connections 1106, where each gateway is connected to specific endpoints 1102 and 1110. Reference letters A and B are used to distinguish the components of each gateway in the two gateways 1104 and 1108.

[0499] Each gateway 1104 and 1108 is configured to include multiple network interfaces for transmitting data over multiple network connections and is an apparatus (e.g., including configured hardware, software, or embedded firmware) comprising a processor configured to: monitor time-varying network transmission characteristics of the multiple network connections; parse at least one packet in a data packet stream to identify a data stream category, wherein the data stream category defines or otherwise associates at least one network interface requirement of the data stream; and route packets in the data stream across the multiple network connections based on the data stream category and the time-varying network transmission characteristics.

[0500] Buffer manager 1150 is configured to set up buffers within the gateway to more efficiently manage traffic (both single flows and combinations of multiple simultaneous flows through the system). In some embodiments, buffer manager 1150 is a discrete processor. In other embodiments, buffer manager 1150 is a computing unit provided as a processor configured to perform activities such as buffer management 1150.

[0501] Operation engine 1152 is configured to apply one or more deterministic methods and / or logical operations based on received input datasets (e.g., feedback information, network congestion information, transmission characteristics) to inform the system of constraints to be applied to bound connections, said constraints being per user / client, destination / server, connection (e.g., latency, throughput, cost, jitter, reliability), and stream type / requirement (e.g., FTP, versus HTTP, versus streaming video). For example, operation engine 1152 may be configured in one instance to restrict certain types of streams to specific connections or data connection sets based on cost, but reliability and low latency may be more important for different users or stream types. For example, different conditions, triggering conditions, or methods may be utilized based on one or more elements of known information. For example, operation engine 1152 may be provided on the same or different processors as buffer manager 1150.

[0502] The operation engine 1152 can be configured to generate, apply, or otherwise manipulate or use a set of rules for one or more defined logical operations to control routing on N data connections 1106.

[0503] The flow classification engine 1156 is configured to evaluate each data flow received by the multipath gateway 1104 for transmission and is configured to apply flow classification methods to determine the type of traffic being transmitted and its requirements (if not already known). In some embodiments, deep packet inspection techniques are suitable for performing the determination. In another embodiment, the evaluation is based on heuristic methods or data flows that have been tagged or labeled at the time of generation. In another embodiment, the evaluation is based on rules provided by the system's user / administrator. In yet another embodiment, a combination of methods is used. The flow classification engine 1156 is configured to interoperate with one or more network interfaces and can be implemented using electronic circuitry or a processor.

[0504] Scheduler 1160 is configured to perform determination, i.e., about which packets should be sent through which connections 1106. Scheduler 1160 can be considered an improved QoS engine. In some embodiments, scheduler 1160 is implemented using one or more processors or stand-alone chips or configuration circuitry (such as comparator circuitry or an FPGA). Scheduler 1160 may include a series of logic gates that validate the determination.

[0505] While a typical QoS engine manages a single connection, the QoS engine (or scheduler 1160 in this case) can be configured to perform flow identification and classification, and the end result is that the QoS engine reorders the packets before they are sent out on a connection.

[0506] In contrast, while scheduler 1160 is configured to perform flow identification, classification, and packet reordering, in some embodiments scheduler 1160 is further configured to determine which connection to send packets to in order to improve the transmission characteristics of the data flow and / or satisfy policies set by the user / administrator for the flow (or policies stated in various rules). For example, scheduler 1160 can modify network interface operating characteristics by transmitting a set of control signals to the network interface to open or close the network interface, or to indicate which interface should be used for routing data. Control signals may be a set of instructions indicating specific characteristics of the desired routing (such as packet timing, network interface reservation for specific types of traffic, etc.).

[0507] For example, consider two connections with the following characteristics:

[0508] Connection 1: Round-trip time (RTT), 1 millisecond; estimated bandwidth, 0.5 Mbps; and

[0509] Connection 2: RTT, 30 milliseconds; estimated bandwidth, 10 Mbps.

[0510] Scheduler 1160 may attempt to reserve connection 1 exclusively for Domain Name System (DNS) traffic (small packets, low latency). In this example, excessive DNS traffic may reach the capacity of connection 1, causing scheduler 1160 to be configured to overflow traffic to connection 2. However, scheduler 1160 may do so selectively based on other deterministic or factor-based decisions (e.g., if scheduler 1160 is configured to provide fair determination, it may be configured to first overflow traffic from IP addresses that have sent a large amount of DNS traffic in the past X seconds).

[0511] For example, scheduler 1160 can be configured to process determinations based on a process or method that operates in conjunction with a similar implementation in one or more processors or hardware (e.g., an FPGA). Scheduler 160 can be configured to operate under the control of operation engine 152 to deassemble data streams into packets and then route the packets to a buffer (managed by buffer manager 150) that feeds packets to the data connection according to rules designed to optimize packet delivery while taking into account the characteristics of the data connection.

[0512] In some embodiments, the scheduler 1160 is not configured to communicate packets in the correct order, but rather to communicate packets across different connections to meet or exceed desired QoS / QoE metrics (some of which may be defined by the network controller, and others by the user / client). In cases where packets may be communicated out of order, the sequencer 162 and buffer manager can be used to reorder the received packets.

[0513] A series of packet bursts are transmitted across a network interface, and a bandwidth estimate for the first network interface is generated based on the timestamps of when packets in the bursts were received at the receiving nodes and the size of the packets. Then, based on the generated bandwidth of the first network interface, packets in the data stream of the consecutive packets are routed across the network connection set using the estimate. As described below, in some embodiments, the bandwidth estimate is generated based on the timestamps of packets in the burst that are not merged with the initial or final packet in the burst, and lower bandwidth values ​​can be estimated, and higher bandwidth values ​​can be estimated (e.g., through packet replacement). The transmitted packets can be test packets, test packets "carried" with data packets, or mixed packets. In cases where data packets are used for "carrying," some embodiments include marking such data packets to increase redundancy (e.g., to enhance tolerance for lost packets, especially those used for bandwidth testing purposes).

[0514] In some embodiments, consecutive packets may be received sequentially or within an acceptable order deviation, enabling sequencer 1162 to rearrange the packets for consumption. In some embodiments, sequencer 1162 is a physical hardware device that may be incorporated into a broadcast or networking infrastructure that receives signals and generates an output signal of reassembled signals. For example, the physical hardware device may be a rack-mounted device that acts as a first stage for signal reception and reassembly.

[0515] Sequencer 1162 is configured to order received packets and transmit them to the application at the endpoint in an acceptable order to reduce unnecessary packet retries or error corrections in other flows. In some embodiments, the order is consistent with the original order. In other embodiments, the order is within an acceptable range, allowing sequencer 1162 to still reassemble the data stream. For example, sequencer 1162 may include buffers or other mechanisms to smooth the received stream, and in some embodiments, the sequencer is configured to control the transmission of acknowledged packets and the storage of packets based on monitored transmission characteristics of multiple network connections and uneven distribution in the data stream receiving consecutive packets.

[0516] For example, sequencer 1162 may be provided on a processor or implemented in hardware (e.g., field-programmable gate array), provided under the control of operation engine 1152, which is configured to reassemble the data stream from received packets extracted from the buffer.

[0517] The sequencer 1162 is configured on a per-stream basis to hide the latency differences between multiple connections that would be unacceptable for each stream.

[0518] The operation engine 1152 can operate as an aggregator of information provided by other components (including 1154) and guide the sequencer 1162 through one or more control signals, that is, instruct the sequencer 1162 how to operate on a given stream.

[0519] When configured to receive packets for protocols such as TCP, the system is typically configured to expect (but not require) packets to arrive in order. However, the system is configured to establish time bounds (typically multiples of round-trip time or RTT) for when it expects out-of-order packets to arrive. The system can also be configured to retransmit missed packets before the time bounds based on heuristics (e.g., fast retransmission triggered by three DUP ACKs).

[0520] When packets arrive at sequencer 1162 on connections with significantly different delays, sequencer 1162 (on a per-stream basis) can be configured to buffer packets until the packets have approximately the same age (delay) before forwarding them to their destination. This would be done, for example, if the streams require consistent delay and low jitter.

[0521] The sequencer 1162 does not necessarily need to provide reliable, strictly ordered packet delivery, and in some embodiments, it is configured to provide the necessary packets so that systems using protocols (e.g., applications on top of TCP or UDP) do not prematurely determine that packets have been lost by the network.

[0522] In some embodiments, sequencer 1162 is configured to monitor (based on data maintained by operation engine 1152) the latency variation (jitter) and packet loss rate of each data connection to predict which data connections are likely to delay packets beyond the expected flow based on connection reliability (meaning endpoints 1102 and 1110 will consider the packet loss rate and invoke their error correction routines).

[0523] For example, in the case of out-of-order packets, sequencer 1162 can utilize a large jitter buffer on connections exhibiting significant latency variations. For packet retransmission, sequencer 1162 can be configured to immediately request lost packets via the “best” (most reliable, lowest latency) connection.

[0524] In some embodiments, a network controller apparatus (e.g., a router) is described, the apparatus including a processor coupled to a computer memory and a data storage device, the processor being configured to: receive one or more datasets indicating monitored network communication characteristics; maintain a hierarchical representation of a plurality of connections in a data structure stored on the data storage device, the plurality of connections being divided into a plurality of groups, each group being established at least based on a minimum probability associated with successful packet communication across one or more of the plurality of connections residing in the group; and control multiple communications of packets such that the packets are transmitted at least once across one or more of the plurality of connections, such that the total number of communications results in the transmission of the packets satisfying a target probability threshold.

[0525] In some embodiments, the reception of data packets defines reliability with respect to a target probability threshold, and in such respect, multiple communications result in the reception of data packets satisfying the target probability threshold.

[0526] On the other hand, multiple groups are organized into multiple corresponding layers, each layer representing the number of times a data packet must be transmitted across any corresponding connection in order to reach a target probability threshold.

[0527] On the other hand, multiple communications involve retransmissions of connections across multiple layers, where the number of retransmissions is based on the number of times a data packet must be transmitted across the corresponding connection of a layer to reach a target probability threshold for that layer. In different embodiments, retransmissions can be performed on the same connection, but at different times (i.e., different bursts) to prevent the loss of all retransmissions due to a temporary decrease in network reliability (i.e., burst loss due to buffer saturation of intermediate hosts in the path).

[0528] On the other hand, different connections across layers perform retransmission.

[0529] On the other hand, multiple communications involve retransmission of connections across different layers in multiple layers.

[0530] Due to configured priority routing preferences, connections may not participate in transmission events, potentially deactivating lower-priority connections because higher-priority connections have sufficient bandwidth to handle priority data streams. In the absence of other traffic, this limits the system's knowledge of lower-priority connections, and the scheduler 1160 or connection manager 1154 must account for outdated observations of connection characteristics.

[0531] This article describes embodiments that can be extended based on the methods and systems described above.

[0532] In some embodiments, new scheduling methods (e.g., algorithms) may be introduced to address the following three problems that arise when sorting connections in an absolute preference order according to their characteristics (as described above): these algorithms are implemented on the physical computing controller and / or circuitry.

[0533] A) Each flow with the same preferences uses connections in the same order. Flows may unnecessarily compete with each other on each connection, increasing the probability that the flow's traffic will be split across multiple connections. Where possible, the preference is to keep the flow's traffic sticky only to a single connection to avoid unnecessary packet reassembly work on the other side.

[0534] B) Connections that are used less frequently are those that are further down the absolute preference order, and therefore are more likely to be obsolete.

[0535] C) Connections with very similar measurement / synthesis characteristics may end up frequently swapping positions in the absolute preference order, even after rounding / bucketing, resulting in packets from the stream being unnecessarily split between connections.

[0536] Another technically challenging situation to meet is in real-time applications with real-time deadlines, such as video chat and voice calls, where timely packet reception with minimal jitter / loss is required due to the deadlines. In some embodiments, both latency and jitter requirements can be specified. For example, to make voice and video smooth, they need to enter at a specific frame rate; otherwise, the user may experience freezes and pauses.

[0537] In a non-limiting instance, the stream types considered may include those generated by streaming video applications. Using these applications, the sender captures and encodes video at a given frame rate, which is typically constant (e.g., 50 frames per second), but can be variable in some cases. The sender typically packages these encoded video frames into UDP packets that include a timestamp representing the relative capture time of each video and delivers them to the receiver. For example, for a frame rate of 50 fps, if the timestamp of the first video frame is initially t = 0, then the timestamp of the next video frame would be t = 20 milliseconds (1 second divided by 50 frames per second).

[0538] In these streaming video applications, the sender then transmits UDP packets to the receiver, which needs to receive the packets in a strictly time-sensitive manner. This is because the instant the receiver begins playing back the first video frame, all subsequent frames (consisting of one or more UDP packets) must be received within 20 milliseconds of the previous frame; otherwise, the video played back to the viewer will be perceived as having artifacts (discontinuous or stuttering), resulting in a poor viewing experience.

[0539] From a networking perspective, UDP packet delivery typically occurs on a best-effort basis, meaning there is sufficient variability for the UDP packets that make up each video frame to not be continuously delivered to the receiver within 20 milliseconds following the previous video frame. This could, for example, be due to queuing delays caused by contention for available capacity at one or more network links between the sender and receiver. Receivers typically attempt to address this by storing video frames in a jitter buffer, sized large enough to handle predicted variations in delivery time. For example, a jitter buffer storing two 50fps video frames would allow network variability in UDP packet delivery time to be hidden up to 40 milliseconds. However, a larger jitter buffer results in longer glass-to-glass latency (the delay between when a video frame is captured and when it is played back). When setting the size of its jitter buffer, the receiver must make trade-off decisions to balance perceived playback smoothness (keeping video frames exactly 20 milliseconds apart) with the overall latency experienced by the viewer.

[0540] For this streaming video instance, the opportunity and challenge presented by having multiple connections lies in the existence of multiple network paths between the sender and receiver. The queuing delays and resulting variability of UDP packet delivery times on each connection will generally be uncorrelated, so the sender may use scheduling techniques to transmit each UDP packet on one or more connections, resulting in video frames always arriving at the receiver before their respective deadlines. This allows the receiver to use a smaller jitter buffer and reduces the overall latency experienced by the viewer.

[0541] Throughput-based applications (such as bulk data delivery, including FTP, and large file downloads) may pose different challenges because there is less concern about latency and jitter. Latency may not be an issue as long as the connection provides throughput. For example, for file transfers, an initial pause before the transfer begins may often be acceptable.

[0542] Previous schedulers may have ordered connections in an absolute order from best to worst, which presents a technical problem because every application using connections in the same order is not necessarily good, especially when the application has competing requirements (e.g., throughput, relative to real-time). The best connection for real-time purposes may be different from the best connection for throughput purposes. Using connections in different orders allows the load to be split to reduce contention for the first connection, making it more likely to meet real-time streaming requirements. In some embodiments, it may be good to treat some connections as “functionally equivalent,” and the embodiments consider maintaining stickiness of the stream to one of several “functionally equivalent” connections. For example, if stickiness is present, it may be less necessary to switch between connections or reorder packets on the other (receiving) side or put the packets back in order.

[0543] Consider a hybrid router with constrained real-time applications and multiple WAN connections that need to route their packets to meet application-defined limits. Multiple applications with different requirements may run concurrently. The challenge is to achieve high performance while balancing these various constraints. This technical challenge is further complicated when many applications are running and competing for bandwidth, where the diverse needs of all applications must be met. In many cases, some applications will be real-time, some will require throughput, and balancing various trade-offs (e.g., bandwidth, latency, jitter) is technically challenging due to WAN capacity limitations.

[0544] According to some embodiments, an improvement is to group connections into layers, where each connection in a layer is “functionally equivalent” from the perspective of flow preferences.

[0545] Figure 12A This is a schematic diagram of a per-stream communication system according to some example embodiments. The per-stream communication system is...

[0546] In the embodiment shown in Figure 1200A, device 1202 can receive input packet 1204, and then in 1206, sort and prioritize the input packet into a stream based on its requirements. Available connections can then be sorted into connection layer 1208 based on a combination of stream requirements, measured / synthesized connection characteristics, and any predefined rules (e.g., connection priority rules specified by an administrator). As shown, connection layer 1208 includes layers 1, 2, 3 through N. Connection layer 1208 may have variations, and in one variant embodiment, the layers are further subdivided into sub-layers. Other layers, different layers, alternative layers, more layers, and fewer layers are all possible. The input packet 1204 can then be transmitted over the determined connection 1210 corresponding to the appropriate layer and parameters.

[0547] Figure 12B This is a schematic diagram of a per-stream communication system 1200B according to some example embodiments. At layer creation 1212, the connection and request processor 1214 takes into account various requirements, such as, but not limited to, connection monitoring, flow characteristics, packet characteristics, administrator configuration, location, etc., as well as multiple connections.

[0548] In some embodiments, the connection and request processor 1214 can group multiple connections into layers, such as layer 1, layer 2, layer 3, ..., layer N as shown in 1200B. Layer creation 1212 may occur at any point before flow sorting, as shown in... Figure 12A At position 1206. In some embodiments, communication system 1200B may be included in communication system 1200A, for example, layer creation 1212 occurs in stream ordering and packet scheduling 1206.

[0549] For example, during layer creation 1212, connections are established and requests are ordered into the layer by the processor 1214, allowing packet streams to be transmitted. The connection scheduler 1216 can schedule packet transmissions iteratively through layer levels and iteratively through connections ordered into individual layer levels. The packet transmission and scheduling methods will be described further in this paper.

[0550] Functionally, layers can group "functionally equivalent" connections as described above; however, in these embodiments, "functional equivalence" has been extended. From a system perspective, management can be managed using database records or dynamically updated or appended metadata. For example, each connection can have a layer assignment that is tracked and updated via corresponding database values ​​stored in a database table (such as a routing table) in coupled computer memory (such as a non-transitory computer-readable medium) that is referenced or traversed to obtain layer information then used in the connection control methods described herein.

[0551] According to some embodiments, the following definitions can describe "functional equivalence":

[0552] For flows with throughput preferences, layers can be based on the same (Mahiss coefficient, capacity coefficient) composite characteristic pair calculated from raw or rounded / bucketed / quantified measured characteristics. In some embodiments, the composite characteristics may also be rounded / bucketed. In some other embodiments, additional rounding / bucketing / filtering may be introduced onto the measured characteristics and composite characteristics to account for edge cases. For example, without this additional filtering, a connection with a very high loss probability (98+%) and low latency may produce a better Mahis coefficient compared to a connection with a lower loss rate but much higher latency.

[0553] For flows with throughput preferences, one interpretation of grouping / hierarchical connections based on unique (Mathis coefficient, capacity coefficient) pairs is that these coefficients measure the connection's throughput capacity and how efficiently it can utilize that throughput. Recall that the Mathis coefficient predicts throughput based on connection RTT and packet loss rate (a high packet loss rate means the connection will have to retransmit lost packets, and if the RTT is large, retransmission will take longer). Recall that the capacity coefficient is a measure of the transmission rate a connection can achieve per millisecond of its RTT (generally, higher throughput and lower RTT are preferred). The following table describes the characteristics of several example connections:

[0554]

[0555] The "functional equivalence" grouping / layering of these three connections lies in the fact that A and B belong to the same layer because both are reliable, high-performance connections. Neither experiences packet loss (therefore, their throughput can be used efficiently since retransmissions will not be required), and both achieve similar throughput per millisecond of latency. They are "functionally equivalent" because if a throughput-preferred flow is scheduled on either connection, that flow will be able to fully utilize the connection's capacity. C intuitively belongs to a different "functionally equivalent" layer due to its high packet loss rate compared to B. The actual throughput achieved on C will be much lower than on B because many packets will need to be retransmitted.

[0556] For streams with real-time preferences, the layer can be based on packet connections, which are based on the probability that the connection will be able to deliver packets of the stream before the deadline for each packet.

[0557] By comparing the characteristics of available connections with the real-time requirements of the stream, it is possible to observe how and why different connections can be classified as “functionally equivalent” for these types of streams. For example, if a real-time video chat application has a target latency requirement of 150 milliseconds, this means that the video sender expects to deliver each packet in its UDP packets to the video receiver before the age of each packet reaches 150 milliseconds, where the age is measured from the moment the sender initially transmits each packet. This is a common requirement for video chat applications, where the overall round-trip latency must be kept to 300 milliseconds or less to make the chat feel interactive to human participants. The following table describes the characteristics of several example connections:

[0558] connect One-way delay Packet loss rate A 50 milliseconds 0％ B 80 milliseconds 0％ C 50 milliseconds 10％ D 80 milliseconds 5％ E 200 milliseconds 0％ F 300 milliseconds 0％

[0559] The "functional equivalence" grouping / stratification of these six connections lies in the fact that A and B belong to the same layer because both have a one-way latency of less than 150 milliseconds for the application's target, and because they experience a 0% packet loss rate, each packet transmitted will arrive along with its measured one-way latency. From the application's perspective, they are "functionally equivalent" because regardless of which connection is used to transmit its UDP packets, they will arrive before the application's 150-millisecond deadline.

[0560] Similarly, E and F clearly belong to the same "functionally equivalent" layer because, although their one-way latency differs, both have an application objective of greater than 150 milliseconds. Regardless of which one is used to transmit UDP packets, it is impossible for them to arrive in time to meet the deadline.

[0561] Finally, aside from the fact that C and D clearly do not necessarily belong to either of the other two functionally equivalent layers, their place is less intuitively obvious. Their application target is a one-way latency of less than 150 milliseconds, but due to their significant packet loss rate, many transmitted packets will be lost and need to be retransmitted, potentially leading to missed deadlines. The following is a more detailed and formal approach to grouping / layering connections based on the "functional equivalence" of real-time application streams.

[0562] The probability that a connection will be able to deliver packets before its deadline is calculated based on a comprehensive characteristic of packets transmitted on the connection, known as the "Estimated Time of Delivery" (EDT), which is defined as the expected delivery time of a packet after retransmission due to a lost packet.

[0563] Figure 13A This is a diagram 1300A illustrating the calculation of EDT according to some example embodiments. In some embodiments, such as Figure 13AAs shown, this can be calculated as the sum of the probabilities of successful delivery on each retransmission multiplied by the delivery time of that retransmission. Assume a uniform probability P of packet loss on each retransmission. drop 13103. Constant one-way propagation delay T prop 13101, Response timeout T after retransmission of unacknowledged packet rto 13102, this can be like Figure 13A The result is determined as follows:

[0564]

[0565] Other variations are possible.

[0566] In some embodiments, the receiver may be able to quickly identify when a packet was lost and transmit a NACK. Figure 13B The diagram 1300B illustrates the calculation of estimated delivery time according to some example embodiments. Figure 13B A variation is shown here, where the propagation delay T of the ACK or NACK packet is... rprop The duration of 13204 may vary, and the total round-trip propagation delay T of the packet and response is also different. rtprop =T prop +T rprop The duration of ACK or NACK may vary. rdrop 13205 is lost; the probability mentioned is the probability that the packet is lost in the opposite direction. Taking these factors into consideration, T in the equation above... rto You can use P rdrop T rto +(1-P rdrop )T rtprop The substitution makes the EDT calculation as follows:

[0567]

[0568] In some embodiments, a simplified assumption T can be used. prop =T rprop and P drop =P rdrop To estimate EDT.

[0569] In some embodiments, the EDT can be estimated by observing the actual time elapsed from when a packet is first scheduled for delivery on the connection until the packet is successfully delivered; however, this is only applicable to packets specifically retransmitted on a single connection. Variations in EDT derivation are also possible, reflecting either a more complex or simpler approach. For example, in a variation, different P values ​​exist for the forward and reverse directions. drop T prop and T rpropIt may not be symmetrical, and / or ACK packets may be lost.

[0570] Figure 14 This is a schematic diagram 1400 illustrating wide area network connections in different layered packets based on estimated time delivery, according to some example embodiments. In diagram 1400, connections can be grouped into appropriate layers according to the methods described below, given specific packet characteristics.

[0571] For real-time streams (those with latency / jitter preferences, denoted as the nominal "target latency"), the layer can base its latency on the per-packet deadline of the stream (referred to as the adjusted target latency or ATL) and the connection's EDT and T. prop The relationships between characteristics. For example:

[0572] Layer 1: T prop ≤ATL&&EDT≤ATL

[0573] Layer 2: T prop ≤ATL&&EDT>ATL

[0574] Layer 3: T prop ATL & EDT > ATL

[0575] The interpretation of each of these layers is as follows:

[0576] Layer 1: Even considering packet loss and retransmission, the one-way latency of the connection is low enough to meet the deadline.

[0577] Layer 2: Apart from considering packet loss rate and retransmission, the one-way latency of the connection is low enough to meet the deadline.

[0578] Layer 3: The one-way latency of the connection is not low enough to meet the deadline, whether or not there is retransmission.

[0579] ATL can be a dynamic value that changes per packet and as packets age: ATL = Packet Deadline - Current Time, where: Packet Deadline = T PacketReceived +Nominal Target Latency.

[0580] In some embodiments, fewer than three layers may exist, depending on the total number of connections and their relationship to the flow cutoff time. For example, T for all connections prop and P dropBoth are likely to be very small, meaning their overall EDT (Electronic Data Depth) characteristics are also very small. Alternatively, ATL (Average Time Limit) can be very large because real-time applications are not interactive. For example, unlike an interactive Internet Video Protocol (VoIP) call with an ATL < 150 milliseconds, a broadcast video application's ATL might be < 8 seconds. In both scenarios, all connections can be classified as Layer 1 because they are "functionally equivalent" in terms of having an EDT small enough to satisfy the ATL, but may not be equivalent in terms of efficiency.

[0581] Conversely, there may be embodiments where there are more than three layers, where efficiency is considered, for example. Consider two connections where the EDT of both satisfies layer 1(T) prop The required relation for ≤ATL&&EDT≤ATL), but a connection of P drop This is significant, meaning it may require several retransmissions per packet to reach its EDT. If another connection's P... drop If the value is zero, then each packet can reach its EDT with only a single transmission.

[0582] In this example scenario, where efficiency optimization is desired, additional restrictions can be added to layer membership to include, for example, P. drop The measured characteristics are as follows:

[0583] Layer 1: T prop ≤ATL&&EDT≤ATL&&P drop <0.02

[0584] Layer 2: T prop ≤ATL&&EDT≤ATL&&P drop ≥0.02

[0585] Layer 3: T prop ≤ATL&&EDT>ATL&&P drop <0.1

[0586] Layer 4: T prop ≤ATL&&EDT>ATL&&P drop ≥0.1

[0587] Layer 5: T prop >ATL&&EDT>ATL&&P drop <0.35

[0588] Layer 6: T prop >ATL&&EDT>ATL&&P drop ≥0.35

[0589] In some embodiments, layers can be recalculated based on events that occur. For example, if a WAN connection has updated its measured characteristics, its corresponding layer membership can be adjusted based on the new characteristics. Adjustments may include updating the corresponding records or data values ​​in a stored reference table or database table used for routing control.

[0590] For real-time streaming, in some implementations, layer recalculation can be performed on a per-packet basis, since each packet may have a different ATL.

[0591] There may be more than two types of streaming preferences (in addition to throughput and latency), and therefore, there may be more than two hierarchical / ordering methods. In some embodiments, both can be used in combination (for low-latency, high-throughput streaming, such as high-resolution real-time video).

[0592] In some embodiments, tiering may only include a subset of available connections. In one embodiment, this is taken into account because time of day affects the expected usage or demand on some connections. For example, during regular work hours, connections may have a high probability of becoming congested and providing poor performance. Such connections will be excluded from the tiering of mission-critical connections. In another embodiment, administrator configuration is taken into account because there may be factors that are not automatically detectable or measurable and may affect tiering, and must be configured manually. For example, a network administrator may exclude a connection due to regular maintenance by the connection provider, high excess costs, or other administrative reasons.

[0593] In another embodiment, advanced or complex connection characteristics are considered. For example, if it is determined that there is a direct proportional relationship between the throughput and latency of a connection (e.g., the connection can send at very high throughput, but the latency will increase significantly, or the connection can maintain very low latency as long as a certain throughput is not exceeded), then if it is known that this connection is currently serving a high-throughput stream, the connection can be excluded from tiering for real-time streams with strict latency requirements.

[0594] In another embodiment, location is considered due to functional or policy requirements. For example, data residency requirements may specify that certain data can only be transmitted within a network with a designated area.

[0595] Then, connections that can be routed to networks outside the region can be excluded from the layering. In another instance, a service with geo-locking restrictions can only serve requests received from a specific geographic location. Connections known to route / go out through locations outside these locations (e.g., where a concentrator providing the bound connection is installed) can then be excluded from the layering. In another embodiment, security requirements can be considered. For example, unencrypted network connections (such as those not provided via a VPN) can be excluded from the layering.

[0596] In some embodiments, layering can be expanded by intentionally incorporating more connections into one or more layers. This can be achieved in more than one way, including, but not limited to, relaxing layering requirements to allow more connections to qualify for the same layer; combining subsets of the resulting layers after layering is complete; moving connections from one layer to another, etc. In some embodiments, security / obfuscation requirements that may stem from administrator configuration (management configuration) are considered when expanding layers. For example, obfuscation can be implemented by including more connections in the first layer, distributing a user's packets and / or flows across multiple networks, thereby preventing a man-in-the-middle attacker capable of eavesdropping on a single network from intercepting all of the user's packets and / or flows.

[0597] In some embodiments, tiering can be entirely manual, based on administrator policies. In one embodiment, tiering is based on the cost of using connections, grouping connections with similar costs into the same tier and traversing tiers from lowest to highest cost. For example, the administrator can use connection prioritization (also known as priority routing), where low-cost connections are designated as Tier 1, medium-cost connections as Tier 2, and high-cost connections as Tier 3, and configure throughput activation thresholds to justify additional costs when total service drops below a certain bit rate. In another instance, a modified version of connection prioritization can use different types or combinations of activation or deactivation thresholds, such as total usage over the billing period, total cost over the billing period, number of connected users, priority of connected users, etc.

[0598] In some embodiments, determining functionally equivalent WAN connections and the resulting hierarchical / grouping of those connections is based on machine learning or other similar classification pattern recognition techniques. For example, a model is trained using characteristics / features that include measurement / synthetic properties of the connections, where labels are provided directly or indirectly by feedback from applications or users.

[0599] According to some embodiments, when finding connections for scheduling packets on a flow, each layer can be iterated from best to worst. Within a layer, iterations may occur on each connection, but the initial connection for each flow can be selected in a pseudo-random manner (e.g., based on a hash of the flow tuples). The initial connection selected at each layer can remain constant, so if the members of the layer change (e.g., new connections are added or removed), the initial connection can remain unchanged (the flow remains sticky to the first selected connection).

[0600] Figure 15A Examples of pseudo-random hashing and flow viscosity are provided to illustrate some embodiments. Figure 15B It is provided to illustrate, according to some embodiments, sticky expiration after idle time and after TCP FIN exchange. Figure 15C Examples are provided to demonstrate how pseudo-random selection can help keep packets from a single stream on the same connection, and how additional requirements can be met by traversing multiple layers. Other variations are possible.

[0601] Figure 15A Example 1500A shows a new connection D added to a layer that already has connections A, B, and C when the TCP and UDP streams are active.

[0602] Before adding connection D, the TCP stream is pseudo-randomly hashed to the connection at index 3 in the layer (i.e., connection C), and the UDP stream is pseudo-randomly hashed to the connection at index 2 in the layer (i.e., connection B). When connection D is added in 1502, it is placed at index 2, thus replacing connections B and C with indices 3 and 4, respectively. Although the TCP and UDP streams are pseudo-randomly hashed to indices 2 and 3, respectively, the TCP and UDP streams remain sticky to their original starting connections.

[0603] In some embodiments, the system can be configured such that selected connections can eventually be forgotten based on idle time. For example, in the case of a given tier, a connection can be selected as a preferred connection in a pseudo-random manner. This connection can be selected for a stream (e.g., a video call), and packets belonging to the stream can be made to remain sticky to the connection. Sticky connections are useful because using the same connection whenever possible can help improve overall communication characteristics.

[0604] However, the stream may eventually become idle due to, for example, the end of a video call. The system may not have explicit notification that the stream has ended, but it can infer this after a period of time when no packets are seen, at which point sticky connection preferences may have expired / reset.

[0605] Continuing with the above example, Figure 15BExample 1500B shows a UDP stream resetting a sticky connection after an idle period and then remapping it.

[0606] Suppose that both the TCP and UDP streams become idle for more than a minute at point 1520. This causes the UDP stream to be considered terminated, and its sticky connection preference (connection B) is forgotten. At point 1522, both the TCP and UDP streams resume, and the UDP stream is remapped after its expiration. Here, since no FIN packet has been detected, the TCP remains sticky. For example, if the UDP stream restarts, assuming the hash function produces the same index 2, the sticky connection preference is now determined again and becomes connection D at index 2 in the layer at this point.

[0607] In some embodiments, sticky connections can be forgotten based on various factors. For example, some protocols (such as Transmission Control Protocol (TCP)) have explicit stream termination sequences (FIN exchanges). Connections can be forgotten based on the identification of such state transitions.

[0608] Continuing with the above example, Figure 15B An example of a sticky connection is shown where, after a FIN exchange is detected at 1524, the TCP stream is reset and subsequently remapped. It is assumed that the TCP stream has completed and FIN exchanges have occurred on both sides of the connection. This causes the TCP stream to be considered terminated, and its sticky connection preference (connection C) is forgotten. In some embodiments, the TCP connection may then be established again, with a new handshake at 1524, and can be remapped. For example, if the TCP stream restarts, assuming the hash function produces the same index 3, the sticky connection preference is now determined again and becomes connection B, which is now at index 3 in the layer.

[0609] In some embodiments, connections may be intentionally omitted for security / obfuscation purposes. For example, packets may be intentionally split between connections to prevent an attacker eavesdropping on a single connection from capturing all packets associated with the stream.

[0610] Forgetting preferred connections after idle periods can be important to avoid the problem of “excessive stickiness,” in which many long (but bursty) flows eventually become sticky to the same connection and compete for capacity on the same preferred connection, thus requiring the traffic to be split across multiple connections and reassembled at the receiver.

[0611] According to some embodiments, pseudo-random selection of preferred connections addresses various technical challenges for starting iterations on "functionally equivalent" connections. Specifically, a flow is more likely to remain entirely on a single connection because it will not compete with other flows for the same connection in the same order, which is more likely to cause packets to overflow to the next connection. Furthermore, all connections in the active layer will tend to have freshness characteristics because they are constantly being used by different flows.

[0612] Figure 15C An example 1500C is shown where two streams send packets over multiple connections across multiple layers. It is assumed that each connection can only send 6 packets at a time to avoid congestion. In the upper half, each of streams 1 and 2 has 4 packets to send.

[0613] Without pseudo-random selection at 1540, the two flows shown in the example, flow 1 and flow 2, always choose the first connection in layer 1 (connection A) as their starting connection, and the packets of the two flows are eventually split between connection A and connection B. Figure 15C In the lower half, with pseudo-random selection at 1542, each stream can choose a different starting connection.

[0614] For example, if flow 1 chooses connection A and flow 2 chooses connection B: all packets from flow 1 end up on connection A, and all packets from flow 2 end up on connection B. In other words, each flow assigns all its packets to a different connection.

[0615] When the demand for one flow increases and the scheduler needs to iterate through all its traffic across multiple connections / layers, another flow remains unaffected. In this example, the demand for flow 2 increases, resulting in 5 additional packets to send. The flow is iterated over all connections in layer 1, and if all connections are filled with packets, it is moved to layer 2. More specifically, the 5 additional packets are distributed as follows: 2 on connection B in layer 1, 2 on connection A in layer 1, and 1 on connection C in layer 2. It is observed that although flow 2 is split across 3 connections in 2 layers, this does not affect flow 1, which continues to have all its packets on a single connection, namely connection A in layer 1.

[0616] In some embodiments, sticky preference connections are remembered only for the first layer because once more than one layer is used, packets are split across multiple connections by definition, thus requiring reordering of packets at the receiver.

[0617] In some embodiments, if a preferred connection no longer exists in the first layer, the preferred connection can be forgotten (and a new connection can be selected). For example, its measured characteristics and the resulting EDT may lead to its being moved to another layer.

[0618] Pseudo-random selection of preferred connections can be performed in many different ways. In some embodiments, the characteristics of the flow can be hashed (destination IP, port, etc.), or random numbers can be generated, and in both cases, the value modulo the number of connections in the layer becomes the index of the selected connection.

[0619] In some embodiments, selection can be made within a sequence. For example, a first stream is assigned to the first connection in the layer as its sticky connection preference, a second stream is assigned to the second connection, and so on, looping back to the first connection each time the number of connections in the layer is reached.

[0620] In some embodiments, sticky selection can be performed by biasing based on available capacity. For example, selection can be based on the least currently used connections, which can be determined based on factors such as the number of flows that have already selected each connection as their preferred flow or the currently unused capacity of each connection.

[0621] In some embodiments, an estimate of the flow throughput requirement can be used as part of a preference for sticky connection selection, such that the selected connection is the one least likely to cause the flow to overflow to other connections. While connections in a layer are functionally equivalent, they are not identical, so packet overflow to multiple connections can lead to out-of-order packets that need to be reordered at the receiver. The likelihood of this occurring is reduced when the flow uses only a single connection.

[0622] In some embodiments, an initial connection is selected in a layer, and then connections within the layer are iterated in order from shortest backlog to longest backlog (“shortest backlog scheduling”). Backlog is measured in units of time (typically milliseconds) and is defined as the volume of bytes in flight (transmitted but not acknowledged) divided by the connection’s transmission rate. Iterating in this order minimizes the latency experienced by the stream when the stream requires more bandwidth than any single connection can provide on its own.

[0623] For example, consider the following scenario:

[0624] Applications that transmit packets at a rate of 1 packet per 1 millisecond.

[0625] • Two identical WAN connections, where each connection:

[0626] • Capable of transmitting one packet every 2 milliseconds

[0627] • One-way latency is 5 milliseconds

[0628] • CWND (Congestion Window) is 5 packets

[0629] Using sticky connection selection, application-generated packets (one every 1 millisecond) are scheduled to connections C1 and C2 and transmitted as follows:

[0630] • t = 0 milliseconds: At t = 0 milliseconds, Packet 1 is scheduled to be transmitted on C1 (Remaining CWND = 4).

[0631] • t = 1 millisecond: At t = 2 milliseconds, schedule packet 2 to be transmitted on C1 (remaining CWND = 3).

[0632] • t = 2 milliseconds: At t = 4 milliseconds, schedule packet 3 to be transmitted on C1 (remaining CWND = 2).

[0633] • t = 3 milliseconds: At t = 6 milliseconds, schedule packet 4 to be transmitted on C1 (remaining CWND = 1).

[0634] • t = 4 milliseconds: At t = 8 milliseconds, schedule packet 5 to be transmitted on C1 (remaining CWND = 0).

[0635] • t = 5 milliseconds: At t = 5 milliseconds, schedule packet 6 to be transmitted on C2 (remaining CWND = 4).

[0636] The receiver accepts out-of-order packets as follows:

[0637] Packet 1: Received via C1 at t = 5 milliseconds.

[0638] Packet 2: Received via C1 at t = 7 milliseconds.

[0639] Packet 3: Received via C1 at t = 9 milliseconds.

[0640] Packet 6: Received via C2 at t = 10 milliseconds.

[0641] Packet 4: Received via C1 at t = 11 milliseconds.

[0642] Packet 5: Received via C1 at t = 13 milliseconds.

[0643] Once the receiver reorders the packets back into the correct sequence and releases them to the destination, the destination experiences both high jitter (packets are no longer consistently spaced 1 millisecond apart) and overall latency longer than the 5 millisecond one-way latency on each connection:

[0644] Package 1: Released after reordering at t = 5 milliseconds.

[0645] Packet 2: Released after reordering at t=7 milliseconds.

[0646] Packet 3: Released after reordering at t=9 milliseconds.

[0647] Package 4: Released after reordering at t=11 milliseconds.

[0648] Package 5: Released after reordering at t=13 milliseconds.

[0649] Packet 6: Released after reordering at t=13 milliseconds.

[0650] Conversely, connection selection and packet scheduling based on minimizing backlog will result in packets being scheduled to C1 and C2, and transmitted as follows:

[0651] • t = 0 milliseconds: At t = 0 milliseconds, schedule packet 1 to be transmitted on C1 (remaining CWND = 4).

[0652] • t = 1 millisecond: At t = 1 millisecond, schedule packet 2 to be transmitted on C2 (remaining CWND = 4).

[0653] • t = 2 milliseconds: At t = 2 milliseconds, schedule packet 3 to be transmitted on C1 (remaining CWND = 3).

[0654] • t = 3 milliseconds: At t = 3 milliseconds, schedule packet 4 to be transmitted on C2 (remaining CWND = 3).

[0655] • t = 4 milliseconds: At t = 4 milliseconds, schedule packet 5 to be transmitted on C1 (remaining CWND = 2).

[0656] • t = 5 milliseconds: At t = 5 milliseconds, schedule packet 6 to be transmitted on C2 (remaining CWND = 2).

[0657] The receiver receives packets in sequence without additional jitter (the interval between packets remains consistent at 1 millisecond), and the latency experienced by the application is equal to the 5 millisecond one-way latency on each connection.

[0658] Packet 1: Received and released via C1 at t = 5 milliseconds.

[0659] Packet 2: Received and released via C2 at t = 6 milliseconds.

[0660] Packet 3: Received and released via C1 at t = 7 milliseconds.

[0661] Packet 6: Received and released via C2 at t = 8 milliseconds.

[0662] Packet 4: Received and released via C1 at t = 9 milliseconds.

[0663] Packet 5: Received and released via C2 at t = 10 milliseconds.

[0664] In some embodiments, shortest backlog scheduling is instead simplified to polling between each connection in the layer. This approach is easier to implement and results in similar performance when the connections are known to have the same characteristics.

[0665] In some embodiments, an administrator can specify connection priority (CP) rules or priority routing to express a preference for the order in which connections are used. In previous solutions, these rules were global, thus applying to all flows, and also had the ability to limit bandwidth usage on low-priority connections. Consider the following example set of connections with CP configuration:

[0666] • Connection A: Level 1

[0667] • Connection B: Level 2 / Threshold = 10Mbps

[0668] According to these previous embodiments, unless the capacity of connection A drops below 10 Mbps, connection B will not become active, and once active, connection B will only contribute the difference to restore the total capacity to 10 Mbps.

[0669] According to some current embodiments, the granularity of CP rules can be the same as that of a single flow. Limiting bandwidth usage for lower-priority flows may also be removed, since a high priority for one flow might be a low priority for another, and attempting to limit bandwidth usage would conflict with the requirements of both flows. For example, suppose flow 1 has the same CP rules as described in the previous example, but flow 2 has the opposite CP rules:

[0670] • Connection B: Level 1

[0671] • Connection A: Level 2 / Threshold = 10Mbps

[0672] Flow 1 can consume the full capacity of connection A because A is its first-level node, and Flow 2 can consume the full capacity of connection B because B is its first-level node. Neither can be restricted as much as they could be if they used global CP rules.

[0673] In some embodiments, for example, the bandwidth limiting function is a global setting that limits the bandwidth usage of each WAN connection (regardless of flow or CP rules) or a bandwidth limiting replacement applied to matching flows.

[0674] CP rules can affect layer construction because if two connections are at different CP levels, they may be placed in different layers even if they are "functionally equivalent" considering all other characteristics. For example, consider the following:

[0675]

[0676]

[0677] Given a packet with an ATL of 100 milliseconds, in the absence of connection priority rules, the layer structure will be as follows:

[0678] Layer 1: A, B(T) prop ≤ATL&&EDT≤ATL)

[0679] Layer 2: C, D(T) prop ≤ATL&&EDT>ATL)

[0680] Layer 3: E, F(T) prop >ATL&&EDT>ATL)

[0681] However, using connection priority rules, even if T prop The EDT feature still satisfies the constraints, but each change in priority results in the creation of a new layer:

[0682] Layer 1: A(T) prop ≤ATL&&EDT≤ATL&&CP=Level 1)

[0683] Layer 2: C(T) prop ≤ATL&&EDT>ATL&&CP=Level 1)

[0684] Layer 3: E(T) prop >ATL&&EDT>ATL&&CP = Level 1)

[0685] Layer 4: B(T) prop ≤ATL&&EDT≤ATL&&CP=Level 2)

[0686] Layer 5: D(T) prop ≤ATL&&EDT>ATL&&CP=Level 2)

[0687] Layer 6: F(T) prop >ATL&&EDT>ATL&&CP = Level 2)

[0688] Additionally, if one connection is "functionally better" than another, it may be placed in a less preferred layer because it belongs to a lower priority CP level. For example, although connection B has a lower T... prop And EDT value, but due to the connection priority rules, the above connection E is placed in layer 3, while connection B is placed in layer 4.

[0689] In some embodiments, scheduling methods are introduced to help ensure that packets belonging to a real-time stream arrive before their deadline, but this needs to be balanced with bandwidth usage (e.g., an early implementation might simply broadcast real-time packets on all available connections).

[0690] In some embodiments, real-time packets can be scheduled across multiple connections to maximize the probability of the packets arriving before the deadline while balancing bandwidth usage. As a non-limiting example, according to one embodiment:

[0691] For example, assigning scores to packet transmissions at each layer:

[0692] Level 1: Rating = 3

[0693] Level 2: Rating = 2

[0694] Level 3: Rating = 1

[0695] Packets can be scheduled across multiple connections as needed to ensure that their cumulative score is greater than or equal to a predefined target score. For example, the target value could be a parameter such as 3.

[0696] Connections lacking fresh statistics or unreliable (with a near 100% loss probability) may contribute a score of 0 regardless of their tier. This can have the intentional side effect of refreshing statistics on these connections if / when packets are acknowledged.

[0697] Figure 16 An example 1600 is shown with various connections 1602 sorted into various layers 1604 for sending a large number of packets 1606. Similar to the above, layer 1 connections have a score of 3, layer 2 connections have a score of 2, and layer 3 connections have a score of 1. Unreliable connections or connections with outdated statistics, regardless of which layer they belong to, have a score of 0. For each input packet, the connections are iterated from layer 1 to layer 3 (in other words, from connection A to F). Each packet is sent over the connections it traverses, and its score is accumulated until a target score of 3 is reached or exceeded. This produces the following exemplary results:

[0698] Packets 1 through 4 are sent on the first connection (connection A) and the target score is reached immediately. At this point, the capacity of connection A is completely consumed.

[0699] After being sent over three connections, packet 5 achieved the target score: Layer 1: Connection B, Layer 2: Connection C, and Layer 2: Connection D. It accumulated scores of 0, 2, and 2 respectively, resulting in a total score of 4, exceeding the target score of 3. Note that Layer 1: Connection B received a score of 0 because it is unreliable. At this point, the capacity of Connection D was completely exhausted.

[0700] After being sent over four connections, packet 6 achieved the target score: Layer 1: Connection B, Layer 2: Connection C, Layer 2: Connection E, and Layer 3: Connection F. It accumulated scores of 0, 2, 0, and 1 respectively, resulting in a total score of 3, satisfying the target score. Note that Layer 2: Connection E received a score of 0 because its statistics are not recent. At this point, the capacity of Layer 1: Connection B and Layer 3: Connection F was completely exhausted.

[0701] Even after being sent on two connections, packet 7 did not reach the target score: 2 for connection C and 0 for layer 2 connection E. It accumulated scores of 2 and 0 respectively, resulting in a total score of 2, failing to reach the target score of 3. At this point, the capacity of layer 2 connection C and layer 2 connection E was completely exhausted. Since the capacity of all connections was also completely exhausted, all transmission ceased. Unless packet 7 is among those confirmed as received, it will continue to be sent in the next round of transmission when more connection capacity becomes available (e.g., the packet is confirmed as received or a new connection has been added).

[0702] Packet 8 did not reach the target score because all connection capacity was fully utilized and the packet was not sent on any connection. Packet 8 may be sent in the next round of transmission when more connection capacity becomes available.

[0703] Some implementations may have different predefined target values ​​on a per-flow basis. For example, a very high-priority flow that wants to broadcast its packets on all connections, or other methods that allow administrators to control the trade-off between bandwidth usage and reaching deadlines.

[0704] In some embodiments, to achieve the target score, the CP rule can be ignored, assuming that the user will reach their preference by the deadline, even if activating lower priorities incurs additional costs. For example, if the cumulative score does not reach the target after iterating through all connections in all higher priority layers (considered active by the CP rule), iteration can continue through connections in lower priority layers (considered inactive by the CP rule).

[0705] In some embodiments, preemptive scheduling can take into account predictions of impending location changes (e.g., buses, airplanes, etc., moving along predictable paths, whose measured characteristics are subject to predictable changes). For example, a current Layer 1 connection may soon become a Layer 3 connection, and vice versa; therefore, in some embodiments, target scores can be adjusted in advance, and more redundancy can be transmitted as the transition approaches.

[0706] Compared to existing solutions, the described embodiment presents a technical improvement because preemptive retransmission is only applied to streams with deadlines (real-time applications), and when deciding whether preemptive retransmission is needed, both the stream deadline and the connection EDT can be considered, rather than just the probability of connection loss. Throughput streams without deadlines or other similar types will not incur bandwidth costs or the overhead of preemptive retransmission.

[0707] In other embodiments, there may be preemptive retransmissions of packets that are approaching their deadline but are still in flight and have not yet been acknowledged. Such embodiments can balance meeting deadlines with effective connection usage (cost—dollars, bandwidth, computation). In some embodiments, there may be per-flow rules that allow administrators to influence this balance. For example, if the flow is so critical that preemptive retransmissions of packets can be broadcast on all connections regardless of any potential additional costs, then "cost" will be irrelevant. As a non-limiting example:

[0708] In one embodiment, if a packet has a deadline and has not yet been acknowledged as received, and a reliable connection with minimal latency, available capacity, and fresh statistics has just enough time for the packet to cross over to another time (while also considering some variations, such as those for handling overhead time), then an early retransmission of the packet is sent on this connection. This is done as a last resort to ensure the packet crosses over in time before its deadline. Figure 17 Example 1700 of such an embodiment (Example 1) is shown, wherein an unacknowledged packet originally sent on connection A at t = 0 milliseconds with a cutoff time of 150 milliseconds has an early retransmission at t = 150 - 50 - 15 = 85 milliseconds on connection A, where 50 milliseconds is an assumed time difference and 15 milliseconds is a known delay of connection A.

[0709] In one embodiment, the time difference is a fixed amount that can account for some variation of processing time and actual connection latency, such as 50 milliseconds.

[0710] In another embodiment, the time difference is a variable that is a function of the connection type or a measured characteristic. For example, if the round-trip time of a connection has a statistically large standard deviation, the time difference for early retransmission may increase. In another instance, satellite connections are known to have relatively stable processing times compared to, for example, cellular connections, and therefore may therefore be allocated a correspondingly lower time difference when considering early retransmission.

[0711] In one embodiment, if the packet has already been sent using a reliable connection with the lowest latency, available capacity, and fresh statistics, an earlier retransmission of the packet will be sent using another reliable connection with sufficiently low latency, fresh statistics, and available capacity to allow the packet to be transmitted before its deadline. This is done to increase the diversity of connections sending the same packet over, and because the first selected connection is considered potentially unreliable since it has not yet delivered a copy of its packet. Figure 17 Example 1700 of such an embodiment (Example 2) is shown, in which connection A is avoided because it has already been used to send packets originally, and an early retransmission of packets is performed on connection D at time t = 150 - 50 - 20 = 80 milliseconds, where 50 milliseconds is the assumed time difference and 20 milliseconds is the known delay of connection D.

[0712] In one embodiment, unacknowledged packets that are approaching their deadlines are retransmitted early on some or all unreliable connections that lack fresh statistics or both. Such connections may have become reliable or now possess more advantageous characteristics (e.g., they may have become less congested due to not being used for some time) and therefore may be able to allow packets to traverse before their deadlines. The number of such connections used for early retransmission can depend on a balance between meeting deadlines and effectively utilizing the previously mentioned network connections. Figure 17 An example 1700 of such an embodiment (Example 3) is shown, in which early retransmissions are sent on all unreliable and / or outdated connections. The calculation of each transmission time is similar to the previous embodiment in this figure. For connection F, the past time is calculated (t = 150 - 50 - 150 = -50 milliseconds); therefore, the early retransmission on connection F is sent simultaneously with the earliest early retransmission, i.e., the early retransmission on connection C in this example (t = 75 milliseconds).

[0713] In one embodiment, one or more of the above-described early retransmission methods are combined. For example, when a service should meet the deadlines of a highly critical stream at any cost, an embodiment may combine all of the above methods. Thus, early retransmissions are sent on: the original connection on which packets are sent; other reliable connections with appropriate latency and EDT, available capacity, and fresh statistics; and all other connections that are unreliable or have outdated statistics. Figure 17 Example 1700 of such an embodiment (Example 4) is shown, which combines the advance retransmission of Examples 2 and 3.

[0714] In one embodiment, the above-described advance retransmission method is considered sequentially (in any order) until at least one or more connections for sending the current packet are found to advance retransmission.

[0715] Early retransmission may ignore CP rules, assuming that early retransmission is the last chance to reach the packet deadline, therefore all available connections must be considered.

[0716] In some embodiments, a modified scheduling method is introduced for packets belonging to flows with throughput preferences, assuming that the method has a higher tolerance for latency / jitter / loss.

[0717] According to one embodiment, apart from a contribution score of 3 for each layer (excluding outdated or unreliable connections that still have a contribution score of 0), the scheduling algorithm is similar to the description above for real-time streaming. Therefore, packets belonging to the throughput stream may only be transmitted once (excluding outdated or unreliable connections).

[0718] Figure 18 This is a diagram illustrating, according to some example embodiments, an instance 1800 of multiple connections 1802 arranged in multiple layers 1804 for sending a throughput stream of many packets 1806. Figure 18 It shows Figure 16 This is a variant of the instance in the example, but this time it's a throughput stream. Layer 1804 and connection 1802 are of the same type, except that all layers 1804 now have a score of 3. Unreliable connections or connections 1802 with outdated statistics, regardless of which layer they belong to, continue to have a score of 0. This produces the following exemplary results:

[0719] Packets 1 through 4 are sent on the first connection (connection A) and the target score is reached immediately. At this point, the capacity of connection A is completely consumed.

[0720] After being sent over two connections, packets 5 and 6 reached the target score: Layer 1: Connection B and Layer 2: Connection C. They accumulated scores of 0 and 3 respectively, for a total score of 3, achieving the target score of 3. Note that Layer 1: Connection B scored 0 because it is unreliable. At this point, Connection B's capacity was completely exhausted.

[0721] • After being sent on connection C at layer 2, packet 7 immediately reaches the target score. At this point, the capacity of connection C is completely consumed.

[0722] • After being sent on connection D at layer 2, packet 8 immediately reaches the target score. At this point, the capacity of connection D is completely consumed.

[0723] After being sent on two connections, packet 9 reaches the target score: 2: connection E and 3: connection F. It accumulates scores of 0 and 3 respectively, for a total score of 3, thus achieving the target score of 3. Note that layer 2: connection E has a score of 0 because its statistics are not recent. At this point, the capacity of connection F is completely exhausted.

[0724] Even after being sent on one connection, packet 10 does not reach the target score: 2: connection E. It accumulates a score of 0, failing to reach the target score of 3. At this point, the capacity of layer 2: connection E is completely exhausted. At this point, the capacity of all connections has also been completely exhausted, therefore all transmissions cease. Unless packet 10 is among those acknowledged as received, it will continue to be sent in the next round of transmissions when more connection capacity becomes available (e.g., the packet is acknowledged as received or a new connection has been added).

[0725] In some embodiments, prioritization between flows can be implemented at a stage prior to packet scheduling (input queue). A packet queue can be created for each flow entering the system, and queues are served according to a differential round-robin (DRR) and / or differential weighted round-robin (DWRR) queue scheduler algorithm (described in RFC 8290), which by default attempts to be fair to all queues. The embodiments introduce technical improvements as a means to allow administrators to configure additional modifications that could be intentionally unfair.

[0726] DRR / DWRR incorporates the concept of a quantum (i.e., the number of target bytes served from each queue in each round). Typically, this quantum is equal for all queues, but this can be modified via weights (specified as rules by the administrator). In some embodiments, weights can be specified in other ways, such as automatically determined based on application type, time of day, machine learning, etc.

[0727] For example:

[0728] Stream #1: Weight = 2

[0729] Stream #2: Weight = 5

[0730] Stream #3: Weight = 1

[0731] In this example, stream 1 receives twice the nominal quantum per round, stream 2 receives five times the nominal quantum, and stream 3 receives one times the nominal quantum. In some embodiments, the quantum can be scaled, for example, by a factor, while preserving relative weights. For example, nominal quantum * k, multiplied by 2 / 8, 5 / 8, 1 / 8 respectively, where nominal quantum and k are parameters. In some embodiments, weights can be specified for rules, and all streams matching the rules share a common quantum pool.

[0732] In the above example, k will be k=1, and the nominal quantum has a default value, which can be manually specified by the administrator or automatically specified based on application type, time of day, machine learning, etc.

[0733] In some embodiments, the concept of queue importance levels is introduced. Queues with higher importance are typically served before those with lower importance. In some embodiments, lower importance queues may be starved if there is only sufficient capacity to serve high-importance queues. Queues with the same importance level can have their capacity split according to their assigned weights.

[0734] This embodiment improves upon existing DRR / DWRR because it describes a multi-level DRR / DWRR priority queue. The characteristic of conventional DRR / DWRR queues is that each stream "i" will reach a minimum long-term average data rate.

[0735]

[0736] Q i R is the quantum of the flow “i”, and R is the total available WAN capacity.

[0737] In some embodiments of the multi-level DRR / DWRR queue implementation, flows predefined as having higher “importance” are allowed to starve flows with lower “importance” such that a given flow “i” in priority level “M” will reach the minimum long-term average data rate.

[0738]

[0739] Q i It is the quantum of flow "i" in N flows of priority level M, where priority levels 0 to M-1 are all higher than priority M, and R j It is the WAN capacity used by all flows at a specific priority level.

[0740] Figure 19An example of an embodiment 1900 of a two-level DWRR queue management algorithm according to some example embodiments is shown. DWRR level 1 is more important than DWRR level 2. DWRR level 1 has 3 flows with weights as described above, i.e., flows 1, 2, and 3 have weights of 2, 5, and 1, respectively. DWRR level 2 has a single flow 4 with a weight of 2. Assume there are 3 connections in this example, with a total capacity of 16 packets per second. Assume the packet rates of the flows are as follows: flow 1 produces 4 packets per second, flow 2 produces 10 packets per second, flow 3 produces 3 packets per second, and flow 4 produces 3 packets per second. Since DWRR level 1 is more important than DWRR level 2, it will always be served before DWRR level 2 as long as DWRR level 1 has available packets from any of its flows. In this example, the flows of DWRR level 1 produce a total of 17 packets per second, which is higher than the total capacity of the connections; in other words, DWRR level 1 will always have available packets, causing DWRR level 2 to become starved and never be served.

[0741] Figure 19 Table 1902 at the bottom shows the number of packets that each stream sees at the input and output over time.

[0742] Streams 1 and 2 in DWRR level 1 are of high importance and produce packets proportional to their quantum number; therefore, the number of output packets always keeps up with the number of input packets.

[0743] Stream 3 in DWRR level 1 is of higher importance and is served; however, it creates a backlog of packets because it produces more packets than its quantum share, which is considered to be the case that the number of input packets grows faster than the number of output packets.

[0744] Flow 4 in DWRR level 2 is of low importance and is not served because the packets produced by the higher-important DWRR level 1 are sufficient to consume the full capacity of the available connection. Flow 4 is starved, and this is considered to be a situation where the number of input packets is constantly increasing while the number of output packets remains at 0. If Flow 4 were in DWRR level 1, some of its packets would be served before the DWRR returns to serve Flow 1, meaning that Flow 4 would not be completely starved.

[0745] This embodiment introduces technical improvements to the sequencer described above for more efficient operation on a per-stream basis. The sequencer can operate using the following measurement characteristics:

[0746] a) Measure the one-way latency of each WAN connection using a statistical filtering method.

[0747] b) The set of WAN connections that a specific flow is actively using.

[0748] Figure 20 This is a block diagram of a per-flow packet receiver 2000 according to some example embodiments.

[0749] At 2001, the receiver of each flow maintains an estimate of the one-way delay of the WAN connection, and at 2002, it maintains an estimate of the set of WAN connections most recently used for a particular flow. These can be used to determine the appropriate time to release packets from the flow output queue at 2003 to minimize jitter in the propagation time of packets in the flow within the system.

[0750] The basic implementation measures and stores these two characteristics on a per-flow basis. The embodiment described above provides an improvement on the observation in (a) above, namely that the statistically filtered measurement of the one-way delay for each WAN connection does not change on a per-flow basis, thus allowing the estimation of the one-way delay for 2001 WAN connections to be shared across all flows.

[0751] In some per-flow packet receiver embodiments, the one-way delay estimator can accept conditioning parameters to adjust the quality of the estimate based on the expected characteristics of the flow or WAN connection. Conditioning parameters may include low-pass filter parameters and the duration after a single packet measurement is lost. In this embodiment, there may be more than one estimate of the one-way delay per WAN connection, and these estimates can be shared among all flows using the same conditioning parameters.

[0752] In some embodiments, the estimated one-way delay of the WAN connection at 2001 can be used to calculate the EDT of the WAN connection.

[0753] The aforementioned embodiments, including improvements to layering, scheduling, and sequencing (collectively referred to as "real-time streaming support"), have been tested in simulated scenarios mimicking real-world usage patterns and have demonstrated quantitative improvements in jitter and latency. Specifically, a common scenario is the use of real-time video chat applications, such as Zoom, through a multi-WAN router containing both wired broadband connections (e.g., consumer cable or DSL) and 3xLTE connectivity. TM Microsoft Teams TM Or Google Meet TM Connection Priority (CP) rules are used to set wired broadband connections as Level 1 and LTE connections as Level 2.

[0754] Then, a scenario simulating a sudden failure of a wired broadband link (e.g., 100% packet loss and / or zero bandwidth capacity) was performed, and the resulting impact on packet loss rate, latency, and jitter was measured. The results are as follows:

[0755] Example Packet loss rate Jitter (average) Jitter (maximum value) Latency (average) Delay (maximum value) Real-time streaming support 0％ 797 milliseconds 1158 milliseconds 40.7 milliseconds 62 milliseconds No real-time streaming support 0％ 986 milliseconds 2070 milliseconds 51.1 milliseconds 351.2 milliseconds

[0756] The improvements in jitter and latency stem from preemptive retransmission (target scoring system) and early retransmission logic. In this scenario, when the wired broadband connection suddenly fails, in-flight packets lost on the wired broadband connection are preemptively or early retransmitted on the LTE connection. In contrast, embodiments without real-time streaming support must wait for the full RTO (Retransmission Timeout) to occur before retransmitting lost packets.

[0757] Figure 21 This is a block diagram of the workflow 2100 of a per-stream communication system according to some example embodiments. For a user connecting a smartphone to a multi-LTE WAN router (such as one described above) via WiFi, the experience may be similar to making a video call via a wired broadband connection.

[0758] According to one embodiment, the per-stream communication system workflow 2100 may follow such a process as outlined below, but in other embodiments, it may include more or fewer steps and may be performed in a different order.

[0759] In the illustrated exemplary workflow 2100, starting at 2110, a target score is assigned to the packet based on the requirements of the flow to which it belongs. Then, in 2111, the connection can be layered based on the flow requirements. For example, for real-time streaming, the per-packet deadline ATL and connection characteristics (such as EDT and T) can be used. prop This is used to build layers. For throughput streams, layers can be built based on unique Mathis and capacity coefficient pairs.

[0760] Next, at 2112, a score can be assigned to the packet transmissions occurring on the connections at each layer. Connections without fresh statistics (outdated connections) or unreliable connections (connections with a near 100% loss probability) contribute a score of 0 regardless of the layer. In 2113, the packet scheduling process begins, iterating through each layer from best to worst, and iterating through each connection in each layer, transmitting packets on multiple connections as needed, to achieve a cumulative score greater than or equal to the predefined target score previously assigned in 2110.

[0761] Then, in step 2120, a connection within the currently selected layer can be chosen in a pseudo-random manner. If the current layer is also the highest (optimal) layer, the selected connection will be remembered as a sticky-preference connection. If this connection still exists in the optimal layer, future packets belonging to this flow will always start from that connection (instead of choosing a connection in a pseudo-random manner).

[0762] Next, packets can be transmitted at 2121, and the cumulative score can be updated. Next, it can be determined at 2122 whether the target score has been reached. If not, it is checked at 2123 whether any more available connections have occurred. If so, at 2124, iteration may occur in the next connection in the current layer, thus exhausting all available connections in the current layer before moving to the next layer.

[0763] If the connection in the current layer is still available at 2125, the process can continue from 2121, transferring over the next connection in the layer and updating the score accordingly. If all connections in a layer are exhausted, it can iterate to the next layer, and the process can continue at 2120, where connections are selected in a pseudo-random manner in this new layer.

[0764] Once the target score has been reached at 2122, it can be determined at 2126 whether there are any remaining packets to be transmitted. If so, the process will restart at 2110 for the next packet. If all available packets have been transmitted at 2126, packet scheduling pauses at 2130 until more packets are available for transmission.

[0765] At 2122, the packet may not have yet reached its target score, but at 2123, there are no available connections at any layer. The lack of availability could be due to factors such as all connections having consumed their available capacity (in-flight packets exceeding or equaling their congestion window), or connection priority rules preventing some connections from being activated. If this is the case, packet scheduling pauses at 2130 until an event occurs, such as connection capacity becoming available due to in-flight packet acknowledgment. This allows the packet to resume scheduling and transmission at 2110, but using a portion of its accumulated score.

[0766] As a non-limiting example implementation that can follow the workflow shown in 2100, consider a multi-LTE WAN router comprising X connections, where connection 1 is equivalent to the old single LTE connection. Given this, connections 2 through X now exist. A user can initiate a voice call with a latency / jitter sensitive flow (packet 1, packet 2, packet N). The traffic itself may also have given latency / jitter requirements. Consider a maximum latency of 150 milliseconds, a maximum jitter of 30 milliseconds, and a maximum packet loss rate of 1%.

[0767] In some embodiments, a flow may consist of packets with various latency / jitter requirements, but similar to standard routers, the flow can be identified and separated based on fields in the standard IP / TCP / UDP header ("SrcIP, SrcPort, DstIP, DstPort, Protocol, DSCP" tuple). Typically, applications are aware of this standard router feature, and therefore, if they have different latency / jitter requirements for different packets, they will create different flows for them.

[0768] Now, consider the initial starting assumptions of the example implementation through exemplary workflow 2100. Recall that, as described above, according to some embodiments, a score can be assigned to packet transmissions on each layer. In this example, layer 1 gives a score of 3, layer 2 gives a score of 2, and layer 3 gives a score of 1. Regardless of the layer, connections without fresh statistics or unreliable connections contribute a score of 0. Packets can then be scheduled on multiple connections as needed so that the cumulative packet score is greater than or equal to a predefined target score. In this example, consider a target score of 3. It is also assumed that the application's nominal target latency requirement is 150 milliseconds, and the initial characteristics of the available connections are as follows:

[0769]

[0770]

[0771] Next, consider packet 1 and its layers. Recall that, as mentioned above, according to some embodiments, layers are based on the per-packet deadline (referred to as ATL) of the flow and the EDT and T of the connection. prop The relationship between characteristics. Given a nominal target latency of 150 milliseconds, and assuming packet 1 ATL is also 150 milliseconds, the resulting layers are as follows:

[0772] Package 1:

[0773] Layer 1: Connects A, B, and C

[0774] ii. Layer 2: F, G

[0775] iii. Layer 3: J, K

[0776] iv.Layer 4:D,E

[0777] v. Layer 5: H, I

[0778] vi. Level 6: L, M

[0779] As described above, iteration can begin at layer 1. In this example, assume the first transmission of this flow is chosen pseudo-randomly as sticky to connection B. An attempt is made to transmit on B, but it currently has no available capacity. Iterating through the layers to connection C, a transmission is made once, but it is outdated, so its contribution to the goal is 0. Iterating to connection A, a transmission is made once, and A is fresh, contributing 3 to the goal.

[0780] Next, consider packet 2, where the layers are the same, but the characteristics of B have changed, making it now have some usable capacity. B exhibits a sticky connection preference, so packet 2 is transmitted once on B, which is still fresh, and its contribution to the target is 3.

[0781] For packet 3, assume connections B and C become unavailable (100% packet loss). B and C are moved to layer 2 (EDT becomes infinity, T...). prop (Still the last measured value), and it is marked as obsolete. B's sticky connection preference is no longer in layer 1, so it is forgotten. It is transferred once on only the retained layer 1 connection A (fresh, with a contribution of 3 to the goal), and A is selected as the new sticky connection preference.

[0782] Package 3:

[0783] vii.Layer 1:A

[0784] viii. Layer 2: B, C, F, G

[0785] ix. Layer 3: J, K

[0786] Layer 4: D, E

[0787] xi.Layer 5:H,I

[0788] xii. Layer 6: L, M

[0789] For packet 4, assume connection A becomes unavailable (100% packet loss). A is moved to layer 2 (EDT becomes infinity, T...). prop (Still the last measured value), and is marked as obsolete. Layer 1 is empty, so the sticky connection preference is completely forgotten, and iteration begins from Layer 2. Assuming that B is selected from Layer 2 in a pseudo-random manner as the initial connection to start the iteration, packet 2 is transmitted once on B, which is obsolete and therefore its contribution to the goal is 0.

[0790] Package 4:

[0791] xiii. Layer 1:

[0792] xiv. Layer 2: A, B, C, F, G

[0793] xv. Layer 3: J, K

[0794] xvi. Level 4: D, E

[0795] xvii. Layer 5: H, I

[0796] xviii. Layer 6: L, M

[0797] Iterate to the next connection C and transmit once (outdated, contributing 0 to the target).

[0798] Iterate to the next connection F, and transmit once (fresh, contributing 2 to the target).

[0799] Iterate to the next connection G, but there is no available capacity, so it is skipped.

[0800] Iterate to the next connection A and transmit once (outdated, contributing 0 to the target).

[0801] Now, layer 2 is exhausted and the goal has not yet been reached, so we iterate to layer 3. For layers greater than 1, there is no sticky connection preference, so we assume a pseudo-random starting choice for J. We transmit packet 4 (outdated, contributing 0 to the goal) once on J.

[0802] Iterate to the next connection K and transmit once (fresh, contributing 1 to the goal, resulting in a cumulative score of 3).

[0803] For the final packet 5, assume connection K becomes unavailable (100% packet loss). K remains in layer 3 (EDT becomes infinity, T...). prop The final measured value (the target latency is still >150 milliseconds) is marked as obsolete.

[0804] Package 5:

[0805] xix. Layer 1:

[0806] Floor 2: A, B, C, F, G

[0807] xxi. Layer 3: J, K

[0808] xxii.Layer 4:D,E

[0809] xxiii. Layer 5: H, I

[0810] xxiv. Layer 6: L, M

[0811] It is assumed that all stale connections previously transmitted for packet 4 are still stale. Normally, this is not actually the case, because those stale transmissions that occurred with packet 4 would refresh connection statistics, allowing updated characteristics to be measured on each of the stale connections. However, this example will assume this to demonstrate how to achieve the goal using secondary priority.

[0812] Repeating all the same iterations in package 4, the inventors found that layer 3 had been exhausted and the target score still had not been reached. Therefore, the iteration proceeded to layer 4. Layer 4 does not have sticky connection preferences, so a pseudo-random initial choice of E is assumed.

[0813] Transmit once on E (obsolete, contributing 0 to the target).

[0814] Iterate to the next connection D, transmitting once (outdated, contributing 0 to the goal). Then, layer 4 is exhausted and the goal is still not reached, so iterate to layer 5. There is no sticky connection preference in layer 5, so a pseudo-random initial selection of H is assumed. Transmit once on H (fresh, contributing 2 to the goal, with a cumulative score of 4).

[0815] Assuming no more packets are available, scheduling and transmission are paused until an event occurs (e.g., a new packet is received at the input, a negative acknowledgment of a transmitted packet is received, etc.).

[0816] Furthermore, considering all the aforementioned stale and preemptive packet transmissions, there may be many duplicate and out-of-order packets. In some embodiments, reordering and deduplication may occur, and endpoints may receive packets in sequence.

[0817] According to one embodiment, the implementation of the per-stream communication system may follow such a process outlined above, but in other embodiments, it may include more or fewer steps and may be performed in a different order.

[0818] Consider another non-limiting example implementation for a stream with throughput maximization requirements (as opposed to the real-time stream in the previous example), which can follow the exemplary workflow 2100. Recall that, as described above, according to some embodiments, a score can be assigned to packet transmissions on each layer. In this example, the throughput stream is configured across all layers to contribute a score of 3. Connections without fresh statistics or unreliable connections contribute a score of 0, regardless of the layer.

[0819] In this example, the target score for throughput flow is also 3, meaning that a single packet transmission on any fresh and reliable connection is sufficient to achieve the target. The threshold for activating secondary connections is also configured to 10 Mbps, meaning that the total capacity of all primary connections must drop below 10 Mbps before secondary connections become available for transmission.

[0820] The initial state of available connections is assumed to be as follows:

[0821]

[0822] Next, consider the transmission of packet 1 and its layering. Recall that, as mentioned above, according to some embodiments, the layering of the throughput flow is based on a unique tuple of a rounded Mathis coefficient, a rounded capacity coefficient, and a connection priority. Given that smaller rounded Mathis coefficient values ​​are better, and larger rounded capacity coefficient values ​​are better, this results in the following layers:

[0823] Package 1:

[0824] Layer 1: Connects A and E

[0825] Layer 2: B

[0826] Floor 3: F

[0827] Layer 4: C, D

[0828] As mentioned above, iteration can begin at layer 1. In this example, assume the first transmission of this flow is chosen pseudo-randomly to be sticky to connection A. A transmission occurs on A, but it is outdated, and therefore its contribution to the goal is 0.

[0829] Now, layer 1 is exhausted and the goal has not yet been reached, so we iterate to layer 2. Layers greater than 1 do not have a sticky connection preference, and there is only one connection in this layer, so pseudo-random selection provides option B by default. Capacity is available, so a transfer occurs. The connection is also fresh, so its contribution to the cumulative score is 3, thus reaching the desired goal.

[0830] Next, consider the transmission of packet 2, which belongs to the same class. Except that the transmission of packet 1 updated the measured characteristics of A, making it no longer obsolete, all connectivity characteristics are assumed to remain the same as packet 1. The flow has a sticky connectivity preference for A, therefore packet 2 is transmitted over it, thus achieving the target score 3.

[0831] Next, consider the transmission of packet 3, which belongs to the same class. Assume that connection A has now become unreliable and obsolete, causing its rounded Mathis coefficient to increase to a maximum of 5,000.

[0832]

[0833]

[0834] Connection A now has a unique tuple, thus causing it to be placed in its own layer:

[0835] Package 3:

[0836] Layer 1: Connecting E

[0837] Layer 2: B

[0838] Layer 3:A

[0839] Floor 4: F

[0840] Layer 5: C, D

[0841] The iteration begins at layer 1. Connection A no longer exists in layer 1, so its sticky connection preference is forgotten. An attempt is made to transfer it on E, and it becomes the new sticky connection preference, but it does not have available capacity, so it is skipped. The iteration proceeds to layer 2.

[0842] Only layer 1 exhibits a sticky connection preference, and layer 2 has only one connection; therefore, pseudo-random selection provides option B by default. Capacity is available, so transmission occurs. The connection is also fresh, thus contributing 3 to the cumulative score, achieving the desired goal.

[0843] Next, consider the transmission of packet 4, which belongs to the same class. Assume that connection B has now become unreliable, causing its rounded Mathis coefficient to increase to a maximum of 5,000, and that it has become obsolete.

[0844]

[0845] This makes its tuple identical to the connection F, so it is now in the same layer:

[0846] Package 4:

[0847] xxv. Layer 1: Connecting E

[0848] xxvi. Level 2: A

[0849] xxvii. Layer 3: B, F

[0850] xxviii. Layer 4: C, D

[0851] The iteration begins at layer 1. Connection E is a sticky connection preference and exists in layer 1, but it has no available capacity, so it is skipped. The iteration proceeds to layer 2. Only layer 1 has a sticky connection preference, and there is only one connection in layer 2, so the pseudo-random selection provides choice A by default. It is outdated and unreliable, so even if a transfer occurs, its contribution to the cumulative score is 0.

[0852] The iteration proceeds to layer 3. Only layer 1 exhibits a sticky connection preference, so it is assumed that pseudo-random connection selection will initially choose connection F. A transmission occurs, but since it is outdated, its contribution to the target score is 0. The iteration continues within the layer to connection B. It is outdated and unreliable, therefore it also contributes to the target score after transmission.

[0853] The iteration stops and does not proceed to layer 4 because all first-level connections have been exhausted, and the connection priority rules are configured to activate second-level connections only when the total fresh, reliable capacity drops below 10 Mbps. Connection E still reaches this threshold, but it currently has no capacity.

[0854] Packet 4 does not have any more available connections for transmission, but it has not yet reached its target score of 3. The system will wait until an event occurs that allows additional transmission to occur. For example, acknowledgment of an in-flight packet on connection E will increase its available capacity; a failure of connection E will meet the threshold for activating a secondary connection; or the measured characteristics of any other primary connections will be updated so that they are no longer obsolete and / or unreliable.

[0855] Assuming an event occurs that causes the capacity of connection E to drop to 8 Mbps, this also reduces its capacity factor:

[0856]

[0857] The layers remain unchanged, and the iteration steps are repeated from the beginning again, but this time instead of stopping at layer 3, the iteration proceeds to layer 4, because the total available first-level capacity has dropped below the configuration threshold of 10 Mbps (only 8 Mbps is available on connection E).

[0858] Only layer 1 exhibits a sticky connection preference, so it is assumed that pseudo-random connection selection first chooses connection D from layer 4. Transmission occurs, but since it is outdated, its contribution to the target score is 0. The iteration proceeds within the layer to connection C. It is fresh, therefore its contribution to the packet score is 3. The target is reached, and packet 4 is transmitted.

[0859] Assuming no more packets are available, scheduling and transmission are paused until an event occurs (e.g., a new packet is received at the input, a negative acknowledgment of a transmitted packet is received, etc.).

[0860] Considering all the aforementioned outdated packet transmissions, there may be many duplicate and out-of-order packets. In some embodiments, reordering and deduplication may occur, and endpoints may receive packets in sequence.

[0861] According to one embodiment, the implementation of the per-stream communication system may follow such a process outlined above, but in other embodiments, it may include more or fewer steps and may be performed in a different order.

[0862] Figure 22 This is a block diagram illustrating a workflow 2200 according to some example embodiments, the workflow which can occur Figure 21 Between 2121 and 2122, these are particularly used for real-time streaming. When a packet's deadline is approaching but it has not yet been acknowledged, these streams can trigger an early retransmission of the packet.

[0863] According to one embodiment, exemplary workflow 2200 may follow such a process as outlined below, but in other embodiments, it may include more or fewer steps and may be performed in a different order.

[0864] In the illustrated exemplary workflow 2200, starting at 2210, the transmitted packets (which occur...) Figure 21 At point 2121, check if this is the first transmission of the packet. If not, it means the packet has already been transmitted on a better connection, and therefore the early retransmission eligibility check has been completed. In this case, the packet returns to... Figure 21 2122 in the middle.

[0865] However, if this is the first transmission of the packet, a check occurs at 2211 to determine if there is sufficient time before the packet deadline to complete an early retransmission attempt. In some embodiments, this check consists of three parts:

[0866] The optimal duration of this first transmission and its corresponding ACK (which will typically be the connection RTT)

[0867] In the absence of an ACK, the worst-case duration after the best-case duration has expired is the time to retransmit and attempt a successful retransmission (which will typically be the connection EDT).

[0868] A predefined "overhead" value is used to describe the system processing time (in some embodiments, this is a constant of 50 milliseconds).

[0869] If the sum of these three parts exceeds the current packet ATL, it is marked at 2212 as not eligible for early retransmission. In both cases (eligible and ineligible), the packet proceeds to the return. Figure 21 Step 2122 in the process.

[0870] As a non-limiting example implementation (which may follow the workflow shown in 2200), consider packet 1 from a real-time stream, with an ATL of 200 milliseconds, scheduled for its first transmission on connection A, where the RTT is 25 milliseconds and the EDT is 100 milliseconds. At 2211, the sum of the three parts (RTT + EDT + 50 milliseconds = 25 + 100 + 50 = 175 milliseconds) is less than the ATL, therefore the packet is eligible for retransmission. The explanation for this comparison is that, in the nominal scenario, connection A is expected to deliver packet 1 and the ACK within its 25-millisecond RTT. If 25 milliseconds have elapsed and no ACK or NACK has been received, the system may begin to suspect that the transmission has been lost and consider retransmission early to meet the deadline. For this early retransmission to be effective, there must be a reserved additional EDT + processing time buffer before the packet 1 deadline.

[0871] In this example, there is sufficient time, so packet 1 is eligible to be considered for early retransmission, and it proceeds back to step 2122.

[0872] Next, consider packet 1 re-entering the workflow, as it requires multiple transmissions as described in workflow 2100 to achieve its target score. This time, at 2210, packet 1 is not on its first transmission, so it directly returns to 2122.

[0873] Next, consider packet 2 from the same real-time stream arriving at 2210, with an ATL of 150 milliseconds. Assume it is also scheduled for its first transmission on connection A, where the RTT and EDT parameters are the same as packet 1. This time, the sum of the three parts (175 milliseconds) is greater than the ATL, so packet 2 is marked as ineligible for retransmission at 2212. It then returns to the 2100 workflow at 2122.

[0874] According to one embodiment, the implementation of the per-stream communication system may follow such a process outlined above, but in other embodiments, it may include more or fewer steps and may be performed in a different order.

[0875] Figure 23 This is a block diagram illustrating a workflow that considers the early retransmission of packets in flight, based on some example embodiments. Specifically, Figure 23 The diagram shows a block diagram of workflow 2300, which is used to consider the early retransmission of in-flight packets (those packets that have been transmitted and reached their target score through workflows 2100 and 2200, but have not yet been confirmed as received or lost).

[0876] According to one embodiment, the exemplary workflow 2300 may follow such a process outlined below, but in other embodiments, more or fewer steps may be included and may be performed in a different order.

[0877] In the exemplary workflow 2300 shown, starting at 2310, an iteration of in-flight packets belonging to a real-time stream occurs in ascending order of packet deadline.

[0878] At 2311, packets with a deadline in the past are skipped, and at 2312, packets that have already been retransmitted early are skipped. Any packet marked as ineligible for early retransmission as part of workflow 2200 is also skipped at 2312.

[0879] If the packet has not been skipped, its ATL is compared with the EDT of all available connections at 2320. For all fresh connections, a connection with the best EDT less than the packet's ATL and not the first connection originally used to transmit the packet is returned at 2321 as a candidate for early retransmission. If no such candidate exists, a plurality of candidates consisting of all stale connections plus the first connection originally used to transmit the packet (if it meets the EDT < ATL requirement) is returned at 2322.

[0880] At 2323, the candidate with the worst EDT is checked according to the packet deadline. If the deadline has become too close such that this candidate would need to transmit the packet now for it to reach the destination in time, the packet is scheduled at 2324 for immediate early retransmission on all candidates. In this exemplary embodiment, the definition of "too close" is a predefined constant of 50 milliseconds, meaning that the current time plus the connection EDT is less than 50 milliseconds from the packet deadline.

[0881] Otherwise, the packet's deadline is not close enough to authorize early retransmission, and the system prefers to wait longer in case the initial transmission of the packet through workflow 2200 was successfully delivered.

[0882] The iteration continues at 2313 to the next in-flight packet (if one is available), otherwise the early retransmission workflow ends at 2314 and resumes later. In this exemplary embodiment, for the packet with the earliest deadline, workflow 2300 is triggered by a timer set according to the above "too close" deadline. The workflow is also always triggered immediately before workflow 2100.

[0883] As a non - restrictive exemplary implementation (which may follow the workflow shown in 2300), consider Packet 1, which is the only packet in flight, where the future deadline is T = 1000 milliseconds and the current time is t = 100 milliseconds. Assume the following connections exist in the system and the first connection on which Packet 1 was initially transmitted is C:

[0884] connect EDT Fresh / Expired A 150 milliseconds Fresh B 300 milliseconds outdated C 50 milliseconds Fresh D 195 milliseconds Fresh

[0885] At 2310, Packet 1 is the only packet in flight, so it is selected for iteration.

[0886] At 2311, it is determined that Packet 1 does not have a past deadline (the current time is t = 100 milliseconds and the deadline is T = 1000 milliseconds).

[0887] At 2312, it is determined that the packet is eligible for early re - transmission (not previously marked ineligible by workflow 2200) and has not been re - transmitted early.

[0888] At 2320, the ATL is 900 milliseconds (T = 1000 milliseconds minus t = 100 milliseconds), so all four connections A, B, C, and D in the system satisfy the (EDT < ATL) requirement. However, B does not satisfy the freshness requirement and C was the first connection used to transmit Packet 1, so it is also excluded. This leaves connections A and D as the only eligible connections.

[0889] Connection A has the best EDT out of the two, so it is returned as the only eligible candidate at 2321.

[0890] At 2323, connection A does not satisfy the "packet deadline approaching" requirement. Recall that in this exemplary embodiment, it is (now + EDT + 50 milliseconds) >= packet deadline, or (t = 100 milliseconds+150 milliseconds + 50 milliseconds)=300 milliseconds, which is not >= T = 1000 milliseconds.

[0891] Therefore, Packet 1 is not re - transmitted early. At 2313, more packets are iterated, but there are no remaining packets, so the early re - transmission workflow terminates at 2314.

[0892] In this exemplary embodiment, the timer is set to restart workflow 2300 at t = 800 milliseconds. If Packet 1 is still in flight at that time, this is considered the time when the packet 1 deadline is considered close enough to require early re - transmission on connection A.

[0893] Assume this timer starts, Packet 1 is still in flight, but the connection characteristics have now changed such that connection A is obsolete:

[0894]

[0895]

[0896] The previous steps 2310, 2311, and 2312 remain unchanged.

[0897] At 2320, the ATL is 200 milliseconds (T = 1000 milliseconds minus t = 800 milliseconds), so only connecting A, C, and D satisfies the (EDT < ATL) requirement. However, A no longer satisfies the freshness requirement, and C was the first connection used to transmit Packet 1, so it is also excluded. This leaves connection D as the only eligible connection, so it is returned as a candidate at 2321.

[0898] At 2323, connection D satisfies the "packet deadline approaching" requirement. Recall that in this exemplary embodiment, it is (now + EDT + 50 milliseconds) >= packet deadline, or (t = 800 milliseconds + 195 milliseconds + 50 milliseconds) = 1045 milliseconds, which is >= T = 1000 milliseconds.

[0899] As a result, at 2324, Packet 1 is retransmitted early on connection D.

[0900] The next iteration for the packet occurs at 2313. Assume there are no other in - flight packets, so the early retransmission workflow terminates at 2314.

[0901] Assume there is now an in - flight Packet 2 with a deadline of T = 2000 milliseconds, initially transmitted on connection C, and the current time is t = 1900 milliseconds. Assume the connection characteristics have not changed since Packet 1:

[0902] connect EDT Fresh / Expired A 150 milliseconds outdated B 300 milliseconds outdated C 50 milliseconds Fresh D 195 milliseconds Fresh

[0903] Assume Packet 2 passes the checks at levels 2311 and 2312 and thus moves to 2320. The ATL is 100 milliseconds (T = 2000 milliseconds minus t = 1900 milliseconds). Only connection C satisfies the EDT < ATL requirement, and it is also the first connection used to transmit Packet 2.

[0904] As a result, 2320 has no option but to proceed to 2322, where it returns all the outdated connections plus the first connection in the initial transmission. This means that all of connections A, B, and C are returned.

[0905] At 2323, the connection with the worst EDT is B, and it satisfies the "packet deadline close" requirement. Recall that in this exemplary embodiment, it is (now + EDT + 50 ms) >= packet deadline, or (t = 1900 ms + 300 ms + 50 ms) = 2250 ms, which is >= T = 2000 ms.

[0906] As a result, packet 2 is retransmitted early on all candidates, namely connections A, B, and C. At 2313, the next packet iteration occurs. Assuming there are no other packets in flight, the early retransmission workflow terminates at 2314.

[0907] According to one embodiment, the implementation of the per-stream communication system may follow such a process outlined above, but in other embodiments, it may include more or fewer steps and may be performed in a different order.

[0908] Figure 24 This is an illustration of a per-stream communication system implementation scheme 2400 based on some example embodiments.

[0909] During an emergency, first responders should not need to worry about their communication connections, nor should they be IT experts. First responders and public safety agencies rely on advanced technologies to improve situation awareness and response time. Transmitting real-time video and data from the field requires significant bandwidth, and relying on a single connection can leave an organization vulnerable.

[0910] Emergency situations can occur in various environments, including those where network conditions are unreliable and a reliable connection is still required. This connection needs to provide adequate security, diversity, redundancy, latency, and bandwidth for communications such as real-time video and data. These communications typically have latency / jitter requirements, and with current technology, available WAN connections cannot always meet real-time demands. This inability to meet requirements may be due to connection overload (contention / competition for connection capacity) or connection failures / inconsistencies (e.g., fiber optic cable cuts, wireless interference, equipment malfunctions). Connection contention can occur anywhere along the path (first mile / middle mile / last mile), wherever demand exceeds the capacity of a particular network link.

[0911] Although individual WANs may be unreliable, the above embodiments use multiple WAN connections to meet these real-time requirements.

[0912] Consider an exemplary implementation scheme 2400, such as Figure 24As shown. In this example, a police officer 2404 can be seen through body camera 2402. For example, police officer 2402 may be conducting a chase in a challenging environment across the spectrum, such as an apartment building, and body camera 2404 may be broadcasting live video and audio of the situation to the 911 dispatch center.

[0913] As seen in 2400, its connection to network 2414 may be overloaded, or it may have experienced a failure of the main connection 2406. According to one of the above embodiments, router 2408 is a multi-LTE WAN router and has multiple connections 2410. Router 2408 can maintain reliable communication with the dispatch center through multiple other connections. Data packets for video and audio communication can be transmitted according to the process described in the above embodiments, and latency and jitter requirements can be ensured to be met, making the video feed clear to the dispatch center (including, for example, dispatch worker 2412).

[0914] Similarly, since connection failures can occur with any type of connection, the dispatch center itself can connect to the network using a multi-WAN router 2408 that supports multiple wired and wireless broadband connections 2410 (e.g., fiber optic, cable, DSL, LEO satellite, LTE, etc.). According to the above embodiments, connection priority rules can be used to classify unmetered connections as Level 1 (e.g., fiber optic, cable, DSL, LEO satellite) and metered, more expensive connections (e.g., LTE) as Level 2. Data packets for video and audio communication can be transmitted according to the process described in the above embodiments, and latency and jitter requirements can be ensured even if multiple WAN connections at the dispatch center experience congestion or failure simultaneously. This typically occurs in dispatch centers in areas with limited unmetered connection options, such as rural areas with only access to low-quality DSL lines and oversubscribed LEO satellite services.

[0915] Other public safety-related use cases may include mobile command, where reliable connectivity is required for connected command vehicles to send and receive data from cameras, sensors, and edge devices. First responders must have rapid and reliable access to the information they need to make life-saving decisions.

[0916] As public safety departments worldwide increase their use of real-time video, the amount of video generated and consumed by law enforcement agencies is also rapidly increasing. The embodiments described present a solution to this increased demand for high standards of reliability, implementing multiple connections for sending data packets, a process for sorting / hierarchizing these connections, and packet delivery to ensure timely arrival while adhering to stringent requirements.

[0917] The embodiments described can be used for fleet connectivity, where real-time video can be sent back to the command center, enabling "mobile stations" to help make appropriate response decisions or retrieve critical data from the scene to keep the team safe and enable faster responses. For example, mobile cameras can capture license plates and send them to the edge for real-time analysis to alert police whether action should be taken.

[0918] Real-time video applications (such as those described in example implementation 2400) can also ensure that the central command has real-time video during sporting events, marches, protests and other events that attract large numbers of mobile people to support decision-making.

[0919] The embodiments described herein can enable unmanned aerial systems or drones to provide real-time video from a secure sky, thereby improving situational awareness for law enforcement officers and fire commanders.

[0920] Figure 25 This is an illustration of a per-stream communication system implementation scheme 2500 based on some example embodiments.

[0921] As VoIP evolves, broadcasting needs, processes, and deliverables continue to evolve. For example, the embodiments described can provide a single service for uninterrupted live streaming from remote locations. Some embodiments may be easy to use and integrate into broadcast workflows, and can be adapted to different environmental uses (e.g., vehicle-mounted, backpack-wearable, integrated into drones, etc.).

[0922] Broadcast communications typically have latency / jitter requirements, and with current technology, available WAN connections cannot always meet real-time demands. This inability to meet requirements can be due to connection overload (contention / competition for connection capacity) or connection failures / inconsistencies (e.g., fiber optic cable breakage, wireless interference, equipment malfunction). Connection contention can occur anywhere along the path (first mile / middle mile / last mile), wherever demand exceeds the capacity of a given network link.

[0923] Although individual WAN connections may be unreliable, the above embodiments use multiple WAN connections to meet these real-time requirements.

[0924] Consider shown in Figure 25 The example implementation is shown below. Reporter 2502 may be reporting breaking news from a live event, where reporter 2502 is filming from a remote broadcast location using a broadcast camera 2504, and because a large number of people at the event are all using their mobile phones 2506, the main connection 2508 becomes very slow due to overload and competition. A slow connection is obviously detrimental to good broadcast production.

[0925] Reporter 2502 may need to transmit live video and make VoIP calls with the main broadcaster to receive their video and audio, along with key points of the conversation, via teleprompter at a remote location. Low latency and jitter are required to ensure the remote team and Reporter 2502 remain synchronized with the central production team 2514.

[0926] Router 2510 allows this low-latency return video and teleprompter feed to be sent to the on-site live production team and the central production department, and allows them to view these feeds simultaneously on tablets and mobile devices. According to one of the above embodiments, router 2510 is a multi-LTE WAN router.

[0927] Router 2510 can maintain reliable communication with the central production department through multiple other connections 2512. Video communication data packets can be transmitted according to the process described in the above embodiments, ensuring that latency and jitter requirements are met, making the video feed clear for broadcast.

[0928] Router 2510 can provide a reliable connection to network 2516, which can provide high-throughput wireless internet to a remote team with journalists 2502, improving their production efficiency by providing access to cloud services, online resources, media assets, and newsroom systems from any location. Router 2510 can also be a portable and / or vehicle-mounted device. The embodiments described can provide remote connectivity, allowing remote access to media assets and newsroom systems, rapid file transfers, and easier communication with field personnel.

[0929] Figure 26 This is an illustration of a per-stream communication system implementation scheme 2600 based on some example embodiments. Figure 26 Another example of a broadcast implementation scheme is shown.

[0930] Storms, floods, and other extreme weather events can disrupt broadcast operations, and damage or severance of fiber optic cables can take days or weeks to locate and repair. As shown in 2600, the main connection 2602 has been damaged and severed due to a severe weather event and is no longer able to transmit data.

[0931] Due to multiple LTE WAN connections 2606, router 2604 can provide both portable and permanent backup connectivity solutions. As shown in 2600, this allows broadcast 2608 to remain on air and keeps remote teams connected to network 2610 during emergency weather events. In such situations, for public safety, it may be crucial to keep news broadcasts active in the event of large natural disasters such as earthquakes to provide the public with accurate and up-to-date information from the ground. For example, in the event of damage to broadcasting facilities or studio 2612, the embodiments described can keep broadcast 2608 on air and maintain content distribution capabilities.

[0932] Other example broadcast implementations and use cases described in the embodiments can be used for remote contributions, such as... Figure 25 As shown and as described above. The embodiments can also assist in remote production scenarios, where the production activity itself is generated at a remote location.

[0933] For example, the embodiments described may also allow content to be distributed cost-effectively to network affiliates, group television stations or other broadcasters and media organizations.

[0934] Figure 27 This is a schematic diagram of a computing device 2700 that can be used to implement systems 1200A and / or 1200B according to one embodiment.

[0935] As shown in the figure, the computing device 2700 includes at least one processor 2702, a memory 2704, at least one I / O interface 2706, and at least one network interface 2708.

[0936] Each processor 2702 may be, for example, a microprocessor or microcontroller (e.g., a dedicated microprocessor or microcontroller), a digital signal processing (DSP) processor, an integrated circuit, a field-programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or any combination thereof.

[0937] The memory 2704 may include various types of computer memory located internally or externally, such as suitable combinations of random access memory (RAM), read-only memory (ROM), compressed optical disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), ferroelectric RAM (FRAM), etc.

[0938] Each I / O interface 2706 enables the computing device 2700 to interconnect with one or more input devices such as a keyboard, mouse, camera, touch screen and microphone, or with one or more output devices such as a display screen and speakers.

[0939] Each network interface 2708 enables the computing device 2700 to communicate with other components, exchange data with other components, access and connect to network resources, service applications, and perform other computing applications by connecting to a network (or multiple networks) capable of carrying data, including the Internet, Ethernet, Common Old-Style Telephone Service (POTS) lines, Public Switched Telephone Network (PSTN), Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), coaxial cable, optical fiber, satellite, mobile networks, wireless networks (e.g., Wi-Fi, WiMAX), SS7 signaling networks, fixed lines, local area networks, wide area networks, and various combinations thereof.

[0940] For simplicity only, one computing device 2700 is shown, but systems 1200A and / or 1200B may include multiple computing devices 2700. The computing devices 2700 may be the same or different types of devices. The computing devices 2700 may be connected in various ways, including direct coupling, indirect coupling via a network, and distributed over a wide geographical area and connected via a network (this may be referred to as "cloud computing").

[0941] For example, and not limited to, computing device 2700 may be a server, network device, set-top box, embedded device, computer expansion module, personal computer, laptop computer, personal data assistant, cellular phone, smartphone device, UMPC tablet computer, video display terminal, game console, or various other computing devices that can be configured to perform the methods described herein.

[0942] Figure 28 Illustration 2800 depicts a physical computer server rack according to some example embodiments. In the computer server rack shown in 2800, there are multiple networked computing components, including rack-mounted computing devices such as network routers, switches, hubs, gateways, etc. In this example, the components interoperate with each other to control routing, for example, by establishing and / or periodically updating routing tables and / or routing rules. In another embodiment, the components interoperate according to routing tables and / or routing rules to control routing by controlling network connectivity, routing data packets, etc.

[0943] As described herein, the method is a technology and computing solution that can be implemented in the form of a physical data router or other networked device configured to control the communication routing of data packets. The device may include one or more processors operating in conjunction with computer memory, and the one or more processors may be coupled to a data storage device. Correspondingly, the method also includes a non-transitory computer-readable medium (e.g., floppy disk, solid-state storage device, hard disk drive) storing machine-interpretable instructions thereon (e.g., software that, when executed by a computer processor, causes the processor to perform the methods described herein in various embodiments).

[0944] Figure 29 The illustration depicts a practical variation using a computing device with reduced computing power, based on some example embodiments.

[0945] exist Figure 29 Figure 2900 illustrates a variant embodiment of the per-stream system described herein. In this variant, the system is adapted to use a computing device 2904 with reduced computing power (e.g., a Raspberry Pi) at one or the other side of the communication. TM Device or Arduino TM The device is used for operation.

[0946] In this example embodiment, computationally expensive or computationally resource-intensive operations (e.g., processing time, speed, capacity, memory / storage device requirements) are performed on the side with more available resources, while the other side is allocated to perform basic operations that require only minimal computational resources.

[0947] This is useful when the actual situation requires asymmetric levels of investment and computing power, due to factors such as the availability of computing devices in specific regions or the difficulty in transporting expensive and heavy equipment to such regions (e.g., emergency areas, remote areas, or communications personnel in areas with poor transportation infrastructure).

[0948] Therefore, asymmetric computing scenarios are possible and have been considered, and the system can also be adapted to operate in these types of configurations.

[0949] Specifically, the side with more resources can perform computationally expensive operations and communicate the results over the air to the other side, such as network models for each connection (e.g., capacity, cwnd, loss rate, RTO, all of which require data collection over time and mathematically demanding statistical and filtering operations).

[0950] A negotiation agreement can be implemented, in which both sides can communicate their capabilities and jointly agree on who will calculate what and communicate it to the other side.

[0951] Specifically, each side can be associated with a data structure that corresponds to the advertised capabilities, and an automated co-agreement process can be performed using a scoring / ranking algorithm.

[0952] In another variation, a discovery protocol is used, where each side knows what the other can do and manages packet transmission so that even if the network allows high bit rates, the other side will not be computationally overwhelmed in terms of packet processing. In practice, the discovery protocol can be implemented using data processes that run, for example, asynchronously or synchronously on both sides, to maintain an up-to-date list of available computing power. This can be useful when devices draw resources from a shared capacity pool, such as in distributed computing systems where resources can be dynamically allocated and distributed.

[0953] Therefore, as Figure 29 As shown, the dynamic allocation of computational tasks for network routing can be shared between two devices, allowing the more computationally powerful device to control routing and send routing instructions to the other device in the form of data messages and packets. Specifically, the more computationally powerful device can be used to manage the management structure and establish priority routing, while maintaining and / or being aware of the priorities of various network connections. The more computationally powerful device can manage the comprehensive characteristics representing estimated delivery times and apply various scheduling methods to address problems that arise when prioritizing connections by characteristics.

[0954] Specifically, the layered approach allows for functional equivalence based on various connections defined as belonging to a specific layer, such as per-stream-based specific methods (e.g., different methods for streams with throughput preferences versus real-time streams that may have latency / jitter preferences). This distinction can be used to control retransmission / reception probability methods with finer granularity, making, for example, preemptive retransmission methods suitable only for real-time streams.

[0955] Therefore, a more computationally powerful device can manage and track the ATL value as a dynamic value that changes based on each packet and over time (e.g., as packets age). The more computationally powerful device then identifies connections to schedule packet delivery and sends routing instructions accordingly. Furthermore, connection priority rules and logical flows can be used to establish granular control methods, or to influence how layers are constructed and / or maintained.

[0956] During the scheduling process, the more computationally powerful device performs score allocation, scheduling, and controls communication across various network connections for any device.

[0957] Furthermore, in some embodiments, the occurrence of early packet retransmission can be controlled by a more powerful computing device. In certain example cases, such as when transmitting over all stale connections and / or original connections, a deadline for each packet can be maintained, and early retransmission can be controlled.

[0958] By controlling which device performs which activities, multiple devices with reduced computing power can be considered, and the load can be shared or completely transferred to a single device (e.g., the device with more resources). This can also be used to manage other types of computing characteristics, such as device wear / lifetime management, thermal hierarchy, etc. Therefore, from a computing load balancing perspective, asymmetric computing can be a useful variation to address technical problems arising from differences in computing characteristics or computing environments. While the examples described primarily involve completely transferring the load to the more powerful device, this is not necessarily required; instead, the load for specific activities can be balanced between the two devices.

[0959] The terms “connection” or “coupled to” can include direct coupling (where two elements coupled to each other are in contact with each other) and indirect coupling (where at least one additional element is located between the two elements).

[0960] Although embodiments have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope. Furthermore, the scope of this application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, components, methods, and steps described in this specification.

[0961] As will be readily understood by those skilled in the art from this disclosure, existing or later-developed processes, machines, manufactures, material compositions, components, methods, or steps that perform substantially the same functions as those in the corresponding embodiments described herein or achieve substantially the same results as those in the corresponding embodiments described herein can be utilized. Therefore, the appended claims are intended to include such processes, machines, manufactures, material compositions, components, methods, or steps within their scope.

[0962] As you can understand, the examples described and shown above are intended to be illustrative only.

Claims

1. An apparatus for coordinating data communication across multiple wide area network connections, the apparatus comprising: A processor coupled to computer memory and a non-transitory computer-readable storage medium, the processor being configured to: For each of the plurality of wide area network (WAN) connections, at least one of the following characteristics—latency, packet loss rate, and throughput—of the packets transmitted over the WAN connection is monitored. For each packet among the multiple packets being routed that has a latency / jitter preference, an adjusted target latency (ATL) is identified based at least on the deadline of each packet in order to deliver the packet to the target endpoint; Grouping is performed using the characteristics of each packet ATL and the plurality of WAN connections to establish a plurality of hierarchical packets, such that each WAN connection in the plurality of WAN connections is grouped into a corresponding hierarchical packet in the plurality of hierarchical packets; and The packet is transmitted using one or more selected wide area network connections selected from the plurality of wide area network connections, wherein the one or more selected wide area network connections are selected using at least the plurality of hierarchical packet selections; The plurality of layered packets include at least a first layer (layer 1), a second layer (layer 2), and a third layer (layer 3), the layers being based on the characteristics EDT and T of each packet's ATL connection to the plurality of wide area networks. prop The relationship between them is defined as follows: Layer 1: Layer 2: Layer 3: .

2. The apparatus of claim 1, wherein the WAN connection latency and packet loss rate characteristics include an estimated delivery time (EDT) generated based on the following relationship: in: P drop Defined as the probability of packet loss; T prop Defined as one-way propagation delay; and T rtprop It is defined as round-trip propagation delay.

3. The apparatus of claim 1, wherein the plurality of hierarchical groups include additional layers, the additional layers being subdivided into layers 1, 2 and 3 based on one or more predefined connection priority rules, such that the result is a new set of layers from layer 1 to layer N.

4. The apparatus of claim 1, wherein the selection of one or more selected WAN connections from the available WAN connections iterates from the available WAN connections in layer 1, and then layer 2, and finally layer 3.

5. The apparatus of claim 4, wherein transmitting the packet using one or more selected WAN connections selected from the plurality of WAN connections includes scheduling the packet on the plurality of connections selected from the plurality of WAN connections for preemptive retransmission.

6. The apparatus of claim 5, wherein wide area network connections in each layer are assigned layer-based scores, and the scheduling of packets on multiple connections is performed to achieve a cumulative score greater than or equal to a predefined target score.

7. The apparatus of claim 3, wherein if scheduling the packet on multiple connections at a higher layer fails to achieve a cumulative score greater than or equal to a predefined target score, then the connection priority rule is ignored.

8. The apparatus of claim 6, wherein a connection designated as unreliable or lacking recent monitoring characteristics is assigned a layer-based score of 0, regardless of its corresponding layer level.

9. The apparatus of claim 4, wherein selecting the selected WAN connection from the available WAN connections in the hierarchical grouping comprises selecting the available WAN connection in a pseudo-random or algorithmic manner.

10. The apparatus of claim 9, wherein the initial selection of the selected WAN connection for a data stream corresponding to a packet includes an initial pseudo-random selection, and the selection of the selected WAN connection is maintained for future packets corresponding to the data stream.

11. The apparatus of claim 10, wherein the selection of the selected WAN connection is maintained only until a connection dormancy period threshold has been reached or an explicit flow termination notification has occurred, after which additional packets on the data flow utilize a different initial pseudo-random selection.

12. The apparatus of claim 1, wherein the processor is further configured to: Data streams are classified into (i) throughput-biased streams and (ii) real-time streams; and only packets from real-time streams are identified as packets with latency / jitter bias.

13. The apparatus of claim 12, wherein for packets from the throughput preference flow, a grouped layer is established based on the same composite characteristic pair based at least on one or more measured characteristics of the plurality of wide area network connections.

14. The apparatus of claim 1, wherein the processor is further configured to: Track the deadline of each packet sent on the first wide area network connection, the packet is approaching its deadline but has not yet been acknowledged; Using a T with an available congestion window (CWND) and a sufficiently short T to transmit packets that are approaching the deadline but have not yet been acknowledged to the corresponding target endpoint. prop The second WAN connection with the backlog performs early retransmission of packets that are approaching the deadline but have not yet been acknowledged.

15. The apparatus of claim 1, wherein the processor is further configured to: Track the deadline of each packet sent on the first wide area network connection, the packet is approaching its deadline but has not yet been acknowledged; It is determined that there is no T with an available congestion window (CWND) and a sufficiently short T to transmit the packet. prop And the available, non-outdated second wide area network connection for the backlog; and In response to the determination, one or more outdated WAN connections are used to perform early retransmission of the packet that is approaching the deadline but has not yet been acknowledged.

16. The apparatus of claim 15, wherein the processor is further configured to: An early retransmission of the packet is performed on the first wide area network connection that was originally used to transmit the packet.

17. The apparatus of claim 8, wherein for packets associated with throughput preference flows, each layer is assigned the same layer-based score that matches the predefined target score.

18. The apparatus of claim 1, wherein the processor is further configured to establish priorities for packet queues defining the data stream among packets within the data stream, the packet queues being unloaded according to a multi-level differential weighted round-robin (DWRR) scheduling method, wherein a higher-priority queue can starve a lower-priority queue, thereby preventing the lower-priority queue from being served, regardless of the weight assigned to the lower-priority queue.

19. The apparatus of claim 1, wherein the ATL is a dynamic value that varies per package.

20. The apparatus of claim 19, wherein the ATL changes as the package ages.

21. The apparatus of claim 20, wherein the ATL is defined as the packet cutoff time value minus the current time value.

22. The apparatus of claim 21, wherein the current deadline time value of the packet is equal to the original reception time of the packet (T). PacketReceived In addition, the nominal target latency requirement of the data stream.

23. The apparatus of claim 1, wherein the apparatus is a network router.

24. The apparatus of claim 1, wherein the apparatus is a physical chipset residing within a network router.

25. The apparatus of claim 23, wherein the network router resides within a data center and is configured to control the transmission of data packets transmitted through one or more coupled endpoint devices.

26. The apparatus of claim 25, wherein the data packet comprises both a packet having a throughput preference and a packet having a latency / jitter preference.

27. The apparatus of claim 25, wherein the network router controls communications of the media broadcasting station or emergency dispatch center.

28. The apparatus of claim 4, wherein selecting one or more selected WAN connections from available WAN connections in a hierarchical group comprises selecting the WAN connection with the shortest backlog, wherein the backlog is measured in time units and is defined as the number of bytes in flight divided by the connection bit rate.

29. The apparatus of claim 10, wherein the selection of one or more selected wide area network connections is maintained only for periods permitted by the management configuration.

30. A method for coordinating data communication across multiple wide area network connections, the method comprising: For each of the plurality of wide area network (WAN) connections, monitor the latency and packet loss rate characteristics of packets transmitted on the WAN connection; For each packet among the multiple packets being routed that has a latency / jitter preference, an adjusted target latency (ATL) is identified based at least on the deadline of each packet in order to deliver the packet to the target endpoint; Grouping is performed using the characteristics of each packet ATL and the plurality of WAN connections to establish a plurality of hierarchical packets, such that each WAN connection in the plurality of WAN connections is grouped into a corresponding hierarchical packet in the plurality of hierarchical packets; as well as The packet is transmitted using one or more selected wide area network connections selected from the plurality of wide area network connections, wherein the one or more selected wide area network connections are selected using at least the plurality of hierarchical packet selections; The plurality of layered packets include at least a first layer (layer 1), a second layer (layer 2), and a third layer (layer 3), the layers being based on the characteristics EDT and T of each packet's ATL connection to the plurality of wide area networks. prop The relationship between them is defined as follows: Layer 1: Layer 2: Layer 3: .

31. The method of claim 30, wherein the WAN connection latency and packet loss rate characteristics include an estimated delivery time (EDT) generated based on the following relationship: in: P drop Defined as the probability of packet loss; T prop Defined as one-way propagation delay; and T rtprop It is defined as round-trip propagation delay.

32. The method of claim 30, wherein the plurality of hierarchical groups includes additional layers that subdivide layers 1, 2 and 3 based on one or more predefined connection priority rules, such that the result is a new set of layers from layer 1 to layer N.

33. The method of claim 30, wherein the selection of one or more selected WAN connections from the available WAN connections iterates from the available WAN connections in layer 1, and then layer 2, and finally layer 3.

34. The method of claim 33, wherein transmitting the packet using one or more selected WAN connections selected from the plurality of WAN connections includes scheduling the packet on the plurality of connections selected from the plurality of WAN connections for preemptive retransmission.

35. The method of claim 34, wherein wide area network connections in each layer are assigned layer-based scores, and the scheduling of packets on multiple connections is performed to achieve a cumulative score greater than or equal to a predefined target score.

36. The method of claim 32, wherein if scheduling the packet on multiple connections at a higher layer fails to achieve a cumulative score greater than or equal to a predefined target score, then the connection priority rule is ignored.

37. The method of claim 35, wherein connections designated as unreliable or lacking recent monitoring characteristics are assigned a layer-based score of 0, regardless of their corresponding layer level.

38. The method of claim 33, wherein selecting the one or more selected WAN connections from the available WAN connections in the hierarchical grouping comprises selecting the available WAN connections in a pseudo-random manner.

39. The method of claim 38, wherein the initial selection of the selected WAN connection for a data stream corresponding to a packet includes an initial pseudo-random selection, and the selection of the selected WAN connection is maintained for future packets corresponding to the data stream.

40. The method of claim 39, wherein the selection of the one or more selected WAN connections is maintained only until a connection dormancy period threshold has been reached or an explicit flow termination notification has occurred, after which additional packets on the data flow utilize a different initial pseudo-random selection.

41. The method of claim 30, comprising: Data streams are classified into (i) throughput-biased streams and (ii) real-time streams; and only packets from real-time streams are identified as packets with latency / jitter bias.

42. The method of claim 41, wherein for packets from the throughput preference flow, a grouped layer is established based on the same composite characteristic pair based on at least one or more measured characteristics of the plurality of wide area network connections.

43. The method of claim 30, further comprising: Track the deadline of each packet sent on the first wide area network connection, the packet is approaching its deadline but has not yet been acknowledged; Using a T with an available congestion window (CWND) and a sufficiently short T to transmit packets that are approaching the deadline but have not yet been acknowledged to the corresponding target endpoint. prop The second WAN connection with the backlog performs early retransmission of packets that are approaching the deadline but have not yet been acknowledged.

44. The method of claim 30, further comprising: Track the deadline of each packet sent on the first wide area network connection, the packet is approaching its deadline but has not yet been acknowledged; It is determined that there is no T with an available congestion window (CWND) and a sufficiently short T to transmit the packet. prop And the available, non-outdated second wide area network connection for the backlog; and In response to the determination, one or more outdated WAN connections are used to perform early retransmission of the packet that is approaching the deadline but has not yet been acknowledged.

45. The method of claim 44, further comprising: An early retransmission of the packet is performed on the first wide area network connection that was originally used to transmit the packet.

46. ​​The method of claim 37, wherein for packets associated with throughput preference flows, each layer is assigned the same layer-based score that matches the predefined target score.

47. The method of claim 30, further comprising establishing priorities for packet queues defining the data stream among packets within the data stream, wherein the packet queues are unloaded according to a multi-level differential weighted round-robin (DWRR) scheduling method, wherein the queues with higher importance levels can starve the queues with lower importance levels, thereby preventing the lower importance queues from being served, regardless of the weights assigned to the lower importance queues.

48. The method of claim 30, wherein the ATL is a dynamic value that varies per package.

49. The method of claim 48, wherein the ATL changes as the package ages.

50. The method of claim 49, wherein the ATL is defined as the packet deadline time value minus the current time value.

51. The method of claim 50, wherein the current deadline time value of the packet is equal to the original reception time of the packet (T). PacketReceived In addition, the nominal target latency requirement for the data stream is added.

52. The method of claim 30, wherein the method is performed on a network router.

53. The method of claim 30, wherein the method is performed by a physical chipset residing within a network router.

54. The method of claim 52, wherein the network router resides within a data center and is configured to control the transmission of data packets transmitted via one or more coupled endpoint methods.

55. The method of claim 54, wherein the data packet comprises both a packet having a throughput preference and a packet having a latency / jitter preference.

56. The method of claim 54, wherein the network router controls the communication of the media broadcasting station or the emergency dispatch center.

57. The method of claim 33, wherein selecting one or more selected WAN connections from available WAN connections in a hierarchical group comprises selecting the WAN connection with the shortest backlog, wherein the backlog is measured in time units and is defined as the number of bytes in flight divided by the connection bit rate.

58. The method of claim 39, wherein the selection of one or more selected WAN connections is maintained only for a period of time permitted by the management configuration.

59. A non-transitory computer-readable medium storing a machine-interpretable instruction set, which, when executed by a processor, causes the processor to perform the method according to any one of claims 30 to 58.

Images