Packet transmission system and method

Abstract

To provide a network gateway for routing data flows across a plurality of network connections, a method therefor, and a computer readable medium.SOLUTION: Each of multipath gateways 104, 108 is a network gateway for routing data flows across a plurality of network connections 106. The multipath gateways each include a plurality of network interfaces for transmitting data over the plurality of network connections, and at least one processor configured for: transmitting a sequential burst of packets across the network interface; on the basis of, timestamps recorded when packets in the sequential burst of packets are received at a receiving node, and a size of the packets, generating a bandwidth of the network interface; and routing a data flow of sequential packets across the plurality of network connections on the basis of, the generated bandwidth.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is a nonprovisional application of, and claims the full benefit, including priority, of, (1) U.S. patent application Ser. No. 62 / 437,635, entitled "Packet Transmission System and Method," filed December 21, 2016, and (2) U.S. patent application Ser. No. 62 / 558,610, entitled "Packet Transmission System and Method," filed September 14, 2017. Both of these applications are incorporated by reference in their entirety.

[0002] The present disclosure relates generally to the field of electronic communications, and more specifically to improved packet transmission for protocols with competing requirements such as latency, throughput, jitter, and packet sequencing, and to combining multiple data connections to achieve improved transmission speeds and reliability in situations where individual data connections may be lossy, unreliable, or have significantly different or variable latency, transmission speeds, or packet loss. [Background technology]

[0003] Transmission of electronic data over wired and wireless networks can be accomplished using various packetization methods. For example, packets can be provided under various protocols, such as the User Datagram Protocol (UDP) and the Transmission Control Protocol (TCP). For UDP packets, packets are provided on a "best-effort" basis, meaning that there is no acknowledgment of receipt of packets sent at the transport layer for each packet. Therefore, efficient transmission at the transport layer is simplified, leaving it up to applications to handle out-of-order or lost packets reliably. For TCP packets, error checking is performed as part of the transport protocol, and TCP packet sequence numbers and acknowledgments are used to ensure that applications receive all data reliably and in order. Lost packets can be explicitly retransmitted, for example.

[0004] With TCP, receiving packets out of order can affect transmission efficiency due to the acknowledgment and re-request process built into TCP, as congestion control algorithms consider out-of-order packets to be caused by network transmission errors and reduce the transmission rate to compensate for these perceived errors. This issue is particularly important when simply combining data connections with significantly different latencies, which can result in significantly reduced transmission rates (known as the "head-of-line blocking" problem). Summary of the Invention

[0005] Combining different networks and / or connections together is technically challenging. A single network and / or connection is particularly prone to transmission errors, signal loss, and lack of redundancy. Combined networks and / or connections are particularly useful in various scenarios, such as news gathering and reporting, because these scenarios require reliable, high-bandwidth transmission through a variety of environments where individually reliable connections may not be readily available (e.g., rural areas, high altitudes), or, if available, may be congested (e.g., sporting events) or underperforming (e.g., after natural disasters).

[0006] In some embodiments, applicants have developed improved data transmission management solutions that apply non-traditional approaches to utilizing technical features of network technologies in establishing hybrid network connections. As described in further detail, additional technical uses and approaches are particularly difficult to combine with TCP due to the need to establish viable connections (e.g., to avoid "head-of-line" blocking issues, as described above). These improved hybrid connections can be used, for example, to transfer high-bandwidth video, voice, or other data. For example, in disaster situations, the improved hybrid connections can be utilized to establish emergency communications, provide ground coverage, and enable emergency telemedicine, among other things.

[0007] A network gateway for controlling data flow through a network connection is described herein, along with corresponding methods, apparatus, and computer-readable media. The gateway may, in some embodiments, be a physical hardware device with components such as a processor, interfaces, buses, power supplies, memory (ROM, RAM, Flash), etc., configured to receive and route data packets according to an instruction set representative of software and embedded firmware in the form of program code or computer logic.

[0008] The network gateway may, in one example, be a specialized computing device specifically optimized for bonding TCP connections, as described in some examples herein. In other examples, the network gateway may be implemented using a processor and / or other computing hardware as part of a larger system. The network gateway may be a single device, or in some cases, multiple devices operating in concert. A corresponding de-bonding server may be provided to reconstruct the signal upon receipt.

[0009] In one embodiment, the network gateway is configured to route a data flow through a plurality of network connections, the network gateway comprising: a plurality of network interfaces for transmitting data through the plurality of network connections; and at least one processor configured for monitoring time-varying network transmission characteristics of the plurality of network connections; analyzing at least one packet of a data flow of packets to identify a data flow class for the data flow, the data flow class defining at least one network interface requirement for the data flow; and routing packets in the data flow through the plurality of network connections based on the data flow class and the time-varying network transmission characteristics.

[0010] In another embodiment, the network gateway comprises a plurality of network interfaces, including a first network interface for transmitting data through a plurality of network connections, the network gateway comprises a processor configured to transmit sequential bursts of packets through the first network interface, the processor generating a bandwidth for the first network interface based on timestamps recorded when packets in the sequential burst of packets are received at a receiving node and sizes of the packets, and routing the data flow of the sequential packets through the plurality of network connections based on the generated bandwidth of the first network interface.

[0011] In one aspect, generating the bandwidth for the first network interface includes generating the bandwidth based on timestamps of the first or last packet in the burst and packets in the burst that are not coalesced.

[0012] In another aspect, generating bandwidth for the first network interface includes replacing a receive timestamp of a particular packet in the burst with a transmit timestamp of a packet sent after the particular packet.

[0013] In another aspect, when the receiving node processes packets received at regular intervals, generating the bandwidth of the first network interface includes (i) generating a lower bandwidth value by using a reception timestamp of a packet in a burst selected as an end packet in the bandwidth determination, and (ii) generating an upper bandwidth value by replacing the reception timestamp of the packet in the burst selected as the end packet with the reception timestamp of a packet in the burst preceding the end packet.

[0014] In another aspect, the processor is configured to provide, for each packet of sequential packets in the data flow, the packet for routing through one of the plurality of network connections based on the monitored latencies of the plurality of network connections and the network connections of other packets in the data flow such that the packets in the data flow arrive at the destination node in a desired sequence.

[0015] In another aspect, the desired sequence is the original sequence of sequential packets in the data flow.

[0016] In another aspect, the desired sequence is a sequence that includes at least one misordering of packets that does not trigger a retransmission of packets within the sequence.

[0017] In another aspect, the processor is configured for receiving a packet from a source interface for routing to a destination node over a plurality of network connections, sending an acknowledgment to the source interface before routing the packet to the destination node, and storing the packet in at least one buffer before routing the packet to the destination node.

[0018] In another aspect, the at least one processor is configured for dynamically controlling a size of the at least one buffer based on a bandwidth-delay product associated with the plurality of network connections.

[0019] In another aspect, the at least one processor is configured for controlling the sending and storing of packet acknowledgments based on monitoring transmission characteristics of the plurality of network connections and unequal distribution in the reception of the data flow of sequential packets.

[0020] In another aspect, the at least one processor is configured to provide the packets based on at least one of a bandwidth of the first data connection and a reduction in the number of network connections through which the data flow is routed.

[0021] In another aspect, at least one processor is configured to route packets for a plurality of data flows, where the at least one processor is configured for providing packets of the plurality of data flows for routing through one of a plurality of network connections based on classifications of the plurality of data flows.

[0022] In another embodiment, the classification of one of the plurality of data flows changes after a threshold volume of data has been routed.

[0023] According to another aspect, there is provided a data transmission manager ("gateway") suitable for controlling the routing of data packets belonging to a plurality of flows through a plurality of simultaneous data connections, each data connection carrying a plurality of packets belonging to a plurality of flows, wherein at least one of the plurality of flows has data packets transmitted substantially sequentially, the data transmission manager comprising: a buffer manager configured to receive flows (sequences of input data packets) from one or more input data connections, the buffer manager storing the input data packets in one or more buffers extracted for routing, each input data packet being associated with a corresponding sequence number representing its order in the flow or sequence; a connection control device configured to interface with the plurality of data connections, each of the data connections having different transmission characteristics; and a scheduler configured to control the operation of the buffer manager by generating instructions to be executed by the buffer manager to control the routing of the input data packets from the one or more buffers over the plurality of data connections, the routing determining, for each individual input data packet, (i) a corresponding data connection over which the individual input data packet is to be transmitted, and (ii) a connection-specific timing or sequence in which the individual input data packet is to be transmitted on the corresponding data connection.

[0024] In another aspect, the apparatus further comprises a network characteristic monitoring unit configured to monitor transmission characteristics over each of the plurality of data connections monitored together or individually.

[0025] In another aspect, the apparatus further comprises an operations engine configured to control operation of the scheduler by generating instructions executed by the scheduler to control routing of input data packets stored in one or more buffers, the routing being controlled based on transmission characteristics monitored over a plurality of data connections, the operations engine configured to modify the routing based on an input data set providing information regarding at least one of throughput, packet loss, latency, Explicit Congestion Notification (ECN) cost, reliability, jitter, user / administrative preferences, and other network connection constraints.

[0026] In another aspect, the apparatus further comprises a sequencer configured to reconstruct input data packets received by the data transmission manager based on respective sequence numbers of the input data packets.

[0027] In another aspect, the apparatus further comprises a flow identification engine designed to identify flows based on unique characteristics and provide flow information to the action engine and the sequencer.

[0028] In another aspect, the apparatus further comprises a flow classification engine configured to classify input data flows and their requirements and, based on the characteristics and requirements of the flows, provide this information to the action engine for modification of the routing of the input data flows and to the sequencer for modification of the manner in which data packets are reassembled.

[0029] In another aspect, the buffer manager is further configured to provide excess buffering to compensate for variability in transmission characteristics of multiple data connections and uneven distribution of transmissions ("bursty transmissions").

[0030] In another aspect, the operation engine is further configured to determine a bandwidth-delay product for each data connection of the plurality of data connections, the bandwidth-delay product being used to modify the routing as controlled by the scheduler.

[0031] In another aspect, the action engine is further configured to perform bandwidth estimation when incomplete information is available.

[0032] In another aspect, the one or more types of data traffic (flows) include at least one of FTP, DNS, HTTP, SSH, and SSL / TLS.

[0033] In another aspect, the operations engine is configured to determine which of the one or more data flows are latency-sensitive, and for the one or more latency-sensitive data flows, maintain preferential routing of corresponding incoming data packets over data connections of the multiple data connections having lower and / or similar latency characteristics. In this scenario, the operations engine may also prevent routing of non-latency-sensitive flows over the multiple data connections to prevent excessive buffering / buffer bloat.

[0034] In another aspect, the operation engine is further configured to modify header information of known protocols to optimize their performance across multiple data connections. For example, if the multiple data connections include one or more high-latency terrestrial satellite connections, the optimization engine may adjust the advertised TCP window size to allow the flow to take advantage of a high bandwidth-delay product (known as TCP acceleration). These changes may also require the buffer manager to buffer additional packets on a per-flow basis. Another example is modifying the TCP MSS header option to match the smallest MSS (maximum segment size) supported by the multiple data connections to prevent data packet fragmentation.

[0035] In another aspect, there is provided a data transmission management method for controlling routing of data flows (comprising data packets) through a plurality of data connections to provide improved transmission performance in which at least one of the plurality of data flows is transmitted substantially sequentially, the data transmission management method including: receiving, from one or more input data sources, streams or sequences of input data packets associated with corresponding sequence numbers representing their order in the data flows or sequences, and storing them in one or more buffers extracted for routing; interfacing with a plurality of data connections, each of the data connections having different transmission characteristics; generating routing instructions by a scheduler that controls the one or more buffers to route the input data packets through the plurality of data connections, the instructions providing, for each individual input data packet, (i) a corresponding data connection through which the individual input data packet is to be transmitted and (ii) a connection-specific timing or sequence in which the individual input data packet is to be transmitted on the corresponding data connection; and routing the input data packets in accordance with the routing instructions.

[0036] In another aspect, a non-transitory computer-readable medium is provided that stores instructions that, when executed, cause a process to perform steps of a data transmission management method for controlling routing of data flows (comprising data packets) through a plurality of data connections to provide improved transmission performance in which at least one of the plurality of data flows is transmitted substantially sequentially, the method including: receiving, from one or more input data sources, a stream or sequence of input data packets associated with corresponding sequence numbers representing their order in the data flows or sequences, and storing them in one or more buffers extracted for routing; interfacing with a plurality of data connections, each of the data connections having different transmission characteristics; generating routing instructions by a scheduler to control the one or more buffers to route the input data packets over the plurality of data connections, the instructions providing, for each individual input data packet, (i) a corresponding data connection through which the individual input data packet is to be transmitted and (ii) a connection-specific timing or sequence in which the individual input data packet is to be transmitted on the corresponding data connection; and routing the input data packets in accordance with the routing instructions.

[0037] In various further aspects, the present disclosure provides corresponding systems and apparatus, as well as logical structures such as sets of machine-executable coded instructions for implementing such systems, apparatus, and methods.

[0038] In this regard, before describing at least one embodiment in detail, it is to be understood that the embodiment is not limited in application to the details of construction or the arrangement of components set forth in the following description or illustrated in the drawings, and it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.

[0039] Many additional features and combinations of the embodiments described herein will be apparent to those of ordinary skill in the art after reading this disclosure.

[0040] The embodiments in the figures are shown by way of example, and it should be expressly understood that the description and figures are given by way of example only and are intended to aid understanding.

[0041] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0042] [Figure 1] FIG. 1 is a block schematic diagram of a system having two gateways linked by N data connections, each gateway comprising a buffer manager, an action engine, a connection controller, a network characteristic monitoring unit, a flow classification engine (handling flow identification and classification), a scheduler and a sequencer, and with each gateway connected to a specific endpoint, according to some embodiments. [Figure 2] FIG. 1 is a block diagram illustrating a scenario in which an Internet user connects to the Web through a combination of gateways utilizing transport links with different network characteristics, according to some embodiments. [Figure 3] 3 is a block schematic diagram illustrating the operation of a sequencer by demonstrating the ordering of six packets according to an example based on the network of FIG. 2, in accordance with some embodiments. FIG. [Figure 4] FIG. 1 is a block schematic diagram illustrating a scenario in which a scheduler is configured to consider the latency of a data connection, according to some embodiments. [Figure 5] FIG. 1 is a block schematic diagram illustrating a bandwidth-delay product (BDP), according to some embodiments. [Figure 6] 1 is a block schematic diagram illustrating a system suitable for operation based on at least the dimensions (size and sensitivity to latency) of flow requests, according to some embodiments. [Figure 7] 3 is a block schematic diagram illustrating the operation of a network characteristic monitoring unit according to some embodiments. [Figure 8A] 1 is an illustration of an example bandwidth estimation method illustrating an example scenario in which a transmitter transmits a burst of eight packets, according to some embodiments. [Figure 8B] 1 is a diagram illustrating an example of a bandwidth estimation method that illustrates non-idealities in practical implementations, according to some embodiments. [Figure 8C] 1 is an illustration of an example bandwidth estimation method illustrating an example technique for correcting errors due at least in part to interrupt latency, according to some embodiments. [Figure 8D] 8D illustrates a variation of the example described in FIG. 8C for correcting errors due at least in part to interrupt latency under different conditions, according to some embodiments. [Figure 8E] 1 is a description of an example bandwidth estimation method illustrating an example in which interrupt arbitration techniques are considered, according to some embodiments. [Figure 9] FIG. 1 is a block schematic diagram illustrating a system utilizing satellite offload (single input) according to some embodiments. [Figure 10] FIG. 1 is a block diagram illustrating an example of mixing with multiple inputs, according to some embodiments. [Figure 11] FIG. 10 is a block diagram illustrating a scenario configured for uncoupled disaster recovery / load balancing use (each flow is sticky to a specific data channel with no coupling whatsoever, since no buffers are required on the other side, i.e., there is no second gateway), according to some embodiments. [Figure 12] FIG. 1 is a block diagram illustrating a system configured for a combined disaster recovery / load balancing application in which traffic is combined and routed through a second gateway according to some embodiments. [Figure 13] 1 illustrates a series of multipath gateways connecting two or more endpoints, according to some embodiments. [Figure 14] 1 illustrates two multihomed sites connected to the Internet in a traditional Border Gateway Protocol (BGP) transit / peering configuration, according to some embodiments. [Figure 15] 1 shows the same two multi-homed sites connected using a multi-path gateway that allows granular control of how both outbound and inbound traffic between the two sites is routed, according to some embodiments. [Figure 16] 1 is an illustration of a computing device according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0043] Embodiments of the methods, systems and apparatus are described with reference to the drawings.

[0044] The Transmission Control Protocol (TCP) utilizes a three-way handshake to establish a connection and is therefore a reliable protocol (as opposed to the User Datagram Protocol (UDP), which is a connectionless protocol). These protocols are described, for example, in RFC 793 and RFC 7323, which are incorporated herein by reference.

[0045] TCP requires a more complex implementation and is suitable for guaranteed in-order delivery of packets. The connection must first be established between the sender and receiver before any data can be transferred, and the connection is often terminated at the end of the data transmission (to free up allocated resources).

[0046] The initial handshake to establish a connection provides an initial sequence number (e.g., an arbitrary sequence number) that is used during data transmission to identify the order in which data is provided during the data transmission.

[0047] During data transmission, data generated by an application is segmented into data packets, and these data packets or segments have information (e.g., a "header") prepended to them when they are encapsulated for transmission. The header contains information used during transmission, such as indicating a sequence number associated with each segment of data. The sequence number is used by a receiving device so that the segments of data can be reordered and reconstructed even if they are received out of order, lost (e.g., a retransmission may be required), delayed, corrupted, etc.

[0048] The TCP method utilizes additional processes to help reduce transmission problems, including flow control, error detection, retransmission protocols, congestion control, and selective acknowledgment.

[0049] In some scenarios, improved data communication performance may be obtained through the use of multiple data connections or networks operating in concert. These data connections or networks may be considered to be "coupled" to one another such that the networks are logically joined or connected. Two or more networks operating in concert may perform one or more communication functions, such as transmitting data, providing error correction, etc., among other functions. In this context, one or more networks need not necessarily be separate networks, but may include different channels, different data connections, etc., through the same network.

[0050] A combined network is desirable because there may be improved data transmission characteristics (such as increased reliability and increased speed) that can be obtained by advantageously taking advantage of the combined characteristics or using the diverse characteristics and capabilities of each network, such as cellular, Ethernet, WiFi, and satellite networks, among others, which have different characteristics and qualities.

[0051] For example, a particular network connection (e.g., path, channel) may be ideal for a particular type of communication (e.g., low latency but low bandwidth for error detection and control signals), while another network connection may be ideal for another type of communication (e.g., high bandwidth for high-throughput data transfer). In some embodiments, network coupling may be provided such that the endpoints (e.g., transmitter, receiver) do not need to be aware of the communication path or technique being utilized for communicating data.

[0052] Because TCP is the protocol used for a significant number of networks and traffic, it is particularly desirable to have a combined TCP solution with improved transfer characteristics, but combined TCP solutions are difficult to implement.

[0053] A challenge with TCP (particularly because it requires an acknowledgment of each packet) is that TCP connections are technically difficult to "bond" together, especially when the data connections are provided over networks with significantly different characteristics, such as different latencies. These difficulties affect the efficiency of communication systems that utilize bonding for networked TCP communication.

[0054] Data is transmitted to one or more endpoints over a combination of data connections (e.g., over different networks), and the transmission of the data is facilitated so that the data can be recombined (e.g., by a "decombining" system) and the original data can be recreated from the transmitted data.

[0055] Such "binding" of connections and allocations of functionality is described, for example, in applicant's US Pat. No. 8,873,560, which is incorporated herein by reference.

[0056] Particular technical challenges exist in the context of using bonding with TCP because the bonding process typically requires segmentation of the data packets comprising the data flow transmitted over the bonded network or data connection. Different types of traffic (e.g., voice data, control packets, video data, error control) are affected differently by ordering requirements.

[0057] Technical considerations for systems implementing bonded networks for the transfer of data include, but are not limited to, ease of bonding / debonding (e.g., required computational resources and total computation time compared to available computational resources), transfer latency, packet loss, redundancy, security, maximum throughput, average throughput, and congestion management.

[0058] Simple round-robin solutions using multiple network or data connections typically only provide benefit if the connections have identical or nearly identical characteristics (e.g., latency, capacity). If one connection has significantly higher latency, all other packets will be delayed and potentially treated as lost or delayed by the sender and receiver, thereby causing the sender to reduce its congestion window (cwnd) and resulting in reduced throughput.

[0059] If one connection has a significantly higher capacity, when the lower capacity connection reaches its limit, all other packets sent on that connection will be lost.

[0060] Thus, simple round robin solutions typically help with load balancing and redundancy, but at the expense of throughput when connections have different characteristics.

[0061] Multipath TCP (RFC6824) is a transport layer solution commonly used to combine wireless networks such as WiFi and cellular. As described below, the system of some embodiments can provide advantageous performance compared to other solutions such as Multipath TCP (MPTCP) for the following reasons. 1) MPTCP requires both endpoints (client (C) and server (S)) to be able to process using MPTCP. Because both endpoints know about the multiple paths, they run independent congestion control algorithms on each of them, and the endpoints do not encounter the problems described above with standard TCP stacks. In contrast, the system disclosed herein, in some embodiments, functions when the endpoints are not configured for use with MPTCP, or the multiple paths are not directly connected to the endpoints (i.e., the multiple paths are known only to intermediate hops between the endpoints). 2) MPTCP attempts to be fair when competing with other non-MPTCP flows over congested links ("concatenated congestion control"). Without Explicit Congestion Notification (ECN), which is not widely deployed, an MPTCP system must guess whether any congestion it sees is on a link shared by all subflows. It often guesses incorrectly, resulting in lower-than-expected throughput.

[0062] As illustrated in Figure 1, system 100 is shown configured to utilize an improved scheduling technique on the transmitter side of the system and a buffering system on the receiver side that provides packet ordering. The components shown are, in one embodiment, hardware components configured to interact with one another. In another embodiment, the components are not separate components, and two or more of the components may be implemented on a particular hardware component (e.g., a computer chip that performs the functions of two or more components).

[0063] In some embodiments, the components reside on the same platform (e.g., the same printed circuit board), and system 100 is a single device that can be moved and connected to a data center / field carry-able device (e.g., a rugged mobile transmitter), etc. In other embodiments, the components are distributed and may not all be located in close proximity, but rather communicate electronically through telecommunications (e.g., processing and control is not performed locally, but rather by components that reside in a distributed resource environment (e.g., the cloud)).

[0064] In mobile scenarios, providing combined connectivity is particularly desirable when signal quality, network availability, network quality, etc., are suboptimal (e.g., professional reporting / video production may occur in locations without a strong network infrastructure).

[0065] Shown are many different data connections 106 (e.g., "paths") representing one or more networks (or network channels) labeled as Connection 1, Connection 2...Connection N. There may be multiple data connections / paths through a single network or multiple data connections that can use one or more networks.

[0066] The system 100 may be configured to communicate with various endpoints 102, 110 or applications that do not need to have any knowledge about the multiple paths / connections 106 used to request and receive data (e.g., the endpoints 102, 110 may function independently of the paths or connections 106). Received data may, for example, be reconstructed so that the original transmission can be recreated from the contributions of the different paths / connections 106 (an example usage scenario is the regeneration of video by a receiver configured to be plugged into a server rack in a data center facility and integrated with existing broadcast infrastructure to provide improved networking capabilities).

[0067] The system 100 receives input (data flow) from a source endpoint 102 and schedules the improved distribution of data packets over various connections 106, then sequences the data packets at the other end of the system 108 before transmission to a destination endpoint application 110. In doing so, the system 100 is configured to increase bandwidth to approach the sum of the maximum bandwidths of the various available paths. Compared to using a single connection, the system 100 also provides improved reliability, which can be an important consideration in time-sensitive scenarios such as live event coverage while the event is taking place. These events may have high signal congestion (e.g., a sporting event) or unreliability along one or more paths (e.g., news coverage after a natural disaster).

[0068] In various embodiments, both the scheduler and sequencer may be provided from the cloud computing implementation, or at the endpoint (before the data is consumed by an application at the endpoint), or various combinations thereof.

[0069] System 100 may be tuned to optimize and / or prioritize, among other things, performance, best latency, best throughput, minimum jitter (variation in latency on packet flows between two systems), connection cost, and combinations of connections for particular flows (e.g., if system 100 has information that a transmission (data flow) is of content type X, system 100 may be configured to use only data connections with similar latency, while content type Y may allow a wider mix of data connections (or may require a greater net capacity that can only be achieved by combining data connections)). This tuning may be provided for the system generally, or may be specific to each flow (or set of flows based on location, start point or endpoint owner or a combination thereof, transmission time, set of available communication links, security required for transmission, etc.).

[0070] The system 100 may generally be bidirectional in that each gateway 104, 108 generally has a scheduler and sequencer for handling TCP traffic (or UDP traffic, a combination of TCP and UDP traffic, or any kind of general IP traffic), although in some embodiments only one gateway may be required (see, e.g., FIG. 11, which shows simple disaster recovery as a service).

[0071] The scheduling portion of the present system features a novel approach to estimating the bandwidth of a given connection (see, for example, Figure 8).

[0072] System 100 can be utilized for a variety of scenarios, for example, as a failover or addition to an existing Internet connection (e.g., a VoIP telephone system, or a corporate connection to the Web), whereby an additional network (or path) is added to seamlessly replace a dropped primary Internet connection, or where combining is used to include only more expensive networks when the primary Internet connection is saturated (as shown in Figures 11 and 12). Another use is to provide a means of maximizing the use of high-cost (often sunk costs), highly reliable data connections such as satellite by allowing traffic to be offloaded to other data connections with different attributes (Figures 9 and 10).

[0073] In some embodiments, the system is a network gateway configured for routing data flows through multiple network connections.

[0074] 1 provides a schematic of a system having two gateways 104 and 108, each comprising a buffer manager 150, an action engine 152, a connection controller 154, a flow classification engine 156 (responsible for flow identification and classification), a scheduler 158, a sequencer 160, and a network characteristic monitoring unit 161, and linked by N data connections 106, with each gateway connected to a particular endpoint 102, 110. Reference letters A and B are used to distinguish between the respective components of the two gateways 104 and 108.

[0075] Each gateway 104 and 108 is configured to include multiple network interfaces for transmitting data through the multiple network connections, and is a device (e.g., comprising configured hardware, software, or embedded firmware) comprising a processor configured to monitor time-varying network transmission characteristics of the multiple network connections; analyze at least one packet of a data flow of packets to identify a data flow class for the data flow, the data flow class defining or otherwise associated with at least one network interface requirement of the data flow; and route the packets in the data flow through the multiple network connections based on the data flow class and the time-varying network transmission characteristics.

[0076] The buffer manager 150 is configured to configure buffers within the gateway that are configured to more efficiently manage traffic (both individual flows and combinations of multiple simultaneous flows passing through the system). In some embodiments, the buffer manager is a separate processor. In other embodiments, the buffer manager is a computational unit provided by a processor that is configured to perform, among other activities, buffer management (150).

[0077] The action engine 152 is configured to apply one or more deterministic methods and / or logical operations based on received input data sets (e.g., feedback information, network congestion information, transmission characteristics) to inform the system about constraints that apply to the combined connections, either per user / client, destination / server, connection (e.g., latency, throughput, cost, jitter, reliability), or flow type / request (e.g., FTP vs. HTTP vs. streaming video). For example, the action engine 152 may be configured to restrict certain types of flows to a particular connection or set of data connections based on cost, while reliability and low latency may be more important for different users or flow types. For example, different conditions, triggers, and methods may be utilized depending on one or more elements of the known information. The action engine 152 may, for example, be located in the same or a different processor as the buffer manager 150.

[0078] The action engine 152 may be configured to generate, apply, or otherwise manipulate or use one or more rule sets that determine the logical operations that control routing over the N data connections 106.

[0079] The flow classification engine 156 is configured to evaluate each data flow received by the multipath gateway 104 for transmission and to apply a flow classification technique to determine, if not already known, the type of traffic being transmitted and its requirements. In some embodiments, deep packet inspection techniques are configured to make the determination. In other embodiments, the evaluation is based on heuristics or on data flows that are marked or labeled as they are created. In other embodiments, the evaluation is based on rules provided by a user / administrator of the system. In other embodiments, a combination of methods is used. The flow classification engine 156 is configured for use with one or more network interfaces and may be implemented using electronic circuitry or a processor.

[0080] Flow identification can be performed by analyzing the information provided in the packets of the data flow, for example by examining packet header information (e.g., source / destination IP, transport protocol, transport protocol port numbers, DSCP flags, etc.).

[0081] As provided in some embodiments, different levels of discrimination may be performed, for example, the contents of the packet body may be further inspected, for example using deep packet inspection techniques.

[0082] Once the flows are identified, classification may involve categorizing the flows based on their requirements. Examples of classification include: 1. Low latency, low to moderate jitter, out-of-order packets, high bandwidth (live HD video broadcast), 2. Especially low latency, low to moderate jitter, packets that may be out of order, medium bandwidth (Skype™, FaceTime™) (jitter can cause artifacts or degradation, which is problematic for real-time communication), 3. Low latency, low to moderate jitter, out-of-order packets are acceptable, low bandwidth (DNS, VoIP), 4. Low latency, no jitter requirements, in-order packets preferred, low bandwidth (interactive SSH) 5. No latency requirements, no jitter requirements, in-order packets preferred, medium to high bandwidth (e.g., SCP, SFTP, FTP, HTTP), and 6. No latency requirements, no jitter requirements, in-order packets preferred, no bandwidth requirements, sustained large data transfers (e.g., file / system backups), etc. is.

[0083] The one or more dimensions that can be classified include, but are not limited to: a.Latency, b. Bandwidth / Throughput, c. Jitter, d. Packet ordering, and e. Data transfer volume Examples include:

[0084] As described further below, these classification dimensions are useful for improving efficient communication flows. Latency and bandwidth / throughput considerations are particularly important when flows with conflicting requirements exist. Example embodiments in which jitter is addressed are described further below, and the system may be configured to accommodate jitter, for example, by buffering in the scheduler or by maintaining flows sticky to particular connections. Packet ordering is described further below with examples specifically for TCP, and data volume is relevant when it can be used as an indicator that allows flows to be reclassified from one type (low latency, low bandwidth) to another (non-latency sensitive, high bandwidth).

[0085] Other classification dimensions and classifications are possible, and the above are provided as example classifications. Different dimensions or classifications and / or combinations thereof may be created. For example, in media broadcasting, the system may be configured to classify video data and metadata associated with clips (e.g., GPS information, timing information, labels) or FEC data associated with a video stream.

[0086] Flow classification can be used to remove and / or filter out transmissions that the system is configured to prevent (e.g., peer-to-peer file sharing in some cases, or material known to be under copyright) or traffic that the system may be configured to favor in the context of providing tiered services (e.g., other favoring particular users or software programs).

[0087] For example, in an Internet backup usage scenario, the system may be configured so that VoIP calls to / from a support organization receive a higher level of service than calls within that organization, even though the combined backup may be limited in availability (where the system may generate instructions for endpoints to degrade the voice quality of some calls more than others when under constraint, or to degrade certain calls altogether when there are bandwidth constraints).

[0088] Scheduler 160 is configured to make decisions regarding which packets should be sent to which connections 106. Scheduler 160 may be considered an improved QoS engine. Scheduler 160, in some embodiments, is implemented using one or more processors or standalone chips or configured circuits, such as comparator circuits or FPGAs. Scheduler 160 may comprise a series of logic gates that are checked to make decisions.

[0089] A typical QoS engine manages a single connection, and although it may be configured to perform flow identification and classification, the end result is that the QoS engine reorders packets and then sends them out on a single connection.

[0090] In contrast, scheduler 160 is configured to perform flow identification, classification, and packet ordering, but in some embodiments scheduler 160 is further configured to make decisions about which connections to send packets to in order to provide improved transmission characteristics for the data flow and / or satisfy policies set for the flow by a user / administrator (or set forth in various rules). Scheduler 160 may modify network interface operating characteristics, for example, by transmitting a set of control signals to the network interfaces to switch them on or off or indicate which ones data should be routed through. The control signals may be a set of instructions that indicate specific characteristics of the desired routing, such as packet timing, reservation of network interfaces for particular types of traffic, etc.

[0091] For example, consider two connections with the following characteristics: 1) Connection 1: 1ms round trip time (RTT), estimated bandwidth of 0.5Mbps, and 2) Connection 2: 30ms RTT, estimated bandwidth of 10Mbps.

[0092] Scheduler 160 may attempt to reserve connection 1 exclusively for DNS traffic (small packets, low latency). In this example, there may be so much DNS traffic that it reaches the capacity of connection 1, and scheduler 160 may be configured to overflow traffic onto connection 2, although scheduler 160 may selectively do so based on other decisions or factors. For example, if scheduler 160 is configured to provide fair decisions, scheduler 160 may be configured to first overflow traffic from IP addresses that have already sent a significant amount of DNS traffic within the past X seconds.

[0093] Scheduler 160 may be configured to process decisions based on a process or method operating, for example, in conjunction with one or more processors or similar implementations in hardware (e.g., FPGAs). These devices may be configured for operation under the control of action engine 152 to divide the data stream into data packets and then route the data packets into buffers (managed by a buffer manager) that send the data packets to the data connection according to rules that attempt to optimize packet delivery while taking into account the characteristics of the data connection.

[0094] The primary criteria are latency and bandwidth-based in some embodiments, but in some embodiments, the path maximum transmission unit (PMTU) may also be utilized. For example, if one connection has a significantly smaller PMTU than another connection (e.g., 500 bytes versus 1500 bytes), it may be designated as a bad candidate for overflow (and may be avoided or deprioritized, for example) because packets sent on that connection will need to be fragmented.

[0095] Scheduler 160, in some embodiments, need not be configured to communicate packets through in order, but rather to communicate packets over various connections to meet or exceed desired QoS / QoE metrics (some of which may be defined by a network controller and others by a user / customer). In cases where packets may be communicated out of order, sequencer 162 and buffer manager may be utilized to reorder received packets.

[0096] Sequential bursts of packets are transmitted through a network interface, and an estimate of the bandwidth of the first network interface is generated based on timestamps and packet sizes recorded when packets in the sequential burst of packets are received at the receiving node. The estimate is then utilized to route packets in the sequential packet data flow through the set of network connections based on the generated bandwidth of the first network interface. As described below, in some embodiments, the bandwidth estimate is generated based on timestamps of the first or last packet in a burst and packets in uncoalesced bursts to estimate a lower bandwidth value and an upper bandwidth value (e.g., by packet substitution). The transmitted packets may be test packets, test packets "piggybacking" on data packets, or hybrid packets. When data packets are used for "piggybacking," some embodiments include flagging such data packets for increased redundancy (e.g., to increase tolerance for lost packets, particularly for packets used for bandwidth testing purposes).

[0097] The sequential packets may be received in order, or within an allowable deviation from that order, so that sequencer 162 can reconstruct the packets for consumption. In some embodiments, sequencer 162 is a physical hardware device that can be incorporated into a broadcast infrastructure that receives the signal and generates an output signal that is the reconstructed signal. For example, the physical hardware device may be a rack-mounted appliance that serves as the first stage for receiving and reconstructing the signal.

[0098] Sequencer 162 is configured to sequence packets received at an endpoint and deliver them to an application in an acceptable order, which in some embodiments follows the original order, to reduce unnecessary packet reordering or other error correction for the flow. In other embodiments, the order is within acceptable limits of order such that sequencer 162 can still reconstruct the data flow. Sequencer 162 may include buffers or other mechanisms to smooth the received flow, for example, and in some embodiments is configured to control the sending of acknowledgments and the storing of packets based on monitoring the transmission characteristics of multiple network connections and unequal distribution in the receipt of sequential packets in the data flow.

[0099] The sequencer 162 may be implemented in hardware (e.g., a field programmable gate array) that is, for example, provided on a processor or under the control of an action engine 152 configured to reconstruct the data flow from received data packets extracted from a buffer.

[0100] The sequencer 162 is configured to hide, on a per-flow basis, latency differences between multiple connections that are not tolerated for each flow.

[0101] The action engine 152 can act as an aggregator of information provided by other components (including 154) and instructs the sequencer 162 via one or more control signals that indicate how the sequencer 162 should operate for a given flow.

[0102] When a system configured for a protocol such as TCP receives packets, the system is typically configured to expect (but not require) packets to arrive in order, although the system is configured to establish a time bound (usually some multiple of the round trip time, or RTT) for when it expects packets to arrive out of order. The system may also be configured to retransmit lost packets as soon as a heuristic-based time bound is reached (e.g., the first retransmission triggered by three DUP ACKs).

[0103] If packets arrive at sequencer 162 on connections with significantly different latencies, sequencer 162 (on a per-flow basis) may be configured to buffer the packets until they are approximately the same age (delay) before sending them onward toward their destination. Sequencer 162 does this, for example, if the flows require matching latency and low jitter.

[0104] The sequencer 162 does not necessarily need to provide reliable, strictly in-order delivery of data packets, but in some embodiments is configured to provide what is necessary so that the system using the protocol (e.g., TCP or applications over UDP) does not prematurely determine that packets are being lost by the network.

[0105] In some embodiments, sequencer 162 is configured to monitor the latency variation (jitter) of each data connection along with packet loss (based on data maintained by operations engine 152) to predict, based on connection reliability, which data connections are likely to cause packets to be delayed by a flow more than expected (meaning that endpoints 102 and 110 will consider them lost and invoke their error correction routines).

[0106] In the case of an out-of-order situation, sequencer 162 may, for example, utilize a larger jitter buffer on connections that exhibit greater latency variation. In the case of packet retransmissions, sequencer 162 may be configured to immediately request the lost packet through the "best" (most reliable, lowest latency) connection.

[0107] In one example scenario, the bandwidth-delay product estimate may not be entirely accurate and latency spikes are experienced in the connection, resulting in packets being received out of order at intermediate gateways.

[0108] In these embodiments, sequencer 162 may be configured to make predictive decisions regarding how a protocol (and / or associated application) can act with respect to out-of-order packets and to generate instructions to reorder packets so that downstream systems are less likely to erroneously assume that the network has reached capacity (and thus throttle its transmission rate) and / or unnecessarily request retransmission of packets that were not lost.

[0109] For example, many TCP implementations use three consecutive double acknowledgments (DUP ACK) as a hint that the packet after the DUP ACK may be lost. In this example, if the receiver receives packets 1, 2, 4, 5, and 6, it will send ACK(2) three times (once for each of packets 4 / 5 / 6). The sender then recognizes that this event may hint that packet 3 may be lost in the network, and is configured to preemptively retransmit it before any normal retransmission timeout (RTO) timer expires.

[0110] In some embodiments, sequencer 162 may be configured to compensate for such predictive decisions. As in the example above, if sequencer 162 has buffered packets 1, 2, 4, 5, 6, and 3, sequencer 162 may reorder the packets to ensure that the packets are transmitted in their proper order. However, if packets were already buffered in the order 1, 2, 4, 3, 5, and 6, sequencer 162 may be configured not to bother reordering them before transmission, since in this example no predictive decision would be triggered (given the positioning of packet 3).

[0111] The connection controller 154 is configured to perform the actual routing between the different connection paths 106, which is provided, for example, to indicate that the connections 106 to the combined links do not have to reside on the physical gateways 104, 108 (e.g., a physical gateway may have several links (Ethernet or otherwise) to physical transmit / receive equipment or satellite equipment that may be located elsewhere (and may be at different locations with respect to antennas, etc.)). Thus, the endpoints may be logically connected and physically separated in various ways.

[0112] In one embodiment, system 100 is configured to provide a method known as TCP acceleration, in which the gateway creates a pre-buffer as it receives a packet and provides an acknowledgment signal (e.g., an ACK flag) to the sending endpoint as if the receiving endpoint had already received the packet, allowing the sending endpoint 102 to send more packets into system 100 before the actual packet is delivered to the endpoint. In some embodiments, pre-buffering is used for TCP acceleration (opportunistic acknowledgment (ACKing) combined with buffering of available data).

[0113] This pre-buffer may reside before the first link to the transmitting endpoint 102 or elsewhere in the transmission path to the endpoint 110. The size of this pre-buffer may vary depending on feedback from the multipath network, which in some embodiments is based on an estimate or measurement of the bandwidth-delay product or a set of predetermined logical operations (where a particular application or user receives a pre-buffer with particular characteristics such as speed, latency, throughput, etc.).

[0114] Pre-buffers may exist at various locations within an implementation, for example, a pre-buffer may exist either at the entry point to gateway 104 or along the transmission path (before the final destination) to 110. In one embodiment, there is a series of pre-buffers, for example, data flows from endpoint 1 to endpoint 2, so there is a pre-buffer on both gateway A and gateway B.

[0115] It becomes the responsibility of the system 100 to reliably transmit data that has been accepted into the pre-buffer and opportunistically acknowledged by the endpoint 102 to the endpoint 110. The acknowledgement tells the original endpoint 102 that the endpoint 110 has received the data and therefore no longer needs to handle retransmissions via its normal TCP mechanisms.

[0116] Pre-buffering and opportunistic acknowledgments are advantageous because they remove the time limit that system 100 has available to handle connection 106 loss and other non-ideal behavior. The time limit without TCP acceleration is based on the TCP RTO calculated by endpoint 102, a value outside the control of system 100. If this time limit is exceeded, endpoint 102 a) retransmits data that system 100 may already have buffered, and b) decreases its cwnd, thus reducing throughput.

[0117] The size of the pre-buffer may need to be limited to limit memory usage, which requires the communication of flow control information between the multipath gateways 104 and 108. For example, if the communication link between the gateway 108 and the endpoint 110 has a lower throughput than the aggregate throughput of all connections 106, the amount of data buffered at 108 will continually increase. When the buffered amount exceeds the limit, a flow control message is sent to 104 instructing it to temporarily stop opportunistically acknowledging data sent from the endpoint 102. When the buffered amount eventually drops below the limit, a flow control message is sent to 104 instructing it to resume opportunistic acknowledgment. The limit may be a static threshold or may be dynamically calculated, taking into account factors such as the total BDP of all connections 106 and the total number of data flows currently being handled by the system. The thresholds at which flow control start / stop messages are sent need not be the same (e.g., there may be hysteresis).

[0118] Similarly, flow control signal information may exist within the multipath gateway itself. For example, if the aggregate throughput of connection 106 is less than the throughput between endpoint 102 and gateway 104, the pre-buffer within 104 will continually increase. After exceeding a limit (which may be calculated as described above), opportunistic acknowledgment of incoming data from endpoint 102 may need to be temporarily halted, and then resumed when the amount of data drops below an appropriate threshold.

[0119] The previous example describes the TCP acceleration method for data sent from endpoint 102 to 110. The same description applies to data sent in the opposite direction.

[0120] In another embodiment, the buffer manager is configured to provide excess buffering for ongoing transmissions per communication link to compensate for variability in the behavior of the connecting network and other activity on the network and the potentially "bursty" nature of the source transmissions.

[0121] Excessive buffering may refer to, for example, being able to handle intentionally accepting more packets at the input than the connection's BDP at the output. The difference between "excessive buffering" and "buffering" is that the buffer manager can buffer different amounts based on flow demand and on how the connection BDP changes in real time.

[0122] This over-buffering causes the gateway 104 to accept and buffer more data from the sending endpoint 102 than it would otherwise be prepared to accommodate (e.g., more than it can "easily handle"). Over-buffering can be done globally (e.g., the system is configured to utilize more than the system estimates is available in total throughput), or it can be moved and managed on a per-connection basis into the connection controller, or a combination of both (e.g., multiple over-buffers per transmission) can be provided.

[0123] For example, even if system 100 only estimates that it can transmit 20 Mbps over a given set of links, system 100 may receive more than that (e.g., 30 Mbps) from transmitting endpoint 102 for the time being and buffer data that it cannot immediately transmit, perhaps based on statistical and historical knowledge of network characteristics provided by network characteristic monitoring unit 161 to determine that network conditions may change, or that there may be times when transmitting endpoint 102 (or other incoming or outgoing transmissions) may slow down its data transmission rate.

[0124] The flow classification engine 156, in some embodiments, is configured to flag certain types of traffic, and the action engine 152, in some embodiments, may be configured to instruct the buffer manager to size and manage pre-buffering and / or over-buffering on a per-flow basis, selecting buffer sizes based on any number of criteria (type of data, user, historical behavioral data, flow requirements).

[0125] In some embodiments, the size of these buffers is determined per transmission and also per gateway (since there may be many transmissions being routed through the gateway at one time). In one embodiment, pre-buffering and over-buffering techniques are utilized in parallel.

[0126] In some embodiments, the size of the excess buffering is determined to be substantially proportional to the bandwidth-delay product (BDP). For example, the system may be configured such that if the network has a high BDP (e.g., 10 Mbps @ 400 ms => 500 KB), the buffers should be larger so that there is always enough data available to keep the network / pipeline full of packets. Conversely, for low BDP networks, the system may be configured such that there is less buffering to prevent excessive buffer bloat.

[0127] Buffer bloat may refer to excessive buffering in a network, for example, resulting in high latency and low throughput. Given the advent of cheaper and more readily available memory, many devices now utilize excessive buffers without considering the impact of such buffers. Buffer bloat is described in more detail in papers published by the Association for Computing Machinery, such as a paper entitled "Bufferbloat: What's Wrong with the Internet?" (December 7, 2011) and a paper entitled "Bufferbloat: Dark Buffers in the Internet" (November 29, 2011), both of which are incorporated herein by reference.

[0128] As an example of determining excess buffering size in relation to bitrate and latency, a rule may be implemented requiring the system to not add more than 50% to the base latency of the network due to excess buffering. In this example, the rule indicates that excess buffering size is "bitrate x base latency x 1.5".

[0129] In one embodiment, the action engine 152 may be included in the multi-path gateway 104, 108. In another embodiment, the action engine 152 may reside in the cloud and apply to one or more gateways 104, 108. In one embodiment, there may be multiple endpoints 102, 110 connecting to a single multi-path gateway 104, 108. In one embodiment, the endpoints 102, 110 and the multi-path gateway 104, 108 may reside on the same device.

[0130] In one embodiment, the connection controller 154 may be separate from the multi-path gateways 104, 108 (and may be physically associated with one or more connection devices (e.g., wireless interfaces or wired connections)). In another embodiment, the connection controller may reside on the gateway, but the physical connections (e.g., interfaces or wired connections) may reside on separate units, devices, or multiple devices.

[0131] Although endpoints must be logically connected, they may be connected independently of the connections 106 (e.g., communications handled by multipath gateways 104, 108). In some embodiments, the set of connections 106 available to a given gateway may be dynamic (e.g., certain networks only available at certain times or to certain users).

[0132] In one embodiment, the traffic coming from the endpoint 102 may be controllable by the system 100 based on dynamic feedback from the system 100 (e.g., the system may be configured to change the bit rate of a video transmission originating at the endpoint). In another embodiment, the traffic coming from the endpoint 102 may not be controllable by the system 100 (e.g., web requests originating from the endpoint).

[0133] In other embodiments, there may be more than one set of multipath gateways 104, 108 in the transmission path (e.g., FIG. 13). For example, in some implementations, there may be a TCP transmission with a remote multipath gateway connecting to a gateway in the cloud, which then provides a non-multipath connection to another multipath gateway at the edge of the cloud, which then transmits to another remote multipath gateway.

[0134] Various use cases may be possible, such as a military use case where a remote field operator may need to transmit large volumes of data to another remote location. The operator's system 100 may be set up with a transmission mechanism where multiple paths are utilized to provide the data to the wider Internet. The system 100 then uses a high-capacity backhaul to transmit somewhere else on the edge of the Internet, where it then requires another multipath transmission to reach a second remote endpoint.

[0135] In one embodiment, gateways A 104 and B 108 may be configured to transmit control information between each other via one of the available connection paths.

[0136] Figure 2 illustrates a scenario in which an Internet user connects to the Web through a combination of gateways that utilize transport links with different network characteristics.

[0137] Figure 2 is related to Figure 1, but shows different examples of interactions between client 202 and server 204. Although the system may ultimately be a two-way system, there may be situations where transmission may be unidirectional or initially unidirectional (e.g., requesting data from a web page that provides the data).

[0138] A device that may be a client 202 may be any type of device capable of communicating, such as, for example, an Internet of Things (IoT) device. Another endpoint may be a server, such as an IoT hub. The illustrated connection 206 may include, for example, an LTE network, a 3G network, a satellite connection, etc. Other connection types include wired networks, WiFi networks (e.g., 802.11 standards), Bluetooth, microwave connections, optical connections, and other wireless connections, among others. The reliability and connectivity of each of these networks and / or connections may be affected by various factors, such as multipath fading, interference, and noise.

[0139] Bandwidth, latency, jitter, packet loss, etc. may be affected, and in some embodiments, the endpoints 202, 210 and / or multipath gateways 204, 208 may be moving, and even the number and type of available connections 206 may become dynamically available or unavailable. For example, a telecommunications-equipped VAN may move in and out of cellular coverage, satellite coverage, etc.

[0140] Figure 3 is provided to show a more detailed example of the method performed by sequencer 160 by demonstrating the ordering of six packets according to an example based on the network of Figure 2. In the example of Figure 3, packets are transmitted in a simple round-robin fashion. The example is non-limiting and other packet transmission methods are possible.

[0141] With particular focus on latency characteristics, we consider three types of connections between the scheduler in the multipath transmitter and the sequencer in the multipath receiver: 1.LTE: 30Mbps, 50ms latency, 2.3G: 2Mbps, 120ms latency, and 3. Satellite: 10Mbps, 500ms latency Consider the following.

[0142] In this example, assume that a client machine (C) 302 connected via a multipath transmitter 304 is attempting to send TCP traffic to a server (S) 310 reachable via a multipath receiver 308, and that the system 300 is not using the TCP acceleration techniques described above.

[0143] Consider a simple algorithm that round-robins IP packets from 6 of these connections, and does nothing but be "smart." Due to the extreme difference in latency, the server (S) receiving the TCP stream will send packets in the following order: Packet 1 @ t = 50 ms (sent over LTE) Packet 4 @ t = 50 + ms (sent over LTE) Packet 2 @ t = 120 ms (sent over 3G) Packet 5 @ t = 120 + ms (sent over 3G) Packet 3 @ t = 500 ms (sent via satellite) Packet 6 @ t = 500 + ms (sent via satellite) Observe the packets arriving at

[0144] TCP is not well designed to handle major reordering of packets and large latencies between them. The server will send multiple double ACKs for packet 1 (to make sure the client knows the server is still waiting for the lost packet), causing any / all packets 2 / 3 / 5 / 6 to be retransmitted, which is unnecessary since the packets are not lost, they are just still in transit. For this reason, many existing solutions provide very poor throughput, especially when trying to combine connections with very varied (or different) latencies.

[0145] The ordering function of the sequencer 162 of the gateway 104, 108 provides a technical solution by buffering packets and sending them in order so that the server in question ultimately receives the packets in the order in which they were sent. The resulting order is: Packet 1 @ t = 50 ms (sent over LTE) Packet 2 @ t = 120 ms (sent over 3G) Packet 3 @ t = 500 ms (sent via satellite) Packet 4 @ t = 500 + ms (sent over LTE and buffered by the sequencer) Packet 5 @ t = 500 + ms (sent over 3G and buffered by the sequencer) Packet 6 @ t = 500 + ms (sent via satellite) is.

[0146] In determining how large the buffer should be, scheduler 160 may be configured to base the determination at least in part on the bandwidth-delay product. In some embodiments, the length of the buffer may be sized based on, for example, flow demand, jitter, historical data for the performance of the set of connections (e.g., how variable delivery has been for those connections, how often the system has observed re-requesting packets that arrived quickly (e.g., the buffer may be too small), or the number of times minimal sorting was required by the sequencer (e.g., the buffer may be unnecessarily large)).

[0147] For example, if many similar networks with similar latencies are used, scheduler 160 may provide a smaller receive / sequencer buffer since there is less opportunity for mis-ordering to occur.

[0148] On the other hand, the system may be configured to select larger buffers on the scheduler to ensure that the network / pipeline can always remain full if either the sending endpoint or the connection is "bursty." Buffer sizing may depend on either side or flow requirements. If lower latency is required for endpoint 2 and the connection is otherwise good / predictable, or if the flow does not have strict requirements for packet ordering, buffer sizing will tend to result in smaller buffers on the sequencer.

[0149] The buffer on scheduler 160 may be configured to ensure that connection 106 is filled by endpoint 1, and the buffer on sequencer 162 may be configured to minimize the need to re-request packets, but not to be too large, causing unnecessary delays in delivery of packets to endpoint 2.

[0150] In some embodiments, scheduler 160 and sequencer 162 are not configured to provide completely guaranteed in-order, reliable delivery of packets; instead, scheduler 160 and sequencer 162 only need to reorder packets sufficiently so that a protocol (e.g., TCP) or application does not erroneously treat packets that are still buffered / forwarded as lost. For example, various protocols / applications may use multiple RTTs as an indicator to determine whether a packet is lost. Accordingly, scheduler 160 and / or sequencer 162 may be operable to reduce extreme variability in packet buffering times.

[0151] In one embodiment, the system provides buffers on both sides of the system to maximize throughput, where a first buffer releases content to the network upon a trigger. In one embodiment, the two buffers can be coordinated together (in the scheduler 160 of gateway A and the sequencer of gateway B) to maintain a particular overall buffer length. Thus, the buffers may operate in unison to provide an overall buffer length and improve throughput while avoiding excessively long buffers (buffer bloat) on either the sender or receiver. In another embodiment, the buffers may operate independently of each other.

[0152] Figure 4 illustrates the case where scheduler 160 is configured to be "smart" and to utilize latency in its decisions. In Figure 4, a simplified example in which three networks have the same bandwidth but different latencies is provided for illustrative purposes.

[0153] In this example, the bandwidth is defined as "1 packet" for computational simplicity, although other bandwidths are possible. This assumption allows for a simplified analysis in which packet 1 arrives at t=50 ms and packet 2 arrives at t=100 ms. If the bandwidth of the connection were greater than 1 packet, packet 2 would arrive at t=51 ms because it is pipelined after the first packet (assuming a processing / pipelining delay of, say, 1 ms). In another example, if one desires to define the analysis in terms of Mbps or Kbps, the derivation may include the assumption that all packets are 1500 bytes and that the connection has a bandwidth of 12 Kbps (1500 x 8). Other assumptions are possible, for example: Connection 1 - 1 packet bandwidth, latency 50ms, Connection 2 - 1 packet bandwidth, 120ms latency, and Connection 3 - 1 packet bandwidth, 500ms latency is.

[0154] Smart scheduler 160 is configured to route packets such that the potentially time-varying latencies of the various links are taken into account in real time, which results in instances where scheduler 160 only sends packets to connections 1 and 2 because all packets can be acknowledged faster (e.g., based on RTT) than it can send just one packet over connection 3.

[0155] The sequencer 162 on the multipath receiver 408 performs the following sequence: Packet 1 @ t = 50ms (sent over connection 1) Packet 2 @ t=100ms (sent over connection 1) Packet 3 @ t=120ms (sent over connection 2) Packet 4 @ t=150ms (sent over connection 1) Packet 5 @ t = 200 ms (sent over connection 1) Packet 6 @ t=240ms (sent over connection 2) Receive packets.

[0156] Assuming there are no dropped packets or congestion on the network, packets will arrive in order, and sequencer 162 simply passes packets through in order without the need for reordering or buffering. In an embodiment where multiple connections have similar characteristics, the system may select to transmit packets based on historical connection characteristics and reliability.

[0157] If packets are received out of order or lost, sequencer 162 may be configured to request that the packet be retransmitted over the determined most reliable connection. Multipath receiver 408 may, for example, be configured to attempt to maintain the total latency of the flow so that it has low variability. If multipath receiver 408 expects a packet and it has not yet arrived (because the packet was lost or the connection reordered it unexpectedly), whether to make a re-request or just continue (depending on the protocol (TCP) or application for handling loss) may depend, for example, on the flow and the determined impact on the request.

[0158] 5 is provided to illustrate the use of bandwidth-delay product (BDP) and stickiness for multiple data connection routing of TCP packets. An input device 502 is shown sending packets to an output device 510 via a transmitter 504, connection 506, and receiver 508, and the system 500 is shown not employing the TCP acceleration techniques described above.

[0159] Considering the example in Figure 5, each of the three types of connections shown in 506 has a different bandwidth-delay product (BDP). The BDP and latency shown are provided merely as examples, and other values ​​are possible. The BDP may be determined, for example, by multiplying the bandwidth by the latency.

[0160] The 3G link (e.g., connection 1) has the smallest BDP (2Mbps x 120ms = approximately 30KB), and once it becomes full (i.e., approximately 30KB of unacknowledged packets have been sent and are in-flight in the pipeline), no other data packets should be sent over that link; otherwise, the data packets will be dropped and / or buffered, which will cause TCP flows to falsely believe they have reached channel capacity and increase the latency of the connection.

[0161] Therefore, the correct approach is to instead send the data packet over a different link (e.g., connection 2 or 3). Only when all links have reached their full BDP should system 100 be configured to drop the packet (or, optionally, send it over any of the three connections), thereby causing the TCP flow to ultimately determine that it has reached its total channel capacity and forcing the TCP flow (e.g., sending source 502) to reduce its total transmission rate in response.

[0162] A technical challenge that arises is that system 100 typically does not have information about the capacity of each connection. In some embodiments, bandwidth estimation techniques are applied to estimate bandwidth values ​​taking into account incomplete information so that improved routing decisions can be applied (e.g., probabilistically improving expected throughput over simple bandwidth estimation).

[0163] A bandwidth estimate (even if only slightly accurate) is useful because without it, the system 100 can only determine when the connection 106 has reached channel capacity if the system 100 observes increased packet loss and / or latency. With an estimate, the system 100 can be advantageously configured to use techniques such as adding a percentage of FEC when the transmission rate begins to approach the estimate.

[0164] When the receiver 108 needs to start using FEC, the transmitter 104 may determine that this indicates that the channel capacity has been reached. This is an improvement over systems that do not use FEC because the channel capacity is determined without introducing any unrecoverable loss.

[0165] In some embodiments, a more accurate bandwidth estimation technique may be useful in determining that the system 100 should invoke an improved transmission mechanism, such as weighted round robin.

[0166] However, even with a perfect bandwidth estimation algorithm, system 100 may not be optimal for utilizing a weighted round-robin mechanism. If a flow does not require the full channel capacity, all of its packets should remain "sticky" to a single connection so that no buffering / reordering is required at the receiver. That is, once a flow is assigned to a data connection, it should continue to use that connection. Only if the needs of the flow exceed the capacity of its assigned data connection should the flow "overflow" to another data connection.

[0167] The selection of the overflow connection can incorporate any number of parameters into the decision logic to provide various advantages in terms of latency, throughput, reliability, jitter, cost, etc., or any combination of these factors. For example, the LTE BDP is 30 Mbps x 50 ms = approximately 187.5 KB, and the satellite link is 10 Mbps x 500 ms = approximately 625 KB. If the satellite link is expensive and the goal is low latency and low cost, the LTE connection will be "filled" with data before the satellite connection is used. Alternatively, if the satellite connection is significantly more reliable than the LTE connection and the goal is reliability, the satellite connection will be "filled" with data before the LTE connection is utilized.

[0168] Connections 106 may also be advantageously pooled so that connections with similar characteristics (latency, throughput, jitter, and packet loss) are grouped together. This simplifies the connection management that must be performed by the connection controller 154, as pooling means that fewer decisions need to be made within the connection controller. As connection characteristics change over time, connections may be merged and separated from the pool in real time as needed to ensure reliable data communication. Any connection with characteristics that are significantly different from all other connections is not pooled and is used as an individual connection.

[0169] Pooled connections also make it possible to meet the needs of flows with conflicting requirements. For example, consider a set of connections 106 that all have consistently low latency but widely varying throughput. In this scenario, significant excess buffering is required to achieve the maximum possible throughput, which can result in a significant increase in buffer bloat. If two flows are currently active, one requiring high throughput and the other requiring low latency and low throughput, by dividing the available connections 106 into two pools, one utilizing significant excess buffering and the other not, both flows can achieve their ideal quality of experience.

[0170] An illustration of a bandwidth estimation technique is provided in Figures 8A, 8B, 8C, 8D, and 8E, which includes an example estimation that uses known information about network capabilities to estimate available bandwidth based on the number of packets to be transmitted and packet size. In actual implementations, corrections are applied to compensate for non-idealities, which may result in coalescing or other issues with respect to interrupt arbitration. An error level associated with each packet may be determined and used to improve bandwidth estimation, and this error may depend on the bit rate (e.g., as the bit rate increases, this error may become insignificant).

[0171] Congestion control techniques well known in the art (Reno, Vegas, CUBIC, and newer techniques such as BBR) can be used to estimate the throughput and latency of a connection (and thus the bandwidth-delay product, or BDP, of the connection). It can be advantageous to dynamically use different congestion control techniques depending on the characteristics of individual (or pooled) connections and / or the requirements of the incoming data flow. Parametric combination of congestion control techniques for a single connection (or pool of connections) can also be advantageous, as it allows for techniques that can better fit the needs of the connection (or pooled connections) and / or the incoming data flow.

[0172] 6 adds the dimension of flow requests (size and sensitivity to latency). There may be different input devices 602 and 603, which may provide different types of data as shown (602 is an FTP request and 603 is an SSH session). These flows may require classification by gateway 604 and may advantageously be provided via connection 606 to receiving gateway 608 based on flow classification, and via SSH connection 612 connected to FTP server 610 and / or input device 2.

[0173] Thus, with appropriate identification and classification of flows, further refined allocation of data packets to data connections (or pools of connections) can be implemented. For example, if the gateway 604 is operable to identify a flow as DNS traffic, the gateway 604 can determine that the data stream is small and latency-sensitive, so that the gateway 604 keeps this data stream "sticky" to the connection or pool of connections with the lowest latency. Conversely, if the gateway 604 identifies the data stream as a large FTP transfer, the gateway 604 should be configured to first prioritize routing to connections with higher latency, and only overflow to lower latency connections or pools of connections when necessary and if excess capacity exists.

[0174] While the illustrated embodiment includes a traffic identifier in the first line, other solutions can use deep packet inspection, labeling of data packets or data streams to identify the type of data stream and ensure appropriate handling by the flow classification engine. Such an approach may aid in routing (e.g., SSH: use lower latency connections, HTTP: use higher latency connections). In one embodiment, some traffic types are always sticky (so a sequencer 162 is not needed on the receiving side).

[0175] In one embodiment, packet marking by the transmitter (e.g., via Differentiated Services Code Point (DSCP)) can be used to aid in flow identification / classification to further improve the allocation of packets to connections and data flows.

[0176] In another embodiment, heuristics based on observed behavior are used to infer the nature of the flow (e.g., given the unidirectional nature back to the initiator and the high bandwidth, gateway 604 may be configured to assume by default that this is an FTP flow and adjust connection allocations accordingly).

[0177] In another embodiment, the classification can change dynamically based on the behavior of the flow. For example, when an SSH flow first starts, it may be classified as a low-bandwidth, latency-sensitive flow. However, as the flow continues to transfer data, once it passes a certain threshold volume, its classification may change to a large data transfer that is not latency-sensitive (e.g., SCP / SFTP operation).

[0178] In FIG. 6, the flow classification engine is configured to determine (presumably) that the nature of the traffic from input device 1 (602) is FTP traffic, and in anticipation of a large flow of data that is not latency sensitive, the system 100 initially routes the traffic through the high BDP satellite connection, overflowing to the LTE connection if necessary.

[0179] The flow classification engine 156 may also be configured to determine that the flow from input device 2 (603) is an SSH session, and the gateway 604 may then be configured to select low latency connection 1 (or a pool of low latency connections) to handle the flow. The flow classification engine 156, in some embodiments, monitors and / or receives statistical information about the incoming data stream that is utilized to help identify the type of stream.

[0180] In one embodiment, system 100 combines its awareness of multiple flow types in real time to dynamically adjust the mix of connections or the mix of pools of connections used.

[0181] 7, system 100 incorporates a network characteristics monitoring unit 701 in an outgoing multipath gateway 704, an incoming multipath gateway 708, or both multipath gateways. Various factors are monitored and / or estimated by system 100 to control how data traffic is allocated to various connections.

[0182] These factors may be established, for example, by determining connection reliability, which is used to determine how to improve packet transmission (e.g., in some cases, overall reliability is more important than peak speed).

[0183] Connection reliability may be measured as a composite of one or more factors including, for example, bandwidth, latency, jitter, packet loss, and variability, among others.

[0184] In one example, the network characteristic monitoring unit 701 collects raw data such as throughput, packet loss, jitter, latency, packet ordering, etc., and calculates statistics on this collected data, typically but not limited to, mean, variance, kurtosis, order statistics, rank statistics, number of times above or below a pre-set / predetermined threshold, and other derived statistics.

[0185] If network characteristic monitoring units are present in both the receiver 704 and the transmitter 708, these network characteristic monitoring units may be configured to share their derived statistics (as shown by feedback 702) in real time or at predetermined intervals over one or more connections 706. For improved reliability, the shared derived statistics may be repeated over multiple connections (or a pool of connections) to increase reliability.

[0186] In one embodiment, the network characteristic monitoring unit 701 at the receiving side takes measurements of how many bits are corrected by forward error correction (FEC) for each packet sent on a particular connection 706. This information may be processed to provide insight into the error rate for the connection and is provided as part of the derived statistics.

[0187] 8A, 8B, 8C, 8D, and 8E are illustrations of bandwidth estimation methods according to some embodiments. Bandwidth estimation is provided to obtain a reasonably accurate estimate of the bottleneck bandwidth (bandwidth constraint) between a transmitter and a receiver without transmitting at a rate anywhere near the actual bottleneck bandwidth.

[0188] The estimated bandwidth is used along with other factors to determine an overall score or fitness level (e.g., network "goodness") for a given data connection. The system may be configured to determine the expected bandwidth of all available data connections (e.g., networks) in order to weight the transmission of packets to the various data connections.

[0189] 8A, as shown in diagram 800, the example provided involves a transmission burst of eight packets of 1400 bytes each from transmitter 802. In an ideal model of the network, the bottleneck link would transmit these eight packets at exactly the rate that would effectively insert an equal delay between each packet. In this example, the network is capable of transmitting at 25 Mbps, so it inserts a delay of 448 microseconds between each packet.

[0190] At the receiver 804, packets are received and timestamps are recorded at precise 448 microsecond intervals. The bandwidth estimation is based on the sum of the packet sizes (excluding the first packet) divided by the time required to transmit all packets. The size of the first packet is excluded because the time it spends in the network is unknown. However, the time it is received indicates the time when the second packet will begin its transmission from the bottleneck link.

[0191] As shown in Figure 8B, in practice, the transmitter, receiver, and intermediate routing systems may introduce errors into the bandwidth estimates due to competition with other traffic, variability in processing times, network interface card (NIC) optimizations such as interrupt arbitration, etc. Error compensation may be performed, for example, through the use of various statistical methods (e.g., mean, median, etc.). Error compensation may increase the efficiency and throughput of the system, and in some embodiments, may result in adaptive modification of routing characteristics.

[0192] Another technique for error compensation is to adaptively vary the number of packets in each burst depending on the most recent bandwidth estimate. Generally, the longer the burst, the more accurate the estimate, since only errors introduced in the inter-packet arrival time at the head and tail of the burst affect the bandwidth estimate calculation. The longer the burst, the more insignificant the head and tail errors become compared to the total burst duration. However, if the network has low bandwidth, long bursts may not be practical.

[0193] For clarity in the examples of Figures 8A, 8B, 8C, 8D, and 8E, the bursts contain eight equally sized packets, but the number of packets per burst and the size of each packet within a burst need not be fixed. Calculating the bandwidth estimate only requires knowing the total size of the packets and the total duration of the burst. However, excessively small packets and / or short bursts can magnify errors introduced at the head and tail of the burst.

[0194] In some embodiments, packets in a burst may also contain true data (essentially piggybacking metadata needed for the bandwidth estimation algorithm on top of the data being transmitted). The first packet (which is discarded) may be used for this metadata. For improved reliability, subsequent packets in the burst may repeat this metadata.

[0195] If piggybacking is not possible or there is insufficient real data to transmit, this burst may be transmitted with "dummy" data, but the burst length may be even shorter so that the burst interferes as little as possible with the actual data transmission.

[0196] Techniques may be used to partially compensate for some types of errors, such as techniques related to handling of NIC interrupts.

[0197] 8C illustrates an example timing diagram for a receiver in which NIC interrupts are modeled as being handled with significant delay, according to some embodiments. The method illustrated in FIG. 8C is utilized to compensate for latency and packet bursts resulting from NIC interrupts.

[0198] This example illustrates a decision to compensate for interrupt latency, as opposed to a simple bottleneck bandwidth estimation. These interrupt delays may be caused, for example, by a network link between the transmitter and receiver that creates a bottleneck in the received packets, resulting in packets being received in bursts as opposed to being evenly spaced as in the example of FIG. 8A.

[0199] An interrupt occurs when a packet is received, but is not executed until after a significant delay (500 microseconds in this example). During this delay, more packets can arrive, so when the handler finally executes, it collects all packets (the first packet that triggered the interrupt + any packets that arrived during the delay) and delivers them to the application simultaneously.

[0200] An error is introduced into this model because some packets in a burst are coalesced into groups, with all packets in the group being stamped with the same receipt time. This error can be corrected by observing that if the coalesced packets are "trimmed" from the head and tail of the burst (meaning they are excluded from the bandwidth estimate calculation), the resulting packet distribution pattern is identical to the non-delayed interrupt case. The only difference is that the effective duration of the burst has been further shortened by the trimming, and each of the packet receipt times has been shifted in time by an equivalent amount due to the interrupt delay.

[0201] Trimming from the head of the burst requires finding the first group that contains only a single packet, while trimming from the tail requires only ignoring all but the first packet in the last group of the burst.

[0202] Thus, in some embodiments, the network characteristic monitoring unit 701 tracks packet reception characteristics (e.g., by determining an estimate of the bottleneck bandwidth). As shown in Figure 8C, the network characteristic monitoring unit 701 of some embodiments is configured to further compensate for interrupt latency, for example, by trimming coalesced packets and incorporating a time shift.

[0203] FIG. 8D illustrates a technique that can be used to compensate for interrupt latency when all groups containing two or more packets do not allow trimming from the head of the burst.

[0204] This technique may be implemented, for example, by the network characteristic monitoring unit 701 to compensate for interrupt latency occurring at bottleneck links. Thus, the network characteristic monitoring unit 701 can adapt to interrupt latency without requiring that it be able to trim from the head of a burst.

[0205] The network characteristic monitoring unit 701 determines whether the reception time of a packet is equal to the transmission time of the subsequent packet from the bottleneck link (i.e., t(N-1) 受信 ==tN 送信 ) is configured to apply an idealized model of the bottleneck link. As can be seen from this example, this permutation (in this case, t2 送信 From t1 受信 It is clear that after performing (a) and (b), estimates can be calculated even without trimming the head of the burst.

[0206] FIG. 8E shows a receiver in which the NIC is modeled by the network characteristic monitoring unit 701 using an interrupt arbitration technique, which may mean that the interrupt handler wakes up on a fixed timer (every 1000 microseconds in this example).

[0207] Any packets received since the last execution of the handler are collected and delivered to the application.

[0208] This introduces error into the estimate, since the actual receipt time of the packet is no longer visible to the application, and the estimate is based on the time the interrupt occurs.

[0209] If the interrupt moderation interval is known (e.g., because it is a system constant or perhaps measured in real time based on the frequency at which groups of packets arrive at the application), upper and lower bounds can be calculated for the bandwidth estimate. Thus, in some embodiments, the network characteristic monitoring unit 701 is configured to maintain a data structure representing the most recent estimated interrupt moderation interval as obtained from a corpus of tracked data values, and this data structure is utilized to estimate the interrupt interval, including the upper and lower bounds for the bandwidth estimate.

[0210] The upper limit is calculated assuming that the last group in the burst is actually received immediately after the second final interrupt, and the lower limit is calculated assuming that the last group is received just before the final interrupt.

[0211] In one embodiment, the floor value is the threshold at which FEC begins to be generated and transmitted on the connection.

[0212] On some systems, despite interrupt coalescing, the actual receipt time of each packet can be made available to applications by the NIC or operating system kernel. On these systems, explicit compensation for errors due to interrupt coalescing is not required.

[0213] The network characteristic monitoring unit 701 is configured in some embodiments to utilize lower and upper bounds in determining packet reception characteristics for reconstruction.

[0214] 9 is a diagram 900 configured to show a simple example (single input) of satellite offload between Gateway A 902 and Gateway B 908. Dashed lines represent logical connections between components, while physical connections represent the flow through the combined links.

[0215] Consider an example where there is one network (e.g., satellite or microwave) that the traffic manager 904 can control (can control who is on the network and who gets priority bandwidth, can adjust network parameters, etc.), and other networks (e.g., third-party cellular networks) that the traffic manager 904 does not control. Consider further that the controlled network may have certain characteristics (e.g., bandwidth is purchased monthly in fixed "chunks" because it has already been paid for) that make it advantageous to ensure that it is "filled" with traffic (to reduce costs or maximize revenue), even if that means offloading some of the traffic to other networks.

[0216] An example is transmitting video, where it is desired to deliver a certain level of transmission reliability that cannot be guaranteed or even provided by the cellular network. In this case, it may be desirable to transmit as much of the video over the satellite network, since it is necessary to guarantee the target level of reliability. Also, if the satellite connections are not at full capacity (assuming one or more gateways are transmitting simultaneously), it may be advantageous to use as many satellite connections as possible to reduce the incremental cost of using additional cellular data. Note that other data combined with the video, such as FEC information or metadata or the actual portion of the video stream, may also be transmitted over the combined connection. In another embodiment, it may be advantageous to offload data from satellite to cellular under control of the network blender / traffic manager 904, such that gateways with good cellular bandwidth are instructed to use less satellite bandwidth.

[0217] The Network Blender / Traffic Manager 904 is configured to determine the optimal combination of networks (in some embodiments, more than one network is controlled, in which case the Network Blender / Traffic Manager 904 is still useful as it provides oversight for all networks and routes based on real-time measurements of throughput, reliability, packet loss, etc.). In one embodiment, one or more available network connections are "controlled" (as described above), and the Network Blender / Traffic Manager 904 is configured to receive real-time information from the gateways and control the bandwidth used by the gateways ("edge nodes") so that the entire network is managed by the Network Blender / Traffic Manager 904.

[0218] In one embodiment, the controlled network management system (CNMS) 906 may have a set of logical operations (e.g., static logical operations, dynamic logical operations) to determine admission of certain traffic under certain conditions (e.g., with respect to user or traffic type, cost-based priority, reliability, etc.) The set of logical operations may be based on what is provided in various data stores and stored in the gateway, source / endpoint, or cloud.

[0219] In some embodiments, a set of logical operations is also stored in the traffic manager 904, and the traffic manager 904 may utilize logical operations from the controlled network management system 906 in combination with other logical operations that apply to the entire transmission (e.g., a user may not have to pay as much for a controlled satellite and may be intentionally pushed off a controlled satellite, but may pay for the entire transmission and therefore receive priority for the rest of the data connections in the system 100 so that there may be preferential treatment regarding the selection or allocation of higher cost / reliability non-controlled networks).

[0220] In one embodiment, the network blender / traffic manager 904 may interact with the CNMS 906 to inform the system, the multipath gateway 902, or devices connected to the multipath gateway regarding general network conditions so that the system 100 can be configured to allow the network blender / traffic manager 904 to better utilize both the controlled network and any other transmissions that the network blender / traffic manager 904 may manage.

[0221] For example, multiple connections may be available, and network conditions affecting each of the connections may vary over time. For example, congestion, loss, and noise may be experienced by each of the connections, and these connections may be advantageously assigned to uses or functions based, for example, on their connection characteristics. Such "binding" of connections and assignment of functions is described in applicant's U.S. Patent No. 9,357,427, which is incorporated by reference.

[0222] FIG. 10 is a diagram 1000 that provides an example of a hybrid where there are multiple gateways connected to multiple controlled and uncontrolled networks.

[0223] The dashed lines represent logical connections between components, and the physical connections represent the flow through the coupled links.

[0224] There may be N multipath gateways (e.g., 1002, 1003), each working in concert with the network blender / traffic manager 1004 and CNMS 1006 to ensure that flows from multiple endpoints through the gateways are provided with the best quality of experience, taking into account factors such as contention for resources on the controlled networks (under the control of 1004 and 1006), the service level purchased by the owners of these gateways, the quality and availability of connectivity on uncontrolled networks attached to these gateways, and the requirements of the flows and applications generating traffic through these gateways. Transmissions may ultimately be provided to multipath gateways AA-NN (e.g., 1008, 1010). These gateways provide combined functionality with a known and controlled quality of service for each gateway on the network. In this context, quality of service refers to the allocation of bandwidth for specific applications or classes of applications through smart management of bandwidth provided by the network blender / traffic manager 1004 and CNMS 1006.

[0225] These gateways may be sold (or guaranteed) with different levels of connection reliability. If a gateway is guaranteed (or it dynamically requests) high reliability and it is not receiving high reliability, the network blender / traffic manager 1004 may be configured to provision (e.g., "steal") more reliable (e.g., satellite) bandwidth from another gateway and grant this increased reliability to the gateway that requested (or was guaranteed) higher reliability.

[0226] The Network Blender / Traffic Manager 1004 manages the allocation of traffic through controlled networks from multiple gateways, optimizing the set of transmissions based on real-time measurements of network characteristics (provided by the gateways) and configured static or dynamic logic (quality of service rules).

[0227] In one embodiment, the network blender / traffic manager 1004 may be specialized to manage traffic for a particular user / organization. In another embodiment, the network blender / traffic manager 1004 may be configured to provide services to a wide variety of users. Where multiple network blenders / traffic managers 1004 may exist, their resources may communicate with each other as needed and be provisioned as needed, where resources are provisioned / deprovisioned (“spun up or down”) as needed. The network blender / traffic manager 1004 and the controlled network management system 1006 may, in some embodiments, be provided in the form of distributed network resources (e.g., run “in the cloud” in a cloud configuration).

[0228] Dynamic logic operations may be provided to address how the controlled network is utilized when a new transmission through the controlled network occurs (e.g., the priority given to any particular transmission may depend on other transmissions in progress).

[0229] Figure 11 is a diagram 1100 provided to illustrate a one-to-one scenario configured for uncoupled disaster recovery / load balancing applications. In this situation, individual flows may remain sticky to a particular connection without any coupling; here, there is no second gateway because no buffering is required at the receiving end. The algorithm for allocating flows to connections may be based on static or dynamic rules that take into account factors such as flow requirements, connection quality / reliability, transmission costs, and computational limitations on the gateway.

[0230] In one embodiment, the gateway 1102 is configured to add or remove connection links as needed (e.g., depending on throughput or traffic type) that set up, rather than bond, individual sockets between two endpoints of a communication.

[0231] In some embodiments, the connection may be coupled but not carry any data; such an implementation may provide technical advantages in failover scenarios, where it may be necessary to know that the connection is up and running; activating the connection allows the system to periodically test the connection to ensure it is working, and / or move the data stream (or flow) through a new connection if the need arises.

[0232] In one embodiment, certain types of traffic from a site experiencing a disaster recovery scenario may be configured to be routed over a particular backup link (or set of links) to reduce costs, increase security, minimize latency, jitter, or any combination of such factors, etc.

[0233] For example, there may be uses for the system for disaster recovery as a service tool (or adaptive load balancer) that does not require routing through a receiver / second gateway, where embodiments of the above solution may be utilized to provide more robust connectivity.

[0234] FIG. 12 is a diagram 1200 provided to illustrate a sample combined disaster recovery / load balancing application, where traffic is combined and routed through a second gateway 1204 according to some embodiments.

[0235] As shown in Figure 12, in one embodiment, a large organization (or an organization needing security) may have two dedicated gateways 1202 and 1204 in a (relatively) closed system. In another embodiment, a customer may simply be assigned a public gateway 1204 "in the cloud." The implementation of Figure 12 functions to provide high security because the use of tiered gateways isolates the network from potentially malicious attacks from the Internet and provides a collaborative part for enforcing security policies and protocols, including quality of service, for incoming and outgoing data streams (or flows).

[0236] FIG. 13 is a diagram 1300 illustrating a series of multipath gateways 1304, 1306, and 1308 connecting two or more endpoints 1302 and 1310. FIG. 13 illustrates a scenario in which a field operator at Endpoint 1 (1302) needs the benefit of having multiple paths for transmission (e.g., soldiers at remote locations transmit video and other data that provides situational awareness information to headquarters). A combination of multipath gateways (e.g., gateways 1304, 1306, and 1308) may be required to accomplish this. For example, the first hop of transit may require using multiple WiFi networks to send data from the field to a nearby forward operating base. At the forward operating base, a gateway may then transmit the data over a different set of communication links (e.g., multiple cellular networks) to a national headquarters, where another gateway may then use one or more fiber links to send the data to the national headquarters, which may then use a different combination of links to send the data back to a different field location.

[0237] In one embodiment, the data is available to applications at one or more stops along the transit, while in another embodiment, the data is transmitted end-to-end without being accessed by applications along the way.

[0238] In some embodiments, a system configured for managing multiple start / end points using overlapping pools of networks may be provided, where an intelligent engine uses knowledge of the activity of various devices and various networks to better manage the flow of transmissions. In such implementations, the system may be configured to perform sequencing or manage network usage according to data residency logic operations (e.g., logic rules).

[0239] 14 is a diagram 1400 illustrating two sites connected to the Internet through multiple links using a typical transit and / or peering configuration involving Border Gateway Protocol (BGP)-based edge routers. In this example, Site A advertises a BGP route for Subnet A to its upstream peer / transit provider, and Site B advertises a BGP route for Subnet B.

[0240] BGP is a "coarse-grained" distance-vector routing protocol, which means that the primary factor in how it decides to route a packet is based on the number of autonomous systems (AS) the packet traverses to reach the destination subnet. It does not consider congestion within each AS, nor does it natively support multihoming.

[0241] For example, if network path Transit A1 is considered the shortest path for traffic from Subnet B to Subnet A, then all inbound traffic to Subnet A will arrive via Transit A1, even if it is congested. All other links (Transit A2, Peer A1, Peer A2, etc.) will be unused.

[0242] A technique to address this limitation is to partition subnet A into smaller subnets and selectively advertise them over a subset of links. This is suboptimal because the traffic patterns in each of the smaller subnets are likely not to represent the desired partitioning within a given link at all times. Smaller subnets also contribute to an increase in global BGP table size, but generally speaking, subnets smaller than / 24 are not accepted in the global BGP table.

[0243] While the comparison with BGP routing between two sites is used as an example in this diagram, this is not a requirement. The two sites may simply be multihomed through different providers that do not have publicly advertised subnets. In such a situation, a fully connected graph between the two multipath gateways is still established, with traffic between the two sites being routed over the connection according to any of the previously mentioned criteria (e.g., connection cost, flow demand, connection quality / reliability, etc.).

[0244] 15 includes a diagram 1500 showing the same two sites connected to two multipath gateway systems 100. The connection controller 154 forms a fully connected graph connecting transit A1 to all of (transit B1, transit B2, peer B1), transit A2 to all of (transit B1, transit B2, peer B1), and so on.

[0245] Flows generated by hosts and applications on subnets A and B are no longer routed based on advertised BGP routes, but instead are encapsulated within a protocol between the two multipath systems 104 and 108 that communicate directly via IP addresses on transit and / or peering networks.

[0246] This allows the multipath system to route packets based on any number of the above criteria (e.g., latency and throughput requirements of each flow, detected congestion / contention, jitter, cost, administrator policy, etc.).

[0247] In some embodiments, one or more aspects of the systems and / or methods described herein may seek to maximize the throughput of the aggregate flow of transmissions over multiple links while satisfying all latency and packet loss constraints for each transmission.

[0248] In some cases, some of these issues may be addressed by managing buffer bloat, which is latency caused by excessive buffering on a link, which can result in packet delay jitter and increased latency.

[0249] When a connection is overloaded, latency increases on the overloaded connection, which increases the bandwidth-delay product (BDP), but without a corresponding increase in throughput.

[0250] In some embodiments, managing buffer bloat may require selecting buffer sizes that reflect the BDP of various connection types, such as:

number

number

[0251] throughput i (t) and RtProp i For simple connections where (t) does not vary over time or varies very slowly, maximizing total throughput while still satisfying the latency constraint is All RtProps i There exists a subset of connections where (t) is <= the latency requirement of the low latency flow, and · Pass through at least one of the subsets [k i ×bdp i (t)] is >= the volume of data generated by the low-latency flow, This can be resolved as long as the following conditions are met.

[0252] If there is a subset that satisfies this criterion, then [k i ×bdp i (t)] (and including the partial BDP of the connection that is not completely consumed by the low-latency flow) can be used for the non-latency-sensitive flow.

[0253] The maximization problem is throughput i (t) and RtProp i (t) increases rapidly, and this system i This becomes apparent when there are connections that vary over such a large range of values ​​that they cannot accurately follow the instantaneous value of (t). For these types of connections, the system uses bdp_min i (t) and bdp_max i (t) (Throughput i and RtProp i The oscilloscope can be configured to measure / observe the following:

[0254] throughput i (t) and RtProp i In the case where (t) varies, the observed throughput i (t) and RtProp i Statistical methods based on the distribution of (t) can be used. In some cases, this may rely on conditional probability or iterative heuristics. Machine learning or artificial intelligence methods may also be used.

[0255] The system is configured to address a maximization problem when determining the optimal split of connections while satisfying latency constraints. There may be multiple subsets of connections that satisfy these constraints. Low latency subset RtProp_min i Find all connections where (t) <= the latency requirement of the low latency flow, and bdp_min iFind all subsets through these connections where the sum of (t) is >= the volume of data generated by the low latency flows High Latency Subset For each feasible subset calculated above on the remaining connections (connections not in the subset), bdp_max i (t), the goal is to find the subset that maximizes this sum.

[0256] k i Decision In a simple case, k i is a constant, so we choose the buffer size for a connection as some fraction of the BDP for the connection. i We have found that a value of =1.2 works well in practice (eg, our system is configured to buffer over 20% of the BDP of a given link).

[0257] In complex cases, k i is the type of flow that travels through connection i (and any constraints on those flow types), bdp i (t) variance (k i This may be calculated in real time and used to adjust k in real time), and the connection characteristics (latency, jitter, throughput), which determine how much latency is needed for the types of flows and how much throughput is needed (and what combination is minimal and acceptable for those types of flows), which is k i This means that is a function of the flow type, BDP distribution and connection characteristics.

number

[0258] With WiFi and LTE connections, latency variations can be quite dramatic and can cover a range spanning several orders of magnitude (e.g., WiFi can vary between <10 ms and >100 ms).

[0259] The amount of buffering chosen results in a latency of <10ms (i.e., a small k i ), whenever the actual latency is >100ms, there is insufficient data available to keep the network full, resulting in suboptimal average throughput.

[0260] The amount of buffering is >100ms latency (i.e., large k i ), it is possible to keep the network full or close to full at all times (which gives the best average throughput), but there will be additional latency due to buffering (buffer bloat) whenever the actual latency is <10ms.

[0261] In this example, if there are flows that require low latency, the system compromises and reduces buffering on the connection used for low latency flow transmission to a level that assumes a latency of <10 ms (i.e., k i and bdp_min i (conservative / small value for t). In some cases, this may result in a lower average throughput overall, but it does not introduce buffer bloat that would impede the demands of the flows. The system can be configured to converse on the remaining connections for flows that require high throughput and are not latency sensitive.

[0262] The above discussion provides many exemplary embodiments of the inventive subject matter. While each embodiment represents a single combination of inventive elements, the inventive subject matter is considered to include all possible combinations of the elements disclosed herein. Thus, if one embodiment includes elements A, B, and C, and a second embodiment includes elements B and D, the inventive subject matter is considered to include any other remaining combinations of A, B, C, or D, even if not explicitly disclosed.

[0263] The device, system, and method embodiments described herein may be implemented in a combination of both hardware and software. These embodiments may be implemented in programmable computers, each computer comprising at least one processor, a data storage system (including volatile or non-volatile memory or other data storage elements, or a combination thereof), and at least one communication interface.

[0264] Program code is applied to input data to perform the functions described herein and generate output information that is applied to one or more output devices. In some embodiments, the communication interface may be a network communication interface. In embodiments where elements may be combined, the communication interface may be a software communication interface, such as an interface for inter-process communication. In still other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combinations thereof.

[0265] The technical solutions of the embodiments may be in the form of a software product. The software product may be stored in a non-volatile or non-transitory storage medium, which may be a compact disc read-only memory (CD-ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that can cause a computer device (personal computer, server, or network device) to perform the method provided by the embodiments.

[0266] A network gateway for controlling data flow through a network connection is described herein, along with corresponding methods, apparatus, and computer-readable media. The gateway may, in some embodiments, be a physical hardware device comprising components such as a processor, interfaces, buses, power supplies, memory (ROM, RAM, Flash), etc., configured to receive and route data packets according to an instruction set representative of software and embedded firmware in the form of program code or computer logic.

[0267] The program code and / or computer logic may include control logic and devices, which in some embodiments may operate as controllers, routers, switches, access points, etc., and said devices may be configured to operate in a manner that is transparent to the end user (apart from possible performance improvements, especially since large amounts of data pass through said devices).

[0268] The network gateway may, in one example, be a specialized computing device specifically optimized for joining TCP connections as described in some examples herein. In other examples, the network gateway may be implemented using a processor and / or other computing hardware as part of a larger system. The network gateway may be a single device, or may be multiple devices, possibly operating in concert.

[0269] The embodiments described herein are implemented by physical computer hardware including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and specifically configured computer hardware configurations. The embodiments described herein relate to electronic machines and methods performed by electronic machines suitable for processing and transforming electromagnetic signals representing various information.

[0270] The embodiments described herein broadly and integrally relate to machines and their use, and the embodiments described herein have no meaning or practical applicability other than their use with computer hardware, machines, and various hardware components. For example, substituting physical hardware configured to perform various operations for non-physical hardware, particularly with mental steps, may substantially affect the way the embodiments operate. Such computer hardware limitations are clearly essential elements of the embodiments described herein, and they cannot be omitted or replaced with mental means without significantly affecting the operation and structure of the embodiments described herein. Computer hardware is essential to implementing the various embodiments described herein, and is not simply used to perform steps quickly and efficiently.

[0271] Although the embodiments have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the scope thereof.

[0272] Moreover, the scope of the present application is not limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods, and steps described in the specification. As one skilled in the art will readily understand from this disclosure, any now existing or later developed process, machine, manufacture, composition of matter, means, methods, or steps may be utilized which perform substantially the same function or achieve substantially the same results as the corresponding embodiments described herein. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0273] As can be understood, the examples described and illustrated above are intended to be illustrative only.

Claims

1. 1. A network gateway for routing data flows through a plurality of network connections, the network gateway comprising: a plurality of network interfaces for transmitting data over said plurality of network connections; at least one processor, monitoring time-varying network transmission characteristics of the plurality of network connections; analyzing at least one packet of a data flow of packets to identify a data flow class for the data flow, the data flow class being associated with at least one network interface requirement of the data flow; routing packets within the data flow through the plurality of network connections based on the data flow class and the time-varying network transmission characteristics; transmitting a sequential burst of packets through a first network interface of the plurality of network interfaces; generating an estimate of a bandwidth of the first network interface based on timestamps recorded when packets in the sequential burst of packets are received at a receiving node and sizes of the packets; routing packets through the plurality of network connections based on the generated bandwidth estimate of the first network interface; a processor configured for Equipped with generating an estimate of a bandwidth of the first network interface when the receiving node processes received packets at periodic intervals; generating a lower bound bandwidth value by using a reception timestamp of a packet in the sequential burst selected as an end packet for the bandwidth estimate; generating an upper limit bandwidth value by replacing a reception timestamp of a packet in the sequential burst selected as the end packet with a reception timestamp of a packet in the sequential burst preceding the end packet; Network gateways, including:

2. 2. The network gateway of claim 1, wherein monitoring the time-varying network transmission characteristics includes generating a bandwidth-delay product of at least one network interface of the plurality of network interfaces based on the monitored time-varying network transmission characteristics, and routing packets in the data flow is based on the bandwidth-delay product of the at least one network interface.

3. The at least one processor analyzing at least one packet of each of a plurality of data flows of packets; and routing packets in each of the plurality of data flows based on a data flow class of each of the plurality of data flows and an available bandwidth of the network connection corresponding to the data flow class of each of the plurality of data flows; 10. The network gateway of claim 1, configured to:

4. 2. The network gateway of claim 1, wherein generating an estimate of the bandwidth of the first network interface comprises dividing a sum of packet sizes of packets between a first packet in the sequential burst and a second packet in the sequential burst by a time elapsed between a timestamp for the first packet and a timestamp for the second packet, wherein the first packet is not a first packet in the sequential burst and the second packet is after the first packet in the sequential burst.

5. 2. The network gateway of claim 1, wherein generating an estimate of the bandwidth of the first network interface comprises generating the estimate of the bandwidth based on timestamps of packets in the sequential burst that are not coalesced with a first or last packet in the sequential burst.

6. The at least one processor for each packet in the data flow, providing the packet for routing through one of the plurality of network connections based on the monitored latencies of the plurality of network connections and network connections of other packets in the data flow such that the packets in the data flow arrive at a destination node in a desired sequence.

10. The network gateway of claim 1, configured for:

7. The network gateway of claim 6 , wherein the desired sequence is the original sequence of packets in the data flow.

8. 7. The network gateway of claim 6, wherein the desired sequence is a sequence that includes at least one misordering of packets that does not trigger a retransmission of packets within the sequence.

9. The at least one processor receiving a packet from a source interface for routing to a destination node via the plurality of network connections; sending an acknowledgment to the source interface before routing the packet to the destination node; storing the packet in at least one buffer before routing the packet to the destination node; 10. The network gateway of claim 1, configured for:

10. 10. The network gateway of claim 9, wherein the at least one processor is configured for dynamically controlling a size of the at least one buffer based on a bandwidth-delay product associated with the plurality of network connections.

11. 10. The network gateway of claim 9, wherein the at least one processor is configured to control sending and storing of the packet acknowledgments based on monitoring transmission characteristics of the plurality of network connections and an unequal distribution in receiving sequential packet data flows.

12. 2. The network gateway of claim 1, wherein the at least one processor is configured to route the packets based on at least one of an estimate of bandwidth of the plurality of network connections and a reduction in the number of network connections through which the data flow is routed.

13. 4. The network gateway of claim 3, wherein the at least one processor is configured to group packets of the plurality of data flows having similar data flow classes and route the grouped packets through the plurality of network connections based on the classification of the plurality of data flows.

14. The network gateway of claim 1 , wherein the data flow class of one of the plurality of data flows changes automatically when a threshold volume of data for the corresponding data flow is routed.

15. 10. The network gateway of claim 1, wherein the packet data flow is data packets containing at least one of video and audio data.

16. 10. The network gateway of claim 1, wherein the sequential burst of packets includes both test packets utilized to make bandwidth estimate determinations and data packets including at least one of video and audio data.

17. 17. The network gateway of claim 16, wherein the data packets containing at least one of the video and audio data transmitted through the sequential bursts of packets are redundant data packets configured to withstand a quantity of lost data packets.

18. 1. A method for routing a data flow through a plurality of network interfaces for transmitting data over a plurality of network connections, the method comprising: monitoring time-varying network transmission characteristics of the plurality of network connections; analyzing at least one packet of a data flow of packets to identify a data flow class for the data flow, the data flow class being associated with at least one network interface requirement of the data flow; routing packets within the data flow through the plurality of network connections based on the data flow class and the time-varying network transmission characteristics; transmitting a sequential burst of packets through a first network interface of the plurality of network interfaces; generating an estimate of a bandwidth of the first network interface based on timestamps recorded when packets in the sequential burst of packets are received at a receiving node and sizes of the packets; routing packets through the plurality of network connections based on the generated bandwidth estimate of the first network interface; Including, generating an estimate of a bandwidth of the first network interface when the receiving node processes received packets at periodic intervals; generating a lower bound bandwidth value by using a reception timestamp of a packet in the sequential burst selected as an end packet for the bandwidth estimate; generating an upper limit bandwidth value by replacing a reception timestamp of a packet in the sequential burst selected as the end packet with a reception timestamp of a packet in the sequential burst preceding the end packet; A method comprising:

19. 20. The method of claim 18, wherein monitoring the time-varying network transmission characteristics includes generating a bandwidth-delay product of at least one network interface of the plurality of network interfaces based on the monitored time-varying network transmission characteristics, and routing packets in the data flow is based on the bandwidth-delay product of the at least one network interface.

20. analyzing at least one packet of each of a plurality of data flows of packets; routing packets in each of the plurality of data flows based on a data flow class of each of the plurality of data flows and an available bandwidth of the network connection corresponding to the data flow class of each of the plurality of data flows; 20. The method of claim 18, comprising:

21. 20. The method of claim 18, wherein generating an estimate of the bandwidth of the first network interface comprises dividing a sum of packet sizes of packets between a first packet in the sequential burst and a second packet in the sequential burst by a time elapsed between a timestamp for the first packet and a timestamp for the second packet, wherein the first packet is not a first packet in the sequential burst and the second packet is after the first packet in the sequential burst.

22. 20. The method of claim 18, wherein generating an estimate of the bandwidth of the first network interface includes generating the estimate of the bandwidth based on timestamps of packets in the sequential burst that are not merged with a first or last packet in the sequential burst.

23. for each packet in the data flow, providing the packet for routing through one of the plurality of network connections based on the monitored latencies of the plurality of network connections and network connections of other packets in the data flow such that the packets in the data flow arrive at a destination node in a desired sequence.

20. The method of claim 18, comprising:

24. 24. The method of claim 23, wherein the desired sequence is the original sequence of packets in the data flow.

25. 24. The method of claim 23, wherein the desired sequence is a sequence that includes at least one misordering of packets that does not trigger a retransmission of packets within the sequence.

26. receiving a packet from a source interface for routing to a destination node via the plurality of network connections; sending an acknowledgment to the source interface before routing the packet to the destination node; storing the packet in at least one buffer before routing the packet to the destination node; 20. The method of claim 18, comprising:

27. 27. The method of claim 26, comprising dynamically controlling a size of the at least one buffer based on bandwidth-delay products associated with the plurality of network connections.

28. 27. The method of claim 26, comprising controlling the sending and storing of the packet acknowledgments based on monitoring transmission characteristics of the plurality of network connections and unequal distribution in the reception of sequential packet data flows.

29. 20. The method of claim 18, comprising routing the packets based on at least one of a bandwidth of the plurality of network connections and a reduction in the number of network connections through which the data flow is routed.

30. 21. The method of claim 20, comprising grouping packets of the plurality of data flows having similar data flow classes, and routing the grouped packets through the plurality of network connections based on the classification of the plurality of data flows.

31. 20. The method of claim 18, wherein the data flow class of one of the plurality of data flows changes automatically when a threshold volume of data for the corresponding data flow is routed.

32. 20. The method of claim 18, wherein the packet data flow is data packets containing at least one of video and audio data.

33. 20. The method of claim 18, wherein the sequential burst of packets includes both test packets utilized to make a bandwidth estimate determination and data packets including at least one of video and audio data.

34. 34. The method of claim 33, wherein the data packets containing at least one of the video and audio data transmitted through the sequential bursts of packets are redundant data packets configured to withstand a quantity of lost data packets.

35. A computer readable medium storing a set of machine-interpretable instructions that, when executed, cause one or more processors to perform the method of any one of claims 18 to 34.

Images